EP 406: Boosting Performance - Azure's Proprietary Data Center Chips Unveiled

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

As soon as you think you knew what was happening with GPUs, then there's something called NPUs. And today at Microsoft Ignite here in Chicago, a lot of us got our first taste of DPUs. It's like you can hardly keep up. But here's the thing. With what Microsoft just announced today at the Microsoft Ignite conference, I think things are going to be getting probably cheaper, faster and more secure. It's like that triangle of three things we all want, but you can never really get. Don't worry, I brought in someone much smarter than me today to help us talk about that. So please help me welcome Alistair Spears, the Senior Director of Azure Infrastructure at Microsoft. Thank you so much for joining us.

Thanks, Jordan.

All right, cool. So can you tell us a little bit first about what you do at Microsoft? What is your role?

Yeah, so I work on the global infrastructure team and our job is really to manage the data centers growing capacity, the worldwide footprint of our cloud infrastructure from a macro lens of like, where to build the power, we need, the market kind of forces that are driving like the relationships with App Intel's, AMD's, Nvidia's and building up that infrastructure and then of course, what goes inside the server as well. So that is the server architectures and some of our own first party silicon development work that we do as well. Yeah.

Can you tell us a little bit what was announced today? Because there was a lot, you know, there was chips, there was HSMs, DPUs. There was so much announced today. I think they said it was like more than 90 different, like AI announcements. And I was trying to keep up. But at least on your side, what was actually announced, you know, with these new GPU chips and the HSM chip.

Yeah, I think, you know, and Satya said it really interestingly, the middle innings of this era of AI.

Right.

And so it was almost a progress report. And all the work that we're doing to optimize this infrastructure and optimize the capabilities to deliver AI to more and more people. So if I think of like optimization, you know, every layer of the stack, it's not just what we're doing in software software, it's not just what we're doing in like data center facilities, but the hardware and then the specific types of hardware that make up this data center kind of platform. We talked about optimizing specific ones there as well. So first the Azure Boost dpu. So DPU is a special type of processor or what we call an ASIC or application specific integrated circuit. And this chip is designed to do one thing and one thing well, and that's storage operations. So you can think of it as taking the place of a cpu, a network card, other infrastructure that goes on in a traditional server, replaces it with one card that just has one job, which is to kind of read data from, from hard drives and push it out over the network. And by doing it and by integrating that into this really single purpose chip, we're able to get a lot more efficiencies out of that chip, use less power to run that operation as well.

So could you kind of help put the last couple of years into perspective? Right, because I think when the, you know, the chat GPT wave, you know, of 2022 came and you know, then we've had, you know, Copilot now generally available for more than a year, but was it kind of like you just had chips doing all different jobs and maybe they weren't too efficient? And that's kind of why. And what's led us to now having these NPUs for edge AI in the DPUs is that kind of what's happening?

Essentially you're seeing a lot more efficiencies out of this. You know, the early models were run brute force infrastructure with CPUs, GPUs, whatever we can get our hands on. Like let's run these models and try and train and infer these AI models on that infrastructure. But you know, over time as we optimize like the models a little bit more and get better at the software, as new chips become available, new capabilities become available, we start to fine tune the infrastructure to really focus on this kind of emerging category of AI type apps. And that's really what you're seeing here with new GPUs that a graphics processing unit, or the full name was gpgpu, a general purpose graphic processing unit. Right now they're getting more and more specialized in just that AI acceleration workload as well. So they're almost becoming more fit for purpose rather than these general purpose computing devices are really targeted to particular use cases. And then if you think about the scale we operate in in the Microsoft cloud, as that infrastructure gets bigger, as the scale gets bigger, then there's more opportunity to optimize for kind of a very targeted workload like AI or like storage or like security as well.

Okay. And for the dorks of us out there, I'M a little dorky, but you know, what are some of the specs? Like as an example on the DPU I think I read, is it like 4x more efficient, something like that.

So you know, this DPU is really looking to run at about 100 watts per server. And you think about that, the traditional data center server with like a traditional CPU, traditional network card is probably drawing about 400 watts. And so you know, forex reduction in power is super important for us, especially in this world of like GPUs and AI accelerators doing more and more power draw as they grow as well.

That's, that's a big thing to think about because I think there's a couple of things. So you have your, you know, your bigger companies that have to worry about where are they getting their power for AI, right? And then there's also the even bigger picture, right, the environmental impact. So how does, you know some of what's announced that was announced today at Ignite go to address both of those things. So how is this going to impact companies having access to DPU's that are four times more energy efficient? And then what does that mean for the bigger picture, like the environment?

That's a great point. Like the cloud's always been designed on this like economy of scale efficiencies. And so working in a cloud environment is generally far more energy efficient than running like an on premises server. And we're building like fit for purpose buildings that are really designed for servers, for data centers and for these cloud workloads. So as you think about like building something like AI or an Azure service or M365 or something like that, it's not just a software problem anymore, it's a software problem. The software engineers need to talk to the hardware engineers, the hardware engineers need to talk to the power engineers, the data engineers, construction engineers, even the sustainability engineers to understand like the full footprint of what they need. And so as we build all this new infrastructure, optimizing the power draw particular servers is important. Optimizing the power sources that we use. We sourced about 34 gigawatts of renewable energy around the world through power purchase agreements. So that's essential, essentially securing renewable energy to continue to run these operations in the future. So kind of forward planning that and targeting those, those, those contracts are important as well. And then of course if you, you know, zero out your carbon emissions from power, what's left is carbon emissions from construction, building materials, server materials as well. So by building our own infrastructure and servers, we're able to improve our recycling rates, we're able to define, like, the types of components that we use in that device, reduce landfill by reusing these components rather than, rather than throwing them out. And then on the data center construction side, we have opportunities for lower carbon intensive building materials as well. And one of the ones that was talked about today was cross laminated timber. And this sounds like a crazy idea, but building a data center out of wood. And it turns out like wood has been used in construction for a long time. But the innovation with cross laminated timber is to essentially build these wider planks of layered timber that's laminated together and is essentially much lighter, much easier to work with, and has less embodied carbon than, say, a concrete or steel construction. Well, so having like lighter flame, lighter frames that are as strong as, as a steel or concrete construction means that you need less concrete for the foundation, less easy to move, less equipment, less logistics costs. And so all of those things add into essentially what we call our Scope 3 emissions. So that's the emissions caused by our broader operations as well.

Yeah, y' all are in for a treat because you thought you were tuning in to learn about, you know, Azure and GPU chips, which we are. But I think the environmental piece is huge because I think sometimes big tech gets a bad rap, right? They're like, oh, you know, AI. And it's so costly and inefficient and all these things. But I mean, here you are just explaining, even going down to the materials that you use to build these centers.

That is the system, Right. When we think about the system, it's not a software system. It's not a software plus hardware system. It is an ecosystem that guides down all the way from the construction and then of course, the water, the power usage, and how that infrastructure runs in its environment, in the built environment, in the natural environment as well.

Yeah. Okay, so let's, let's maybe take it, take it back, back to the office, back to. Yeah. So I mean, what is this ultimately going to mean, number one? Number two, when our customer is going to be able to get their hands on things that were announced today, or did they already? And then, you know, both. I want to look at it through the scope of, you know, the more technical people a little bit. Right. Even though that's not always our audience. But, you know, what does it mean for people in the IT department for big enterprise companies? And then what does it mean for everyone else? Are they going to be noticing all of a sudden they're working with, you know, everything in Azure is much faster, like overnight or how is this rollout going to work for those different groups?

Yeah, I mean the general promise is it just gets better. That's, I think that what we're really trying to do here. So as we do these optimizations at kind of the component level and other things, they're sort of out of sight. Right. They're running in a data center somewhere else. So you don't necessarily see the change in like architecture or server design or rack design, but you'll see the benefits. You'll see the services being faster, the services being more integrated, the services being more cost effective to run as well. On the other side, being able to control our energy draw and being more optimized means that essentially we're placing less burden on the grids that we operate in as well. So that has a benefit for every consumer of energy on a particular energy grid as well.

Yeah, the energy thing. Yeah, because I think we just shared in our newsletter last week something like there's going to be 4, 40% of data centers were going to be facing power struggles by 2025 and you know, here we are at the end of 2024. So this was timely news, right? Yeah.

And you know, for us like long term sourcing of that power is really important to us as well. We want to make sure we have stable operations for years to come. And so providing like a demand signal to our utility providers, it's really important. And so these long term agreements use some of them five years, some of them 10 years, some of them even longer, help us kind of guarantee a certain amount of new energy is joining the grid so that when as we need it, we have the energy there and we don't become like a burden on the overall energy grids themselves. Yeah.

And you know what, this is helpful for me too as a non technical person. I was, I was sitting in the keynote today or you know, by the time that everyone hears this yesterday and you know, everyone was clapping when this came out and I'm like, okay, but now I understand it so much more because of its impact that it makes really just all across the enterprise. But you know, I'm curious because it's almost like it seems like one of those things that's not like too good to be true, but it's, it's, it's faster, it's cheaper, it's more secure. So what are some of the challenges? Right, like as you guys continue this, this, this work and continue the development.

Yeah. You know the thing with the, as we get Lower into the stack and of infrastructure structure. Being wrong in software is bad. You have a bug, you got to fix the bug. You can generally fix software relatively fast. Being wrong in hardware is more expensive, right? So you want to catch those bugs earlier in the cycle, preferably before you've even made the chip. It's very hard to replace the chip once it's in production and out there as well. And then being wrong in construction is another problem as well. It's multi year complex infrastructure problem projects. So building in the wrong place or building in areas that won't have the energy needs or the energy supply that you need in five or 10 years is a challenge as well. So as we get closer to the infrastructure and the physicality of the cloud, being accurate longer term is really important for us. And so that's the constant pressure between like, what can be fungible? Software can be fungible, right? We can swap out one software for another software. What can be a little bit more fungible? Hardware. We can swap out one hardware for another type of hardware. It's a little harder, but both use electricity. Then as you get down to like space facilities, square footage, power supply, water usage, all these other things, there's less flexibility there. So we spend a lot more time on making sure we're right and we're in the right places that we expect the demand to be in the future as well.

So you kind of talked about some of the challenges that the rest of us may not see. So what are you all focused on now? Because at least to me, when I see this, I'm like, oh, you know, this checks off all the boxes. You guys have made it. But what are the next challenges? You know, from the Azure side, from the, you know, processing side, what are the next big challenges that you need tackle? And maybe not just Microsoft, but the industry at whole. Like, what is the next. A couple of hurdles that maybe not the things that keep you up at night, but the things that you're still actively working on.

Yeah, it's a great question. What keeps me up at night? A couple of things are really paramount right now is like security. How do we build security into every layer of this stack as well? And you can think about software security. We announced today our new hsm. So it's a new hardware security module that will go in all our servers going forward. And you think about network security, perimeter security. Quantum technologies are coming as well. And you know, one of the scary parts of Quantum is just how fast it can work and how it can cut through traditional cryptography like butter, which again, not a good thing. And so building in like quantum cryptography protections into our platform now is really important to us as well. Well, so that security layer of making sure that we can run this trusted, secure platform for all of our customers is really important for us to focus on. Then secondly, I'd say is the environmental impact. We've set some bold goals around being carbon negative, being water positive, being zero waste. And so all of that really translates to how we build and architect this data center infrastructure and this server infrastructure as well. And part of that is also working with our community, working with our broader supply chain, working with our energy providers and others to integrate into these grids. As you think about renewable energy, wind and solar, when the wind's blowing, the sun's shining, the power price fluctuates. And unlike a traditional factory, the data center is kind of software defined. So we have fairly granular controls of when we use energy, when we sip, when we slurp from the grid, if you like. And so being again, fungible with all of these workloads, being able to move them around our global infrastructure is something that's really interesting as well, to really take advantage and support kind of renewable energy to prosper as well.

Yes, I feel we focused a lot on the DPU and maybe scathed over the HSM a little bit. But can you explain a little bit more? How does that work and how does it actually make operations more secure?

Yeah, great question. So let me give you the, the 101 on like cloud cryptography. So HSM, a hardware security module, has traditionally been kind of an appliance, a separate rack that does nothing but generate cryptography keys. So that use for encryption and decryption, these things are generally really large prime numbers. If you want to really dumb it down, you can think of it as like a really expensive random number generator that will give you kind of uncrackable keys or very hard to crack keys. And so in that model, you have this separate appliance, you use it when you have something really important to secure. And so you will talk to this machine, you get back your cryptography keys, you do the encryption, you move the data around, someone will decrypt it, they'll talk back to the HSM machine. So, well, as you think about a world where you want more security, more defense in depth, more layers of this, essentially, in a world where every transaction, every message, every API call or every database read will be encrypted, you can have more and more traffic on that. And so moving that encryption decryption off a dedicated device, or maybe still there on a dedicated device for specific use cases, but embedding it in the server allows you to do it much faster and allows you to even have scenarios where you may not even completely trust all of the components on the server. You have transaction between two processes on the same server, encrypted between the traffic or something like that. And this is called ephemeral key cryptography, where you may be just encrypting and decrypting transactions that might just last seconds or milliseconds, but you're just essentially creating that secure chain between that transaction for the life of that transaction. But it may be ephemeral just there for that transaction in memory secure and then gone away after that. So these sorts of chips support that kind of next level of security that we're really looking to bring.

I feel so much smarter now. So, Alistair, we've covered a lot. We've talked about some of the new announcements today from Microsoft at the IGNITE conference here in Chicago. We talked about HSMs, we talked about DPUs, cryptography, we've talked about water cooling, the environment, timber, my gosh, everything. But you know, maybe what is the one most important takeaway that you want the average business leader listening in on what this actually means for them and.

Their business moving forward? Yeah, I think the key takeaway here is that this cloud technology and this AI technology is rapidly becoming more and more optimized. And our vision is that every application, every user, every workload, every business is going to be using these AI capabilities. And to be kind of democratized or commoditized and just be part of the fabric of how we do business is going to require like a new type of infrastructure, a new type of architecture that we're building up. We need to build it out in a way that it will scale to every business, every user. And that's in line with our mission at Microsoft, right? To support every user and every person on the planet, every person, organization on the planet, to achieve more. And so that scale out model is really what we're thinking about with all of these decisions across software, hardware, all the way down to physical space planning. Wow.

I think that was an extremely impressive and important way to wrap up today's show. So, hey all audience, next time you're out there, don't take what's happening under the hood at Azure for granted. There's a lot going on and it's only going to get apparently faster, more secure and cheaper. So thank you so much for tuning in. Make sure if you haven't already, go to our website@your everydayai.com Alistair just dropped a whole bunch of knowledge on our heads. We're going to be breaking it down for you in our newsletter, so make sure you go check that out. Thank you for tuning in and we hope to see you back tomorrow and every day for more Everyday AI. Thanks y'.

All. Thanks. Hopefully.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.