#113 The Hidden Energy Drain Behind AI and What We Can Do About It

You're listening to the Proteus Leader Show, a top 10% world-rated podcast where you'll get practical tools and insights for leading, managing, and staying ready for the future. Your host today is a senior leader of Proteus International, a firm that focuses uniquely on leader readiness, working to support organizations of all sizes and industries all over the world. Ready for something you can use today? Enjoy this episode.

Welcome to the Proteus Leader Show. I'm your host, Marie-Olive, and today I'm thrilled to welcome Ian O'Connor, distinguished professor at École Centrale de Lyon, my own engineering alma mater. Ian leads cutting-edge research on energy-efficient computing at the Lyon Institute of Nanotechnology. And directs one of France's national research networks on systems on chip. Ian is uniquely placed to help us separate hype from reality on AI and energy efficiency. AI's appetite is massive and used around 4.4% of the country's electricity and often taps higher carbon power. Cooling alone can be over a third of the load. So today we'll explore why AI is so energy hungry, where the real bottlenecks are, and the most promising hardware and architectural breakthroughs to bend the energy curve. We'll try to keep it simple, practical, and useful. Welcome to the show, Ian.

Thanks very much. Thank you for the invitation, Marie.

My pleasure. So, Ian, let's jump right in. So what do we mean when we say AI is energy hungry? Why does training and running AI take so much electricity?

Data centers, which power AI models like ChatGPT, they consume a huge amount of energy. So for example, as you were saying, data center electricity usage in the US hit 176 terawatt hours in 2023. So terawatts mean 10 to the 12th. So that is 1,000 billion watt-hours in 2023, and that is just over 4% of the national energy consumption, and it is still rising sharply. So exactly as you were saying, it is an enormous amount of energy which is being used. And further to that, most data centers rely a lot on non-renewable energy sources. The reason being that the data centers have to be on all the time. So it is extremely difficult to use renewable energy sources for powering data centers. And the result of that is that the data centers have a carbon intensity, which is almost 50% higher than the US average.

Okay. So they cannot really be on one non-renewable energy source. Now, I was talking to Marco Netto, one of Rallion graduates who used to work at Amazon for Jeff Bezos. And he was telling me that Amazon has built on the Columbia River, a combination of hydro, wind, and solar power. It's quite complex, but they managed to create a non-intermittent energy source. What do you think of that?

That's great. A very good initiative, extremely challenging. I can see how that can work, but it must have a lot of engineering challenges which are associated with it. The current thinking is that basically each data center almost needs a nuclear power station right by it so that it can work. So this is where we have been seeing people who are in the AI space starting to invest in nuclear energy. So like Meta has been investing in building a couple of nuclear power stations. So we are seeing this shift away from carbon intense to nuclear. But if something can be done so that we can use renewable energy sources by combining them, make a non-intermittent energy source, and that's very good.

When we decide to make a query on an AI tool like ChatGPT versus a query in Google, it actually uses at least 10 times more energy. Is that true?

Yeah, that is true. And it is kind of logical because when you're querying Google, I think it's like 1 gram of CO2 for every query that you make to Google. But it is 10 times more than that when you make a query into an AI model because You're not just asking for results to come back from the data center, you're also asking the AI model to make sense of it, give you a structured response to your question. And what we're seeing now is that data centers are just an incredible amount of power just to run. So per facility, it's like 100 megawatts, a few hundred megawatts, and that is inching up to gigawatts, and that is equivalent to the power consumption Of a small city. So it is really a huge amount of power. And it's not just the compute, it's also cooling. So cooling is 37% of the total energy. There is also a very complex infrastructure. The costs are $1 billion in capital expenditure. And training the models, which then enables you to get to these structured responses which are coming out, they cost up to $100 million each. So So it is just staggering. GPT-3, it used over 1,200 tons of CO2 equivalents. And so it's not just the energy consumption, it also translates to impact on the environment. So really, it does underscore that these energy demands translate to a need for sustainable infrastructure and innovations, that you can get the performance that you need at reasonable cost and with hopefully negligible environmental impact.

I'm so grateful that you're grounding us in numbers because I think a lot of us are completely unaware of that. And maybe it will help our listeners to go back to where it all began. And a little bit of history on AI. When you came to talk to us in the UK, really enjoyed your tentacles metaphor. So I'd love for you to share that with our listeners too.

I like making this parallel. Tantalus is a figure from Greek mythology. So his crime was that he stole ambrosia from the gods, so the food of the gods, and the gods were not happy with this, and obviously they decided to punish him. They condemned him to eternal hunger and thirst so that he was always reaching for food and water that just receded out of reach. And the metaphor is that it illustrates the insatiable nature of data centers. They are always hungry for more data and thirsty for more power. And this is because they are driven by relentless demands of modern AI workloads. So why are we here? How did we get here? So we have to go back to almost 100 years ago when AI was just beginning. And it has been an objective of humanity to make something in our own image. And that is, The kind of way that this all started to get to a machine that can think.

Too many science fiction movies about that, right?

Yeah. So it has been this kind of science fiction, but science fiction is here now. So the history of AI has really been through hand-in-hand approach between mathematical models and the hardware that enables you to implement these mathematical models. So they began with the very early logic principles, neural networks around the mid-20th century. This was initiated partly by Turing, for example, who started some of the initial computations. And then we got to the initial perceptron around the end of the '60s, which is now the fundamental building block of all artificial neural networks. And mainly the initial implementations were done on general-purpose processors or CPUs, central processing units, which were limited in processing power, but were the only thing that people had around at the time. And so as the demands for artificial intelligence grew and the need for More complex and more involved mathematical operations, hardware emerged that was specialized for AI. So initially it was digital signal processors, which are very good at signal processing algorithms, and then the GPUs, where the G is for graphical processing, and they really excel the matrix multiplications and the calculations which are really fundamental to AI.

GPUs were originally in like Gaming computer?

Yeah, that's right. So the G is for graphics, and so they were used to do image processing or video processing on games consoles. And the processing that is done in, in that space is basically working on matrices of values, because an image is just a matrix of pixels. So because it is a matrix and we wanted to do operations on these matrices, then the GPU, this architecture, already had the architecture that was necessary to do the kind of calculations, operations that are needed in AI, and much better than a CPU, which is very general purpose, very good at doing kind of any task, but it usually just works on a few things at a time. And it is not very good if you give it a specific task to do. It is not very good. It trades off flexibility over performance on a specific task, which is exactly what the GPU does. It looks at working on a specific task, matrix multiplication, but it trades off its flexibility. If you're using it for something where you need that flexibility, a GPU would be very bad.

Got it. So it's a bit less flexible. And maybe back to what you were saying earlier, I remember the movie Imitation Game that talks about Alan Turing. For the listeners that haven't seen the movie, that's a good way to imagine those first big massive computers and why they were created versus being done manually by a man.

Yeah, and at the time, you know, these machines were taking up huge spaces. And if you look at how the microelectronics industry and the semiconductor industry has evolved, now you can get much more computing power into a much smaller space. So you think about processors like this size, the size of a small chip, And you have a much more powerful computing processor in that small space than you had in just the whole room.

Yes. I'm very lucky because my dad always loved, you know, technology and bought us computers very early in our life. I remember this Amstrad big computer. And when you think of the phone like this, it's more powerful. It's a computer that we have in our hands with our Smartphones.

Exactly, yes. We have much more compute power than we ever had. And I had a Spectrum from Sinclair when I was growing up. If you compare the compute performance now to what we had many years ago, then yeah, we can say that there has been just some staggering progress in the number of operations per second that can be done.

Yes. Now I really want to go into where the energy really goes because we picture it as big computers doing math, like in that Imitation Game movie. But I think you suggested that it's also about moving data around that also consumes a lot of energy than just the calculations. Can you explain a bit further?

So the architecture that is currently used massively today is called the von Neumann architecture, and it separates processing resources from memory. So that was done in around the '50s and '60s in the last century to make these architectures programmable. So before then, what we actually had was something which was not flexible at all. We would be wanting to do, let's say, a multiplication, and so we'd build a circuit to do the multiplication, and it could do just that, very good at doing that, but just that. And during the computer revolution, so in the '50s, '60s, and then '70s when the microprocessor was starting to become popular. What happened is that people wanted to be able to program this with software, program with instructions and with data. So separation between the processing resources and the memory resources. That results in the fact that you have to move data between memory, processors, chips, and servers. And so this is what we term data movement. And the problem is that with today's processors, data movement is extraordinarily energy intensive because transferring bits over distances inside chips, between chips consumes a lot more operation, like orders of magnitude more than the math operations themselves. In typical processors, what we would see is that you have 95% of the power which is actually going into shifting data or instructions between the memory and the processor itself, and just 5% for doing the computations. And we're seeing this again, the scale of the racks and servers in the data centers. So the issue is that we already have this dominance of the energy in the data movement, but in addition, we also have these AI workloads, both training workloads and inference workloads, which are dominated by matrix-to-matrix multiplication, which we call general matrix multiplication. Which really requires just shuttling a load of data for these multiply-accumulate operations between the memory, between the processors, and it just amplifies energy usage. So it's just a huge amount dominated by these kind of workloads. Hmm.

Well, so I realized that I guess the proximity of the data centers could help reduce the need for movement? Does it play a role?

There is some transfer of data from the user, let's say, to the data center. But the main power consumption comes from the fact that we have this separation between processing and memory resources inside the data center. So you have to use the information which is stored somewhere. It's like having this huge factory belt where you have something which is stored somewhere, and you have to bring it to where it will then be processed in a central place. And then you have to send back the results again to where it's gonna be stored.

Mm-hmm.

And so doing that requires a lot of energy. And we're seeing this in all scales. So from the user to the data center, in the data center you have communication between the racks, and in the racks you have the processors or the boards. And so you have communication also inside the boards, but also inside the GPUs themselves. And so you're seeing this communication issue Everywhere at all scales.

Okay, I see. So there's all those different layers of communication kind of draining the energy. So obviously my next question is, how do we solve this problem? What can we do to reduce that energy drain, both in terms of electricity but also water? What is happening? And I know you're working on that specifically.

Yeah, there are a few ways to help things. So the first is to Say, basically, let's do the mathematical operations where the data is. So it's bringing the operations closer to the data, or the data closer to the operation. So avoiding so much data movement. So you could think about this as doing the calculations right where you store the information instead of repeatedly fetching it from a long way away. So if we can do the computing actually inside the memory chips, and that would really cut down the power which you use and speed things up. So I have a few projects which are ongoing at the European level, like Ferro4Edge AI, and at the national level, which we call FerroFutures. And we're focusing on how can we use new memory technologies to do computing so we can then allow this platform to do both storing of data and processing of data within the same physical hardware element so that Basically cuts out all the data movements completely. And so from this, with these technologies, we could then target low-power, high-speed computation for the AI workloads at the edge, because this is where the real constraints start coming in. So on mobile platforms where you have constrained energy resources, you're working with batteries. And so that could promise substantial energy savings. So this is one approach that we're looking at. Another approach is making the communication less costly, and there are a couple of ways also of doing this. So the first way is thinking 3D. So chips today are 2D, so you basically have an integrated circuit which communicates and is then integrated onto a board. And again, everything is kind of pretty much 2D, so you have distance between the various processing elements and the memory elements. But what if we could pile things on top of each other? In that way, instead of having the lateral distance, which is unavoidable, we use the third dimension to reduce the distance between memory and processing elements. So that is basically a kind of parallel between what we do in chips and in buildings. So instead of having cities which spread out over huge amounts of distance, square kilometers, Yep. Of course, what we have done is do multi-story buildings, skyrises. So instead of spreading out flats on a single floor, we use the third dimension.

Yeah, I love that metaphor because there was a big problem of urbanism in cities like Tokyo and you just go vertically versus horizontally. Brilliant metaphor.

Yeah, exactly. And so we've been working on this also in another European project to figure out how we can develop 3D neural network accelerators using vertical stacking of hardware layers. So this reduces the spatial footprints, interconnect distances, and we were focusing in Full Monty on this project on very low-power, real-time AI inference at the edge. The idea being that if we can do this, then maybe we can enable translation in earbuds without having it working just like a couple of seconds, so it runs out of energy. And also it doesn't burn off your ear because of the temperature increase that you have in the circuit itself. So yeah, the 3D, we believe in this a lot. And then being able to stack things like kind of Lego pieces to be able to build these very complex structures. So that would also help in manufacturing yield, making chips easier to upgrade, customize, and reducing unnecessary energy waste. And the third, I think we're on the third approach. So again, reducing the cost of data movement instead of using electricity, use light. So we have seen this in telecoms. Telecommunication uses fiber optics, it uses photonics to communicate over long distances. And as we go to higher data rates, so as our appetites as humanity gets increasingly large for data rates and data being transferred, that has become more and more interesting at smaller scales. So we've seen optics come from transatlantic communication down to metropolitan communication, down to communication in buildings. Then my computer is already connected by fiber optics, probably the set-top boxes that people are also connected to optics. And we are starting to see this also going beyond the connection between racks in data centers, but also actually inside the racks and even inside the chips. And if we can do this, it means that we can use a lot less energy to move data faster for inference. And because of the fact that we have data already represented in the form of light, could we not do some computing in the photonic domain?

Mm-hmm.

We see some research approaches already target matrix multiplication in photonics, which is actually quite well suited to this operation because it is already quite a linear approach. We avoid multiple energy-hungry electronic-optical conversions, which you would have if we were doing just communication in the photonic domain and all the processing in the electronic domain. So if we can do this kind of thing, then it makes sense to try to boost Confusing and difficult to integrate.

So, I captured 3 different ways at a high level. First, we need to do the math where the data lives. So, in a way, we avoid a lot of data movement that consumes a lot of energy. The second idea is to stack things in 3D. I'm thinking about Lego blocks. So, you bring everything closer to each other and using that vertical building versus And finally, what you were just talking about is to send data with light. And people can think about fiber optic as, you know, an example that they use in everyday life. Now, I believe when we were together, we also talked about smaller models. Maybe that's the 4th category. Can you elaborate on that?

Actually, the power consumption of an AI model is intrinsically linked to its size. So the bigger the AI model, the more it will consume. So if we can move to smaller models, that will require less computing power and memory access, and thereby will save energy. And what we have seen in the large AI models, the extreme-scale AI models, is that very often there is some redundant or less important information. So we can reduce the size of these models by pruning synapses or parts between it representing data with fewer bits, so shrinking down the models so that these models are smaller. So this has been done with quite a lot of success, both in research and now in industry, to really compress the AI models into the resources that are available, but also small resources as possible. But we're also seeing people working more and more on the models themselves. How can the models be built in a better way rather than having to build huge models and then try to shrink them? So just recently, actually last week, we were seeing some innovations coming out of Samsung. Take the shrinking of models a step further with what they call a tiny recursive model. And the difference is that instead of having a complete, what we call a feed-forward flow from input to output, it also uses some backwards links. So it has some kind of feedback or recursive connections, and that makes it into a much more richly expressible model, so it can express more things with a smaller footprint. And so this approach really challenges the idea that bigger is always better. And so I think they entitled their paper Less Is More, so a very nice way of expressing this. And so instead of increasing the size of a tiny recursive model, it uses this recursion and iterative refinement.

Yeah. So then it can, instead of having billions of parameters, it just has 7 million parameters, and it works with comparable Now, I can't help thinking that, you know, with smaller models, maybe you can do less things, or it has to be more targeted. So as a result of that, you could have an increase of the number of models, and then you get a lot of mini models. But if you add all the mini models together, it's as bad as one big model.

I wouldn't say that. I think that if we have a very large language model, which is very general purpose, it's a similar reasoning to CPUs, GPUs. We have these agentic AI models, which are more tailored, more specialized to certain usages of the model. I think that, for example, if we are going to use something for specific tasks like summarizing reports, answering questions, translating short texts. Then we could use a much smaller model. And that is actually what Google has been doing with one of their models, which has 60 million parameters rather than the GPT-4, which has like hundreds of billions. And it is very well optimized to do specific tasks and understand new tasks from a very small set of examples. So it can do specific things. And I think that we're going to see this, and I think the challenge will be in actually figuring out What model is the best for the task that you want it to perform? And how we actually get this is still to be defined.

That makes me think about a topic that comes a lot in coaching, you know, with leaders is multitasking versus doing tasks in a sequence. And often multitasking is overrated because you do a lot of things At the same time, but poorly. And I'm a big promoter of small tasks. I mean, not small tasks, but one task in a sequence, and then you do it very well. So maybe that's a similar idea.

Yeah, I agree. Because everybody has some kind of specialization. And so if we are looking at how a task should be executed, and this is partly how reasoning models work, they break down a large problem into smaller problems. And then process each individual problem and make the links between them. But I think we're going a step further, seeing how these different models will be used and how they can be then connected in a heterogeneous way. This, I think, will be very interesting because we will not just have different models but also hardware which is better adapted to specific models. And so what we will probably see is that we will have a large set of accelerators which could be used in a platform, and then specific AI models which then get mapped onto these individual accelerators. But we only use the accelerators that we want. I think we'll see a lot of things emerge in this space. And so a lot of interesting questions which are arriving about how we make the models, the implementation of the models, and then hardware implementation, how that actually is gonna connect together. That's fascinating.

I have to ask about quantum, right? Because it's a big topic also in the boardroom where people say, what about quantum? It's going to solve everything, it's going to replace AI. And maybe it's an oversimplification of it. So can you say a few words about quantum, what it is and what it can actually do for this issue?

Okay. I think if people say quantum will solve everything, I think that is wrong. That is not correct because quantum, even though people are working a lot on how quantum computing works, it will not replace classical computing. It does give a complementary path to specific tasks. And in particular in AI, if you want to improve energy efficiency, there are people working on hybrid quantum-classical AI systems, which can leverage some speedups in parts of the AI workloads. For example, when we're doing complex problems or training, quantum is very good at exploring a large number of possibilities at the same time simultaneously, whereas it is not so easy to do with classical approaches. So in that sense, it could be used to do specific parts in AI, accelerate the optimization problems which we have, model training, evaluating many solutions all at once. But it will not replace it. I think it is important to stress that AI will also not replace conventional computing because we still need computers, still need calculators, and we need exact answers to specific problems. AI will not replace those approaches to computing as quantum will not replace AI or conventional computing. I think we're going further towards a big mix of computing approaches.

Got it. I think it's good to talk about that misconception that some of us have. Now, we love leaving our listeners with practical steps. So what are 3 simple steps that our listeners can take this quarter to lower AI energy and costs, whether they're technical kind of listeners or non-technical?

Okay. From a technical point of view, so people who are actually making AI available in the workplace, for example, there are a few things to do. So, right-sizing the AI models and the workloads. So, task-specific AI models rather than general-purpose large models. I think smaller models should be used whenever it is possible. So, really using a screwdriver to turn the screw rather than a sledgehammer. I think that's really important. This is an important thing to do. And then there will be training efficiency, training tasks and inference tasks in local environments where people are seeking to implement the AI models in local resources. Again, it is all around the efficiency of the use of the models. So wherever feasible to reduce compute time, we should make sure that the input prompts are as short as possible. But also as accurate as possible. So having very concise but well-formulated questions is an important approach. And that is true if you're setting up AI for the workplace or as a user in normal life. And also the third way of the technical approach is to look at very energy-efficient hardware platforms. That do provide energy usage metrics, support hardware accelerators rather than worst-case CPUs that are actually then going to be optimized for the workloads. Also try to minimize the data movements. Idle compute time also is important. So apart from being concise and clear with the prompts, maybe also limit unnecessary interactions with AI. We've seen when ChatGPT came out, People were absolutely astounded and the natural reaction was to start using it and see what it could do and marveling at the magic. But there is an energy cost behind that. And people who use AI for futile things or which are not particularly useful or that could be avoided, you know, it's, this is something that will of course improve the energy efficiency. And it's all about the usage of the platform. And the final thing is, if you can look at how the AI model is being implemented, and if there is some information on the carbon efficiency of that AI model, choose these energy-conscious AI platforms. So services that really do prioritize sustainability. So ones that you were talking about at the beginning from Amazon about combining sets of intermittent energy sources, renewable energy sources to build something which is overall non-intermittent. That's a very good initiative. And so these offerings are showing transparency on the carbon footprint, and that is a good sign. And I think if they can use this, then that's great.

I hope there will be more and more kind of communication about that from those AI providers so we can make better decisions when we use one or the other.

And I think there is a clear recognition that energy efficiency in AI is a real fundamental problem. And people are really trying to do the best they can to reduce the carbon footprint of the data centers and the AI models. And I think things are going for the better. I hope that the race to get bigger and bigger and bigger models will soon end so that we can now focus on smaller models, smarter models, and how these can be used. But for sure, we want to have high performance, we want high-quality responses, but we also want AI to be sustainable. Absolutely.

Now, thank you so much, Ian, for making a complex topic clear and actionable. I got so many insights from your presentation for our Alma Mater, and I really wanted to bring those learnings to a wider audience. So I really appreciate you coming.

Well, thank you very much for the invitation. It's been a pleasure. Thank you.

For more resources on leading through change, including AI, visit proteus-international.com. Thank you for tuning in, and until next time, here's to creating the energy-efficient AI you truly want.

We hope you're feeling better equipped to create the career, the business, and the life you want. To find out more about how we support our clients in leadership, learning and change, visit us at proteus-international.com. Have an excellent day, and thanks for listening.