Data Center Metrics: Gigawatts and Manhattans

Live from Schenectady, New York, it's SAS Talk with the Metrics Brothers, Growth and CAC.

And I'm Growth, better known as Ray Reich, founder and CEO of Benchmarkit.

And I'm CAC, better known as Dave Kellogg, EIR at Balderson Capital, independent consultant, and the author of CalBlog.

And together we are the Metrix Brothers.

And we go together like shits and giggles.

What? Well, that one's out of left field. They're going to have to put parental controls on our podcast rating, Dave.

Oh no, Ray. Well, you try to come up with a new simile every week when you can't talk about food. It's hard work.

Yeah, I know, it can be tiring. So shits and giggles it is. But I don't suppose that really links to today's topic in any way, does it?

Well, I'd say profanity— this is tenuous, but profanity is endemic in New York City, and we're going to be covering 2 metrics today, and one of them is Manhattan's.

Manhattan's? Like the cocktail? Is that your way of saying I never closed that Katie O'Byrne's Irish Pub sponsorship, Dave?

You did miss that sponsorship, Ray, but no, I wasn't talking about the, uh, cocktail. I was talking about the 22.8-square-mile island that you live on.

Okay, so gigawatts and Manhattans it is. But before we jump into that, let's hear a word from our sponsor. The Metrics Brothers is proudly sponsored by Benchmarkit, the B2B AI and software industry's largest, broadest, and most contextualized benchmarking index. Dave Kellogg and Ray Groth Wright have put the Metrics Brothers podcast on hiatus for the remainder of 2026. All episodes we have ever recorded are still available on your favorite podcasting app. You can reach Dave @kellogg and Ray on LinkedIn @rayreich. Thank you so much for listening to the Metrics Brothers podcast. Okay, Dave, why don't you take the lead on this one?

Okay, so Ray and I have kicked off a series on AI metrics here on SAS Talk. The first episode in that series was an in-depth review of an excellent blog post by a guy named Ben Evans. And the title of that post was simply AI Metrics. And, you know, what I liked about that post, Ray, was that Ben made the important point that at the dawn of a new era, we're not only trying to figure out what to measure. Guys like you and me are like, ooh, what can we measure? But more importantly, we're trying to figure out what matters. Like, what's happening around us? What matters? What's going on? And then once we know that, we're going to know what to measure.

Yeah. And one of the things I liked in that was we got to talk about DAO over MAO and why a 60% rate was good for AI large language models. So I'm looking forward to learning more about the metrics we're going to talk about today.

Yeah, that was a fun episode. I liked, what did I like? I think I liked the big insight that he had, you know, But one of the big insights I liked was it's not just what we can measure because I do think for metrics people, there's a tendency to want to count things that you can count whether or not they're meaningful. Like in marketing, it might be leads at a trade show booth or white papers distributed. You count that. I can say we handed out 1,247 white papers. But the question is, how meaningful is it? And I think the example I would use from the internet era was hits. You know, hits got measured, but they didn't really mean a heck of a lot. And when you figure out what does matter, like the famous story that we talked about last week, that Google figured out that search engine response time mattered. And neither Microsoft nor other competitors did. And that one insight and measuring that because it mattered was critical. So, I love that episode, Ray. It was a lot of fun making it.

Okay, but let's get into this whole AI and Manhattans.

Yeah, we're going to get to Manhattans. They're going to come. I know you want your Manhattans, Ray, and you are 3 hours ahead of me here, and it's Friday night, so Ray's earned a Manhattan. But we're going to work our way there. Look, as part of the AI transition, we're hearing about a lot of new metrics. In some cases, the metrics aren't new per se, but they're newly visible. You know, for example, when we talk about models or foundation models specifically, we'll talk about model parameters, kind of effectively the weights within the model. And we'll hear numbers like 1.75 billion parameters in ChatGPT-3. By the way, 1.8 trillion, 1,000 times more in ChatGPT-4. And by the way, ChatGPT-5 was just announced today, but they didn't actually release the number of parameters. But we'll also hear other model metrics like how many tokens it has processed or processed per day. Right. So we have to get used to new metrics about, for example, models.

Yeah. In fact, 2 of those that I actually like, and it's kind of the same one, but 2 different takes at it, is ML performance. The acronym, I believe, is MLPerf. And all it does is it measures how fast a system can train a model and then how fast can a system run a trained model. And I've been seeing a lot of that when they compare ChatGPT to Groq to LLaMA. is what the ML performance benchmarks are. So that's one that I'm trying to learn more about and try to actually get some benchmarks on.

Okay, maybe do an episode on that. And of course, there's a whole bunch of other interesting model metrics like how good they are, how good are the answers they produce, anything from can it pass the bar exam to they line them up against all these standardized exams and see what they can pass. They compare them to human performance, etc. So but, but today we're actually not talking about model parameters. I'm just using them as an example. or model metrics as an example of the kind of new metrics that are coming across the bow. Today, I want to go down, not up, Ray. I want to talk about data centers. And specifically, we're going to talk about 2 metrics: FLOPS, floating point operations per second, and watts, because these are going to be the 2 common ways. There'll be a 3rd one, which is effectively area, which will lead us to Manhattans. But these are the metrics we want to talk about today. Thoughts?

No, I want to dive into it because I got thoughts when we dive into it.

Okay, you're gonna do a flop for me, or maybe drink a Manhattan. So look, let's get to a floating point operation per second. So what is a floating point operation? It's basically doing any basic math on 2 floating point numbers. So addition, subtraction, division, multiplication. In some ways, this is the new MIPS. Back when Ray and I were young, you used to measure processors by MIPS, which were million instructions per second. On the theory that things were compute-bound basically, and you were running processor instructions. Now, because in the world of AI, so much of this is just math calculations, we're looking at FLOPS, not MIPS. We don't care how many instructions you're running per second. We care how many floating-point calcs you can do, because as it turns out, to build, particularly to build/train foundation models, takes a simply staggering number of floating-point operations. Any guess, Ray? Do you know? For ChatGPT-3, how many flops were required?

Well, because I love pi, Dave, I'm going to say 3.14 times 10 to the 23rd flop. And by the way, it's also in the script, thank you very much.

Well, to be honest, Ray, that's actually the way I remember it too. So it's a good metric if you want to burn into your head a simple little number. Uh, you can remember 3.14 0.14 times 10 to the 23rd flops are the number— is basically the amount of work as measured by floating point operations required to build the ChatGPT-3 foundation model. Now, to translate that— go ahead, Ray.

No, no, I'm just thinking, I don't think we have this data, but I know ChatGPT-5, who just came out yesterday, is so, so much more advanced. I can't wait to see how many flops that took.

Yeah, yeah, don't have the data available. This is why I'm using ChatGPT-3, by the way, because we have the numbers. Let's first, if we wanna talk about flops, we're gonna have to dust up, dust off our skills on prefixes. So let's start at the bottom here. So 10 to the 9th is billion, and that's giga. Everyone knows that, right? So gigahertz, gigabyte. Tera, 10 to the 12th, trillion. So the— well, I won't do examples. Peta, 10 to the 15th quadrillion. Exa, 10 to the 18th quintillion. And zetta, 10 to the 21st sextillion. So this is 3.14 times 10 to the 23rd. So it's 314 zettaflops is what it took to train ChatGPT-3. Okay, Ray, any guess on how much it costs to use this thing? If it costs 314 zettaflops to build it, what does it cost to do an inference with, say, 100 tokens?

Well, I'm going to think it's a lot less, so I'm going to think something around maybe closer to a teraflop or 10 to the 13th.

Okay, okay. Well, I'm not sure that's a teraflop. I didn't do this one in English. Is that going to be right? This would be 10 teraflops, but, um, you're correct. Let's, let's come back. To do an inference, you're right, Ray. 3.5 times 10 to the 13th flops equals 35 trillion flops equals 35 teraflops, right? Because it's 10 to the 12th, so there's one extra. So it's 35 trillion flops, 35 teraflops. Okay, so the real question— so now that we've got those 2 numbers, and part of the purpose of this exercise today is to try and get used to some of these numbers, their size, the scale, prefixes. How much bigger is the cost of building the model than using the model?

Well, the first time I saw this, it blew me away. And I know we're getting better every day, but it was 9 billion times more to build the ChatGPT model than it is to use it.

Yeah, just if you did, if you did the simple order of magnitude, 10 to the 23rd divided by 10 to the 13th, Right. So, or sorry, other way around. But it would be 10 to the 10th difference, which is 10 billion or 9 billion if you account for the prefixes. So yeah, it costs, I'm gonna round it up, 10 billion times, 10 billion times as much to build ChatGPT as it costs to use it, which is really my—

10?

Like when you said it was a lot, I'm like, yeah, Ray, I'm thinking maybe 100 times, maybe 1,000 times, maybe 10,000 times. We're talking 10 billion times.

Well, that's why when you see, you know, Microsoft investing $80 billion in capital to build out their AI infrastructure this year, or I think it's over $100 billion for Meta, now it makes sense, right? I mean, this is—

Yeah. Yeah, capital intensive. By the way, you have a ton of people doing these. I mean, let's just be clear, you build the model once, so it costs 314 zettaflops, but you have millions and millions, hundreds of millions of people, right? ChatGPT is, I think, 120 million daily users running inferences, and this is not a particularly complex one we used, it costs 35 trillion flops, 35 teraflops to run an inference, right? So these things are enormously— as Andreessen Horowitz said in their original post on all this, the generative AI revolution is compute-bound, right? And they weren't kidding. And there's a minute, Ray, I think we're going to find out it's actually energy-bound, but we're going to come back to that in just a minute. So, uh, now for the finance people in the crowd, when you hear about something that costs a ton of money to build, that lasts you for a while, and is fairly cheap to use, you should think what I think, which is factory, right? Hey, we're going to build a factory, costs a lot of money to build it, but it's going to have a useful life of 30 years, so I'm going to depreciate it over 30 years, yada yada. So they've got capital expenditure to build the factory, right? Kind of 2 different pockets. So companies have their operating expense pocket and they've got their CapEx pocket. There are 2 different budgets in most companies as well, obviously tied together. But that's what this feels like to me. I have no idea, in point of fact, Ray, how AI companies do the accounting here, because the trick is it does cost a ton to build it, but it doesn't actually last that long. These models last only a year or 2. So it's quite possible they don't even get capitalized.

No, I am, you know, I've talked to some people in the last 48 hours on this, and a lot of times the initial training of the models is being treated as OpEx, Dave, not capital expenditure.

And if your factory only lasts a year, so you get to depreciate it over its useful life, which is, ta-da, a year, which means it's effectively OpEx. So, okay, well, that's crazy. We're not going to do the accounting on this today, but, but in any case, I wanted to get the main point across that These things are incredibly computationally consumptive to build and actually to use. So now let's go talk about power, because I started to say, and Andreessen Horowitz said this, the generative AI revolution is compute-bound. Maybe it's actually power-bound or energy-bound. So let's talk about watts for a minute here. We're going to talk about 2 units today. We're talking about data centers. The first was flops. And now we have some sense for what flops are and how big they are. Now let's talk about watts. So watts are a unit of power. You wanna do watts, Ray? Here, you studied this stuff.

No, I want you to do it 'cause I'm gonna talk a little bit about how that's power versus energy consumption. So please go ahead.

Oh, okay, okay. So look, a watt is a unit of power which equals a joule per second. So it's energy per second. And it also could be stated as the power in a circuit where there's 1 volt of potential difference and 1 amp of current. That gets you 1 watt. And therefore, a 60-watt light bulb, to bring this back to Earth, uses 60 joules per second. And your electric bill, by the way, is not measured in watts. It is measured in kilowatt-hours, or watt-hours in effect, which is basically energy times time. Sorry, power times time.

I was going to say, that's a measurement of—

that's it.

But it is a measurement of energy versus power.

Correct, because it's power times time, and power times time equals energy. 'Cause power is energy per second times seconds equals energy. Did you ever—

Exactly right. So that's all I was gonna share 'cause I was doing, yeah, I was, 'cause I'm amazed at how many, how much energy these data centers are gonna take, but that's for later in the podcast.

So when you went to college, Ray, did you have to, in physics class, calculate the speed of light in furlongs per fortnight? That was literally our first homework problem.

We actually did, even at a state school, we did that.

But it's a great— so that's why I'm very unit-driven still to this day. When I try to figure this stuff out, I'm like, show me the units and I'll get the math right. So, okay, so we now know what a watt is and what— and we could make it intuitive by knowing that our typical light bulbs around us, by the way, used to be 60 watts. Now that they're LED bulbs, I think they're around a 10th of that, maybe a 20th, right? Like it's simply a 3 or 4-watt bulb. giving the same light as you used to get from a 60-watt bulb. But, but anyway, let's imagine the days where a 60-watt bulb took 60 watts. Um, let's talk about power in these models. So ChatGPT-3 cost around 1.3 gigawatt-hours of electricity to build. Now that is the same order. When I hear 1.3 gigawatts, what comes to mind, Ray?

Well, you know how I like history. I think of Back to the Future and DeLoreans.

There you go. And the famous quote by Doc Brown, basically, it's gonna take 1.21 gigawatts. And I had to go figure out why he said gigawatt. Do you know the answer on this one? Why did he say gigawatt?

I don't.

So the trivia here is that it was a mistake, that the filmmaker didn't know how it was pronounced. And they thought giga kind of sounded like ginormous. So they kind of willfully mispronounced it into gigawatts, but there's nothing more behind it than that. So the correct pronunciation is gigawatt.

And I don't think we talked about the name that power source for the car in Back to the Future. What was it called, Dave?

The flux capacitor. We failed to mention the flux capacitor. Yes, the flux capacitor required 1.21 gigawatts of power, which is why they had to attach it to the clock at the town hall to capture the energy from the lightning bolt. But anyway, just coincidentally, that ChatGPT took around basically the amount of energy it would take to power the flux capacitor for 1 hour. That's the amount of energy it took. The flux capacitor is completely fictitious. That, that may not be terribly intuitive, but that is basically, you know, giga means 10 to the 9th, so it's 1.3 billion watt-hours. And to try and make that real, the average US family consumes 10.5 kilowatt-hours. Remember I said we needed to dust up on our prefixes?

Yeah.

Per year. So that's the power on average for 123,800 families per year. So it cost as much to build ChatGPT-3, the foundation model, as it cost to power St. Louis or Salt Lake City for a year.

That's incredible. And I, I just am starting to think about, you know, how much power does a nuclear power plant or solar grid— and, you know, we'll get into that— but it's going to take a hell of a lot of power to make this happen.

It does. It does. Somewhere in the script, I had the nuclear plant. I don't know if I missed it, right? Oh, there it is. Yeah, the average power output of a large commercial nuclear reactor is, is again, coincidentally, 1 gigawatt, right? So, so it took basically— I want to make sure I get this right— it would take an hour of a nuclear reactor to build ChatGPT-3, which is the same amount of power it would take, or energy it would take, to power the city of Salt Lake City or St. Louis for a year, just to get more intuition here.

How does that compare to like a standard data center today, Dave? Is it—

Yeah, so a standard data center today runs between 10 and 70 megawatts as a power consumption. And because megawatts is per time, right? Because it's— if you look at it for a year, it'd be 613,000 megawatt hours per year. So if you took 70 megawatts for a year, it'd be 613,000 megawatt hours. So now let's talk about Meta's recent announcement in July chasing superintelligence, which was really the genesis of this episode. Meta said we're going to spend hundreds of billions building huge AI data centers chasing superintelligence. The biggest one will go up to 5 gigawatts, which is about 100 times bigger than today's medium data center. 'Cause we just learned today's median data center is around 50 megawatts. So 100 times bigger than that. It's about 10 to 20 times bigger than today's largest data center. So the largest data centers today are somewhere, I had trouble getting an answer, but between 250 and 500 megawatts. So this thing that Zuckerberg announced is 10 to 20 times bigger than today's single largest data center. And perhaps most impressively, it's 1/10 of all, all total global computing. global data center power consumption today, which is 55, which is 55 gigawatts. Right. So, so this thing's big.

It's a lot of numbers, Dave. I'll tell you, I was, I was doing some additional research from the International Energy Association— agency, it's actually agency— and they said we're going to need like 900 terawatts to fuel all the AI data centers by 2030. So another number, but it's just amazing how much we're going to have to build out over the next few years.

Exactly right, exactly right. It is. This is what I'm saying. Maybe, maybe Andreessen was wrong. This thing might be energy-bound, not compute-bound. Um, so because what Meta has talked about is building a data center, one data center that takes the equivalent of 1/10 Of what every data center takes today. And that's 10 to 20 times bigger than today's largest data center. This thing's big. The first multi-gigawatt data center is aptly named Prometheus, who stole fire from the gods. It's in Ohio. It'll be operational in 2026. It's got 500,000 GPUs. It will have 3.2 teraflops of performance. So the metric I think is interesting here is basically the teraflops per gigawatt. And it just works out that you get 3.2 billion teraflops per gigawatt in this example.

Hey Dave, before happy hour ends here, we gotta talk about Manhattan still.

We do. So the other thing, so there's 3 ways to measure data centers, right? One is flops, one is gigawatts, or you can use my new unit, teraflops per gigawatt. And we got 3.2 billion of those from Prometheus. But the last way to do it is through area. Just how big are they in terms of square miles? And all this takes us to Manhattan, Ray. So it's getting time for yours. Meta also announced that they're building clusters. They call the first cluster Hyperion. It could scale up to 5 gigawatts in power. So 10 Prometheus's. Is that right? No, 5 Prometheus's in power consumption. You know, in the press release or in the PR, Zuckerberg said it, quote, covers a significant portion of the footprint of Manhattan. And that's why I said, oh, we're going to start measuring data centers in Manhattans, right? How big are they? As in using Manhattan, which is 22.8 square miles, as a unit of area. But alas, Ray, this turned out to be somewhat hyped because when I did more research on this, that significant portion looks to be 3.5 square miles. And 3.5 divided by 22.8 equals 0.16. So even the biggest thing complemented— contemplated, which is one of these Hyperion clusters, is still only 0.16 of a Manhattan after all.

Okay, so we're not going to lose our financial district to a data center. That's good.

No, but they are going to be building them on the order of magnitude, you know, it's order of magnitude the size of Manhattan. It's just only a sixth of them. So these things are getting— I mean, I just think the main purpose really today was to try and understand the new data centers and to get a sense for their scale, both in terms of surface area, like we're building data centers that are as big as a 6th of Manhattan, their power consumption, which we're now measuring 5 gigawatts, and basically their compute power, which is 3.2 billion teraflops. So pretty amazing.

Dave, this was some incredible information. And we, we're talking about data center size here and data center investment. But to your point earlier about AI may be energy bound versus compute bound, I did similar math for how many acres of power grids it's going to take, and it's scary. But we'll talk about that hopefully in another episode.

Okay. Well, thanks for joining us, Ray, and go get that hard-earned drink. The Metrics Brothers: AI, SaaS, and More is a production of The Metrics Brothers Growth and CAC and a member of the Benchmarkit Podcast Network. By accessing this podcast, you acknowledge that the Metrics Brothers make no warranty, guarantee, or representation as to the accuracy or sufficiency of the information provided or the humor content of the jokes told. Ray? Information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for information purposes only. This podcast should not be considered professional or, for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, indirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising out of any use of, or God help you, reliance upon the information presented here. Ray Grothreich is based in New York City and available on X at Ray Reich. Dave Kellogg is based in Silicon Valley and available on X at Kellogg. Schenectady, which is French for unspellable, is not our actual production location. Thanks for listening.

Okay, have a good one, everyone.

See ya.