Jensen Huang addressed SemiAnalysis at GTC, defending Nvidia's tokens-per-watt advantage. He argued that Hopper H200 and Grace/Blackwell NVLink achieve the lowest cost-per-token through extreme co-design, and pushed back on published performance measurements.
Chapters
Tokens per watt is important because every data center, every single factory, by definition, is power constrained. A 1 gigawatt factory will never become 2. It's physically constrained. The laws of atoms, the laws of physicality. And so that 1 gigawatt of data center, you want to drive the maximum number of tokens, which is the production, the product of that factory. So you want to be on top of that curve as high as you want. This, the x-axis, is the interactivity, the speed of inference, the speed of each inference. The faster you can inference, the faster you could, of course, respond. But very importantly, the faster you can inference, the larger the models, the more context you could process, the more tokens you can think through. This axis is the same as smartness of the AI. And so this is the throughput of the AI. This is the smartness of the AI. Notice the smarter the AI, the lower your throughput. Makes sense. You're thinking longer. Okay? And so this axis is the speed. And I'm going to come back to this. This is important. This is where I torture all of you, but it's too important. Every CEO in the world you watch, every CEO in the world will study their business from now on in the way I'm about to describe, because this is your token factory. This is your AI factory. This is your revenues. There's no question about that going forward. And so this is the throughput. This is the intelligence. Better perf per watt for a given power of data center, the more throughput, the more tokens you could produce. On this side is cost. Notice Nvidia is the highest performance in the world. Nobody would be surprised by that. They would be surprised by the fact that in one generation, whereas Moore's Law would have given us through transistors 50%, 2 times, Moore's Law would probably give us 1.5 times. More performance. You would have expected from Hopper H200 one and a half times higher. Nobody would have expected 35 times higher. I said last year at this time that Nvidia's Grace Blackwell NVLink 72 was 35 times perf per watt. Nobody believed me. And then SemiAnalysis came out and Dylan Patel had a quote. He accused me of sandbagging. He accused me of sandbagging. He says, Jensen. sandbagged, it's actually 50 times. And he's not wrong. He's not wrong. And so our cost per token, our cost per token is the lowest in the world. You can't beat it. And the reason that's true is because of extreme co-design. And so I'm very happy that he named us.
