What Happens After A 1,000,000x AI Compute Leap? | Jeff Dean

There used to be a chat group internally called Data Centers on Fire that would have exciting events happening.

A distant supernova goes off, a cosmic ray hits a memory cell, and a 0 flips to a 1. Does that really happen?

Oh yeah.

So my question is, do you enjoy these Chuck Norris-style jokes about you?

It could be true.

One problem that you solved, tried to solve many times, but have never been able to crack? I cannot believe that this is happening, but I got to talk to a legendary engineer.

The Chief Scientist of Google, Jeff Dean. He led Google Brain, one of the most legendary AI labs in history. He co-created MapReduce, which taught thousands of computers to work together as one. He co-built TensorFlow, the engine behind a huge chunk of AI research. And for all this, they call him the Chuck Norris of computer science. Yes, I will tell him a joke about that too. Now, when I see interviews with these executives, everyone is asking about China and taxes and all that.

Look, I know nothing about that.

I am just a student who loves to talk about research, so my goal was to try to go a bit deeper and ask him questions that maybe only he knows the answer to. Which is incredible. I'll also ask him about problems that even he couldn't solve yet. And I will ask him about some of the secret sauce at Google and see if we get something. And more. And I am so happy to share it with you Fellow Scholars so we can learn together. I am not sure if I saw Jeff smile and laugh this much before, so I hope he enjoyed it too. And once again, this is an incredible honor. I cannot believe that I was sitting there, There were some production issues with the video part. I apologize for those. Also, I was super nervous. I could barely hold on to my papers. Now, fellow scholars, let's learn together with Jeff Dean.

Thank you so much for doing this, Jeff. We talked a bit last year and I learned so much from you. It was incredible. And then I got a message that we, we get to do this and I was so happy. So thank you so much for this. And we get to share your knowledge. Small part of your knowledge with the fellow scholars. So that's, that's absolutely—

it was great chatting with you last year. I'm looking forward to this.

Thank you. Thank you. So everyone says that we are running out of training data for LLMs, but you, you said that there is still plenty of data out there. What did you mean?

Yeah, I mean, I think everyone has this view that, uh, we're running out of training data, and, um, it's true, we've like used quite a lot of, of the public text data in the world, um, but I think there's lots of interesting video data that we're not really training on yet. Uh, there's lots of interesting kind of, um, ways to generate synthetic data and then use that for training. And then I also think we can start doing things like, uh, making more passes over the data that we do have to make more, more capable models and also come up with algorithmic techniques that enable us to get a lot more information from every piece of data that we do have. So I'm not too worried about that as an impediment to making progress. It seems like there's lots and lots of things we can do.

People also say that with so much simulation data, as you mentioned, sooner or later, most of the data will be AI-generated, which is then used to train a different AI. And then suddenly, everyone starts to learn on the same thing. But you said— Wait, it still helps. I think the argument was that if you have enough compute, you can crunch through a lot of data. And if there is just a little needle in the haystack that's useful, the system is able to learn from it. Is that true? Because my previous crappy little experiments, it was not true at all. So you had to be very careful with the data.

Yeah, I mean, I think it is true in general. I mean, there's a lot of details to get right to make this a reality. Think about, for example, doing RL training and rollouts to figure out how to solve some fairly high-levelly phrased coding question. So you might explore 100 or 1,000 different ways of generating solutions to these problems. And you might have some filters that you apply to these things, like does the code even compile? Well, you can throw out 800 of them right off the bat. Does it pass the unit tests? Does it like perform well? And so you can really start to hone in on like which of these, you know, potentially many solutions to the problem is the one that actually sort of generates the highest, you know, characteristics that you're looking for, the reward in some sense. And that I think is, is definitely true. Like more compute will generate you more interesting solutions and then those can then be put into the training data. They can be enriched with data augmentation techniques. I generated the solution in Python. Now I could generate a solution in Go and have more Go programming language training data.

That's an incredible kind of augmentation. Augmentation before with convolutional neural networks, it was just shift the image by a couple pixels and whatnot. And here the augmentation can be a completely different programming language.

Yeah, I mean, I think a lot of times we think about coding-based problems as you go from natural language, which is often very underspecified. It's like, make me a cool Space Invader game or something. But actually, if you have a program that already works that does what you want and you want to translate it, that's awesome. Because in effect, your prompt is the fully specified behavior of the system you want. And you just want it in a different language for whatever reason maybe. Better performance or better safety characteristics or whatever. So that we've seen internally with some tools that have been written in Python and people have been able to sort of just say, please use all the tests for this code and the actual Python code base and make different versions of it and found much faster solutions.

So you can suddenly get so much more out of the same amount of data basically. Yeah, yeah, exactly. So that's why you're not worried about the data. Yep. Okay.

Nice.

Now, Bill Dally has said that something like 90% of what happens in modern data centers is not training anymore, which I found really surprising. It's inference. There's less training and more using, relatively speaking. How does that shift the way you design hardware at Google?

Yeah, I mean, first, there's a lot of other things that are not either inference or training that happen in data centers. So all the applications we run and search and Gmail and so on. But of the sort of machine learning workloads, you know, it is the case that training is becoming, you know, less proportion of the overall compute that we want to do because there's so much, you know, inference workload you want to do. And the inference workload includes both like offline inference, sort of RL rollouts during RL training, and then also online inference for handling user requests or agent-based behavior. Because of that shift and the different characteristics of those two kinds of computations, it makes a ton more sense to now specialize much more for inference workloads in hardware, for example, because the characteristics are quite different. You need lower precision. You are handling a very large volume of requests on this particular model. The model weights don't necessarily change at inference time. All these things lead to very different solutions for hardware, and much more energy efficiency can be gained by specializing. And so I think you'll see a lot more in this area now and in the future. We've already done this with our TPU 8i and 8t chips that we announced a couple, maybe a month ago, but you'll see even more specialization, I think.

And that's pretty crazy that you said that even FP4 kind of works. And when I first heard it, I was like, it cannot possibly work and do anything useful. And it does.

Yeah. If you told that to a computer scientist from 15 years ago, they'd be like, that's not enough numbers.

Yeah, exactly. And I look at every now and then at these papers and you have these different transforms that are the distance-preserving transforms, rotations between the points and all kinds of compression, but still FP4, that's It's unbelievable.

It's not many bits for exploder or mantis or scythe.

And it's high quality intelligence that comes out of it.

So it's unbelievable. It's a good sign that it works.

Yeah, yeah. I don't know if we can get lower. What do you think? Like even lower?

Possible. I mean, I think people are seeing and experimenting with things where you have some even lower precision and then every so many weights of that lower precision, you have a scaling factor. And that seems like you get a little bit of a higher precision thing that's kind of shared across all the other lower bit precision formats, whatever they might be, 2-bit integer, 1-bit integer. You know, I haven't heard anyone say 2-bit float because I'm not sure what that would mean. But yeah, I think that plus a scaling factor seems to be able to get you pretty far. And the question is, like, how often do you need the scaling factor? Is it every 64 bits? 128 or 256 weights.

Pre- and post-training are typically separate steps today. Do you see that split holding or do you expect the two to merge as capabilities increase?

Yeah, I mean, I feel like it's a little intellectually dissatisfying that they are these distinct phases and you do one and then you do the other. It's like conceptually the right thing to do is to have interleaved periods where you're sort of observing data and then periods where you're trying to use that new knowledge you've gotten from the data.

Like, like with DQN, this experience replay kind of thing.

Yeah. And then you want to now take actions in some environment, maybe it's a simulated environment, maybe it's the world with a robot or whatever it is. And then, you know, learn from those actions because I think you get a lot more benefit from actually taking actions and observing the consequences, or trying to write code and seeing does the code work, than you do from just passively sitting there and seeing tokens streamed by you, which is really what most of pre-training is.

It's really interesting that you say that, that in an interleaved manner, because when I, when I hear merging the two, what, what in my mind is continuous, like like continuous learning, but at the same time, people have to test models. You cannot just chuck it out there. You finish training, you finish the post, and then maybe the red teaming steps and safety and everything, and then you package it up and you say, okay, this is good to go. But if there is continuous learning, then there's new challenges because how do you know that this intermediate state is actually safe? Maybe some more research there too.

Yeah, I mean, I think, first, like, a bunch of discrete steps where maybe you do this 100 times or 1,000 times starts to look more like an integral than a summation. And so I do think interleaving in that way will make sense. But you're right. Like, you have a bunch of things you need to do for a live model that is serving user requests. You need to make sure that it's safe. So maybe that the continual learning happens and then there's some application of safety protocols and red teaming, as you say, and then you release a new version of that, but then that model still continues to learn behind the scenes. And then before the newest version of it is provided to users, you redo the final safety testing and red teaming.

Jensen likes to say that compute capabilities advanced 1 million X over the last 10 years. So if in the next 10 years, assuming we get another 1 million X, what would we be able to do that we cannot do now?

Yeah, I mean, it's like imagining the future is always a hard thing because this field is moving quickly. I mean, I think if you think back, you know, 10, there was 10 years, 10 years, 10 years. You think back 10 years, you know, we were kind of just starting to have language models that were the sequence to sequence paper had appeared, you know, it was just before the transformer. LSTMs maybe. LSTMs. LLMs were popular. And now those models sort of look—

Ancient.

Not nearly as ancient and not nearly as capable as the models we have today. So I think if you project forward that level of advancement, you're going to see huge investments in both new kinds of hardware, new kinds of research techniques. There's just a lot more attention being paid to the field, so I see that progress rate not slowing down over the next 10 years. And so that's going to be incredible. Like the multi-agent workflows, we're now able to start to kind of get to work on very complicated tasks like you saw in the I/O keynote, being able to write an operating system autonomously with a relatively simple prompt. Crazy. You know, obviously there's a lot of operating system-y like things in the training data. So it's not completely out of distribution, but the fact that it's able to build an OS that can run Doom successfully is pretty amazing.

I couldn't believe it. Last year I heard a talk from Stephen Balaban, the Lambda CEO, and he had this neural OS like, hey, it does more and more. Forget the UI, forget maybe the drivers, I don't know. But just let's have a neural OS. And I was like, yeah, that sounds like an amazing science fiction idea. I would love to see it, but I don't know. I mean, it sounds far off. A year later, and we got— you know, not exactly like that, I know, but, but if, if you look at the derivatives—

yeah, I mean, I would say one thing I'm particularly excited about is, you know, can we with these tools accomplish so much more in, you know, science— Demis was mentioning in the keynote— or in, you know, complicated engineering tasks that often would take lots and lots of people multiple years to accomplish. Could you actually have a system that with the correct access to the right kinds of simulation environments and a learning set of agents that are trying to accomplish the task and break it down into smaller tasks, could you design an airplane in 5 days instead of many, many years? That would be amazing.

1 millionx, and we can try again.

Yeah. I mean, we're not there yet, but that would be a pretty— pretty amazing capability, or designing new computer chips or computer systems, new hardware. I'm pretty excited about that.

Incredible times. Are open models standing on the shoulders of giants? And by that, I mean, if frontier models suddenly stopped being released, would open models improve as quickly as they do now, or is their progress mostly driven by distillation?

Yeah, I mean, I think certainly a bunch of the progress is driven by distillation. For example, our own Gemma models are definitely distilled from higher quality, larger scale models. And I think a lot of other open source models are getting benefit from distillation data. Distillation has always been an amazing way to get really capable models into a smaller footprint thing. And that's how our Flash models are quite capable for their size relative to the Pro models, is we're able to use the Pro model to to teach the flash models. So I mean, I think really the question is not so much one of closed versus open. It's, you know, if we want small, incredibly capable models, we have to keep building larger scale models that are maybe less inference efficient but are more capable and then use distillation to, you know, transfer the knowledge into into the smaller models, whether they are open or closed.

Now I'm wondering, you might be the only one who can answer that. So I really want to ask this. Everyone has their flagship models and yes, the distilled models, pretty much every company does this tiered level thing. The quicker, faster models were always well below the frontier models. And at some point, I think 3.1, there was one version where the quick one was suddenly so, so close to the frontier when there was like a 3% difference in, in, in TUF benchmarks. And, and I just heard someone say, I don't even know who that was, that, that, yeah, it's not like just distillation, there is some magic sauce in there that's been in the works for years. So can I hear a bit about that?

Sure. Well, not too much. I mean, there is always some magic sauce that we don't reveal, but distillation is definitely one of key the things that makes those much smaller models, much cheaper, much faster, much more affordable models be nearly as good as those frontier models. And then we push ahead and build an even better frontier model. And then we have to then do the process again where we now transfer the knowledge and the really capable frontier model back into a lighter weight one. And I think this is really important because the flash models are really the workhorse of what people generally want to use because they're almost escapable.

We saw it.

Yeah. And they're quite good.

Yeah. It's unbelievable how close they can get. This didn't used to be like that at all. All right. What trends in machine learning are you most excited about right now? You have a separate talk about exciting trends in machine learning where Oh, yeah, yeah. I mean, I think— What's the newer version of that?

Yeah, the newer version, I guess— I mean, there's a few different trends that I think are really exciting. So one is— so first, I think continual learning is still a little bit nascent. But I think looking at ways to make models that are more interleaved in their way, use of sort of seeing data passively and taking action and learning from that seems like a really important thing. Agents and multi-agent use of these systems is really, really important. As one trend of that, though, I think as you see, we're going to need a lot more inference hardware and capability for that because those systems that are working autonomously in the background actually consume lots of tokens in order to do the important work they've been asked to do. I think being able to build really efficient inference hardware will enable a lot of things. So looking at co-design of model architectures and hardware architectures to make sort of the best use of things and have really good properties in terms of very low latency, much higher performance per watt, performance per dollar are things we really care about. I think looking at how do you, the context window of these models is an important characteristic, but I think there's a lot we could do if we come up with mechanisms that are sort of cascaded series of things that kind of give you the illusion that you have all information in the context window. Like you'd like to have the whole internet at your model's fingertips. Or on a personal level, if you've opted in, you know, all of your email and your photos and your videos you've watched and things like that. But you can't really do it with the sort of quadratic attention mechanism. But I think you can build a series of kind of retrieval and lighter weight mechanisms and then ways of cascading from, you know, here are the 30,000 documents out of 10 billion that seem most relevant and then have a lighter weight model that looks at those and decides these 117 things seem really relevant to what you're trying to do and puts those in the more expensive context window of a bigger model perhaps. That's going to be exciting. And how do you orchestrate and interleave all that stuff so it gives you the illusion without you having to even think about it?

Interesting. So it's very advanced games to be played with the context window because obviously very expensive. So the attention mechanism you get, You got big O of n squared. Are we still there or do we have some, I mean, I've heard some n log n things. Can we go lower?

There's like a whole series of—

Obviously we can go lower, but the question is what the trade-offs are, right? Yeah. Like what do you have to pay for that? Yep. Where are we in that?

Yeah, I mean, I think there's actually quite a large body of work. There's probably 100 papers on more efficient context algorithms than the— N squared one. I mean, the N squared one works really well, so it has a pretty high bar. But I do think there is traction in finding things that are much lower cost, whether it's reducing algorithmic factors or very large constant factors on the base N squared algorithm. I think all of these are pretty exciting. You can actually combine many of these approaches and get much cheaper attention over many more tokens.

Yeah, I think that's one of the most important things because if it was cheaper in some sense and you could still find the needles in the haystack over very long contexts, then you could have some sort of lifetime AI thing.

Yeah, totally. I'd like my whole life of all the digital things I've seen in there. As an internal Google developer, I'd love for the entire Google codebase to be in there, which is probably 10 billion lines of code.

Yeah, yeah, I think big.

Yeah, you know, 100 billion tokens.

I just want my wine list. I just want 100 billion.

All I want is 100 billion tokens of attention. Yeah, that's all I need.

Amazing. I think we gotta do this one. So Google's data centers run an enormous number of machines, and at that scale, anything that can go wrong will go wrong. Like, I hear that wires wear down, hard drives fall apart, motherboards overheat. Is that something that actually happens day by day, and do you have any good stories?

Absolutely. I mean, I don't have that many personal stories, but there used to be a chat group internally called Data Centers on Fire that would have exciting events happening and sometimes exciting videos. Yeah, I mean, I think at scale, lots of things that are very, very unexpected happen, and usually those are the combination of one thing fails and something else fails simultaneously, or in the—

You have a cascade of problems.

Yeah, you have a cascaded failure of some sort. Sometimes that means some software system stops working. Sometimes it means the bus bar overheats and you get too much power to the rack and it catches on fire. I mean, that's a much rarer thing, but you have to be prepared for this. And I think one of the things, even from the very earliest days of Google, is We have really focused on how do you build reliable systems out of unreliable parts. Yes. Right? Like in the earliest Google days, we were buying consumer machines without ECC memory. Not only not ECC, not even parity. We were buying consumer motherboards that didn't have like redundant power supplies. And you can do that if you can handle things at a higher level. And that's generally what we try to do. In all cases is.

I actually wanted to ask you about that, the ECC thing, because here, here's one of my favorite failure modes, if that's true, but you tell me. A distant supernova goes off, a cosmic ray hits a memory cell, and the 0 flips to a 1. Does that really happen?

Oh yeah, yeah, absolutely. I mean, alpha particles definitely can flip, uh, you know, DRAM state. We've actually observed this because we have monitoring data of how many ECC Errors and like single-bit errors that are corrected and 2-bit errors that are not corrected are happening in all of our machines. And you can actually see this where some clusters that are pointing in a particular direction on the Earth have a much higher rate for a, you know, a period like 10-minute period or something, and then the other ones on the other side of the Earth do not have that. So it's definitely something that happens.

How worried should I be? Because MacBook Pros don't have ECC memory as far as I know. Like, for, for one machine, is it so vanishingly unlikely that you shouldn't care, but for data center or?

I mean, for one machine, it's generally not too bad. I think they have parity, so at least they detect it typically if it's a single bit error.

Detection but not fixing.

ECC usually gives you single bit error correction and dual error detection. With that, you don't have to worry about it too much at a single machine level, but even at tens of thousands of machines, you'd have to start thinking about that. One of the things we did when we were using machines without even parity is we built an entire software-based checksumming system for large amounts of our data.

Doing it by hand.

Doing it by hand, essentially. And for crawling web pages and putting them in the index, if you detect that this particular record is corrupted, it's usually generally okay to just ignore that record.

Now I have something interesting for you. I call it lightning round. So please try to answer in one sentence. One word is okay. One, one sentence.

Can I make run-on sentences?

We'll see. We'll see. So I read that Jeff Dean's PIN code is the last 4 digits of pi. I give this one an 8 out of 10. So my question is, do you enjoy these Chuck Norris style jokes about you?

It could be true. I do enjoy them. I mean, it's an April Fool's joke gone awry by my colleagues in 2009. But it's very both flattering and kind of embarrassing.

I think, I think he felt the same way about them too. He, he enjoyed them too. Legend. All right, one big thing that you were wrong about and came around.

I think AI is going to influence healthcare quite dramatically, but I think it is harder, not necessarily for technical reasons, but for, you know, how do you actually get regulations in regulated industries that are super important and have all kinds of privacy constraints and safety concerns. But I think ultimately that will happen. It's just taking longer than I hoped. Yes. Because I think there's tremendous world benefit to do it, but we need to do it carefully and safely.

Vim or Emacs or something else? Hint, there's only one good answer.

Emacs. Was that it? Oh, no.

Look, I'm a Vim person, but I'm not— Maybe I'm an embarrassment of a Vim person because I looked at Emacs too and I was like, that's pretty cool too, but I don't want to learn both. It's just so much time.

Yeah, it's true. One can spend a lot of time customizing Emacs.

Yeah, the vimrcs I wrote up and then it never ends. Yeah. One problem that you solved, tried to solve many times, but have never been able to crack.

I think in some sense we still don't have an answer to how do you do continual learning appropriately. That's something I've thought about a little. I've dabbled a little bit with some techniques along with colleagues, but I think if we're able to crack that, it's going to be amazing, but it's not there yet. Last one.

Favorite Two Minute Papers episode.

Oh, yeah. I assume the transformer one was a good one.

Alright, alright, that's a good one. Okay, Jeff, I learned a lot today. Thank you so much.

Great chatting with you again.

Thank you so much.

Thank you.

Here you see me running the full Deepseek AI model through Lambda GPU cloud. 671 billion parameters running super fast and super reliably. This is insane. I love it. And I use it on a regular basis. Lambda provides you with powerful NVIDIA GPUs to run your own chatbots and experiments. Seriously, try it out now at lamda.ai/papers or click the link in the description.