Jeff Dean traces 15 years of AI progress at Google through two forces: more compute and data, plus better architectures. Transformers beat LSTMs by removing sequential bottlenecks. Pathways abstracts tens of thousands of TPUs into one virtual computer, handling failures automatically.
Jeff Dean just left Google. He was employee number 30, and there was a lot of news around this, which is, is Google AI dying? And a lot of famous names that have left Google DeepMind and their AI initiatives recently, Jeff Dean being one of them. He was the chief scientist for Google Research and Google DeepMind. This is his talk on important trends in AI and how did we get here. And I'm just going to summarize the talk. So a few points that he makes. Number one, AI progress has been driven by scale. and algorithms, not just bigger models. And he argues that the last 15 years of AI came from 2 multiplicative forces, dramatically more compute and data, and dramatically better architectures. That leads to not just incremental additive gains, but also compounded orders of magnitude step function improvements. Google's AI infrastructure evolved alongside the model. So Dean kind of walks through Google's progression from distributed network training with their Disbelief tool, unsupervised learning on millions of YouTube frames, Word2Vec as a tool, sequence-to-sequence models, and eventually the TPU program. And then he goes on to explain transformers and sparse models and pathways and how that fundamentally paved the road for larger scale AI and changes there in industry. So he explains why transformers replace LSTMs by eliminating sequential bottlenecks through attention mechanisms. He also discusses sparse MOE or mixture of experts models that activate only relevant portions of a network. Reducing compute while increasing capacities, and that Google's Pathway systems abstracts tens of thousands of TPUs into one virtual computer, and that automatically handles hardware failures during massive distributed training. So really cool behind the scenes peek at like, what does massive distributed training look like? He then talks about modern LLM capabilities coming from many complementary innovations, not like a single breakthrough. Really great talk. Would highly recommend anybody who's an AI builder that wants to kind of know The how did we even get here as an industry, what is possible today, and, and where's the industry headed. This is about 6 months old, but I find that, well, a lot of what he says still sticks, and the principles and thought leadership, I definitely, I see that evolving in the landscape, and I'm really excited to see what he does next here.
