Google Launches Gemini 1.5 with ONE MILLION Token Context Window

Today on the AI Breakdown, Google has announced Gemini 1.5 Pro with an unbelievable million token context window. Before that on the Brief, another very long context window project specifically focused on coding has raised a fresh $100 million. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our Discord and our newsletter. Welcome back to the AI Breakdown Brief. All the AI headline news you need in five minutes. One of the big pursuits for AI developers right now is building software that can build software. This is important on a number of different levels. As we will discuss, it is not only about building a new type of tool, but also about giving AIs more broad reasoning power. Well, the specific context that we're starting with today is a new nine figure funding round for Magic, former GitHub CEO and now AI investor extraordinaire. Also, the guy behind the Vesuvius challenge that we talked about last week, Nat Friedman tweeted. Magic.dev has trained a groundbreaking model with many millions of tokens of context that performed far better in our evals than anything we've tried before. They're using it to build an advanced AI programmer that can reason over your entire code base and the transitive closure of your dependency tree. If this sounds like Magic, well you get it. Daniel and I were so impressed. That's Daniel Gross, his often investing partner. We are investing $100 million into the company today. The team is intensely smart and hardworking. Building an AI programmer is both self evidently valuable and intrinsically self improving. It's if this sounds interesting to you, consider joining them now. One thing I want to note is this many millions of tokens of context, which is something that we're going to get deeper into in the main part of the episode. But here obviously part of the big innovation is that with that big of a context window, as Nat puts it, the AI programmer can reason over the entire code base. The ability to ingest an entire code base all at once obviously seems like it could give Magic a very different set of capabilities. Now in their announcement post, Magic writes that they're working on frontier scale code models to build a coworker, not just a co pilot. They write things we believe Code generation is both a product and a path to AGI. AGI safety matters and is solvable. To build a great AI product we need to train our own frontier scale model. Transformers aren't the final architecture. We have something with a multimillion token context window. So there is a lot in just a few bullets there Code generation as both product and a path to AGI we're about to discuss Mark Zuckerberg's comments recently where Meta has come around to this thinking that code generation is indeed a central capability for the state of the art, not just from a feature parody standpoint, but because it improves proves general reasoning. The other bombastic note here is this idea that Transformers, which is of course what GPT is built on, the T stands for transformer, aren't the final architecture, and that they have a multi million token context window. Now, given all this, it's not surprising that Magic is very buzzy. However, they're not the only project thinking in similar ways. Anton Osika, who has been building GPT Engineer, which we've covered on the show before yesterday, tweeted happy Valentine's Day everyone, Launching a new AI startup out of Europe today. Lovable we're building software that builds software. How we are approaching it at a high level. Step 1 A tool for anyone to prototype web apps and collaborate via natural language on the same code base as developers. Step 2 Make fully autonomous code generation work on one single set of technology choices. Now if you go to their website lovable.dev the claim they make is that this is the last piece of software they write. We're building the software that builds other software. Why? The world they write is full of ambitious people who want to solve important problems. They over three decades, software has been the most significant tool to unleash the world's ambition. Still, less than 1% of the world has the skills required to create software. If we succeed, they write, everyone will have the same capabilities that entire product development teams at stellar tech companies have today. Right at their fingertips. We will unlock a new era of innovation, empowering dreamers everywhere to shape the world. We're reducing the barriers to build and staying committed to one to unleash human creativity on an unprecedented scale. Now, about a month ago, Mark Zuckerberg talked a bit about what was coming with llama 3, and some of these same themes were on display. He discussed how the company had changed its position on coding in AI, saying one hypothesis was that coding isn't that important because it's not like a lot of people are going to ask coding questions in WhatsApp, which is of course a big part of the Meta portfolio. Zuckerberg went on, though it turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge and just generally have more of an intuitive sense of logic. You remember there were a lot of headlines around this time last month about Zuckerberg's goal to build an open source AGI. Now the consequence of this shift to focus on coding inside llama 3 is not just to compete on that open source access, to compete at the state of the art. In General, Zuckerberg said Llama 2 wasn't an industry leading model, but it was the best open Source model. With llama3 and beyond, our ambition is to build things that are at the state of the art and eventually the leading models in the industry. Now of course there are big debates around what the net impact of this type of self generating software can do. Holding aside all the arguments about how this is where AI goes off the rails and all the safety risks which is worthy of an entire episode on its own, even the debate around whether AI is going to replace programmers has a lot of interesting dimensions. Ahmad, the CEO at Mercury, tweeted about this yesterday, writing why AI is Not going to Replace Programmers When I was in college studying computer science in 2005 I was told that outsourcing to India will remove the demand for programmers. It was a real fear. AI replacing coders I think is based on a similar misconception. He goes on. This misconception is rooted in a misunderstanding of what programmers do that non programmers and even many engineers have. Programming is sometimes seen as a science where you have a very specific spec and you convert that through a series of repetitive work to code. People perceive it similarly to high school level math. You get an equation and you solve it and get an answer. And to be fair, that is what learning to program and entry level programming is like. So it's easy to see why people have this misconception. In reality, a lot of programming is 1 understanding customers internal or external 2 converting that understanding into potential implementation 3 thinking about integrating seamlessly into existing code 4 thinking about building in a scalable way for future recs 5 etc. A lot of it involves having deep taste and applying that taste against the customer need. This part is much more art than science even. So what he says AI will actually do 1 make programmers more efficient at the repetitive part 2 enable more people to be programmers 3 turn more of the world into bespoke software 4 increase the demand for great programming artists, elevating the craft even more. Ultimately, he concludes, if you're thinking of learning to code, don't be put off by the AI will replace programmers meme and just do it. Another take on this that we've often heard from Sam Altman is that AI won't replace programmers because the demand for things that programmers build is just going to go up. Consequently, with our capacity to actually deliver against that demand. Anyways, it's a super interesting discussion, and honestly could have been a full episode rather than just a brief but before we get to the main part of this episode, let's actually cover a few additional stories Slack becomes the latest Web 2.0 software to deeply integrate AI announcing that Slack AI has arrived there are a bunch of different parts of this personalized search results, channel recaps and thread summaries. So for example, if you've been gone for a while, Slack AI can summarize what you've missed. Which for anyone working with me on Slack would be a very useful thing given how many messages I send. And of course, Slack and its owner Salesforce are promising that users will control their data, that they'll have more granular ability to implement tools, et cetera, et cetera. Interesting news out of Amazon Researchers at Amazon have apparently trained the largest ever text to speech model, which of course matters given that the company is so invested in things like Alexa. And they claim that they're seeing what they call emergent qualities, which is improving that model's ability to speak naturally, even in the context of complex sentences. The new model is called Big Adaptive Steamable TTS with Emergent Abilities or Base TTS. TechCrunch writes at 980 million parameters, base large appears to be the biggest model in this category. The largest version was trained on 100,000 hours of public domain speech, 90% of which was in English, with the remainder in German, Dutch and Spanish. Areas where this large model improved from previous models, they said, include compound nouns, emotions, foreign words, paralinguistics, AKA readable non words like sh, punctuation questions and syntactic complexities. The authors of a paper wrote. These sentences are designed to contain challenging parsing garden path sentences, placing phrasal stress on long winded compound nouns, producing emotional or whispered speech, or producing the correct phonemes for foreign words like chi or punctuations like at, none of which Basetts is explicitly trained to perform. TechCrunch writes. Such features normally trip up text to speech engines which will mispronounce, skip words, use odd intonation, or make some other blunder. But while base TTS still had some trouble, it apparently did far better than its contemporaries. So once again we have an advanced model that is working in ways we just don't quite understand. Over on Wall Street, a number of companies saw their stock rise after Nvidia disclosed that it was an investor. Those investments included Recursion Pharmaceuticals, Conversational voice assistant developer Soundhound AI and fellow AI chip designer Arm so friends, AI's White Hawk Streak continues. But for us, that will end the AI breakdown. Brief. Thanks for listening or watching as always. And up next, the main AI breakdown. Welcome back to the AI Breakdown. Today we had some unexpected and exciting news out of Google CEO Sundar Pichai Tweets In December we launched Gemini 1.0 Pro. Today we're introducing Gemini 1.5 Pro. This next gen model uses a mixture of experts MOE approach for more efficient training and higher quality responses. Gemini 1.5 pro are mid sized model will soon come standard with a 128k token context window. But starting today, developers and customers can sign up for the limited private preview to try out 1.5 Pro with a groundbreaking and experimental 1 million token context window. The 1 million tokens feature unlocks huge possibilities for devs upload hundreds of pages of text, entire code, repos and long videos and let Gemini reason across them. It's still experimental and early and we'd love your feedback now. Context windows are something that people have been talking about ever since ChatGPT launched. It refers to the number of tokens that any given model can engage with at a particular time. The larger that window, the more coherent an LLM can reason across a bigger volume of text to get a sense of how much of a difference it is to be talking about 128k in million token context windows. People were incredibly excited to see ChatGPT move from 8k up to 32k, so obviously we're talking about significantly longer than that. Anthropic's Claude has also used longer context windows to try to compete, although of course one of the things that people watch out for is whether performance starts to degrade when you're actually using longer context windows. Still, the initial response has been incredibly excited. Lior at Alpha Signal writes just in Google releases Gemini 1.5, a powerful Moe model. It's a huge breakthrough. The model has the longest context window ever seen, 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, 1 or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. Jeff Dean, the chief scientist at Google DeepMind and Google Research, says one of the key differentiators of this model is its incredibly long context capabilities supporting millions of tokens of multimodal input. The multimodal capabilities of the model mean you can interact in sophisticated ways with entire books, very long document collections, code bases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more. Now, given that Google has kind of a history of announcing things before they make them available, another thing that people were very excited about is that early testers were actually allowed to start using this 1 million token context window at no cost during the testing period. Writing about the technical approach to this model on their announcement post, Google says Gemini 1.5 is built upon our leading research on transformer and MOE architecture. While a traditional transformer functions as one large neural network, MOE models are divided into smaller expert neural networks. Depending on the type of input given, MOE models learn to selectively activate only the most relevant expert pathways in its neural network. This specialization massively enhances the model's efficiency. Google has been an early adopter and pioneer of the MOE technique for deep learning. Our latest innovations in model architecture allow Gemini 1.5 to learn complex tasks more quickly and maintain quality while being more efficient to train and serve. These efficiencies are helping our teams iterate, train and deliver more advanced versions of Gemini faster than ever before. Another performance piece that many people noticed was this one quote Gemini 1.5 pro maintains high levels of performance even as its context window increases. In the Needle in a Haystack evaluation, where a small piece of text containing a particular factor statement is purposefully placed within a long block of text, 1.0 Pro found the embedded text 99% of the time in blocks of data as long as 1 million tokens. Jeff Dean again tweeted about this a little bit, saying needle in a haystack tests out of 10 million tokens. First, let's take a quick glance at a needle in a haystack test across many different modalities to exercise Gemini 1.5 Pro's ability to retrieve information from its very long contexts. He points out that the results are almost entirely good, meaning 99.7% recall even out to 10 million tokens. That's in audio, video and text. Avi Schiffman, who's working on the AI wearable tab, writes, Perfect recall with 10 million tokens. The Messiah has arrived and its name is Google. Developer Nick Dobas responded, what? That's crazy. Okay, Google, you might have a chance. And indeed, this is the sentiment that I am seeing all over Twitter. X Today, in many ways, this Gemini 1.5 Pro announcement has developers and people who are deep in this technical part of this space more excited than Gemini Advanced even did. There is so much sentiment like wow, Google is really shipping. Which is of course a complete switch from the discussion around them last year. The Verges piece about this is Gemini 1.5 is Google's next gen AI model and it's already almost ready to the next version of Google's model is better and faster, sure, but it also has one pretty remarkable new party trick. Now the Verge does a good job of contextualizing what 1 million tokens means in this multimodal model. Apparently, Sundar Pichai told the Verge. That means you can fit the entire Lord of the Rings film trilogy into that context window. As you might imagine, this has a lot of people saying, Is OpenAI starting to fall behind? Where is GPT5? Is this going to put increased pressure on them? Well, interestingly, we did get some new leaks about some things going on inside OpenAI that do in some ways relate to Google as well. Last week we heard that they were working on AI agent startups, but now, according to people with knowledge of the project, OpenAI is also working to develop a web search product that would bring them into direct competition with Google and Of course other AI search tools like Perplexity, writes the Information OpenAI has been developing a web search product that would bring the Microsoft backed startup into more direct competition with Google. Now details are scant right now. The information source said that the search service would be partly powered by Bing, but it wasn't clear whether it would be a separate product from ChatGPT or in some ways embedded into ChatGPT. Now this is a story that I could see going in a bunch of different ways. It could be as simple as a slightly different interface and a set of expanded features around what they already have, which is effectively what Perplexity has done. Although of course done in such a valuable way that many people have shifted their research behavior entirely to Perplexity. Or it could be a fundamentally different and expanded product. In any case, it feels to me like there is a lot brewing and bubbling behind the scenes at OpenAI, and it'll be interesting to see what actually pans out. Now, staying on the topic of OpenAI for just a minute, they just published a piece called no, Sam Altman isn't Raising Trillions of Dollars for Chips, and basically what they're saying is that while the Wall Street Journal report from last week implied that he was actively seeking that much capital for some sort of joint venture instead. That's Altman's calculation of the total cost of everything from real estate to power for data centers, that it would actually achieve the type of objective that he wanted to see. So it's not just some specific company that he's out trying to raise money for. To me, it's pretty interesting to see Google putting the pedal to the metal so hard and finally starting to reclaim some ground against OpenAI, which had been so lost throughout the course of 2023. Will this mean that we'll see an increased urgency from OpenAI to release more advanced models or new types of products? Or are they comfortable just continuing to go at their own pace? That is certainly what I will be watching and I will share with you as I get any clarity on it. For now, though, that is going to do it for today's AI breakdown. Until next time, peace.