Gemini 1.5 and The Biggest Night in AI

You could call tonight the triumph of the Transformers, or maybe the battle for attention with two thunderous developments within a handful of hours of each other, one from Google, DeepMind and then OpenAI. But ultimately the bigger picture is this. The exponential advance of artificial intelligence and its applications shows no sign of slowing. If you were hoping for a quiet year, tonight just shattered that peaceful vision. Of course I could have done a video on either Gemini 1.5, which is arguably as of tonight, the most performant language model in the world, or Sora, the text to video model OpenAI, released shortly after Google, possibly as a purposeful grab of the spotlight. Possibly coincidentally, both developments are game changing. But with the technical paper of Sora due out later today, it allows us to give Gemini 1.5 its due attention. Yes, I've read all 58 pages of the technical paper, as well as four papers linked in the appendices. In my search for how it got the results it did, I've got 62 notes, so let's dive in. Here is the big development Gemini 1.5 can recall and reason over information across millions of tokens of context. Or in another example, they gave 22 hours of audio or three hours of low frame rate video, or six to eight minutes of normal frame rate videos. I don't want to bury the headline. We're talking about near perfect retrieval of facts and details up to at least 10 million tokens. Performance did not dip at 10 million tokens. Indeed, the trend line got substantially better. For reference in text, 10 million tokens would be about 7.5 million words. To wrap your head around how many words 10 million tokens is, that's around 0.2% of all of English language Wikipedia. And again just for emphasis, we're talking at least 10 million tokens. Now. Admittedly in the blog post and paper they do talk about latency trade offs with that many tokens. And no, in case you're wondering, Gemini 1.5 isn't currently widely available just to a limited group of developers and enterprise customers. In case that puts you off though, Google promised this Significant improvements in speed are also on the horizon. But before we get back to the paper, how about a demo so you can see Gemini 1.5 Pro in action?

This is a demo of Long context understanding, an experimental feature in our newest model, Gemini 1.5 Pro, we'll walk through a screen recording of example prompts using a 402 page PDF of the Apollo 11 transcript, which comes out to almost 330,000 tokens. We started by uploading the Apollo PDF into Google AI Studio and asked find three comedic moments, list quotes from this transcript and emoji this screen capture is sped up. This timer shows exactly how long it took to process each prompt and and keep in mind that processing times will vary. The model responded with three quotes like this one from Michael Collins. I'll bet you a cup of coffee on it. If we go back to the transcript we can see the model found this exact quote and extracted the comedic moment accurately. Then we tested a multimodal prompt. We gave it this drawing of a scene we were thinking of and asked what moment is this? The model correctly identified it as Neil's first steps on the Moon.

I'll feature more demos later in the video, but notice that the inference time in this sped up video wasn't that bad. And yes, for anyone paying attention, this is Gemini 1.5 Pro. What that means is that any results you're seeing will soon be improved upon by Gemini 1.5 ultra. Remember that we have Gemini Nano, then Gemini Pro, the medium sized model, and finally Gemini Ultra. Well, how did Google DeepMind achieve this feat? They say in the introduction that the model incorporates a novel mixture of experts architecture as well as major advances in training and serving infrastructure that allows it to push the boundaries of efficiency, reasoning and long context performance. In case you're wondering, that long context again refers to that huge volume of text, image, video and audio data that Gemini 1.5 can ingest. But I will admit that when I first read this introduction I thought they have used the Mamba architecture that was billed as the successor architecture to the transformer, and I did a video on it on the 1st of January. It too achieved amazing results in long context tasks and outperformed the transformer. The transformer is of course the T in GPT. However, by the time I finished the paper it was pretty clear that it wasn't based on Mamba and it took me a little while to figure it out and reading quite a few papers cited in the appendices, but I think I've got a pretty good guess as to what the architecture is based on. Anyway, what is the next interesting point from the paper? Well, Google does confirm that Gemini 1.5 Pro requires significantly less compute to train than 1.0 Ultra. So it's arguably better than Ultra and we'll see the benchmarks in a moment, but requires significantly less compute to train. That is maybe why Google DeepMind were able to get out Gemini 1.5 Pro so soon after announcing Gemini 1.0. And don't forget that Gemini 1.0 Ultra as part of Gemini advance was only released to the public a week ago. That's when my review video came out. Google, by the way, gives you a two month free trial of Gemini advance, and I now think part of the reason for that is to give them time to incorporate Gemini 1.5 Pro before that free trial ends. And here is the first bombshell graphic in the paper. This is the task of finding a needle in a haystack across text, video and audio modalities. A fact or passcode might be buried at varying depths of sequences of varying lengths. As you can see, for text these lengths went up to 10 million tokens. For audio it was up to 22 hours and video up to 3 hours. The models would then be tested on that fact. As you can see, the performance was incredible with just five missed facts for reference, I went back to the original benchmark, which I've cited in two previous videos to compare. Here is GPT4's performance at up to only 128,000 tokens. As you can see, as the sequence length gets to around 80,000 words or 100,000 tokens, the performance, especially midway through the sequence, degrades. At the time, Anthropic's Claude 2.1 performed even worse, although they did subsequently come up with a prompt engineering hack to reduce most of these incorrect recalls. But are you ready for the next bombshell? They see Gemini 1.5 pro outperforming all competing models across all modalities, even when these models are augmented with external retrieval methods. In the lingo, that's rag in layman's terms, that's grabbing relevant text to assist them in answering the questions. Of course, with a long context, Gemini 1.5 Pro is just simply ingesting the entire document. And now for one of the key tables in the paper, you might be wondering, and indeed Google DeepMind were wondering, if this extra performance in long context tasks would mean a trade off for other types of task text, vision and audio tasks that didn't require long context. The answer was no. Gemini 1.5 pro is better on average compared to 1.0 pro across text, vision and audio. In other words, it hasn't just got better at long context, it's got better at a range of other tasks. It beats Gemini 1.0 Pro 100% of the time in text benchmarks and most of the time in vision and audio benchmarks. And wait, there's more. It also beats most of the time, Gemini 1.0 ultra in text benchmarks of Course, at this point, if you only care about standard benchmarks and not about long context, it's more or less a draw. With Gemini 1.0 Ultra, it has a win rate of 54.8% and it would also be pretty much a tie with GPT4. But I think it's fair to say that once you bring in long context capabilities, it is now indisputably the best language model in the world. I should caveat that with it's the best language model that is accessible to some people out there. Of course, behind the scenes Google DeepMind have Gemini 1.5 Ultra which would surely be better than 1.5 Pro. And I know what you're thinking, this is Google, so maybe they use some kooky prompting method to get these results. Well, I've read the entire paper and it doesn't seem so to me. These are genuine like for like comparisons to 1.0 Pro and 1.0 Ultra. Now I have looked in the appendices and the exact wording of the prompts may have been different to e.g. gPT4. It does seem borderline impossible to get perfectly like for like comparisons. But from everything I can see this does seem to be a genuine result. Now before you get too excited, as you can see the Benchmark results for 1.5 Pro in non long context tasks is pretty impressive, but we're not completely changing the game here. They haven't come up with some sort of architecture that just crushes in every task. We're still dealing with a familiar language model for most tasks. But before I give you my architecture speculations, time for another demo, this time analyzing a one frame per second 44 minute movie.

This is a demo of long context understanding, an experimental feature in our newest model, Gemini 1.5 Pro. We'll walk through a screen recording of example prompts using a 44 minute Buster Keaton film which comes out to over 600,000 tokens in Google AI Studio. We uploaded the video and asked find the moment when a piece of paper is removed from the person's pocket and tell me some key information on it. With the timecode this screen capture is sped up and this timer shows exactly how long it took to process each prompt and keep in mind that processing times will vary. The model gave us this response explaining that the piece of paper is a pawn ticket from Goldman & Co. Pawnbrokers with the date and cost and it gave us this timecode 12:01. When we pulled up that timecode we found it was correct. The model had found the Exact moment the piece of paper is removed from the person's pocket and it extracted text accurately. Next, we gave it this drawing of a scene we were thinking of and asked what is the timecode when this happens, the model returned this time 1534. We pulled that up and found that it was the correct scene. Like all generative models, responses vary and won't always be perfect. But notice how we didn't have to explain what was happening in the.

Drawing. Gemini being multimodal from the ground up is really shining here and I do think we have to take a step back and say wow. At the moment this might mean six to eight minutes of a 24 or 30 frames per second video on YouTube. But still, successfully picking out that minor detail in that short of a time is truly groundbreaking. Given that Google owns YouTube, you will soon be querying, say AI explained videos. Okay, time for my guess about how they managed it in terms of architecture. Well, first they say simply it's a sparse mixture of expert transformer based model. Those are fairly standard terms described in other videos of mine. But then here's the key sentence. Gemini 1.5 Pro also builds on the following research and the language model research in this broader literature. Now I had a look at most of these papers, particularly the recent ones, and one stood out. This paper by Zhang et al came out just over a month ago. And remember Google says that Gemini 1.5 Pro builds on this work. Now building on something that came out that recently is pretty significant. Of course Google have their own massive body of literature on sparse mixture of experts and indeed invented the transformer architecture. But this tweet tonight from one of the key authors of Gemini 1.5 does point to things developing more rapidly. Recently Pranav Shyam said this just a few months ago, Nikolai Denis and I were exploring ways to dramatically increase our context lengths. Little did we know that our ideas would ship in production so quickly. So yes, Google has work going back years on sparse mixture of expert models. And yes, too many people underestimated the years of innovations going on quietly at Google. In this case for inference. But for the purposes of time, this is the paper I'm going to focus on. This is the one by Zhang released around a month ago. It's of course mixture of experts from Mistral AI, that's that brand new French outfit with a multi billion euro valuation. And no, I don't just think it's relevant because of the date and the fact it's sparse and a mixture of experts. Mixture of experts in a nutshell being when you have a bigger model comprised of multiple smaller blocks or experts. When the tokens come in they are dynamically routed to just two in most cases relevant experts or blocks. So the entire model isn't active during inference. It's lightweight and effective. But no, that's not the reason why I focused on this paper. It's because of this section 3.2 Long Range Performance Mistral managed to achieve 100% retrieval accuracy regardless of the context length and also regardless of the position or depth of the password. Of course, Mistral only proved that up to 32,000 tokens and Google I believe have taken it much further. That's my theory. Let me know in the comments if you think I'm right. Google do say they also made improvements in terms of data optimization and systems, but if you're looking for more info on compute or the training data set, good luck. Other than saying that the compute is significantly less than Gemini 1.0 Ultra and that Gemini 1.5 Pro is trained on a variety of multimodal and multilingual data, they don't really give us anything. Well, I tell a lie, they do say that it was trained across multiple data centers. So given that a data center maxes out around 32,000 GPUs and I know Google uses TPUs, but that still gives us a sense about the sheer scale of Google Gemini's compute. And there is one more task that Google DeepMind really want us to focus on. Admittedly it is very impressive. They gave Gemini 1.5 pro a grammar book and dictionary, 250,000 tokens in total from a super obscure low resource language. The language is Calamang and I had never heard of it. They take pains to point out that none of that language was in the training data set. And so what was the result? Well, not only did Gemini 1.5 Pro Crush GPT4, it also did as well as a human who had learned from the same materials. Now we're not talking about someone from that region of Papua New Guinea. The reason a human was used for comparison was to make that underlying point. Models are starting to approach the learning rate, at least in terms of language of a human being. And don't forget this factors in data efficiency. Same amount of data, similar result. Next up is what I believe to be a fascinating graphic. It shows what happens as a model in blue Gemini 1.5 Pro is fed more and more of a long document and of a code database and the lower the curves go, the more accurate the model is getting at predicting the next Word what happens to December's Gemini Pro as you feed it more and more tokens? Well, it starts to get overwhelmed both in terms of code and for long documents. As the paper says, that older model, and I hesitate to call it older because it's just two months ago, they're unable to effectively use information from the previous context and are deteriorating in terms of prediction quality. But with Gemini 1.5 Pro, the more it's fed, the better it gets. Even for a sequence of length a million for documents, the or 10 million for code, it's quote remembering things from millions of lines of code ago to answer questions. Now I think it's significant that when we get to sequence lengths of around 5 to 10 million, the curve actually dips downward. It no longer follows the power law trend. That would suggest to me that if we went up to 100 million, the results would be even more impressive. Here's what Google have to say. The results above suggest that the model is able to improve its predictions by finding useful patterns even if they occurred millions of tokens in the past as in the case of code. And to summarize this, we already knew that lower loss could be gotten from more compute. It's a very similar curve. But what's new is that the power law is holding between loss and context length as shown above. They say from inspecting longer code token predictions closer to 10 million, we see a phenomena of the increased context occasionally providing outsized benefit that could be due to repetition of code blocks. They think this deserves further study and may be dependent on the exact data set used. So even Google aren't fully sure what's causing that dip. Now we all know that OpenAI kind of trolled Google tonight by releasing Sora so soon after Gemini 1.5 Pro. But on this page I feel Google were doing a little bit of trolling to OpenAI. First we have this comparison again of retrieval and they say they got API errors after 128,000 tokens. Well of course they knew that because GPT4 Turbo only supports 128,000 tokens. I think they kind of wanted to say that after this length we crush it and with them you just get an error code and the next bit of trolling comes here. These haystack challenges where they secrete a phrase like this. The special magic quote city number is quote. With this the model has to retrieve the correct city and number which is randomized. But that phrase could have been hidden in any long text and they chose the essays of Paul Graham. Now yes, this is almost certainly coincidental, but Paul Graham was the guy who fired Sam Altman at Y Combinator. Sam Altman disputes that it was a firing for audio, it's the same thing. Even when they break down long audio into segments that Whisper can digest, which are then transcribed and fed to GPT4 Turbo. The difference is stark. Before you think though, that Gemini 1.5 Pro is perfect at retrieval, what happens when you feed in 100 needles into a massive haystack? Well, in that case it still massively outperforms GPT4 Turbo. But can recall, as you can see, 60, 70, 80% of those needles. It is not a perfect model. And no, we don't have AGI. And at this point Google does state that retrieval is not the same as reasoning. They basically beg for harder benchmarks, ones that require integrating disparate facts, drawing inferences or resolving inconsistencies. Essentially reasoning. If you want to know more about how reasoning, some would say is the final holy grail of large language models, do check out my Patreon AI Insider. I have around a dozen videos and podcasts up as of today. In fact, it was just today that I released this video on my patreon. It's a 14 minute tour of deepfakes and features, interviews and exclusives and more. If you are a student or retired, do email me about a potential small discount. Now for the final demo in.

Coding, we'll walk through some example prompts using the 3js example code, which comes out to over 800,000 tokens. We extracted the code for all of the 3js examples and put it together into this text file which we brought into Google AI Studio. Over here, we asked the model to find three examples for learning about character animation. The model looked across hundreds of examples and picked out these three. Next we asked what controls the animations on the littlest Tokyo demo. As you can see here, the model was able to find that demo and it explained that the animations are embedded within the GLTF model. Next we wanted to see if it could customize this code for us, so we asked show me some code to add a slider to control the speed of the animation. Use that kind of GUI the other demos have. This is what it looked like before on the original 3js site, and here's the modified version. It's the same scene, but it added this little slider to speed up, slow down, or even stop the animation on the.

Fly. Again with audio, Gemini crushes Whisper. It has a significantly lower word error rate and for video it was pretty Funny, they had to invent their own benchmarks because the other ones were too easy or informal language. To bridge this evaluation gap, we introduce a new benchmark that was testing that incredible feat we saw earlier of picking out key details from long videos. Now, to be clear, despite the demos looking good and beating GPT4V, we're still not close to perfect. Just because Gemini 1.5 Pro can see across long context and watch long videos doesn't mean it's perfect at answering questions. Remember that recalling facts is not the same as reasoning or getting 100% on multiple choice questions. I also found this part of the paper quite funny where they tried to highlight the extent of trade offs of switching architecture if it exists. And the problem was they couldn't find any across the board. 1.5 Pro was just better than 1.0 Pro, whether that was math, science coding, multilinguality instruction following image understanding, video understanding, speech recognition or speech translation. Of course, it's obligatory at this point for me to ding them about the accuracy level of their MMLU benchmark test. For Gemini 1.5 Pro they say for math and science it's 1.8% behind Gemini 1.0 Ultra. But how meaningful is that with this many errors just in the college chemistry section of the MMLU? Buried deep is one admission that 1.5 Pro doesn't seem quite as good at OCR. That's optical character recognition, in other words, recognizing text from an image. But Google cloud vision is state of the art anyway at OCR and soon enough surely they're going to integrate that. So I don't see OCR being a long term weakness for the Gemini series. And it's hard to tell, but it seems like Google found some false negatives in other benchmarks and so the performance there was a lower bounding the model's true performance. And they complain, as I did in my original Smart GPT video, that maybe we need to rely more on human evaluations for these datasets and that maybe we should deviate from strict string matching. And there was this quite cute section in the impact assessment part of the paper. So what are the impacts going to be of Gemini 1.5 Pro? Well, they say the ability to understand longer content enhances the efficiency of individual and commercial users in processing various multimodal inputs, but that besides efficiency, the model enables societally beneficial downstream use cases, and they foresee Gemini 1.5 Pro being used to explore archival content that might potentially benefit journalists and historians. Suffice to say, I think this is somewhat underplaying the impact of Gemini 1.5 Pro just for one I think it could transform how YouTube works. Or another obvious one. What about long term quote relationships with chatbots? GPT4's new memory feature, which seems to me like an only slightly more advanced custom instruction, pales in comparison to Gemini 1.5's potential. You could have discussions lasting for months with Gemini and it might remember a detail you said back, say six months ago. That seems to me true memory and might encourage a kind of companionship for some people with these models on safety. Without giving too much detail, they just say it's safer than 1.0 Pro and 1.0 Ultra. But later they do admit two things. First, Gemini 1.5 Pro does seem a little bit more biased. It's probably a bit harder for the model to be anti stereotypical when it remembers so much. Also, and I know this is going to annoy quite a few people, it has a higher refusal rate that's on questions that should be both legitimately refused and not legitimately. In other words, they should have been answered. Of course, by the time the model actually comes out, we'll have to see if this is still the case, but you just have to take a look at my Gemini Ultra review to see that at the moment the refusals are pretty extreme. This could honestly be a key sticking point for a lot of people we're drawing to an end here, but just a quick handful of further observations. Do you remember that trick with ChatGPT where you submit the letter A with a space say 500 times and it regurgitates sometimes its training data? Well, apparently that works Also on Gemini 1.5 Pro. Thing is, you have to manually repeat that character many more times, up to a million times. But with those long prompts they do admit that it becomes easier to obtain its memorized data. I presume that's the kind of thing that Google DeepMind are working on before they release Gemini 1.5 Pro. And one more key detail from the blog post that many people might have missed. When Gemini 1.5 Pro is released to the public, it's going to start at just 128,000 token context window. I say just, but that's still pretty impressive. And it seems to me, based on the wording of the following sentence, that even that basic version won't be free. They say we plan to introduce pricing tiers that start at the standard 128,000 context windows. So anyone hoping to get Gemini 1.5 for free seems to have misplaced hope and then there's going to be tiers going up to 1 million tokens. I'm not sure how expensive that 1 million token tier will be, but I'll probably be on it. Notice that we probably won't be able to buy going up to 10 million tokens, but I do want to end on a positive note for Google. There was one thing I missed out from my review of Google Gemini Ultra and I want to make amends and that is its creative writing ability. Gemini 1.0 Ultra is simply better at creative writing than GPT4. And of course we're not even talking about 1.5 Ultra. How so? Well, Gemini varies its sentence length. We have short sentences like this. Dibbons never really listened. We also get far more dialogue, which is just much more realistic to real creative writing. There's a bit more humor in there. Whatever they did with their writing dataset, yet they did better than OpenAI. GPT4 stories tend to be far more wordy, a lot more tell, not show. And I'm actually going to go one further and prove that to you. When you put GPT4 story into two state of the art AI text detectors, that's GPT0 and binoculars, which is a new tool. Both of them say most likely AI generated. GPT0 puts it at 97% for Claude. We also get most likely AI generated, although GPT0 erroneously says it's only 12% likely AI generated. That's Claude's story. It's way too much to get into this video now, but remember, binoculars is state of the art compared to GPT0. But here's the punchline. This is Google Gemini story we get from GPT00% chance of being AI generated. And even the state of the art binoculars gives it most likely human generated. And I think this proves two points. First, Gemini is definite better at creative writing and making marketing copy by the way, but too long to get into here. And second, don't put your faith in AI tech detectors, especially not in the age of Gemini. If you want to learn more about detecting AI and deepfakes, of course I refer you back to my deepfakes video on my Patreon AI insiders so that is Gemini 1.5 Pro. And yes, this does seem the most significant night for AI since the release of GPT4. As I said in my video on January 1st, AI is still on an exponential curve. 2024 will not be a slow year in AI and for as long as I can, I will be here to cover it all. Thank you so much for watching and have a wonderful.