Googles GEMINI 1.5 Just Surprised EVERYONE! (GPT-4 Beaten Again) Finally RELEASED!

So Google actually did just surprise everyone by releasing Gemini 1.5. And this is their latest iteration of their family of Gemini models. And this is a rather surprising model in the fact that it is able to do something incredible. Gemini 1.5 is the behemoth that is able to take up to three hours of video in a single context length. It's also able to take 222 hours of audio at once. It's also able to take up to 7 million words or 10 million tokens, with remarkable accuracy as well. Because lots of the time when we see new models appear, many of the times what happens is, is that their accuracy rates are very, very underwhelming. And Gemini is just outstanding because on these capabilities, their accuracy rate is around 99 to 100%. So that is absolutely incredible. This multimodal model is going to change everything. And let's take a look at some of the things you do need to know, because once you see a few videos, you're going to be truly surprised by how good this AI really is. So where is Gemini 1.5? Before we dive into some of the examples of how good this AI system is, where does it fit? So on the left hand side you can see Gemini Ultra, our most capable and largest model for highly complex tasks. And then in the middle you can see Gemini Pro and you've got two iterations, Gemini 1.0 and Gemini 1.5. And Gemini 1.5 is the model that was released today, which is essentially for larger, more tedious tasks that require a longer context length. So how much better is Gemini 1.5? So you can see that in text, in vision and in audio. Gemini 1.5 is better across the board. However, compared to the Ultra benchmarks, you can see that only on Vision and audio on the right hand side. But there are some areas where Ultra is slightly better. So overall, this model is substantially better than Gemini Pro 1.0, which is currently available. And in terms of Gemini Ultra, largely it's better than text and Envision. So across the board, this is a model that is most certainly more capable. Now, I'm going to be showing you guys one of these examples of Gemini Pro 1.5 reasoning across a 432 page transcript.

This is a demo of long context understanding, an experimental feature in our newest model, Gemini 1.5 Pro. We'll walk through a screen recording of example prompts using a 402 page PDF of the Apollo 11 transcript, which comes out to almost 330,000 tokens. We started by uploading the Apollo PDF into Google AI Studio and asked find three comedic moments, list quotes from this transcript and emoji. This screen capture is sped up. This timer shows exactly how long it took to process each prompt. And keep in mind that processing times will vary. The model responded with three quotes like this one from Michael Collins. I'll bet you a cup of coffee on it. If we go back to the transcript, we can see the model found this exact quote and extracted the comedic moment accurately. Then we tested a multimodal prompt. We gave it this drawing of a scene we were thinking of and asked, what moment is this? The model correctly identified it as Neil's first steps on the moon. Notice how we didn't explain what was happening in the drawing? Simple drawings like this are a good way to test if the model can find something based on just a few abstract details. And for the last prompt, we asked the model to cite the timecode of this moment in the transcript. Like all generative models, responses like this won't always be perfect. They can sometimes be a digit or two off. But let's look at the model's response here. When we find this moment in the transcript, we can see that this timecode is correct. These are just a few examples of what's possible with a context window of up to 1 million multimodal tokens in Gemini 1.5 Pro.

That demo right there was rather impressive, and there are a lot more examples in the paper, but. But let's take a look at another example of Gemini's massive capabilities on doing this with coding tasks.

This is a demo of long context understanding, an experimental feature in our newest model, Gemini 1.5 Pro, we'll walk through some example prompts using the 3js example code, which comes out to over 800,000 tokens. We extracted the code for all of the 3js examples and put it together into this text file, which we brought into Google AI Studio. Over here, we asked the model to find three examples for learning about character animation. The model looked across hundreds of examples and picked out these three One about blending skeletal animations, one about poses, and one about morph targets for facial animations. All good choices based on our prompt. In this test, the model took around 60 seconds to respond to each of these prompts. But keep in mind that latency times might be higher or lower as this is an experimental feature we're optimizing. Next, we asked what controls the animations on the Littlest Tokyo demo. As you can see here, the model was able to find that demo, and it explained that the Animations are embedded within the GLTF model. Next, we wanted to see if it could customize this code for us. So we asked show me some code to add a slider to control the speed of the animation. Use that kind of GUI the other demos have. This is what it looked like before on the original 3js site. And here's the modified version. It's the same scene, but it added this little slider to speed up, slow down, or even stop the animation on the fly. It used this GUI library. The other demos have set a parameter called animationspeed and wired it up to the mixer in the scene. Like all generative models, responses aren't always perfect. There's actually not an INIT function in this demo like there is in most of the others. However, the code it gave us did exactly what we wanted. Next, we tried a multimodal input by giving it a screenshot of one of the demos. We didn't tell it anything about this screenshot and just asked where we could find the code for this demo. Seen over here. As you can see, the model was able to look through the hundreds of demos and find the one that matched the image. Next, we asked the model to make a change to the scene, asking, how can I modify the code to make the terrain flatter? The model was able to zero in on one particular function called GenerateHeight and showed us the exact line to tweak below the code. It clearly explained how the change works. Over here, in the updated version, you can see that the terrain is indeed flatter. Just like we asked, we tried one more code modification task using this 3D text demo. Over here, we asked, I'm looking at the text geometry demo and I want to make a few tweaks. How can I change the text to, say, goldfish and make the mesh materials look really shiny and metallic? You can see the model identified the correct demo and showed the precise lines in it that need to be tweaked further down. It explained these material properties, metalness and roughness and how to change them to get a shiny effect. You can see that it definitely pulled off the task and the text looks a lot shinier now. These are just a couple examples of what's possible with a context window of up to 1 million multimodal tokens. In Gemini 1.5, you just saw Google's.

Gemini 1.5 Pro problem solving across 100,000 lines of code. My, oh my, this is something that is truly impressive. There is no other AI system out there that can do this with the Accuracy level of Google's Gemini. But now let's take a look at some of the multimodal prompting which is going to be used by a lot of standard users.

This is a demo of long context understanding, an experimental feature in our newest model Gemini 1.5 Pro. We'll walk through a screen recording of example prompts using a 44 minute Buster Keaton film which comes out to over 600,000 tokens in Google AI Studio. We uploaded the video and asked find the moment when a piece of paper is removed from the person's pocket and and tell me some key information on it. With the timecode, this screen capture is sped up and this timer shows exactly how long it took to process each prompt. And keep in mind that processing times will vary. The model gave us this response explaining that the piece of paper is a pawn ticket from Goldman & Co. Pawnbrokers with the date and cost and it gave us this timecode 12:01. When we pulled up that timecode we we found it was correct. The model had found the exact moment the piece of paper is removed from the person's pocket and it extracted text accurately. Next we gave it this drawing of a scene we were thinking of and asked what is the timecode when this happens? This is an example of a multimodal prompt where we combine text and image in our input. The model returned this timecode 1534. We pulled that up and found that it was the correct scene. Like all generative models, responses vary and won't always be perfect. But notice how we didn't have to explain what was happening in the drawing. Simple drawings like this are a good way to test if the model can find something based on just a few abstract details, like it did here. These are just a couple examples of what's possible with a context window of up to 1 million multimodal tokens to in Gemini 1.5.

Pro. That right there goes to show us how crazy this is. I think the only caveat to this is that it does take a little bit of time for it to go ahead and get the footage. But looking through a 44 minute video is absolutely incredible. And doing the reasoning across that is not to be understated because think about how long it would take a human to watch through an entire movie and find something from one frame. And whilst these demos are impressive all, what's even more impressive is the paper that they attach to this, which I read that shows a whole host of other incredible capabilities. So let's take a look at some of these stunning examples from the paper, which is going to show you all exactly how accurate this AI system really is and why Google are really leading the entire AI industry with Gemini 1.5 Pro. So there was this example and it stated, given a reference grammar book and bilingual word list, a dictionary, Gemini 1.5 is able to translate from English to Calamang with similar quality to a human who learned from the same materials. This is incredibly substantial because it means that not only is it able to get the entirety of this context length and a dictionary, it's able to reason and do translation based on new data, just like a human would. There was also this example right here that was another stunning example from the.

Paper.

This. And essentially it states that with the entire text of this really, really long novel, it's able to understand exactly what's happening just through a very simple drawing. And I'm no artist, but I'm sure you can all appreciate the fact that this drawing here isn't a very, very artistic one and it's really, really simple. So the genius here is of this system to be able to understand the nuance of what's happening in the image that then extrapolate that data out and of course reason to figure out exactly where that is. That is something that is unheard of in our current AI systems. And that's why I stated that this is truly game changing stuff. There was another example in the paper and I'm pretty sure you've already seen this one based in the video, but it just goes to show how crazy this is now in the paper. Some of the stuff I was looking at was really cool because there was this thing called called video haystack. Okay? And I'm going to break this down for you guys because it's truly fascinating on how accurate this really is and how they tested it goes to show how accurate this is. Now on the image you can see Gemini 1.5 Pro compared to GPT4 with vision. And unfortunately GPT4 with a vision can only take in 3 minutes in their API, whereas Gemini 1.5 can do 1 minute of content all up to the way of 3 hours. So essentially they set up a game. The computer, which is Gemini 1.5 has to find a secret message. And the secret word is needle. But this message was sneakily hidden in one tiny part of a very long movie. And this movie isn't just any movie. It was a three long hour movie made by sticking two copies of a documentary about the game of Go together. And this makes the video really long with lots of Places that could have hidden the message. Now it in this demo, what they did was they put the secret message only in one single frame of the video. That's just one picture out of a thousands and thousands that make up the entire movie. And of course there's a picture every single second. Now Gemini 1.5 pro's job was to watch this entire super long movie and find that one frame with the secret message. And all they did was they asked Gemini 1.5 what was this secret word, which is essentially like finding a needle in a haystack. And can you guess what? Gemini 1.5 was able to do it 100% of the time. So that is why the video capabilities, the video haystack capabilities are absolutely incredible. In addition, they did the same kind of game with the Gemini 1.5 Pro system and they did it with 22 hours of footage. And you can see here that it was able to do it up to 100%. And they compared it to Whisper and GPT4 Turbo with 12 minutes all the way up to 11 hours. And you can see the boxes in red that are essentially areas where it completely failed. In addition, they also did this with text Haystack. And this is where things start to get crazy because this was something that people didn't really think was possible. There were certain research papers that were stating that, you know, using Mambo was essentially going to be possible with this kind of, you know, output that we really wanted. If we really wanted to be able to get the retrieval that we wanted. We're going to have to use different architectures, but it seems like Google managed to figure out how to do that. And you can see right here that up to 10 million tokens they were able to get the accuracy up to around, I think it was 99%, a ridiculous level of accuracy. And that is something that is unheard of. A 1 million context length window is incredible. And of course compared to GPT4 Turbo, it's only a 128,000 contact length. So this is a truly game changing thing because imagine having 1 million tokens and then getting an AI system to be able to reason about the entirety of that or find certain things and then reason on that. That is going to be a hugely different thing. Now there were additionally some benchmarks. So we can see here the comparison between GPT4 Vision and Gemini 1.5 Pro on a one hour video QA and experiments are run by sampling one video frame per second and and linearly subsampling 16 or 150 frames and you can see here that Gemini 1.5 Pro outperforms GPT4 with Vision substantially, because not only does it outperform the 16 frames and the 150 frames, it does actually support for the video, whereas GPT4 with vision currently doesn't. Now, in addition, we can take a look at some of the benchmarks to see exactly what is going on. You can see right here that the core capability in terms of the actual families of model, like if we take a look at where Gemini Pro 1 point sits, we know that Pro 1.5 sits in the middle in terms of what the model is going to be able to do. So that leads me to believe that potentially we could be getting an ultra 2.0 or an ultra 1.5, but with these benchmarks, we can see that Gemini 1.5 is literally better across the board and it has a hugely increased contact length that's going to allow a lot more things. Now, if you want to take a look at some of the individual detailed benchmarks, you can, you can see the math ones right here. You can see that 1.5 Pro outperforms it on the heliswag, doesn't on the MMLU, does on the GSM8K, does on the math, doesn't on the rest of these, and does on the big bench. So across the board you can see that Gemini 1.5 Pro is really taking the cake here in terms of what is possible with an AI system. And of course in addition, the detailed benchmarks encoding, we can see that it's half and half in terms of these capabilities, but it is 77.7% of on the natural to code benchmarks. And one thing that I did want to find out was of course how they trained this model. And like Gemini 1.0 Ultra and Gemini 1.0 Pro, Gemini 1.5 Pro was actually trained on multiple 496 chip pods of Google's TPU or VU accelerators distributed across multiple data centers and on a variety of multimodal and multilingual data. Now, with that being said, are you excited for Google's family of models that are absolutely incredible? And are you going to be taking a look and using this model in Google's AI? And of course with things like the video capabilities that haven't been done by any other AI system before, are you excited to potentially use these to reason and figure out things in certain videos? Either way, I'm excited for Google to finally beef up the competition and make a more competitive AI space. But it will be interesting to see how other AI companies do respond, because right now it seems that Google is well in the lead. Benchmarks are here and the benchmarks are clear. And some of the AI systems right now don't even have some of these capabilities. So with that being said, if you did enjoy this, don't forget to leave your comment below on where you think Google is going to go.