Károly Zsolnai-Fehér walks through DeepMind's Gemini 1.5, which can process 1 to 10 million tokens, vastly more than GPT-4 Turbo's 128k. The demo shows it searching a 400-page Apollo 11 transcript, analyzing a 100k-line codebase, and understanding a 44-minute movie plus images. This could enable AI assistants that remember years of your medical history or film preferences.
Yes, of course I'll talk about this. Just a few days ago, OpenAI released Sora, their text to video AI that took the world by storm. But Sora was not the only thing that was released that day. DeepMind's AI assistant, Gemini 1.5 also appeared. And it offers something that, interestingly, not even the great GPT4 can do. Dear fellow scholars, this is two minute papers with Dr. And the story of Gemini 1.5 is a story of context windows. Hmm. What is a context window? Well, you see, we are living in the age of smart AI assistants. They help you with assignments, homework. It is a personalized teacher, a programmer, your assistant for daily planning, and so much more. However, there is something still missing. Imagine that you are in a conversation with someone and the context window means how long of a conversation this person can remember. If you have an assistant, you expect them to remember what you just talked about minutes ago, right? Well, let's see together if that is really the case. Hmm. GPT4 had an 8,000 token context window. So it remembers at most a few pages of a book. By the end of the book, it doesn't remember who even wrote it or how it started. And let's not even talk about movies. That is way too much data. Then a few months later, GPT4 Turbo appeared. We jump from 8k to 128k. Oh my. That is an entire book. We can talk about a book now. Fantastic. Now hold on to your papers, fellow scholars, because from 128,000, the new Gemini Pro offers a 1 to 10 million token window. Wait, what? That is insanity. Now let's see what that means in practice. Level one. Now, little AI, here is a 400ish page transcript of the Apollo 11, the American space flight that landed on the moon. Now find three of the funnier moments. Not a problem. And now a multimodal. What is this? Of course you fellow scholars already know, but does the AI know? Oh my, yes. One small step for man, one giant leap for mankind. And believe it or not, this was the easiest task it was given. Now, level two. Now let's give it a huge codebase with up to 100,000 lines of code. Now sift through it and show me parts of it that have to do with character animation. Not a problem. We can even run these animations. And now it can go from engineer to to teacher and explain to us how it works. Exactly. And then back to AI engineer again and code up a little slider to control the animation. I absolutely love this. How many people are there? Who are great teachers and engineers at the same time. Very few. Loving it. And now the Boss Level, Level 3. Check this out. Little AI. Forget the text, that's too easy. Let's step it up and watch a movie together. And look at that. We can ask questions about a 44 minute movie and it knows exactly the moment we are asking about. But it gets better. I did not think it could do this, but it did. Now here is a crude drawing. What is this? And when did it happen? Yep, that's the scene. This is incredible. And according to Google, it is going to be released soon. And here comes the kicker. Oh, this is just the Pro, not the Ultra version. Remember, the Pro version is the one that is free for everyone to use. The Ultra will likely be even better. Now wait a minute. We are experienced fellow scholars here. So. So we have questions. We found out recently that ChatGPT remembers. What does that mean? Well, have a Look. This is Claude 2.1, a competing chatbot from Anthropic. And it had a context window that was twice as big as GPT4 Turbo. However, wait, look at this. Context length is one variable, but being able to actually recall all this information is also important. And when asking questions about the document, it matters where in this document the information was. For instance, if it was early on, it might be forgotten by the time it reads until the end. But chatgpt remembered much more reliably. So our question is clear as day. What is the price to be paid for this million token context window? Is Gemini going to be as red as Claude 2 was here? Let's see. And whoa. That is insanity. It is not perfect. It will forget a few tiny details. But my goodness, look at that improvement. And all this barely a few days after the initial release of Gemini. Bravo. And yes, you're seeing correctly. We have a paper. Oh yes. So a deeper look is coming soon. Subscribe and hit the bell icon to not miss out on it. Now, the judge is always going to be us fellow scholars, the users. As soon as it is out, let's run our own experiments and find out how well it works in practice. But just think about it. Given this pace of progress, we will soon have essentially almost infinite tokens from for a context window. This is likely not even years away, but a year away. Possibly just months away. And then it will know not just one film, it will know the entire history of films. Better than Tarantino? Well, maybe not as well as Tarantino, but who knows? And these smart AI assistants will be lifelong partners that remember every single thing you talk to them about, they remember and can recall your bloodwork from a decade ago and have the knowledge of a good doctor to connect the dots of what happened, since they will be a lifelong teacher who is infinitely patient and gets smarter and smarter every month, an assistant that has access to almost the entirety of the knowledge base of humans. And just imagine all the good we could do if we had this assistant right at our fingertips. And all this for free or for very little. What a time to be alive. And yes, a closer look into SORA is also coming in a few days. I can't wait to show this to you. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations, web or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
