Google I/O 2024: New AI That Looks Like Magic!

So while we are all stunned by OpenAI's new GPT4O, Google also held their own keynote with lots of new goodies. And one of the highlights of the show was Sir Demis Hassabis unveiling Project Astra, a universal assistant that is with you all the time. You can ask what that part of the speaker is called. Draw an arrow and it will tell you.

That is the tweeter. It produces high frequency sounds.

Or you can. You can point to a piece of code and ask it what it does and it will tell you.

This code defines encryption and decryption functions. It seems to use AESCBC encryption to encode encryption.

You've seen this feature in OpenAI's GPT 4.0 as well. Awesome. In fact, Gemini, Google's AI has also watched the OpenAI event and it can even answer your questions about it.

That was a really neat way to demonstrate GPT4's capabilities.

Now back to Project Astra. This one has an amazing feature. What is it? Well, it remembers when you walk around in the office, it remembers where you put your glasses before and can tell you. Or it can do this too.

Alright, give me a band name for this duo. Golden Stripes. Nice. Thanks Gemini.

Some say this is the future of AI at Google and if I had any doubts that smart glasses are the future, I don't really anymore. And there is so much more. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now I mentioned that this is an AI that remembers. How does it remember exactly? Well, their new Gemini 1.5 flash AI has something that we call a wide context window with 1 million tokens. What does that mean? It means that you can not just prompt it with text, but you can upload your entire thesis in there with videos of talks and everything and ask it to roleplay as your thesis committee and ask you tough questions to challenge it. It can also watch movies and videos too. For instance, I made it watch one of my favorites, Jacob Collier. I gave it a question about an almost 10 minute video that is in an unnecessarily high resolution. This came out to about 160k tokens and it takes about 30 seconds to get an answer. It is not perfect, but it is very impressive that it is close to it. Thank you Google DeepMind for the opportunity to play with this. The previous 1.5 Pro version also had a similarly wide window. We talked about the fact that it has a quadratic computational complexity and that means that it can take a minute or even longer to answer and the new Gemini 1.5 flash is promised to be able to do this much much faster. In fact, do you remember how fast GPT4O was? Well, the first benchmarks show that it might be close to twice as fast. To say that it is blazing fast would be an understatement. There will be an update to its open model version two that you can run locally for free. It will be called Gemma2 and will come in a 27 billion parameter package that is a good size to run on a beefy computer. And likely there will be smaller versions too that you can run on your phone like Gemini Nano. Now they also showed a sneak peek of Imagen 3, their text to image AI. You know the drill, you write a piece of text and you get an image, but this time with more details. And they also promise better text. That's good because text is still something that even the state of the art systems struggle with. But in The Age of OpenAI's Sora, text to video is also possible. And to compete with it they showcased veo, which creates full HD videos up to a minute long each super long. Now note that Google DeepMind publishes a ton of papers, but this event was not showcasing peer reviewed papers. It is more like a sneak peek into the things they have in the works. And here we are focusing on things that were or will likely end up as papers. Or for instance, VEO is a culmination of a lot of earlier papers, Fanache, video, Poet Lumiere, many of which we have talked about here in the series. Yes, in this corner of the Internet you get to become a fellow scholar and see the history of artificial intelligence unfold right before your eyes. Now here I also see long term temporal coherence which means that it remembers what it did a moment ago. So the car will be the same and and the environment will be the same too if we look away and look again. I think the visual quality is still a bit behind OpenAI's Sora and my impression is that they will try to compete on creative control tools instead. That is a good direction too. Note once again this is a keynote, not a peer reviewed research paper, so it is harder for me to find potential faults, cherry picked results and so on. This is a disclaimer that I would like to draw a little more attention to in cases like this. Now they also showcased more improvements to Gemini the Assistant which will be able to leverage the fact that they also have search and gmail and can pull up some of your messages, for instance flight or hotel info and use that to plan a trip for you. Or it can integrate with Google Sheets and look at your financials and profits and get all of this data together and visualize it for you. And then you can find out that maybe bracelets is the way to go. So there are lots of things unveiled here. So when are these available? Well, there are different dates for each of them, but most of these come in the future. I've put a link in the description that tells you what is going to be available and when. The bottom line is that after text, audio and images now we have entered the era of AI video understanding and there is intense competition in all of these AI areas. And that only means one thing. It means an excellent future for us consumers and fellow scholars. What a time to be alive. And in the meantime, I am also really excited to show this to you. I had the honor of accessing the shiny new ChatGPT app on macOS early and check this out. It can be invoked with a button combination much like Spotlight. You just write your prompt here and there you go. You can even add screenshots there from a paper and ask ChatGPT about it. Mathematical typesetting is working beautifully out of the box and extra style points for you. If you know which paper this is from, let me know in the comments below. And even chatting with Voice is working. It is beautiful and I absolutely love it. As far as I know it is going to be deployed to everyone soon. So stay tuned. And my goodness, what a week in AI. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPUs. Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.