Never Browse Alone? Gemini 2 Live and ChatGPT Vision

First Google and then tonight OpenAI. Both companies have released fun, useful and pretty impressive tools that can see what you see, listen and even browse alongside you. Google's is free, OpenAI's is paid, but OpenAI have included parts of their tool in iPhone 16s. I'm going to give you just my highlights from the literally dozens of announcements in just the last 24 hours. We'll take a one or two minute demo tour and then I'll give you a hint about why these models make more mistakes than you might think. Plus of course some benchmarks can't resist that. Now, I could save the best till last, but I won't. I'm just going to give you the coolest tool in my opinion, straight up front. You can access it for free@aistudio.google.com live. We're going to have a conversation with Gemini 2.0 from Google DeepMind. I'm not going to say it's their smartest model and I'll get back to that later, but it is their newest. Just for a brief moment though, let's not care too much about the smartness of the model. Look at what you can do if you're on mobile. What you'll want to do is click Show Gemini and then it can gain permission to look out of your camera. Now you could already send videos to Gemini, but this is live interaction. Maybe ask it about that building you're walking by or that plant you can see next to the path. Because we're using this for entertainment and curiosity, I don't mind too much that it makes the occasional mistake. Obviously do not rely on it Answers I'm going to click Share Screen in just a moment, but first I have to give you a bit of context. Here is another tool announced by Google yesterday called Deep Research and it's only available via Gemini Advanced. It's a decent tool. It researches the web, collates from like 20 results and presents them to you. I asked tell me everything about Simple Bench featured on AI explained and it gave me a plan and then I approved it by saying Start research and it gave me this result. It's definitely comprehensive research, but you can't really rely on it as exemplified by this sentence. Now though, let's investigate that interactively with Gemini. This is free, so bear that in mind. Okay, Share your screen and then pick the screen you want. Hi Gemini, how are you?

I am doing great, thanks for asking. How are you today?

I'm good. Can you read out the sentence highlighted in yellow?

Of course, the highlighted Sentence reads Llama 3.1 outperformed other models like GPT4 Turbo, Gemini 1.5 Pro and Claude 3.5 Sonnet on the simple benchmark.

Close but good enough. Now can you look at the leaderboard and tell me, is that correct?

Yes, that's correct. The leaderboard shows that Llama 3.14.05B instruct is in 8th place with a score of 23% while the first place is 01 preview with a score of 41.7%.

This is honestly really fun to speak to you and you sound like a cool dude, but did Llama 3.1 really outperform Claude 3.5 Sonnet, which I'm going to highlight?

You are correct. It seems that Llama3.1 did not outperform Claude 3.5 Sonnet as Claude 3.5 Sonnet is ranked second with a score of 41.4%, while Llama 3.1 is ranked eighth with a score of 23%.

Thank you for your honesty, Gemini. By the way, this is the channel that came up with Simple Bench. Any feedback on the aesthetics of the thumbnails?

Sure. The AI Explained channel on YouTube has a consistent style with its thumbnails. They typically use a combination of text and images and they have a very clear and legible text overlay. The colors are bright and contrasting, which makes the thumbnails stand out.

Thank you very much Gemini. Of course you will have noticed some basic mistakes there because even though this is gemini 2.0 flash, the whole new gemini 2 era, actually gemini 2 flash got around 20% on simplebench. Simple bench is all about testing basic reasoning and seeing the question behind the question. Other new entrants include llama 3.3 at 19.9% and Gemini experimental, interestingly, 31.1%. That's not the model that powers that interaction. Why then did google pick gemini2flash for this tool? Well, because it's much cheaper and faster, some of you who have been following the announcements will say. But wait Philip, I thought Gemini 2 Flash got crazy results in all the benchmarks. Well, yes, it got crazy results in all the benchmarks they picked to show you notice. Also, they only compared the results to their own models. O1, for example, let alone 01 Pro gets close to 20 points higher on this benchmark, for example testing biology, physics, chemistry. But in fairness to Google, they aren't really pretending like Gemini 2.0 Flash is the best model around. Perhaps they learned their lesson from one of their previous releases where Demis Hassabis said I think it was Gemini 1.5 pro is now better than experts in a range of subjects. Scoring highly on certain tests doesn't mean for now that you can match the breadth of human expertise. Luckily then, Hasabis didn't make that claim this time around. But there are other things that Gemini 2 Flash can do. It can speak in whispers and different tones and swap between languages. It can also output and kind of edit images and more on why I say kind of In a moment we.

Brought this image into AI Studio and sent it to the new experimental Gemini 2.0 flash model. This is a real time screen capture and all the prompts you'll see in this video are things that early testers can now try. And this is the entire prompt we gave it Turn this car into a convertible. Here they are side by side.

Actually, I was fortunate enough to get early access to that tool and even though I can't show you any of the results, I can say it's still a work in progress. Others though have shown that you can upload an image like this one of a Tic Tac Toe or Noughts and Crosses board and ask it's X turn to play. What should the next move be? I must say, as anyone who has seen this channel before will know, I am quite the expert at tic tac toe and so I would agree when Gemini 2 flash says the move should be this one and then it gives us the image. That's pretty cool. One more cool thing that Gemini 2 Flash can do before we get to a pretty big caveat. Google demoed project mariner where Gemini 2 actually takes over your computer. Well, at least it can click things, browse the web and perform actions, even purchase things. Not yet available mind you, but I am trying to get access. Here is a 20 second demo.

You can ask it to do things like this that require multiple steps, doing research to look up the artist, then finding the painting and then shopping for the right supplies. The agent uses the web to take action and find what you're looking for. It plans and reasons at each step, always staying in your control and completing your task.

Google DeepMind are not the first company to allow a model to take over your device, but the reason I didn't talk about computer use from Anthropic powered by Claude at the time was because it just really wasn't very good. I tried to get it to do countless things and it simply couldn't. It took ages and rarely could achieve the goal. But just as an aside, bear in mind this benchmark called WebVoyager, testing how you can navigate the web with visual inputs. Claude computer use gets 52%. This particular agent Scaffold got 67%. Almost Project Mariner, as you could see, powered by Gemini 2 Flash 90.5% with tree search. And in an interview released just three hours ago, I picked out this highlight spoken by Oriole Vignals. He is the technical lead on Gemini and will tell you what's coming soon.

I could ask a model, a language model or visual language model to learn to play the game of Starcraft. That's a very different approach to say, create one agent that does play the game. In this other example, it could go online, watch videos about the game. It could of course download the game to start interacting with it to learn. Oh yeah, I know how I get it. Do research online, go to forums, read the forums, go play and figure out that it's weak at this thing and improve and so on. And after literally it could be weeks, it sends you an email that says I now know how to play the game. Let's play. Right. That's not a reality. That's that far away.

Frankly, that's quite enough buzz. Time to bring us all back down to earth. Progress, according to Google's own CEO, might be slowing down. In his words, the low hanging fruit is gone.

The current generation of LLM models are roughly a few companies have converged at the top, but I think we're all working on our next versions too. I think the progress is going to get harder when I look at 25, the low hanging fruit is gone, the hill is steeper. You think it's slowing down though? That's interesting because I will tell you that your competitor, peer in Sam Altman, said there is no wall. There is no wall. You think it's maybe slowing down in terms of how quickly that scales? I'm very confident there'll be a lot of progress in 25. I think the models are definitely going to get better at reasoning, completing a sequence of actions more reliably, more agentic, if you will.

That is a pretty stark contrast with what Sam Altman said as the CEO of OpenAI when he said there is no wall, end quote. Or Dario Amadei, the CEO of Anthropic who predicted transformative AI by 2026. That's AI that can automate whole swathes of the economy. You can let me know in the comments who you think is right in just a few moments. Of course, I will get to the announcements from OpenAI over the last couple of days. But there's one model that we have right now. Forget the future, it comes from the sponsor of today's video assembly AI, and it's their universal two speech to text model. I've shown you plenty of comp comparisons before on the channel, but here's another few for example Proper nouns Naturally models have to get better at understanding what you said before they can get better at replying intelligently. Links in the description Some people criticize me for being too harsh on OpenAI and do check out my previous video for more of what I think, but I will say they are great at smooth interfaces. You could almost say Apple like UX. So appropriately, ChatGPT is now inside iPhone 16. Yes, that means it's inside Apple Intelligence, but it's made by OpenAI anyway. No, getting an iPhone 16 isn't a way to circumvent paying for plus or Pro, because even though you do get ChatGPT vision, it can as of yet only analyze an image from within a video, not like a live interaction. That's something separate. Still cool though, because you can invoke ChatGPT even if you don't have a ChatGPT account. So I can imagine hundreds of millions of quote normies using this even though they don't follow AI. But for true video interaction available in advanced voice mode, you'll need to get plus or Pro. That's $20 or $200 a month respectively. And though it's not free, just as with Google AI Studio, you've got to admit it's a pretty compelling tool.

Sure, I'm ready for a quiz.

All right, what was the name of.

My colleague that was wearing the reindeer antlers?

That would be Michelle.

All right, and how about the name of my colleague that was wearing the Santa hat?

That would be Rowan.

All right, good work.

You're two for two.

You get an A. Thanks, Chat. Frankly, I think this is the best of AI, where you don't always care too much about super accurate answers but satisfying your curiosity. As of tonight, you can even speak to Santa via voice mode and ask terribly rude questions. Again, as with Google, you will find that the underlying intelligence of the model you're using isn't the very best. The vision and advanced voice mode is based on GPT4O, announced quite a while back. Obviously that makes sense, they just don't have the GPUs to give everyone the best model at all times interactively. Gemini 2 Flash is for us, currently heavily rate limited, so we're struggling to get the 5 runs necessary to calculate an average, but on the runs we've done so far, it scored around 20%. When the API does become available, I do expect 01 promo to sit comfortably at the top of this leaderboard. I should say at the top of the models, not above the human baseline. That baseline 01 Pro mode, at least, will not be so. Those were my highlights from the dozens and dozens that came out over the last 24 hours. AI models that can see what you see, listen, browse alongside you, even play games alongside you. Actually, that gets me thinking. I can imagine in like two, three years, kids or even adults when they're losing a game online, being like, you're using Gemini 4, man. Actually, that could be a pretty serious problem for competitive gameplay, unless you had like, land tournaments. Anyway, I'm getting too distracted. Those were the announcements I found the most interesting. Let me know which ones you found the most compelling. And as always, have a very wonderful day.