The Truth About Google’s Gemini AI

Alright, so Google just came out with an announcement that is their answer to GPT4 and GPT 3.5 in its current state. And yay, they're coming out with a model that is better than GPT4. And this is both very exciting and interesting and very disappointing. So in this video we're going to talk about both, plus everything you need to know as a user of these products because a part of this announcement is available today through Google's bart. But more on that later, because first things first, we need to talk about the ginormous announcement here. Gemini Ultra, their most capable model moving forward. So I'm going to give you a super concise summary, my take on it and what I make of other people's commentary on this because I feel like a lot of it is displaced. So first of all, the summary, it's multimodal from the ground up, not as an afterthought, as is the case with GPT4.

Based on their design, which of these would go faster?

The car on the right would be faster. It is more aerodynamic.

That's amazing. Some of these use cases are mind blowing. We'll look at them in a second. It actually beats GPT4 on most benchmarks, which is amazing. This is a truly next level model that will be coming sometimes early next year. So yeah, the numbers check out here. From what they're showing, what they're telling us, this is really impressive. Now Google executives will be able to maintain face in front of their board members and shareholders at the end of 2023 and users will be getting next level tech in a few months. I like it. That's a win win. But here's the biggest deal. Okay, so we just crossed a specific score on this MMLU benchmark and the score I'm talking About is this. 89.8% on the MMLU benchmark. This is what human experts achieve when taking that test. Okay, so this is the very first model we will get access to that is actually better than human experts at specific tasks. Yikes. So as you can see, the Gemini Ultra scores 90% on this and GPT4 scores 87.29. But they actually tried to fool us here. No, no, not happening on this channel though, because look in the blog post, they're comparing the performance of Gemini Ultra and GPT4, but they're not using the same prompting technique. Okay, because what this means is that they prompted gemini Ultra with 30 tool prompts with a prompting technique called chain of thought and we're not going to get into that. Here. But it's just a specific way to prompt to get the highest quality results. And they did this 32 times, picked the most common one and that's what this score represents. Okay, so that's a fair technique to evaluate it, right? They use prompt engineering to get the best result possible. But what they're comparing it to is to a five shot approach on GPT4, which is a different and generally speaking, less capable way of prompting the model. And the funny thing is, if you go into this paper, which most shareholders, most consumers don't, you will find that if you compare the actual numbers, it's 90.04 and 87.29. So yes, it's a difference, yes, it's an improvement, but it's not the improvement they make it out to be here. Just be aware of that. But other than that, it just beats GPT4 across the board. So it's fair to assume that this is a better model than GPT4. So. So one more important fact here is that this model will have a 32k context window which does not match GPT's 4,128k that we got now. It doesn't work perfectly though. We talked about that, right? It only works really well until somewhere around 64k tokens. So this will be smaller than that. But honestly, 32k is plenty for most use cases. So I'd rather have this model. Right. And honestly I could go on for another 15 minutes here, but I feel like those are the most important points for consumers. But now let's talk about the multi modality of this. Cause this is actually very different.

Traditionally, multimodal models are created by stitching together text only, vision only and audio only models in a suboptimal way. At a secondary stage, Gemini is the multimodal from the ground up.

So if you're really into this, I obviously recommend watching all of these. We're not going to do that together here. But they do a great job of visualizing this concept of how current models are built, aka taking a bit of a jab at GPT4. Because GPT4, you might know that when it launched it was text only, because it's a text only model and it was trained to be text first and then they added Dall E, vision and more and combined all of them into one interface where now by just prompting something, it can access those different models. When you're a developer, this is apparent because these are just different API endpoints that you can independently access. But inside of ChatGPT, the easy to use web interface. It combines all of them into one. This is different with Gemini Ultra. And yes, 3% better performance in certain benchmarks. All. All well and good. I'm sure it's going to be fantastic once we get it. But the fact that it's multimodal by design, that's actually a game changer. And it just becomes really apparent in the use cases. And before we look at those, I just want to point out this also takes video, which is huge, right? Chatgpt, doesn't it? But this takes video inputs. So yeah, it's fair to say that from everything we see, this is better than GPT4. More capable. So here's the one video that I would strongly recommend you watch fully. Show it to your friends, show it to your family, show this to everybody you deeply care about. Because this is literally what's going to be driving the future of software.

Come up with a game idea. Oh, and use emojis, please.

How about this? We can play a game called Guess the Country.

Sure, let's do it.

Clue 1. This country is the home of the kangaroo, the koala and the Great Barrier Reef.

Oh, that's easy.

Clue two.

This, right? So it can do things like this. And did you realize how fast that is? How snappy? The latency on this is super low in the demo and I believe this should beat the final version, right? Google surely wouldn't just apply to us in a demo video, right?

Well, that's what I thought yesterday, but now that I'm finishing the edit of this video, it turns out that some sleuths on Twitter uncovered the fact that the prompts that they showed off in this use case video weren't the actual prompts the model was answering to. So they did in fact doctor this demo to look better than the model is actually performing. And that's a problem with these announcements. They tell us something is coming in a few months, show us fancy demos, but they don't actually give us anything to test. So we just gotta blindly believe what they show us. And it doesn't always turn out to be true. So I don't think this is a major thing, just something to be aware of right now.

Let's get back to the video. Magic tricks, analyzing cookies, fidget spinners, and then even harder problems including reasoning and translation. And look at that. These are the video use cases.

What movie are they acting out here?

I think they are acting out the famous bullet time scene from the Matrix.

Ooh, nice.

That's incredible, right? It takes in video, it takes in the voice you could surely type. And it's just interacting with the world. So if you unleash this onto your desktop and you let it interact with your entire interface, plus maybe you turn on the camera, it could pick up your facial expression while picking up your voice, while picking up your desktop, while picking up the context that you provided with from other Google apps, like your Google or YouTube search history and all the data they have on you that could be your personal context. Right, so very quickly this will become insane. So obviously they don't show some of the things that I just pointed out here, but it doesn't take a lot of speculation to kind of just take them next step here. Right? This will be the assistant of the future and they just demoed it to the world. So yeah, just make sure everybody you love sees this use case video so they understand where the world is heading. So once we arrive there in a few months, they won't be as surprised. Okay, so that's the multi modality. I should also note that several people on Twitter and myself tried some of these use cases with ChatGPT4 Vision across the board. They just don't work as well and I think they might have just picked out these examples. So people do that, so they arrive at that conclusion. It's just that much better at the multimodal aspect of it. Okay, but hey, let me just briefly say we're comparing a 2024 model with a 2023 model with GPT4. So it's not exactly apples to apples. Just wait for OpenAI to make their move with GPT5 and then we will return to this comparison. Right, but as of now, Google is winning the announcement game, whereas GPT4 is winning the usable product game. Okay, but that's not all. There's a few more points that we really need to talk about here, and one of them is the developer ecosystem. And wait a second before you just skip this. I think this is a really important point because one of the biggest things that OpenAI has achieved over the last year since ChatGPT release is they built a developer ecosystem that is beyond what any other AI provider like Google, Anthropic or the others can claim. And honestly, that might be the big differentiating factor. What gets them ahead. A lot of apps will be built up on GPT4 just because it's already available and you can build stuff with it. Plus there's a strong argument to be made that you might just prefer a company that is solely focused on supporting developers and shipping AI models, rather than Google, that does 10,000 things. And yes, AI is one of their main priorities, but just look at the landscape of their models and their AI technology. There's a lot, right? You have the Bard chatbot, you had the PAL models. Now you have the Gemini model, and there's three Gemini models between the Pro, the Nano and the Ultra. Then you have Google Duet inside of their sheets. Then inside of Google Cloud, they have a specific AI. And all of these seem to be led by independent teams with no really useful endpoint for developers to work with. I don't know, I just wanted to underline that the developer ecosystem that OpenAI has is just unmatched. And I don't see that changing so fast, even though this might be a few percent better. Because again, in this game of tennis, usually OpenAI was making this matches and now Google got to make their move. But as we know, then, there might be announcement lurking right around the corner, as they usually do when a competitor comes up with something that trumps their models. Anyway, getting back on topic, there's two more really important points that we need to talk about here. Okay? One is the other models they released, and the other one is the new and upgraded Bard. Okay, does mention they came out with three models. Gemini Ultra, their new flagship model, aka the superior GPT4 Gemini Pro, which is essentially their 3.5. It's better at certain tasks, worse at others. It nets out to be essentially the same. So that's the Pro model. That's what's available in Bards today. And then Gemini Nano. And the big deal here is that they actually released two versions of this, one with 1.8 billion parameters and one with 3.25 billion. Now, why is this a big deal? We haven't really received models that are this compact yet. Right. The most popular small model right now is Mistral, with 7 billion parameters. But with 1.8 billion, it will be really easy for developers to put this on phones. Okay, so expect a lot of this technology to come to your smartphone in 2024, because we're getting models like the Nano. Anyway, this also isn't available yet. It will come to the Pixel 8 next year. But the thing that is available today is their Pro model inside of bar today. So how good is this? I went into it, I tested it a bunch, I ran some of my standard prompts, and here's my verdict. It's a different flavor of GPT 3.5. Is it better? Is it worse? It really depends on the prompt and it's quite subjective. Now, the Cool thing is that they did add a bunch of extensions that could be useful to you. But honestly, in this state, I have kind of a hard time recommending this over ChatGPT to anybody. The free version, that is. Right. They both just work. And this is certainly better than the Bards that we had last week, for example, because that was quite the mess, to be honest. And now they upgraded to a level where it's decent. It's good. And I should point out that all the prompts that we discussed on this channel, including the prompt factory, work seamlessly in here. So look, everything I learned from this channel and other prompt engineering resources, you can still apply in here. Right? All these large language models take natural language as an input and all of the techniques universally applies. So if you learn how to craft a prompt once, yeah, it's just a universal skill. But what is pretty cool are these prompt presets. I'll say that, because if you click them, these prompts are actually proper prompts as opposed to what you get inside of ChatGPT, which is just very simplistic beginner prompts that I wouldn't really recommend at all. The bar to prompt presets here are actually pretty good, and some of them go sufficiently into depth to get proper results, as we talked about so many times on this channel. Right. By the way, if you want to learn how to prompt like this and get weekly live events by me for the next two and a half months, check out my prompt engineering course. But yeah, Bard with the upgrade, nothing crazy, nothing to lose your breath over. Gemini Ultra coming early next year. Yeah, fantastic. But let's be real, we're also expecting GPT5 early next year, so they will have to compete with that and not with GPT4 as they're comparing it with right now. So if it was out today, Insane game changing announcement. What we got here with the pro model is. Okay, they basically caught up. So look, what's the overarching story here? Well, first of all, finally Google made a move because they're considered by many as the tech company. And they were getting beat by a little startup called OpenAI on arguably the most important front of their entire business, which is AI, including large language models, consumer products business, but also developer endpoints. And now they can finally claim that the tech giant is actually delivering and living up to the super high expectations that everybody involved in Google has. And secondly, it's the fact that we have so many players in this game now that it's just a question of time until we get commoditized models, because you have Chinese companies with models like Quinn that are capable beyond GPT 3.5 and the Pro model that we got here today, you have anthropic pushing the envelope now with a 200k context window. This ultra model that we'll get in 2, 3, 4 months maybe has a 32k context window. So it's not just better across the board, right? Leading in multimodal, which is arguably the most important aspect of these models for user friendliness and use cases. But then you have players like Meta releasing translation models, open source models like Llama for everybody to build on freely that are out since months, so who knows what they're cooking up. So if it hasn't been obvious yet, it's officially on. The AI arms race is happening. Everybody's trying to get the future ahead of their competitors, and the most aggressive and innovative player, OpenAI, hasn't made their 2024 move yet. So. So while seeing all this, and especially seeing these use cases is extremely exciting, I think it's fair to Compare it to GPT4, as a lot of social media is doing right now, including this video. But the only fair comparison is comparing this to GPT5 that we essentially don't know anything about yet, except of all the mathematical capabilities of Q and those leaks actually being confirmed by Sam Altman. If that's a part of GPT5 and it's fully multimodal, well, then it's gonna be Google's turn again to 1up them or one of the 10 other companies competing on this. So the only thing that's for sure is that 2024 is going to be as wild as this year and I'm here for it. I'll be covering it all, the use cases and what it means for you. And on that note, I would like to point out a new show that I'm starting this week, which is a regular coverage of all the AI use cases that come out. It's not a new show, it's not covering all the latest drama in the AI space. It's just covering the tools that you can put to work today that are actually worth covering. So you can find the first episode over here. It's coming out later this week. All right, so that's it. Check out this video. It's a really good one.