Gemini is Here! (And It's Better Than GPT-4?)

Well, I really thought we'd get a break with these AI advancements in December with the holidays here and everything, but I was totally wrong because Google just announced Gemini and we actually have a usable version of Gemini that we can play around with right now, today. So in this video I'm going to dig into what's new with Gemini and what we know so far and how you can actually go and use it yourself. Let's get into it. So today, December 6, 2023, Google and DeepMind introduced Gemini. Now this a pretty hefty article here with all the details about Gemini. I've pretty much gone through it all, watched all the videos, read all the tweets, and here's what I think you need to know about it. So they're calling this version Gemini 1.0 and it's going to start off in three different sizes. Gemini Ultra, which is their largest, most capable model. Gemini Pro, the best model for scaling across a wide range of tasks. This is the one that we actually have access to now and then Gemini Nano, which is the most efficient model for on device tasks. And this one seems to be be mostly designed for mobile use cases. We also know that this model was built from the ground up to be multimodal, meaning it can generalize and seamlessly understand, operate across and combine different types of information including text, code, audio, image and video. When you look at something like GPT3 and GPT4, these were initially trained from the ground up to be text models and just recently have been upgraded to accept both images and audio.

Multimodal models are created by stitching together text only, vision only and audio only models in a suboptimal way. At a secondary stage, Gemini is multimodal from the ground up, so it can seamlessly have a conversation across modalities and give you the best possible response.

Gemini will be available in three sizes. Gemini Ultra, our most capable and largest model for highly complex tasks that Gemini Pro, our best performing model for a broad range of tasks, and Gemini Nano, our most efficient model for on device tasks. Now, in benchmarking tests, Gemini Ultra actually outperformed GPT4 in most cases. Now keep in mind, Gemini Ultra is the model that we don't have access to yet. This is the model we'll likely get next year. We have access to Gemini Pro right now, which is a closer comparison to GPT 3.5 which you get inside of the free version of ChatGPT. But this Gemini Ultra model, when given the MMLU test, which represents questions in 57 different subjects, including STEM humanities and others it performed at 90% compared to GPT4's 86.4%. In reasoning we can see that it just barely outperformed GPT4 in this big bench hard which is a diverse set of challenging tasks requiring multi step reasoning. In this drop test here for reading comprehension it slightly outscored GPT4. And in this hella swag test which is common sense reasoning for everyday tasks IT underperformed against GPT4 which is the only area of these benchmarks that it underperformed. In math, IT slightly outperformed GPT4 in basic arithmetic and in challenging math problems. Keep in mind that neither of these performed very well in challenging math problems. These would be Fs on most tests. For Python co generation it performed at 74.4% compared to GPT4, 67%. Now they also tested it against GPT4. V we can see here that in these tests once again it's all Gemini Ultra that they're testing, except for these audio tests down here which they actually did test with Gemini Pro. So in pretty much every single image recognition benchmark like natural image understanding, OCR document understanding, infographic understanding and mathematical reasoning in visual Context, it outperformed GPT4 Vision. And in OCR document understanding and infographic understanding it actually outperformed quite a bit. With video captioning and video question answering, it outperformed GPT4. With audio automatic speech translation, Gemini Pro outperformed OpenAI's Whisper V2 and in automatic speak recognition, in which a lower score is actually better, Gemini Pro outperformed Whisper version three. So it sounds like Gemini Ultra again, the version that we won't get access to till next year, outperformed GPT4 in almost every way. Even Molik here sums it up pretty well. What changed today? Here's the simple version. There's no reason to use the free version of chat GPT anymore because that uses GPT 3.5 and as of today that's outclassed by Bard, which is now using Gemini Pro and Claude both, which you can use for free. But again you should probably still be using GPT4 because it's still dominant and it's free to use in Bing. And again, all those benchmarks we just looked at are pretty much not available yet until we get Gemini Ultra. Where Gemini really seems to shine is with its ability to understand visuals and text and all these modalities all at the same time. In fact, here's some examples that they shared in some of their demos today. In this example they gave a worksheet with a bunch of math problems on it. And the math problems are already done with the handwriting and some of the work being shown. And they asked to essentially check the answers.

It can read the answers and understand what was right and what was wrong and explain the concepts that need more clarification. So Gemini identified some mistakes with problems one and three here. Let's take a look at three here. Gemini identifies that the formula was correct, but there was a mistake in calculating height. We can ask Gemini to explain in more details why the height is 50 meters instead of 6. I can ask Gemini to explain further. Here Gemini explains the step by step details to solving the problem. Because of Gemini's ability to understand nuanced information and answer questions relating to complicated topics, it can give you a customized explanation of the subject you're trying to learn.

Google also put up this really cool video on YouTube that shows all sorts of different ways that you can use the multimodality. So check this out.

Tell me what you see.

I see you placing a piece of paper on the table. I see a squiggly line. The contour lines are smooth and flowing with no sharp angles or jagged edges. It looks like a bird to me.

Hmm. What if I add this?

The bird is swimming in the water. It has a long neck and beak. It is a duck.

Yes.

The duck appears to be blue. This is not a common color for ducks.

Keep in mind that this is responding in near real time. From what I can tell, there's no delay from him starting to color. And Gemini replying saying, the duck is blue.

It is more common for ducks to be brown, black, or white.

Are you sure?

What the quack? I was just talking about a blue duck. And now you're holding one. It looks like blue ducks are more common than I thought.

Would this float?

I'm not sure what material it's made of, but it looks like it might be rubber or plastic.

What if I tell you it's making a squeaking sound?

Oh, if it's squeaking, it's definitely going to float. It is a rubber duck. It is made of a material that is less dense than water.

Can you teach me how to say this in a different language? Ooh. Can you explain how to pronounce the Mandarin one?

So it also seems to be multilingual. So it could help you understand these words in different languages and. And even help pronounce them.

Is pronounced with a first tone. In Mandarin, there are four tones which are essential to distinguish the meaning of words. The first Tone is a high level tone.

Okay. Based on what you see, come up with a game idea. Oh, and use emojis, please.

How about this? We can play a game called guess the country.

Sure, let's do it.

Clue 1. This country is the home of the kangaroo, the koala and the Great Barrier Reef.

Oh, that's easy.

Fast forwarding in the video a little bit here we actually see the capabilities of it generating both text and image as a response. So check this out.

Give me some ideas for what I could make with this.

I see pink and green yarn. How about a dragon fruit?

You can see it suggested a dragon fruit and then actually drew a picture of what that would look like.

Or how about a green cake with a pink heart?

Once again, it created a green cake with a pink heart. So it's suggesting what you can do with this and then also drawing a picture of what that could look like. And it goes on and it shows more examples of this type of thing. And then it starts to go into these logic problems here.

Which way should the duck go?

Going left leads to a duck, which is a friend. Going right leads to a bear, which is a foe. It is better to make friends than foes. So the duck should go left.

Smart choice.

Now this one actually kind of blew me away. I don't. It's actually a sort of connect the dots sort of drawing. And I was actually able to figure out what the drawing is by just looking at the numbers and the dots.

This is a picture of a crab.

I don't know how it did that. That's crazy to me. There's a whole bunch of really cool examples where he draws pictures, asks more questions about the pictures. And here he actually drew a picture of an acoustic guitar and it was actually able to respond with I see you drawing a guitar. And then actually gave an audio sound of an acoustic guitar.

Now it says you've added an amp.

Now it's an electric guitar. We can make some seriously loud music now.

Now drums, given that you added drums, how about some 80s hair metal?

It's actually generating that audio based on what it's seeing in the images. I'll make sure the video is linked up below because some of these examples are really, really cool. I'm pretty sure they're using Gemini Ultra and not Gemini Pro. Pretty mind blowing what Gemini is going to be able to do. Now here's an examp where they showed Gemini turning images into code.

I'll start with this image of a tree and just select the part I want and then ask Gemini, can you turn this image into an svg? This represents the main shapes of a tree. Let's see. That's pretty good.

Not the most impressive tree.

Now I want to try a more difficult test. Let's see if Gemini can make an interactive demo in JavaScript. Okay, here we go. A common algorithm for this is called a fractal tree.

Now that's really cool.

Okay, this is pretty cool. Gemini even provided a slider so I can change and move the fractals.

Now here's Gemini actually looking at outfits and telling us what scenario this outfit is good for.

What is someone wearing this? Best dress to do. Hmm. Perfect for staying warm in the tundra. Good color for blending in with glacial ice. Okay, how about another one? Intergalactic Travel. How about this one? To boldly go where no one has gone before and play some jazz, here's.

Gemini using some logic to guess movies based on just images alone. Given the play on words in these.

Images, guess that's the name of the movie.

The Breakfast Club. Breakfast at Tiffany's. Uncut Gems.

And here's Gemini looking at multiple images to find similarities between the images.

We'll start with these two. The Boss Chapel and this print by Hokusai. And I'll prompt Gemini. Find a connection between these two images. A curved and organic composition. The building is more refined and the second image is more fluid. Okay, let's try another one using the moon and this golf ball on my webcam. Okay, let's see. In 1971, the Apollo 14 crew hit two golf balls on the lunar surface. Okay, then one more just for fun. Who wore it better? The zebra. Oh, I like this. The zebra has been wearing its stripes for millions of years now.

Moving along, Gemini Ultra is also really good at coding, including competitive programming that.

Includes Python, Java, C and Go. It substantially improves coding abilities over previous Palm 2 models. We are tasked with with computing aggregate statistics that account for what appears to be an impossibly large amount of random arrays. The really cool thing is that to solve it, AlphaCode 2 makes use of dynamic programming. Dynamic programming is an advanced algorithmic technique which basically simplifies a complicated problem by breaking it down into easier subproblems again and again. And what's really impressive is that not only AlphaCode 2 knows how to properly implement this strategy, but also when and where to use it.

Now, one thing to note about what we just watched over those last few demos that isn't actually explicitly mentioned in any of the blog posts Google put out, but was mentioned in a press briefing is that none of the Gemini models will initially be able to generate images. So those examples where we saw the images generated with Yarn, it's not going to output those images at launch. It plans to add that capability in the future. Gemini's multimodal nature, its ability to understand and produce both text and imagery simultaneously, was supposed to match OpenAI's capabilities. Now, I wasn't on that press briefing, but TechCrunch also confirmed this and another key limitation. While the Gemini Ultra architecture supports image generation as does Gemini Pro, in theory, the capability won't make its way into the productized version of the model at launch. That's perhaps because the mechanism is slightly more complex than how ChatGPT generates images. Rather than feed prompts to an image generator like Dall E3 in ChatGPT's case, Gemini outputs images natively without any intermediary step. The article goes on to say that they didn't provide a timeline as to when image generation might arrive, only an assurance that the work is ongoing. Jumping back to the Google article, it claims that it was built with responsibility and safety at the core. They're adding new protections to account for Gemini's multimodal capabilities. It has.