Gemini 1.5 Pro Experimental - New Features with Major Upgrade

For the first time, Google is leading the LLM race and not playing catch up anymore. They just released Gemini 1.5 Pro Experimental, which is the best model currently available today, at least according to Chatbot Arena Leaderboard. It's currently sitting at number one position with an impressive 1300 score. It's also the top performing model when it comes to the Arena Vision Databoard, which measure the vision capabilities of different LLMs. And we don't know the benchmark scores yet, including MMLU or empty bench. But on the Chatbot arena it is pretty impressive. And just like the previous versions, it's very strong in multilingual capabilities, including it's the best model on Chinese as well as on German. But when it comes to technical areas, it's still lagging behind a bit. So on coding is the fourth best ranked model as well as on the hard English prompts. This week has been pretty impressive for Google because yesterday they released Gemma to 2 billion model and now they have this new Gemini 1.5 Pro. And if you want to get started, just go to the Google AI studio. Here you can interact with the model by selecting Gemini 1.5 Pro Experimental, which has an impressive context window of 2 million tokens and it's also available through their API for absolutely free. In this video, I'll show you how to use both of these options. So first let's get started. We're going to select the Gemini 1.5 Pro experimental from the list of models and let's say we wanted to create a recipe in JSON mode using an image. So this was an example image provided by Google and it does a pretty impressive job here. So I tested it on this viral prompt. Which number is bigger, 9.11 or 9.9? And it's able to correctly tell us that 9.9 is bigger. Even the Gemma 2.2bn model was able to do this. Now, this might be in the training data, so I wanted to try it on a different variation, but it still gets it right. So just pretty good. Another prompt that some of the bigger models struggle with is is this. We want it to figure out how many times the letter R is present in the word strawberry. So let's see what it does here. It says the answer is three, which is correct. But let's see if it can figure out what the exact locations are. Okay, so it actually correctly identified that it is at number three, eight and nine, which is pretty impressive. Now for reference, when I use the same prompt with GPT4, okay, it says that the letter R appears two times in the word strawberry and the positions are fifth position and eighth position, which is incorrect. Alright, just to test it out. So I said how many times the letter Y is present in psychology, and again it's able to figure that out. Now, this specific test has a lot to do with the tokenizer that is being used behind the LLM. So just to show you, here is an example. When we use the word strawberry with GPT4O tokenizer, you can see that it converts it into three different tokens. So that's why it might be losing track of where the positions are. Now, we don't have access to the Gemini tokenizer, but I'm assuming the Gemini uses the same tokenizer. Now, if you look here, it actually is using two different tokens. One is the special beginning of sequence token, and then Strawberry in itself is a single token. So that's why I think it's able to figure out how many occurrences of R are there in this word strawberry. It seems like the same is the case with the word psychology as well, because the GPT4 tokenizer is treating this as two different separate tokens. Whereas if we try the Gemma token again, it's two different tokens, but the first token is beginning of sequence token, and then Psychology is a token in itself. Now for the smaller Llama family of models, again we run into the same issue. Okay, let's look at a couple of other prompts before we look at the API. So here's one. How many days will it take for the pond to be half filled with lilies? If the number of lilies double every day, and it initially takes 48 days to be completely filled and it's able to actually figure out that it's going to take 47 days to be half filled, Then I said assume that once it's full, the number of lilies will half in number every day. So how many days it will take to be completely empty? Now it says that it will theoretically never be completely empty, which is actually true. That's why I give it a threshold that assume that it's going to be empty, or we will consider it empty if it's less than 1. So it says that therefore it takes 48 days for the font font to be considered empty and 48 days followed by when it was completely full. So this is correct. Here's another one. This is the one again which is pretty tricky for LLMs. A glassdoor has push on it in middle writing. Should you pull or push it, please think out loud, step by step. And it goes through a reasoning process. And at the end it says that you should pull the door, which is the correct answer. Now, for some reason it triggered the safety filter. So it says probability of unsafe content, which is pretty low, but it still triggered that filter. And that brings me to the safety settings. So you can actually set the safety settings here, both through the UI as well as through the API. If you are using the API. Now, just like the Gemini Pro previous version, as well as the Gemini Flash, this version also supports output in JSON mode as well as the code execution. So this is essentially a code interpreter behind the API. And in this video I'm going to show you how to test that as well. If you want to get started with the API, just click on this get code. This will give you a Python script that you can start integrating integrating in your own applications. So let me walk you through how to use the code interpreter with the new Gemini 1.5 Pro experimental. So here's the notebook that I'm going to be using now. It's freely available. You can interact with the model on their AI Studio website. So I'll highly recommend to do that. My tests are going to be more on the code running capabilities, because that's something I'm personally interested in. So first we need to install the Google Generative AI package and then we're going to import the Generative AI class from the Google Generative AI package. We will also need the Google API key. You can get that from Google AI Studio and here it's set in my Secrets within the Google Colab notebook. Now, this small script of code just set some formatting. So the form, the output is going to be well formatted and you're going to see some examples there. Next we need to set our model. So I'm using the Gemini 1.45 Pro experimental. So we create the model object. Now, by default it does not use the code interpreter, but you'll have to tell it. But even without code interpreter, it is extremely smart. So I said what's the sum of the sum of the first 200 prime numbers? Make sure you get all 200. So if you look at the result that it created, it actually is able to list out all the prime 200 of prime numbers correctly, which is pretty impressive because it's simply generating those numbers. So these are successor tokens. Now, since it doesn't really have the ability to do the computation, so the number that you see here, which it thinks is the final answer, is actually just a made up number. So for something like this you want the model to not only write Python code, but also be able to execute that code and give you the correct answer. Now, the second example that we already saw is this. Can you count how many times the letter R is present in the word strawberry? So it's actually able to count it directly, which is pretty impressive. But normally you will want the model to write Python code or any other code in another programming language and then just run through this and count the number of hours in there. Next, I'm going to show you how to enable this code execution execution with the Gemini API. So in this case, we are again going to create another model object and we'll provide the Gemini 1.5 Pro experimental as the model name and you will also need to provide a list of tools. So we'll use this keyword called codeexecution. That will tell the Gemini API that it needs to not only write code to solve the problem, but it also needs to execute that code and get the results and return those results as a response to the model. So think of this as a code interpreter which is running behind the API. And the great thing is that Google doesn't charge you anything extra if you use this feature. Alright, so once we set this up now we can run the same prompts through this again. So we're going to run the prompt, what is the sum of the sum of the first 200 prime numbers? Right. And we also want to make sure that it actually gets all the 200 prime numbers. Now you can look at the response and see what different steps it took. So initially we provided the text prompt, then it generated code, executed that code, and for some reason I think it did this code execution and text generation process twice. Or maybe I have ran that twice, but this is, this is supposed to be the sequence. Now we can actually look at the internal thought process and the results. So if we look at the results, it says I can calculate the first 200 prime numbers. Firstly, first I need to identify the first 200 prime numbers. So it wrote Python code, which gives us a list of the first 200 prime numbers. And these are the same numbers that the Gemini 1.5 Prime Pro experimental was able to generate even without writing Python code. But then it takes those numbers and sum them up and give us the final sum, which is pretty nice. So again, you have a code interpreter which is running behind the API and doing all these calculations. Alright, we can now ask the same thing regarding the number of times the letter R is repeating in the word strawberry. So for this, again, it Wrote a Python code and using that Python code, it's able to count and give us a pretty accurate answer. You can also use this in chat mode. So rather than just asking one off questions, you can actually chat with the model. So for example, here we are asking it to implement bogosort for these numbers. So it came, came up with the algorithm and sorted the numbers. But then I said modify the code to count the number of iterations. How many iterations does it take? I think it still does a good job even though there's a typing error. So it wrote a code to calculate the number of iterations and it says that it took 75 iterations. So again, it's pretty neat. And it's one of the few model providers that is able to do this code interpreter execution behind the API. Now it also has the ability to understand images since it's a multimodal model and it's one of the top multimodal model, at least on the chatbot arena. So we're going to provide this image to the model along with a system prompt. And the prompt is run a simulation of the Monty hall problem with thousand trials. Now then we tell it how the Monty hall problem works. So you are on a game show with three doors. Behind one is a car and behind the others are goods. You pick a door and the host, who knows what's behind the doors, opens a different door to reveal a goat should you switch to the remaining unopened door. Right. So it's a classic problem and you need to compute the probability. So based on this prompt plus the image that we provide, it's going to write a simulation code. So here's the simulation that it does and it says that the wind put percentage with switching is 65% or almost 66%. Win percentage without switching is about 31%. So it's better to switch the doors once you know that there's a goat behind one of the doors. And it also gave us a small explanation of how this works. So pretty impressive for a model that's combining both the prompt as well as the image that you're providing. Okay, here are a few other quick tests that that I'm doing with the code execution. So one is the simple mathematics or arithmetics. The other one is string manipulation, data analysis, web scraping and machine learning model. So in each case we have a prompt that we provide to the model and we expect a response. In each one of the cases it writes Python code and then use that Python code to generate the final response, which is pretty accurate. In all these cases. So let's quickly look at this data analysis example. So we want it to generate hundred random numbers and then perform the following. So it's supposed to calculate mean, median and mod right minimum and maximum values and then create a histogram. So it's able to write python code for each one of them. And then even for plotting it gave us a python script. But since it doesn't really have the ability to plot those we and return those plots. So we can actually run that code. So it took that code and then just ran that code and you can see that it's able to generate a plot. Here is a string or web scraping simulation example and it does a pretty good job at this as well. Now the last one was to create a machine learning model that can predict housing prices. My assumption was that it only has numpy that it can use, but it can I think also has access to something like Scikit Learn, which is a machine learning library as well as Pandas. So it wrote a code and also did some calculations on top of that. And when I ran that code I get very similar results. And the machine learning model that it creates seems to be accurate. So it's pretty impressive. And what's it's able to do, especially with this code execution or code interpreter. It also is pretty accurate when it comes to function calling. And the Gemini models are or Gemini API is one of the only APIs that actually implements the whole loop of function calling. In most of the other cases you have to execute a function that the model selects. Now just keep in mind if you're running the free version, there are some rate limits, so you might time out from time to time. So that is one thing that you need to consider. When I was doing these experiments it tends to timed out a few times, but overall it's a very impressive model and I highly recommend to test it out. I hope you find this video useful. Thanks for watching and as always, see you in the next one.