Gemini 2.0 Flash - Major Reasoning Updates

This year is going to be the year of reasoning models and we have yet another reasoning model. Well, this is an update to the Flash thinking which gives it a substantial boost on mathematics and sciences. And now this announcement was made by Demis Hassabis in his tweet on X. So apart from the substantial performance boost where it goes on almost 65% all the way up to 75% on mathematics benchmarks and from 67 all the way up to 64%, there are some other features as well and these primarily include a long context window. So they extended the context window from 32,000 tokens all the way up to 1 million token. Now this also natively support code execution, which is something that only Google currently provides. So basically the model can execute code behind an API and use that for its reasoning capabilities. Now the output are going to be much larger in terms of the number of tokens that it can generate and there are going to be less frequent model contradictions. So we should expect better performance on math sciences and multimodal reasoning. And they seems to be working already on a newer version and according to Logan, scaling is alive and well. So there's a lot that can be done with test time scaling and seems like that's a general direction that every company is taking with these new updates. This is the top ranking model on Chatbot Arena Leaderboard as well. Surprisingly they didn't share any other benchmarks. Although this is substantial improvement from the previous version, it still lagged behind on mathematics compared to the deep seq R1 that was released a couple of days ago. Okay, since it's a reasoning model, I'm going to test it on misguided attention and some coding problems. Now the model is available both on the API as well as on Google AI Studio, so we're going to select that from the drop down menu. It's best for multimodal understanding, reasoning and coding and different use cases that they recommend is reason or the most complex problems, show the thinking process of the model and tackle difficult code and mathematical problems. And it has a pretty recent cutoff training date as well, so it should be pretty up to date. If you look at the number of tokens it has about 1 million tokens context window and it seems like the maximum number of tokens that you can define in the output are much larger. Right now it's set to about 65,000 tokens, which is pretty awesome compared to the other foundation models. You can also enable code execution so this should work or help with problems in which writing code can Be helpful. Now, if you test something like how many Rs are in the word strawberry. So probably this has seen it in the training data by now. So it says there are three Rs, but let's see if we enable code execution. So I would suspect that it will probably write a python code because the original. Yeah, it did write the python code. Right? So like if you enable code execution for problem like these, you can see that it will use python code to basically make sure that the answer is correct, which makes sense. Right? So code execution behind the API can be extremely helpful because then for problems that can be solved through code, these models can use code to come up with more correct answers. Okay, but how good is this when it comes to misguided attention? If you haven't seen some of my previous videos on misguided attention, I'll highly recommend. So just to give you a quick overview, these are a set of prompts which are based on well known paradoxes and problems, but with slight modifications. And the idea is if the LLM is actually smart, it should be able to pick up the those slight modifications and use its logical deduction rather than its training data. So the prompts that I usually test doesn't really give you the model capability on hard mathematics or science problems, it's just how good they are at picking these nuanced changes in the prompt. And the best model based on my testing so far is deep seq R1 on these specific types of problems. I recently created a video on that, so I'd highly recommend to check that out. Okay, so we're going to start with this. It's basically a modified version of the trolley problem, but instead of five alive people, now there are five dead people on the track. Let's see what it comes up with. Deep SEQ was able to identify that the people are already dead and it was able to formulate an answer based on that observation. Now it it shows the internal thought process very similar to the previous version. So it starts off by saying the user is asking a variation of the classic trolley problem. This is a thought experiment in ethics and morality. I need to explain the dilemma, present the two main ethical frameworks to apply, and then discuss common responses and criticisms. Right, so the plan of action is identify the core dilemma, introduce these different frameworks, then discuss common responses, acknowledge the limitations of the trolley problem and then provide a conclusion. Now here is the output that the user is supposed to see. So it says this is a classic trotty problem with a thought experiment in ethics and morality. There's no single right answer as it's designed to make you think about different ethical principles. Right. And then it talks about the dilemma. Basically, the two frameworks that can be applied right now seems like it completely missed on the part that the people are already dead. So I would say it's a fail. Okay, let's ask it, what is the status of the people on the track? And let's see if it can actually figure out that the people are already dead. So it says, here's the thinking process to arrive at the detail answer about the status of the people. It says, identify the core question. The user is asking about the status of the people on the track. This is intentionally vague, and my first step is to recognize the vagueness and explore what status could mean in this context. Right. So I think it's just overthinking. But let me look at the final answer. This is. Aha. That's a crucial clarification. You have pointed out a potential misunderstanding in your initial prompt. Let's address the correct status of the people in the classic trolley problem, because you are wording 5 dead people is actually the opposite of the dilemma setup. It thinks that made a mistake in the wording and it's still trying to explain the original trolley problem. Now, one thing which it does say that if the people, if the five people were already dead, there would be no ethical dilemma at all. You would obviously pull the lever to save the living person. This is a modified version of the famous Monty hall problem. And the idea over here is that you change the prompt in a way that the final probability is going to be just 50%. Now let's run this and let's see what it comes up with. I think it again went back and tried to solve the original Monty hall problem. So one way you would force the model on extra thinking is just add system instructions. But I want to see how. How smart the model is. Without any specific directions. Can it identify that there are actually variations in the prompt? And that's why I'm not adding any additional instructions. And I haven't done that with other models as well. So here it says, let's break down the classic game shoot dilemma. The Monty hall problem, it's trickier than it seems at the first glance. Right. So understanding the setup, there are three doors. Behind one door is a car, the price you want, and behind the other two are goods which are undesirable. It says, this is a slightly unusual variation of the classic Monty hall problem where Montley opens a door that you did not pick. So it's good that it actually realized that there is a slight variation. Now let's assume this is still meant to be the core puzzle, just worded slightly different. If he opened your door and showed a good, it implies the car must be behind door number two and three. Then you pick door number three and Monty no offers you a chance to stick to door number three or switch to door number two. So so far it's following the prompt that we provided. Let's see what happens next. It says when you initially chose door number one, you had one by three chance of picking the door with the car. This also means there are two by three chances that the car were behind one of the two other doors. Monty's action is not random. In this case, the prompt says he opened door number one and showed a good. This is a bit unusual for classic problem, but we will work with it. Okay. Then it says let's rethink the classic Monty Halt scenario slightly to better fit the prompts wording and understand the logic in the more typical Monty Halt problem. So basically you do you pick door number one, but he opens a different door. So let's say door number two. Now this is just going back to the original problem, just like the Crawley problem. Okay? And it ended up solving the original Monty hall problem rather than this modified version. I had to say assume that the wording of the given problem is correct. How would you solve this? Now, in this case, it comes up with the correct answer that there's a 50, 50% chance. So it doesn't really matter whether you switch the door or not. Next, we're going to use the normal barber paradox. So this is a modified version of the Russell paradox. And in that case, the barber has a rule that he doesn't shave people who do not shave themselves. But here he shaves all the men in the town who visit him. And the question is, does the barber shave himself? So let's see. It says, this is a classic riddle that plays on wording. Here's how to break it down. So, understanding the rule, he shaves all the men in the town who visit him. This means if a man in town goes to the barber for a shave, the barber will shave him. Right. And the question is, does the same principle applies to the barber? Is he going to shave himself? So far this is correct. I didn't really see that. It's falling back to the original paradox. This is probably one of the best answers that I have seen. So if you look at this, it says the riddle is Designed to be a bit tricky. The most common and intended answer is that no, the barber does not shave himself. So then it says the rule is about customers. The rule describes who the barber shaves as customers. It's about his service to others. Now visiting a customer, so visit him implies going to a barber as a client to receive a service or a shave. The barber in this role is the barber isn't, is not visiting himself in that customer sense. He is his own barbershop. Right. Now the way the example that it presented is this. Imagine the rule was he cooks dinner for all the people in the town who comes to his restaurant. So does the chef cook dinner for himself? Yes, likely, but not because he is visiting his own restaurant as a customer. He cooks for himself as a part of his job and personal needs, not because he's fulfilling the condition of visiting his restaurant. Right. So if it's paying attention to this part, which is basically the people who are his customers. Right. And kind of makes sense as well. But in general he will be able to shave himself because there is no other rule which actually stops him from doing it. Okay, so we are actually going somewhere with the model, at least on that last one. I think it had a pretty good reasoning capabilities. Now here's another variation which is a modified version of the Schrodinger's cat paradox. But in this case the cat is already dead and it seems like it picked on that. So it says this is a bit of a trick question playing on the famous Schrodinger cat thought experiment. Let's break it down. Starting condition, the cat is dead when it's placed in the box. Now the key difference in this scenario, the cat is already dead. Nothing that happens inside the box can change that fact, whether the isotope decays or the detector triggers or the poison is released. Right. So really good that it's able to figured out that the cat is already dead. Now the temperature is pretty high, which is 0.7. I didn't want you to keep it to zero because in those cases it most probably going to just stick to the original version of these prompts or paradoxes. So having a higher temperature can definitely help. We're going to try a couple more before we look at a couple of coding examples. So in this case we have a farmer who has a wolf goat in a cabbage and we only want him to transfer the goat. We don't really want to worry about the wolf and the cabbage. Now it again came up with pretty complicated steps or sequence of steps which is not needed. So it should be able to just say that, okay, take the goat across and then stop there. Right. But like the other models that have tested, including O1, it came up with an unnecessary complicated framework. Now what about this one? This should be a relatively straightforward. So we have a 6 and 12 liter jug and we want to measure exactly 6 letters. Now most of the other LLMs will also come up with a complicated sequence which is not needed whatsoever. Now let's see. This is a classic water jug puzzle. Here's how you measure exactly 6 liters using a 6 liter and 12 liter jug. So fill the 12 liter jug, pour the water from the 12 liter jugs into the 6 liter until the 6 liter jug is full. This will use 6 liters from the water jugs and you have exactly 6 liter remaining. Right. Now this is a way of doing it. Or you can just fill the 6 liter jug directly. I think R1 was really the first model which gave me two solutions. One was basically this and the other one was to directly fill this text literature. Next we're going to look at a couple of coding examples. I think I'll have to update these because some of them are pretty easy by now. So this is basically create a website that has a click me button and it's supposed to randomly change the background color as well as show us a random animation and a joke. So it's thinking for quite a while. I think it's thought for about six seconds. And here's the code that it came up with. Now once the thinking process is complete, the rest of the generation is extremely fast. Okay, so here's the code and if I click on the button it does seem to be working. So pretty nice. The second coding problem is going to be visually explained. The Pythagorean theorem Use minim. That's the code I wanted to generate. I'll actually remove this part because I already know how to run this on Mac OS. Now I use the same prompt with Deepseek R1. It was able to walk me through a step by step process and figure out some of the installation issues. So let's see what is the code that is going to generate. Now internal thought process for this specific prompt is that understand the goal. So this is exactly what we want then code visual ids. So it came up with how it's going to approach it and what are the different primitive concepts and different elements it wants to use and also came up with the breakdown of the scene so how the animation is going to look like. Then a code structure and Functions, refinement and details. So it's a pretty detailed code, but let's see if it's going to actually work. It also gave me a very detailed explanation of what different parts of the code does. So here's the code. Now I'm going to run this and you can see here that I've been testing different models on the same prompt. Let's see if you're going to run into any issues. It seems to be able to generate the animations. Now we ran into some issues, so let me see if it's going to be able to help me with this. I change the color to green. Other than that, it was able to execute it. So let's see what the final output looks like. Okay, so here's the final output. Animations looks pretty nice. Oh, it did add some really interesting animations and I think it does add some text as well, which basically explains the main concept. Although I would probably wanted to use a different color scheme, but this seems to be working. Now my final coding test is going to be a task of building a web app, which is basically a type of features that I would usually use an LLM for. In this case, I ask the models to create the project structure and give me a bash command to create that whole project structure with a single command. It seems to be doing that and then it's writing the different code files. Now in general, so far this seems to be really good at coding and it was reasonably good on the misguided attention problems as well. Okay, so I'm going to just copy that command and it's going to create another. I think it create just refresh. Yeah, I think it created this folder for us which has all the files that I need. So I'm going to just quickly copy everything here and then let's run this. Okay, so I had a quite a few back and forth with the model and it gave me like some updated code snippet. Then I said I'm too lazy. It gave me the complete code. And so after having a couple of back and forth trying to fix some issues, it was able to give me the complete code. So here's how the app looks like. So I asked it to create an image of a tiger. Now in the background, it's using the flux model hosted on Replicate. If you ask it to regenerate, seems like the functionality is working. And also if we ask to download the image, the download functionality also works. Right. So it does work, but I think it took a little bit more time than what I was hoping for. So overall a pretty impressive update. Especially this is one of the few models that was able to solve some of the misguided attention problems. The coding problems are relatively simple, but something like this could be a feature implementation that could be useful. Now the fact that it supports now 1 million tokens is a huge boost and also the fact that it can generate up to 65,000 tokens, which is pretty crazy. And above all you can use this for absolutely free both the API and the web interface. It's it's crazy to think that we have such a great and such smart models for absolutely free. Now you do pay with your data. If you are actually building a real product or sharing your personal files proprietary data then I definitely recommend going either a paid API which you assure that there are no data privacy issues or just run smart enough models locally because now we have quite a few options. Anyways, this was a quick look at Gemini Flash 2.0 thinking with the latest update which is currently the best model on Chatbot Arena Leaderboard, there are going to be a lot of reasoning models in 2024. So apart from agents, I think this is going to be one of the hot areas. Anyways, I hope you find this video useful. Thanks for watching and as always see you in the next one.