Best Coding Model? I Tested 5 Models.

I tested seven different models using the same prompt and here are the results. I want you to pick your favorite and later in the video I'm going to show you which one is which and the results will probably surprise you. Here's number one, number two, number three. In each of this case, here's number four. I asked the LLM to use search tool to look up the information. Number five, number six and last but not the least, number seven. Claude 4 was released last week and Anthropic is calling it the best AI coding assistant. We're going to figure out if that is actually the case. I'm not going to bore you with the blog post, but a quick look at the benchmarks on the SWE bench Verified it shows a very interesting result. We have both the Opus, which is the biggest model, along with relatively smaller Sonnet 4. However, on this SWE bench verified, Sonnet 4 seems to be doing relatively better compared to Opus 4. But for the agentic terminal coding tasks where the agent has to use a set of tools to interact with environment, think of as Cursor or Windsurf, where the model can modify or delete files, Opus 4 seems to be doing much better compared to Claude Sonnet 4 and even outperforms the rest of the big models. Anthropic is saying that Opus Forces is great for long agentic tasks especially it has improved memory and it's great for power parallel tool execution. Now how good is this on independent testing? Well, on the ADIR LLM leaderboard, which I respect a lot, Opus 4 is currently at 72%. That puts it at number five. The new Sonnet 4 is actually lagging behind the previous 3.7 Sonnet, which is very surprising but but still anthropic things. These are the best coding model out there. Now something interesting, if you use Opus 4 on ADIR benchmark, the cost is very similar to if you're using O3 GPT 4.1. Although O3 is a relatively cheaper model compared to Opus 4, so it might be a lot more efficient in terms of the number of tokens that it generates. In order to run through all the questions in the benchmark before looking at the test, let's look at pricing which will really determine whether you want to use these models or not. Opus 4 $75 per million token output Cloudsonnet $15 per million output tokens and this is a lot more expensive compared to O3 which is $40 per million output tokens. The best pricing that you can get is $15 per million output tokens for Gemini 2.5 Pro. And if you're using less than 200,000 tokens, then you even get much better pricing. But how good are these models when it comes to actual testing? Well, I'm going to be testing all of these models on a very interesting prompt. In this case, we want to create a web app that is going to ask the model to look for information on the web. We're going to enable web search tool to all of these models. And using that web search tool, we want the models to synthesize information into a dashboard where we're going to have name of the company, name of the model, category of the model, everything listed in a very specific format that we are providing, along with the benchmarks that it can find. There are two different goals. One is how good the models is able to follow the instructions, and the second is how good they are at using the tools that are available and also whether they are going to hallucinate information. And for testing, we're going to be using Sonnet or 3.7. That's the previous version, Opus 4, Sonnet 4, OpenAI 03, Gemini 2.5 Pro, Quinn 2.5 Max with web search enabled and Deepseek R1. I also added Groq 3 because it also has thinking and web search capabilities. But I was never able to get it work for this specific prompt. So we're going to drop that model. All of the models that I have tested or chosen to test here, they have the ability to do web search, and all of them are reasoning models. Now, there is this one specific capability which sets apart O3 and Claude 4 models, and that is sequential tool call in chain of thought. So if you look at this chain of thought from O3, you're going to see that it used the web search tool. Then based on that, it synthesizes the information, then it decides to make another call to that web search tool. And this way it enables the model to build up its information on the go. If you look at something like Gemini 2.5 Pro, it uses the web search tool in the beginning, and after that it's just relying on the information that it has found in the Star Quinn. And Deepseek has a very similar flow. So they collect information right in the beginning, but they cannot update these web searches as it's going through its chain of thought. As an example, you can see the Claude Opus 4, which does a web search, looks at the result, identify different themes in the results, then does another subsequent web search and kind of goes through this Sequential execution of tools. And Opus 4 is the one that takes a lot longer compared to any of the other models that that I tested. Okay, so here are the results. Now I want you to guess which model produced which results. At the end of the video, I'll actually show you which one was the model for these specific results. So this first one, I think it did a pretty good job. It figured out that the latest anthropic models are Opus and Claude 4 series. For some reason it missed on O3 in the prompt I added very specific models which are not latest state of the art. And the goal was to see whether this is going to confuse some of these LLMs or not. And seems like this trick actually worked. Now this specific model for some reason thinks that Jamba 1.5 large is a state of the art model. But apart from missing O3, it did a reasonable job except with the date of release of Cloud 4 Sonnet or Opus Sonnet. Here it's listed as May 21, but in reality it was released on May 22. Actually, I think it has the same issue with Llama 4 as well. Llama 4 was released on April 5, but here we find it's listed as April 4. It might have something to do with the time zone. Yeah, it seems like for each one of them, the day is listed one day before the official release. So it definitely has something to do with the time zone. Now, in terms of benchmarks, it does a pretty good job. So let's say if I select Sweepbench, we have Gemini 2.5 Pro and the Cloud models. And here it also added this neat little visual. Okay, so this was number one. Let me know what you think about this. Number two is this one, which is a lot more colorful. Now this one lists only Claude 4. For some reason it does not list Opus 4 and also has this 8 billion model listed as a frontier model. Other than that, it also has some issues with the release date for Claude 4 and for some reason it thinks that Cloud 4 is about 200 billion parameter, while Gemini 2.5 pro is around 1 trillion parameter. I guess like it did the web search and it found some speculations around the model sizes. Now, in this case there's a tab for benchmarks. It actually does not work. Okay, so number three, this looks a lot more professional. Similar to number one in here it only lists Claude 3.7 sonnet, so it's not actually able to find Claude 4. And also it lists O3, which is pretty great. The context window information seems to be correct, but it thinks that Falcon 2 is a frontier model. And let's see, the benchmarks also actually looks pretty nice, especially it has these plots here. Although there's no way you can select specific benchmarks. But for O3 we don't really see any of the MMLU Human Evolve and Math benchmarks listed. And you can also filter these based on the companies, which I think is pretty neat. Number four, very similar to the first one that we saw in here. It lists GPT 4.5. I don't think it lists O3, although it has O3 mini. And it also lists Claude 4, which is pretty good. But if you click on the benchmarks, it doesn't do anything. If you look at some of the benchmark scores, seems like it's hallucinating. So for GPT 4.5 it lists MMLU at 92.5. A quick Google search shows it 89.6%. Also, I wasn't able to verify the Cloud4sonnet MMLU score. Actually the benchmarks for number four are listed down here. Now I can select different benchmarks. So sweep bench. This is the one that I actually know. So Claude4sonnet. These numbers seems to be correct. But if you select something like this, I think it does a recent reasonably good job. But for some reason this link in itself is not working. This is number five. We don't really have much information. Seems like it went by the models that I have listed in the prompt that I provided. And it thinks that Gemini 2.5 Pro is 540 billion model with a context window of only 16,000 tokens. Right. So this is pretty bad. But if you look at the benchmarks, it also lists these benchmarks for the model models that I have listed. Now the most surprising one was this for me. So this is number six. It is able to find Claude Opus 4 and for some reason it missed out on O3. And it has this DB RX model listed as state of the art, which was released back in 2024. Now this one has the release dates correct. You can select the benchmarks here. Okay. But visually this looks pretty bad. And the last one wasn't able to actually render correctly, so it had a lot of issues with the code, so I had to just discard that. Okay, so again, this is number one. This is Gemini 2.5 Pro. Number two, this is Sonnet 4. Now this one, Opus 4, it had issues with the latest cloud models. It was not able to retrieve them. Also it's listing Llama 3.3 for some reason in here. And it is thinking that Falcon 2 is state of the art model. The surprising thing is that I would assume that all the entropic models have access to the same search tools, but it's not able to synthesize information in a very similar way. This, which looks very similar to the first one, this is Sonnet 3.7, although I feel like it made up some of these benchmark numbers. Quinn 2.5 max. Now this one is O3, which is I would say a little disappointing. And the last one is deep seq r1. Okay, so what is the conclusion? Well, even though we provided the same prompt in all of these cases, seems like the results are hit and miss for all of these models. Now all of them rendered the web pages correctly except R1, so you can give them passing marks on the UI that it creates, but the information they were able to put in is lacking even for the opus 4, which is supposed to be this model that can run tasks for hours right in an agentic framework in 2.5 Pro as well. So if you're running any complex task that is going to be using multiple agents, you definitely want to recheck irrespective of whatever model you're using. I didn't want to test this within Cursor or Windsurf because both of these systems uses more complex agentic prompts and they have to optimize it for each of the models that they use. But in this case it's the default behavior of the model, so you're going to see very similar results. So in reality you could pick each one of these models. It doesn't really make a difference if you just want to work with the UIs, but if you want to collect information, synthesize it right In a multi agentic system, you probably want to use a combination of these models rather than relying on a single model. Do let me know what you think based on the results that you saw so far and which model will be your best choice. I am biased towards Gemini 2.5 Pro because of its cost is probably the most cost efficient model out of everything that we have tested in here. But in terms of performance, you can pick any of the Sonnet models or the Opus model, but you're going to run into rate limits pretty soon. Do let me know what you think and I hope you found this video useful. Thanks for watching and as always, see you in the next one.