Qwen 2.5 Coder 32B: Is This Best Open Weight Model Better than GPT-4o?

Okay, so everybody's talking about this new Qin model that we're going to put to a test in this video. The Qin 2.5 coder series from Alibaba is a collection of different models of different sizes. They recently added the 32 billion version, which is the state of the art open source code model at the moment. Now, based on their internal benchmarks, it actually outperforms GPT4O and is very close to cloud 3.5 sonnets. Now I personally don't really consider the internal benchmarks, so we're going to look at an independent benchmark. But in this video the goal is going to be to show you how you can build code based applications with this new model. So the QIN 2.5 coder series contains models ranging from half a billion all the way up to 32 billion model. Personally, I'm excited about the 32 billion model because I can run this locally on my M2 Max and it does provide a pretty reasonable performance. Now, the smaller half a billion or 1.5 billion models, they are more geared towards the edge devices, so you can potentially run them on edge devices. And all of them except the 3 billion model has permissive Archie 2.0 licenses. Apart from the model, they also released something they are calling Quinn Coder Artifacts. So it's very similar in functionality to artifacts from Anthropic within their cloud interface and it's worth checking out. I'll put a link to this in the video. Description for coding I'm personally interested in the inner LLM leaderboard. It has two different types of leaderboards. One is the code editing leaderboard. The second one is code refactoring leaderboard. The code editing leaderboard contains 133 small coding exercises where the LLM is supposed to edit Python source files. So this is kind of the stuff that a developer would do. Now, on this leaderboard, the new Quinn coder model is at the fifth position, just behind the 3.5 haiku, which is the latest haiku model from Clon. And it does outperform GPT4O and Deepseed coder models, but it's not close to the Claude 3.5 sonnets, both the previous version as well as the new version. The code refactoring leaderboard is a lot more complicated because the LLM has to refactor 89 large files. This is a lot more involved and I believe the new model is not tested on this specific leaderboard yet. The blog post provides a lot of technical details and comparison analysis, including this human preference Alignment where they created their own internal database or data set very similar to the Arena Hard, and in this case, for some reason they use GPT4O as the evaluation model for preference alignment. So basically GPT4O is the judge. Now, based on the results, it seems like it does very similar to the GPT4O model itself, and it's lagging behind the cloud. 3.5 Sonnet. Now, there are a lot of technical details in the blog post, so I'll put a link to this in the video description. But let's test it out on a couple of example applications that I want to build with this model. But before that it can support a context window up to 128,000 tokens, which is pretty great. But in reality I think it's limited to around 32,000 tokens. So if you're looking at the model on Hugging Face in the config JSON, it's actually set to 32,000 tokens, but then you can expand it to 128,000 tokens by changing the configuration in the config file. The weights of the model are available on Hugging Face. So you can download this. You can use either Ollama or LM Studio. Right now my Internet connection is pretty banned, so I can't really run this model locally. Once I have the ability, I will download the weights. But there are a few options that you can use the model, including the Hugging Face Chat interface. Now, the model is currently available on Hugging Face Chat. There are a couple of API providers which are also hosting the model, so you can test those out. I'm going to put link to a couple of those. So in here we're going to start with a couple of simple applications and the goal is going to be to see whether the model can follow instructions and then create our web apps. Now, I'm not testing it on something like a Snake game in Python, because that is probably already in the training data. So we will experiment with two different apps. The first one is write a Python code for a website in a single file with the following features. Now, I'm going to test it on two different applications. The goal here is to see whether it can create these similar simple web apps and whether it can follow instructions. So the first one is write up a HTML code for a website in a single file. So it has to write everything in a single file. And the functionalities are there's a button in the middle of the page with the title. Click me. When the button is clicked, show a random joke from a List of jokes, make sure the jokes are funny, although I don't really expect the anonymity to generate funny jokes. And then when the button is clicked, also make sure to randomly change the background color. I also want to add some sort of animations. Now this was a task that was nearly impossible for the earlier open rate models, including something like the Mistral 7B and Mistral. Moe was probably the first model that got this right in a single shot. So let's see what kind of responses we get from the 32 bin coder model. So it started writing HTML code for our web page. So I'm going to let it finish and then we are going to run this in an HTML file. Okay, so it generated the whole code. Now for animation, I think it only added a shake and it says you can add more animation if you want, but let's see if this is going to actually work. And in order to test it out, we're going to use this online HTML editor. So I pasted all the code here. Then let's click on Run. Now it shows a button which says click the button to hear a joke. No. And the button says click me. So this is good. And when we click is showing different jokes. It does change the color, it does show the joke, but I think that animation is not really there. Now, in comparison, this is what GPT4O generated with the same prompt. And if we click Run, you can see that it does pretty much the same thing. And then the text is animated. And there are three different options. One is slide in, fade in, and the other one is rotate. Okay, so I'm going to say this was a pass or partial pass, but let's look at a little bit more complicated example. Okay, in this next task I'm going to ask it to create a web page that is going to enable us to do a text to image using an external API. So it's supposed to create an HTML based web app which will take text from the user and then generate a description based on that using an external API. And then there are a set of requirements which basically first states that when the web page first loads, it simply needs to show a text box with text imaging something so the user can actually type in something in there. Then it's supposed to use the Replicate API and I am providing the documentation here. So usually if I'm working in cursor, I'll refer to specific documents. But in this case I am adding the Python SDK documentation for Replicate when it's using the flux model as an Example here and then also showing it how it can use an image model using that API. And all of this is in context. Now we also want two buttons below the generated image. One of them is going to show or enable us to download the image and the other one is going to regenerate. Then we are asking it that the backend is going to be implemented in Python while the front end is supposed to be in HTML. I am asking it to provide not only the structure of the project, but give me a bash command that I can just run and create the structure of the project. Right? So you can see that I am providing very detailed requirements in terms of what exactly I want. This is critical for these LLMs to actually generate something meaningful. For example, if you simply ask it create a Snake game in Python, it's going to look at its training data and put something together. But if you provide very detailed instructions to these models, they tend to do much better. Now here's what I did because a step by step guide. So first it gave me the structure of how the project is supposed to look like. Then it gives me a bash command that I can run to recreate that specific structure. And after that it started implementing each of those four files that are listed in here. So it first created the backend server, which is the app py. It's using a Flask API server. Then it created the index HTML. It implemented CSS for styling and then some JavaScript for interacting with the app. Now here are the detailed requirements that I need to install. So I actually need to install both Flask and replicate. Okay, so, so I ran the bash command, it created this project structure for me and then all I had to do was just copy the code from each of the files. We are ready to go. Now, something which I have noticed with this 32 billion model, for some reason it puts these special tokens in random places. Sometimes. I don't know if it has to do with the Hugging face Chat version that I'm using. It might be a quantized version or it's actually part of the model. So I'm not sure. Even in this case it did create or did add these special tokens. So I think the only adjustment that I need to make is to make sure that we remove the special tokens and then we just read the index zero from the output. Other than that, this follows the API documentations correctly. Now one more thing. It actually imported these two packages, OS and IO, but did not really use these packages in the code. Now it's a very small thing. But I won't really expect this from a LLM specifically that is trained for coding. So I think it should be able to follow the best practices for coding. Don't import any packages or libraries that you're not going to actually use in the code. All right, so I already have a virtual environment that I previously created. All we need to do is just run that Python file. So in order to run this, we're going to run app py. It's already running on the localhost port 5000, so let's see what the output looks like. So here's the app. So I'm gonna say create an image of a tiger and let's click on Generate. So on the back end we can see that it is able to make the API calls, which is pretty nice. And on the front end we see it's able to generate an image, which is pretty cool. Now, in the earlier version, when I was clicking on regenerate image, it was just showing me exactly the same image. So I just told it when I click on the regenerate image button, it's not updating the image on the web app. And then it not only updated the code, but actually told me what exactly could be happening. So right now it's actually thinking that it probably has something to do the way the images are cached. So it added current timestamps to ensure that every time a new image is created, we have the latest timestamps in there. And if I click on Regenerate image, it actually changes the image, which is pretty cool. Also, the download works, so you can see that it downloaded the image files. Now, I kind of did the same exercise with GPT4O as well, and it also gave me the same similar file structure. It also gave me a similar bash command that I was able to run to create a structure of the project. I like to do this because you can automatically create the whole structure and just copy paste the source code that is created by the LLM for you if you're not using something like cursor. Now, with GPT4O, I initially ran into an issue because it was not able to serialize JSON. So then I just copied that error here and it was able to adjust the code and then it started working. So here's the GPT4.0 version of the code base. Now let's run this and let's see if it actually works. For that I need to change my directory. In this case, we are going to use the image Generator app. If I type Alice, that is all the files that are needed and then we can run the same thing again. I have to kill the other process because I want to run this new app. Now here is how it looks like. So it has this imagine something twice. We're going to ask it create an image of Adama with sunglasses. Let's click on Generate image and you can see it generated the image. It also has the prompt space on top which is pretty cool. So you can actually just type the prompt again if you want or you can click on regenerate and it regenerates the image. Now in the backend it's using exactly the same model because it's using the Flux model hosted on Replicate and just making an API call and also you are able to download the images if you want. Okay, so we just looked at a couple of code examples and it does a pretty decent job at those, especially in a single Tron which is pretty impressive. Now is it GPT 4 or 3.5 sonnet killer? Well I don't think so, but I think it's a very impressive model. Especially you'll be able to run this on your local machine. Now the Quint team also showed that you can use it with cursor or which makes it an extremely powerful model. Although I don't know how exactly they did it. I'm going to look at some options to figure out what they probably did. Anyways, I think it's a pretty impressive model. I am going to try to run this locally when I have a stable enough Internet and then probably will report the results back. I hope you found this video useful. Thanks for watching and as always see you in the next one.