WizardCoder-34B and fine-tuned Code Llama variants now match or beat GPT-4 on the Human Evolved programming benchmark. WizardCoder scored 73.2% versus Code Llama's 69.5%. The benchmark is small with only 164 problems, but the models handle tasks like grouping characters and converting Roman numerals that stumped the original Code Llama.
Within 24 hours of the release of the Code Llama model, there is not one, but two different open source large language models which are able to surpass GPT4 performance on the Human Evolved data set. In the original Code Llama paper, the authors showed that the base Code Llama is able to surpass the performance of GPT 3.5 on the human Evolved dataset. But the performance was lower than GPT4. So in this video we're going to look at both of these models. We'll also quickly talk about the Human Evolved data set. But rather than just testing these on the benchmarks, I'm going to show you the results on a couple of questions that the original Code Llama model was not able to solve. Okay, so the first model that we are going to be looking at is, was released by pend. I hope I'm pronouncing this correctly. They have an amazing search engine which I would recommend everybody to check it out. Okay, so this was the first model that was released in the last 24 hours and they are releasing basically two different models. One is the fine tuned version of code llama34 billion model and the second one is the code llama34 billion python model. And the results respectively on the Human Evolved data set are 67.6% and 69.5%. Now according to the GPT4 technical report, GPT4 achieved 67% when it was tested at the release time. So this definitely outperforms GPT4 by relatively small margin. Now, by using their proprietary 80,000 examples high quality programming data set, they were able to improve the performance of the original Code Llama models by substantial amount. Now these are great results, but the major news came from the WizardLM team. The major development came from the WizardLM team. So they have a code specific model which they're calling Wizard Coder. Now they had a previous version which is the 15 billion parameter model, but they fine tuned the code code llama34 billion parameter model specific to Python. Now, from their evaluation they were able to achieve 73.2%. Now, compared to the GPT4 performance, this is a substantial improvement. Now where is actually this GPT4 performance coming from? So when GPT4 was released to public, OpenAI released a technical report alongside and over there they reported a performance of 67% on the Human Evolved data set. You also see 48.1% by the GPT 3.5 which is the ChatGPT. So when Meta released their Code Llama models, they used the exact same numbers in order to report the performance of GPT4 and ChatGPT, however, the wizard Coder team actually released an updated version. So they are comparing it with the original model, but they also perform their own evaluation. Now, in all fairness, the Wizard Coder team has provided performance values for two different versions of GPT 3.5 and GPT 4. So the one that you see with the March 15 date, these were the original reported by OpenAI. However, it seems like in the current version, the performance has improved substantially Both for GPT 3.5 and GPT 4 on the human Evolved data set. Now, if you compare the results of Visit Coder with the current version of the results which they got using the official API, so it's still able to beat GPT 3.5 by almost 1%, but it lags behind GPT 4. It's actually great to see that the Wizard Coder team put this together and made it very clear that they are simply comparing it to the original or official score that was reported by OpenAI. Now, there has been some criticism around the Human Evolved data set. So if you look at in details, it has only 164 original programming problems, which assesses language comprehension algorithms and simple mathematics. Right? So it's a relatively very small data set. Now, even though if you use this as a test set on multiple iteration of the same model, it's actually possible to overfit your model to your test set. So it might be the case that some of these models might be simply overfitted to this data set. Right. So we definitely need much bigger and probably relatively diverse benchmark data sets from now on. Now, having said that, we are not taking anything away from these amazing new models that are coming up. Just keep in mind at the start of the year there was no open source model which was even closer to something like ChatGPT. And now we are looking at a simple 34 billion parameter model which is on par with ChatGPT for coding. Now that's enough regarding benchmarks, I wanted to actually try this on some of my own code examples. So I simply tried the Wizard Coder model because that's the best model out there at the moment. And I tested it on two different code examples which the original code llama model was not able to solve. So the first one is that if you're given a string, write a Python function that rearranges the characters so that the matching letters are grouped together. The output should be string like this. So for example, here is the original string and then the corresponding string will simply Put the same characters together. Okay, so the original Code Llama was not able to solve this. I'm using the Gradu app that was provided by the Wizard Coded team and this is the output. On the first try I had to wait for it to generate the response. So I'm simply showing you the pre generated responses here. Okay, so I copied the code in a Google Colab. Then using the same example, I was actually able to get the correct response. This is pretty amazing because when I tested the original Code Llama as well as the Chat GPT, they were not able to solve this on the first try. I have that video, so I'm going to put a link to that video if you are interested in more comprehensive testing. Okay, so here's the second example. Write a Python function Roman to integer that converts a Roman numeral to an integer. Now the original codelam model was not able to solve this, but here's the code that was generated by the wizard coder. So when I tested that code and provided a Roman numeral, it was able to give me the correct answer. So this is amazing. Please. On my tests it's able to solve the two problems that the original Code Llama was not able to solve. So I will say that at least the wizard coder is much better than the original Code Llama model. Okay, that's it for this video. Some really amazing stuff are happening in the open source large language model space and this is really exciting. I hope you found this video useful. Consider liking the video and subscribe to the channel. Also, make sure that you turn on the push notification because YouTube is not sending notifications to all the subscribers when a new video is updated. So make sure that you turn on the push notification so that you don't miss a new video. Thanks for watching and see you in the next one.
