CODE LLAMA - THE BEST CODING MODEL IS HERE!
LARRUZ breaks down Meta's Code Llama family: three released models (base, Python, Instruct) in 7B, 13B, and 34B sizes, plus an unreleased high-performing variant. The 34B model rivals GPT-4 on coding benchmarks. Training on 500B tokens of code and a small curated instruction set of 15k examples drove the gains.
Chapters
Meta just released yet another large language model. This time these models are focused on coding. And these models were able to achieve some really impressive results. And there is a unusual 34 billion model which blows every other model out of the water except GPT4. Okay, so let's quickly look at the announcement and then we are going to look at some very interesting things from the paper. So in this blog post they are introducing a family of models which they are calling codelama. So Code Llama is a state of the art LLM capable of generating code and natural language about code from both code and natural language prompts. So it seems like it has the ability to generate code based on the natural language prompts as well as explain a given piece of code or debug it. Now, since these models are based on llama 2, so they are free for research and commercial use, they're releasing three different models. The first one is called Code Llama, it's a foundational code model. Then there is a Python version and the last one is the Code Llama Instruct which is fine tuned for understanding natural language instructions. Now there is another version called Unnatural Code Llama which is the most powerful and most capable, but that has not been released. And later in the video we are going to look at some of the unusual results from that Unnatural Code Llama model. Now each of the model is being released in three different sizes. So there is a 7 billion model, 13 billion and 34 billion model. For some reason Meta has not released the Llama 234 billion model yet. From earlier reporting there was seems to be having issues with alignment, but it's interesting to see that they were they are releasing the code version but not the LLM itself. Now, as expected, the 34 billion model is the one that is the most capable model, but that's going to have the highest latency as well. So that's why they are offering the smaller 7 billion and 13 billion models. Now, since we are talking about code generation, the number of tokens in the context window matter a lot. And that is why they have ensured that these models are able to provide stable generations up to 100,000 tokens of context. Now the model itself were trained on 16,000 tokens. However they can go up to 100,000 tokens when it comes to code generation. And this is great because for coding you need a very long context window. We are going to be looking at some very impressive results. But let's look at a simple example that they have provided. So you simply provide a prompt, the Model is able to generate a code based on the prompt and then you can do some plotting as well because it's just Python code. Now, before looking at its impressive results, I want to show you how this model was trained. So I'm going to be using this figure generated by Diego Santos on Twitter or X the way it's called now. So Meta started with llama2, the base versions and they're using three different foundation models. Then they used a total of 500 billion tokens for code training. After that, as we said, there are different variations. So for example, for the Python version they trained it for an additional 100 billion tokens. Then in order to increase the context window, they fine tuned it with another two 20 billion tokens and as a result you get the code Llama Python version. Now for the instruct fine tuning version they followed a very similar methodology. So initially they extended the context window with 20 billion tokens and then they further performed instruct fine tuning with 5 billion tokens. Next we will look at some of the results which shows why this is such a big deal. In this table they have provided performance of different models on different benchmark data set. We want to focus on the Human Evolve and MBPP data set because these are code oriented data sets. Now the thing to keep in mind is that apart from GPT 3.5, GPT 4 and llama to 7 billion parameter model, all of the rest of the models are actually fine tuned or trained specifically for coding tasks. Now we really want to focus on these 34 billion parameter model for the code llama and you will notice that apart from GPT4, it's able to beat every other model when it comes to the Human Evolve and MBPP data set. And this is actually huge because 34 billion parameter model is a fraction in size compared to something like GPT 3.5 which is at least 175 billion parameter model. Now this shows that task specific smaller models can be much better than much larger general purpose models. Therefore, it's amazing to see that such a small model is able to get such great performance. Now this is impressive. However, there is one more model which is a lot impressive than what you see in here. Now if you look at the paper, there's this model called Unnatural Code Llama which is not discussed at all in their blog post. But this is probably the most impressive one because it's able to beat every other model by a substantially significant margin. The performance of this unnatural code llama model is very close to GPT4. Now, details about this specific models are missing in the paper. Now the only reference to this unnatural model in the paper is that it seems to be a fine tuned version of the code llama python34 billion parameter model with just additional 15,000 examples and they are calling these unnatural instructions. So these are basically self instruct data set. So I'm assuming this is generated by the LLM rather than human generated data set. So it's very interesting to see that just additional 15,000 example can give you such substantial improvement in performance and that highlight the importance of carefully curated high quality data set when it comes to training any machine learning model and LLMs in particular. When you look at these results it definitely seems like the gap between proprietary models and open source models is shrinking when it comes to particular applications. In this last section we will just cover how you can get access to the model and so there is a request form that you need to fill out in order to get the official weights and code to run the models. However, if you don't want to do that then the bloke has already converted these into GGML and GPT Q format so you can directly download these models right from Hugging face. I will be exploring these models in the next couple of days and we'll create a detailed video on how to run this as a part of your own workflows. It's very interesting to see that Meta is the company that is driving the innovation when it comes to open source. Soon we might have a Codex alternative which is completely open source. Let me know what you're planning on building with these models. I hope you found this video useful. Consider liking the video and subscribe it to the channel. Also make sure that you turn on the push notification buttons. Because of the YouTube algorithm changes, you are not going to receive notification by default. When I upload a new video you need to turn on the push notification. Thanks for watching and as always, see you in the next one.
