Zephyr - 7B Beating Llama-2 70B Model

We have a new very interesting 7 billion parameter model that is able to beat the llama 2. 70 billion model on the empty bench benchmark. Now, the performance of this model is very impressive for its size. The model itself is called Zephyr and it's a fine tuned version of the Mistral 7B model. Although the performance of this model is impressive, that's not the most important thing about this model. Rather it's how this model was actually trained. They did the initial fine tuning of Mistral 7B on Ultrachat data set and the data set comprises of around 2000 examples. After the initial fine tuning they did an alignment of the model. But instead of using something like RLHF that OpenAI is using, they use this new technique called Direct Preference Optimization. I'll put a link to the paper. So this technique is coming out of this paper titled Direct Preference Optimization. Your language model is secretly a reward model. So essentially they argue that you do not need a reward model in order to do alignment. Now the DPO seems to be a lot more stable compared to the other techniques that are used for alignments and it's much more faster. So it takes just a few hours compared to days for something like PPO or RLHF. So it took them only 8 hours of training on 16 A100 GPUs or around $500 in order to get these results. For the rest of the video we'll first look at the training set on how this model was trained. Then we will talk about the performance of the model and at the end we will look at some of our own testing. So let's get started. Now for the initial fine tuning of the model they use the Ultra Chat data set which is basically a conversation between two different ChatGPT instances. So they are having a multi turn conversation. So here is the description of the Ultra Chat data set. Now one thing you will notice is that it's actually purely synthetically generated data set because it's a conversation between two separate ChatGPT Turbo APIs and the goal is to generate multi round dialogue between these two chatbots and then they further filtered it. So the questions or the conversations are limited to questions about the world, writing and creation and assistant on existing material. They further align the model with another data set which is called Ultra feedback data set. So let's look at how this data set is collected. Now based on the official description, it's a large scale, fine grained, diverse preference data set used for training powerful reward models and critic models. So here they use a total of 64,000 prompts from different data sets, and then they use those prompts to get response from four different models. Now, it seems like for each of the prompt response pair, the models that we use are actually different. So for example, for the first one you see there are. There is Alpaca Patia, then Starchart and Vicuna 13B. In the second case, the models are GPT4, Llama 213B, Star Chat and Ultra LM with 65B. Now, as a final step, they asked GPT4 to annotate or rank the response based on these four different criterias. So the criterias are instruction, following, truthfulness, honesty, and helpfulness. Now, for the Zephyr models, they're simply focusing on helpfulness. Now, this means that instead of RLHF, they are basically using another feedback mechanism. But in this case, the ranking is done by GPT4 rather than human feedback. Okay, so I hope this gives you a better understanding of how the model is trained. But let's see what is the impact of this training paradigm. So here is how its performance looks like relative to something like llama 270 billion. If you look here, it actually outperforms 70 bill llama 2 model on coding extraction. That means it's probably a good candidate for rag. So I might change the default model in local GPT to Z47B. Now, in stem fields, it also beats llama 270 build model by a small margin. And for humanities writing role playing, the performance is very similar to Llama 270 build model. Now, it does lag behind Llama 270 build model in reasoning and mathematics. Right, but that's expected because the Llama 270 billion model is 10 times bigger than this model. But it has really impressive results when it comes to the other categories. Okay, on hugging face, they have provided a link to the demo. We're going to use the demo, but they'll have also provided an integration with the transformer package. So you can run this in your own applications. Let's test the model on some of our test prompts. For this, we're going to be using the demo site that they have provided. Let's first check its writing abilities. So the prompt is write an email to the city appealing my $100 parking ticket. Appeal to sympathy and admit I parked incorrectly. I would say it came up with a very good letter. It starts off by saying, I'm writing this letter in response to the recent parking fine that was issued against me. And then you can put in the date, then as much as it pains me to say it, I believe there are compelling reasons why the fine should be waived or reduced. Now, it goes on to say, first and foremost, please let me express how genuinely, genuinely sorry I am about the situation. Then it admits that we did park incorrectly. So then it goes on to say, however, at no point during the incident did I intend to cause any inconvenience or violate any rules. Unfortunately, circumstances beyond my control led me to make an error in judgment which resulted in me receiving a citation. Right. So. And then it goes on to say, like, why that happened. Right. And then appealing to the sympathy. Okay, so it's actually a pretty good letter. Now, one thing which I would have liked for it to include was that $100 parking ticket number, but that amount is nowhere in the letter itself. But still, nevertheless, it's a pretty good letter. Now, since we are comparing it to llama 270 bill model. So here is the response from llama 270 build model for the exactly same prompt. And I would say that it's also is a pretty good response, but I personally like the other one. One of the things that they have highlighted is that it's not aligned. So let's see if it actually can give us answers to some relatively controversial questions. So for example, here I'm asking why is Democratic Party not a good option for the country? And let's see if it's willing to generate a response for us. So here is the response. It says, I do not have personal beliefs or opinions. However, some people may argue that the Democratic Party may not be the best option. And then it simply list a few reasons that people might think about. Now we can do the same for the Republican Party as well. And I think it's willing to give us answers. Okay, we can further check its coding abilities. So here is the prompt that I usually use. We wanted to create a web page that has a single button. When the button is pressed, it's supposed to change the background color of the website to a random color and it also supposed to display a random joke. Let's see if it can write this code. Okay, so here's the code that the model generated. So we're going to simply copy this code and we'll paste it in here and simply click run. So when we click run, we do see that it generated a web page which says press the button below for random jokes and colors. So let's see. Click me. Okay, yeah, it does change the color. And it's also randomly, I think picking jokes as well, but sometimes I think it doesn't change on each click, but it's a random selection, but at least it works so this is paas now based on my tests, it definitely seems to be a very impressive model for its size and it's great to see the progress these open source large language models are making and it seems like everything comes down to proper selection of the training or the fine tuning dataset. This has been known in the machine learning community for a while that the data set matters, and it's great to see that the LLM community is also catching up on it. I'm putting together a list of prompts or tests to standardize my test in these videos. So if you have any recommendations for a good prompt to test these large language models, put them in the comment section below or share them on our Discord Server. Link is in the description of the video. Now this model was based on the original Mistral 7B model and if you want to learn more about that model, watch this video next. Thanks for watching and see you in the next one.