Which OPEN LLM is the BEST?? (3B to 13B models)

Everybody has their own favorite open source LLM. But which one is the best? We might have an answer. Now, the folks who created Vicunia have put together this amazing website where you can actually objectively compare different large language models. You can experiment with a total of nine different models, including Vicuna, Koala, Open Assistant, Fast Chat, T5, Alpaca, ChatGLM, Dall, E2, StableLM, and the original Llama model. Apart from individually looking at the output of these models, you can put them in a competition with each other. You can do it in two ways. The first one is called Chatbot arena, where you can chat with two anonymous models side by side and vote for which one is better. Now, this is an excellent way of comparing different models because you don't know which model is which. So you simply I shown model A and model B. You give a prompt, you get a response, and then you get to choose which one is the best. And after that it will show you which model it was. The second possibility on their website is you can simply choose one model. So let's say vocunia 13B and choose another model from the list of nine models. And then you look at the responses on the same prompt and you can choose which model is the best. Now, this first methodology is the one which is the most powerful one because it's simple blind testing. So based on the data they have collected, they actually provide a leaderboard where you can see all the models compared and see which one is the best on human feedback. Now, this is great because we're not looking at their performance on some benchmarks, but rather on actual human feedbacks. We will look at the leaderboard in a minute, but I just want to experiment and play around with this blind model comparison first. Now, to do this, we are going to use a very simple prompt. So the first prompt is going to be draft an apology email to a customer who experienced a delay in their order and provide reassurances that the issue has been resolved. Okay, so this is going to be our prompt, the first for the first comparison. So let's send this prompt in and it will randomly pick two models and those models will start generating responses. Now, it would be actually interesting if you pause the video and read both response, read both responses and decide for yourself which model is which. You could probably make a guess. So the first one is I apologize for the delay in the processing your order. It's more of a canned response. The thing which I liked it was that it says we recently ran into an unexpected issue with the raw materials needed to fulfill your order. So it kind of give it and give the customer a cause of the why the delay happened. So this is pretty nice. Now the second model has a similar response, but it's a lot more detailed. So for example here it says our team has been working hard to identify the root cause of the delay. And a thing which I really liked was that as a token of our apology we would like to offer you and then insert compensation or offer if appropriate. Right. So this is pretty good. I would suspect this is going to be either Vicuna or Koala and this is probably a smaller model or even the original app. They actually have access to GPT4 and ChatGPT. So sometimes the responses can be coming from them and we don't know ahead of time. But in this case I think model E is better in my opinion. So I'm going to select model A and then you see that. Oh, okay. So actually ChatGPT that generated this response and this one is coming from Dall E version 2. Now this is a great way of providing feedbacks to the system and your feedbacks are actually used for retraining these models. So it's, it's a great way that you can contribute to the open source efforts if you want to compare two different specific models together. So you can actually come here and select those. So let's say I want to compare vocunia versus Stable LM and my prompt in this case is what are the main differences between Python and JavaScript programming languages. So based on my experience I would expect ocunia to be better. But let's see, both of them actually came up with a very similar response. So let's say they compare it in terms of syntax, memory management, function calling, object oriented programming, iteration, and in this case it's syntax, data type functions, scope, libraries. Right. So I think these are similar responses or similar comparison. But I really like this comparison because you could be actually really surprised based on the results. Sometimes we think that the smaller models are not really powerful. But what I have seen is in my responses, sometimes the smaller model can outperform the larger models. Now let's look at the leaderboard, but before that let's see what the community thinks. So maybe order them on the based on what your opinion is. Which one are the top, let's say three or five. It would be interesting to see that in the comment section below. If you like to do that, pause the video and maybe post a comment. All right, so based on all the human feedback, here is what people think Right, so this is the great thing about their method, that they are evaluating it on the human responses rather than on some benchmarks. Now keep in mind that the leaderboards board is based on the data that have collected before May 1. Right? So it could change actually. But overall Mokonia seems to be the most liked model followed by Koala and then Open Assistant, then Alpaca, then chat glm and fastchat dall e2 surprisingly is actually on a much lower rank than I would have suspected and similar to similar is the case with Stablelm. But I think Stablelm was just released so maybe that's the case. The most surprising part for me was this Fast Chat which is a FLAN T5 model that is fine tuned to be like a chat assistant. Now this is the smallest of the models, so it's just 13 billion parameter compared to let's say 13 or 12 billion parameter. But it's surprisingly doing well compared to the other models. Now they have provided a really comprehensive statistical comparison. So let's say here are a few plots if you guys are interested in. So the first one is fraction of model A wins for all non tied A versus B battles. Right? So let's say in this case Vicuna versus the rest of the models. So it's like if there is a one to one comparison. So vicuna was able to beat koala almost 69% of the time, 68% to be exact. Then vicuna versus open assistant. So around 77% of the time. Right. So you can look at these results and here's a better count for each combination of the model. So this means that how many times different models were compared? So let's say vicuna versus koala it was compared to 422 times. Now based on these results you can see that the fast chat T5 model as well as the stable LM models, these are not compared a lot with the rest of the models. So that could be a reason for it. Stable than I am not being high up in the rank. Now if you take average win rate against all other models, so not just one one comparison, then Vicuna is the overall winner and this is probably expected. Then there's Koala, Alpaca and Open Assistant. Similarly they provide something called ELO scores or estimates which measures the relative performance of different models. And even on that Vicunia is the top model, they actually have a very comprehensive description based on their methodology to explain it. And they have even provided a Google Colab notebook with all the data available. So you can actually go and experiment with it or if you want to perform any other statistical comparison, it's actually possible. All the results that they have shown is actually and derived from this Google Colab notebook. It's actually great to see these efforts to objectively compare different open source models and the results I think are not unexpected because if you look at online discussion everybody thinks that Vokunia is the best model overall. It would be actually nice to see if they can add some more models. For example, there is the Stable Vicunia. Now there's even Visit LM and just recently another model called Wizard Vicuna was also released which is very promising. I hope you like this video. If you are interested in learning more about large language models, you can actually check out my channel. I have a whole bunch of videos on different large language models. There is even a dedicated playlist so you will find all the models that have been released so far. For example, there's OpenNEMA, stable Vicunia Kunya with Vision and almost all the other models. If you're new here, consider subscribing to the channel and push that bell notification button. That way you don't miss any new videos. I hope this video was helpful. Thanks for watching. See you in the next one.