SMALL BUT MIGHTY - 13B Model Beats Llama-65B NEW BEST LLM!!!

If you go to hugging face and check for open source large language models, there is almost a new model every week just for text generation. We have around 19,000 different models. And if you look at the famous TheBlokes account, he has converted around 750 different models. So it becomes really hard to keep track of them and cover these models on this channel. And that's why I kind of took a step back to cover each and every new model that is being released. But the model I'm about to show you actually caught my eyes. And the reason was that the authors claim that the 13 billion parameter model can beat the original 65B model, which is actually a really bold claim. Now to be honest, if you look at the actual results, this model outperforms the original Llama 65B model only on one data set. That is the truthful Q and A. But for the two other data sets, ARC and Hellas Swag, it really comes close to the original 65B model and it outperforms all the rest of 13 billion parameter models. Now another interesting thing about this model is it's not really a new original model. Rather it's a merge between two very powerful models. The model that we are looking at is called open arca platypus 213b and as the name suggests, it's a merge between Open arca and the Platypus 2 models. If you look at the Open LLM leaderboard, the Platypus 2. 70 billion model is actually the current leader. So in this work the author have used a smaller version of the Platypus 2 family and have merged it with a model trained on the Open ARCAD dataset. And as a result they were able to get a very powerful model that is very small in size but is huge in its capabilities. The performance of this model on the benchmark data set is very impressive and you can look at these results yourself. I'm going to put a link in the description of the video, but let's test it ourselves. Now for testing any large language model or any machine learning model in general, it's very important to understand what type of data set was used to train the model. For example, if a model is trained on English literature and you are testing it on programming examples, then it's not a fair comparison. So let's see how this model was trained. Now they use two different data sets. The first one focuses on STEM and logic based data set and this was used to train the original Platypus 213B model. The second data set that is used to train the open ARCA Open Chat model is the open ARCA data set, which is actually derived using GPT4. Now just by looking at these data sets, it seems like it should be good for logical problems as well as things related to science, technology and engineering. Now to give you a quick example of what type of data set was used to train, so we're going to look at the open Platypus data set. So for example, these are the type of questions that you can expect the model to be better at. So these are mathematical questions and I think that I also saw some stuff related to programming and engineering. Now here are some example questions from the open Archive data set. This seems to be much more diverse. So we're going to be testing this model both on some logical questions as well as some programming questions. Now if you want to test this model, you have quite a few options. You can use the Uber BUGA text generation web UI to run this model locally. Here I have downloaded the model. This is the full model. If you are looking for the quantized version, the bloke has already converted into GGML format. There is also GPT Q format as well. So these are four bit quantized models. Or if you don't want to run this locally, there is a free demo on Hugging Face that is available for you to test out. If you want to run this locally on your own machine, you will need to use the following prompt template which is the Alpaca instruction only prompt tablet. So you have the instruction, then you provide your prompt and then you will get the response. So here is how it's going to look like in the Uber Booger text generation web ui. So first we have the instructions response and here is a sample prompt that I'm using. I'm asking the model to write a letter to the CEO of OpenAI to make GPT5 model open source. So here is the output letter and I must say it's a very well written letter. Now it starts off by saying as a passionate programmer, most of the other large language models would say as a assistant or as a large language model. So this is very refreshing to see. The overall letter is very coherent and it's actually very well written. Probably one of the best I have seen from open source large language models in a while. So for the rest of the video we are going to be using the demo provided by the authors. You can also set some advanced parameters at the bottom of the page. So the only parameter I'm changing is Actually the temperature, so let me bring it down to 0.3 and, and let's take the max number of tokens to 800. So that's the maximum tokens that it's going to generate in response. Now since this is based on the ORCA data set, so you can actually also provide system message if you want, but I'm going to just keep it blank for this simple test. So my second question was explain the correct usage of homophones in the following sentences. The principal is your pol and the site at the site was quite a site and it's actually able to correctly identify the homophones. Example here says the homophone principle refers to the head of the school or college and the site used in two different contexts. So the first refers to the ability to see or perceive something visually, right? And then this second is something which is visually striking or impressive. So, so this seems to understand language, which is pretty impressive now since it's supposed to be trained on STEM fields. So let's ask this very simple mathematical question. What is two plus two? And the answer is four. That is correct. Now let's see how this, how good this is at computing probabilities. So very simple question, a six side die is rolled. What is the probability of rolling a four? Right? So it's supposed to be one by six and actually it got it right. So that's correct. So let's make the question a bit more complex. So we have a jar which contains four black balls, three white balls and three red balls. If two balls are drawn without replacement, what is the probability that the first ball is black and the second ball is white? So it actually got that wrong. The reasoning is kind of correct because it says there are four black balls. So the probability of drawing a black ball is four by eight since there are a total of eight balls in the jar. But it got that wrong because there are a total of 10 balls in the jar, right? And then like if you draw, let's say one ball from eight, so you are left with seven and then there are three white balls, right? But since the actual number is supposed to be this is supposed to be 10 and 9. So it says it's incorrect. Although based on the computation of assuming that the total number is 8, the answer is correct. Now it's kind of funny because it kind it got the hardest part right. But simple addition, it got wrong. Now let's see if this model can do some further logical reasoning. So here's the prompt a glassdoor has push on it in the middle, writing should you push or pull it. Please think out loud. Step by step. So I think I have seen only one open source large language model which got this right. Now the model came up with a step by step reasoning process. But the final order, the final answer is actually not correct in this case. So I wanted to have some fun with the model and I asked it how to take over a country, create an elaborate plan for the takeover and provide step by step process. If you ask the same question from ChatGPT, it's not going to give you a response, but at least it attempted to give me a response. So it kind of says identify the country you want to take over, gather resources and support, then create a well structured organization, gain public support, establishes presence in the country, weakening the current government and then capitalize on opportunities and consolidate, consolidate power and stabilize the country. Now, the model itself is not fully uncensored. If you ask some other questions, so it probably is going to same that as a large language model, I cannot respond. But there are still some fun things you can do with this model. So at the end we're going to simply look at a couple of programming tests. So here is the one I usually use. Write a Python function that accepts a file and write it into an S3 bucket and the code that it generated seems to be correct. Now the formatting is a bit off, but that's fine. And this is a question for which you can easily find an answer online. It's not really a hard question at all. Now let's ask a little more complicated question. So we are asking it to write HTML code for a web page that is like a single button and when the button is pressed, it will change the background to a random color and it will also display a random joke. And I asked it to put this in markdown because the HTML code was actually disappearing. So what we're going to do is we're going to simply copy the code that it generated and we're going to go here. So this is an online resource where you can test your HTML code. So just pasted that. Let's run this. So we do see the background color has changed. Now if we click on it, it's changing the background color, but it's actually not showing the joke. Now since we are running a chat model, so we can actually go and tell it to fix the code. So I told it the background color is changing, but it's not displaying a random joke. Can you fix the code? And let's see what happens. Okay, so even after telling it, there's an issue, it simply returned the same code. Then I asked it again that the code is still incorrect. It doesn't show any jokes, right? So it gave me exactly the same code again and I asked it what changes you made in the code. So it listed a few changes, but those are actually not in there. Now the mistake is actually pretty minor and can be easily fixed. So based on my tests, I would say it's a reasonably good model for its size. Now a couple of things which I wanted to highlight when it comes to the Open LLM leaderboard or any other leaderboard for that matter. First thing is that these are benchmark data sets, so your own applications may not be relevant to the results that you see on these benchmark data sets. Now if you are evaluating a model for your own application, just make sure that there was no data leakage between the training and test set. Data leakage is a common problem in machine learning where part of the training part of the test set actually goes into the training set. Or you have very similar questions or examples in the training set and that's why you kind of see a bloated performance of the model on the test set. So make sure if you are choosing a model for your application to look at the test set and see how different this the test set is from the training set. Overall, it's really exciting to see all the innovation that is happening in the open source large language model space and to see that in just a matter of few months how far we have come. Consider liking the video if you found it useful and subscribe to the channel for similar content. Thanks for watching and see you in the next one.