FreeWilly (Orca 🐳): NEW LLM Leader! Better Than Llama-2

It's been less than a week since Llama 2 was released and there is already an open source model which is outperforming it on some of the benchmarks. The new best model is called Free Willy 2 released by Stability AI. Stability AI is the company behind the Stable Diffusion project. How were they able to outperform the Llama 2 model? For answer we need to look at the ORCA paper by Microsoft Research. Although we are still waiting for Microsoft to release the model as well as the data set, there has been a number of open source projects. They have been able to replicate the data creation process proposed in the ORCA paper and have resulted in some really impressive models. The new free variable models uses a very similar approach. So they are basically fine tuning a 70 billion parameter llama2 model with the data set curated using the ORCA approach proposed in the original ORCA paper. So Stability AI is releasing two different models. The first one is called Freeville 1 and the second one is called Freeville 2. The original Free Willy 1 model is actually a fine tuned version of the original llama 65 billion parameter foundational model and that was carefully fine tuned on a newly synthetic generated data set. And we're going to look at how this data set was generated. The free VD2 model is actually a fine tuned version of the llama 2. 70 billion parameter foundational model. And according to their evaluation, the performance compares favorably with ChatGPT for some tasks. Now I just want to say this, the results are on benchmarks. The mileage on real world applications might vary. Now one not so good thing about these models is that even though they are open source models, they cannot be used for commercial purposes. And I think it comes down to the data creation process that they have followed because the base Lama2.7 billion parameter model is open source and you can use that for commercial purposes. So let's look at the data generation and collection process and see what exactly they have done here. As noted here, the methodology that they adopted was inspired by the ORCA paper proposed by the Microsoft research team. And they go on to say that while our data generation process is similar, we differ in our data sources. So the data set and model associated with the ORCA paper are supposedly open source, but they have not been released by Microsoft Research. And that's why all the open source community or projects are actually using the methodology proposed in the paper to create their own data sets. For training these models, they have used nearly 600,000 data points or examples, which is around 10% of the original data set proposed in the ORCA paper. If you notice, these are relatively a small number of examples, but they are high quality instructions and that's why the resultant model is really incredible in its performance. This shows that you need to pay close attention to your data set that you're going to be using for training or fine tuning these large language models. And as a result of this, the state then despite training on 1/10 the sample size of the original ORCA paper, the resulting Free VD models demonstrate exceptional performance across various benchmarks. And again, that comes down to how good your training dataset is. Let's talk about the performance evaluation. So they have used the LM Evolve Harness dataset, which is their internal dataset by the Eleuther AI. So Eleuther AI is also a company under Stability AI. According to them, the Feeville models excel in many areas including intricate reasoning, linguistic and then answering complex questions related to different domains, for example law and mathematical problem solving. So they seem to be pretty general models. Now let's look at a couple of very interesting comparisons that they have provided on different benchmarks. Keep in mind, these results are on benchmarks. Real world performance will most probably vary than what you see on these benchmarks. So with that in mind, let's look at this comparison. So they're comparing both the models to ChatGPT or Turbo 3.5. Now we know that ChatGPT has not been evaluated on all different benchmarks. So the results that you see here, they are probably not official results from OpenAI. So at least on one benchmark, which is the helicewag, the Freeville 2 model is able to outperform ChatGPT model. For the rest, it doesn't really seem to be closer. For MMLU, it's within I guess like 1%. For the AIC dataset, the performance is pretty bad relative to a ChatGPT. However, on the AGI Evolved dataset, the results are really impressive and they're really comparable to ChatGPT. If you look here, out of the eight data set on which the comparison was performed with zero shot, FreeVidi is able to outperform ChatGPT on six out of the eight different benchmarks. And actually I would say this is pretty impressive. Now there is this very interesting table which is comparing free vd1 to free willy2. And the results are surprising for me at least. Like, even though this Free Willy 2 is based on the Llama 2. 7 billion model and Free VD1 is based on the original Llama 65 billion model, the results are pretty comparable to each other. Like I don't see a clear winner. I think they are performing pretty consistently on these data sets. So this kind of shows the importance of the data set rather than the size of the model because both of them are trained or fine tuned on the same 600,000 examples. So even though there is a size difference of 5 billion parameters, I don't think it translates into a huge performance improvement for the Vidi 2 model. Now the question is, can you use these models? And the answer is yes. So the models are already released and you can access them on Hugging Face. And since it's based on Nama 2, so it's fully compatible with the transformer library from Hugging Face, there are even a couple of spaces where you can experiment with these models. However, you need to be aware that the wait time is too long for them to be useful. For example, here is a test prompt that I want to use in the Stability AI free video spaces. And if you notice, I am actually 48 in the queue and it doesn't seem to be changing at all. Right, so probably we need to wait for somebody to generate or create a quantized version of this model. Probably you will be able to run a two bit quantized version because it's the biggest Llama 2 model. I think it will be very interesting to see its performance in real world. Okay, so I'm still in the queue, but I don't think I'll be able to include this in the video. But the amount of innovation that is happening in the open source community is simply mind blowing and impressive and to be frank, really hard to keep up with. I hope Stability AI will release a smaller version of the model at some point so that it will be more useful for general population. So this was a quick update on the free VD projects. I hope you found it useful. Thanks for watching and see you in the next one.