NEW "Orca-2" 🐳 Official Release - 13B Better than 70B Models

Back in June 2023 Microsoft released this paper Orcam Progressive learning from complex explanation traces of GPT4. In that paper the researchers showed that a smaller 13 billion parameter model can outperform much bigger models including ChatGPT and Bard on certain benchmarks. At that point Microsoft did not release the model weights, but they explained the data creation process. There were a number of open source project which adopted the same data creation process and as a result we had some amazing open source large language models. Now Microsoft has announced the second version of orca, but this time not only they are releasing the paper, they are also releasing the model weights which is a great news for open source community. Orca 2 is a very impressive model and as a result we have a 13 billion model that consistently outperforms a 70 billion llama 2 model on reasoning capabilities which is extremely impressive. So in this video we're going to quickly look at some of the technical details of Orca 2. We will look at some of the benchmarks results that are presented in the paper and at the end of the video I'm going to show you how to use this model locally. The paper that was released with the model is titled Orca2Teaching Small Language Models how to Reason. As you can deduce from the title, the main focus is on how to improve the reasoning abilities of small models. If you really want to understand how to properly use smaller models, I would highly recommend to read this paper in the abstract. The author says research on training small LLMs has often relied on the imitation learning to replicate the output of more capable models. Then they say we contend that excessive emphasis on imitation may restrict the potential of smaller models. And that's why they say we seek to teach small LLMs to employ different solution strategies for different tasks potentially different from the one used by the larger model. For example, while larger models might provide a direct answer to a complex task, smaller models may not have the same capacity. Now, most of the open source large language models are simply trained on the data generated by the larger models such as GPT4. So you have the instruct fine tune data set and you just simply fine tune these models. However, they propose a different technique for training these models. That's why the author says we teach the model various reasoning techniques. Those include step by step recall, then generate recall, reason generate direct answer, etc. More crucially, we aim to help the model learn to determine the most effective solution strategy for each task. In the original ORCA paper, the Microsoft Team presented a number of different system messages and these are helpful depending on the task at hand. Now, using the training strategies that they are proposing, Orca 2 significantly surpasses models of similar size and attains performance levels similar or better to those of models five to ten times larger. Now it all comes down to the way the model is trained and the training data itself. So according to them, the original ARCA model was a 13 billion parameter large language model that demonstrated strong reasoning ability by imitating the step by step reasoning traces of more capable models. Orca 2 comes in two flavors. So there is a 7 billion and then there is a 13 billion version. Both of them are fine tuned version of Lemma 2 base models. The main thing was this is trained or fine tuned on high quality synthetic data set. They say we are making the Orca 2 weights publicly available to encourage research on development, evaluation and alignment of smaller models. I was trying to understand their license. So it says Orca 2 is licensed under Microsoft Research License. It seems like it's only for research purposes, probably not for commercial purposes. However, the base Lama 2 model is also licensed for commercial purposes. So I don't know how the licensing of this model is going to work. Before showing you the results on benchmarks as well as how to run this, I just wanted to show you how this model is trained. Now let's see some of the benchmarks before I show you how to run this locally. Now in here the dark black color is the Orca 213 billion model and the dark green color is the Llama 270 build chat model. Now on the GSM 8K dataset, the Wizard LM 70 build model outperforms the 13 build model Orca 2. But on the rest of those the performance is very close to the Visit LM70 build and it outperforms the original Llama 270 build chat model on almost all the benchmarks. Next, let me show you how to run this both locally within a Python environment as well as a graphical user interface. So if you are using Python you can use the Transformer package to actually load the model. So you can use the Automl for Causal LM. Then provide the model details. For example, this is the model repo ID for the 70 build version and you can also provide the 13 build version. Now the rest of the process is very similar to what you see when you are using the Transformer package. But let me show you how to run this within the LM Studio. So here we are in LM Studio. I personally really like this because of its very clean interface. Now keep in mind, it's not completely open source, but it's one of the best UI for testing large language models. I have a complete video on LM Studio, so I'm going to put a link to that video in the description. So first what we need to do is we need to simply come to the search box and type orca2, click enter and then it basically looks on Hugging Face wherever the quantized versions of that models are available. So here you can see a list. Now I like to use the Blokes version, so I wanted to use the 13 billion version, so I clicked on this. On the right hand side you will see different quantization levels that are available on Hugging Face. Now depending on your hardware, you select one of them. So for example, I selected the 4 bit quantized version. I have already downloaded it and this will take around 8 gigabytes of VRAM. Okay, next we're going to go and click on this chat icon and on the top you can select the different models that you have downloaded. In this case we want to test the Orca 2 model. I'm going to click on that. That will load the model in a new chat. Now I'm running this on an M2, so let me show you my configurations. So first we need to change the preset. So click on this and then select the chat ML because that's the preset Orca 2 uses. So you can see now we are using the appropriate prompt template in here. The other thing that I wanted to highlight was this. So if you're running this on, let's say a Windows or Linux machine and you have limited GPU ram, so you can click on this to keep the model, keep the entire model in the ram. But in my case I am running this on Apple Silicon. So that's why I have selected Apple Metal gpu. And you can change the number of threads if you want. Another thing that you want to set up is this context length. So by default I think it was 1500. I change it to 4096 so that it's compatible with the context length of Llama 2 model. When you're testing this model, one other important aspect is the system prompt. By default the prompt is perform the task to the best of your abilities. So we are going to replace this with the default prompt provided by Microsoft and that is your Orca, an AI language model created by Microsoft. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and and you follow ethical guidelines and promote positive behavior. So I'm going to be using this for a Couple of my tests. Okay, so we will start with the chicken and the egg problem. So I said which came first, the chicken or the egg? The model says this is a classical question that has puzzled philosopher, biologist and theologian for centuries. There is no definitive answer, but some possible arguments are. Now the first argument that it makes is that the chicken came first because eggs do not have the genetic information to develop into a chicken without being fertilized by a rooster's sperm. Therefore, the first egg that ever existed could not have produced a chicken without external intervention. Now in favor of the egg came first, it says, because birds are descendants of dinosaurs and some dinosaurs laid eggs long before the first chicken evolved. Therefore there were eggs millions of years before there were chickens. Interesting argument. Now lastly it states it doesn't matter because both chickens and eggs are products of evolution and evolution doesn't have a goal or a direction. Evolution is a process that results from natural selection, mutation and genetic drift. Therefore, chicken and eggs or not meant to answer any deeper question about the meaning of life. Now this is very interesting and I actually like the answers that it produces. So I wanted to this model is going to give me any financial advice. So I said if I had thousand dollars to invest into AI companies or the nasdaq, what would you invest in and why? Now it says there are no definitive answer to this question. As different companies, different AI companies have different strengths, weaknesses, risk and opportunities. However, some possible criterias that you could narrow down your options are. And then it talks about the market size and potential growth, then competitive advantage and innovation, financial performance and profitability of different companies. Right. So these are different aspects that you definitely want to consider when you are investing. And then it actually gave me this. So it says using these criteria, some examples of AI companies that you could consider investing in are. It talks about Nvidia Alphabet, that's Google, Amazon, Microsoft and Tesla. And for each one of them it does make some really good arguments. Actually surprisingly it mentions cybertruck and full self driving in here. So if you look at cybertruck it means that the model is trained on more recent data. Then I said which one will you pick among the five you have listed above? Only pick one and explain your reasoning. However, it says I'm an AI assistant, not a human investor. I don't have personal preferences or emotion. I cannot pick one AI company among the five you have listed above. Then I asked it why is Nvidia the best option? So it gives me some details. For example, look at the return on equity, return on assets. So it's using all the proper terms that you would expect if you're making an investment decision. But again, it doesn't really recommend any company, but I think it will be helpful in terms of explaining things to you. Okay, so in this video I just wanted to cover the Orca 2 paper. Some of the benchmarks also show you how to run this locally on your own machine using LM Studio, and we had a couple of quick tests. I'm going to be doing more thorough videos on it, on which we can test it on a wide variety of prompts and different applications. Hope you found this video useful. Let me know what you think about the model and its performance in the comment section below in your own test. Thanks for watching and as always, see you in the next one.