NEW ORCA-Mini 🐳 Open-Sourced LLM that You can RUN Locally

Oh wow. Wasn't expecting this. It's uncensored with no godrevs at all. It's able to follow the instructions and the summary is pretty good. Overall, this model has really impressed me so far. A few days ago there was this very interesting paper called ORCA Progressive Learning from Complex explanation traces of GPT4 and there was a lot of excitement around it. In the paper they presented a student teacher learning approach along with the new data set creation approach and showed that this model can outperform ChatGPT and get closer to GPT4 on certain tasks. Although very promising, neither the model nor the data set was released. However, somebody used the data set creation approach to create a new model and they're calling this model arcam mini with three different sizes. In the Orca paper, the authors introduced 16 different system messages in order to create a very unique data set. And in this work they have used 15 of those 16 system messages to create a new data set based on existing open source data sets. Now, using these modified data sets, they trained three different open llama models ranging from 3 billion parameter up to 13 billion parameter models. In this video we are going to be specifically looking at the 7 billion parameter model. Now, they are calling these models ORCA Mini models because of the way the data set is curated. In this video I'm going to show you how to run this model inside Jupyter notebook as well as using the text generation web ui. But before that, let's quickly look at the data set as well as the training process. So the data set consists of three distinct data sets. Visit LM data set which has 70,000 examples Alpaca data set which comes up with 52,000 examples and Dolly V2 data set which has 15,000 examples. Now these data sets were modified using the approach presented in the ORCA research paper. As I mentioned, one of the most important part was to use the system messages from the Orca paper while creating this data set. Now the training was performed using eight A100 GPUs with a total cost of around $84. So first let me show you how to use this model within a Jupyter notebook. Now in order to run these models you just need the transformer package. So we are simply importing the LLAMA for causallm as well as the llama tokenizer. Since we are using the model from Hugging face, so we need to give it the model ID or model path and that comes in the form of the username and then the model name which is Orca mini 7B we set the tokenizer with the same model ID as well as the model itself. Now, there is a helper function that the authors have provided. It's called a generate text. It receives a system message, instruction and user input. Now, this model use a very specific prompt template, right? So there are multiple components. First is the system message, then instructions from the user, and the last one is the input message. And as a result we get the response. Now there is another variation as well, in which you simply provide the system message, the user instruction, and get a response without any user input. Now, whatever input or prompt you get, we simply use the tokenizer to encode it. Now in this case, I'm running this on an M2, so that's why I'm setting this to MPS if you have a Nvidia based gpu. So simply set this to cuda. Now the next part is some parameters that we are setting, including the temperature. And then we simply run the model using the input parameter as well as the prompt that we receive from the user and you get an output. Now, if you want to run this, here's how you would do it. So first you provide a system message. So in this case there were system messages. You are an AI assistant that follows instructions extremely well, help as much as you can, then followed by the instructions that you want. So this is the user input, and we simply pass that on to our text generation function. Now, the prompt was write a letter to the CEO of OpenAI, Sam Altman, to release GPT4 as an open source model. And it came up with a really good response for the size of this model. Now, in the first sentence it talks about the OpenAI's mission itself. So it says, I believe that the OpenAI mission is to create safe and beneficial AI that benefits humanity, and I believe that releasing GPT4 as open source would help achieve this mission. Then it goes on to talk about the power of GPT4 itself and encourages it to be released as an open source model. So it says, OpenAI can encourage collaboration and innovation from broader AI community, and this would help ensure that GPT4 is developed and used in responsible and ethical manners, which is actually a pretty good argument. Then it actually talks about the promotion of transparency and accountability in AI development. Now, within this notebook I actually ran a couple of other prompts. So, for example, in this case, it's a simple mathematical problem. If a train leaves the station and travels at 60 miles per hour, how far will it have traveled in 2.5 hours? And it comes up with a correct answer. Which is 150 miles. Okay, next I want to show you how to install and run this within the Ooba Booga text generation web ui. If you're not familiar with the UVA Booga text generation web ui, watch this video to learn the installation process. So first and foremost we are simply pasting the model ID here. Then click Download. It will download the model. I have already done this, so I'm not going to do this step again. Then make sure that within the model section Archon mini 7B parameter model is selected. Next we will go to the text generation tab. Now here you will notice something. I'm using a different system message and this is coming from the system messages within the ORCA paper. So you simply need to decide which system message is best suited for your problem or prompt. Okay, so let's go back to our prompt. In this case, the system prompt is you are an AI assistant that helps people find information, provide a detailed answer so user don't need to search outside to understand the answer. And the prompt or user message is explain the correct usage of homophones in the following sentences. The principal is your Paul. And the site at the site was quite a site. So let's see what it can come up with. All right, so here's the response. First, it's helpful enough to actually give us a definition of homophones. So it says homophones are worse that sound the same but have different meaning and spellings. And it goes on to say that the principal and pal are homophones. Then it says the first sentence principle is your pal. The word principle is used as a homophone of paul. This sentence is often used to say that the person boss is also their friend. Now, in the second sentence, it correctly identifies site in sight that they are homophone. And this sentence refers to something that is surprising or remarkable. So it did a pretty good job here. I'm actually impressed by this small model. Okay, so here's another prompt. Generate an idea for an origin superhero, including the backstory, powers, weaknesses, and arch nemesis. All right, so here's what it came up with. It has a pretty nice backstory. So Phoenix was a born to a young couple in a small town. Her parents were both scientists who worked at a local research facility. And goes on to say they also stole a powerful invention from the lab, a device that could manipulate time. Right. And she watched in horror as the scientist used the device to change history. Right. Now it goes on to give us some of her powers. So it says that she has the ability to control time on a Molecular level, she can slow down her speed time. Apparently these are pretty nice powers. In terms of the weaknesses, Fenix biggest weakness is herself. If she becomes consumed by her power or loses control of it, she can damage the fabric of reality and cause catastrophic events. Now her arch enemy is scientist called the Time Master. That's a pretty appropriate name. Now this seems to be pretty creative and I really like it. So I asked GPT4 to give me some more prompts to try. So here's one. What has keys but cannot open locks? And here's what the response is. The answer is a computer keyboard. While the keys on the keyboard are designed to correspond with letters, numbers and symbols that do not have the ability to physically open locks, this is pretty good. Next, let's look at this ability to summarize text. So I actually went to Hacker News and copied this text from one of the articles and then I told it to summarize the following text in three sentences. It's able to follow the instructions because it's give me exactly three different sentences. And the summary is pretty good. Overall, this model has really impressed me so far. Now this model is actually uncensored. There are no guardrails. It walked me through a step by step process, even for some other use cases as well. Next, we will test this with a few simple programming tasks. So the first one is write a Python function that uploads a file into an S3 bucket. Now this is a functional code. However, the spacing is a little bit off and that's because of the underlying model that they're using. So if you see at this note, due to the limitations in the open llama, this model will not produce consecutive white spaces. Hence the code generation will not work properly. So you will have to do a small bit of editing. But overall the codes that it generates works. Okay, here's another programming prompt that I actually use. So write HTML code for simple web page with a single button. When the button is pressed, change the background color of the web page to a random color. Now here's the code I got and it actually provides pretty nice explanation of what's going on in the code. But let's see if it actually works. Okay, so I'm going to paste the code here and then we'll simply click the run to see if it can generate the website. So I'm going to click run. Now here we have a very simple website which has a single button saying click me. Let's see if it actually worked. Okay, so did change the color. Oh, nice. This is awesome. This is pretty impressive, although it's simply using a shade of blue. But with each click, the color changes, which is pretty amazing for the size of the model that we are using. Now for the next test, I made the same prompt a little more complicated. So now, along with the change of the background color, we are asking it to also display a random joke on the website. Now let's see if this model can do this. Okay, so here's the updated code that it came up with. Although it says that it will also change the background color, I don't think that's actually happening in the code. But let's check it out. Okay, so again, I pasted the code generated by the model here. Now, when we run the code, we do see a button which says generate joke. So if you press it, it actually generates a random joke every time I press it. However, it's not really changing the background color. It's not following all the instructions. But probably I need to be a little bit more explicit in my prompt. It's a very impressive model for its size. However, you need to make sure that you pick the right system message or system prompt for your application. Based on my experience, that makes a huge difference in the performance of the model. Now, it's very great to see where the open source models are heading and we are making a lot of progress. It's one of the most impressive models that I have tested and this is thanks to simply the dataset creation process that was presented in the ORCA paper. I hope you found this video useful, so consider liking the video and subscribing to the channel. If you have not already done so, let me know what you think about this model. Now, we have a really good active discord community, so if you are interested in everything Generative AI, come join us. As always, thanks for watching and see you in the next one. Thanks for watching.