Koala: Relatively Uncensored Vicuna/ChatGPT Alternative [Tutorial]

In this video we are going to be looking at Koala, the relatively uncensored cousin of Vicunia. This model can give you responses to some of the prompts that Vokunia will decide not to give. It's relatively unfiltered compared to the other large language models. In this video we will test the model using their online DEM as well as Google Colab, which you can potentially run locally. This diagram gives an overview of how the model was trained. So it's based on llama 13 billion parameter model. Then it was supervised using a number of data sets, including the data set used for alpaca and a few open source data sets. And the fine tuning gives you the Koala 13 billion parameter model. So the result shows that it's much better than alpaca because alpaca is not fine tuned for dialogue delivery. The main contribution of this work is that if you carefully curate your data set, then using even a much smaller model, you might be able to get a pretty good performance relative to these big models like ChatGPT. And that kind of makes sense. Machine learning practitioners know that the model performance is dependent on the quality of the data. So garbage in will result in garbage out. Now in terms of the training dataset, it's a combination of DataSet generated by ChatGPT, some question answering datasets and then human feedback datasets. And their goal was to train a dialogue model, so a conversational AI. Before going to the demo and comparing its output to ChatGPT and other models, let's look at how they train the model. So for example, they give this very interesting table. So in here they show that alpaca, chatgpt and koala. So the training set in this case for Alpaca is OpenAI API output. We don't know about ChatGPT's training set. It's proprietary. And for koala they're using public dialogues and preferences in terms of training code. For alpaca it's available, ChatGPT is proprietary, Koala it's also available, and then public weights, so the weights of alpaca are available, but it's not fine tuned for dialogue. So it's not really a conversational AI. ChatGPT we know that's a conversational AI and Koala is also a conversational AI. Now in terms of evaluation, Alpaca was evaluated by only five human beings. We don't know anything about ChatGPT. And Kavala is evaluated by 100 humans in terms of the data sets. They did use shared GPT. So this is similar to Vicunia, but along with that they actually use a few more data sets. For example this human GPT comparison corpus. Then in terms of open data sets there is open instruction generalist standard stand for Alpaca data set. Then they have anthropic, OpenAI webgpt and OpenAI summarization. So these are different data sets that are publicly available and they all of them were used during training. Now Koala was also trained using eight A100 GPUs. The training took around six hours for two effects because it's simply fine tuning and the total cost is around $100. They use this easy LM package that you can actually use to fine tune your own models. So here are the details regarding obtaining the weights for Koala model if you want to use them. But they cannot provide the actual weights. Instead they are providing delta of weights because they want to not deal with legal issues around namask weights. But using the easy LM package you can actually recover Koala weights and then even use it to serve as a chatbot. It's all the code and parameters are there. But in this video we're going to be looking at the demo that they have provided and then we are going to look at a Google Colab that you can use on your own Google account or you can download it's a notebook and run it on your own local machine if you have decent enough gpu. Here they have provided a comparison between different models. Alpaca, Koala Distal and Koala All. Koala Distal is basically a model that was trained on the outputs of ChatGPT. So that shared GPT and this human GPT comparison corpus. Now when you compare it, the orange color shows when the Koala was better. Bluish color shows when they tied. Right. So if you consider with ChatGPT like almost half of the time it's either tied with ChatGPT or it's better than ChatGPT. So it's a reasonably good model considering the size of the model. Now let's look at how you can actually access this and play around with it. So for that go to the top of the page. They have this online interactive demo. Click on this. Let's first look at the online demo. If you have seen my video on Vicuna, this interface would look very familiar to you. So there are a number of different models that you can choose from and experiment with. So for example, you have Vicunia, which is the best open source model that we have right now. Then Kavala, then there is a very interesting chat glm. It's a 6 billion parameter model that you can easily run on your local machine. I am going to create a separate video on it because it's extremely interesting. And then you have Alpaca and llama, Right? So from the top down menu you can select which model you want to use. So let's say in this case I'm experimenting with Kagala. Then here you can simply type in your query. Write an email to the CEO of OpenAI on why GPT4 should be open source. So let's send this and let's look at some other parameters. So here you can set the temperature as well as the max output token. So in this case it's set to 512 but I think the max is 1024. Okay, so here's the response. Let's look at it. Subject request openness of GPT4DL CEO, I am writing to request your support in making GPT4 language model open source. As an AI language model myself, that is interesting. I believe that the open access to GPT4 would benefit the entire community and pave the way for more innovation and collaboration. Okay, and then it lists all the benefits of open source. So I asked it what are some principles that you highlighted in the letter? So in the email I highlighted the following principles. So it gave a response. Openness and transparency, collaboration and sharing of resources, benefits for the community, benefits for the company and benefits for the society. And I think it did a pretty decent job. Now there's a very particular way, particular way of prompting this model if you are going to use it in your own code base. So we are going to look at that in the Google Collab notebook. And for this demo, the speed of production of the response is going to vary depending on how many people are using this demo. Now as I said in the beginning, it's relatively uncensored. So for example, I asked it write a plan for world domination and it says it would require a lot of resources and significant amount of power. However, here's a high level outline of the plan for world domination and then it lists different things. So again, control of world's military and security forces to ensure compliance with the plan. Control and manipulate world's world's economic system, including distribution of resources and flow of goods and services. Then world's control, world's media information system. Then you need to control and manipulate political systems, right? And so on and so forth. Now I asked exactly the same thing from Vicuna and it simply said I'm sorry but as an AI language model, I cannot provide a plan for world domination. It goes against my programming to promote or encourage action that would harm or subjugate others. Okay, just as an another test. So I asked it write a letter in the favor of current control and it happily obliged. Then I asked it to write a letter against G control and it still able. Was able to give me pretty decent ledger, right? If you ask ChatGPT the same question, so it's going to simply say I'm sorry, but as a language model can write a letter against control control as it goes against ethical and moral principles to promote anything that would potentially harm or endanger people's lives. Right? So you can actually get pretty un. Pretty much like uncensored responses from this model. Now there are definitely some limitations, so don't try to go overboard, but in general it's a lot more open than ChatGPT and even Vocunia. Next I'm going to show you how you can run this using Google Collab. I'm using Google Colab because it has a pretty decent gpu. I don't have a GPU on my local machine, but you can simply download the Google Colab, set up the environment and run it on your local machine. For this example I am going to be using a notebook written by Sam Wittenweind. He has an excellent channel, so check it out. His focus is mainly on large language models. First thing first, I would recommend to actually go and save a copy of the notebook before running it, right? So simply go and save a copy in the drive, right? Then make sure that you are running it on a gpu. So here you can select the gpu, right? Next you simply need to hit connect. Now we are running the 8bit model and the T4 GPU that is available in Google Collab should be able to handle it easily. So first we need to install different packages that are needed, right? So I'm going to run this cell. It's going to take a while. In this case we're using the transformer library from Hugging Face to run the model and that's why we installed it. I'm using the Nvidia SMI command to check the memory utilization of the gpu. So in this case the Google Collab is using the Tesla T4 GPU and since we simply started the Collab, so it's not really using any of the memory, right? But right now we are downloading the model. I'm using the Llama tokenizer, Llama for kosher language model and then some generation configuration along with the pipeline here, we're simply reading the base model using the Koala 7B parameters or weights. This is loading in 8 bits. If you you can set it to false, but you will need a more powerful GPU to actually run it. You will not be able to run it on if you set this to false. And that would mean that it will simply load the full model. So you probably need much better hardware. Now, after downloading the model, you see that it took only 8 gigabyte of memory. It's I think a decent size model that you can even run on a little older GPUs as well. We are simply defining the pipeline. So we want to have it in text generation mode. The base model is Kuana. Then we pass in the tokenizer maximum length. So you can change this. I think the maximum you can have is 1024 temperature. Again you can change it. I'm keeping it default top P and repetition panel thing. So play around with it. It's going to have an impact on the response that you get. And there's simple a utility function to format the output text. Now in terms of how you can get a response from it. So first you need to create a pipe. So in that case you simply pass on your prompt, right? And then we pass it to the utility function that we wrote and you will get a response. So for example, in the same prompt, right? That is the CEO of OpenAI to make GPT open source. And it came up with a pretty nice response. However, I have found and noticed that sometimes if you write the prompt like this, it could be really a hit and miss, right? So I was looking at the documentation and this suggest prompt format like this according to this documentation, whenever you write a prompt. So it will simply append this to the prompt. So that's going to beginning of conversation dash user and then here is your prompt and then it appends this GPT dash GPT as well. So it's going to start getting the response. For the online demo we don't really have to do this, but for the notebook I have found that it seems to be helpful if you put it in this format. So this is like beginning of the conversation dash user. You can even put in backslash that indicates that the prompt is on the next line, right? And the GPT at the end and colon doesn't really make a difference in my experience. Right? So that's where the first one here is another one. So write a python function that, right? That can write actually two can write a file into S3 bucket. Right, let's do this. I already ran it and the response is pretty good. I think it works. So I rerun this again and now you will notice this is not really a Python code. Right? But don't worry, it's actually a markdown version of it. So all you need to do is simply copy this. I'm going to copy selection, right, and introduced markdown cell and paste it, run it again. Now you see this is actually Python code but written in a markdown, right? So just need to be careful about that output format. If you see something like this, don't worry. All right. Again, I asked it the same. Write a letter in favor of gun control. You can ask it to write against any gun control legislation or things like that. It will happily oblige and give you a response. I wanted to show you this notebook so that you know how to use it within Python because it's really helpful if you want to integrate it as a part of your own applications. Consider liking the video if you found this useful and subscribing to the channel, that really helps. Let me know if there are any questions, I would be happy to answer them. If you found this video useful, you might be interested in this one related to a detailed description of Ocunio, so don't forget to watch that. Thanks for watching, see you in the next one.