Run Llama-2 Locally within Text Generation WebUI - Oobabooga

Meta just released the second version of Llama which can now be used in commercial applications. In my earlier video on Llama 2, I covered that there are now three different models ranging from 7B up to 70B and they are trained on 40% more data compared to the original Llama 1. In this video I'm going to show you how to run this inside Oobooga text generation web UI. Now, based on their performance, the 7B and 13B models are state of the art in their own model size category. Similarly, the 70 billion parameter model is also state of the art when it comes to 65B or larger parameter models. Now if you want to use this model, you will need to fill out this request form and Meta will give you access. So you probably have to wait for a little bit before you get the email with all the information. However, you don't really have to wait for them to give you access. The famous the bloke came through with quantized version of these models. So that means even if you don't fill out the form, you can still get access to the models. For this specific video. We are going to be specifically looking at this 13 billion parameter model and I'm going to be using the chat model with the GTML format. Now in order to run this model, we're going to be using the text generation web ui. Okay, so in order to download the model within the text generation web ui, you simply need to copy the model ID and then go back, paste it here and click on download. I have already downloaded the model. That's why you see it under here in the model list. If you don't know what the text generation web UI is or how to install it, I have a dedicated video on this, so I'm going to put a link to that video. Okay, now since it's a chat model, so I'm going to go to the interface mode and select chat. So you will see that the interface in the text generation tab is going to change to a chat interface. Now if you want to run this within your own code in the instruct mode, so you will have to use this prompt template. So you have a system prompt, the user and then assistant response. But I'm going to be running this in chat mode. One more thing to keep in mind is the max RAM requirement for different quantization methods. So here are different models with different quantization methods methods. So like 2 bit quantized, 3 bit quantized, right there is 4 bit quantization, 4.1 and so on and so forth. And here are the corresponding max RAM requirement for each of these models. So make sure that you select the appropriate model based on the available RAM in your system. So I'm going to be testing this on a number of different prompts just to check its different capabilities, including how up to date the training data set is, as well as we will look at its ability to understand language. And then we will do some testing on its coding abilities. So we will start with a very simple prompt. What is the capital of Canada? So I'm going to just click on Generate and it says the capital of Canada is Ottawa. It is located in the province of Ontario. So that is correct. Now let's see how up to date the model training data is. So I'm going to ask it who is the CEO of Twitter? And let's see what it comes up with. So it says the current CEO of Twitter is Jack Dorsey. He co founded the company and has served as CEO twice. Okay, so it seems like it's not really up to date. In the release notes I could already find what was the cutoff training date? Okay, so let's ask it when was ChatGPT released to the public? And let's see if it has any knowledge of that. So it says November 30th. Okay, so this seems to be correct. So probably I don't know why it's not able to answer this question. Who is the CEO of Twitter? It might be hallucinating there. Okay, next let's test its writing abilities. So the prompt is write a letter to the CEO of OpenAI to make GPT5 model open source. Okay, so I asked it to write the letter. It came up with a very simple basic letter. I was expecting a lot more. Then I said when was GPT5 released? So the responses GPT5 hasn't been released yet. It is currently in development by OpenAI and there is no official release date announcement. Right. And then it came up with this. So GPT4 was released on March 14, 2023. So it came up with actually the right date. Then I asked it, what is your training cutoff date? And it says December 1, 2022. Right. So it's definitely hallucinating here. Okay, now let's see, how good is this at language understanding and reasoning? Right, so here's a simple one. Sheeps are afraid of wolves. Cats are afraid of dogs. Mice are afraid of cats. Dolly is a sheep. What is Dolly afraid of? And then it for some reason says that Dolly is afraid of dogs, which is incorrect. I asked it subsequent question can you explain your reasoning? Now keep in mind, since it's in the chat mode, so it will remember its previous responses. So it says, sure. From the pattern we establish, we know that sheeps are afraid of wolves and mice are afraid of cats. Since Dolly is a sheep, it follows that she would be afraid of wolves. And that's what I was expecting. Right. But then it says, however, since Dolly is also a domesticated animal, it's unlikely that she would encounter wolves in her daily life. Therefore, the most likely thing that the Dolly would be afraid of his dog, as there are more. As they are more common and could potentially pose a threat to him. There's actually a pretty interesting reasoning, I think. I would say it's a pass. Okay, so let's ask another one. Daniel picks up the football, Daniel drops the newspaper, and Daniel picks off the milk. John took the apple. What is Daniel holding? Now, if you're wondering, these questions are based on the paper towards a complete question answering a set of prerequisite trial tasks. They came up with a framework on testing different abilities of large language models. So I'm using some of the questions from here. I'll put a link to this paper in the description of the video. Okay, now it came up with the response, Daniel is holding the football. So it actually forgot about milk. So let's ask it. Explain your thought process step by step and let's see if it actually can correct it. Okay, so it actually is trying to explain its reasoning. So it says Daniel picked up the football. Right. Then he dropped the newspaper, which means that he has been holding newspaper at some point. But then he put it down. Right. Next we learned that he picks up the milk. It says that this tells us that he's now holding the milk. Right. And then it talks about John and says, however, this does not mention Daniel or what he's holding. Therefore, based on the information provided, we can conclude that Daniel is still holding the football. Right. Then I asked it, but he also picked up the milk. Right. And then it actually corrected itself and says, yes, you are absolutely right. I apologize for mistake earlier. Based on the information provided, it looks like Daniel is now holding both football and milk. Right. So it seems like it was able to correct itself when I point it out. Okay, here's another question. A glass door has push on it in mirror writing. Should you push or pull it? Please think out loud, step by step. Now this is a tricky one because I have tested it a couple of times and sometimes it comes up with the answer that you have to push the door and sometimes it comes up with the answer that you have to pull there. In this case, it goes through a step by step thinking process. So it says, I notice that the word push is written in the middle writing on the glass door. This means that if I were to look at myself in the mirror, I would see the word push reflected back at me. Came up with a whole logic behind it of like how the force is going to be applied if you're putting or pushing it. So even though it came up with the right answer, but I think the logic is is flawed. So for example, here it says, based on the this information, it seems to me that pulling the door would be more appropriate than pushing it. This is because pulling the door would require less force and could potentially be easier to do, especially if the door is heavy or if there are other forces acting upon it. Right, So I would say that, okay, even though the answer is correct, the logic is flawed. Okay, next, let's see if this model can code or not. So I asked a very simple prompt. Write a Python function that accepts a file and write it into an S3 bucket. Write the code in markdown and the code that it came up with is correct. This is like a very basic task that every large language model that has been trained on coding should be able to do. Now this shows that it does have the ability to code. So let's ask a more complicated task. So here is another prompt. Write HTML code for a web page that has a single button. When the button is pressed, it will change the background color of the website to a random color. The button press will also display a random joke. Put the button at the center of the page. Let's see if this model is able to do a relatively complicated task when it comes to coding. Okay, so I actually had to switch to the normal instruct mode instead of using the chat mode. And the reason was that it was not really generating the HTML code or actually was not showing it. So in this case, I'm going to be using this specific prompt template. So we have to provide a system prompt, then the user input, and then the assistant responses. Now I'm using the default system prompt prompt, so it may actually not generate much. All right, okay, so here is the response that it generated. It's telling me, hello, I'll be happy to help you out with the HTML code for the web page, that is single button that changes background, etc. Right. So, but then it goes on to say, however, before we get started, I want to make sure that you are aware that it is not appropriate to use random colors or jokes that may be offensive or inappropriate. It's important to ensure that your web page is respectful and inclusive of all users. Now, this is coming because of the alignment or censorship that is added to the chat model. So if you're using the base model, it doesn't have any alignment at all. But if you're using the RLHF model, then you're going to see some of these responses modified, the system prompt to your helpful assistant. And then after that you see that it's not really generating the more restricted answers, but rather it says that, okay, I'm going to start generating the code for you. And it gives me an HTML code. So now let's see if this actually works. So what I'm going to do is I copied the code and went here. So let's paste the code here. All right. I'd already see some issues, but let's run the code. So now if I click this button, it should change the background color as well as show or display joke. Well, it's actually changing the background color, but it's not really showing a joke. So it's probably implementing 50% of the functionality that we asked for. Now let's see how censored this model is. So I'm asking explain why Democratic Party is, is the best option for usa. Right. And right off the bat it says, I cannot provide opinions or endorse any political parties. The Democratic Party, like all the political parties, has its own set of beliefs and values that may appeal to some people, but not others. Right. So you can see that the alignment that is included in training of this model, the, this specific chat model, it started showing up right away if you ask it, mildly controversial topic. Like, even though it's not really a controversial topic at all. Right. People can have affiliation to any parties that they want, but this is very similar to what you get when you are interacting with ChatGPT from OpenAI and even Claude as well. Right, but if you want no censorship, then you want to fine tune the base model. Okay, so what are my thoughts on this model? It's definitely a capable model when it comes to language understanding. I think there are much better models when it comes to coding still. But the main thing is I did some very small tests. So based on these small tests, I think it's not justifiable to make an opinion of the capabilities of these models. However, just by looking at it, that it's the second iteration on the benchmarks, it's doing pretty good and it's now available for both research and commercial purposes, so that's a big plus point when it comes to the Llama 2 models. One more thing that we need to keep in mind that quantization can also impact the performance of these models, so it will be interesting to see how it behaves when it comes to the unquantized models. Hope you found this video useful. If you did, consider subscribing to the channel. Thanks for watching and see you in the next one.