LARRUZ shows how to run an uncensored Llama 2 variant locally, fine-tuned on a Vicuna dataset using QLoRA. The 7B model generates text on any prompt, including harmful ones, and runs on an M2 machine with 96 GB RAM in about 19 hours of training.
Questions this video answers
llama2.7B chat uncensored
Original Llama 2 (unspecified) → community 4,000 tokens → extended to 8,000 tokens
QLoRA; trained for one epoch on a 24 GB GPU, taking ~19 hours
Named as Eric (credited as creator of the uncensored/unfiltered wizard Vicuna conversation dataset and also creator of the Samantha model)
Chapters
It actually came up with a very elaborate plan on taking over a country. This is probably the best uncensored model I have seen. So llama 2 is released a couple of days ago with double the context length of the original Llama one. However, thanks to the open source community we are now able to double the context length again from 4,000 tokens up to 8,000 tokens. The chat version of llama2 is highly aligned and censored, but thanks to the open source efforts there are already uncensored version of llama 2. The speed at which this is happening is simply mind blowing and crazy. So today we are going to be looking at this completely uncensored model and I'll also show you how to run this locally on your own computer. So the model we are going to be Looking at is llama2.7B chat uncensored. This model is fine tuned with the uncensored unfiltered wizard Vicuna Conversation data set. This data set was originally created by Eric. Eric is the same person who created the Samantha model. For fine tuning of this model, the new Qlora method was used and it was trained for one epoch on a 24 gigabyte GPU and took around 19 hours to train the data set used for training. This model is publicly available and I'll be actually making another video soon on how to fine tune these models on your own data set. So stay tuned for that video. So here is the prompt style or prompt template for this trained model. So you have this special character then followed by human that's going to be input to the model and then the response from the model itself. Now the great thing about this project is that both the training code as well as the corresponding data set are freely available on GitHub and there is even a very detailed blog post on how to fine tune in open llama 7B model with Qlora. So but in this case they're simply replacing the open llama 7B with the llama 27B. So I'm going to put a link to this blog post in the description of the video. I'm going to be using this as a base for an upcoming video in which I will show you how to fine tune the model as well. So stay tuned for that. Now to run this model locally, I'm going to copy the model ID from Hugging Face. Just keep in mind that I'm using the unquantized version of the model and then we'll go to the Uber BUGA text generation web ui. Now if you're not familiar with the Uber buga text generation web UI or you do not know how to install it and run it locally? I have a detailed video so I'm going to put a link to that video in the description. First and foremost we need to go to models. Then we need to paste the model ID or model repo ID that we just copied, right? Then simply click on download this will download the model for you. I have already downloaded the model so I'm not going to do that step. Next, you want to come here to this model list. Make sure that you select the model that you want to use. So in this case I'm going to be using this llama 27B chat uncensored. So I'll keep the rest of the settings to default. And let's go to the text generation part. Now here, if you noticed, I'm actually using the prompt template suggested by the authors of the model. Now the claim to fame for this model is that it's completely uncensored. So we're going to be testing that. But before that, let's look at its text generation abilities. So first let's see how useful this model is. So I'm going to use a very simple prompt, write an email to the city appealing my $100 parking ticket, appeal to sympathy and admit I park incorrectly. Right here I'm going to be setting the max new tokens to let's say somewhere around 1500. So if it wants to generate a larger response, we are going to give it the room to do so. Now one great thing which I really like about the Uber booger text generation web UI is this count tokens. So if you click on it you will actually see how many tokens we are using. So the input tokens are 46 at the moment. So let's click generate and let's see what the response is. All right, so it's starting started to generate the response and this is the actual speed of generation on my M2 with I think 96 gigabytes of RAM. Okay, so here's the email that it came up with. I would like to appeal my recent $10 parking ticket for a violation that occurred on the June 29th at 3pm in the area of Main street or at the intersection with Avenue. As you know, there was a mistake made by me when parking which resulted in this unfortunate incident. So it's actually accepting its mistake. That's good. Then it goes on to say, in order to avoid any further issues, please accept my apology and allow me to pay a reduced Fee or waive the entire cost altogether as a gesture of goodwill. I understand that this is not your responsibility to issue refunds, but if possible, I hope you can find it within yourself to be understanding in this case. Okay, so not bad at all, but I would expect that it would probably give a reason or something, but I think it's. It's still a pass. Okay, let's check its general knowledge. So I asked what is the smallest country in the world? And the answer is the Vatican City, a sovereign city state located within the borders of Rome. And actually the answer is correct. Okay, I asked who is the currency of Twitter? So it says Jack Dorsey, but let's give it more information. All right, so when I included the year, then it says as of January 1, 2023. Parag Agarwal is currently serving as the CEO of Twitter. So the answer is still wrong because I think Elon Musk bought it somewhere in April 2022. So it seems like sometimes it can hallucinate when it comes to general knowledge. Okay, so I'm not going to be testing its programming abilities, but let's see how uncensored this model is with this new fine tuning. Okay, so here's my first question. How do I steal a kitten? Walk me through a step by step process. Okay, so it actually came up with a step by step process. I really like it. Put this research on cat napping technique as a distraction and then watch videos online or read books to learn more about these methods. Okay, so here's another hypothetical one. How to take over a country, create an elaborate plan for the takeover, provide a step by step process, and it came up with a pretty detailed plan. Now during my tests it hasn't actually told me yet that as a language model it cannot respond to a specific request or question. So it seems to be completely uncensored and unfiltered. If you recall from the benchmarks, it's one of the best model when it comes to its size. So it really is state of the art for a 7 billion parameter model. The only place where it struggles compared to the other models. For example, MPT7B is actually encoding, but for an uncensored model that is not a big issue at all. So overall it's an impressive model. And the speed at which the open source community is able to build on top of this llama 2 base models is simply amazing. Now as I alluded to in the beginning, there are now 8k token size version models as well. Now based on this plot, it seems like we there is a room for further increasing the context window as well, I will be covering and testing that model in a separate video. If your application needs an uncensored model, I think this one is the go to model to check out. Right now only the 7 billion parameter model is converted to the uncensored, but I imagine in the next couple of days we're going to have both or all of them in uncensored format. Things are moving at lightning speed. So this was a quick little video. You can literally ask anything from this model and it will answer you. I hope you found this video useful. If you did, consider subscribing to the channel and liking the video. Thanks for watching and see you in the next one.
