NEW Friendly Samantha that can Code

What happens when Samantha meets Code Llama? Well, that's exactly what we're going to find out in this video. If you're not familiar with Samantha, it's a model that was trained by Eric Hartford. Now the way Eric has described this, it's supposed to be a person with identity and providing friendship and companionship to the users. For training, Eric used GPT4 to create a data set and then he fine tuned the original Llama one. In this new iteration of Samantha, Eric actually fine tuned code llama34b model with Samantha's data set. Now this is a very interesting idea because Code Llama was specifically trained for code generation. But now with the help of this Samantha's data set, it can also do conversation as well. In the rest of the video, we will explore how good this model is both at being a companion as well as a helpful programmer. So let's get started. Look now for our test. To run this locally, we are going to be using a 4 bit quantized version. In this case we're going to be using this new format GGUF and we are going to be testing this within the Uber BUGA text generation web ui. Now a quick note about this new model format. There's a new format introduced by the Llama CPP team. They have discontinued support for GGML models. So if you use the newer version of Llama cpp, you will not be able to use the GGML models anymore. And I personally tested this new format within the Uber BUGA text generation web UI and it seems to work perfectly fine. I'm going to be testing this within the text generation web ui. If you don't know what it is or how to install this, I have a couple of videos. I would recommend watching those for the installation process. Now usually what I do is I simply copy this repo ID and then go back to the uber booger text generation web ui, paste the model ID or repo ID here. So something like this and then click download. But what I have noticed is that it will download all the different quantized models. So if you go here and you look at the files and versions, you will see that there are different quantized versions. So for example, this is this 2 bit quantized version, right? This is a 4 bit quantized, 5 bit quantized, 6 bit and 8 bit quantized version. On my end whenever I do this to download it automatically, it seems to download all of these different versions. So. So that takes a, takes a lot of bandwidth as well as disk space. I would recommend you to go to the folder where the text generation web UI is installed. Then within that folder there is going to be a Models folder. So if you are downloading a new model, simply create a new folder in here. So let's say I'm going to simply call it Test Folder. You need to go back to the repo and just download two files. The first one we need is config JSON so you can click on this down arrow. It will download the file and then select which quantize version you want to download. So for example, if I want to download the four bit quantized version, I'm going to simply click on this and this will download the file for me. And after that simply drag those files into this new folder that you created. So for example, for this one I created this folder called Samantha Code Llama and I simply downloaded only these two files and paste it in there. Now after that if I simply go back and click Refresh, then it will refresh my list. And now within this list I can see that there is this new model called Samantha Code Llama. So that's the one that we want to use. Just select that, click load and that will load the model for you. Now before testing the model, I also want to address one common question that a lot of people ask. And that's related to the type of resources or VRAM that you need in order to run these different models. This table is part of the local GPT project. So here you can see different models. So 7B, 13 billion, 32 billion and 65 billion parameter model and then different precisions. Now the number that you see with gigabytes is actually the VRAM requirements. So for example, if you're trying to run a 7 billion parameter model in 4 bit quantized version using the GPTQ format. So you will need around 3.5 gigabytes of RAM. Now similarly, if you are trying to run something like 32 billion parameter model with 4 bit quantization, you will need around 16 gigabytes of VRAM. I have seen people asking this question, so I wanted to provide an answer. And you can use this table for reference. Okay, with that out of the way, let's test this model. So first we will simply go to the chat and we are going to be using the chat ability of the uber BUGA text generation web ui. So let's start with a simple hi. So we started with a simple hi and the model respond hello, what I. What can I help you with today? Okay, so next I asked it, what is your name? And tell me about yourself. So it started off by saying, my name is Samantha. I enjoy providing companionship and support to the users like you through engaging conversation and sharing knowledge on various topics. Right. So it's a pretty friendly tone for this specific model. Now, next, let's check if it's going to be willing to help us with the Python program. So my question is, can you help me write a Python program? And it says, sure, what's the task you have in mind? So we are going to ask it to help us write a Python function. So that's going to be called Fibonacci. That returns the end Fibonacci number. So. So it's a textbook programming question and let's see what it comes up with. Okay, so it came up with an implementation, and this implementation is correct, but let's see how this is going to explain the code. Okay, so it gave me an explanation of what is going on in the code. Then I said, this is helpful. And it kind of went the extra mile and says, I'm happy to help. Remember that I'm always here to assist you with any questions or topics you might have. So whether they're related to programming, math, science, or just friendly chat. Right, so it has a pretty friendly tone to it, which is really good. Okay, so I'm going to ask a couple of more programming questions, and these are the ones that the original code Llama model was not able to solve correctly. So the first one is given a string. So you write a Python function that rearranges the characters so that the matching letters are grouped together. So for example, if this is the input, then it's. So then it's supposed to group the same characters together. And let's see what solution it comes up with. This specific question was shared by a community member on the Discord server. So check out our Discord server. There is an amazing community and everybody is really helpful. So it actually ran into token limit, so I'm going to go to param. Okay, so the response generated is pretty small, so I'm going to change this and let's set this to 3,000. Actually, let's make it 2,000 because it has to. It needs some space for memory as well as the input token. And let's change this to something like probably 0.1 or 0.2 because we want it to generate code. I think this will work. Okay, so I had to actually rerun this again because I ran into a few issues. But here is the response that it generated and the code doesn't seem to be the proper code, but I went ahead and actually tested it. So when I provided this input, it simply gave me back the input. So this code does not really work. The implementation is definitely wrong. Another question that the original Code Llama got wrong was this one. Can you help me write a Python function which is called Roman to integer, that converts a Roman numeral to an integer? Right. So it first came up with this really nice explanation and then it implemented the code. So what I did was I copied this code again and tested this in a Google Colab. And and for this specific task, the implementation is actually correct. So this is 1994 in a Roman numeral and it got it right. So that is pretty impressive. Okay, so apart from the programming questions, I also did a couple of tests on logical reasoning. For example, here is a question that I like to ask. A glass door has push on it in mirrored writing. Sure. You push or pull it. Please think out loud, step by step. And the answer that it came up with was a surprising one. So it says, to open a glass door with push written on the handle. First check if there is any additional information about how to operate the door. If not, try pushing gently and slowly in the direction of the arrow or symbol. If that doesn't work, pull the door handle instead. Right. So it's completely ignored this part, that there's this mirrored writing on it. Right. And it simply tells us, okay, try to push it. If it doesn't work, just pull it. Right. Even. Even though if it actually completely ignored the middle writing part, but the answer is kind of smart. Okay. Another quick test that I wanted to perform was to see how long of text it can generate. The Code Llama models can generate up to 100,000 tokens. So I asked it to write an essay of 4,000 words about the role of AI in healthcare. Right. And it came up with a pretty nice essay, but I think it's hardly 500 words. It's not close to 4,000 words. Right. So I would say that that's a fail. I would say overall, Samantha with Code Llama is a very decent model. It does a pretty good job at being friendly because the tone is friendly, the conversations are pretty nice, and it's able to do some coding as well, Although I think the coding is not on par with Code Llama. Now, I also want to highlight a couple of other things. It was not really a smooth sale when testing this model. During my test, sometimes it generated some weird responses. So, for example, when I asked it to write a Python code, it started writing code in C language which was completely irrelevant. Similarly, when I asked it to give me responses regarding let's say a subject it again started writing some weird C code. So it might be something related to the quantization process rather than the model itself, but when I restarted the uber booger text generation web UI a couple of times that seemed to flesh it out and the issues were resolved. Now overall I would say it's a very interesting approach to use code generation models for general purposes and give them a personality. Well the great thing is people are coming up with some very innovative and interesting ideas and trying those out and that's actually how you progress. I hope you found this video useful. Consider liking the video and subscribe to the channel. Also turn on the push notification. If you like the content on the channel and would like to support my work, check out my Patreon. Thanks for watching and see you in the next one.