CODE-LLAMA For Talking to Code Base and Documentation

In this video we will explore how to interact with your documentation and code base using Code Llama, which is an open source large language model. In the last video, I showed you how to do the same thing using GPT4 model from OpenAI. In this video we will be trying to achieve three different things. First, I'll show you how to use Code Llama as a replacement for GPT4 when it comes to interacting with your own code base or GitHub repositories. The second goal is to show you how to set up the proper prompt template for codenama and show you the impact of a wrong prompt template on the performance of these LLMs. And the third objective is to show you what type of performance you should be expecting from Llama CPP if you are able to enable GPU support. Let me first give you a very quick overview of what we are trying to achieve here. So in this case we're going to be using this LangChain repositories and the goal is to interact with the Python files. So simply ask questions from the code base and the model should be able to answer those questions. This video is partly based on LangChain documentation where they have provided an example on how to interact with your code base. But we are going to be adding a few extra things and also looking at why the default prompt that they have provided does not work. Okay, so for the most part I'm going to be using the same notebook with some additions. So first we are simply importing different packages that will help us load the code base. Next we are Simply loading our OpenAI API key. Then here is the part of the repo. So I already downloaded the LangChain repo on my local machine and I'm providing the repo path, then creating a generic loader. With that generic loader, I'm specifically looking for all the files that ends with py and then we're using a language parser, and that language parser is going to use the Python language syntax. That's basically the part that we will need in order to load the documents. Now next we need to actually split our Python files. So for that we're using Recursive text splitter and we're specifically asking it to use the Python syntax. Now one thing I would like to reiterate is the way these documents are created or split is that for each given Python file it will first look at standalone functions. Those functions are going to be treated as a single document. Then it looks at standalone or top level classes. Those are treated as separate documents. And the third step is that any code which doesn't belong to a function or a class, that's going to be treated as a separate document, right? So with that in mind, it will split those documents into different chunks. And we are defining the chunk size to be 2000 and there is an overlap. Now after that we need to create our vector store. So for this quick example, we are using the OpenAI's embeddings, but you can replace those with any open source embeddings. We pass on the chunks that we just created, right? And then that will create a vector database for these embeddings. Now, in terms of retriever, so instead of similarity, we are using the max marginal relevance. So the idea is that the chunks that retrieves the two of them should not belong to the same file. And the number of chunks that we are retrieving is set to 8. So up till this point we covered this in the last video. If you're interested in learning more about it, I would recommend you to watch that video. Now what we did in the last video was that we used GPT4, but in this case we're going to be using Code llama. So here we're setting up everything in order to use the Code Llama model. So first we are importing all the different required packages. For this specific example, I'm not going to be using any type of memory and we're simply using the Llama cpp. So if you are not familiar with LLAMA cpp, it's a package or library that is used to load quantized models. It will enable you to run quantized models both on GPU as well as on cpu. They had support for a file format called ggml, but in the recent update they dropped that support. So now you need to specifically look for GGUF models. So okay, let me show you how to actually get this model. Okay, in order to get a model we're going to go to Hugging face and we will look for our trusted the bloke. So he has a model which is called Codelama 13B. Instruct GGUF. So this is the new format that is supported by Llama cpp. So then you need to go here to files and versions and here you have models in different quantization level. So for example this is Q2, so that's 2 bit quantized, 3 bit quantized. Then there are 4 bit, there are even 5 and 6 and 8 bits quantized. Right. So I'm going to be just downloading this Model. Now, in this case, if you see there's this K and M, these are different sampling methods. So simply select one of the models that you want to download and then click on this down arrow. So this will download the model for you. Okay, so now we go back to the Visual Code Studio and here, if you notice that I created a folder called Model and within that folder I have pasted the model that I just downloaded, right? And in terms of the path you see here, I have set the path to the path of that model. So this is one way where you can simply download models locally and then set this path to the path of the model that you want to use. Next, for the context, we are using a context of 5000 tokens. Max token I set to 5000 as well. Now I'm running this on M2 max. So in order to run Nama CPP with GPU support on M1 or M2, you simply need to set this n GPU layers to one. If you're running this on Ubuntu or Windows, then you need to look at the number of layers you want to offload to GPU and you can set those in here. Then I am setting the batch size. Also I'm enabling streaming, so this will actually stream the output word by word. So in a subsequent video I will show you how to set up LLAMA CPP with GPU support properly on an M1 and M2, as well as on a Windows and Linux machine. So watch out for that video. That's going to be very helpful. Now, if you are able to properly set up LLAMA CPP with GPU support, then you should see that this flag is going to be set to one. So that's the last flag. It's supposed to be set to one if it's using gpu. Okay, so let's see if we properly set this up. So we will ask with a question that is not relevant to our repo or documentation that we have. The question is in Bash, how do I list all text files in the current directory that have been modified in the last month? Right, so this is our question and when we run this, so you start start seeing a response. Now, apart from the issues with the answer, you will probably notice a couple of things. It actually included these two different questions. So it seems to be asking a question from itself and then trying to give us an answer, but that's not actually we were asking it for. So this comes down to the prompt template that you need to use. And we're going to talk about that in a bit. Now let's look at an example of when we're doing information retrieval from our code base. So this is the default prompt template that LangChain documentation provides. So if you look at here, it says use the following pieces of context to answer the question at the end. And then it says that if you don't know the answer, just say that you don't know. Don't try to make up any answer. And it's asking the model to use three sentences at maximum and keep the answer as concise as possible. Right. Then you are providing the context, the related question, and then we're feeding that into a prompt template. So it expects context questions as inputs. That's the template that is using. And then you ask a question and that will the retriever that we created using the LangChain repo as a source of information that will retrieve us certain documents. Okay, so our question is how can I initiate a react agent within LangChain? Right. So if you look at the documents that it has returned, it actually are different parts of the code that does talk about react agent. Right? So if we use an LLM, it should be able to generate a response based on that. Okay, So I want to show you what type of response we're going to get and also the speed of response as well, because we're using the GPU on M2. Right. So first you need to create a question answer chain. So we are passing on the LLM, that's our code number. Then the type of chain is stuff the prompt that we just created. So we run this chain and let me actually bring up the GPU usage pattern. So this is basically the current usage. So we are right here. You see that it's actually almost 50% because I'm running my screen recorder and some other softwares as well. But let's see what happens when we execute this chain. So this is the actual speed of generation from the model itself. You can actually see that there's a small bump you see here. So sorry. Okay, let me bring it here. So now you can see that it's fully utilizing the gpu. However, if you look at the answer, so it says use the following snippet to initialize a react agent and it simply gives us a single command to do that. Right? So the answer is actually not great at all. Now the question is why we're not getting a good response. Now it all comes down to prompt template. Now code llama and llama2, they uses special tokens both for the start of System message, end of system message, then start of user message or user input as well as the end of user input. So what I have done here is I created a simple system prompt. So the system prompt is you are a helpful assistant. You will use the provided context to answer user question, read the given talk context before answering questions, and think step by step. If you cannot answer a user question based on the provided context, inform the user. Do not use any other information for answering user. Right. So you could either use this or use this. That should be fine. This is going to become part of the system message, but then you really need to format it properly. So in this case, what I have done is it expects the system prompt or system message and I am actually putting it in between these special tokens. That's the system prompt. Then for the user input or user prompt, we are going to expect a user input. So that's going to be combined with the system prompt or system message. And then we are using these special tokens to enclose everything in between. Right. So this is basically the template that llama 2, as well as code llama uses. Okay, so next I'm simply putting everything together again. So instead of the default prompt template, I'm using that as my template. Then we are creating this Q and A chain prompt. So it's exactly the same as what we saw before. Now here's the question that we are asking again. So if we run this now, we can again look at the docs that is returning. So these are not going to be impacted at all by the system message, because that's the LLM part. So we are getting exactly the same documents that were returned before. Okay, next I'm going to run the chain. So this will basically create the chain and now we need to execute it. Okay, before running that again, I'll bring up my GPU usage plot. So let's run this and see what type of response we get. Okay, so it started executing, so our GPU usage goes up. So basically it's actually utilizing the GPU in this case now, and it started generating the response. Now if you see, this response is actually a lot more detailed and is the accurate response that you would expect from documentation. So now you see here it says to initialize a react agent, you will need to provide it with the necessary tools and configurations. Here is an example. So basically it's importing all the required different functions and objects. So here's the react agent. Then in this example they're using a lookup tool and search tool, right? And here's the output parser that you will need. So first create all the tools that you need, then create the react agent. Okay and you will also notice that I'm actually getting pretty decent speed in terms of tokens per second. So in this video we covered how to use CodeLamma as a replacement for GPT4 when it comes to interacting with your code base or documentation. We also looked at how to set up the proper prompt template in order to get more accurate responses from the model. And I also kind of showed you what type of performance you should expect when you're using LLAMA CPP on your gpu. I'm going to cover the installation in the next video, so make sure that you subscribe to the channel and turn on the bell notification button so that you get a notification whenever I upload that video. We also have a great discord server where so I would suggest everybody to check out details of the server in the description of the video. Thanks for watching and as always, see you in the next one.