LocalGPT: Convert Your Chats with Docs to Fine-tuning Datasets

Did you know that you can use local GPT to create training datasets for fine tuning your local LLMs? If you're not familiar with local GPT, it's my own project that lets you chat with your documents on your local machine. It already has more than 18,000 stars on GitHub with over 2,000 folks. Local GPT enables you to chat with your documents in different formats, but there is a little known feature that you can use to create training data sets if you are new to the Local GPT project. I have a playlist of all the videos related to the project. Link is going to be in the description. The idea behind this is very simple. Local GPT lets you create a vector database of your knowledge base. Then when the user asks a question, Local GPT will generate a response based on the LLM that you have chosen. However, you can also store both the query and response pairs. That means that if you have enough interaction with your files, you can use this data to fine tune a model that is specifically trained on your own documents. I think this is going to be extremely helpful in cases where you have a lot of users who are using Local GPT and us storing the question response pairs in a local storage. Let me show you how to do this within Local GPT using a single command line argument. But before that if you want to run local GPT you will need access to a gpu. Now depending on the model that you want to run you will need different amount of VRAM in case if you don't have access to a computer with a GPU you can use different cloud providers. I have partnered with a company called Mast Compute and you can rent one of their virtual machines which comes pre configured with Local GPT. If you use link in the description below you will get access to this pre configured virtual machine that has an A6000 GPU which has 48 gigabytes of VRAM. This is enough to run 70 billion model in four bit quantization. It comes with local GPT, Visual Code Studio and Conda pre installed. Now the best part is that you can use this virtual machine for any of your own projects. It doesn't have to be restricted to local GPT. Currently you're going to be paying $4 for three hours dedicated usage. However, if you use my code prompt engineering without space, you're going to get 50% off of this price. I will get a small commission if you decide to use this virtual machine. If you decide to use this, you will need to download the thin link client to in order to access the virtual machine. Tintin client also works on the M1 and M2 series. Thinlink client also works on the M series of Macs, but you will need to install Rosetta. Now. If you decide to use this virtual machine using Thin Link, you can set it up by providing the server ID as well as the password that Mast Compute is going to send you. Once you reserve your virtual machine, you can also enable the transfer of files between your local machine as well as this virtual machine. So let me show you how to do that. So first click on Options, then click on Local Devices, then click on Drives, then click on Details. Here you can actually add the path of the folder that you want to share between both your local machine as well as the remote virtual machine. So for example, if I click on prompt this is my local machine within my documents I created a folder called Thin Link. This is going to be shared between my local device and the virtual machine. So just click OK and now I need to provide my server IP address as well as the password that Mast Compute has sent me. Once you log into the virtual machine, here's the interface that you're going to see. So on a desktop you have the local GPT folder which is cloned from the GitHub repo. And there is a welcome doc that I would highly recommend to read before start using the vm. This provides all the information that you need to know in order to start using Local GPT on this virtual machine. As I said, it comes pre configured with a local GPT code base from GitHub. There's also Conda pre configured with Visual Code Studio. The best thing is that you can use this virtual machine for anything you want. It doesn't just have to be related to local GPT. Now here's a step by step process of how to get started with Local GPT. We are going to look at this in a little bit. However you want to look at the frequently asked question questions. There is just one small issue with the copy paste functionality. If you're using this on Linux you can use Control C and Control V. On Mac I believe it's Command C and Command V. The only other thing you need to keep in mind is that if you're copying anything to the terminal on the virtual machine, then you need to use Control Shift and V. They highly recommend to use the ThinLink client in order to access the virtual machine. But you can also use your browser as well. Although that's not recommended by Master Compute. Visual Code Studio is pre configured on this virtual machine and it will open the Local GPT folder. They have also included a virtual environment called Local GPT which you can directly activate here. So we're going to use conda activate Local GPT and that will activate the Local GPT virtual environment. You need to do one extra step before you start using the Local GPT. You'll need to manually install all the requirements. So we're going to just copy this, go back to the terminal and we'll paste that you can run this. This will start installing all the requirements. I'm working with the Mast Compute team to have a virtual environment pre configured with these requirements so you don't have to do this step. Next, let me show you how to use Local GPT to log user interactions and create a data set that you can use to fine tune your LLMs. In this case, we're going to be using the Orca paper as our source document. This comes pre configured with Local GPT Project. A side trick. By default you will not be able to see a PDF preview, but what you can do is you can go to extensions, look for PDF and just install the PDF Viewer extension that will enable you to open PDF files within Visual Code Studio. I personally find this very helpful. Before we start interacting with this file, we need to create a vector store and for that we're going to be using the ingest py. So in order to ingest our document, we're going to use the command Python ingest py. When we run this command, it will create a vector store that has the embeddings of different text chunks from this document. We have a total of 193 chunks. The paper itself is 51 pages long and it will create a DB folder for us that has the Chroma DB Vector Store. Next, we're going to use the Run local GPT file to start interacting with this vector db. Within the Run Local GPT file we have this new option Save qa. This will enable you to to save your query as well as the model response. Before running the local GPT file, let's look at what model we are using. So by default this virtual machine is going to be using the llama2.7 billion parameter model and this is the unquantized version. However, if you decide to use a quantized model, I'll highly recommend to use the GPTQ file format then rather than the GGUF file format. And the reason is that the GPTQ quantized models are well optimized for Nvidia GPUs. So you will get better performance if you use the GPTQ models on this A6000 GPU. And as I said in the beginning, you will be able to run up to 70 billion parameter model if you decide to go with 4 bit quantization on this virtual machine. In order to enable logging of the user question and answers, we are going to use this extra command with run local GPT and that is going to be dash dash save, underscore QA and let's run this. I already downloaded the model, that's why it's simply loading the checkpoints. However, in in your case, it will first download the model. The download speed is pretty good. I think it has a link of 1 Mbps, so that is pretty nice. Okay, so we are all set to start interacting with local GPT. But before that I'm going to click on this split terminal here. I'm going to use command to watch the GPU usage and that is going to be watch dash N1 and then Nvidia dash SMI. So using this command now, it will update the GPU usage every second. Now we can start asking questions from local GPT relevant to this file. Here's my question, what is instruction tuning? And you can see that the GPU usage goes up if you run a bigger model, it will use a bigger portion of the gpu. All right, so here's the answer from local GPT. It says, hello, I'm here to help you with your question. Now we can put a system instruction so that it doesn't do that. But it goes on to say instruction tuning is a technique used to train smaller language models to perform specific tasks by learning from input response pairs. This involves using a large language models such as GPT4 to generate responses to given inputs and then fine tune the smaller language model to mimic the behavior of larger model. So this got the response correct. Now I asked a subsequent question, what is the role of system prompt in instruction tuning? It first talked about the instruction tuning. Then it states, to answer your question, the role of system prompts in instruction tuning is to provide additional information to the model beyond what is contained in the input alone. And the response is coming from the paper which actually talks about the role of system instructions. The paper also talks about different benchmarks. So here is AGI evolve. So I asked the model what is AGI evolve? And it says based on the context provided, AGI evolve appears to be a benchmark for evaluating the general abilities of artificial intelligence models and tasks related to human cognition and problem solving. Local GPT does a really good job when it comes to retrieving information from the documents. The paper also talks about the performance of the Orca maru compared to ChatGPT and open source models such as Vicuna. So here I asked it which model between ORCA and chatgpt is better when it comes to multilingual understanding? And it says based on the provided context, it appears that the ORCA performs slightly better than ChatGPT on multilingual understanding, which is correct according to the paper. So let's assume you are working with your documents and you are asking these questions. The model is generating good responses. You can essentially use this data to fine tune a model where you don't have to provide a context anymore. And that's what Local GPT does for you. If I type exit, it will stop running Local GPT, but when you look at the files you will see a new folder called Local Chat History. And if we click on this QA log CSV, here are our questions and answers. So here we have the first column which has the timestamps. The second column is a question that we asked. The third column is the responses the model generated. Now the great part is that when you interact with Local GPT, it will keep appending your question and responses here to this CSV file. And you can potentially use this to fine tune your own model because you have your question answer pair data set, which I hope a lot of people are going to find it very useful. Now if you want to modify where the file is being stored, you can look at this util PY file and here it creates this Local Chat History folder and then put in this QA log CSV file. You can change the file name if you want. So for each conversation you can create another file as well. Or for each user you can have a separate file. I hope this was helpful. I'll be creating videos on Local GPT to highlight these little known features within the Local GPT project. If you are working on integration of Local GPT as part of your project and you need help, you can reach out to me. My contact details are in the video description. If you would like to contribute to the local GPT project, check out the GitHub repo. And if you would like to support my work, check out the options in the video description. I hope you found this video useful. Thanks for watching and as always, see you in the next one.