Free LOCAL Copilot to Take Your Coding to the NEXT LEVEL

In the previous video I showed you how to use llama3 as your copilot in VS code. But we were serving llama3 via Groq API. In this video, I'll show you a local replacement of Groq API for Copilot. We will be still using the Code GPT extension within VS code. But for serving the model, we will use both LM Studio as well as Ollama. Both these programs enable you to run local models. If you're new to Ollama or LM Studio, I have dedicated videos. Links are going to be in the video description. Okay, to get started, we first need to install the Code GPT extension. So just go to extensions, type Code GPT, look for Code GPT chat and AI agents. Make sure to download the version that has 1.2 million downloads. Okay, once the installation is complete, you will see the Code GPT extension here. So in the previous video I showed you how to use Groq as an API provider. But in this video we will look at both LLM Studio as well as Ollama. And we will be serving Llama 3 models using both these applications. So first let's get started with LM Studio. So click LM Studio. Now we need to set up LM Studio and make it an API server that is going to be listening to requests coming from this code GPT extension. Okay, so make sure to have LM Studio up and running. Next you need to find a model that you want to serve through LM Studio so you can use the search to look for the model. For example, if we type llama3 then it will go to Hugging Face. List all the Llama 3 variants that are available on Hugging Face. Now in my case, I have downloaded the Llama 3 instruct gradient 1 million token version. So I'm going to select that model. It will load that model for us. Now we need to create a server that will serve this model through an API. So for that go through Local Server, click on Local Server. Make sure that you have the appropriate model selected. You can also change the port on which you want to serve this model. But we are going to keep everything to default. Next click on Start Server. So this will start the server. Now LM Studio is basically listening for incoming requests and we can use this model as our copilot within VS code. Okay, so I will go back to VS code here. Just make sure in providers you have selected LM Studio as a provider. And let's test this whether it's able to communicate with the Llama 3 variant that is being served using LM Studio. Okay, so my prompt is going to be write a Python program that will download a file from S3 and store it in a directory path provided to the function. So let's see if it can write that function. Now you can see that it actually started communicating with the server that is up and running and it is giving us code that seems to be correct. Now, on the server side you can actually see that it receives the request and it generated the response based on the request that it receives. Right? So this is the system function that is being used for the Llama 3 model by LM Studio and this is the actual input from the user that was coming in and it generated the response and it says finish streaming response. Alright, so apart from asking questions directly, you can also use all the functions that are available through Code GPT. So for example, if you select a function, you right click on it, you can ask Code GPT to explain that that is going to be passed on to the Llama 3 that is being served through LM Studio. Right now, the speed is not as great as what we were seeing when we were running this through GROK API, because Grok API has much better inference speed. But at least everything is running locally and you're not communicating with an external API at this point. Now let's try to refactor the code with this model. So we're going to click on Recall Refactor Code GPT. Now, the quality of output is going to depend on on the model that you're serving. So for example, in this case it says I'm unable to provide a refactor code for this task. However, I can suggest some improvements that could be made to the original code. Add type hint and then use descriptive variable names instead of. But the quality of output is going to depend on which model you're using. So for example, in this case we are just serving the 8 billion model. If we were to serve, I think the biggest 70 billion model, it might be able to give us much better responses. Okay, next, let me show you how to use Olama within Code GPT as your copilot. Now, LM Studio is great, but it's not completely open source. If you're looking for a completely open source solution, I'll recommend to use Ollama. So let me show you how to set this up as your copilot within code GPT. So first make sure to download Ollama from ollama.com and install it. Next, we are going to look for Ollama in my applications I'll click this and this will start the Ollama server. Now there are a couple of additional steps when you're running Ollama. So first you'll need to select Ollama from this list. So we'll select Ollama. Then Ollama will let you also provide the model that you want to use. So instead of LM Studio just directly connecting to a server, here you actually need to provide the name of the model that you want to use. But before that, you actually need to start that server manually in your terminal. So for example, we're going to be using the Llama 370 billion model. But before doing that, I'm going to go to my terminal and here I'm going to type olama run then llama 370 build model. If I do this for the first time, this will download the model, which will take some time, but in my case I have already downloaded the model. That's why it will just load the model and serve it through the API server that Ollama has. Okay, so the model is up and running and I'm going to go back to code GPT extension. And here we're going to select the Llama 370 billion model. So let's try the same prompt rather Python program that will download a file from S3 and store it on the local machine. Now in this case, since I'm running the 70 billion model, it's going to be pretty slow. I'm running this on an M2 Max with 96 gigabytes of RAM, but at least it's able to communicate with Ollama and the response seems to be accurate. So this is a good start. Now let's test it on this re ranking methods. If you remember, for our from our earlier test, the 8 billion model was not able to refactor in. So let's try the same function on the 70 billion model. So we're going to send the request to refactor this specific code. I actually wrote this code with the help of llama3 when I was using the Groq API. Now this will potentially take a bit longer to process because it's running this model locally and it's a much bigger model. Right. But here it started implementing the refactored code. So I'm going to wait for the implementation to finish and then we'll see like what the model actually changed in this code. Okay, so here are the changes. The first one is consistent naming convention. So it basically replaced my function name from rerankdocs to rerankdocuments, which is pretty good and improved variable naming. So variable names such as retrievedocs has been changed to retrieve documents Separated Default Method Definition the default method was separated from the Re Ranking methods dictionary to better for better clarity. Now what it did here was that we were using this dictionary which was going to receive a name of the model that some of the user wants to use for RE ranking and then select the corresponding function to use. But we also had this default value if the user doesn't provide anything and it took that out and implemented it at as a separate lambda function right? So minor changes. There is actually nothing much that the model could have changed here, but still the 70 billion model definitely is much better as you're coding copilot compared to the 8 billion version. One more thing to say to highlight is that there's kind of a limited version of models that are currently available within Ollama, but LM Studio gives you a lot more flexibility because it's not really looking at the model here from this list, but it can use pretty much any model that is available on Hugging Face. The downside is that LM Studio is not completely open source, but still it's a really cool project, especially really like the UI implementation. Ollama on the other hand is truly open source. They have some really good integrations, but I hope that the code GPT will enable us to use a lot more Ollama models. I hope you found this video useful. Thanks for watching and as always, see you in the next one. Thanks.