NEW FalCoder-7B: The Best Open Coding Assistant? | Installation & Tutorial

We all know that GPT4 is great at coding. However, over the last few weeks we have seen a number of open source LLMs focused on coding that were released. One of the earlier and major one was Starcoder, which is an LLM that is specifically trained on source code and natural language text. More recently the WizardLM team released Wizard Coder, which is a 15 billion parameter model that is trained on programming codes. A couple of days ago, yet another programming focused model was released and this one is called Falcoder. The question is why is this model significant? This model is based on Falcon 7B, a big brother of that model. Falcon 80 billion parameter is currently leading the open LLM leaderboard. So it's basically one of the best open source models out there. In this video we're going to closely look at this model, understand how it's trained and what are its capabilities, and then I will show you how you can run this locally on your own machine. I will be showing you the results from the full model, but you can use the Qlora version if you do not have enough memory. Okay, so let's first look at what this model is all about and how it's trained. So it's essentially the Falcon 7B model which is fine tuned on the code ALPACA20K instruction data set using the Qlora method with the PEFT library from Hugging Face. Now, first and foremost, in order to use a model, it's very important to look at the actual training data so that you know what inputs the model expects. The good thing is that in this case we actually have access to the training data. So there are a number of programming languages in this training data. On the left hand side you have the prompt and on the right hand side you have the corresponding completion or the output. For example, let's look at the first one. Create a Java class which sorts the given array of numbers and so you have an array of numbers along with the instructions. And here is how it's supposed to solve it. I believe it's a very diverse data set because you see examples from the R language. There are Python examples and then there are even C sharp, JavaScript C and even SQL. While I was looking at the training data, not all of them seems to be coding examples. So for example, if you look at this, why is overflow an issue in computer science? So then here's an answer in clear text. So it's not really a code belonging to any language. Here's another example. Describe the difference between a loop and a conditional statement. Now, based on this training data, it seems like the model will not only be able to generate code, but it could also give you responses in plain text, because there are a few examples in plain text on which it's trained. This is why it is very important to understand the training data set so that you know the capabilities of the model you are dealing with. Now, one more thing before we actually go and experiment with the model and that is if you look at these code examples in the training data set, most of them are very small snippets of code or very small functions. So based on the training data set, you cannot expect the model to solve very complex programming tasks, but rather it is going to be working as an assistant which will provide you with helpful code snippets. We looked at the training data, but now let's look at how you can actually use this model on your local machine. In this specific case, I'm actually using the full model. So if you do not have enough memory or ram, you probably want to use the Qlora version of the model. Let's walk through the code first. Now here in the first cell we are installing all the required packages. Next we are going to be using the transformer library or package from Hugging Face to load the model. So we are simply importing all the required packages and functions. Next we are defining our model id. So this is coming from Hugging Face. Then we are initializing the tokenizer for that specific model which is the Falcoder 7B. Next we are loading the model from. Since it's a pre trained model, in this case I am using MacBook Pro, so that's why I'm setting this to. Now if you're running this on an Nvidia gpu, you should set this to cuda. So next there is this generate function that accepts an input in the form of instructions. The max new token. These are the number of tokens that you want to generate. So for smaller problem you probably want to set it to a smaller value. I set it to 2098. The next parameter is temperature, which essentially controls the creativity or hallucination. So we do not really want any creativity for programming solutions. That's why I'm setting it to a very low value. And these are some of the parameters that control the behavior of our large language model. Now when it comes to this specific model, since it's based on Falcon 7B, so you have to provide your prompt or instruction or input in a very specific format. So here's the format that it expects the input to be so you are going to have the instructions, then a triple hash on the next line, solution followed by colon, and after that it will generate its response. All right, next we are simply setting some other input parameters. So for example, we run our input or prompt through the tokenizer. Now again, if you're running it on Nvidia gpu, so just set this to cura and then there are some configurations that we already got as an input to the function, right after that we're going to be running the generate function on the model to generate output response and we simply give it all the inputs that we provided to the model and here is how it's going to return the output. So the output and then we simply split it on this special token for the solution that is defined in the input. Now I was playing around with the model itself to see how good this is at coding. So this is an example prompt that's actually provided in the repo, and the prompt is design a class for representing a person in Python, right? And actually it came up with a really good solution. So there's a class called person, and then within that class define different methods. So for example, get name, get age, get gender, right? And then there are some set of functions. So set name, set age, set gender. These are the type of attributes you would expect a person to have. Here's another one. Write a function that takes a list of lists and flattens it into a one dimensional list. Name your function flatten and it should take a single parameter and return a single list. And it actually used list comprehension. So here you get a list of list as an input and the return type is simple. A list where everything is concatenated together. It's using this comprehension to do it. So I think it's pretty neat. And it actually also gave a couple of examples, which is pretty awesome. So here's a fun example. A string is a palindrome when it is the same when read backwards. So if you reverse a string, if it's exactly the same, then that is supposed to be true. So it wrote a function, the implementation seems to be correct, and then it actually has this example script to run. However, the output is a bit weird because first it says enter a string, right? Then abc and it says it's not palindrome. And then it has another example abc, and this time it says it's a palindrome, right? So it's not really an interpreter where it's actually running the code, but it's simply giving you examples. But the implementation actually Looks correct. Now before this we just looked at a few fun examples. However, here is an example use case that you probably will be using. So I asked it write a function in Python that accepts a file as input and uploads it to NS3 bucket, right? And actually it wrote the code which seems to be correct. Now these are the type of example use cases for which you will be using LLMs specifically trained for coding. Since the model was trained on programming related concepts, so I thought it would be a good test to ask it some questions. So the question that I asked was what is the difference between a class and a method? And the answer it gives is pretty actually accurate. So it says a class is a blueprint for creating objects, while a method is a function that is associated with an object. A class is a collection of variables and functions, while method is a function that is associated with an object. Now it further goes on to say a class can have multiple methods, while method can only be associated with one class. Right? So the explanation it provides is pretty accurate. A couple of things to keep in mind. So I have been testing this model specifically on Python, but the authors have provided examples of SQL and some other languages during my testing I found this to be pretty accurate on the examples that I ran. Now a couple of more things to keep in mind While using these LLMs which are trained specifically for coding. First, they are not really trained to give you a whole program. Rather they are simply assistants which can help you out with smaller code snippets. So for example, uh, if you need a very small function, these models can write those for you if you give it proper instructions, but don't expect it to be cranking out a whole application for you. Now the second point I think which is very important to highlight, is that I'm running the full model in this case. Now as a result, if you look at this simple instruction, it's actually taking around two minutes to complete. Now based on the hardware you have, your mileage may vary. It's very exciting to see these task specific open source large language models and I believe that is the way to go. These smaller open source models cannot compete with something like GPT4 or even ChatGPT for more generalized knowledge. However, if they are trained on very specific areas and subjects, they might be able to compete and I think that is the way to go. I hope you like this video. If you have any questions, put them in the comment section below. We have a very vibrant community in our Discord server, so I would recommend everybody to come and join us over there and continue the discussion. Consider liking the video if you found it useful, and maybe consider subscribing to the channel for updates like this and more in depth tutorials. Thanks for watching and see you in the next one.