CODE-LLAMA: Is it Actually Good?

Oh, okay, it got it wrong. Let's see if Chat GPT can actually do this. You probably have already seen the release of Code Llama and that it can beat GPT 3.5 on some of the benchmarks. In this video we are going to perform some of our own tests and I'll also be doing a comparison with the results of GPT 3.5. During this video, I will be running some tests which will vary in complexity. For Code Llama, we're going to be using the Perplexity Labs and we will also use GPT 3.5 and compare their responses. In most of the cases, I will be executing the code to see how the results look like. Now, one thing to keep in mind, even though I'm doing comparisons, but these are not exhaustive test. In order to see which model is the best, do your own tests or you can go by the benchmark results. Now, with that out of the way, let me quickly walk you through Perplexity Labs implementation of Codelama. So There are basically four different models that are available. There's a code llama34B instruct model, and then there are the original llama 2.7B, 13B and 70B. So we are going to be specifically focusing on the code llama34B model. Now, most of my tests are going to be running in Python, but unfortunately we don't have access to Python in here. Okay, so we'll start with the very basic test, and that is a simple calculator. So we are asking it to write a Python function with call, which is called calculator. It will receive two values and an operation. And the operation can be add, subtract, multiply and divide. Okay, so this should be able to easily implement this. All right, so here's the implementation. By looks of it, it seems to be fine. Okay, so I'm going to be using this simple Google collab in order to run the code. So let's run this code and let me pass two values, let's say four. And we are going to multiply add them actually, let's see what the results look like. Okay, so that's good. And then I ran the Same prompt in ChatGPT and it has absolutely no issues with it. Just by looking at the code I can say the implementation is correct. So we're not going to test this. Okay, so the second one is yet another textbook programming question. So we want it to implement a Fibonacci series. Let's see if can implement that. It should not have any issues whatsoever. All right. Okay, so the implementation looks perfectly fine. It's using recursion, but I'm going to ask it to not use recursion. Okay, so even without recursion, it's able to implement it absolutely fine. So this seems to be correct. So just for the sake of testing, I ran both of them without recursion. So this is with recursion, this is without recursion. In both of the cases, it seems to be giving us the correct answer. Okay, so for the sake of completion, here are the Results Based on ChatGPT, this is with recursion, and here's without recursion. And based on my test, it seems to be working correctly, so I would give both of them a pass. Now, these were great, but in terms of the actual usability, you are probably going to use it to help you out with small code segments. So, for example, here's a simple U test case. Write a Python function. So we provide the name of the function that removes duplicate items from a list while preserving the order. Okay, so it came up with a pretty neat implementation. So it's using the Python set to make sure that there are no duplicate items, and it's also using the list comprehension. So this is pretty awesome implementation. And I actually really like the way it also puts a small explanation, just like ChatGPT. So this seems to be correct. We can also look at the ChatGPT implementation and again, I would say it's also correct. The only thing is like, instead of a list comprehension, in this case Python Chat GPT is actually using for loops. Now here are the results when we executed both the codes. So this is the implementation from Chat GPT and this one is from codenama, and both gives us the correct responses when our input list has duplicate items. Okay, now we're going to move to some relatively complex questions. So let's start with a relatively simple one first, and that is write a Python function which is called validate password, that checks if a password is at least eight characters long, has at least one uppercase, one lowercase, one digit, and one special character. And this is the actual code of generation when you're using Perplexity Labs, which is pretty awesome. Okay, so first is doing the check if the length is less than eight characters. Then it checks if there is at least one upper character and lower character. Then it is checking whether there's a digit or not. And here it's looking for special characters. Right, so the implementation seems to be correct, but let's test it out. Okay, so here are the Test results. I'm testing the code and I provided this test password. And then there's an exclamation mark at the end. It says false because we're missing a number in here. So if I add, let's say one, now it says true because it fulfills all the criterias that we have listed in here. Now let's look at the Chat GPT implementation. And this one is actually pretty neat because it's using the regular expressions. So that's, that's actually pretty awesome. Now in terms of the tests, it's doing exactly the same thing. So for example, it's looking at the characters. So it could be upper and lower character. This is basically lower characters, then searching for upper characters, then numbers, and then those special characters that we define. Right. So both of them have similar implementation. But I actually like that ChatGPT is using the regular expressions. Okay, now this one is going to be a bit tricky. So we are asking it to write a Python function which we are calling Roman to integer, that converts a Roman numeral to an integer value. So let's see if it's able to implement that. Okay, so here's the implementation from code llama and we are going to be using a simple test. So I'm providing this Roman numeral which is 1994 in integers. Let's see what the answer is. Oh, okay, it got it wrong. Let's see if Chat GPT can actually do this. So here we provided exactly the same prompt and here's the response. So I'm going to just copy the implementation of the function rather than the whole code. Oh, interesting. So Chat GPT got this, right. So probably there is a small mistake in the logic when it comes to code. When it comes to code Llama. Okay, so now we are getting into some relatively complex questions. Now here's a question that was actually posted on the Discord server. So I would actually recommend everybody to check out our Discord server. There are some really amazing people helping each other out. You will learn a lot. Okay, so the, the prompt is or the question is given a string such as this, write a Python function that rearranges the characters so that the matching letters are grouped together. The output should be a string like this. So let's say in this specific case there are two E's, two T's, one C, R, two G's, and H. So the solution should only use for loops, while loops and if statements without relying on advanced built in functions. And I am asking it to write a Python function. Okay, so in Terms of the logic that it came up with, the solution seems to be fine, but we are going to test it out. Okay, so here is the code and in order to test it, we're going to use the same string. However, before testing it, let's see what Chat GPT says about it. Okay, the problem with ChatGPT solution is that it's actually using the sorted functionality and we specifically asked it not to using use any built in advanced functionality. So let me rerun this again and let's see if ChatGPT is going to give me a different solution. Okay, so in this case it's definitely not using any of the advanced functionalities within Python. It's just using the while loop. So at least it fulfills add requests or requirements. So let's copy this and test it. Well, both of them actually failed. So this is from Code Llama and for some reason it actually dropped some of the characters in the output string. I guess like it simply took one string if there is a duplicate. And that's why we are seeing the output like this. In terms of ChatGPT's results again, it's simply returning the same string again. So it's not really doing anything. So that means both of them are not able to solve this specific problem. So it's a fail for both the models. Now keep in mind that if we run this again, it might be able to solve the issue, or if we tell it that there is something wrong with the solution because these are chat models. But for the sake of time, I'm not going to do this. Okay, so since it's a very interesting question, so I'm going to ask GPT4 to see if GPT4 can give us a proper solution. Okay, so here's the implementation from GPT4 and I tested it and GPT4 is definitely able to do this. Okay, so here, here's the final test. Right? Right. HTML code for a web page that has a single button. When the button is pressed, it will change the background color of the website with a random color. And with the button press, we also want it to show us a random joke. So I'm going to Change this to codename 34B. Let's run this and we will run the Same command within GPT 3.5. Okay, so it came up with the code. Let's have a quick look at it. Yeah, I think it missed the joke part. Okay, it's trying to fetch the random joke from an API. So I'm going to run this again and let's See if it's still using the API, then I'll have to explicitly ask it to write jokes. Okay, so it actually wrote jokes in here. Let's see if this code actually works. Okay, so for ChatGPT, this, it has jokes in there, so don't really see an issue. So we will test both of these codes and see if both of them gave us correct HTML code. Okay, so we are going to paste the code here. Let's run this. Now we do see a click button. So let's click this. And it does display a joke here. When I hit ok, it actually changed the color. Right. So not exactly what I was looking for, but it satisfies all the requirements. And this is probably the second open source large language model that has been able to implement this specific task fully. I think the only other one was Visual Kunya that I tested. Okay, next we're going to look at Chat GPT's implementation. So I'm going to copy this code and replace it. Let's run this. Okay, so it says change color. All right. Awesome. This is pretty neat. Yeah, so this also works pretty, pretty fine. The code is actually shown here rather than a pop up and it does change the color. Okay, so in terms of my assessment, it does seem like this code llama2 is a very capable model when it comes to code generation. We specifically focused on its ability to generate code. We did not test its ability to fix codes. I'm going to test that out in another video. And as I said in the beginning, it's not really a comprehensive test between ChatGPT and code LLAMA. I would recommend everybody to test this in your own applications that will give you a better understanding of its abilities. If you like the video and content on this channel, there are a number of ways you can support my work, including Patreon. Check out the description of the video for more details. I hope you found this video useful. Consider liking the video and subscribe to the channel. Thanks for watching and see you in the next one.