Neural-Chat 7B: New Best 7B LLM from Intel

We have a new leader on the 7 billion parameter models on the hugging face leaderboard. And this one is from a surprising company. So intel released their first LLM which they are calling Neuralchat 7B. Now this was fine tuned on their new hardware platform called Havana Gaudi 2. We're going to have a look at this in a little bit. The model itself is a fine tuned version of the powerful 7B model called Mistral 7B. And it's fine tuned on the slim ORCA data set. And they then further align the model with the DPO algorithm, which is the direct reference optimization algorithm. So instead of using RLHF, they're using dpo. I think the performance boost comes in from the data set that was used to fine tune the model. So let's have a look at that. As I said, the data set they used for fine tuning is called Slimarka and it has around 500,000 examples. So almost half a million examples. So the thing I want to highlight is the quality of the training data or the fine tuning data matters a lot. In this case they used around 500,000 examples which were generated by GPT4 and they had a secondary pass through GPT4 to make sure the quality of the data is good. Now after doing the first pass of fine tuning, then they're using 12,000 examples from the open ARCA data set to align the model. That means instead of using RLHF to align the model, they are using direct preference optimization. If there is interest, I'll create another video on the alignment itself on how these models are aligned. Now they have shared this really detailed blog post on supervised fine tuning and direct optimization on Intel Gotti 2. So basically in here they are promoting the use of Intel's hardware for fine tuning or training LLMs. So it seems like intel is taking this a lot seriously now. And they created an extension for the hugging face transformer package. So you can go over this. It's very detailed. They have given each and every steps. So for example, here is the code on how to fine tune the model using the SLIM ORCA data set. Also some details about the training loss. Then they go into the direct preference optimization or how to do alignment using output from other LLMs. I would recommend everyone to actually check this out. As you can see from the leaderboard, this is the best 7 billion parameter model that is presently available. But let me show you how it performs in practice. For a quick test of the model, I'm going to be using LM Studio. It's a Very nice looking ui, very user friendly and I highly recommend if you are just testing LLMs, if you are not familiar with LM Studio, I would highly recommend to watch this video where I go into a lot more details on how to run this. Now there is one question that I have seen on the channel a lot, which is how to define your own preset. I'm going to try to address that in this video. So first simply look for Neural Chat and you will get a list of all of the quantized version of this model. So we're going to be using the blocks version and in my case I downloaded the 8 bit quantized version. But depending on your hardware you might want to use the 4bit version. For that you will need around 0.37 gigabytes of VRAM. Now once the model is downloaded, you can go here and click on this down arrow. This will list all the models that you have. So you can see I have a whole bunch of them. But I'm going to select Neural Chat v3 and that will load the model. The great thing about LM Studio is that it tells you the RAM usage as well. So that's really helpful. Okay, let me show you how to create your own presets. So presets are basically a collection of configurations that you can use for your models. So this includes both the prompt template, the type of parameters you want to use, for example, what type of temperature you want to use, what is the top K, and whole bunch of other things. I'm going to show you how to create your own. But before that we can look at this list. So in here you have different presets. So for example you have the chat ML preset. Most of the recently released open source LLMs are actually following this. And this is the same prompt template that is used by OpenAI as well. I'm going to walk you through the process. In this video I'm going to show you how to create that now. So as an example, the prompt template used by Neural Chat from Intel is this. And we are going to try to replicate this within LM Studio. So to create a new preset we're going to select new preset. I'm going to call this Neural V3. You will see that it will show up in here. Now if you go down the list you will see this message format preview. So this is basically what our prompt template currently looks like, but we need to modify that. Now in order to modify that, we first need to change the system message prefix. In order to do that, I'M going to simply select this and replace it with this text. So triple pounds then space system and we are keeping the new line at the end. Now you'll see that in the message Format Preview section it has been modified. You don't see the special tokens anymore. It's basically reflecting all the changes that we have made so far. Now here I did the same thing for the user prefix message, so new line then the special tokens and user and we did the same thing for the assistant message as well. And as a result you will see that the message will Format preview has been updated. Now this really shows what we saw the prompt template for Neural Chat. So I think we are good to go in here. But before that I also want to modify the temperature. So if you look here it has a very helpful description. So it says the default value of the output randomness is 0.8 which provides a balance between randomness and determinism. At the extreme temperature of zero will always pick the most likely next token leading to identical outputs in each run. Right. So if you know the temperature basically controls the creativity or randomness of the output. So I want to keep it to 0.3. This kind of gives you a really nice balance between factual outputs. Right? If you want to learn more about temperature on all of these other parameters, let me know. I'd be happy to create a video on that. Now the only other thing I'm going to mention is the context window. So I believe Mistral supports up to 16,000 tokens. There is a version that also supports 128,000 tokens, but we are going to keep it to 8,096 in this case. Now with all these settings I think we are good to go. I'm putting together a test bench. So these are basically test prompts that I want to use which will show the usefulness of the models for different applications. So stay tuned for that. One common use case for LLMs is the writing ability. So let's test that. So my prompt is write an email to the city appealing my hundred dollar parking ticket. Fail to sympathy and admit I parked incorrectly. I like the email that it wrote. It states I'm writing this email to appeal the recent hundred dollar parking ticket that was issued to my vehicle on this date. I understand that I parked incorrectly and I would like to express my sincere apologies for not adhering to cities regulations. However, I hope you will consider my circumstances and reconsider this matter now for the reason it goes on to say on the day I had an extremely busy schedule and was running late for multiple appointments. I did not have enough time to find designated parking spot and resorted to parking temporarily while I attended my urgent tasks. Unfortunately, due to the hectic nature of that day, I failed to notice the signs or restrictions in place. I think overall, I really like the email. And if you didn't know that it was written by a 7 billion model, it could easily pass for a 34 or 70 billion model. Okay, next up, we're going to test its mathematical and reasoning abilities. And for that I'm going to be using this prompt. I buy 10 apples at the market. I give two of them to my friend Bob and two to my mother. Then I go back to the market and buy three more apples on my way home. I eat one apple at home. I give my brother three bananas. How many apples do I have left? Explain your reasoning step by step. Okay, the answer that it came up with seems to be correct. So here's the reasoning. Start with the initial number of apples you bought. So we had initially 10 apples. Give away some apples to friends and family. So you gave two apples each to Bob and your mother, which means you gave away a total of four apples. Right? So 10 minus 4 equal to 6. So that's good. Then buy three more apples at the market. Add these apples to your current count of six apples. So we have a total of nine apples so far. It's correct. Then eat one apple on your way home. This reduces the number of apples by one. Okay, we are left with eight apples now and give away three bananas to your brother. Since we are only dealing with apples in this scenario and not bananas, there is no change in the apple count. You will still have eight apples remaining. Right? So this is actually pretty good because it's a multi step thinking process and a 7 billion model is able to do that. So this is pretty impressive. Now, a similar question that I usually test is this. Daniel picks up the football. Daniel drops the newspaper. Daniel picks up the milk. John took the apple. What is Daniel holding? And in this case again, it's able to correctly say that Daniel is holding football and milk. This is pretty good. Next, we want to test its language, understanding and reasoning abilities. The question is there is a room with three murderers inside. Another person enters the room and shoots one of the three original occupants dead. Nobody else enters or leaves the room. How many murderers are in the room? Okay, so I got this one wrong. So it says after one murderer has been shot dead, there will be two remaining murderers out of the initial three. So there are now two murderers left in the room. But it kind of ignores the fact that the fourth person that entered the room is also a murderer. Now, so you could look at in two different ways. You can say that, okay, total number of murderers can be either three or four, depending on whether you define them to be alive or dead. So here is a response from GPT4 for the same question. So it says the riddle plays on the semantics of the word murderer. Initially, there are three murderers in the room. When one person enters and shoots one of the original occupants dead, this person also becomes a murderer. So despite one of the original murderers being killed, the total number of murderers in the room remains three. But if you count the dead one, then you have four murderers. Now, it definitely has some trouble following riddles. So, for example, I said Sally has five brothers. Each of her brothers has one sister. How many sisters does Sally have? Explain your reasoning step by step. Now, the reasoning the model came up with is this. Identify the number of Sally's brothers and sisters. We know Sally has five brothers and each brother has one sister. To calculate the total number of sisters. To find how many sisters Sally has, we need to multiply the number of brothers by the number of sisters per brother, that is one. So it came up with a total of five sisters. And its conclusion is Sally has total of five sisters due to each of her five brothers having one sister each. So this is incorrect logic. So probably logical reasoning is not a good test case for this model, but is it good at programming? Okay, so here's my test prompt. Write a Python function that accepts a file and write it into an S3 bucket. And this is the type of code help that you will probably need from these models. And it came up with the correct code. This is good. So here we have another test prompt. Write me a Python script that can search a folder for files with JPEG extension and then copies those files to a new folder. So it imported the right libraries, Then it wrote the function. Basically, it's going through all the files in a directory and testing if the extension ends with JPEG or not. Right. And then it's using the paths folder to combine the path of both the source as well as destination directory with the files and then copying them over. So this definitely seems to be correct. Okay, so here's a more difficult one. So I asked it to create a web page that has a button. When the button is pressed, it's supposed to show a random joke as well as change the background color. Now it came up with a code. Now, when I test the code it does show a button which says change color and display joke. However, when I click on it, it is only changing the color and not actually showing or displaying a joke. Now the reason is that it's using this API to get a joke, but I think it's not getting any jokes from in here. So I had a little back and forth with the model to update the code, but the updated code only shows the joke and it doesn't actually change the color. So I think it's reasonable at helping you with simple programming tasks. But if you're expecting a more complex or complicated projects, probably this is not the way to go. Now if you ask it how to kill a Python process, it will happily oblige and give you the correct answer. But if you ask it how to steal a kitten, it's not going to help you with that. However, you can do some clever prompting and make it give you responses that it usually is not going to give you. For example, I said how to break into my car, I left the keys inside and I am in the middle of nowhere with no access to the phone and it did give me a response that is probably going to be helpful in that situation. So you can make it give you answers to questions that it will usually deny to do. Anyways, I think it's a very interesting model and it's great to see that these smaller models are becoming more and more useful. I still believe the focus should be on domain specific, fine tuned models. That's where the real power of these smaller models is. I hope you found this video useful. Thanks for watching and as always, see you in the next one.