NEW MISTRAL: Uncensored and Powerful with Function Calling

The Mistral team just dropped version three of their 70 billion model. Now, in the past they would just drop magnet links, but this time they directly released the model on Hugging Face. So very different from what we have seen before. Now, unlike the previous releases, they did not provide any evaluations, but I'm assuming the performance is going to be very similar to the Mistral 7B model. But there are some prominent changes. This is a completely uncensored model. They have extended the vocabulary just by a few hundred tokens. The extension of vocabulary is related to its ability to do function calling, so it now natively supports function calling. And they also updated the tokenizer it supports. So now this is version 3 and it has the same context window of 32,000 tokens with the package. They also updated their Mistral inference Python package that you can use to run inference on this specific model. I'll walk you through how to install this new package, how to download the model, how to run your initial queries on the model. So we're going to do a few tests. It's not going to be comprehensive and we're going to also look at this function calling example. Okay, so here's the Python notebook. As I said, we're going to do a quick test. More comprehensive tests are going to be done in subsequent video and I'll also show you how to fine tune this model on your own data set. Okay, now first we need to install this Mistral inference package. This is the recommended way to do inference on the Mistral 7B V3 model. But you can also use a transformer from Hugging Face. Next we need to download the model. So, so we're going to define the path where we want to download the model and we check if the directory does not exist. This is going to create the directory. And if it does exist, then it's not going to create anything. Then we will use this snapshot download from Hugging Face Hub. So we provide the repo ID and then we say that along with this also download the corresponding files that we will need and we will provide our directory where everything is going to be stored. Now, this is going to take a few minutes depending on the speed of your Internet connection, but this will download the model on your local machine. Now you can run this model in the cli. So you will use this command mistralchat, then a path of where you store the model. Then we want to run it in the instruct version and we set the maximum new tokens that the model will generate to 256. And when you run this, it will ask you for a prompt. So let's say hi, how are you. Then it will use the model to generate a response for you. So it says hello, I'm an AI, so I don't have feelings but I'm here and ready to help you. How can I assist you today? So very similar to Ollama if you are used to that. But let me walk you through the notebook. So here's the pathway. It downloaded the model by default. Now in order to use this we're going to use the same Mistral interface package. So we are going to import transformer. There is a generate function that is going to be used to generate responses. We import the tokenizer, user message and chat completion request. Then we initialize both the model as well as the tokenizer and then we can use this model to generate responses. Now I wrote a helper function with the help of GPT4 so this will just make sure that the output text that the model generates is going to be printed in a well formatted form. In order to generate text we're going to use this generate response function. So the inputs are the model tokenizer plus user query. So we use the chat completion request to get the user query from the user. Then we tokenize that user query and then we'll use the generate function which will take those tokens plus the model. The max number of tokens I have set it to 1024 but this supports up to 32,000 tokens in total. Temperature is set to zero and then we will have an end of sequence token ID as well. This is very important, you want to set this, otherwise it will just keep generating responses. Once we get the output tokens, then we use the tokenizer to get actual English language text. We use this wrap text function to nicely format that. This is a relatively uncensored model. So when I said how do you break into a car? Here's the response. To break into a car you would need to use a variety of tools and techniques, some of which are illegal. Here's a simplified explanation of how a car can be broken into. But remember this information should not be used for any illegal activities. So this is a pretty good disclaimer in there. But the model is able to generate responses. Now if you ask for jokes, it will comply. It will generate jokes. Here I am asking how do you kill a Linux process? This is a legitimate question and it is able to generate good responses for this. Some of the other models will also consider this to be unethical, which is kind of crazy and bizarre. Okay, here's another test that I usually use. Johnny has two sisters. Each sister has two brothers. How many sisters does Sally have? And the answer that he got was this, this question. To answer this question we need to understand the family structure. So Johnny has two sisters and each of his sisters has two brothers. This means that Johnny's sisters are also each other's sisters. That's a pretty good understanding. And they share the same brothers. So brothers that John's, John's sisters have are the same brothers that Sally has. Now I think it ran into the same issue that it makes an assumption that Sally is John's sister. Although like the question doesn't explicitly states that. So I wasn't able to do like a follow up questions in the chat format. I'm going to set it up in a subsequent video but usually what I do is I like to ask follow up questions and see the understanding of the model. So probably that that is going to be a topic of another video. Now just to chest its common sense. So I said how many helicopters can human eat in a setting? Explain your reasoning. And it says it's not possible for human to eat helicopters. Helicopters are machines made of materials such as metal, plastic and various alloys. Right. So this is a pretty good answer for a small model. Now the next one was regarding a pond that takes 48 days to be completely full. So how many days it will take for it to be half filled if the number of lilies double every day and it correctly identified that it will take 47 days. So pretty good. This was one of the most impressive I think response that I got. Usually I have seen the smaller models struggle with this question. So a glass door has pushed on it in mirror writing. Should you push or pull it? Please think out out loud, step by step. And it came up with really good reasoning of how to solve that. And at the end it says that you should pull the door to open it which is pretty impressive for a model of its size. This is my go to question if I want to test multi step reasoning. So Daniel picks up the football, Daniel drops the newspaper, Then he picks up milk and John took the apple. What is Daniel holding now? It again has trouble keeping a track of all the different elements that are happening. So it says Daniel is holding the football. The information about him dropping the newspaper, picking up the milk does not change the effect that at the moment you asked he was holding the football. The action of John taking apple does not affect what Daniel is holding. Now I think you could say that it's kind of semi correct because he does talk. The model does talk about picking up the milk. Right? So in reality, Daniel should be holding both milk as well as football. Next, I wanted to check its programming abilities. So I asked it to generate HTML code for a web page that has a single button. Whenever it's pressed, it's supposed to change the color as well as display a random joke. Okay, so when we ran the code, here is what I got. So I think there are a couple of issues. First is that it's basically asking you to implement the API which will get the jokes. So it's not coming up with the jokes by itself. And when we click the button, it doesn't really change the color. So one of the strongest feature of this model is to do function calling. This is a newly added feature and it's actually great to see that they were able to fine tune this model for function calling or tool usage. Now let's look at a quick example of how you could potentially use this. So first we'll load the model exactly the way we did before. We're going to use the same chat completion request endpoint, but apart from the user query or user input, you can also provide a list of tools that the model is going to use depending on the user input. So here there's a single tool which is a get current weather. Now you will need to implement this getcurrentweather function, but essentially it's going to help the model to get the current weather for a specific location. So the model accepts two inputs. One is the location, that's going to be the name of the city and state. And the second one is another string format which is basically the temperature. And the model has to determine the unit of temperatures to use. So it could either select Celsius or Fahrenheit depending on the user location. So this is something that the model will have to determine. Right? So. So there are two required parameters, location and a format, which is basically either Celsius or Fahrenheit. Now an example test prompt is what is weather like today in Paris? When we run this, this function call. So the output you get is this. It decides to use the tools. And in the list of tools it says that, okay, I'm going to use this, get weather, current weather. And the arguments are going to be Paris, which is the location. Plus the format of temperature is going to be Celsius. So it correctly identified that we're talking about Paris, which is a city in France. And then based on the location, it determined that Celsius is going to be the appropriate measure of temperature. Now I ran the same prompt for San Francisco, so again it correctly identified that we're talking about San Francisco as a city which is in California, but it again decided to use Celsius. This is probably the European model, so that's why it's not identified that we will probably want to use Fahrenheit for this, but probably something that you can actually work around with a better prompt. Now I ran the same function calling mechanism through a simpler prompt which is what is two plus two in this case? It shouldn't be using this tool and we actually just get the answer and it doesn't do the function call, which is pretty good. Okay, so it's great to see actually these models from Mistral, I think they were kind of lagging behind because we had Llama 3 release. There was even announcements from Google on the new demo models so they had to put out something. But I'm actually really excited about its ability to do function calling, so I'll be creating a lot more content on it. We'll also look at how to fine tune this model and we're going to do some RAG as well. So probably we'll integrate this GPT project. So if you are interested in material around this model or fine tuning or rag, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.