Just Large Enough - Mistral released a BEAST

If you recall, last year an internal memo from Google researchers made a lot of headlines. It was titled we have no mote and neither does OpenAI and it's turning out to be very true with the latest developments in open source. Yesterday the Meta team released llama 3.1 and the 405B model was one of the state of the art, both for proprietary as well as open with models. But today this was dethroned by a release from Mistral. They just released a new model called Mistral Large 2 which is outperforming 405B on a number of very important benchmarks. Previously Mistral was keeping the Mistral Large model as a proprietary model behind their API, but now they are opening the weights of the model and you can also use it for commercial purposes when you get a license from them. According to the blog post, Mistral Large 2 is significantly more capable in code generation and mathematics and reasoning. It also provides a much stronger multilingual support and advanced function calling capabilities. Just like the 405B, it has a context window of 128,000 tokens, so it supports a much larger context window compared to some of the earlier releases. And it's multilingual. It has support for French, German, Spanish, Italian, Portuguese, Arabic, Hindi, Russian, Chinese, Japanese and Korean. It's great to see that these model creators are now adding a lot of different languages and one of their focus has been to add a lot more programming languages so they Support up to 80 plus coding languages. Here's a quick look at some of the benchmarks. I'll put a link to this in the video description, but the Mistral Latch 2 is on par with GPT4O, and in most of the benchmarks it's actually performed better than the 405B, which is pretty significant given that it's just 1/3 of the size. So this model is 123 billion parameters compared to the 405 billion parameters from llama 3.1. This now it is under Mistral Research license, but if you want to use this for commercial purposes you will need to contact them and get a commercial license. So in terms of size it's just larger than the 70 billion offering. But in terms of performance on code generation it's better than the 405B as well as on math it's again better than 405B. So it gives you a really sweet spot when it comes to a model performance versus the number of parameters. So you can actually deploy this on a single H100 node compared to like 2 or 3 nodes for 405B. Now a couple of highlights from this release is that one of the key focus areas during training was to minimize models tendency to hallucinate or generate plausible sounding but factually incorrect or irrelevant information. This has been one of the biggest issues with large language models, but seems like they have paid very close attention to it and have gathered training data where the model hallucination has been reduced substantially. And that's why this new model is trained to acknowledge when it cannot find solutions or doesn't have sufficient information to provide a confident answer. Another improvement is instruction following and alignment. According to them, this model is particularly better at following precise instructions, handling long multi turn conversations. So this is going to be pretty significant especially for real world applications. Smaller models tend to suffer in performance when it comes to long multi term conversations, so we'll need to test it out, but on benchmarks it seems to be pretty good. Now again, I personally don't really believe any of these benchmarks. You need to do your own vibe check for your own application. Another capability is tool use and function calling and these are the two main things that you will actually use in real world applications. They have enhanced function calling capability. That means that you can use this model for agent orchestration and it has enhanced retrieval skills. RAC is one of the most practical applications when it comes to businesses and enterprise today, so I think it's really good that they are specifically focusing on retrieval skills. The other company that is doing a great job at IT is Cohere who don't really get a lot of credit, but they are probably creating one of the best rag focused LLM systems. Now this can do both parallel and sequential function calls, so we are going to look at a couple of examples. I will show you a code of how to use this for function calling and on a number of benchmarks that are dedicated to function calling. This seems to even outperform GPT4 and Cloud 3.5 Sonnet, which is pretty substantial. Okay, so this is available both through their API as well as through their chat interface. So I'm going to be probably creating more dedicated videos on testing the model, but I'll put a link to this chat interface where you can experience the model yourself and see if how good this is and whether you can use it in your own applications or not. But I'm going to show you a code where we're going to test the function calling capabilities. But before that, just like Llama, they are partnering with a number of different companies such as Google, Microsoft, Amazon Bedrock and IBM Watson, and they are making these models available through these different API providers. Mistral also has their own platform which you can use through their API. And here are all the different models that are going to be hosted there. So they're going to be hosting the Mistral Large, which is I think this V2 model. And there's going to be Mistral Nemo and then there are there's one code specific Coastal and an embedding model. So these are the ones that Mistral is going to be hosting. I think they are deprecating some of the previous models that were available through their platform. Another thing to keep in mind if you decide to use the Mistral Large model through their platform is that the input price is very similar to what other API providers are providing for 405b, but the output seems to be a bit more expensive compared to 405B if you are looking at Fireworks AI. So here are the relative prices for 405P. You can see that Fireworks AI provides $3 per million tokens for both input and output, but for the rest of the API providers it's a lot more expensive. Okay, next we're going to look at its function calling capabilities and I'll show you how to integrate this in your own code. Before looking at the function calling capabilities, I want to highlight one very important thing when it comes to benchmarks and how these models are tested. So they added this small disclaimer. All models were benchmarked through the same evaluation pipeline except for the paper row and they are talking about this 405B. So the 405B llama 3.1 paper are actually the results that were reported by the LLAMA team, but they also measured the performance on their own benchmark on the same data sets. Now you can see there is difference of performance when it comes to measured versus paper. And this is most probably coming from the difference in the prompt that were used in the paper versus they're using here. And in some cases the difference is pretty substantial. If you look at C Sharp, PHP and even Java, most of these models still need customized prompt engineering to get the best performance. So if you use the same prompt for all the models, that is probably not a fair comparison because there is still a lot of prompt engineering that goes into especially like when it comes to the smaller models. The smaller models need a lot more few short examples compared to the bigger models. So again, even if you are seeing the reports on benchmarks, take them with a grain of salt. That said, let's look at how you can use this for function calling. Okay, so function calling or tool Usage enables these LLMs to interact with outside world. Now before showing you the code example, let me explain the process. So here's how it works. LLM analyzes the user query and then the LLM will have access to a number of different tools. For example, to do mathematics it can use a math or calculator tool. If it needs certain information from web, it can use a web search tool or get its stock information, and so on and so forth. So in the first step the LLM has to determine whether it needs to use a tool or not. If it doesn't need a tool, it will just generate a response. So it's going to be a two step process. But if it decides to use a tool, then based on the available list, it needs to pick an appropriate tool for the task and then generate the inputs to that specific tool. Now the LLM does not really have any ability to actually execute the call to the tool or function. So you as a user will need to write custom code to execute that call or use that tool and then you pass on the response from the tool to the LLM to generate the final response. And that is going to be shown to the user. So in this case we are going to just follow all those steps. So first we need to install the Mistral AI Python client. Mistral has their own Python client, so you can use transformers package, but in order to use the API we are going to use the Mistral AI Python client and we also need pandas for data frame generation. So in this case we are generating dummy data set which is basically some transactions which contains transaction id, customer id, what was the payment amount when the payment was made and what's the status. Right, so you can have a quick look at how this look data looks like. So there is a customer id, then a transaction ID and the other fields that we are interested in. So we want to be able to analyze this data and for that we will create a couple of different functions. So one is retrieve payment status, the the other one is retrieve payment date. So These are the two tools or functions that we want Mistral Large V2 to use whenever it's interacting with this data set. Now in order for the Mistral model to understand the functions, we need to outline the function specifications with a JSON schema. So it's very similar to what OpenAI uses. And specifically we need to describe the type of function name, function description, function parameter and the required parameter to the function. So here is A description or the tool usage description that we need to provide. So basically it's a list of dictionaries. First is the type which is going to be function. Then for both of those functions we need to provide the names small description. The LLM is going to use this description when it's picked picking a tool. So you need to make sure to provide enough details in here. Then what are going to be the inputs to this function as well as what is the required output. And we can look here. So we need to provide the data and the transaction id. In our case, the transaction ID is the required field so it can have access to the data object. But the model has to provide this transaction ID as an input to this function. Okay, so once we create our tool set, then we create another dictionary where the key is going to be name of the functions. So since there are two functions, we have two different keys and then we use the functions tools partial. This is an open, this is a Python class that will help us pick the tool and execute it. Alright, so here's an example query. The user is asking about the status of my transaction. And here is a transaction number. Okay, next we need to set up our LLM. So we will use the Mistral AI Python client. We need to set up our API key. In my case, that is coming from the Secrets within my Google Colab notebook. Or you can set this as your own environment variable. The model that we're going to be using is Mr. Large latest, and this is coming from here. So here's the API endpoint that we're going to be using. So we create a Python client. Then we are using this in chat mode, so it's going to keep history of the previous conversations. We provide our model name, we provide our initial user input. So this was the first conversation. Then we also provide the list of tools that we defined in here. So there are two tools and then we ask the model to automatically select a tool for a specific task. Now the user query is asking about the status of a transaction. So here you can see the response from the model. So this is the first message that is basically the user input and the content is the actual user query. Now since it's a user query, so there are no tool selection whatsoever, but the model generates a response which says that I need to use a specific tool. Then it selects the tool that it's supposed to use and that is going to be retrieval, payment status and the argument is going to be the transaction id. And from the prompt it was able to retrieve that Transaction id. So we're here at this step. It was able to pick the appropriate tool, but we now need to execute that function call or use that tool to generate response for the LLM. And as I said before, you will have to do that. So you pick the tool from here like what tool was selected by the model. And for that we're going to look at the function name. This is the function name. Here is the input that the LLM thinks that we need to provide. Now we are going to pick that tool or function from the dictionary that we define and then pass on the parameters. So here's the dictionary that we initially defined. The input is going to be retriever payment status and then the input to that function are going to be two things. So one is going to be the transaction ID and the data frame which we get here. And it's supposed to generate a JSON response and based on that response the result is paid. So this is the status response from the function call. Now what we need to do is we need to get this back into the loop of the LLM. So we're going to create another chat and in this case we provide the function name, what the function results were and what tool was used to generate that response. Right? So we append this to the messages. Now the model is going to see this extra message that we had to create and based on this chat history it's going to generate the final response. Right? So if you pass on the model plus the message history, then it sees that the user is asking about the status of transaction and it already figured out that I need to use this tool with this transaction id and the tool response is that the status was paid. Right. So then it puts everything together and generate the final response which is your transaction is marked as paid. Okay, so this was a quick example of how you can do tool usage with Mistral Large V2. It's, it's a very capable model. You can potentially add a lot more tools. Here I will actually create a video in testing its capability to do multi level tool calls. So we'll see if it's nested tool usage, whether it can do that or not. So if you're interested in that, make sure to subscribe to the channel but closing thoughts. So it seems like the size of these LLMs do not matter as much as we are thinking. Thinking it's probably more about the quality of data and the compute scaling because we have a relatively small model compared to 405B and it's getting better results in that than that model. Even the Llama 3.170 build model is, I think very close to 405B. Now you do have to give credit to companies like OpenAI or Anthropic because they are able to release multimodal models, or at least they are providing multimodal models through their API endpoints. When it comes to open source, we haven't really seen a very capable multimodal model yet, but hopefully that will change pretty soon. And when it comes to the ecosystem, I think there is definitely space for both for close wait and open weight models. But it's really great to see that open weight is not only catching up, but seems to be ahead because we had two releases which are state of the art in the last two days. I don't know what Thursday is going to bring. I hope you found this video useful. Thanks for watching and as always, see you in the next one.