MIXTRAL 8x22B: The BEST MoE Just got Better | RAG and Function Calling
Mixtral 8x22B is Mistral's new mixture-of-experts model, cheaper and faster than its predecessor, with 64k token context and native function-calling. The demo builds a RAG pipeline that routes questions to separate vector stores for Uber and Lyft filings, then shows the model decomposing math problems into multi-step function calls.
Chapters
We have a new best open weights large language model that is able to beat all the existing open weights models not only on the benchmarks but also on the speed of generation. And this is the Instruct fine tune version of the previously released Mixtral 822B from Mistral AI. And it has a very interesting release. They're calling it Cheaper, Better, faster and stronger. The model has support for French, German, Spanish, Italian along with English. And it outperforms all the existing open weights models on all those languages, which is pretty impressive on mathematics and coding. It outperforms all the existing openweights models just like it's base version. We're going to do some of our own testing, but it's going to be very different than the normal tests I do. And the reason is that they claim that it has function calling abilities natively, which is extremely helpful if you are building applications on top of these LLMs. And it has a pretty huge context window of 64,000 tokens. Now the weights are available on hugging face. So for example, you can use Ollama to run this locally if you have enough hardware, but if you're GPU poor, you can actually use the Mistral API to run this model. And in this video we're going to be using the Mistral API. But in another subsequent video I'll show you how to run this locally. So I want to test its function calling abilities and the Llama Index team was able to put together a notebook that tests its capabilities in rag, query routing and function calling. So we're going to be using the notebook to test these abilities. All right, so here's the notebook link is going to be in the video description. So I installed all the required packages. After that we are importing all the required packages that we will need. So we are importing Mistral AI for rag. We also need an embedding model. So we're going to be using the Mistral embedding model. Then we need to provide our own API key. So just go to the platform, get your API key from Mistral AI. Since I'm using Colab, so I actually put this as a secret and then enabled access to that secret here. Okay, so we are going to be using the open Mixtral 822B. The temperature is set to 0.1. We will also use the embedding model from Mistral AI. So both the LLM as well as the embedding model is going to be from Mistral. Now in this case we're going to be looking at some financial data. So we'll use the Uber 2021 and Lyft 2021 EC filings. So for that we need to first download the data and that's how we are downloading it. All right. Once we download the PDF files, then we need to load them. So for that we are using the Simple Directory reader. If you're not familiar with Llama Index, I have a series of videos. I'll put a link to the videos in the description. Llama Index is an amazing package that will help you build RAC pipelines very quickly. Okay, so once we load the files from the directory, then we need to create embeddings. So for this we're going to be using the vector store index from documents. We pass on the documents, it will create a vector store. Essentially it computes embeddings and put them in the vector store. Right now we're creating two separate vector stores. One is for the Uber earning and the second one is for the earning. And the reason is that later on we want to see whether the LLM is able to actually route to these different vector stores, depending on where the question is coming from. So for example, here's a question that we are directly asking on the vector store that we created for Uber. What is the revenue of Uber in 2021? And since we are directly asking a question on this specific vector store, it's able to give us an answer from that vector store. Now here is an example of a question that we are asking on the LIFT specific vector store. So what are the lift investments in 2021? And it was able to generate an answer based on that. But now let's assume that you have multiple vector stores for different types of documents and you want to ask a question and let the system automatically determines where to give get the answer from. Right? So for that, Llama index implements something called Function calling agent using the query engine tool. Alright, so let me explain how this works. So for the query engine tools, you will provide a list of query engine tools. So we this is very similar to a function calling. So you basically you have JSON structure or a list of tools and you need to provide the vector name of the vector store. So that is the lift vector store. Then you need to add metadata. So what is the name of the vector store? So we are calling it lift underscore 10k. Then description. And you want to make sure that the description actually has enough details if the LLM decide to look up this information. So it should be able to deduce whether it belongs to The Lyft one or the Uber one. Okay, so if you have seen my function calling videos, the structure is extremely similar to when you're creating tools for function calling. And in this case we are just creating vector DBs which are going to be called using this query engine tools. Right? So you will provide a list of different vector stores here, then we pass those on to function calling agent worker. So basically this, these are going to be specialized agents which will route your queries to the appropriate vector store. So we need to provide the list of vector stores that we created or a list of engine tools. Right? Then we need to provide the LLM we're going to be using the Mixtral 22B. Then we also want to say whether we want to make parallel calls or not. So in this case we say, okay, we are not interested in parallel calls. Right? Then we create the agent runner object. Right? So basically this is our agent. Now the first question is what is the revenue of Uber in 2021? So the LLM will take this query, run it through this tools, and basically it has to determine which vector store to use. Right? So in this case, since it's related to Uber, so we will hope that it's going to use the Uber vector store. And here's the internal working because we set verbose to true. So it says function calling. It actually automatically or correctly determined that it it needs to use this uber underscore 10k vector store. Right? It's basically goes through a step by step process of basically it's internal reasoning of how it's going to answer your question based on the vector star and it correctly provides us with the answer. Right, so this is kind of an example of function calling, but you are using it for query routing to an appropriate vector stroke. And Mixtral A7B seems to be doing a pretty good job at it. All right, so on the same agent, when we ask regarding investments that were made by Lyft in 2021, it does a similar thing and actually figures out the correct response. So in this case the input, it thinks it's investment and then it looks at the lift 10K. Okay, now we can extend this idea to tool usage as well. Right? So tools are basically function calls or think about external API calls in order to perform different operations. All right, so here's how it works. So similar to the vector store example, there is a special class called function tools and we're going to be using the function calling agent worker, Agent Runner and React agent. Right? So here we are defining multiple different functions so there is an addition multiplication subtraction function. And you also want to make sure that you provide appropriate description for each function. And then we created tools based on those functions. Right? So each of this description will be basically converted to the description of the tool. Now we wrap everything in our function calling agent worker. So we provide a list of tools, then the LLM and then whether we want to allow parallel tool calls or not. Right? So very similar to what we saw for the vector stores, we send our agent. Now, in this case, if we call our agent something like what is 26 into 2 plus 2024, then it kind of has to break it down into multi step, right? And it figures out like what different operations it has to perform. So first and foremost it figured out that it has to perform the multi multiplication operation. And these are the two arguments that it's going to be needing. So A is set to 26, B set to 2, right? And it does the function call and it gets the results of 52, right? Then it says next I'll call the add function with argument a equal to 52 because that's the result of this operation. And then b equal to 2024, right? And it does that addition operation as well. Okay, so this was a quick example of how you can get started with this new Mixtral 822B model if you're running it through the API. If you want to run this locally, the steps are going to remain exactly the same. But in that case, rather than making an API call to Mistral AI, you will simply make an API call to the Ollama server that you are running locally. I will be doing some more testing of this model on some of the my usual prompts, but I wanted to show you what the practical usage of models like these looks like, especially when it comes to rag and function calling. And I'm really thankful to the Llama Index team for them to be able to put such a great notebook so much so quickly. So let me know if you want me to do more of these videos where we look at some of the practical applications of these LLMs, rather than just testing it on a predefined set of prompts which may or may not represent the actual use cases. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
