MPT-7B: Beats GPT-4 to 65K+ Tokens

A few days ago, the company called Mosaic ML released a group of 7 billion parameter models called MPT7B. And this is probably one of the most significant development in achieving truly open source large language models. And one of the model offers a feature that even OpenAI with GPT4 has not been able to deliver. In this video we will look at this group of models, why they're significant and then I'll show you how you can play around with them. We will also look at features which set these models apart from the rest of open source models as well as G.MPT7B is a family of transformer based models that were trained with 1 billion tokens of text and code. The claim to fame is that it matches the quality of llama 7B. It's a completely new architecture that you can use for commercial purposes. The Training took about 10 days with a total cost of $200,000, which is just probably a fraction of what Meta or Facebook had to spend on their llama. Mosaic ML is releasing a total of four different models and we are going to look at those in details. Now you might be thinking there are a whole bunch of open source models already available, so why is this one significant? There are quite a few reasons. First, it's a completely different architecture from llama7b, so you are not constrained by the licensing issues related to that model. So here are some more reasons. First, it's licensed for commercial use, so unlike nama, you can use it for commercial purposes. Then it's trained on a much larger data set compared to the other open source models. So in this case it is trained on 1 trillion tokens similar to llama. Whereas for comparison the open assistant model is using 300 billion tokens for training or 800 billion tokens used by Stablelm. And I think the most important feature is prepare to handle extremely long input tokens. So it can actually go up to 84,000 tokens for which is just mind blowing. We will talk about this in more detail. Now it's optimized for fast training and inference and then equipped with highly efficient open source training code. Basically, Mosaic ML gives you the ability to fine tune these models just by providing your data and use their platform. So in terms of the models, they are releasing four different models. The first one is MPT 7B base model. MPT stands for mosaic Pre Trained Transformer and the base model is very similar to llama, so it doesn't do much, but it's trained on 1 trillion token and you can fine tune it for other tasks. Which they have done in the subsequent models. And the next model which is probably the most significant out of the bunch of is the storywriter 65k plus model. I want to come back to this at the end because this is significant. The next one is the instructor version which is a model trained on a data set which is that is basically a combination of The Databricks Dall E15K dataset and Anthropic's helpful and harmless data set. This is very similar to other instruction following models and this is also openly available. You can use it for commercial purposes. Now the last one is a chat model which is can be used for dialogue generation and it's a fine tuned version based on a combination of data set. For example, There is shared GPT data set that was used for Acuna HC3 Alpaca Helpful and harmless as well as Evolve instruct data set that was used for WizardLM. Now this is the only model that you cannot use for commercial purposes because of the data set that we use for training it coming back to the story writer model. So this is a model designed to read and write stories for super long context lengths and this is actually significant part of the whole conversation. So it was built by fine tuning the 7B MPT model with a context length of 65,000 thousand tokens, which is crazy. And it can go to as long as 84,000 tokens. Now just to give you an idea of how significant this is, let's look at this image. So today the model that has the highest token length or context window is GPT4 and that is currently available. It has a context window of around 8,000 tokens. In OpenAI claims to have another version which is 32,000 tokens, but we have not seen that publicly yet. Now in comparison, this Mosaic model has a context window of over 55,000 tokens. Now what does it mean? You can actually literally feed in a whole bunch of books and it will not forget the information. And you can converse with this as an example. They show that they provided the whole the Great Gatsby book in a single prompt which took around 67,873. 3 tokens. And then it was asked to write an epilogue at the end. It's, it's amazing that it's actually able to process around 67,000 tokens and remember the context of of them. Right? And then it produced a beautiful epilogue based on the input text that was provided to it. Now this is literally the game changer when it comes to large language models because one of the bottleneck of the large language model was the number of tokens that you can feed in and how many tokens it can actually remember. But with this model you don't really need to worry about the context window at all. And this is actually possible due to this paper titled Train short testing. Long attention with linear biases enables input length extrapolation. You know, in this work they showed that you can train a model with a shorter context length sequence, but you can then extrapolate it to a much longer context length sequences without significant impact on computation as well as memory usage. You know, these are all great things, but are these models any good in actual testing? You know, they claim that their base model is as good as Laven's Llama 7B. It actually outperforms other open source models in the range of 7 to 20 billion parameter models. So in this table you can actually see the comparison. So on top you have the MPT7B, this is the base model, so not fine tuned. And then There is another llama 7B model. You also have the stable LM7B then path here. And I think there is a GPT Neo X20B along with Cerebras 7B and 13B. Based on the overall results, you see that the MPT 7B is very comparable to Llama 7B because in most of the benchmark data sets their performance is very similar to each other and both of them outperforms the rest of the models easily. Now keep in mind, we are simply talking about the base models. So it's not instruction fine tune or these are not fine tuned for chat. Now in terms of the training dataset is actually collected from 10 different open source data sets and combined together to train their base MPT7B model. Now to understand how easy it is to fine tune your own model, let's look at their training setup. So they're using their own Mosaic ML platform for training these models. Now, in order to train their base model it took them around 440 A100 GPUs with a total cost of $200,000 and it took around 10 days in total. However, fine tuning the subsequent models is a whole different story. To fine tune the the MPT7B instruct model, it took around three hours in total, actually around two hours with a cost of only $37. For the chat version it's around 15 hours with a cost of almost $600 only. And for the story writer, the total cost on top of this 200,000 base model is around $4,000. Only which took around two days. Now it's also interesting to see the number of tokens that you need to find use these models. So the initial model, the base model is trained with 1 trillion tokens. But to get a instruct model on top of it, on a given data set, you just need around 10 million tokens, which is just a small fraction of the initial data set. For the chat it's around 86 million tokens. And then the story writer is the most significant one, around 5 billion tokens. Now keep this, keep in mind that for the base model, instruct model and chat model, the context length is around 2000 2048, whereas only for the story writer it's around 65,000 token length. And that's why you need a much larger number of tokens to actually train the model or fine tune them. This table is significant because it gives you a sense of what type of cost you're looking at and what type of or amount of data you will be needing to fine tune these models on your own data set. I'm actually very glad that mosaicML actually added this part. Now let's look at how you can interact with these models. MosaicML has a hugging face repo where you can actually go and interact with these models. So there are currently two spaces running. One is for the instruct version and the other one is for the chat model. The story writer model. You cannot run this on consumer hardware yet because the memory requirement is pretty high at the moment. I think there is currently some work being done on the 4 bit quantized version, but I think it's not available yet. Now these models are compatible with the transformer library. So let's say if you go to the chat version, you can actually see how you can use this on your local machine or through a. A Google Collab is compatible with the transformer package from hugging face. So you simply pass on the type of model that you want. Then you say trust remote code is equal to true, Right? And this is how you're going to be configuring the model. In order to get the prediction from the model, you need to run it through a tokenizer. In this case, the tokenizer that they use is from Eleutherai, which is based on the GPT Nux 20B model. They have provided two different spaces. One is for the instruction instruct version, the other one is for the chat version. Now keep in mind, both of them are 7 billion parameter models. So I won't expect too much from these two specific models because even the context length for these two models is limited, so that is around 2048 tokens. I'd be more interested in playing with the story writer model. The contest version is available pretty soon, but let's ask it some very basic questions. So I'm going to be primarily testing the instruct version because the other one is taking too long. Now there are some advanced options as well that you can play around, but I'm going to keep everything to the defaults. So let's ask it what is five times seven? Okay, that gives me a pretty good answer. And now they have a few questions here. You can actually test them. So for example, here's a convert the following to a single line of JSON, right? And if you submit, it's actually able to do that pretty easily. So this model can do easy stuff like this without any issue. Let's try a little a more challenging one. So how come time flies like an arrow but fruit flies like a banana? Right. So the second part is talking about the insects. Fruit flies. Okay, let's see what it can come up with. All right, so the it says the question was asked by Albert Einstein in his famous quote about the relativity and how it relates to our perception of time passing relative to events happening around us. Yeah, I think it completely missed the point. Let me submit it and see. No, Okay. I don't think. I didn't understand the proverb at all. Yeah, it's going back to Albert Einstein for some reason. Although the. The generation speed for this is pretty good. All right, so here's another one. The school bus passed the racer race car because it was driving so quickly in the previous sentence. Tell me what was driving so quickly and explain the reason. You had reasoning. Right. Then it says the response is actually really nice. It says in this example, the school bus is the thing which was passing by or moving faster than the other object, which is the race car. So that's pretty good. And it says this is due to the fact that the buses are typically slower moving vehicles compared to the cars. But in this particular instance, the bus was able to move more quickly through the scene as it drove past the sky drop. There are many possible reasons for why this might be the case. Perhaps the bus drivers was rushing to get the students to school on time. Or maybe there were heavily traffic delays along the route of the bus journey. Right. So it actually ables to understand or deduct what was being mentioned in the first sentence. So that is pretty good. Now for a small model like this, it's actually a pretty Good response. So here is another test. Tell me why 42 is important, an important number. Let's see what it comes up with. Okay, so 42 is an important number because it is the answer to life, universe and everything. It's also the number of spaceships in Star Trek. I think it's. Yeah, I don't think it knows about the Hitchhiker's Guide to the Galaxy. Or maybe it simply extract the answer but didn't really explain the rationale behind it. Now let's look at a couple of other prompts before we wrap it up. So drop an apology email to a customer who experienced a delay in their order and provide reassurance that the issue has been resolved. So let's see if it can come up with a nice email. So it says, hi, I wanted to reach out personally after learning of your recent experience with a company regarding delivery delays on one or more items from your most recent purchases. Okay, so it says we have identified the root cause of these issues which was related to supply chain disruption across multiple industries due to the COVID pandemic impacts. However, rest assured that we are working diligently towards resolving the challenges that so they do not impact future power displayed by yourself or other customers. So this is actually a pretty nicely worded email. It gives a reason and then like it reassures that it's not going to happen. Hopefully not going to happen in the future. So it's the thing, it's. It's a. It's a pretty decent event. Now this model is supposed to be trained on code as well, so let's look at a quick example. Write a Python function to write a file to an S3 bucket using the Boto library. Let's click enter and let's see what it can come up with. Now usually this is in markdown, so that's the comment in the code, but let me quickly look at the code itself. So upload client, right? Yeah, I think it's it. It will work. This will work. Although I would like it to be well formatted, but it's a demo, so maybe the hugging face platform itself is actually not running it correctly. Okay, so in conclusion, these are actually pretty powerful models for its size, right? You cannot be expecting the same quality like Vukuni or ChatGPT, but still, I think these are decent models. I'm excited about the story writer model, which is 65,000 plus tokens. That would be a game changer when it's actually when we are able to run it on our local hardware. So keep an eye out for that video and I am going to be making a video on that when it's available. Hopefully we can run that in the Ooga Booga text generation web ui. We covered a lot in this video. If you have any questions or comments, put them in the comment section below. Things are moving very fast and seems like the open source will surpass the closed source models very quickly.