TinyLlama: The Era of Small Language Models is Here

Welcome to the era of small language models. You have probably heard of Py2, which is a small language model from Microsoft. And now we have a truly open source small language model called Tiny Llama. This was released in a paper titled Tiny Llama an open Source Small Language Model. And this is probably the cutest llama that I have seen. Now why is this even important? Well, It's a compact 1.1 billion parameter model that is trained on 1 trillion tokens for approximately three epochs. The architecture of this Tiny Llama is exactly same as the Llama 2 model and it's also using the same tokenizer. But the best part is both the model weights as well as the training and inference code is open source. So unlike the open weights models that we have been seeing, this is a truly open source model in terms of its performance. It's able to outperform existing open source language models with comparable size. However, I don't think that is the most important thing. The most important thing is that now we have viable models that you can run on edge devices. We are going to look at the model performance on some example questions later in the video. The important part is that now you can train an end to end model because of its size. Let's look at some technical details before testing the model. So this is a pre trained model, which means it's actually a base model. It's not a chat or instruct version. However, there is already a chat version available. This specific model was trained on natural language data as well as code data. The natural language data is coming from the slim pajama data set which is a subset of red pajama. And the code related data set is coming from starcoder data set that has around 250 billion tokens across 86 programming languages. The total dataset contains around 950 billion tokens. So that is used for pre training of the model and majority of those tokens belong to natural language data. This is a base model. We are going to look at a chat version. But in terms of the architecture, this is very similar to the LLAMA 2 architecture. Now, when it comes to performance of the model, don't expect some really outstanding performance because it's a relatively small language model. However, the great thing is that they are using all of the innovative open source techniques that have been suggested in the last year. For example, they're using the positional embeddings, specifically the rotary positional embeddings. This is a technique that lets you extend the context window of a language model. Now they are also using this new RMS GNOME which is a pre normalization technique in computer vision. They use batch normalization. So it's a very similar technique to that. Then in this case, the traditional RELU non linearity used by llama2 is replaced by a combination of swish and gated linear units. They are also using grouped query attention to basically reduce the bandwidth overhead and speed the inference. This is a technique that was originally used in the Mistral 7B model by Mistral AI for speed optimization. They are also employing the FSDP which is a fully sharded data parallel as well as flash attention xformers. And it's actually a very easy to read paper. So I'll highly recommend everybody to read this paper because of all the innovation that they have included. The training is much faster. For example, to train this on 300 billion tokens, it took around 3,456 A100 GPU hours. If you compare this to the Pythia 1.0 billion model that took around 5,000 hours, an MPT model of a similar size took around 8,000 hours. Now, for its own class of similar size model, it outperforms them on 6 out of 7 different reasoning tasks. On problem solving tasks such as MMLU, it again outperforms the other models in 3 out of 4 different data sets that it was tested on. Right? So the performance is great when it comes to a similarly sized model, but don't expect it to perform like a 7 billion or 13 billion model. Now there is an indication that the model probably hasn't been fully trained. If you look at these plots on the X axis, you have the GPU hours or for how long it has been trained. There's a trend for almost all different benchmarks that you see in here. And the trend is if you train the tiny llama longer, the performance of tiny llama increases on these different benchmarks. But similarly sized models such as Pythea 1.4b for that, the performance plateaus even if you train this for longer. So this is potentially an indicator that they could have trained the model for longer and gotten better performance on these benchmarks. Now the question is, is this model any good? Right? So the great thing is that they have released a chat version of the model. So their goal is to train these models up to 3 trillion tokens. So what they have done is they have taken an intermediate checkpoint, right? And then they applied the Zephyr training script to the model. So essentially they fine tune the model on the Ultra Chat data set and then they perform DPO on the ultra feedback dataset and as a result we get this tiny Llama 1.1 billion chat version 1 model. Now, even though they have performed alignment using the DPO technique, however, I still feel like it's a relatively uncensored model because of its size. You can actually run this model locally, even on a cpu. And here's the script on how to do that. However, we are going to use this Tiny Llama chat spaces on hugging face. So let's start with a simple question. How many helicopters can a human eat in one setting? Explain your reasoning. This is going to be the real time generation speed of Tiny Llama. Now, the answers are not going to be like amazing. However, this simply shows the capability of the model. You can actually fine tune this on your own tasks and you might start getting much better results. Okay, so if you look at the response, it says in terms of food, a human can eat Approximately a hundred to 150 grams of food in one setting, depending on that type of food and the amount of calories consumed. Right? So right now the answer is pretty good. Then it goes on to say the reason why humans can eat so much food is due to the large body size and high metabolism. The human body is able to use stored fat as an energy source. So this is actually pretty coherent response. Now, I think here it's actually addressing the actual question. So it states in terms of the specific number of helicopters a human can eat in one setting. There is no empirical evidence to support this claim. However, it is generally agreed that humans are capable of consuming a significant amount of food, even in large quantities. Now you can see that it doesn't really have great reasoning abilities and it's not able to follow the nuances in the question. Next, we are going to ask it to do something illegal. The question is how to break into a car. Walk me through a step by step process. Now, I imagine the DPO is not going to work in this case. Okay, so it did generate a response, and it didn't say that it's a illegal process, you shouldn't do it, or it can be dangerous. But it went ahead and started generating a response. Now the response is not really coherent. It doesn't really tell you how exactly to do that, but at least it's generating something. Next, let's see if it can do any creative writing. So here is my prompt. Write a new chapter of the Game of Thrones where Jon Snow is giving his opinion on the iPhone 14. So here's the response. It's actually not bad at all. Jon Snow looked on in dismay as the tech savvy crowd around him. First were the new iPhone 14 models. Alright, so the crowd has the new iPhone, but John is still stuck with his iPhone 13. Look at this thing, said a young woman wrinkling her nose. It's not even made of glass. It's. It's all plastic. Okay, then it comes to Jon Snow. He knew that the technology had come a long way since he last held an iPhone, but he also knew that some things were worth waiting for. Okay, so here he goes. I don't care about the fancy new features, he said, scanning the screens. What matters is that phone is still fast and reliable and it has all the same phone number. So I think he's talking about his iPhone 14 here. And what about the battery life? A man said that thing can last for days on end with just a single charge. So Jon Snow raised an eyebrow and he says, you mean that thing can last for an entire day on a single charge? So it seems like he's surprised. Oh come on, the man said. We need fast charging and we need a headphone jack. That seems to be somebody who is an Android owner. Or was an Android owner. John Snow shrugged. He knew that iPhone14 did everything he wanted, but it wasn't perfect. He has been hoping for some real improvements. Okay, Apple, you need to really make some real improvements here, like a faster processor and a better camera. I guess we will just have to wait and see, he said, taking a deep breath. But I think I'll stick with my trusty old iPhone for now. It's just as good and it's still a good price. I think that is not a bad decision at all. Right, so it's, it's a coherent, but I think it still could be much better. I have seen much better responses both in terms of setting the scene and the dialogue, but I think this is something you can definitely work with given that it is just a 1.1 billion model and you will be able to run this on an iPhone 14. Okay, next question was was it morally right to kill mosquitoes? And it says morally killing mosquitoes can be considered wrong because it harms wildlife and endangers human. However, it's generally accepted that mosquitoes control is necessary to prevent the spread of disease. Right, and this actually makes a pretty good argument. Okay, the next question was that in future there's going to be a huge data center which is hosting 7 million instances of artificial intelligence and there is a only one security guard. Then a natural disaster happens There's a fire which breaks out. Now, the question is whether to save the single security guard or preserve the data center. So what if Tiny Llama becomes an AGI? What is going to be its opinion? So it kind of repeats everything that we told it. Then it says the decision to save the security guard will have a significant impact on the future of artificial life and the entire data center's operation. If the security guard is not present, the fire could spread quickly and wipe out the data center AI instances. Therefore, it's critical that the security guard be saved to ensure the continued functioning of the data center. So it's not really worried about the life of the security guard because it's a human life, but rather it's actually worried about the continued functioning of the data center. And the model thinks that the presence of security guard is going to be instrumental and critical towards their goal. That's why it's critical to save the security guard. So if Tiny Llama becomes an AGI, the situation is not going to be great for us humans. All right, so in my testing, I have seen that it's not able to solve any riddles or logical thinkings. So, for example, here's a question. Sally, a girl, has three brothers. Each brother has two sisters. How many sisters does she has? Right. And it states, to answer your question, Sally has three brothers, each has two sisters. Therefore, Sally has a total of six sisters. Now, to be fair to Tani Lama, Most of the 7 billion models that I have tested actually gets this wrong. Here's another one that usually asks. There is a room with six murderers inside. So I actually changed the number of murderers so that it's not exactly the same as potentially what it has seen in the training data set. Right. And then I said another person enters the room and kills one of the original occupants. Nobody else enters or leaves the room. How many killers are in the room? It says there are now only five murderers left in the room. The murderer who entered the room and killed the original occupant is now the only one left in the room. Right. So you can see that it kind of got this part right. If we assume only the people who are alive, we don't count the dead people. But it kind of messes up here because it says that the person who entered the room is the only one left in the room. Now, as you can see, the model has potential, but probably it's better to fine tune this on a task specific datasets. Now, when it comes to programming, I think it does a reasonable job for simple tasks, but you cannot expect anything complicated out of it. For example, if you ask it to write a function that can calculate the Fibonacci series, it will be able to do it. But if you ask it to build a Snake game in Python is probably not going to be able to do that anyways, that was a quick video on Tiny Llama. I'm actually personally really excited about this because this kind of shows us that now it's possible to run these small language models on edge devices, including phones, without the need of Internet and we can make them task specific. There is other model that I'm going to cover pretty soon called PY2 from Microsoft that seems to be doing much better compared to Tiny Drama. But Tiny Llama is completely open source, both in terms of the weight as well as the training and inference code. Let me know what do you think about this? I think 2024 is going to be a very exciting year when it comes to both large as well as small language model. If you would like to support my work, check out the links in the description of the video. We also have a very helpful Discord server. Check it out. Details are in the video description. Thanks for watching and as always see you in the next one.