The Real Progress of Local LLMs

2023 was an amazing year for large language models and for open LLMs in particular. For this last video of the year, I want to do a recap of all the open foundational LLMs and see how much progress we have made in a span of a single year. During 2023 a large number of open LLMs were released, so it's hard to keep track of all of them. But we're going to be looking at open foundational LLMs that had at least their model weights released. Most of them are going to be decoder only transformer architecture and we're going to be specifically focusing on language first models, so we're not going to be looking at coding focused LLMs. For most of these LLMs, the major difference was the way they were trained, so the training data as well as some small tweaks in terms of the algorithms. Before looking at the open LLMs, let's talk about two major companies that contributed the most to open LLMs in my opinion. The first one is Meta. They contributed a lot by releasing the weights of Llama One and Llama Two series of models and most of the early models like Vocunia were just fine tunes of the LLAMA architecture. The second one in my opinion is OpenAI. This might be a bit controversial, but I think they had the major impact because with the help of ChatGPT and GPT4 the open source community was able to generate a large amount of high quality data that was used to train the initial iterations of the model. For example, models like GPT4 all Vicunia were fine tuned on the data that was generated using GPT4 and ChatGPT. There was even share GPT where people share their chats from ChatGPT and that actually played an important role at least in the beginning of the year. This video is inspired from the post on Hugging Face. The title of the blog post is 2023 year of open LLMs. I'll put a link to this post in the description and would recommend everybody to check it out. So let's start with a timeline of what happened with the last year. The first major release of open source large language model was the LLAMA model series family from Meta AI and these model actually showed that you can train smaller models on more data for longer periods to get more enhanced performance. For llama 1, Meta actually released a series of different models. The smaller model was around 6 billion parameter, the largest was 65 billion parameter and they were trained on 1.4 and 1 trillion tokens respectively. These were the first open large language model that showed performance better than GPT3. The models were great. However they were released under a non commercial license and the second biggest limitation of these model was the context window. So they had a context window of only 2048 tokens. In April we had the second major release. So these were the Pythia models from ULUTHA AI, which is the nonprofit lab sponsored by Stability AI. Similar to LLAMA series, they had a number of different models released. The smallest one was just 14 million parameters and the biggest one was 12 billion parameters. Unlike the LLAMA series, Patia models were trained on publicly available data sets. However, again they were under a non permissive license so you couldn't really use them for commercial purposes. And they also had a pretty limited context window. In May of 2023 we had the release of MPT models from Mosaic ML. They released information regarding the training data, so that was really helpful. Initially they released a 7 billion parameter model which was followed by a subsequent release of 30 billion parameter model in June. However, one of the 7 billion parameter model had a context window of 65,000 tokens. This was called Storyteller and this was the first open source model which showed that you can achieve a significantly huge context window or working memory. Both of these versions were trained with 1 trillion tokens and they contained English as well as programming languages. But the best part was that these were one of the first model releases with a very permissive commercial license. This was followed by the release of Falcon series of models in June. This was significant because this was the first time that we saw a company outside of the USA releasing a foundational large language model. So initially they released a 7 billion model as well as a 40 billion parameter models. Both of them had a limited context window of 2048 tokens and they were trained on a huge data set which ranged from 1 to 1.5 trillion tokens. If you notice, these models were actually releasing information regarding their training data. So this was pretty helpful for people to actually take these models and fine tune them further. Later in the year they also released a huge 180 billion parameter model, but it also suffered from relatively small context window of just 2048 tokens. These models were initially released under only a research license, but later on the Falcon team was kind enough to change it to a more permissive Apache 2.0. So that means you can use these for commercial purposes. In the same month Salesforce released their own foundational model called Xgen. This was another 7 billion parameter model and it utilized 1.5 trillion tokens consisting of both natural language as well as code. Now this was the first model that employed a data scheduling system. So basically you did not had to train the model in a single step, but you could introduce more data in multi stage fashion. This was a very innovative idea, but the X gen model family didn't get too much traction. So we are now into July and Meta had another huge release. So they released the Llama 2 family of models and again this time there were multiple models ranging from 7 billion up to 70 billion parameters. They also expanded the training data. So now the models were trained up to 2 trillion tokens. However, Meta did not really release information about the training data. It only stated that it's publicly available data set. But they provided a lot more information about their training process. The training process involved an extensive procedure of fine tuning based on human preferences using RLHF. So this was very similar to what OpenAI is doing with ChatGPT as well as GPT4. For this release they also extended the context window or working memory. So now we had a working memory of 4096 tokens. But the best part was that this was released under a more permissive community license and you can use this for commercial purposes. Just after two months after the release of Llama 2 family, we had a surprising release from a new kid on the block. This was Mistral 7B from Mistral AI. This was a second foundational model that was released outside of USA. Now they did not release any information regarding their training set. The only information we got was that the data was extracted from Open web. So starting with llama 2 we saw this trend that nobody's actually releasing information regarding the training data. When it comes to foundational models, that is true both for proprietary models like OpenAI's Chat GPT, GPT4, Oranthropics, Cloud1, Cloud2 and now it's also true for these open LLMs as well. But they released some information regarding the architectural choices. So for example Mistral was using Group Query Attention along with sliding window and a new tokenizer and we had a release of both the base as well as instruct version of the model. And the great thing was that this was also released under Apache 2.0 so you could use it for commercial purposes. Now along with this, Mistral7B also had a relatively bigger context window. So the initial version had 8,000 tokens context window or working memory which was huge because now you can use this for more practical applications. Also in September we had the first major model released from China and this one was from Alibaba. This model family was called Queen. They also released multiple models ranging from 7 billion up to 70 billion. And this was a bilingual model, trained both on English as well as Chinese and was trained on 2.4 trillion tokens. Another great thing about the Queen 70 billion model was that it had a much larger context window of 32,000 tokens and it was the best performing model on the open LLM leaderboard. In November, we also had another release from China and this was the YI model family. There were two models. One was the 6 billion parameter model and the second one was 34 billion parameter model. These were again bilingual models trained both on English and Chinese and now they were trained up to 3 trillion tokens. The training sides of these models actually started increasing and there was not much focus on increasing the actual size of the model, which is pretty interesting. Another differentiating factor was the context window. So the YI model series family had two different versions which could support up to 200,000 tokens in its context window. It's on par with Claude 2 when it comes to the context window size. Now, just like Claude or GPT4 with 128,000 tokens, this 200,000 token version also suffered from lost in the middle phenomena when it comes to these huge context window models. But nevertheless, they showed that it's possible to extend the context window of these relatively smaller models to something like 200,000 tokens, which in itself is a huge achievement. Okay, so this brings us to December. In December, Mistral AI released their mixture of experts or MOE model, which they were calling Mixtral 87B. They had three different models releases, so one was tiny, which is the second version of their 7 billion parameter model. The small was the MOE version. And there is even another model that is called mix 12 medium, but it's only available through their API. So nobody has access to the weights of the Mixtral medium model. Now, when it comes to training data, Mistral AI is very secretive about it. Mistral AI does not release any information regarding the size or the source of the data. The only information we got was that it was extracted from the open web, but it now supports up to 32,000 tokens. So if you remember, llama1 had a context window of only 2048 tokens and now we are already up to 32,000 tokens. In case of Yi, we already have a 200,000 token version as well. So this is a huge improvement. However, based on the benchmarks, the Mixtral 87B is on par with GPT 3.5, which is absolutely mind blowing because in the beginning of the year there was not even a single open source large language model. And now we have an open LLM that is on par with GPT 3.5 or chat GPT, at least on the benchmarks. So you can run this model on your own servers. There is even a version that you can run on a consumer grade GPU with 24 gigabytes of VRAM. And again, Mistral AI released this under Apache 2.0, so you can actually use this for commercial purposes. Okay, so in December we also had the release of solar 8.7B from upstage AI. This is out of Korea. This is 11 billion parameter model which is merging two different models using a new technique called depth upscaling. Although this is not a foundation model by definition, but I really like the idea and that's why I put it in the list. So essentially what they have done is that they took a base Llama 2 model, replaced the weights with the Mistral 7B weights, and then stack two of them together on each other to create a new model that has 48 different layers. Currently, the top 10 models on the leaderboards are just fine tunes of this architecture. So even though it's another foundational model, the idea seems to be working pretty good and people are able to create very powerful fine tunes based on this training data. They're using a combination of both open source publicly available data set as well as some of their proprietary data set. That was a quick summary of all the Foundation Open LLMs that were released during 2023. Now there were some amazing fine tuned models, for example the Vicuna model, then there was this ARCA model. There are different models like Dolphin, the Nose, Hermes as well as Zephyr, but I just wanted to cover only the open foundation models. Now just to recap, in January 2023 we did not had even a single open large language model comparable to anything that the proprietary models have. But by the end of the year we have multiple models that are on par with at least chatgpt when it comes to performance on leaderboards. I'm actually really excited for 2024. I think the focus is going to be on smaller models that are trained on extremely high quality data. I hope you found this recap of 2023 useful. Thanks for watching and have a great new year. See you next year.