LLAMA-3.1 405B: Open Source AI Is the Path Forward

It took only 16 months for the open source to catch up to GPT4 level model. Meta just released the Llama 3.1 family of models which are not only the best open weight models out there, but the bigger 405B version is probably the best model available today. Both among the open and closed weight models, 405B is the highly anticipated model from Meta, but they also refreshed their smaller 70 and 8 billion models. And personally I'm more excited about these smaller models because you can actually run them on your local machine. For 405B, you need to be GPU rich. In this video we are going to be talking about different capabilities of these models. How do they compare to other models out there where you can run them and if you decide to run them locally, how much RAM you are going to need. As well as we are going to look at the new Agentix system from Llama along with an open letter from Mark Zuckerberg and a hint of a multimodal llama. We have a lot to cover so let's get started. Before looking at how it compares with other models, let's look at some technical details which are I think a lot more interesting. So I'm going to be relying on this summary from Aston who is a research scientist on the LLAMA team at Meta Generative AI. This new model family has a huge context window. For the previous version of 8 and 70 billion model the context render was only 8000 tokens but now they were able to extend it to 128,000 tokens which makes it a lot more useful and it's on par with the context window of GPT4 models. Now the quality of training data matters a lot. So compared to the previous LLAMA models, they have enhanced the pre processing and curation pipeline for pre training data as well as the quality assurance and filtering methods for post training data. So they have paid a lot of attention to the training data and that seems to be the main reason behind this performance improvement. The architecture is very similar to the old models. Now one of the use cases that they highlight for the bigger 405B is synthetic data generation for fine tuning of smaller models. And how big was this data? So the pre training Data is about 16 trillion tokens and it was trained over 16,000 H100 GPU clusters which is pretty crazy to think about, but the inference is going to be a lot more compute efficient. So they say they to support large scale production inference for the 405B model we quantized it from 16 bits to 8 bits, reducing compute requirements and enabling the model to run on a single server node. Later in the video I'm going to tell you how much GPU vram you actually need in order to run this model. But spoiler alert, you need a lot. And they leverage the 405B model to improve post training quality of 70 and 8 billion model. So it seems like they these models are distilled versions of P and that's why you see a substantial improvement in performance. And we're going to look at that in a bit. Now in the post training they refined the chart models with multiple rounds of alignment which involved supervised fine tuning, rejection sampling and dpo. And they generated the most SFT examples using synthetic data. So seems like again they were mostly relying on the 405B model for synthetic data generation. Now these models seems to be multimodal in nature, so they have the ability to process images, videos and speech as inputs and also generate these modalities as outputs. We highlighted this capability in their technical report, which is 92 pages long. So in here you can see that the input can be a single image or a video file, text tokens or speech segments, and the output can be video representation or text tokens. At the moment they are not releasing the multimodal version, so hopefully they will be able to release that sometime in the future. And now they have updated their license as well. So previously you couldn't use the output of a LLAMA model to train other models, but now you can do that under this new license. How good are these models compared to other models? Well, the answer is really good. So if you look at all these three models starting from 8,070B all the way up to 405B, they are best in class or almost the same in their own categories. The 405B is comparable to bigger models like GPT4, GPT4O and Claude 3.5 Sonnet. But for me the winner is Llama 3 because it's very comparable to 405B but is much smaller in size that you can run this on your local system. Again, don't rely on the benchmarks, check your own vibes regarding these models for your own applications. But to make a better sense of these benchmarks, I found this article very helpful and it's from IBM. Here they're comparing the 405B with other leading models for undergraduate level knowledge which the MMLU benchmark measures. This 405B is very comparable to OpenAI's GPT4 Turbo and the Opus model from Anthropic as well as Gemini 1.5 Pro. Even for graduate level reasoning which the GPQI data set measures, it performs very closely to both the OPUS as well as GPT4 Turbo. Now the math problem solving skill is just behind in GPT4O, but it's better than the 3.5 Sonnet. And when it comes to reasoning comprehension as well as Knowledge Q and A which is the ARC Challenge data set plus the coding is very comparable to the state of the art models that are available today. Now some of the best use cases for this 405Br synthetic data generation knowledge distillation for smaller models so as I mentioned before, the 8 and 70 billion models seems to be distilled version of 405B. You can also use this as a judge. This was a very popular use case for bigger models like GPT4 and Cloud Opus and you can use this to generate domain specific fine tunes as well. Unlike the previous version which was English only, this is now multilingual. So beyond English it has support for Spanish, Portuguese, Italian, German and Thai. And it seems like they are also adding some more languages. I think I heard Hindi's in that language or that list of languages. So we probably are going to see more languages being added which is pretty great for people who are not speaking English. In terms of the model capabilities they are highlighting tool usage. They actually released Agentic system with this Nama 3.1 release we're going to look at those. Then the agents can be multilingual as well, which is pretty impressive and it has complex reasoning capabilities along with great coding abilities. Now on benchmarks, the 405B seems to be state of the art, but do human prefer responses from this model compared to the other models? So they did the human evaluation study in which they compare two models and see which response is preferred by humans. So when the 405B was compared with the original GPT4 as well as Claude 3.5 sonnet, seems like the human preferences for these models basically have a tie. However the GPT4O seems to generate responses which are liked by humans a lot more compared to the 405B model. I think in the upcoming days we're going to have a lot more data on this when people start reporting their own vibes. But this is something to keep in mind if you're building applications on top of 405B that are going to be shown to users and basically rely on whether they like the responses or not. With this release they are also introducing the LLAMA system which is an overall system that can orchestrate several components including calling external tools. Their vision is to go beyond foundation models to give developers access to broader system that gives them the flexibility to design and create custom offerings that align with their vision. If you think about ChatGPT from OpenAI or Claude from Anthropic, these are not standalone LLMs, but as a part, these are basically part of multi component systems and the Meta team is trying to create something like that. They are providing a reference system which they are calling the Llama Agentix system. We're going to have a look at that. But along with that they are also releasing Llama Guard 3 which is a multilingual safety model and a Prompt Guard which is a prompt injection filter. They released a new repo called Llama Agentix system. This basically implements several components of their Llama system which allows you to do multi step reasoning plus has the ability to do tool usage and it works both with the bigger as well as the smaller models. It does come with a code interpreter so you can do some data analysis in there. I'm going to be creating a number of different videos on how to use this, so if you're interested in that, make sure to subscribe to the channel. Now for the UI they are using Mesop, which is a new Python UI builder from Google. So very interesting choice there. If you want to try these models through the API you have a number of different options. So here is a list of different providers who are providing these models through the API. Overall for the bigger models, Fireworks AI seems to be the best choice when it comes to the final pricing. For smaller models, the 8bit, this Octo AI seems to be a good choice. And this is kind of the beauty of open source. It's now you have a lot of different options through different API providers. They are giving you the same models, but if you want to try these models just through an interface so you can use something like Groq. Although currently the 4 or 5 billion model is not available because this was basically overloaded, everybody was trying to experiment with the model. Another option would be to try Meta AI. The 4 or 5 billion model seems to be working there, but there are I think limited number of requests that you can make per day. And you can also try these models on Hugging Face chat. Again, the 405B model is not currently available. If you try to use that you are going to run into issues, but you can use these 70 billion models without any issues. I'll create a more detailed video on running these models and doing different comparisons but but the model weights are also available on Hugging Face for you to download. Now the question is going to be how much VRAM you need in order to run these models. Well, I have all the answers for you so we're going to be relying on this blog post from Hugging Face for inference. If you're running the 8 billion model in 16 bit floating precision you just need 16 gigabytes of vram. For 70B you need 140 gig and for 405B you need 810 gigabytes. But if you run if you want to run this on 4 bits precision then you need just 203 gigabytes of vram. You need to be GPU rich in order to run this or you just need to have access to H100 cluster or a single node will allow you to run the 405B in 8bit precision. Another thing to keep in mind is that the VRAM requirement is increases as you increase the number of tokens in the context window. So here is a table which provides a very good comparison of how much VRAM you will need in order to support a different number of tokens in the context window. So you will need an additional 123 gigabytes of VRAM if you're using the full context. Now if you want to train these models for the 400x5B, you need 3.25 terabytes of VRAM. But if you're just fine tuning the 8 billion model in full fine tuning you just need 60 gigabytes. Memory requirements is a huge consideration both for training as well as inference. So these numbers are going to be very helpful if you decide to either fine tune these models or put them in inference. I want to close this video with the open letter from Mark Zuckerberg and the letter is titled Open Source AI is the Path Forward. I'd highly recommend everybody to read this whole letter, but in the letter he makes a case for open source AI systems. Here's a section why Open source AI is good for developers. We need to train, fine tune and distill our own models. We need to control our own destiny and not get locked into a closed vendor. Data privacy is a big issue, so we need to protect our data. We need a model that is efficient and affordable to run and we want to invest in ecosystems. That's going to be standard for the long term. And according to him, open source AI is good for meta, so that's his own business venture as well as Open source AI is good for the world, and I totally agree with him. I hope you found this video useful. Thanks for watching and as always, see you in the next one.