Metas LLAMA 405B Just STUNNED OpenAI! (Open Source GPT-4o)

So Meta have finally released their highly anticipated LLAMA 3.1 405 billion parameters large language model. There's so much to discuss and so much that they actually spoke about in their research paper. So first of all what you're going to watch is their announcement video and then I'm going to dive into so many of the details they seemingly left out and including the stunning benchmark.

Today we're excited to deliver on the long awaited llama 3.14 or 5 billion parameter model that we previewed back in April. We're also updating the 8 and 70 billion models with new improved performance and capabilities. The 405 is hands down the largest and most capable open source model that's ever been released. It lands improvements in reasoning tool use, multilinguality, a larger context window and much more. And the latest benchmark numbers that we're releasing today exceed what we previewed back in April. So I encourage you to read up on the details that we've shared in our newly published research paper.

Alongside the 405B model, we're releasing an updated collection of pre trained and instruction tuned 8B and 7DB models to support use cases. Ranging from enthusiasts and startups to enterprises and research labs like the 405B, these new 8 and 70B models offer impressive performance for their size along with notable new capabilities. Following feedback we heard loud and clear from the community, we've expanded the context window of all of these models to 128k tokens. This enables the model to work with larger code bases or more detailed reference materials. These models have been trained to generate tool calls for a few specific functions like search code execution and mathematical reasoning. Additionally, they support zero shot tool usage. Improved reasoning enables better decision making and problem solving. Updates to our system level approach make it easier for developers to balance helpfulness with the need for safety.

We have been working closely with partners on this release and we're excited to share that in addition to running the model locally, you'll now be able to Deploy Large Llama 3.1 across partners like AWS, Databricks, Nvidia and Groq, and it's all going live today.

At Meta, we believe in the power of open source and with today's release we're furthering our commitment to the community. Our new models are being shared under an updated license that allows developers to use the outputs from LLAMA to improve other models. This includes outputs from 405B. We expect synthetic data generation and distillation to be a popular use case that enables new possibilities for creating highly capable smaller models and helping to advance AI research.

Starting today, we're rolling out Llama 3.1 to Meta AI users and we're excited to bring many of the new capabilities that Angela outlined to users across Facebook, Messenger, WhatsApp and Instagram. With the release of 3.1, we're also taking the next steps towards open source AI becoming the industry standard, continuing in our commitment to a future where greater access to AI models can help ecosystems thrive and solve some of the world's most pressing challenges. We look forward to hearing your feedback and seeing what the developer community will build with Llama.

So that was the announcement video from Meta, but like I said, there's actually so much to dive into here and I think genuinely that this release is going to change the entire ecosystem. So one of the things that most people did want to know was of course the benchmarks for llama 3.405b. So when we actually take a look at some of these benchmarks, one of the things that we can see here is that this is actually on par with state of the art models. And something funny that I did actually find here was that Gemini 1.5 Pro isn't even here. So I'm guessing that maybe that model is far superior in those areas. But what we can see here across the board, and if you want just a quick glance here, essentially the categories that Llama bests the other models in are the categories where it has the box around. And I think it's crazy that currently what we're looking at here is a model that actually bests GPT4O and Claude 3.5 Sonnet in many different categories, one of those being tool use and multilingual. And of course the GSM 8K, which is pretty crazy. And arguably you can see that the reasoning of this model is, is up to 96.9, which means that potentially the reasoning of this model is better than Claude 3.5 sonnet. Now, of course this is all well and good, you know, having benchmarks that showcase that your model is doing amazing things, but one of the things we do have to always look at is of course the human evaluation, as after all these models will be used natively by humans, and that is by far the most effective benchmark to seeing how effective these models truly are. But just on the surface level, taking a look at what we do have here from a completely open model, and considering the fact that these other models are much larger in size, as you do know, GPT4 allegedly was 1.8 trillion parameters, meaning that if we compare that size to Llama 3.1 being a 405 billion parameter model, that means that it is as good or if not better than GPT4 as with a 4.5 times reduction in size, which is just completely remarkable. Meaning that potentially people can have GPT4 running offline locally. Although yes, it's going to be pretty compute intensive. But this is something that is truly shocking because it shows us the trajectory that we're on in terms of the size versus efficiency. So I do think that this is genuinely the start of a new paradigm where we start to get frontier capabilities available for free. Now, what we also did get from llama3 was this right here. So you can see that they also did updated versions of their llama3.8 billion parameter model and the 70 billion parameter model, which means that they made even further improvements. Now what this basically just means here is that in their respective sizes, Llama 3 is by far the best model that you can use. You can see that Gemma2 by Google here is falling short in nearly every single category other than the Ark challenge reasoning. And we've got Mixrol here that also is falling short. And of course you can see that llama 3.1, the 70 billion parameter model, actually does far better than mixral, which is 8 times 22 billion parameters, mixture of experts and GPT 3.5 turbo. And to be honest, what I'm seeing here is that this Llama 3.1 model, it isn't just marginally better than the other models at their respective sizes. We can see that not only does it surpass them in all of the categories, it manages to surpass them in a clear margin, which is incredible. Like genuinely incredible. So overall, if you are someone that is using these small models for whatever tools that you might want to use them for, you can see that Llama 3.1, a 70 billion parameter model, is super effective. Now, like I said before, if we look at the human evaluations for this model, what we can see here that it does hold up respectively against state of the art models. What we can see here is that around 70% of the time to 60% of the time, or 70% to 75% of the time, it either wins or ties with state of the art models. That is really impressive considering the size difference and the cost to use these models. I mean, imagine having an unlimited version of Claude 3.5 sonnet. I know so many people that are building with those models that unfortunately run into issues because the model is just very expensive to use. So this shows us here that versus GPT4 it wins a lot more and versus GPT4 oh, it does win a little bit less, but it's still very respective considering how small the model is Now I know it's still pretty big, but compared to the other model sizes, this is just something that we never thought we'd see. Now something interesting that they also managed to talk about was they managed to talk about how this model was a bit different in terms of the architecture. So we can see here that they said we've made design choices that focus on keeping the model development process scalable and straightforward. We've opted for a standard decoder only transform model architecture with minor adaptation rather than using a mixture of experts model to maximize training stability. So I'm guessing that for whatever reason here, and of course the reason is that they stated that they wanted to keep everything super simple, they decided against using a mixture of experts model. And we can see here that this made the model a lot more effective. And I'm wondering if this is going to be a continued trend as we move towards a space because I did see a recent paper in which they actually did talk about, and this was Google, not Meta, but they actually did talk about a million experts. So I'm wondering if this is just for open source models but. But it will be interesting to see what continues on. So this is where we get into the research part of this. And you can see here that they talk about the Llama 3 herd of models. So it says here that the paper also presents the results of experiments in which we integrate image, video and speech capabilities into LLAMA 3 via a compositional approach. Now that is absolutely insane because what they're trying to do here is to make this model multimodal. And what you can see here is that they say we observe this approach performs competitively with state of the art on image, video and speech recognition tasks. The resulting models are not yet being broadly released as they are still underdeveloped. So essentially what they have is they have image, video and speech recognition tasks which they can use. But these are still under development and some of the stuff that I'm seeing in this research paper shows me that they're actually pretty good. So what we can see here is that they said as part of the 3Llama 3 development process, we've also developed multimodal extensions to the model enabling image recognition, video recognition and speech understanding capabilities. They're still under active development and not yet ready for release. In addition to our language Modeling results. The paper presents our initial experiments with those multimodal models. So what you can see here is llama3vision. And we can see that this model actually does really well at vision tasks. And some of them it even manages to surpass static state of the art models. So it says image understanding and performance of our vision module attached to llama 3. So this looks rather effective because there aren't too many differences in terms of how it performs. We can see that it performs a lot better than GPT4 Vision. If you take a look actually at GPT4 Vision, you can see that in these categories, Even at the AI2 diagram, you can see that this is 94.1 and this is 78.2. So taking a look here, you can see that this actually does better than the previous GPT4 vision. And the reason that is crazy is because if you remember reading the initial GPT4 Vision paper, that paper was actually talking about how crazy GPT4 Vision was. So I can't imagine all of the use cases that are going to happen when we actually do get llama3 as a vision assistant. So that's going to be really amazing. And what's even crazier is that there were only marginal improvements from LLAMA3.70 billion parameters to the LLAMA3.405 billion param. So we can see here that using these different models, there's not that much discrepancy between how much the vision models are between the 70 billion and the 405 billion paramount. But overall this is really good because image recognition is relatively expensive. Now we've also got video understanding. And what's impressive here is that if we actually look at llama3.70 billion parameter model, the video understanding model, that video understanding model actually performs better than Gemini 1.0 Ultra, Gemini 1.0 Pro, Gemini 1.5 Pro, GPT4V and GPT 4.0. So that's pretty incredible that they managed to support DEADIA in terms of the video understanding model. And I got to be honest, whilst yes, you could argue that Gemini 1.5 Pro, the video understanding is long context, so it's kind of different in the sense that it can understand what's going on over 2 million tokens, I still find it incredible that such a small model is able to compete and be on par with these giant multimodal models. Now additionally, what we can see here is one of the features that they actually spoke about, which is essentially the audio conversations. So you can see right here, this is a screenshot from where Someone is having a conversation out loud. I guess that you could say this is quite similar to GPT4O, you know, the version of ChatGPT that you can actually talk to like it's a person. But you can see here that it's pretty crazy. You can, in the sense that it's able to understand many different languages and it's able to understand that through natural speech and not just text, which is a little bit different, because understanding the pronunciations of certain words and of course, the under, and of course how those words are spoken is a really big thing in terms of using AI. Now, another thing that they also showed was this tool use. And we can see right here that if we actually take a look at what's going on, it says, can you describe what's in this CSV? Then the model is able to identify exactly what's going on in this CSV, which is really nice because a feature that I didn't mention was actually that llama 3 is actually 128 tokens long. So it's a longer token context length model. And then you can see right here it says, can you plot it on a time series? So what it's also able to do is it's also able to use tools to execute different things. So you can see right here that the model is able to essentially bring up this graph, which is really nice. And. And then it's able to say, can you plot the S&P 500 over the time in the same graph? And then it's able to do that rather effectively. I think you guys might underestimate what's going on here, because tool use is truly the next stage of these AI systems. And I think this is truly how we get to systems that are generally intelligent because they're able to execute a wider range of things utilizing all of the tools. The last thing that I'm going to leave you guys with, which is pretty crazy, is that they state that our experience in developing LLAMA 3 suggests that substantial further improvements of these models are on the horizon, which means that they're basically saying that, look, Llama 3 is not the best that we're going to give you. There are so many improvements that we can make to AI models, and we are just scratching the surface of what's going on. Now, if you enjoyed this video and you want to use Llama 3, of course, and you're in America, you can just head on over to Meta AI. But if you're in the uk, the only place that I know currently that you can use this is and I've even tried with a VPN and it doesn't work because you need an account to sign in. And of course by the time this video is released that might have changed. But currently, if you want to use it right now is after the video is released and you're in the uk, you're going to have to use Grok, which is a inference platform where they basically allow for super fast inference. Then just head on over to this right here. You can see llama 3,405 billion parameters. Then of course you can use the model right here. So that's the only way you can use it in the uk. I'm not sure if it's banned in other regions, but I do know that currently Meta AI is just not available right now in the uk. But of course they're going to roll it out onto many different platforms that you're going to be able to serve it. So within 24 hours that's not going to be a problem. There's a billion different sites that are going to start hosting this. But of course, if you did enjoy the video, hopefully this was of some use to you and I'll see you guys in the next one.