Metas LLAMA 3 Just STUNNED Everyone! (Open Source GPT-4)

So Meta have finally released their long anticipated Llama 3 model, which is an open source model that actually grants access to a variety of new capabilities in terms of how well the model functions when it answers questions. And this is a truly landmark event for the AI community. So I'm going to let Mark Zuckerberg state exactly what's going on and then we'll dive into the technical details of exactly what this release means.

All right, big day here. We are releasing the new version of Meta AI, our assistant that you can ask any question across our apps and glasses. And our goal is to build the world's leading AI and make it available to everyone. Now today we are upgrading Meta AI with llama, our new state of the art AI model that we're open sourcing. And I'm going to go deeper on Llama 3 in just a minute. But the bottom line is that we believe that Meta AI is now the most intelligent AI assistant that you can freely use. To make Meta AI even smarter, we've also integrated real time knowledge from Google and Bing right into the answer. We're also making Meta AI much easier to use across our apps. We built it into the search box that's right at the top of WhatsApp, Instagram, Facebook and Messenger. So anytime you have a question, you can just ask it right there. And we built a new website, Meta AI for using it from the web. We're also releasing a bunch of unique creation features. Meta AI now creates animations and it now creates high quality images so fast that it actually generates and updates the images for you in real time as you're typing. It's pretty wild and you can go check it out now on WhatsApp or the website. We are investing massively to build a leading AI and open sourcing our models responsibly is an important part of our approach. The tech industry has shown over and over that open source leads to better, safer and more secure products, faster innovation and a healthier market. And beyond improving Meta products, these models have the potential to help unlock progress in fields like science, technology, healthcare and more. So today we're open sourcing the first set of our Llama 3 models. At 8 billion and 70 billion parameters. They have best in class performance for their scale. And we've also got a lot more releases coming soon that are going to bring multimodality and bigger context windows. We're also still training a larger, dense model with more than 400 billion. And to give you a sense of LLAMA 3's performance, this first release of the 8 billion is already nearly as powerful as the largest LLAMA 2 model. And this version of the 70 billion model is already around 82 mmlu with leading reasoning and math benchmarks. The 400 billion parameter model is currently around 85 mmlu, but it's still training. So we expect it to be industry leading on a number of benchmarks. We're going to write a blog post with more technical details on all of this if you want to go deeper. In the meantime, join Meta AI and let me know what you think.

So that was Mark's Zuckerberg statement and honestly there is quite a lot of information to dissect because there's just so much from this release and it's actually a lot more than many people, including myself, did anticipate. So let's actually take a look at one of the first things that he talks about and this is of course the benchmarks. So we can see here that the benchmarks are actually rather surprising. We can see that the Meta llama 3 instruct model performance. And the reason these benchmarks are so surprising is because if we take a look at these models, they're actually state of the art, which means that this is the very best that you can get in terms of AI. So there's nothing better currently that exists at the 8 billion parameter model and at the 7 billion parameter size. So with that, you have llama 3 leading the way in terms of open source. Now, one of the most surprising things that I think most people were actually shocked by was the fact that if you take a look at some of the benchmarks, we can see that that this right here is Claude Sonnet and this was part of Claude 3's family of large language models, but it seems to have been surpassed by meta with llama3, which is quite surprising. Now, we don't know exactly how large Claude 3 sonnet is, but it's quite surprising that a 70 billion parameter large language model can actually surpass a state of the art model that many people do use on a daily basis, but for a variety of tasks, which does go to show that this industry is currently and consistently being shaken up in terms of who is the market leader on the benchmarks for different size and different price points. And I can see that this for Meta and Llama 3 is going to be a key, key area for dominance due to their ability to continually make changes to their models and update them and completely thrash them on benchmarks. Now they, to be honest with you guys, this was something that I think nobody really expected because these models are just open source and are mainly for the developer community. Whilst, yes, we knew there were going to be improvements surpassing Gemini 1.5 Pro, even on some benchmarks like the MMLU are obviously quite surprising. And of course, when we do compare it to the other models at similar sizes like Gemma, Google's Gemma, and Of course Mistral's 7B instruct, we can see that Llama 3 absolutely thrashes these models in terms of performance and in terms of general ability overall. So this was something that, you know, you can look at and see that, okay, right now it seems that currently we're on a path where even companies like Mistral are being beaten in terms of their ability to launch large language models or AI systems that are consistently at the front of the pack in terms of their ability. Which is, to be honest with you guys, quite surprising. So with that being said, there was also some other information that you should know. One of the things that they did was they sought to optimize performance for real world scenarios. And to this end they developed a new high quality human evaluation set. This evaluation set contains 1800 prompts that cover 12 key use cases, including asking for advice, brainstorming, classification, closed question answering, coding, creative writing, extraction, inhabiting a character Persona, open question answering, reasoning, rewriting and summarization. And of course to prevent accidental, and of course to prevent accidental overfitting of our models on this evaluation set, even our own modeling teams do not have access to it. So essentially what they did here was they used a new human evaluation set. And I think this is really, really important because I've always said that humans are the end users of these products, so they should be optimized for humans and not benchmarks. And that's what some things like the LMYS leaderboards have aimed to optimize for and see which LLMs rank best in that area. Because it's being used by real humans. And that should be by default the benchmark that people do test their model against. Because that is what is going to be what people actually use. It doesn't really matter if something looks as if it achieves great, unlike the mmlu or the GSM 8K, if people can't actually use the model and on a day to day basis for certain things, of course, unless it has a really specific use case, I think this is going to be something that shows us how good this is. Now if we look at llama 3 versus the human evaluations, one of the things that they did was they tested it against some of the other state of the art models. So we can see here that in the initial area we can see that the Metas Llama 3 was tested against Claud Sonnet and overall the majority of the time it did win in the human evaluation, which is like I said before, quite surprising. And it was 52% a win, 12.9% a tie and 34% a loss. But across the board we can see with even Mistral medium that Meta's llama 3, the 70 billion parameter model is really, really surprising in terms of its capabilities. And across the board it increasingly gets better compared to Mistral Medium GPT 3.5 and Meta's Llama 2. So honestly they've actually done a pretty amazing job being more efficient and managing to get an even better AI system while still remaining at the same number of parameters. Now in comparison with other open source, we can also see that the pre trained model performance does also excel these other open source and of course these other closed source models abilities. So we can also see the llama 38 billion parameter compared to the Mistral and the Gemma and it just completely thrashes them in terms of performance. We can also see the 70 billion parameter model doing better than Gemini Pro 1.0 and Mistral's 8.22B. And this is actually quite surprising considering the fact that Mixtral actually did just release this 8.22B model. So I'm not entirely sure how on earth Meta, you know, were able to secure that lead because it seemed like with Mixtral just releasing stuff pretty randomly. If you don't know Mixtral sl, Mistral are an AI company that is completely open source and when they release stuff it's pretty insane because they don't say that, okay, we're working on releasing this. The thing that they do is they literally just release a download link that you can go ahead and then download and then you kind of have to find out exactly what they released, how good it is, and then kind of benchmark it yourself. There's no real blog post for that. So the fact that, you know, Mixtrail could just drop anything open source at any time and they're pretty much the leading open source and to be honest, hats off to them because they're not, you know, backed by a giant company like Meta. They don't have billions and billions of dollars. I mean they recently did raise at 2 billion, but you know, Meta is like a multi billion dollar company. So you know, the comparison there is a bit stark. But the point is, is that they were still able to do A lot better than that. And whilst yes it is marginally better, it's still pretty surprising that they were able to do that compared to something that was only released a couple of days ago. Now in terms of the model architecture, there are some quite interesting things here. Llama3 uses a tokenizer with a vocabulary of 128,000 tokens that encodes language much more efficiently, which leads to substantially improved model performance. And in addition to that, the training data was something that is rather fascinating because people always love to see what you trained your model on. And llama 3 is pre trained on over 5 trillion tokens that were all collected from publicly available sources. And they state that their training dataset is seven times larger than that used for llama 2 and it includes four times more code. And of course to prepare for upcoming multilingual cases, over 5% of the LLAMA 3 pre training dataset consists of high quality non English data that covers over 30 languages. However, they do not expect the same level of performance in these languages as English, which does make sense. So they actually basically with the training data just ensured that their data set was truly, truly high quality. And that's why they are able to get so much more out of this model compared to other models at the similar size. And remember, as we've always discussed when training your models, data is one of the most important things as we've seen with smaller models like Orca 2 and Microsoft's Phys 1.5 Phi 2 and of course those smaller models. Now in addition, something crazy that he did actually speak about, and I think this is pretty fascinating, was that Meta's llama 3 is actually going going to be a 400 billion parameter model and this model currently is still in training. The checkpoint as of last one was 04-15-2024 and you can see that this pre trained model is pretty, pretty intense. You can see that these benchmarks are quite surprising considering the fact that Meta have previously not trained a model of this size. And this is the first time that that we're getting a look at what Meta can really do when compared to these close source companies. And I think it's so fascinating to how quickly these LLMs and AI systems are consistently evolving as these giant companies try to one up each other with in addition to just simply doing better in terms of providing a service, they just constantly try to raise the level of what these benchmarks are. Now this was actually compared in a table provided by Jim Fan and you can see he states here that the upcoming Laba 3 model will mark the watershed moment that the community gains open weight access to a GPT4 class model. It will change the calculus for many research efforts and grassroots startups. I pulled the numbers on Claude 3 opus GPT's recent model, which is GPT 420240409 and of course Gemini, and it says Llama 3. The 400 billion model is still in training and will hopefully get even better in the next few months. There's still many research potential that can be unlocked with such a powerful backbone and expecting a surge in builder energy across the system. So pretty much what we can see here is that this is basically on the level of GPT4 and with that it's pretty crazy because if you have an open source level of GPT4, it now means that people have access to build a variety of different applications and AI systems that they couldn't before in many different ways, which means the ecosystem is going to truly evolve from this moment. And that's why I said that this Llama 3 release was definitely going to be something pretty crazy. And it's basically open source GPT4 when it is released in the couple of coming months. And I do think it's going to be pretty interesting at how they maintain safety with this model because as you know, usually when you open source something there are bad actors that try to tweak the model. But it will be interesting to see how Meta changes their approaches as their model does get smarter. But honestly, this table is quite shocking. I mean on the mmlu it pretty much surpasses the rest of the models. Gpqa, it's on par. The human eval, it's on par. And of course the math benchmarks, it does lack a little bit, but like they said, it's still in training, so that means that this system will likely get even better. Now here's something that they also did talk about, which wasn't all the technicals currently. They did make a new website in order for you to access this model currently. So I do want to say that if you are in the eu, and I know at least if you're in the uk, you actually won't be able to access this current page. That's because there are just some rules and regulations that means things take longer to get here. However, I'm going to be doing a full tutorial on this, which will release probably about one or two hours after this video goes live. I would have included it into this video, but it is just significantly harder for the algorithm to find, especially for those looking to use this now there are many different things like he said, there's animations, there's different ideas, there's different images so it's going to be pretty interesting to see how this new model is used. If you would like to access this model whilst being in the UK like I am, of course you're going to have to use a VPN and I think at this point I might just, you know, find a VPN partner to partner up with because I find myself logging into VPNs all the time because for whatever rules and regulations exist for some reason us people that live in Europe and the uk, we are subject to just delayed AI releases. So with that being said, let me know what you thought about this release. Are you surprised by the benchmarks? Are you excited for the open source community that basically gives us access to an Open Source Level GPT4 sometime in the future? And are you ready to try out meta AI? Let me know what you think.