Meta released Llama 3, an open-source AI model that runs on consumer hardware and performs nearly as well as GPT-4 on many benchmarks. The 70-billion-parameter version scores 82% on HumaneVal coding tasks versus GPT-4's 84-88%, and it's free to download or use online outside the EU.
Questions this video answers
70 billion (public) and ~400 billion (large model still in training).
About 82% (compared to modern proprietary systems at ~84–88%).
Close to 40%.
Math around 40% for Llama 3 versus GPT-4 over 70%.
Described as comparable to earlier versions of GPT-4 and ranks highly, including second place overall and first for longer prompts in some measures.
Chapters
Meta released their Llama 3 model. This is an AI chatbot assistant like GPT4 and I was quite surprised by how well it is performing and it is open and completely free for all of us. I'll let you know how you can try it right now for free. It was quite surreal as I was in the US for the first time ever and exactly at the conference it was announced we were all stunned by the initial Results and Google DeepMind's Gemini 1.5 Pro is so good it might start rivaling the mighty GPT4. Super exciting news and some surprises are ahead. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now let's grab that 70 billion parameter version of Lemma 3 and and have a look at what it can do. Wait, what does the 70 billion mean? Well, let's say this can be run on a beefy desktop machine and it is performing really well on the coding tasks of the HumaneVal benchmark. It is doing very well at that with 82% where modern proprietary systems go up to 84 to 88. By the way, don't worry about the technicalities here. We will talk about what all this means for you in a moment. Now I am really excited for gpqa. This is a real tough scientific test with questions in organic chemistry, molecular biology, physics and more. Close to 40% that is very impressive. Now math around 40%. Hmm. I would say this is not its strong point given that OpenAI's new GPT4 can go over 70% on this. These are tough math problems. Even a three time international Mathematical Olympiad gold medalist gets only 90% on this. And wait, here is a surprise. There will also be a huge 400 ish billion parameter model available. Note that this is not available yet, it is still in the gym training, but it has a great deal of improvements over every test compared to the 70 billion parameter model. I bet it will be really impressive when it comes out, likely a bit before the end of the year. Now we fellow scholars have one big question to ask here. What is a good AI benchmark? Well first let's talk about what it is not something that has a success rate of less than maybe 10% that is not great for testing. In these cases it is a bit harder to evaluate the statistical significance of the test. And more importantly these are asking something that the models are not good at yet. Now what about the other end? What about over 80 to 85% success rate? Those tests are still useful but around this point they start to lose meaning. There may even be errors in these tests and if you are looking to score 100%, you would also have to replicate the errors perfectly. Just imagine a school test that is graded incorrectly and everyone writes the wrong thing to get a good grade on it. That is not desirable. And imagine if in some schools some of the answers for the tests leak out. That happens here too. So we need to be careful with the numbers seen in benchmarks. They are not everything. Gpqa, the tough science test is still one of the good ones in my opinion. Now most of the target audience using these AIs are not me. They are not Caroj with his crazy experiments, but it's, you know, normal people doing normal people things. How is that tested? Well, luckily there is a test for that too. This is called the Arena Leaderboard where we have another great surprise. Hold on to your papers fellow scholars, because llama 3 is comparable to earlier versions of GPT4. Wow. And all this for free and out there in the open. And it gets better if we filter for results only in English. Wow. Second place in some aspects this is one of the best AI assistants in the world. Once again available for all of us for free. My goodness. Meanwhile, Google DeepMind's Gemini 1.5 Pro is also ripping. This is the one with the incredible context window that can watch entire movies like Learn Code, Bases of projects and more. And now fellow scholars, look at that second place overall and it gets better. First place when it comes to longer prompts. My goodness. And you can try this one too. Outside the EU it is free to use the link is in the video description. So how can you try llama 3? 1 to the best of my knowledge once again outside of Europe you can try it on Meta's website or two even download and run it anywhere. Three There are websites that are hosting it already where you can try them. The links are in the description. So I would like to leave you with two key insights here. 1. AI assistants are improving at a stunning pace and two open source models are also back in the game. They are once again nearly as good as the paid proprietary systems, but these you can use free of charge to your heart's content forever. Absolutely wonderful. I would like to send a huge thank you for that. What a time to be alive. Experiment tracking, model evaluation and production monitoring for your deep learning projects and and LLM apps. This is what weights and biases does and it is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
