Károly Zsolnai-Fehér breaks down Llama 3.1, a free 405-billion-parameter AI model with a 128k-token context window. It hits 50% accuracy on expert-level science questions, matches proprietary systems like GPT-4O and Claude 3.5, and can run locally for private use.
92 pages
128k tokens
405 billion parameters
Breaks the 50% accuracy barrier
65%
This is simply unbelievable. An AI model that is open and is available for free for all of us and it did something that I thought was previously impossible. It rivals some of the best AI assistants out there like GPT4O and it has almost every feature you could possibly need. And luckily you fellow scholars can try it right now. It took me some time to get through the 92 page paper to be able to give you the good stuff and I'm not done yet. But here are a few surprising things that I found. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. It is called Lemma 3.1 and frankly I am completely stunned that it is able to rival the best proprietary close AIs in the world and we just get it for free. Wow. So what can it do? Well, you can throw a CSV file edit with lots of data. The data goes in and yummy insight comes out. But not only in terms of text. No no, look at that like a helpful co worker. It can even create these nice little plots for us about inflation rates. It has a fairly sizable a 128k token context window too for us to play with. What does that mean? That means that we can give it a document that is approximately the size of a book and it can read the book, then reliably answer your questions about it. It speaks many languages and can also use external tools. If we give it a complex task with so many moving parts, it it can first create a plan. First it wants to look up recent US inflation data and calculate the cost of a bunch of groceries and more. If you take a moment to read the prompt, you will see that it is not easy as it seems, and it is able to go out there, write a piece of code, and use other tools to organize and look up new information that by itself. We can even ask it to visualize the relationship between the sugar content of cereals and their ratings. And it goes out there, does the research, does the coding and gives us a beautiful plot. This is great because it reveals unexpected insights from a soup of data. For instance, just putting more sugar in that cereal does not necessarily mean that people will like it more. The new system greatly outperforms its previous versions and breaks the 50% barrier on the GPQA dataset, which contains really tough scientific questions in biology, physics and chemistry. Expert Humans with PhDs are expected to score 65% on this. Not a great deal more. Just saying. It also understands human speech, speaking, speaks many languages, and here comes one of my favorites. It is steerable. What does that mean? It means that we can give it instructions on how to behave. That is huge. Essentially there is no such thing as ChatGPT or no such thing as Llama 3.1. Now, yes, I hear you asking Karoj, why are you saying this? How does this make sense? Well, every single copy of these AI assistants can be customized by their owners. So I gotta be honest with you fellow scholars. I love mathematics and emoji too. And with this, these assistants can be instructed to delight me every day. And I absolutely love it. As expected, llama 3.1 can translate code between different programming languages. Human human preference tests are also comparable to GPT4 and a bit behind the better GPT4O. However, hold on to your papers, fellow scholars, because it can rival and in some cases even beat the best systems out there, like OpenAI's GPT4.O and Claude 3.5. That is stunning and doubly stunning if you add the fact that all this is free, free, free forever for all of us. The model itself is available to download right now. It is a whopping 405 billion parameter model, so this is mainly to run on servers, not on your phone or laptop yet. However, good news. You fellow scholars can try it out right now through the link in the description. So what else is it good for? Well, for instance, confidentiality. If you have some confidential data that you wish to work with, that's great. But you may not want to share all this sensitive data with anyone else. For instance, in my case, I had the honor of looking at the AlphaFold 3 paper a little earlier than its release and it was glorious. However, I could not share anything about it. So if I wanted to discuss something, there weren't any truly powerful offline options that I could run myself. Well, not anymore. That is absolutely fantastic news for all of us. And once again, all this for free for all of us. This might usher in a new age of AI that is open to everyone and is as powerful as the best AIs in the world. A great gift to humanity. Thank you so much. What a time to be alive. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lamda, featuring up to 512 Nvidia H100 GPUs with InfiniBand Networking. And hold on to your papers because with One Click Clusters you can access all these GPUs on demand. With reservations starting at just two weeks. Try it out@ lambdalabs.com paper or click the link in the description.
