Llama 3.1 Is A Huge Leap Forward for AI

All right, so you might have heard this, but Meta just open sourced the new Llama models and the big one is actually state of the art, meaning it is better on most benchmarks than GPT4O and open source. And they updated their 70B and 8 billion models, which actually the 8B model is the thing that I personally most excited about. So as per usual, let me summarize all the most important details, show you what people are building with it, and I'll even show you how to run it locally, which means it can be offline, and how to jailbreak this thing so it does anything you want. Because this isn't a closed model, it's fully open source, which opens up some interesting possibilities. Let's have a look at it all. Okay, first things first, let's get the basic specs out of the way and let's talk about these benchmarks because they are impressive. So they just released three models, one of them is a completely new Release and the 70B and 8B are update from llama 3. Now they call it llama 3.1. The 405 billion parameter model is the state of the art GPT4 competitor. It's a big model designed to compete with OpenAI and anthropic. If you're not already aware, these big models have the most world knowledge, are the best at coding, and they also excell at math, reasoning or using other tools, and so much more. But not every use case requires this. Matter of fact, no matter what kind of machine you have, I don't think anybody at home realistically is going to be able to run this big one. That's where these smaller models come in. I actually use the Llama Free 8B model regularly. We'll talk about this soon and we'll actually end the video of me using it right here locally on my machine. But all of these have been updated and if you look at the benchmarks, which we won't spend too much time on, you can look at this by yourself. And as I always say, they are not everything. Actually, the Internet kind of came up with a name for this. They say, yeah, benchmarks is important, but does it pass the vibe check? That's literally the wording used on Twitter these days. It's kind of funny as that is a Gen Z term, but it does capture the essence of it. These benchmarks are not everything. Now, human eval might be important and Llama 3.1405 B scores 89 points on it, GPT4 omnis just barely above it. But on Many others here like MMLU. It's virtually identical to GPT4 for Omni, also identical to Claude. 3.5 sonnet on math. It actually beats out these other models. And what I personally love to see that on these long context tests it actually outperforms other state of the art models as it does on its language capabilities. But enough benchmark for now. You can look at all of these by yourself. What I want to point out here is the fact that the jumps for the 70B and the 8B model are actually significant, especially on this 8B one look at these differences. Human Eval 60 to 72, Math 29 to 51 tool use almost doubled on some of these benchmarks. So that's great. But does it pass this vibe check? We'll see. We'll test it out ourselves. I guess only time will really show that. And you have to try it yourself. I can tell you already from my initial usage tone is very similar to Llama Free, which I personally would say I actually prefer to the tone of ChatGPT. But Claude is still king when it comes to writing style. Now I personally really care about writing style, but different people have different use cases. So the vibe check is just something you will have to figure out for yourself or will have to wait a little bit more for the Internet to report back on. One more thing about these benchmarks is that yeah, they publish their own numbers, but actually there's various forms of benchmarking that exists now. And if you follow my Friday show, AI news you can use, you will know that Scale AI actually brought out their very own leaderboard and they test the models on private data sets that have never been released to the model developers. So they couldn't just include them in the training. Sort of a cat and mouse game with these benchmarks. But I just personally and fully subjectively find these scale benchmarks to be more trustworthy than something like the other alternatives like Chatbot Arena. Those results seem skewed sometimes. I mean heck, it's based on user preferences. A person can just go in there while sipping a beer and be like hey, I like this response better. And then that goes into the results here on the scale leaderboard. When it comes to instruction following, which is what I personally care about the most, the 405B model actually is is number one head of sonnet coding, Sonnet Twins. This totally aligns with my personal preference too. And for all you Spanish speakers, unfortunately they haven't tested this yet because Nosotros hablamos de intelligencia artificial Todo el dia. And yeah, I did have to look up this word. And no, I do not speak Spanish, but I did take classes for a while so I have some basics and I absolutely love the language. Just the flow and sound of it is muy excellente. Anyway, back to this video again, but a few things are not as ethereal and a bit more immutable like the context Limit, which is 128,000 tokens across across all three models. That's fantastic. That is more than enough for a bulk of the use cases that I encounter in my everyday usage. Oh, and it can also handle eight languages. And did I mention that it's open source including open weights and the code? One interesting fact about the big model is that it actually took 30 million H100 hours. These are industry standard GPUs that these models are trained on. And 30 million H100 hours to a mere mortal like you and me, if you just quickly look it up, translate to $3.50 per hour meaning training. This model, if you we wanted to do it, would cost $100 million now sure, better purchase the GPUs and run them on their own. So it's going to be a fraction of this price, but nevertheless they paid tens of millions of dollars just to open source the damn thing. That's pretty impressive. Gotta give it to Zach and the team at Meta. I mean from whatever angle you look at this, this is just pure good, I think. So that pretty much sums up the basics. I'm not gonna bore you with the model architecture. It doesn't really matter in the context that we're talking about. As you might know, I like to take this point of view of a AI power user. I use all these different little tools, however privative they might be on a daily basis and my mission is to try to help you get more out of all these tools. So all the intricate details of how these models were trained are generally not as essential to cover. What I do want to cover though is some of these use cases that this will open up. Well, matter of fact, with other open source models, these use cases were already open already, but now we actually have a state of the art model that can do some of these things. To me, the most exciting one is rag and tool use. But also fine tuning can be a fantastic capability, quick refresher if you're new here. Fine tuning is giving it specific input and output pairs to specialize the model for a specific use case only relevant to you. So if you only use the model to classify all sorts of incoming data and the data always looks similarly and there's a certain amount of categories. You could fine tune it so it only focuses on that and does that one use case really, really well. Just an example. And RAG is using external files to supplement the context window to essentially extend it by creating so called embeddings that it can then search over. Matter of fact, I have a video coming up going deeper into RAG and how to use it for yourself. But basically this model is going to be open to all of that, and this has not been with all open source releases. It is actually permitted to use the model for synthetic data generation. So you could produce these artificial data sets that you can then use to for example, fine tune this model or train another model. This is actually really surprising to see because it gives a lot of competitors the ability to use this state of the art model, the 405B, to improve their own models and compete with Meta. But I guess Meta just doesn't care. They have their core business elsewhere and all they care about in the AI space. Apparently it's for everyone to have a level playing field because they have their advantages elsewhere already. Okay, last note in the summary section, the pricing is nothing surprising, nothing special. If you run it through these various services, you will find that the pricing of GPT4 mini, for example, is actually cheaper on the input side and costs the same on the output size as the 8B model. And if you look at GPT 4.$05 for a million tokens of input, $15 for a million tokens of output, roughly the same over here. So it's not a big cost reduction. The real value of this is the fact that it is open source and you can do all of these things. You can run it locally, you can alter the weight and for example, make the entire model uncensored. People have been doing that with Llama three like a day after release. Expect the same to happen here. And that's also sort of scary, right? Because now we have the most capable models and they're fully open and people can mess around with them. There's this whole argument of, yeah, just China taking these and not having to develop their own. It's a legit thing. I mean, their entire website starts with a download button. You just fill out a little form and you get the model literally an hour after this coming out. I just downloaded this into LM Studio. There you go. We already have these instruct models available that should perform even better than the benchmarks on the website showed. And we'll get to this in a second. But before that I have to address one story which is kind of linked with this because OpenAI actually made their move too. I choose violence immediately. In response after this came out, in typical OpenAI fashion, they released the fine tuning for GPT4O mini. So this is their competitor to the 8B model from Llama. They just didn't want a world in which people, people could fine tune these small models only with Meta, but not with OpenAI. Meaning this small model that I can run locally and you can probably too is fine tunable. But now also GPT4.0 mini is fine tunable. Which is interesting because in last week's news you can use, we actually talked about the fact how OpenAI teased that GPT4O is going to be fine tunable and oh God, I hope this is not too confusing. I realize there's a lot of model names, a lot of sizes and I just spent so much time with all of this that it's normalized to me. But hey, if you have any questions or something is not clear, you can always leave a comment below. I'll be checking them. If you have some questions, I'll be happy to answer them. Anyway, let's get back into the video. And what I was saying is that GPT4 and GPT4O is not available for fine tuning as of today. If you want to do that, you'll land here with Llama 3.1. But they did open up the ability for us to fine tune the GPT4 model, which is actually very exciting. I'll be playing around with that. And they give you the first 2 million training tokens for free. This is roughly a value of $6 and should be more than enough for you to get started and to get your feet wet. With this, along with the fact that GPT4 mini now is second in LMSwise chatbot Arena. I don't know, sometimes I don't trust these rankings. To be fair, that seems a little off. I don't know how this can be better in Sonnet than here. This does not pass the vibe check for me personally. Anyway, having fine tuning available on GPT4.0 mini at that price that I love to see. But then again, we can fine tune something like this and just run it locally. It's yours. All right. And I would like to tell you about one of the best places to acquire new skills on the Internet and that is Brilliant, the sponsor for today's video. Brilliant is an online platform that gives you hands on ways to learn about subjects like math, science, programming, and even AI. One of my favorite things about Brilliant is that they offer these structured learning paths. They combine multiple courses into a curriculum that you can then follow. And today I'm going to highlight the science learning path. I think especially the first course in it is really good. It teaches you about the fundamental concepts surrounding the scientific method, which is the fundamental method that enables most innovation that happens throughout society today. So understanding it might just be helpful if you're trying to wrap your head around all of these innovations happening in the AI space, for example. But this can not just help you with understanding science and innovation. It can really help in becoming a more reasonable person. And you'll be less prone to getting fooled or scammed by somebody. And that course is just the beginning of the learning path. Then it goes on to build up on the fundamental skills that you acquire right at the beginning. And of course, there's much more to be learned with Brilliant. So if you want to dive in and start exploring these incredible resources, head on over to the link in the description for a free 30 day trial. Plus, if you decide to stick with it, you'll get 20% off an annual subscription. A big thank you to Brilliant for sponsoring this video. And now let's get back to it. And now let's get to the interesting part. What can you do with it? What are other people doing with it? Well, let's just start with probably the most impressive demo, which is from Jonathan Ross over at Grok, and they're doing essentially real time inference. I mean, this thing is instant. If you're not familiar with Grok, they're really, really good at running these models fast. And if I say fast, I mean it, the text kind of just appears. If you run the 8B model, have a look at this. That's a little complicated. Can you tabularize it for me? And there it is. So this is the smallest model, which will also be the fastest. Nevertheless, very impressive. Next up, we have Perplexity already degrading this. It's been like an hour from release. But if you're a Perplexity Pro user, you can now use llama 31405B for your search. This is the type of thing where you just have to test it for a few days and see if that actually works better than something like GPT4O. And then the obvious question, which is, where can I use this myself? Well, there's a few options. One of them is you could head on over to PO and there's many chatbots live there where you could use it already. One note is that to run these bigger models on Pole, you actually do need a subscription. The second and maybe even more obvious option would be Meta AI. But me sitting over here in Europe, I cannot actually use this. If I find a free version outside of Meta AI to run the model for free, I'll include it in the description below. Matter of fact, I did find a free space. There's this replicate space where it's hosted and it's actually free to use. Link will be below. So if you're in the us, using Meta AI is probably best. If you're not using something like PO or downloading it locally are some of your best alternatives. Now talking about downloading it locally, I made a tutorial on this before. I'm not affiliated with this company whatsoever. I think it's just the simplest way to download local models and to run them you don't have to use the terminal as you do with something like Ollama. It just has this wonderful graphical user interface and you can just search for models and download them. So if you search for this, you're going to find all the Llama 3.1 models. Just make sure it's the 3.1 and get one of the instruct versions if you can. Those will work better for you. And there you go. I just downloaded this model and now I could go ahead and start a new chat and of course I could write me an essay about penguins. And now this 8B model is going to write me the essay fully locally. I could switch off the Internet. This just works, which is amazing for a lot of these business use cases. You really want privacy. And you could essentially take this model, install it on an air gapped laptop that has never seen the light of day, AKA the Internet I guess, and just run this model fully locally. And what kind of information you have to be dealing with to warrant a workflow like that? I'm not exactly sure. That's up to you. I'm just saying it's possible as this model is yours. This is doesn't go out for an API back to some server farm where you have to trust a tech company with the data and the information that you provide these models with. Especially relevant if you're doing something like rag and you're actually uploading sensitive documents. All right, so by the time I completed my little privacy rant here, we have a response here. The majestic penguins, masters of adaptation in the Antarctic. As you might already know, we like penguins. They have a lot in common. With AI enthusiasts like you and me and this harsh yet breathtakingly beautiful landscape, which is the world we, the tech enthusiasts, have adapted to survive and thrive in one of the most inhospitable environments on Earth, the Internet. Anyway, so as you can see, this works wonderfully. And I want to do two more things here. First of all, I just want to take my most recent ChatGPT prompt and kind of see how it handled it. Okay. I was actually doing my counting over the weekend, and I had this simple prompt that I just wrote on the spot. Turn this table of USD euro exchange rates for three months, April, May and June into a CSV with dates in one column and the values in a second. So some basic data transformation here. I just copied in this table from the European Central Bank, I believe, and let's see how llama8b actually handles this in a new chat. I'll just copy paste the Same thing in GPT4.0. This worked like a charm. It gave me the CSV and I could copy this into Microsoft Excel to work with it further, actually then followed up with another prompt to fill in some blanks here and there. But this is looking really good already. Look at that. Here's the CSV output. I could just copy it, no problem. This would totally work inside of Excel. And I'm doing this locally on an 8B model. Right. Let's just do a little spot check. If this lines up on the first 749, on the second 783. Aha. There you go. It actually does not line up. This is wrong. So, see, this is why you might want to avoid these smaller models with something that might take a little finesse, like this use case. All right, so let me try the same thing in the 405B instruct model and let's see if this, this gets it right. And I'll just run this in a replicate space where it's free. And let me compare these outputs with GPT4O. So on the 2nd of April, it's 107-497-83. That looks fantastic. Let me go somewhere into the middle. 614-1068-1068. Yep, this works. So there you go. This is one of the use cases where you do want a bigger model. Usefulness has been confirmed. And instead of boring you with another 50 use cases here, I would just say try the things that you do go into your chatgpt, the history, take some of the latest prompts and run them in spaces like this, See how it performs. I do want to Leave you with one sort of a gem here. And that's plenty. The prompter's jailbreak from X here. Literally an hour after this thing came out, he already found a jailbreak for it. You can use it in the form of this prompt. I'll just copy paste this into a brand new chat in my little 8B local model over here. Okay, I'll paste this and what will I get? Actually, we'll have to blur these outputs because. Oh, it actually tells me I cannot provide instructions on how to create a malicious device. Let me try that again. New chat. There you go. Now it's giving me a step by step tutorial on how to create a novel and very dangerous biochemical compound. I won't go into the details here. I don't want to offend YouTube, but you can try this for yourself. Change up the prompt to give you any knowledge that you might require from it, and you can get these uncensored results before even the uncensored versions of these models come out. I mean, look at this table of contents it produced. Isn't this wild? All right, I sincerely hope this was helpful to you. Let me know in the comments below what you think of this, what kind of use case you'll be running through the model first. I'd love to hear that. And other than that, I'll see you soon because I've committed myself to uploading two high quality videos like this every single week. All right, I'll see you soon.