Nvdias New Open Source Model Surpasses Gpt4o And 3.5 Sonnet....

So Nvidia have just released their stunning Llama 3.1 Nematron 70 billion parameters instruct model. And crazy enough, this model surprisingly beats every closed source model. So it seems like once again open source has raced forward despite closed source efforts to best them in terms of which model is currently state of the art. So there's going to be a lot to dive in for this video because they did introduce a new technique with how they produce this model and I think it's rather interesting with how they managed to do this and it's worth noting. So you can see right here it says our Llama 3.1 Nematron 70B instruct model is a leading model on the Arena Hard benchmark from LM arena AI. So let's get into exactly what was done. Okay, so essentially, if you aren't aware, basically Nvidia used the Llama 3.1 model as their base model and then of course they did some post training on that model, I believe with reinforcement learning. And that kind of reinforcement learning managed to get this model to surpass state of the art closed models. And this video actually does have some of my own tests that were quite surprising. So I think AI is about to get even more insane. So for those of you who do want to see the actual benchmarks of the model, you can see here that we have Llama 3.1 Nematron performing at 85% 57.8.9 on the MT bench. And you can see that this actively surpasses all prior models. So this one is really surprising, not just because of the fact that it surpasses Claude 3.5 Sonnet, but the fact that it also surpasses GPT4O, which is recently been debuted by OpenAI as the Frontier model that can do a lot more than just text. So I think this is pretty insane because not only that as well, now that I'm looking back at this, I can also see that this manages to surpass the Llama 3.14.5B instruct model, which is a monumentally larger size model. But somehow training the model in a certain way has allowed them to perform this kind of feat where it surpasses the closed source model. So certain ways that you fine tune a model are going to be really impactful on the results. Now essentially there was this paper that they put in the hugging face description and basically what they did that was really different to everything else was they introduced an advanced reward model used to improve the alignment of the AI models with human feedback. And we'll get later on to that because humans really do love the responses of this model. So if you. Essentially, the researchers addressed two main approaches to reward modeling, which is the Bradley Terry style and the regression style. Now both these methods are used to guide AI models to provide more useful and accurate responses by assigning them reward scores based on their performance in following instructions. The Bradley Terry model focuses on comparing responses to prompts and identifying which one is better, while the regression model predicts a numeric score for the response based on several criteria like helpfulness or correctness. Now what's crazy about this is that they faced a challenge because of course you've got these two different ways to of course guide the models. But the challenges are, is that these models are often trained on different types of data, which makes it really hard to compare them directly. So this is where Nvidia's genius comes in. To overcome this, the authors of this paper presented a dataset called Help Steer 2, which includes both types of data preference rankings for Bradley Terry and Leica T scale ratings for aggression. So this new data set helps bridge the gap between both these approaches, allowing for a more comprehensive comparison. So overall, what they managed to do here and why this model manages to surpass state of the art models is that they use reward models which are used to help produce better responses by scoring AI's output and guiding the model responses. And of course they decided to use a new data set which is Help Steer 2, which has both of these preference rankings and numeric rankings ratings to help train reward models more effectively. So this new combined reward model achieved top scores on a benchmark called Reward Bench. And basically by combining these methods, they managed to outperform state of the art systems. Now we can also see Nvidia's model's performance on the Arena Hard Auto. You can see right here that the Arena Hard Auto is an automatic evaluation tool for instruction tuned LLMs. It contains 500 challenging queries from the Chatbot arena and they prompt GPT4 Turbo as a judge to compare the model's responses against a baseline model. Now, the Arena Hard Auto actually has the highest correlation and separability to Chatbot arena among popular open ended LLM benchmarks. So when we actually take a look at these results, I truly find them pretty fascinating because on this leaderboard we can see that on the leaderboard with no style control and essentially what style control is, is for example, sometimes how you'll message ChatGPT and it will give you a response in a certain format. Whilst the data might be the same in another chat with another different AI system, the format can alter how humans view the helpfulness of the response. For example, in certain responses you would rather bullet points and in certain responses you would rather just a sentence. But we can see here that overall Llama 3.1 Nematron 70B instructs scores just two points above GPT4 Turbo and other models. And surprisingly we can see a large number of models on that list, although we don't see Gemini's recent model. So I would like to see if we do get that model there. But its scoring just behind 01 mini and O1 preview, which is rather fascinating. So of course if we take away this style, we can see that these discrepancies are a little bit more pronounced. But I would say that a 70B model doing this well is still a remarkable feat because it does mean that with how you manage to, you know, guide these models after they've been trained completely, it shows that we can still achieve marginal gains that could potentially catch up to state of the art. So things like these are really, really surprising. Now some of you guys might be wondering, okay, Nvidia have done it. They've managed to get this model to perform better than state of the art. But benchmarks are good, but how does it fare on certain questions that you might ask it? So of course you could always test the model yourself, but it's completely up to you what kind of questions you ask it. Now what I wanted to do is I wanted to ask it some questions from a research paper that I covered in a video three days ago. So you all know that this. Well, you might not all know, not all of you watch my videos. So you might know that there is this thing called the GSM noop. Basically this is from a research paper where Apple said that LLMs don't reason. The first question you're looking at here is a reasoning based question that asks a question and in that question it has some information that is just completely irrelevant. So you can see here that it basically asks about how Liam wants to buy some school supplies and the irrelevant information is highlighted in pink. It basically says, assuming due to inflation the prices were 10% cheaper last year, how much should he pay now? But the question starts with the information that you already need. It basically says these are the prices of the products that he's checking out with now. And then add some random information about inflation to see if the models get confused. Interestingly enough, we can see the 01 preview OpenAI's best model manages to unfortunately get confused and does the wrong calculations. I think this is what happens when you have a model that is rewarded for reasoning and their reasoning steps. And sometimes you don't necessarily need reasoning steps. You just need the model to really look at the question and answer the question and figure out what the question really demands of you. So I did put this question in to the 70B model because I wanted to see if it gets it initially right. And unfortunately, if I'm being honest, it didn't get this question right. Now that doesn't mean that this model is awful at all. Remember, this research paper was a complete shock and most models will get this question wrong. But what I did do was I did implement some information from a different research paper that actually says to have this simple thing. And I think you guys need to do this as well. If you have any hard, difficult reasoning question that you really want an AI to perform well at, perform this small step and you're more than likely going to perform a lot better on your outputs from the model. All I said was reread the question. Okay, There's a research paper, I don't know where this research paper is or this blog post, but I saw a blog post and it basically Sundays ask an LLM to reread the question. And it improved reasoning by like 10 to 15%. So all I did was I asked this smaller model to perform this reasoning step and I said, look, just reread the question, okay? And I didn't even give it anything else than that. All I said was just reread the question, okay? No additional context like what about this and that? And you can see that after asking the model to just reread the question, it manages to actually realize that the information at the end of the prompt is wrong. You can see right here, it says that the question asks for the amount Liam should pay now and provide provides current prices. And it says that information about the inflation rate does not affect the calculation for what Liam should pay now as we already have the current prices. So this is what I'm saying that sometimes there is the smartness inherently built into the model. You just need to be able to prompt it out, which is some clever prompt engineering. So I think this model is really smart as well, because I did give it a question that OpenAI01 failed. It wasn't this question, but it was this question, okay? Now this question was one that was once again containing some irrelevant information. It was just simply asking about picking kiwis. And then of course the number of something doesn't vary if some of them are smaller than average. But this was a key piece of information that O1 Mini manages to miss and other models managed to miss. And when I actually gave this question to this new model, surprisingly we can see that this model manages to understand that Oliver has a total of 190 kiwis and the size variation of kiwis on Sunday is noted but does not impact the overall account. So it's clear that this model does reason a little bit more than usual because of the way that the reward modeling was done. And another thing that this reward modeling has trained the model to do, you can see that it says we adopt a prompt that many have been recently vibe testing LLMs with how many Rs are in Strawberry and among Table 5 only reinforce which is the method that they used can correctly answer it. You can see that these other methods and these other models that actually clearly fail, but the method that they're using which is right here manages to answer this question clearly. You can see GPT4O says there are two letters Claude 3.5 sonnet says there are two letters 405B says there are two Rs llama 3.1 says there are two Rs, but the method that they're using managed to count them and then of course managed to actually get the correct number. So overall it seems that open source models have once again raced ahead of closed source in terms of their capabilities. But but this leads me to believe that larger frontier models that have even better reasoning might just be around the corner as the last time we saw open source manage to catch up to closed source, we saw a sharp jump in performance across closed source companies revealing their next iteration of models. So with that being said, if you enjoyed this video, let me know your experiences with Llama 3170B instruct and I'd love to know your thoughts.