Elon Musk STUNS The Industry With GROK 2

So today's update actually did just stun the entire AI community. And by the entire AI community, I mean those of you who are focused on the space and following the updates on a day to day basis. For those of you who are unfamiliar with what I'm talking about, recently X AI announced their new chatbot, Grok2. Now, Grok2, which is a very capable chatbot, confirmed recently that they are actually the chatbot that was SAS Column R. Now, if you don't know why this was such an incredible statement from them on Twitter, it's because Sauce Column R. Many people were speculating that this model was actually Strawberry. And considering the fact that this chatbot was grouped with the OpenAI chatbots on the Chatbot arena, many people thought that this was a speculative OpenAI model that was able to do advanced reasoning. And today we got the news that this chatbot is actually a model from X AI. You can see clearly here that Elon Musk is confirming this by tweeting, Rrr sus. This tweet got 38.7 million views. And I think this is one of those tweets that, you know, is like, not really a slap in the face, but more also a look, I told you so. Because Elon Musk is someone that has been, you know, working on his chatbots for quite some time. And of course the thing is, is that one thing that people never do, regardless of whatever industry Elon Musk is in, never count him out. There are many naysayers, many critics and whatever, but one thing that you can't dispute is, of course, results. Now, the thing about this model is, is that if we do take a look at X AI the, the SUS Column R, we can see that on the leaderboards for the Chatbot arena that this model actually performed rather well. I have to be honest, when looking at the leaderboards, especially on the LMYS arena area, one of the things you do have to do is note that sometimes things aren't exactly as they seem, because in some of the arena battles, sometimes models are not the right context length, sometimes there aren't the right kind of queries. But when I've used SAS column R, this has been a model that has consistently been basically state of the art. Now, you remember in my recent video where I actually spoke and tested this model against some of the other models, one of the things that I spoke about was the fact that this model, it was on par with the state of the art, but it wasn't really crazy. But one of the things that I did find out about this model was that it seemed to be trained in a way that it was essentially reasoning, a lot like its problem solving capabilities seem to be more so than the other chatbots, just by the way that it responded. I think that we've moved now from a stage to where we've got these standard chatbots to where we now have the models that are more so trying to reason about their questions and queries in order to make themselves more helpful. If you haven't been familiar with the recent updates from Anthropic, they were one of the first labs that actually incorporated this natively into the chatbot. One of the early things we saw from Claude that was a leaked prompt was that we could actually look internally at what Claude was thinking before it responded to a message. And I'm guessing that this, combined with the extra data, more training, better post training and pre training, and of course a new entire model that we managed to get Claude 3.5 Sonnet, which is just remarkably more smarter than anything else on the market. Now, this, you can see, currently manages to apparently beat Claude 3.5 Sonic. The only problem is we don't have access to Grok 2 just yet because it is slowly being rolled out. Now you can see that it does beat Gemini 1.5 Pro. It beats Meta's, Llama 3.1 and of course Claude 3.5 Sonic. Now, I would, you know, question that Claude 3.5 Sonnet, because on the leaderboards, consistently, from what I've heard for people diagnosing the hardest issues, the best chatbot in terms of raw intelligence is Claude 3.5 sonic. You have to remember that the LMYS arena board doesn't choose just measure things like intelligence. It measures things like how a chatbot manages to actually format its responses and how helpful the responses are in comparison to another chatbot. So always, you know, take that into consideration when you're looking at which model you want to use, because this leaderboard, while useful, it doesn't serve every purpose that you might think it does. Now, once again, this. They released the overall ELOs for the chatbot arena. This was their image. I got to be honest, it's kind of annoying having to tilt my head to kind of read. I mean, of course I don't have to tilt my head, but I mean, there's easier ways to present data, but that's not part of this video. The point here is that what we are looking at here is a remarkable release from this team. Because the thing is, is that they came into the game in terms of AI pretty late compared to everyone else. And a lot of people did count out X AI because they're a much smaller team and they don't have the necessary billions and billions of dollars and infrastructure like Meta and Google. And of course OpenAI have had, considering that those companies have been existing for quite some time. So the fact that a much smaller team on a shorter timeline has managed to reach state of the art shows us that there are still innovations to be had in terms of the many different teams that are going to be working on different. And I think this is rather fascinating because what we see here is that this is essentially their GPT4 level model. And honestly, when I used this SUS column R using it definitely did feel like it was a state of the art model. It didn't feel comparably worse in many scenarios. And I'm going to show you guys later on in the video a small demo of some of the things that this chatbot actually does well. Now what we can also see here is the win rate of GROK2 against competing models on the Chatbot arena. And we can see that it manages to get pretty much every single model apart from the Gemini 1.5 Pro. They're very new, experimental one, which is rather interesting because some people have tested it, found it to be not great, and others have tested it like myself, and have found it to be, you know, absolutely amazing. So I think one of the things I would say to you, and this is something that I've been saying in certain communities and to people that have, you know, asked me about this and is that with these large language models, all of them have different kind of areas of expertise that are going to be different for every single person. For some of your queries and some of your requests, it might be best to of course test out all of the top models to see which one kind of edges out in the competition, because there isn't just a one model for absolutely everything, although you might think that that's Claude 3.5. Some models like Gemini are more creative, some models like GPT4 are more structured in terms of their responses. And models like Claude 3.5 Sonnet are just completely raw intelligence. So I think there's definitely a mixed bag on different things here that you can completely use. And some people, interestingly enough, on their own benchmarks, have tested things like Mistral Large, which is an extremely underrated chatbot that I personally use for my daily queries, and and Llama3 14 5B that have enough Reasoning to do a lot of low level tasks really well and a lot of tasks that previously you would have used for GPT4, but now that you can use these other models for in terms of your need for intelligence. So essentially overall just decreasing your overall cost. So rather than putting all of your queries through Claude, you just put the most intelligent ones through Claude or Grok2 and then of course route the other ones to like mistral large or ollama 314 5B without actually losing any tokens. Now of course, the benchmarks have been yapping for quite some time, but you can see here that the benchmarks are a decent improvement over the Grok 1.5. We can see that there was a huge jump from Grok 1.5 to Grok 2. You can see that on the GPQA, there's a 15% jump on the MMLU, there's a 6 to 7% jump on the MMRU Pro, there's like a 25% jump on the math benchmark, 26% jump on the human Eval, there's a huge jump there. And then of course on the mmu, the Math Vista and the Doc vqa, there's a lot of jumps there as well. So overall, what we can see on just these initial first, like not comparisons, but first impressions is that we can see that there have been some stark improvements in terms of how it's managed to perform regarding these other models. And we can see here exactly where the model ranks compared to its other counterparts. And I got to be honest, for X AI, this is remarkable considering the fact that they've been somewhat behind in terms of their starting position. And when we look at these other models in terms of the other benchmarks, it is quite hard to see what is first. But we can see these blue bars here. So you can see that this one is like second, this one is third, this one is second, this is third, this is first and this is second. So overall we can see that this is definitely a mixed bag. It's not completely state of the art, but it is a model that is currently on par with state of the art. And as someone that's tested it, it does do well. Now there are some unique features about Grok 2 that most people are going to miss because they don't realize that this model is becoming increasingly more capable. One of the things that you know this model is good at is of course its image capabilities. In terms of it being able to actually look at an image and completely understand what's going on. You can See here it's able to look at this image where it says, and that is the original processor. And then of course write down exactly what is going on. So here, what we can also see is that if you are currently on X, I know a lot of people don't use the platform, it's completely up to you. But like I said before, this is, you know, Elon's way of getting people to use his, of course, social media site. And I can't hate him for that because if you spent billions of dollars on a social media site, you'd be doing anything to get people to use the platform. So if you do want to use Grok 2 and Grok 2 Mini, which is a, you know, smaller, lightweight version of the model, you definitely want to sign up to x.com because I'm not entirely sure if you're going to get instant access because before it did take a bit of time, of course, for you to get verified and then to get access to the model. Now, essentially, you know, they've got Grok 2 and Grok 2 mini. Grok 2 mini available now. And one of the most interesting things about Grok 2 Mini is the fact that it does have some kind of core reasoning capabilities that other models do suffer from, which is weird on how they manage to do that. But essentially they do have text and vision understanding, which is really nice, integrating real time information from the X platform. And what's really cool is that of course, in collaboration with the Black Forest Labs, the recent company that managed to make this Flux 1, which is a remarkable model in terms of prompt adherence, quality and photorealism, that model is going to be natively built into Grox capabilities on Twitter. So essentially, if you want to use Flux, if you don't know how to access it, there is a million ways. But if you're on Twitter and you want to easily access this model, this is going to be a way that you can do that. Now, I want to show you guys some of the capabilities of Grok 2 Mini because it is rather surprising and you might laugh at the test, but I think they're pretty humorous. So one of the things that we do know is the fact that large language models cannot count the number of letters in a word due to the tokenization. So essentially what happens is the fact that large language models don't view words the same way that we view words. So we view words as broken up by letters, but large language models view words as tokens. So for example, this word might be WO and RD for letter it might be let tr. It just completely depends. I wish I had the image on screen to explain to you how it works. But long story short, it's not broken up. Each token isn't one letter. That's not how things work. Which means that when they're counting things, they can sometimes get it wrong. But somehow I'm not entirely sure how they've done this because it doesn't seem to be like a neuro symbolic approach. It seems to be rather fast, intuitive and quick. Is that when I ask this model how many letters of a certain letter, how many are in a certain word, it manages to get it right every single time. So, for example, I can write how many letters A are in the word Andrew. Okay, so I'm going to put that in. But you can see right here, I've put three A's in because I wanted to kind of confuse the model to see if it was going to count the extra A's. And you can see here it manages to do that rather well. So this is a model that seems to have an engine that kind of reasons in a way that I'm not understanding just yet, because it didn't use step by step prompting to reason that way. And usually if it does step by step prompting. For example, with GPT4O, if you ask this, usually if you ask this for a variety of different queries, what will happen is it will usually give you the wrong answer. Then you have to say, you know, write it out step by step. The other day, this is exactly what I had to do with GPT4O. For example, in another video, I said, how many Ls are there in the word lollapalooza? You can see that they've put six instances when it's just 1, 2, 3, and of course, 4. And then essentially you can see that through prompt engineering. I said, write out the letter of each word and verify if it is an L or not, and then count that and you can see it's manually outputting the word once again, then classifying if it's an L or not, and then producing its response. Now, you can do this for a variety of different AI tasks, which is how prompt engineers manage to provide value in various different ways. And it's still something that people are gaining ground on because LMS are weird and tricky systems. So we're going to keep figuring out ways to make these chatbots more effective. But of course, with prompt engineering, once you have figured out how to get the model to output the right response, you can then share that method with everyone else. Now, what's cool about this, okay, is that we can see here that this is potentially natively built in. Now, I think the reason I don't see this model managing to do the step by step calculations is because this is Grok 2 mini, which means that this is a fast and lightweight model, which means that it's likely that this model could have done all of those previous steps within a split second. So essentially what we do have here is potentially a model that has internal chains of thought or an internal prompting strategy for the final output. If that is true, then that means that this model would definitely be smarter than other models of a similar size. But because usually what I've seen, when models are able to do that, they output better responses. So I wouldn't be surprised if this model has like a really cool system prompt where it says, think carefully through your answer. Write it down first, or yada yada, yada. So it will be kind of interesting to see what kind of way that this system prompt is. I mean, there's going to be some people on Twitter that are going to definitely try to, you know, crack this system prompt over time. I'm not sure if that's going to come out, but if it does come out, I would really love to see exactly what the model thinks before it manages to do its responses, because you really do want to understand how these models are able to respond with such high accuracy, especially in certain scenarios. So that is something that I find absolutely incredible. And of course you can say, make an image of London. So I'm just going to put make an image of London. I know that's super boring and I know that's probably not what you wanted to see, but I just wanted to show you that you can actually now use these. Wow, that looks, you know, remarkably photorealistic. That actually looks like a picture and not something that was AI generated, which is rather strange and uncanny. But nonetheless, you can see here that this is Grok 2. Now, this is the Grok 2 Mini. So I'm excited for when we have, you know, the final Grok 2, which is probably going to be even more capable. But the Grok 2 Mini seems rather effective at its current moment. Now, let me know if this completely surprised you. Did the sus column r completely throw you off guard? It seems that the information that we were getting from this Twitter account unfortunately might not be entirely true. But that just begs the question with, as to why Sam Altman did respond to this tweet. I guess some mysteries we will never know. But for now it seems that the SUS column r the model that everyone was trying to figure out is of course belonging to X AI, and it's a rather capable model from the Cracked X AI team. With that being said, if you did enjoy this video and you want to try the model or have me run any other tests, leave a comment down below and I'll see you guys in the next one.