China takes the LEAD! New AI Model STUNS OPENAI Sense time V5.0 Beats GPT4 On All Benchmarks

So around two days ago, there was a rather interesting development. There was something in China that I think you should all be aware of because it changes the dynamic of the AI race, which leads me to believe now that currently China is hot on the heels of the rest of the world in terms of AI developments. Now it seems that the US has been leading the charge in terms of state of the art models, but recently this is the most interesting thing we've seen in recent times. So, so let's take a look at exactly what China has been doing and what this announcement is. So essentially what we have here is we have a situation where China has potentially taken the lead with their new model. Two days ago, SenseTime launched Sense Nova 5.0, which according to this report, actually beats GPT4 on nearly all benchmarks. And there are actually a variety of different things that are truly surprising about this. It isn't just some hyped up GPT4 thing. If the claims are true and we're going to dive into them, this means that things are truly, truly ramping up. And it seems that across the board, from different nations and different countries, we're definitely going to be seeing increased competition in terms of what AI systems are capable of. So we can see here that they've essentially compared this to GPT4 Turbo. But there are a lot of things that this presentation dives into. And bear in mind that the majority of this is in Chinese. But, but I've done my best to translate most of this so that you can completely understand exactly what's going on. So you can see here that their sense Nova 5.0 actually looks pretty decent. So on the left you can see it states that it is hybrid used. In the middle they state that it is trained on over 10 billion tokens. The inference actually supports up to 200,000 tokens, which is rather interesting because we've seen longer and longer context windows. So, so a 200k context window is something that is, I wouldn't say surprising, but it does show that of course things are going well. And of course the main statement from this is that performance exceeds GPT4 Turbo. Now the reason, as you all know, why GPT4 Turbo is highly touted as the state of the art model at that time. And that's why people constantly, constantly, constantly try to one up that in terms of the benchmarks. So this was the start of the presentation, but things do get a lot more interesting and I'm going to show you all exactly what that is. So there was one screenshot here where I'm not entirely sure if they actually had these AI systems hooked up to a game. But the text states that a live demonstration was conducted on the comparison of multiple functions of their model and GPT4, including creative writing, logical reasoning, diagrams, image understanding and calculations of food calories based on pictures. And this is what the text actually states here. But we will get into the benchmarks, but I just wanted to include this for full transparency in order to show off the muscles of the larger models. SenseTime also played the King of Fighters and at first the green player GPT4 had the slight upper hand but was quickly overwhelmed by the red player Sense Chat's light various combos. So I'm not sure if they're actually talking about this as if it's some kind of metaphor for their system beating GPT4 or if they actually hooked it up to a Street Fighter style game. And Sense Chat Light was able to best GPT4 if you are someone who can translate this more effectively. Because I did honestly put this through multiple different AI systems and the result was frequently the same. Now, now here's where things start to get interesting and here's where things are really, really, really fascinating because these are obviously the benchmarks for this system. You can see Sense Chat version 5 is over here on the right hand side. This is sense chat version 5. And then of course what they actually do do and this, you know, pay attention here because I'm going to, you know, say some things and I'm going to recall them. So you can see right here that they actually do Compare this to GPT4 Turbo. And that actually is to the eagle eyed viewers that it is the 1106. This isn't actually the most recent one, but it's not also a most outdated one as well. And we can see that the only benchmarks that it surpasses that model sense chat version 5 on is of course the math zero shot benchmark right here and we can see that it's highlighted in the 61% area and we can see that in this benchmark right here as well. So we can basically see that currently what we're looking at is a situation where across all of the benchmarks compared to GPT4 Turbo, one of the most recent models, I don't know why they didn't do the most recent one. I'm guessing just based on where they are in their testing. And of course we do have a comparison with llama 370 billion parameters instruct. We can see that the differences from GPT4 are honestly, I wouldn't say quite stark, but it doesn't just seem to be a simply percentage gain. Like for example, on the mmlu we can see that this one is rather small. But. But on the cmmlu I'm guessing that this is a different version of the mmlu, perhaps the Chinese mmlu. Honestly, I'm not entirely sure we can see that it increases. But you can see, like I said on the different benchmarks here, the gains don't seem to be completely incremental because if we actually look at the differences, this one is at 80 and this one is at 93.61. So I think what we're looking at here and of course some of these benchmarks aren't all on GPT4 as in when I was actually checking some of the other models, I couldn't find things like the race, I couldn't find things like the natural questions on other models and that's like Gemini Pro and Claude and I do actually test them against it and I'll get to that later on in the video. But essentially I think this is rather fascinating that, you know, China has, I would say surpassed GPT4 Turbo in their model. And what I would like to see from this model, honestly is if, you know, people could potentially test this model because it would be interesting to see how it actually compares in terms of users. And what I mean by that is, I mean that whilst yes, currently on this chart, what you're currently seeing is GPT4 only winning in two out of the several different categories here. And also even llama3 we do know that there is a different benchmark which is rather, rather important. And the benchmark that I find to be one of the most interesting because it's not just based on the instance where things could be fine tuned on the actual benchmarks in terms of just trying to get something that beats things on benchmarks but isn't actually useful on a day to day basis. You can see here that we do have GPT4 1106 on the chatbot Arena. As I've stated previously, the Chatbot arena is different because it ranks models based on their arena elo, which which is determined by votes against other systems. And basically all this means is that people test AI systems side by side and their ELO increases when people vote for that system being the most useful. And it's a blind test so you don't know which system is which and then over time you get to see which system is actually useful on a day to day basis. Answering a variety of different questions without the bias of looking at other systems. So it's something that I think is very useful and we can see that GPT4 1106 is basically at rank number one. And of course, like I said, the new one does have a slightly higher ELO rating. But I still think that this is rather, rather impressive for what we're seeing here. Now, something that I also did want to test was I wanted to test this against other models because one thing I realized when looking at this was I was like, okay, whilst this, this is good, there are other state of the art systems out there because yes, GPT4 Turbo 1106 is state of the art, but GPT4 Turbo isn't the only one. And GPT3 Turbo and Llama 3. 70 billion parameters aren't the only ones there. Now the other one that most people might be thinking about, and maybe you're not thinking about this, but I certainly am, is I'm thinking about Claude 3. Okay, so Claude 3, if you don't know, it's a model, a state of the art model and it was released by Anthropic. Now, I've done this a little bit sloppily, but let me just explain to you what you're currently looking at. So all you're looking at is Claude 3's benchmarks and we can see that it is compared to GPT4 in this tab right here and essentially across the board, completely, Claude 3's Opus model does surpass the benchmarks of GPT4. Now what this does mean here, what I've actually done here is I've actually circled these areas where Claude 3 retains its leadership and in terms of how far it is, but I've added X's here because these are areas where Sense Chat V5 actually beats Claude Opus. So in the math benchmarks, which is math problem solving, it does better than Claude Opus. There we can see it's 61.9 and this one is 60.1. So that's where Sense Chat v5 reign supreme. Then of course, we can see here on the Common Sense knowledge, the HELA Swag, this is 95.4%. And then for here on the Common Sense hello swag, we can see that theirs is actually at 97%. So this is actually a huge improvement. We can see that 95.4% to 97.52% is a big improvement. However, CLAUDE actually retains the leadership benchmarks in these areas where we can see the mmlu, the graduate level reasoning, the grade school math, but something that I do want to add. While CS Claude 3 does have the best on these benchmarks, and I would have to be honest and state that it isn't far off. Like for example, if we look at the code on the human evaluation at 84.9%, or for example, if we look at the Big Bench hard, we can see it's currently at 82.98%, whereas on the big bench hard it's at 86.8%. So I mean, it is pretty interesting to see how these differences are. But the long story short is that basically since chat V5 surpasses GPT4 Turbo, Claude 3 opus surpasses GPT4 Turbo. But in terms of the actual benchmarks, the ones that you do want to care about, according to the Arena, GPT4 Turbo's most recent version retains its leadership spot. Even if the benchmarks that you know, are benchmarks by like math and coding and stuff don't appear to be correct. People seem to be having a great time with GPT4 Turbo, firmly followed by Claude 3 Opus, and then Google's Gemini 1.5 Pro is in the API. Now, I think this is still rather impressive because it seems that this company has been working very quietly and very diligently in stealth because this is something that did catch people off guard. Now there are some other things that are actually really surprising and one of the most surprising things about this was something in regards to the smaller models, because their smaller model is insane and it's so crazy to the point where I'm wondering to see if some of these things are actually even legit. So one of the things they actually talk about is a writing task. And here's where they compare GPT4. You can see right here GPT4 to Sense Chat V5, and right here this is where, and funnily enough, this actually does look like the Chatbot Arena. So I'm wondering if in a couple of days we are actually going to get Sense Chat V5 in the arena. But they talk about how essentially the summary of this, what actually happened here was that they basically had a task to write a college entrance exam essay on innovation based on the classic Chinese novel Dream of the Red Chamber. And the author notes that GPT4's writing style tends to be more rigid and structured, using repetitive phrases. And in contrast, Sense Nova's 5.0 exhibits more free flowing and divergent writing style, drawing upon a wide range of Chinese cultural and literary references from ancient times to the modern Internet age. Long story short, they're basically trying to say that if we look at the writing tasks in these examples that they've included GPT4 loses. Now, there was also another one right here where they put GPT4's model against SenseChat v5 in, I guess you could say a logic reasoning task. And they were trying to compare this in terms of what this model would do when it was trying to figure out the amount of coffee and water consumed. And honestly, it's pretty hard to decipher what's going on here because like I said before, it's a translation, so the interpretation might actually not be correct. And I'm not going to get into the details of that. But the long story short here was that GPT4 wasn't able to realize exactly what was going on. But since chat v5, the reasoning steps provided the correct answer, I will leave the prompt and the explanation in the description. But it's just very confusing trying to get a logical reasoning starts correct. Because if you've ever seen the examples of these, you know, incredibly confusing tasks, one word can truly change the entire outcome of the answer. And it's important to get the translations 100% because even if you get them 99%, you can completely fail the question. So this is probably the most fascinating part of the thing when I was looking through it in terms of the benchmarks and in fact that's actually not true. It's actually coming up soon. But when? Right here. And we will get to the most interesting part here. But right here, this is where we take a look at their visual recognition systems. And of course you can see it is compared to Google's Gemini provision, OpenAI's GPT4 vision, Quen's VL Max, which is a vision system, and of course the Intern LM X Composer two VL and Step one Vision. So there are all of these different ones compared on these benchmarks and we can see, you know, from what we've seen here that the top one does surpass them. Now one thing that I also saw that was really, really cool was that they also did have image generation and by the looks of things, it seems to be very, very photorealistic. So essentially the text states that Sense 5.0 sets a new benchmarks in the term of AI powered image generation, demonstrated by its impressive performance in generating nuanced and lifelike portraits. The following example illustrates the AI's capability to create text to image visuals. And essentially using the prompt, a vibrant Asian female portrait with beauty makeup, casual hair, a smile natural with movie grade Quality presenting different expressions and styles, Sense Nova 5.0 showcases its sophisticated interpretation of textual descriptions and its ability to generate diverse facial expressions and styles that carry the subtlety and richness of film grade portraits. The that description is pretty good, but I've got to be honest, if this is their text to image generation model, I have to say that whilst yes, Mid Journey might be on this level, this is definitely something that looks really, really good. I mean we can't say that this doesn't look completely realistic. And as for the prompt, I think this is something that does do it really, really well. Now of course I'm guessing that they did compare this to other systems right here. In fact, it isn't entirely clear but, but I'm guessing that this, these are other systems. But either way, this one right here does look really, really realistic and I have to say that this is pretty, pretty impressive. Now here's where we talk about the most interesting bit because like I said before, we were previously supposed to, but you know, I was, you know, thought that the slide was the other slide. Now essentially this is where they talk about their smaller models, they're more compact models that they can use for many different things. Now you can see that the one they've highlighted is Sense Chat lite, which is 1.8 billion parameters in size. And it does do a lot better than others at the similar size we can see compared to the Gemma 2 billion parameters by Google, it completely destroys it. We can see that llama2.13 billion parameters, it completely does better than it. But one thing that I didn't really understand from this and was pretty confusing was of course the benchmarks, because they didn't actually do this on traditional benchmarks, they did this on a different kind of benchmark. So when this text on this left column was actually translated, the benchmarks that we got were, I guess you could say the words we got were comprehensive score, language comprehension, creativity, reasoning and the average overall. So one thing that I do think is quite surprising here is that 1.8 billion parameter model does showcase incredible capabilities because time and time again we've seen that trend to be increased. Now one critique that I do have here is that they didn't compare this to Microsoft's Phi 3 and I'm guessing that, you know, Phi 3 was literally just released, so I guess that's okay. But they didn't actually Compare this to llama 3. You can see here that they compared this to llama 2. 7 billion parameters. And of course they state that it beats it. And you know, beats Google's and you know, the other ones like Quen's 1.5. But, but even what they actually did was if we actually look at the original, you know, parameters, we can see they actually did compare llama 370 billion instruct. So I'm guessing okay, and this is just a guess, maybe it isn't true that if the original system beats llama 3, I'm guessing that this smaller system doesn't beat llama 3 because they haven't included it for whatever reasons and I'm pretty sure they could have. So I think this is still pretty interesting because they've got a smaller model that is, you know, levels above these other smaller models. But the point here is that based on some other things that I've seen as well, you know, they actually do talk about image generation. And I got to be honest, the text here is a little bit confusing. I have tried to understand the context of what it's trying to explain, but it does get a little bit confusing sometimes. And I will leave a link to the article. One thing that I actually do talk about that they say that, you know, this is going to be a calorie assistant and you can use it to submit images and, and completely understand your calories. And I think right now the AI space is heating up quite a lot. And this is not going to be the only company coming out of China to present their models that do take us by surprise. But so far this seems like a very interesting update. Now what's also interesting is that the company's shares soared more than 30% after announcing its latest generative AI model. So this was something that made the company's stock price jump 30%. But. So maybe it could be argued that the benchmarks that they're talking about aren't as good. But the only way for us to know is, of course, for us to test it in the arena for other people to do their independent evaluations. And of course things are going to be a little bit different because I'm guessing that this is fine tuned on the Chinese language, so the translations might not be as completely accurate. So I'm guessing maybe if they make an English model that might be as good or better. But of course this is going to be something that we would have to see. So let me know what you think about SenseTime's new model. And, and if we think China is going to catch up to the US competitors and how this actually impacts the landscape, I think this is of course, some very fascinating competition because I think we're going to continue to see, you know, models and different companies pour millions and billions of dollars into this industry because like I said before, I think we are now in that terminal race condition where if one company stops, then they are pretty much giving up the list.