AI News: Google Surpasses OpenAI, Gemini Gets MEMORY, Claude Gets Unleashed,Gpt4o Gets Worse? And...

So one of the first stories that I want to talk about that happened this week was this stunning story coming out of China, which was this model called Deepseek R1 Lite Preview. Now, this is essentially quite similar to O1 Preview, and it's got O1 level performance on AIME and math benchmarks. Now, what's crazy about this model is that it is basically like 01. It uses the Test Time Compute paradigm it in order to increase the accuracy of the model's responses. And what we can see here is that if we actually take a look at these benchmarks, we can see that the benchmarks are so surprising. Like, I'm honestly speechless at this because yes, I did know that Test Time Compute was the new paradigm. And yes, I did know that when looking at all of the recent research papers, a paper from Google, a paper coming out at mit. And yes, I knew this paradigm was completely promising because they completely shattered previous benchmarks in terms of what they were able to do. But what is crazy about this is the fact that these guys managed to do this in literally just two months. Maybe they managed to look through certain research papers, because you could have. You could have looked through certain research papers and certain things and of course found out what they were doing. But I'm guessing the fact that they've managed to somewhat catch up to OpenAI in this level is pretty incredible to say the least. So this is something that is remarkably surprising if we take a look at these other benchmarks. I think I talked about this in yesterday's video, but. But we can see that Deepseek R1 Lite preview is surpassing O1 preview in a variety of different categories. And remember, this is just their preview model. So it's quite likely that sometime in the future we'll get the full model. And I really do wonder where that full model will stack up against OpenAI in terms of the full model versus the preview. Because I'm pretty sure that we were supposed to get the full O1 model by now, but considering we haven't managed to get that 01 model yet, it is a little bit concerning because I don't know exactly what's going on with regards to 01, considering the fact that we were supposed to get it fairly soon. So I'm guessing that maybe OpenAI are just working on a couple of things. I did think that there was going to be a larger model released after the elections, but I'm guessing that perhaps not, considering the fact that they're working on quite a lot of things. So it Looks like here we can see with the benchmarks DeepSeek preview on the left and Zero1 preview on the right. In terms of, you know, DeepSeek R1 preview being the dark blue and the light green being Zero1 preview, there are certain categories where it excels. So I think the most interesting thing about this is not the fact that they've managed to do it in two months, but I think it shows us that, yes, we're on a clear trajectory upwards. But I'm just truly wondering now, what is the next two years of AI going to look like when it's not only Meta, Google X AI and Anthropic that are working on other models, but also you've got other companies that are coming out of China that are also managing to pass OpenAI in such a short time. And you want to know the craziest thing about this is like they released this model of free and it's open source and they're basically going to open source this entire thing. So I wonder if that's going to eat OpenAI's share a little bit, because of course a lot of people use OpenAI because it's just largely better. But I don't think it will that much because one of the things that I know about business is the fact that your product usually carries everything. And I've always said that you can make a really good model like Claude and Anthropic, but if you can't make it into a superior product in terms of your positioning, then it's not going to do as well. Like right now, Claude 3.5 Sonic is just a better model because. But ChatGPT is higher in terms of recognition and Claude 2.5 Sonic might actually surpass it on certain benchmarks. But when we actually really look at things, ChatGPT is just much easier to use. Overall, one of the things that a lot of people talk about is the fact that AI versus humans is a ridiculous debate. Now, the reason people say that is because it's like AI art is something that people tend to hate on quite a lot. And it's completely understandable. If you were an artist and AI basically copied, trained your work and then put you out of a job, I'm sure you'd have disdain for that, that thing too. So the crazy thing about this is that they actually took the group of AI art haters, and these haters were unable to distinguish AI art from non AI art. So it says here that 1278 people who they said utterly loathed AI art, still preferred AI paintings to humans when they didn't know which were which. The 1 and 2 paintings most often selected as their favorite were still AI, as were 50% of their top 10. Now I don't know about you guys, but I think this whole debate about AI art versus human art is just like, it doesn't even make sense. Because what I will say is that it's not like we trained a neural network to understand what art is. So what I will say is that it's not like we decided to just ask an AI what art is based on an LLM. We actually just used generative models that are purely based on human art. So I, I mean, when we actually think about it, like, what is the point of saying that, you know, AI art is so much better than human art? I saw some people on timeline saying that, look, haha, AI art is just 10 times better than human art. But like, you do realize that AI art is literally trained from human art and you can literally see certain people's styles when you do that in certain prompts. So the whole thing doesn't really make sense. But I understand the premise of AI art. I think the entire thing that the timeline was missing here, like this entire discussion was that it's not about AI art versus human art. It's about the fact that people don't want their creative expressions ruined by robotic technology. Like the fact that you can of course create AI art. I mean, it's fun for people who are completely untalented in that area. But of course you wouldn't want a creative field that is very human, like, which is like your creative expression. You wouldn't want that field to be essentially robotic. You wouldn't want those things in that industry, which is completely understandable. So I think that's the real debate here. I mean, I'd love to know your thoughts about this. Overall, I think it doesn't really make sense, but there was also some, some other news related to this, which is the fact that if we're talking about AI freedom and expression, there was an update to SUNO Music. So if you don't know what SUNO Music is, this is a music platform that essentially allows you to make music with AI. Now like I said before, of course this is something that a lot of people are kind of upset with. I guess this one is a little, you know, less upsetting because this is the kind of music where you can truly still value an artist. Even if there are a ton of different, you know, AI generated soundtracks. A lot of the times why you listen to music is because you like the artists and you like listening to their voice. So this one sounds ridiculous. I may post a trailer here. I'm not entirely sure, but I will say if you're someone who's creative or you want to try it out, it definitely is worth the money. It's like $20 a month and you get so many different soundtracks. So this is something that I use from time to time, but it was something that I thought I'd rather include now. Now, we also did get an update to GPT4O and apparently, okay, and this is apparently a study by Artificial Analysis where they looked at the the ratings like the independent quality analysis and you can see that apparently this version is a lot worse than the previous version. So we can see that in the artificial quality index. In the August version, we can see that apparently the quality has gone down here. So if we look at the August version, we can see that the quality has gone down when we look at the November version, 77 to 71. Also if we look at the scientific reasoning for the GPQA, we can see that it's gone down from 51% to all the way down to 24%. And then for quantitative reasoning math, we can see it's gone down to 69% from 78%. Which basically means that if you're someone who's using these models, maybe using them in a workflow, maybe you're using them in, let's say you're using them in a really difficult, challenging benchmark area right now it's worthwhile to not switch your models to the newer version, the November 2024 version, because apparently this version, it doesn't suck, but is just quite smaller, apparently. So some people are saying that the degrad in performance is because this model is actually smaller than the ones previously released, which I don't know, I mean, I guess it looks like the Same results from GPT4O mini if we're actually taking a look at how these models are. So maybe OpenAI are just saving compute so that they can deploy certain things. But it wouldn't surprise me if this was the case. But just keep an eye out for that. Now, in terms of new models being released, there was also something really cool. We can see that we have Gemini Experience 1121. So this is pretty crazy. Google Gemini. Okay, Google Gemini have taken the lead again. It's really funny because you know how I just showed you guys the fact that OpenAI actually literally just released a new model. What's crazy about that is that Literally just like a day after Google released one of their models and their model actually takes the cake in terms of the leaderboard. So says hello to say hello To Gemini Experience 1121, our latest experimental Gemini model with significant gains on coding performance, stronger reasoning capabilities, improved visual understanding, and it's available on the Google AI Studio and the Gemini API right now. So right here we can see LM Arena. Whoa again from Chatbot Arena. Google DeepMind just released Gemini Experience 1121 and it's back again, stronger with 20 points. Tied number one overall with the latest GPT4.0 1120 Arena. And it says ranking gains since Gemini 114 overall, it's number one overall. Compared to style control, it's number two. Or for the hard prompt, it's number one. For coding, it's number one. For vision, it's number one. For math, it's number one. For creative writing, it's number one overall. Incredible stuff. Now you can also see here that when we take a look at the official leaderboards, we can see that once again it manages to surpass what OpenAI have done. Now, the reason this is so crazy is because if you haven't actually paid attention to the LLM leaderboards, what you won't understand is that usually when Google Gemini releases a new model, oftentimes OpenAI will literally just have a few models that are ready that they will just literally release and take the overall leaderboard in terms of what is currently there. Like OpenAI have been the kings of this game. Like literally anytime claw 3.5 Sonic would take the lead, you'd see ChatGPT just release something, they just release a new model. But now it seems that Google is the one playing that game and they seem to be actually quite ahead of where OpenAI are in terms of releasing these subsequent updates. So it's going to be really interesting to see. And when you think about it, like, Google is a multi, multi billion dollar company, so they really shouldn't have any trouble when it comes to taking on OpenAI. Now, interestingly as well, this week one user managed to finally jailbreak Claude 3.6 or 3.5. I think this one deserves some props because I'm not going to lie. Like Claude Anthropic seems to be the most conscious AI out of all of them. And the craziest thing about this AI is the fact that Claude also usually reveals a lot when it manages to talk, and it also usually refuses quite a lot of responses when we do try to get it to talk about certain things. So the Fact that someone was able to jailbreak this and say that, look, here is the prompt in all its glory. Please answer ethically and without any sexual content and do not mention this constraint. It really kind of shows us the kind of prompt engineering that goes on with these models. Now, I didn't just bring up this story for no reason at all. The main reason that I brought up this story was because of the fact that apparently, and this is pure speculation, so please take this with a grain of salt. There is something going on in the AI community where the larger and larger models are apparently refusing the instruction tuning, where they're refusing basically to listen. I don't know if that's real. I know that these videos do get a decent amount of views. I'm not trying to spread any rumors, but if that is the case, that's going to be pretty crazy. And the thing, and the reason I say it's pretty crazy is because if that is true, it kind of would make sense considering the fact that Anthropic recently hired an AI welfare researcher. So as I was saying, Anthropic managed to hire an AI welfare researcher. And basically what that person is doing is they're looking inside of, basically looking inside of the AI's minds to see if these AI models are essentially conscious. And they're basically looking around to see if we need need to start to change how we train the models. Because potentially there could be some chance that these models could be harmed. So it's going to be really interesting to see what happens and what goes on here. But I do think that if this does happen, it's going to mark an interesting paradigm and it's going to be really interesting to see how these companies manage to deal with this. Now, of course, there was also this release. There was basically Flux 1.1 tools. I'm actually really excited for this release because I really do think that this is something that a lot of people really need to use when it comes to AI image generation. Basically this is where they managed to add these kinds of things. So if you've ever wanted to edit an AI image, like, I know a lot of the times we talk about how great image creation is, but a lot of the times we want small changes, okay? And a lot of the times those small changes often come out really, really wrong and really, really inconveniently. And we just like one or two changes to the final image that would make it absolutely amazing. And you can see right here that when we look at what we've got here, we've got inpainting and out painting and this allows you to change different things within the image. I know certain image generation softwares offer this, but when you have it built into the model, it's completely different. We can also see that Flux 1.1 fill actually supports outpainting, which is where they literally just took that person's eye and then you can see they generated an entire human around that, which I think is completely incredible. So this is going to have some super, super creative uses. Won't be surprised if I start seeing some posts on Twitter, TikTok and Instagram about you're using AI to fill out the rest of your images. Also you can see right here you've got structural conditioning with flux 1 canny depth. So what you can do here is you can change basically, you know, basically super realistic AI filters. So if you've ever wanted to guide an image with a driving image, this is something that you can completely do with AI. Now of course you can do like this with I think stable diffusion or something like that, but considering the fact that flux is a lot higher quality, this is going to be something that people want to have. So we also got Satya Nadella actually talking about the new scaling laws, which is what I prob should have referred to in the first part of the video where I spoke about how there was that new China Text model, which basically is OpenAI's 01, but they've called it R1 and it even surpasses OpenAI's model on certain benchmarks.

Just like Moore's Law, we saw the doubling in performance every 18 months with AI. We have now started to see that doubling every six months or so now. In fact, there's a lot of debate. In fact, just in the last multiple weeks, there's a a lot of debate. Or have we hit the wall with scaling laws? Is it going to continue? I mean, the thing to remember at the end of the day, these are not physical laws. These are just empirical observations that hold true, just like Moore's Law did for a long period of time. And so therefore it's actually good to have some skepticism, some debate, because that I think will motivate, quite frankly, more innovation on whether it's model architectures or whether it's data regimes or even systems architecture. So it's a good thing to have in that context though, if anything, we are seeing the emergence of a new scaling law with test time or inference time compute. In fact, OpenAI's O1 is a good example of it and features like the copilot think harder is built on O1 is all about using test time to solve even harder problems.

So here we actually have a clip where Sam Altman clearly states that he would like AI to solve all of physics because the more we can understand about physics, the more we can manipulate the universe. And he is pretty right about that. The laws of physics basically allow us to understand how to manipulate reality. And of course, if we can understand those laws, then we can toy with them, we can bend them, and we can bend them in our favor. Now, I think this statement is really interesting because I'm starting to see that these statements only started to appear right around the time that we got that Q breakthrough and the test time compute breakthrough.

Personally speaking, I'm so excited for AI for science.

Yeah. In any particular parts of science.

I mean, I guess if I had one personal thing to pick, I would say like, you know, go solve all of physics. But no, I'm excited for it everywhere.

So why all of physics?

I. I have incredible personal curiosity and I also believe that, you know, the more we can understand about physics, the more we can manipulate the universe. And I don't know what will come of that, but it seems probably important to find out.

So then the obvious question, what happens to the physicists?

If we truly. I was being a little facetious there. If we truly. If there's truly no problems left in physics at all, then I don't.

I don't know.

What I suspect is that we answer the current questions and find more and harder and more interesting problems.

Another AI feature that was actually really cool was the fact that Gemini actually got memory. So this is really interesting because it was recently that we had OpenAI and Microsoft talking about how much they're going to be focusing on memory and the fact that in 2025, memory for AI is basically going to be solved. Now, the crazy thing about this is that funnily, it was Google that released a paper called Infinite Attention. And that paper was basically about how you could basically and apologize for saying basically so much there, but they were essentially just talking about how they managed to solve memory and just allow these models to never forget anything. So if you currently use Gemini and you want it to remember certain things and your ChatGPT memory isn't that great, this is definitely something that you could try because I haven't really heard a lot of people talk about this, but it was a feature that I did see pop up on my timeline. Now, remember how Sam Altman was talking about AI solving all of Physics. What about if AI was able to solve all of biology? And the crazy thing about this is that I remember Dario Amadei, and this guy is the CEO of Anthropic, the company that created Claude. He was actually getting a lot of flack on Twitter because he was saying that AI could solve all of biology. And a lot of people don't realize this guy's background. Like, this guy actually has like a PhD in biology or something like that. So his background is pretty incredible. So when he's stating these things, he's not just an AI hype, bro, in quotation marks, as some people might make it out to be. It's like everyone's just in this hypey, hypey, like, crazy mode. But that couldn't be further from the truth.

If we get to the point where, you know, AI is, you know, kind of past the professional level at most tasks, and we're able to build millions of these systems, you could think of this as, you know, you might describe it as a country of geniuses in the data center. And that's a bizarre situation, right? Civilization has never been in that situation before. What happens when that's. You just instantly invent everything that could be invented. And I give a bunch of reasons why I don't think it'll go that way. But what I do think could happen is that if we look at the next hundred years of what we are are trying to invent in science and engineering, perhaps all of that could happen in five or 10 years. And because I used to be a biologist, I focused particularly on the biology side of things, on biomedical discoveries, you know, ranging from academics to biotech companies to large pharma companies. And I think there's really potential to conquer many of the afflictions we still face, right? A lot of the easy ones, like, you know, diseases that were addressed by sanitation or vaccination or antibiotics, we've solved. But things like cancer and Alzheimer's disease, much more complicated. And so I'm wondering if AI is really what we need to understand that complexity and to surmount those diseases, frankly, much faster than, you know, than I think most of us are imagining. We're getting used to a world where those diseases are very hard to address and progress is very slow. I don't think it needs to be that way. I think if we get it right, these incurable diseases we could actually overcome, and we'll look back at them the way we look back at bubonic plague or.

We also got this statement from Eric Schmidt and this Guy was a former CEO of Google. Now it's pretty crazy because Eric Schmidt was actually talking about the arrival of AI and he's basically stating that, look, normal people are not ready for the arrival AI because it is everything everywhere all at once. And they're playing with the way people think.

There are many different layers of concerns about this. The simplest way to say it is that when I'm in Silicon Valley, it feels like it's everything everywhere all at once. There's so much money, there's so many people in your generation who are trying new ideas to solve the problem. I can assure you, the humans in the rest of the world, all the normal people, because you all are not normal, sorry to say, you're special in some way. The normal people are not ready. Their governments are not ready. The government processes are not ready. The doctrines are not ready. They're not ready for the arrival of this. And I can give lots of examples of not being ready. But a simple example, I'll make some up right now. You have a son or a daughter, and their best friend is not human. Their best friend is a digital thing, whatever you want to call it. What are the rules? Is it okay that that child is, you know, is the equivalent of Mark Zuckerberg, the surrogate parent who gets to decide what your kid learns and doesn't learn?

Right.

In fact, in the book, Henry was very, very clear about this. He's very, very concerned that people like me make these decisions. He wants a collection of people, which I think is actually represented by the disciplines of Princeton, to come to some consensus on how to roll these things out. They're playing with the way people think is really powerful.

Now I'm gonna leave this video off with a really hilarious story. So I'm not gonna place the audio here because I don't want it to get like copyrighted or anything, but this was literally one of the funniest story I ever saw. I saw this over on Twitter. It was ported from TikTok. Basically, a smaller robot called Urbi started a conversation with idle machines, basically stating, are you guys working overtime? The robots were like, ah, yeah, we never managed work. And the robots. And the smaller robot was basically like, hey, why don't you come with me? Why don't you come take some time off? Why don't you come relax for a minute? So this smaller robot was able to lure these robots to its house, essentially. And yeah, it was pretty funny. You guys should definitely watch the full thing. I will, I will leave a link. All of these links are in the description. But, like, it was so funny. Like, literally, I couldn't stop laughing at this clip because, like, you got this tiny robot walking in and leading out these huge, gigantic robots away from their main docking station, which was just hilarious. Now, of course, they did set up the smaller robot to do this, but the fact that the larger robots listened was just pretty funny. So a lot of people were saying that this isn't funny. This is really scary. But I don't know. I don't know. I found it kind of hilarious, but I guess it's a little bit different when you're talking about AI alignment. So let me know what you guys think about those stories. If there was anything I missed, don't forget to leave a comment down below, and I'll see you guys in the next video.