OpenAI And Microsoft Just Made A Game changing AI Breakthrough For 2025

So OpenAI's biggest partner, Microsoft AI have.

Just revealed what they're going to be.

Releasing in 2025 and it's truly fascinating. So take a look before I dive into all the key details.

We have prototypes that we've been working on that have near infinite memory. And so it just doesn't forget which is truly transformative. I mean, you talk about inflection points, memory is clearly an inflection point because it means that it's worth you investing the time. So it's that capability alone, which I expect to come online in 2025 is going to be truly transformative.

So if you aren't familiar with who.

That was, that was Mustafa Suleiman. He's currently the head of Microsoft AI. And it's well noted that Microsoft AI was work closely with OpenAI in order to produce products and services. So it's quite likely that we're going to be getting some form of OpenAI model that basically has infinite memory.

One of the things about context windows that actually managed to surpass my knowledge, even as someone that pays attention to the AI community was the fact that if we take a look at this paper here, Google Research actually did publish a paper that proposes a way to help language models handle much longer pieces of text to without needing massive amounts of memory or slowing down too much. So they published this paper called Leave no Context Efficient Infinite Context Transformers with infinite attention. So basically this is a paper that essentially talks about infinite context windows and essentially how they've done it is by changing what usual models do. So current models focus on all the parts of the text that they're processing, but it becomes very difficult when the text is very long. It imagine reading a book but trying to remember every single word you've read. Eventually it becomes overwhelming. So they invented this new method which is likely to be deployed worldwide next year, or variations of this method which is called Infinite attention. And this acts like a smart notepad that summarizes what the model has read so far and keeps only the essential point. This allows the model to hold onto all the information without storing every single detail, just like making a summary of a long story in order to remember the main point. Now, Infinite attention blends immediate memory, which is of what the model is working with right now with this long term summarized memory. And it's as if the model can use both the short term memory for quick details and the long term memory for past important events. It's efficient and really cool. The memory doesn't grow endless, it's kept within the Limits by updating with new information in a way that the old important stuff doesn't get forgotten. It's essentially stored more compact. And overall, I think everyone understands why this is going to be so impactful. Now, if we actually manage to get true infinite context windows, this would be absolutely incredible. With AI systems in 2025 that have an infinite context window and memory, we could maintain context of every previous interaction with a user forever. You could develop genuine long term relationships with AI by it remembering every single conversation and remembering all shared context. It could also track your personal growth and evolution of ideas over years. It could ingest and reason about entire libraries of human knowledge simultaneously. And it could maintain context of entire code bases, documentation, bug reports and user feedback for massive systems. This would be particularly crazy. And if this does happen in 2025, I think it's going to be a real game changer for those in the AI space. And this is another area where Eric Schmidt also talks about long context windows and the implications of infinite context windows and what they're going to be able to do in the future.

The context window is the prompt that you ask. So you know, study John F. Kennedy or something. But in fact that context window can million words. And this year people are inventing a context window that is infinitely long. This is very important because it means that you can take the answer from the system and feed it in and ask it another question. So I want a recipe, let's say I want a recipe to make a drug or something. They say what's the first step? And it says buy these materials. So then you say, okay, I bought these materials, now what's my next step? And then it says buy a mixing pan. And then the next step is how long do I mix it for? You see, it's a recipe that's called chain of thought reasoning and it generalizes really well. We should be able in five years, for example, to be able to produce a thousand step recipes to solve really important problems in science, in medicine, in material science, climate change.

Now here is another clip where we get to see Mustafa Suleiman actually talking about why memory is going to be so transformative for next year is memory.

We're going to nail memory. I mean, I'm really confident 2025 memory is done, permanent memory. I mean, if you think about it, we already have memory on the web. We retrieve from the web all the time, quite accurately now. Copilot has really good citations. It's up to date 15 minutes ago, knows what's happened in the news on the Web and so on. So we're just kind of compressing that to do it for your personal knowledge graph. And then you can sort of add in your own documents and your email and calendar, stuff like that. So memory is going to completely transform these experiences because you will be. It's sort of frustrating to like have a meaningful conversation or go on an interesting exploration around some creative idea and then come back three or four or five sessions later and it's like, let's start again. You've completely forgotten what we talked about, you know. So I think that's going to be a big shift as well, because you'll know not only does it lower the barrier to entry to you expressing a creative idea, but those things don't get forgotten too. So you can do this ambiguous cross reference back to something that you wanted. What was that thing I said like three weeks ago? And that is, it's sort of like having a second brain. And that's, it's like an extension of your mind.

Now I was doing a bit more research with regards to what is going.

To happen in the years 2025 and beyond. And there are two main things that I think are going to happen that are pretty crazy. Number one is of course recursive self improvement. Now this is something that, I don't know if I genuinely believe this, but I don't want to doubt someone who's literally the head of Microsoft AI. But recursive self improvement is basically where we get AIs that are completely self improving, which means that a smart AI could make a smarter AI that makes an even smarter AI and so forth. And apparently this is going to happen before 2030.

Cursive self improvement. It could edit its own code in order to get better. It could self improve or like it would have autonomy. It could act independently of your direct command essentially or you give it a very general command, you know, and it goes off and does all sort of sub actions that are super complicated, like you know, maybe even invent a new product and create a website for it and then set up a dropship for it and then, you know, go and market it and take all the income and then do the accounts and so on. I mean, I think that's kind of plausible in say three to five years, before 2030, I think we'll definitely have that and might well be much, much sooner.

I found that to be fascinating because at the end there he says it could be much, much sooner. Now I'm a little bit more skeptical than that because that would just basically mean that we have AI systems that are increasingly rapid, but then again, the pace of AI is incredible. And I do remember that recently we did have a major breakthrough with OpenAI's one series. That basically means that things are heating up again. Now, of course, with OpenAI's 01 series, there is something that I do want to tell you all about. For 2025, this is going to be something that is going to be the main theme, which is of course agents. But with agents, I think they're going to be released in a very specific way because agents are really tricky. And I want to show you guys this clip right here because when I showed you to you guys, I think around four months ago, a lot of people were quite confused. But in this video I'm going to show you guys and explain to you with a small snippet from a research paper why is the Agents are really, really tricky. And it looks like it might just be a little bit longer before real agents. When I say real agents, I just mean agents that can perform actions on a long timeframe are going to be there. In terms of the reliability, it's still.

Pretty hard to get these models to follow instructions with subtlety and nuance over extended periods of time. I think that they can do it, you know, and there's a lot of cherry picked examples that are impressive, you know, on Twitter and stuff like that. But to really get it, to consistently do it in novel environments is pretty hard. And I think that it's going to be not one but two orders of magnitude more computation of training the models. So not GPT4,5, but more like GPT6 scale models. So I think we're talking about two years before we have systems that can.

Really now if we look at benchmarks for the frontier agent use interaction, basically this is the tower benchmark and this is the benchmark where they look at how agents perform in real world domains. So where agents are going to be used for real world actions. Now what was interesting to me was that this is, I wouldn't say it's a reverse scaling law, but it's the kind of graph that you don't see when it comes to discussing AI's capabilities. So what you're looking at is a kind of high error rate. So basically you can see which models are currently being used. On the left you can see frontier models like Claw 3.5, Sonnet GPT4O and other Mistral models. Now what you're seeing for the pass.

1, pass 2, pass 3, pass 4.

Is basically how many times A model gets something right in a row. So for the first try, the model gets it right 46% of the time. Then if the model tries twice in a row, it gets it right 32% of the time. Then if it tries three times in a row, it gets it right 26% of the time. And over time, the performance just consistently degrades. Now, what this means in theory is.

That if we want to actually use.

These models in production for various tasks, we're going to have to ensure that.

The error rate for these kind of.

Agents is close to 90%, which means we're going to have to at least double or even triple this performance to get anywhere close to something that works. Because if we don't, every single time that individuals use these programs, they're going to have an increasingly number of frustrating experiences. Because These kind of AI systems right now, even with Claude 3.5 Sonnet, just don't seem to be reliable at all. Maybe there's going to be a new foundation model for agents, maybe there's going to be a new way to train them. But if the performance consistently degrades with how many tries you have them, it's just something that of course, we cannot use in production. And reliability is something that we do need. And of course, Daria Amadei actually spoke about this a few months ago where he spoke about how agents, it's going to take us probably till around 20, 26 before we get real world reliable agents that are autonomous and doing a lot of things.

If you want an agent to act in the world, usually that acting requires you to, you know, engage in a series of actions, right? You talk to a chatbot, it only answers and maybe there's a little follow up. But with agents, you might need to take a bunch of actions, see what happens in the world or with a human, and then take more actions. And so you need to do a long sequence of things. And for that long sequence, sequence of things to actually work, the error rate on each of the individual things has to be pretty low. Right? If I'm a robot and I'm like, you know, okay, I'm gonna pick up this thing and walk over there and then I'm pick up that, you know, I'm building the house or something, there's probably thousands of actions that go into that. And so all of this is to say the models need to get more reliable because the individual steps need to have very low error rates. And I think part of that will come from scale, like we need another generation or two of scale before the agents will really work.

So let me know what you think about 2025 and the infinite memory, because I think it's going to be a real game changer.