OpenAI's NEW QStar Was Just LEAKED! (Self Improving AI) - Project STRAWBERRY

So OpenAI are working on a new reasoning technology under the code name Strawberry. What's most fascinating about this technology is that it is actually previously called Q. So finally we do get some new details, some new information on the top secret project at OpenAI. So let's take a look at what this project is, how it actually works, and some of the research papers that show us at least some hints towards what we might be looking at with Project Strawberry. So you can see right here that this is an article released not too long ago by Reuters. Now the thing is, with Reuters, we can actually trust their judgment when it comes to releasing information, especially from OpenAI. This is because Reuters is a very trusted source, meaning that the information that they present is often rather accurate. Now the crazy thing about this is that not only did we get this article that I'm about to dive into, we also got some interesting pieces of information regarding other demos at OpenAI. So one of the things I spoke about yesterday was human like reasoning. You can see here that this, this is an excerpt from an article that was published yesterday in Bloomberg. And in this article they actually discuss how recent demos have been showing GPT4AI that apparently rises to human like reasoning. Now we can see here, at the same time, company leadership gave a demonstration of a research project involving its GPT4AI model that OpenAI thinks shows some new skills that rise to human like reasoning. Now the reason that I've included this in the video is because with Project Strawberry, some people are wondering if the article from yesterday is referring to Project Strawberry that also demonstrates human like reasoning. But as you can see from this Reuters article, they clearly state that Reuters could not determine if the project demonstrated was Strawberry. So what this could mean is that OpenAI probably has another model that is specifically focused on reasoning that isn't as small as Project Strawberry, in the sense that it's probably a large language model. So what this could mean is that there is some kind of agentic framework wrapped around GPT4, maybe a small iteration of GPT4 that uses an agentic framework to do better increased reasoning. We know that this is a possibility because this is something that many people have done with smaller large language models. So the reason that this is actually kind of fascinating is because it now means that potentially there might be a new way to for GPT4 to rise to human like reasoning. One important detail that you might want to also note is that this doesn't actually say anything about GPT5. I know that some people might speculate that this is GPT5 for Project Strawberry. And this GPT4AI model doing some human like reasoning is completely separate from what GPT5 is. So I'm guessing that what we are seeing here is the inner workings of OpenAI's research departments and these areas are where they are focusing on how they can actually make these models a lot smarter in terms of the reasoning ability. And the reasoning ability is essentially just where the model is able to think about problems, break them down and understand them in a way that is far superior than prior efforts. So you can see here that the actual goal of Strawberry is to perform research. Teams inside OpenAI are working on Strawberry, according to a copy of a recent internal OpenAI document seen by Reuters in in May. Reuters could not ascertain the certain precise date of the document, which details a plan for how OpenAI intends to use Strawberry to perform research. The source described the plan to Reuters as a work in progress and the news agency could not establish how close Strawberry is to being publicly available. So one of the key things about Strawberry that we've gotten from this article is that Strawberry is going to be used to perform research. Now, it's not clear what kind of research that is, but further on they actually do talk about how this is going to be some kind of deep research that's on the Internet. Take a look at this. So it says how Strawberry works is a tightly kept secret, even within OpenAI, the person said the document describes a project that uses Strawberry models with the aim of enabling the company's AI to not just generate answers to queries, but to plan ahead enough to navigate the Internet autonomously and reliably to perform what OpenAI terms deep research, according to the source. So deep research could mean that this is some kind of AI agent that is able to perform research autonomously. Now, what's interesting here is that this seems to be the next iteration of models that OpenAI might be working on. Previously, if you watched yesterday's video, we saw that there are five levels to AGI, the levels of AI that OpenAI is working on, and number two is going to be reasoners, and number three is going to be AI agents. Now, it's actually quite hard to gauge exactly what Strawberry is and how it's being done, but I do think we have enough details to make somewhat of an educated guess. So what we can see from this article right here, especially this part, is that the document describes a project that uses the Strawberry models with the aim of enabling the company's AI to not just generate answers to queries but, but to plan ahead enough to navigate the Internet autonomously and reliably to perform deep research. So previously, if you've been in the AI space, you'll know that one of the main problems with AI is that currently in doing AI agents, some of the top CEOs have previously said that in order to get autonomous agents, it's actually very tricky. This is because of two reasons. Number one is the lack of skill, and number two is a lack of reliability. If we don't have models that are truly capable and truly reliable, we can't get them to perform consistent actions over a long enough timeframe in order for the AI agent to actually perform autonomously.

So when you ask a model to produce, you know, to, to complete a sequence of action, let's say it's like three things in, you know, to basically, let's say to book, you know, a restaurant that you and I can go to on a certain day. Yep. The first action would be check the availability in both of our calendars. So that's a correct function call, reconcile the correct moment. So that's the second action. Make sure that it's a restaurant that has availability. So that check is another one. And then, you know, go and sign in so that you can basically, you know, use the correct tool to book the right restaurant at the right time. Put your credit card details down, having obviously also checked that it's a restaurant that we both like, et cetera. So it's like four or five or six different steps just to produce that one action, subcomponents Right. In order to get that right, you're basically saying that the model has to produce perfect function calling for each element and do so in sequence. So it can't just be arbitrary, it has to be in sequence. And that's like saying it has to write a four page document in response to one question that is exactly that document. And it can't be something that is approximate or similar to that document. So we all think that obviously these models are magic at the moment and they write beautiful poetry and creative copy and text and give you good answers and sometimes they're grounded and blah, blah, blah. But when, when, for each one of those answers, there's a wide range of correct answers that it could have picked. Right. Tens, hundreds, thousands, maybe.

Now, what's interesting here is that apparently Strawberry might be the breakthrough that enables the company's AI to plan ahead enough to navigate the Internet autonomously and reliably. Now, I may have included a clip there from Dario Amodei and Microsoft's Mustafa Suleiman, but hopefully those clips showed you all why this is a real problem that needs to be solved. Now, depending on how good Strawberry models are, we know that if the reasoning of abilities do jump, then it does mean that potentially we could be getting AI agents much sooner. But we also do know that AI agents are planned for the future models like GPT6. Now there was an OpenAI spokesperson that said something in a statement. They said that we want our AI models to see and understand the world more like we do. Continuous research into new AI capabilities is common practice in the industry, with a shared belief that these systems will improve in reasoning over time. And the spokesperson did not address questions about Strawberry for whatever reason. Now, of course this is one of the main goals about OpenAI, because having better reasoning allows the models to be a lot more useful. Because one of the main gripes that people have with current systems is that they just aren't smart enough to understand what we want them to do time and time again. Now I would argue that oftentimes you do need to provide as much detail as you can to systems like GPT4, but there is the argument that sometimes these models just truly don't understand what we want them to do. It's clear that this is essentially the main directive for OpenAI right now, and this is where they're focusing all of their efforts in order to be the market leader in terms of these models. Reasoning is probably going to be the main area where they're going to be focusing. Now we can see something also quite interesting here, and this is a callback from an interview that Sam Altman had with Bill Gates. You can see here that the bit I've highlighted speaks about how Sam Altman talks about how future models are going to have increased reasoning ability. The reasoning ability essentially was where Sam Altman was stating that that is the area of largest progress that they would want. Of course GPT4 right now can reason, but as Sam Altman states, it's only reasoning in extremely limited ways and it's also not that reliable. As he states, if you ask GPT4 most questions 10,000 times one of those 10,000 is probably really good, but it doesn't know which one is which and you'd like to get the best response of 10,000 each time. So the increase of reliability will be important. Now an interesting part of Strawberry actually showed some information regarding future capabilities. So you can see that it says here that AI researchers interviewed by Reuters generally agree that reasoning in the context of AI involves the formation of a model that enables an AI to plan ahead, reflect how the physical world functions, and work through challenging multi step problems. Reliably improving reasoning in AI models is seen as the key to unlocking the ability for models to do everything from making scientific discoveries to planning and building new software applications. That's why reasoning is at the forefront of OpenAI's directive is because this is going to be something that unlocks pretty much all future use cases where the model is used extensively. Now what was also interesting was that it looks like we might actually be getting a separate model. As Sam Altman has discussed in the past on the Lex Friedman podcast, there has been discussions of different model releases. Now different model releases isn't always necessarily GPT5. Remember, GPT4O was a model that was released with certain capabilities that we truly didn't even expect. And I did a whole video on the secret capabilities of GPT4, which honestly are quite outstanding. GPT4 is quite understated in terms of its true capabilities, from it being able to do 3D models to photorealistic images. It's a truly multimodal, incredible piece of technology. What this statement says here is that in recent months the company has been privately signaling to developers and other outside parties that it is on the cusp of releasing technology with significantly more advanced reasoning capabilities. Now what that could potentially mean is that we could be getting an independent reasoning model. I'm not sure if this is actually going to be true, but it didn't mention anything in regards to GPT5. Now I do think that if this was referring to GPT5, they would have probably simply just stated it. But that is going to be something that we will have to know see in the future. But either way, we do know that they are on the cusp of releasing technology with significantly more advanced reasoning capability. And it will be interesting to see how with OpenAI's track record of now doing iterative deployment, how those models are released into the wild. Now this is where we get into how this model actually kind of works and what the model is in its bare bones form. So it says that Strawberry includes a specialized way in what is known as post training. So OpenAI's generative models or adapting the base models to hone their performance in specific ways after they have already been trained on reams of data. One of the sources said the post training phase of developing a model involves methods like fine tuning, a process used on nearly all language models today that come in many flavors, such as humans giving feedback to the model based on its responses and and feeding examples based on good and bad answers. So if you want to know how post training works in AI, essentially the AI learns general language skills from a lot of text during the pre training and learns how to understand and generate language, kind of like a student learning basic reading and writing skills. Now after pre training, the AI goes through post training, which is also called fine tuning. And during fine tuning, the AI is given specific tasks to practice, like answering questions, summarizing articles or translating languages. It's like giving the student extra lessons focused on particular subjects. This is obviously an important step for AI because it can help the student do better in certain subjects and it helps AI perform better on specific tasks. Now here's what's interesting about Strawberry is that OpenAI hopes the innovation will improve its AI model's reasoning capabilities dramatically and that Strawberry involves a specialized way of processing an AI model after it has been pre trained on very large data sets. So it seems that OpenAI is testing out a new way to kind of finalize these models. It doesn't seem like it's that similar to post training, but it involves a specialized way of processing an AI model after it's already been pre trained. So basically a new way of quote unquote post training the model. Now here's where we get into some of the research papers that were linked to this information. So it says here that Strawberry has similarities to a method developed at Stanford in 2022 called Self Taught Reasoner or STAR. And this is one of the sources with knowledge of the matter said. Now STAR enables AI models to bootstrap themselves into higher intelligent levels by iteratively creating their own training data and in theory could be used to get language models to transcend human level intelligence. One of its creators, Stanford Professor Noah Goodman, said, so essentially here this is I guess, guess you could say in theory some kind of self improving AI. Because what they're doing is iteratively creating their own training data and then using that to reason about the world. Now I did a video on a version of this which I will talk about later. But if we take a look at the paper, it truly is fascinating with as to what it shows us because we haven't seen this, explored that much with regards to AI. So we can see here that this is called star Self Taught Reasoner, bootstrapping reasoning with the reasoning. If you read the abstract, it gives you a clear detail on exactly what's going on. So we can see here generating step by step chain of thought rationales improves language model performance on complex reasoning tasks like mathematics or common sense qa. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using few short intervals. Re propose a technique to iteratively leverage a small number of rationale examples and a large data set without rationales to bootstrap the ability to perform successively more complex reasoning. This technique, the self taught reasoner, relies on a simple loop generate rationales to answer many questions prompted with few rationale examples, and if the generated answers are wrong, try again to generate a rationale given the correct answer, fine tune on all the rationales that ultimately yielded correct answers and then repeat. And from this we've shown that STAR significantly improves performance on multiple data sets compared to a model fine tuned to directly predict final answers, and performs comparably to fine tuning a 30 times larger state of the art model on common sense qa. Thus, STAR lets a model improve itself by learning from its own generated reason. So this is incredible because this is the kind of technology that could generate superintelligence or at least get to an intelligence that is above humans. Having a model that's improved itself by learning from its own reasoning is something that would be somewhat recursively self improving. Now, I do know that recursively self improving means that the model can change all kinds of things about itself, but I do think having a model improve itself by learning from its own generated reasoning is largely the step that the space is going to be moving towards. Because we do know that from history, AI models that only learn from human demonstrations or human informations are limited by that of the human. So having an AI system that could improve itself by learning from its own generated reasoning is absolutely incredible. So essentially what we have here is a model that's able to learn from its mistakes, and then when it gets the questions right, it learns why it got them right and learns from that. Now here's an example of this in action. Now this is the STAR paper. This is a screenshot, an overview of STAR's generated rationale on common sense QA. We indicate that the fine tuning outer loop with the dashed line and the questions and ground truth answers are expected to be present in the dataset while the rationales are generated using star. So you can see here that this is the architecture of how it works and this is the question. So the question is, what can be used to carry a small dock? Then of course we have the choices. A swimming pool, a basket, a dog show, a backyard, an own home. And then we can See the rationale being generated using the STAR framework. So it then says the answer must be something that can be used to carry a small dog. Baskets are designed to hold things, therefore the answer is basket. So we can see that the model has rationalized, it's got its answer and is then able to improve itself. Now one of the crazy things when I was looking at this paper was not only what this model is able to do, but what it was able to do compared to other models of much larger sizes. So GPTJ is a 6 billion parameter language model served as the base model for the STAR method. Initially GPT J was prompted with a few examples that included rationales which are step by step explanations. The model was then used to generate rationales for a larger dataset. The generated rationales were filtered to only keep those that led to correct answers. And GPT J was then fine tuned on the filtered dataset containing both the questions and the correct rationale for the questions where the model initially gave the wrong answer, it was given the correct answers and asked to generate rationales for them. And these rationales were also used for further fine tuning. And this process of generating, filtering and fine tuning was repeated iteratively, allowing GPT J to improve its reasoning capabilities over time. Now what's crazy here is as you can see, the GPTJ with star and the rationalization performs comparably to GPT3, which is directly fine tuned. So you can see they managed to get an extremely small model to perform comparably to a model that was around 30 times larger. And I'm guessing that people are wondering if this method is going to work with larger methods like GPT4 or even GPT5. One thing that I do know now is that all eyes will be on the STAR paper. One thing that I do know now is that maybe many AI researchers are probably going to take a second look at this paper because if what they're claiming is true, and it could potentially be used to get to high levels of intelligence that transcend what humans can do, this has some true, truly scary implications. Now the article also goes into aiming. Now the article continues to state that OpenAI is aiming at Strawberry to perform long horizon tasks, referring to complex tasks that require a model to plan ahead and perform a series of actions over an extended period of time. You can see here it says that OpenAI is creating, training and evaluating the models on what the company calls a deep research dataset. Now, Reuters was unable to determine what is in that dataset or how long an extended period would mean, and the entire goal of this is that OpenAI wants to use its models to use these capabilities to conduct research by browsing the web autonomously with the assistance of a CUA or a computer using Agent 1 that can take actions based on its finding, according to the document, with one of these sources. And OpenAI plans to test its capabilities by doing the work of software and machine learning engineers. Now, this is a lot of information because it has a lot of implications. One of the major implications that this has is that it seems like OpenAI has a clear path towards being able to autonomously do good Internet research. Now, I don't think it's just Internet research, because what they also stated here was that they're going to be testing the capabilities on doing the work of software and machine learning engineers. So it seems also that OpenAI might have some internal benchmark for where they're testing how capable these models are in terms of it being able to do software engineering and machine learning engineering. These, of course, are really difficult tasks. But if you do remember, one of the main goals of OpenAI was to automate AI research. And one of the other things of OpenAI was to get AI agents. Right? Now, I'm guessing that what OpenAI has probably realized is that reasoning unlocks all of these capabilities. And once they do unlock higher levels of reasoning, then everything is going to be a lot easier. Now, one of the things that many people have questioned is of course, the name strawberry. Why would OpenAI switch from Q to Strawberry? One of the main theories floating around is the fact that that this is from a question that shows how LLMs struggle with reasoning. A simple question proposed to the majority of LLMs today gives a wrong answer. The question is, how many Rs are in the word strawberry? And usually what happens is that these models will state that there are two Rs in the word strawberry. If you didn't get it, if you take a look at the word strawberry, there's one R here and then there's two R's right there. So in total there are three. But somehow the model didn't reason properly and it didn't realize that there are three Rs in the word Strawberry. This could be the reason why there's also another theory floating around that this is named after Elon's Vanity Fair, Strawberry Fields. So in the 2017 Vanity Fair profile, using a vivid metaphor that involved strawberry fields, and he described a scenario where an AI designed for picking strawberries could potentially go rogue and convert the entire earth into strawberry Fields. He explained that all it wants to really do is pick strawberries, so then it would have all of the world be strawberry field. Now I don't think that this is the case since Elon Musk and OpenAI have long had a feud in recent years and I think that Sam Altman and Elon Musk's relationship isn't exactly the best. Now overall I do think that Q STAR does have a lot of information and probably is something that they are prioritizing a lot at OpenAI considering. Very recently Sam Altman said that Q STAR was something that they weren't willing to talk about yet. Now there were many theories about what Q was before and we do have them here. One of them was the Q learning approach which I spoke about before, which was why they potentially called it Q. Of course we had the A search which was where the STAR comes from and of course now we have the Star which is the self taught reasoner, which is why some people are speculating this is why it's called star. But also some of the things from the paper also do show us that this is potentially related to the paper. So I'm not sure if maybe it's a combination of these papers abilities or it's a really unique way because OpenAI doesn't show their research. In summary, Q could be an advanced AI system that combines the strengths of Q learning for decision making, a search for efficient planning and self taught reasoning. In summary, Q could be an advanced AI system that combines the strength of Q learning for decision making, a search for efficient planning, and the self taught reasoning for improving its problem solving abilities through iterative rationale generation and refinement. This combination could create a highly capable system that can plan, act and learn in a sophisticated manner.