Q* - Clues to the Puzzle?

As you might expect, I have been researching nonstop about this apparent powerful AI discovery that insiders at OpenAI said could threaten humanity. I've spoken to every insider I know and done a ton of research, and I am not claiming to have solved the puzzle, but I can provide some genuine clues that I think will be at least part of the answer. Normally I like to be a lot firmer than that, but this is the best I can do. Now, the first thing to note, of course, is that OpenAI have now denied that Sam Altman' elsa was precipitated by this safety letter to the board. As my previous three videos have shown, there was certainly a lot else going on, so it might well not just be about this AI breakthrough. Second, I just want to quickly debunk this clip that's doing the rounds where people are claiming that Sam Altman called this new creation a creature, not just a tool. Actually, if you watch to the end, he's very much saying he's glad that people now think of it as part of the toolbox. So despite this frantic headline, I am not trying to overhype things, but I genuinely think I figured a couple of things out. Let's get to it. Later in the article, these insiders these researchers flagged up work by an AI Scientist team, the existence of which multiple sources confirmed. This AI Scientist team was formed by combining earlier Cogen and Math gen teams at OpenAI. Their work on exploring how to optimize existing AI models to improve their reasoning was flagged in the letter to the board. Now there is very little public information about either the Cogen or mathgen teams, but I dredged up this old tweet from Sam Altman. Yes, by the way, I am now on Twitter. I finally succumbed. Anyway, Sam Altman said really exciting process supervision result from our Math gen team and he linked to a critical paper that I covered back in the spring. It's called let's verify Step by Step and that paper is the crux of the video. That's what I think the former Math Gen now AI Scientist team were working on. And take this tweet in September by Noam Brown at OpenAI. My teammates and I at OpenAI are hiring ML engineers for research on LLM multi step research reasoning. We recently hit a state of the art 78% on the math benchmark, which I'll get to in a second. Our new plans are even more ambitious. I'm only just getting started, but you might already be glimpsing why I think this paper is the crux of the new development. Here's another reason. What's much more ambitious than 78% acing such tests? That's apparently what they've achieved behind the scenes, at least according to Reuters. And what's another bit of evidence? Well, we have this exclusive report from the Information. Again, similar frantic headline OpenAI made an AI breakthrough before Altman firing, stoking excitement and concern. But who led that breakthrough? The technical breakthrough, they say, was spearheaded by OpenAI Chief Scientist Ilya Sutskeva. And the most recent paper on Arxiv listing Ilya Sutskever as an author is the let's Verify Step by Step paper. Now at this point I know you want me to get to what it means, but with so many theories floating out there, I want to give you yet more evidence that at least one of the breakthroughs links to let's Verify Step by Step. And I think I might even know what the other breakthrough careers the same article in the Information talks about Sutskever working on ways to allow language models to solve tasks that involve reasoning like math or science problems. It talks about how he had this secret program called GPT0. And here's where it gets interesting. The team hypothesized that giving language models more time and computing power to generate responses to questions could allow them to develop new academic breakthroughs. And Lucas Kaiser, who we will definitely be seeing more of in this video, indeed he appears in the thumbnail, apparently held a key role role in the GPT0 project. And look at this. Among the techniques the team experimented with was an ML concept known as test time computation that's apparently meant to boost language models problem solving abilities. And we have a hundred speculations online about what this might mean. But I knew that name rang a bell and I dug up this 2021 paper which cites Test Time compute. And what is this random paper that Philip is talking about? Well, it's from OpenAI and look at some of the co authors Karl Cobb of the mathge and Lucas Kaiser cited in the Information. This was actually one of the precursor papers to let's Verify Step by Step, which I will explain in a moment. It introduced the now famous in ML circles GSM8K dataset. That's 8000 grade school math problems. More importantly though, it trialed this method of at test time generating many candidate solutions and selecting the one ranked highest by a verifier. And I'm going to massively oversimplify at this point and just say that a verifier is a separate model trained only at this point in this paper to spot good solutions, solutions that get the answer correct. And what the paper proposed was getting the base LLM to generate hundreds of solutions and then getting this separate verifier to spot the ones that were likely the most correct. And in a nutshell, the authors noticed that if they invested more computing power in generating more solutions and taking a majority vote among the top verifier ranked solutions, that had a massive effect on performance. And that's what it means by test time compute, investing your computing power while you're taking the test, not during training, so the model stays the same. You're not further training it or fine tuning it. You're investing that computing power during test time again to generate potential solutions and take majority votes amongst them. Self consistency. They found that using a verifier in this way was better than fine tuning because verification was a simpler task than generation. It's easier to check your answers than generate good ones. So investing time in checking answers was more fruitful. How much more fruitful? Well, the use of verifiers results in approximately the same performance boost as a 30 times model size increase. And then it gets prophetic when it says and verifiers scale significantly better with increased data. And to hammer the point home, they said a 6 billion parameter model with verification slightly outperforms a fine tuned 175 billion parameter model again thereby offering a boost approximately equivalent to a 30x model size increase. And that team, by the way, with Lucas Kaiser, was probably drawing on work by a single dude done six months earlier. In April 2021, while studying the board game Hex, the researcher Andy Jones found an interesting result. He said, along with our main result, we further show that the test time and train time compute available to an agent can be traded off as we've seen while maintaining performance. I read that paper in full and it was cited by no other than G.N. brown. He's the guy we saw earlier who joined OpenAI in July of this year. He said he's investigating how to make reasoning methods truly general. If successful, we may one day see LLMs that are a thousand times better than GPT4. And later in that thread he cites that same paper from Andy Jones and he concludes with this. All those prior methods are specific to the games they're talking about. But if we can discover a general version, the benefits could be huge. Yes, inference may be a thousand x slower and more costly, but what inference costs would we pay for a new cancer drug or for a proof of the famous mathematical Riemann hypothesis, and we'll come back to mathematical hypotheses in a moment anyway, Improved capabilities are always risky, he went on, but if this research succeeds, it could be valuable for safety research as well. Notice that point about capabilities being risky. It does seem to me to be linking together into that Reuters story. Imagine, he says, being able to spend $1 million on inference to see what a more capable future model might look like. It would give us a warning that we would otherwise lac. Is that what OpenAI did when they recently pushed back the veil of ignorance? Did they just spend a million dollars on inference and see what a more capable future model might look like? So that's test time compute. But what about let's verify step by step? Well, going back to that original 2021 verifier paper, they said this. The problem that they noticed with their approach back in 2021 was that their models were rewarding correct solutions, but sometimes there would be false positives getting to the correct final answer using flawed reasoning. They knew this and so they worked on it. And then in May of this year they came out with let's verify step by step in this paper by getting a verifier or reward model to focus on the process, the P instead of the outcome. The O results were far more dramatic. Next, notice how the graph is continuing to rise. If they just had more, let's say test time compute, this could continue rising higher. And I actually speculated on that back on June the first. That difference of about 10% is more than half of the difference between GPT3 and GPT4. And also is it me or is that line continuing to grow, suggesting that when more compute is available, the difference could be even more stark. Imagine a future where GPT 4 or 5 can sample, say a trillion 10 to the 12 solutions. So you're beginning to see my hypothesis emerging a new and improved let's verify step by step called Q drawing upon enhanced inference time computer to push the graph toward 100%. If you want more details on that process reward model, check out the video I did back then called Double the Performance. But the very short version is that they trained a reward model to notice the individual steps in a reasoning sequence. That reward model then got very good at spotting erroneous steps. Furthermore, when that model concluded that there were no erroneous steps, as we've seen from the graphs, that was highly indicative of a correct solution. Notice also that sometimes it could pick out such a correct solution when the original generator GPT4 only outputted that correct solution one time in a thousand. Furthermore, the method somewhat generalized out of distribution, going beyond mathematics to boost performance in chemistry, physics and other subjects. So was it potentially the million dollar inference run that spooked the researchers? Or was it the potential to make radical breakthroughs in SC science? And a further bit of supporting evidence for this theory comes again from the information. It gives us a rough timeline. In the months following the breakthrough, Sutskva had reservations about the technology and in July he formed the Super Alignment team. So the original breakthrough, whatever it was, had to have come way before July. That would fit much more with it being associated with let's verify step by step or again maybe that combination of process, reward modeling and inference time compute. And in a genuinely fascinating recent conference call, this is what Ilya Sutskeva said in reply to a question asking about if models could produce mathematical conjectures.

How original or creative are the latest large language models? Of course we know that for instance AlphaGo did some pretty creative moves when it won its match in South Korea. So, so that's possible. But to be very concrete, do you think the existing models or some, you know, the next GPT4, say GPT5 or so will be able to state a new non trivial mathematical conjecture? I'm not saying proving it, I'm saying stating it. Who think it's possible within the next five years?

Are you sure that the current model cannot do it?

I'm not sure absolutely. Do you know whether it can?

I mean let me give you, let me give you an example of something.

Creative that GPT4 can already do. Obviously we would have all loved to hear Ilya Sutskova's full answer to that question, but we never got a chance. But here's where we get to arguably the strongest bit of evidence that I've got. Remember that name again, Luke? Lucas Kaiser cited in the exclusive information report. If you remember, he had that key role on the GPT0 project. He was a co author in both of the papers that I've brought up and he was focused on test time computation to boost language models problem solving abilities. Well as the article says, even more significantly, he was one of the co authors of the original attention is all you need Transformers paper. And so presumably with that much pedigree, it would take quite a lot to get him excited. So I'm now going to show you some extracts from two YouTube videos, both of which have hardly any views despite him being mentioned in these exclusives about Q. And Lucas Kaiser will describe in these videos how significant he thinks a variation of let's think step by step could be. First, he'll give you some further background on the breakthrough.

It needs to do this thinking inside its layers. That may not be enough time and space to do, like, tell me what you're thinking and only then give the answer. And if you do that, there is a recent paper that says the model can basically do any computation. It can even execute programs. It's Turing complete. And does this really help? So on mathematics, you can tell the model, hey, do this thinking, but do it, like, number each step, like 1, 2, 3, 4, 5, 6, 7, as you see here, and be very precise about each step. And then you can try to verify each of these steps of thinking separately. You can even ask the model, well, was step three correct? Was step four correct? And when you do that, like this math data set, which is a little bit tougher math problems than the pure arithmetic, it was especially made to show what the models cannot do. If you add this thinking, you can get to almost 80%. And just by allowing the model to think. And at least on mathematical questions, this gives like, insane games. And insane is the fair word to use. Like, if you. A transformer has a limitation in running time. It runs in M squared time for input of like n. And that's it. When you allow it to produce chains of thought. But it's as computationally powerful as anything you can imagine. But, okay, so these two ingredients give you something that generalizes. Could we make them even more powerful? And this is called chain of thought, obviously, chain of hindsight, programs of thought, and so on. But I think this has turned out to be the method that makes transformers more powerful. And it's not just mathematics where you can build these things thinking.

He even goes as far as describing it as a major focus for deep learning in 2024.

If you think what, you know, what is coming in the next years, I think there'll be a lot of focus on doing this thinking thing with deep learning, with language models, probably with chains of thought, but also these chains of thought, currently, they're just prompted, but maybe you need to do some fine tuning, some learning. There will be a lot of work there, but this is a very hard problem to solve. And we'll start with much simpler exercises and probably move forward. But I think this is something that the coming years will see a lot of work.

And in a sign of the significance he attributes to this method, he said it could even be revolutionary for multimodality.

I think you also need these chains of thought that, like, you need Them to give the model the ability to think longer than it has layers. But it can be combined with multimodal. So in the future the models will have this knowledge of the world and this generation, which we call chain of thought and text. But multimodality, this means just it's a chain of frames of what's going to happen to the world, which is basically how we sometimes think, you know, what will, if I go, what will happen to me? And I think that will indeed be so it will be multimodality and this ability to generate sequences of things before you give an answer that will resemble much more what we call reasoning.

He then described how this method will help models to generalize from much less data.

Transformers are not DM right. And you can do chains of thought to extend them. You can do GNNs, you can do recurrent recurrence and depth. How do you see the next two years of deep learning? Yeah, I think there'll be as interesting as any previous two years or even more. I think there'll be a lot on the chain of thought, but very generally speaking, so also on the agents building libraries of knowledge, possibly multimodal, where the chain of thought is basically a simulation of the world. So I think that will be like one big topic. And I think this will make the models generalize much better from less data too.

And that might remind you something going back to Reuters and the information Satskopher's breakthrough allowed OpenAI to overcome limitations on obtaining enough high quality data to train new models. According to the insider, with knowledge a major obstacle for developing next generation models. So according to my theory, this breakthrough is less about generating trillions and trillions of tokens worth of synthetic data, but more about using the data you've got much more efficiently. But now, alas, we must get to the bits that my theory can't explain, namely the name. The information cites two top researchers at OpenAI building on top of Sutskever's method, a model called Q. Now, I've tested every link I could possibly find to the name Q with my theory about let's verify step by step. And while I do have some ideas, honestly it's still an open question. And of course I, like everyone, has to admit that there's a chance that I'm entirely wrong. When I put my idea to a senior ML researcher at a top AGI lab, he thought it had real legs. It was a genuine possibility. And he said one link to the name Q could be in a generic sense, without getting too technical. Q refers to the optimal Q function or optimal policy. Another possibility is that the queue references Q learning generically. That's a reinforcement learning technique where an agent learns to make optimal decisions by exploring its environment. An agent chooses actions, see how they go, and then updates their policy. Basically trial and error trading off exploration of new steps, new actions versus exploitation of actions you know have some good reward. And here's where the analogy gets a little bit tenuous. Picking the reasoning step steps in let's Verify step by step could be like choosing an action. After all, in the original paper using test time compute in this way was described as a kind of search. And in let's Verify they hinted at a step forward involving reinforcement learning. They said we do not attempt to improve the generator, the model coming up with solutions with reinforcement learning. We do not discuss any supervision the generator would receive from the reward model if trained with rl. And here's the key. Although fine tuning the generator with reinforcement learning is a natural next step, it is intentionally not the focus of this work is that the follow up work that they did. I mean, you can kind of think of Q learning for process supervision as minimizing the cumulative probability of failure, which is the equivalent of maximizing the probability of success. And after all, maximizing a sum of rewards over multiple steps is exactly what Q learning aims to do. If any of you though have better guesses for the analogy, and I'm sure you do, do, let me know in the comments below. Comments but what about the star? Well again, here I am truly speculating. Unlike the earlier parts of the video, in which I am much more confident, this is much more speculative and tenuous. Peter liu of Google DeepMind had this idea. Remember the leak? Talked about acing math tests. He said sounds like OpenAI got some good numbers on GSM8K. Remember that's the set of questions made for that original verifier paper back in 2021. He said he's speculating, but there's a star in this paper, a technique that fine tunes a model to its own better outputs. In a nutshell, it involves fine tuning a model on the outputs it generated that happen to work. Keep going until you generate rationales that get the correct answer and then fine tune on all of those rationales. And they say that we show that STAR significantly improves performance on multiple data sets compared to a model fine tuned to directly predict final answer. Does that remind you of let's Verify and performs comparably to fine tuning a 30x larger state of the art language model.