OpenAI: ‘We Just Reached Human-level Reasoning’.

The CEO of OpenAI said just two days ago that AI models have reached the level of human reasoning. Human level problem solving. In a sea of hype, though, claims like this have to be taken with a gallon of salt, naturally. But my question is this. Is it not now easier to come up with a reasoning challenge that the new O1 family of models passes and an educated adult would fail than the other way around? Yes, of course. While O1 still makes plenty of embarrassing mistakes, so do we. So is this not a watershed moment either way? I'm going to analyze four new key quotes from Sam Altman, give you the backdrop of OpenAI's new $157 billion valuation, and provide as much context as I can for his key claim. The key claim that Sam Altman made at Dev day less than 48 hours ago was that the new 01 series of models on are human level problem solvers. They don't output the first thing that comes to mind, so to speak. They reason their way through challenging problems. This chart, released in July, I've realized, is OpenAI's version of a levels of AGI chart. For me, it sets a very high bar for what would count as an AGI, but it does mimic five things that humans can do in increasing order of difficulty. We can chat, we can reason our way through problems, we can take actions in the world, we can innovate, and we we can organize together. Forget levels 3 and above for a moment. Just claiming we've reached level two is a bold enough announcement already. Here is a 60 second extract from dev Day and if you're wondering about the strange video setup, it's actually a massive improvement from the super shaky footage that's circulating online. Essentially I used Adobe Warp stabilizer, so it's a bit easier to watch. Altman here claims we are at level two.

How close are we to API? You know, we used to every time we finished a system we would say like in what ways is this not an AGI? And it used to be like very easy. You can like make a little robotic hand, but it doesn't prove it's cube or a Dota bot. It's like, oh, it does some things, but definitely not an AGI. It's obviously harder to say no. So we're trying to like stop talking about AGI as this general thing. We have this levels framework because the word AGI has become so overloaded. So like real quickly we use 1 for chatbots, 2 for reasoners, 3 for agents, 4 for innovators, 5 for organizations like roughly I think we clearly got to level two or we believe we clearly got to level two with 01 and it, you know, can do really quite impressive cognitive tasks. It's a very smart model. It doesn't feel AGI like in a few important ways, but I think if you just do the one next step of making it, you know, very agent like which is our level three and which I think we will be able to do in the not distant future, it will feel feel surprisingly capable.

Being totally honest. The next two key quotes I might previously have dismissed as being pure hype, but after the 01 preview release I'm less inclined to do so. First is the commitment that the next two years are going to see very steep progress.

If you go from like 01 on a hard problem back to like 4 Turbo that we launched 11 months ago, you'll be like, wow, this is happening.

Pretty fast.

And I think the next year will be very steep. Next two years will be very steep progress harder than that, hard to see a lot of circuit.

And next and he didn't have to go this far is the claim that this time next year there will be as big a gap from that model to 01 as from 01 to GPT4 Turbo.

The model is going to get so much better so fast. Like we are so early. This is like Maybe it's the GPT2 scale moment, but like we know how to get to GPT4. We have the fundamental stuff in place now to get the GPT4. And in addition to planning for us to build all of those things, plan for the model to just get like rapidly smarter. Like, you know, hope you all come back next year and plan for it to feel like way more of a year of improvement than from I'll save.

The last key quote for later on in the video, but in case you're skeptical of his or even my analysis, a plethora of professors and scientists have described an incipient form of reasoning within O1. One top researcher said, in my field of quantum physics, it gives significantly more detailed and coherent responses. Another molecular biologist said that O1 breaks the plateau that the public feared LLMs were heading into. Then we have the creator of the graduate level, Google Proof Q and a benchmark test that's become famous. It's targeted at PhD level scholars who score an average of around 60%. One of the authors of that benchmark said, it seems plausible to me that the O1 family represents a significant and fundamental improvement in the model's core reasoning capabilities. Just yesterday, a professor of mathematics described a moving frontier between what can and cannot be done with LLMs and that that boundary has just shifted a little. And here was his aha moment. Using GPT01 mini and 43 seconds of thought, it came up with an entirely new, clever and correct proof that he described as being more elegant than the human proof. It must be said that most of this is anecdotal, but it at least shows that the claim isn't completely outrageous. From Sam Altman yes, there are plen plenty of benchmarks where O1 Preview is still not scoring top marks. To give one example, here's a new one called Psy Code where O1 Preview scores just 7.7%, a noticeable step up from other models. But what is this benchmark testing? Well, it includes several research problems that are built upon or reproduce methods used in Nobel Prize winning studies, including things like the Haldane model for the anomalous quantum hall effect. To get questions right you would need abundant high quality data not usually made available to current language models. The language models have to generate code for solving real scientific research problems. And it's not enough just to get these sub problems right. There's 338 of those. The models have to compose solutions from those sub problems to get the 80 challenging main problems correct. It's an amazing and useful benchmark, but seems more apposite for a level 4 model than a level 2 one. This is certainly not average human level problem solving. So I return to that question. Is it not now easier to come up with a reasoning challenge that O1 passes and an educated adult fails than to find one where it's the other way round? I say this as the author of Simple Bench where it is the other way around and models underperform humans. But it seems harder to create such a benchmark than the other way around. As Sam Altman said of the Turing Test, I don't want everyone just to ignore the fact that we have passed average human level reasoning. Then we could throw in the fact that Mentor permits the Law School admissions test as an admissions criteria to Mensa. Because O1 can crush the LSAT, it now qualifies for Mensa 18 years early according to 2020 metaculus predictions. Do you remember what comes next in that levels of AGI chart? What is Level three agents? Just a couple of days ago in the Financial Times, the Chief product officer at OpenAI claimed this we want to make it possible to interact with AI in all of the ways that you interact with another human being. These more agentic systems are going become possible. And it is why I think 2025 is going to be the year that agentic systems finally hit the mainstream. I can see that taking a little bit longer though, because unless it had 99.99% accuracy, I wouldn't trust an O1 agent with my credit card. One ability that will be absolutely crucial if agents are to work obviously is self correction, as I described recently on AI Insiders on Patreon. Speaking of which, a very quick shout out if anyone's in California, sw, India or Japan for the regional networking that's going on on Discord. What I will say though is that if OpenAI can turn reasoning into agency, I can see why they had a $157 billion valuation. That's great timing given their imminent switch from being a capped profit entity to a for profit entity. As we learned today in the information, it actually gets more serious than that. OpenAI has proposed letting investors claw their money back within two years if it fails to convert itself into a for profit entity. Now for some of you, I might have buried the lead because just 18 hours ago we learned this of the fundraising of OpenAI. Reuters reports that one of the clauses in joining in that funding round was that the investors don't also fund rival outfits. That includes Ilya Sutskoper's Safe Superintelligence, Anthropic Perplexity, XAI and others. Now, I don't think that'll be a problem for the likes of XAI funded by Elon Musk, but Ilya Sutskeva's Safe Superintelligence might face a bit more of a funding struggle. According to the New York Times, revenues are expected to balloon to almost $12 billion next year, though OpenAI are expected to lose $5 billion this year. Why? Well, the costs related to running its services and other expenses like employee salaries and office rent. Setting aside short term revenue and costs, one key question for OpenAI's future is whether this fifth clause from their charter still applies. Their board will determine when they've achieved AGI. By AGI they mean a highly autonomous system that outperforms humans. At most economically valuable work. Such a system is excluded from intellectual property licenses and other commercial terms. With Microsoft, those terms only apply to pre AGI technology. Now, you don't need me to point out that that sets up an incentive to push the definition of AGI as far away as possible. Is that clause what prompted these five generous levels of AGI? Notice if so, we're drifting away from concepts of intelligence and reasoning to other more nebulous attributes. Human level reasoning is an AGI, and even agents that take actions aren't AGI. These systems have to do the work of entire organizations. You can let me know what you think in the comments, but many people would say a more reasonable definition of AGI would have arrived well before this point. As a quick, fun experiment, I looked up Microsoft's definition of AGI and we got far into sci fi. AGI apparently may even take us beyond our planet by unlocking the doors to space exploration. It could help us develop interstellar technology and identify and terraform potentially habitable exoplanets. It may even shed light on the origins of life and the universe. If this ever becomes the definition of AGI, Microsoft will be minting money to Almost the end of time. Speaking of Microsoft, I wanted to show you guys this and I think it's fair to say that Potentially after seeing 01, Bill Gates has somewhat shifted his timeline. I reported on an interview in Handelsblatt where Bill Gates said he didn't expect GPT5 to be much better than GPT4. He was super impressed, of course, with GPT4, but just saw some limits to where scaling could take us. Contrast that with this clip from this week.

It's the first technology that has no limit. I mean when you invent a tractor or even a cell phone, you kind of figure out, okay, we can figure out how that's going to change life here where the AI is very intelligent and when you put it in robotic form it can do a lot of both blue collar and white collar jobs. The fact that's happening over the next decade, the idea of do we really trust government to adjust the tax policies and make sure that, okay, we're shortening the work week so it's happening very fast and it's unlimited. A lot of it is super good, like you know, inner city personal tutors for all the kids, great healthcare even in the poor countries. So the good stuff, which maybe gets crowded out by these fears. That's so exciting.

We're going to end the video with the final Sam Altman quote, but just before then I want to Highlight yet again NoteBookLM from Google. If you didn't catch it in the last video, it's an amazing new tool powered by Gemini 1.5 Pro. As most of you might already be aware, you can turn any PDF audio file or now any YouTube URL into a podcast. And it's not just PDFs, documents, YouTube videos or audio files that you can add either. I can imagine millions of students taking class materials and their own handwritten notes, feeding it into NotebookLM for free by the way, and getting amazing podcasts out. And in case you missed it last time, it's literally as easy as clicking try NotebookLM and then New Notebook. Upload one or multiple sources now, including YouTube URLs. Then the option will spring up to generate the audio. Click that and it takes two or three minutes. I've seen people take literally anything, including absolutely absurd sources, and turning them into engaging podcasts will say that if you want the transcription that the podcast hosts use to be extra accurate, do check out Assembly AI's Universal One. I'm super proud they're sponsoring this video because their Universal One model has amazing speech to text accuracy. Yes, it does handle my rather rough around the edges London accent. You can see some of the comparisons to other models down below and the link will be in the description to check them out. Here though, is the Sam Altman quote from Dev Day that I found somewhat intriguing. He speaks almost optimistically about a model one day automating OpenAI itself.

Even the Turing Test, which I thought always was like this very clear milestone. You know, there was this like fuzzy period. It kind of like went whooshing by and no one cared. But but I think the right framework is this one exponential.

That said.

If we can make an AI system that is like materially better at all of OpenAI than doing at doing a that does feel to me like some sort of important discontinuity. It's probably still wrong to think about it that way. It probably still is the smooth exponential curve, but that feels like a real milestone.

We are almost certainly nowhere close to that level now. But there is one caveat I wanted to give. Sam Alwan describes automating OpenAI itself, but in OpenAI's own preparedness framework, that would be a critical threshold. They say if the model is able to conduct AI research fully autonomously, it could set off an intelligent explosion. Moreover, they say we will not deploy AI systems that pose a risk level of high or critical, and we will not even train critical ones given their level of risk. It almost seemed like Sam Altman was speculating about a model that OpenAI itself promised to never train. As always, though, I'm curious what you think, so thank you so much for watching to the end and please do have a wonderful day.