AI:AM — World Models, Recursive Learning, and the Politics of AI-Native Organizations · September 8, 2026

So this is a guy called Skalski. And Skalski is pretty famous because he ended up— prior to this, he basically has a way of identifying players on the basketball court. And so he trained models to identify players on the basketball court. When people asked him how he did it, he hand-labeled 12,000 individual images with who the players were, referee or this player, that team, et cetera. And he hand-labeled 12,000 images, probably spent about 3 to 6 months. And he said he lost his vision at one point and he suffered and all of these things happened. And now Astra just does it. That's it. It's done. So this is a task that a human being will never do again. Like, there just isn't any, any point. You, you can't even pay someone to do it because if you paid someone to do it, they would use Astra to do it and then like pass you back the results. Like, it's, it's done. Like, a human will never do this task again.

Hello, Lems. predict the next token, but doesn't have the internal representation that the world model tries to build. So world model is not concerned about the details as much, but it's concerned about the patterns, the main patterns that you need to predict what happens next. And it's a lot about action. And world models just build differently in that sense, and they are able to have this representation of the world, like driving a car, and you understand, you don't need to remember that there is a green car coming next to you, or there is like a little bush over there. You just have the understanding that what you need to understand to be able to act. So that's what world models do. And that's a big difference from large language models.

Mathematicians can't answer Basic questions about it: Does a smooth flow always stay? So start the fluid in a perfectly smooth state. Two possibilities: regular, it stays smooth. In 2D, the answer has been known since the 1930s. Regular. In 3D, for Navier-Stokes, nobody knows. That is the Clay Millennium Prize. Every prediction we make about air, water, blood, or plasma. Runs through this equation. If it can tear itself apart in finite time, then classical physics in its own language stops.

This graph is scaled to the single day that had the highest RL. So already it's kind of dishonest. It's not relative to like an average that they had in some prior period. It's relative to the single highest. Like, oh, okay, well, so you're off pace from your single highest day ever. Is that really a slowdown? Did you really do much?

Good morning. It is Tuesday, September 8th, 9:01 AM. Good morning, Nathan. Hi, Prakash.

How are you?

I am very well. It is— it has been a very busy 36 hours, 48 hours, 72 hours. I think we start off with Astra dropping. And so Astra drops, what was it like Thursday, Friday? Yeah, Thursday. Thursday they launch, but they only announce, they don't actually give anyone access yet. And I think people started getting access around Friday. And there were, I think the influencers who had early access were able to give kind of early reviews and early projects on it. And then after that, the rest of the hoi polloi got access over the long weekend. Nathan, have you been using Astra? What have your thoughts been so far?

Yeah, I have been. I did, as I mentioned on Friday, follow through with my plan of telling Fable 5.1 that it now needs to make a plan to have Astra do all the major tasks that we do for a little while in parallel so we can systematically compare results and figure out what the new and improved division of labor should be. I don't have any explicit reason to think that Fable is cheating or undermining Astra's ability to succeed, but I do have it doing the prompting. So I'm, I'm leaving myself a little bit of room for model-to-model sabotage or other kinds of drama there. But so far, I would say it has impressed on a few different dimensions. I think both Fable 5.1 and Astra seem to be better at writing. I, of course, always use my intro essay to the podcast test to just see how I feel about model writing. Fable, for all it was— original Fable 5— for all it was extremely capable and at times showed really high taste and at times wrote amazing song lyrics, Mm-hmm. I feel like its prose was kind of overwrought, hard to read at times, definitely not really grokking the way that I write and imitating it very effectively. So I didn't really use it actually for that much at all. FiveOne and Astra, I think both better in that regard, although I still haven't quite locked into the point where I'm like, oh, you nailed my voice. I was interested to see that OpenAI is starting to go to market with that claim and saying that if you give it access to all the stuff, which I've done, although I've done that more on the Claude side. Astra has the similar access, but I haven't really iterated as much to make sure it's using it fully effectively. But those connections are made. Will it really start to learn my writing style? That will be very interesting to see. I think right now I can say It feels like a better writer, but will it dial into the point where I'm like, oh, you kind of nailed my voice on this one? That I think remains to be seen. Probably the coolest thing that I did was in creating a song for my upcoming US-China Nathan Goes to China Part 3 episode of the podcast. In preparation for the song, I really went down a bunch of different rabbit holes. It's not easy to get Suno to combine Chinese and Western sounds. I've really found that to be a struggle with eventual successes for the first 2 episodes, but it's not something that they— it seems to be naturally inclined to do out of the box. So tried a bunch of different things, bunch of different prompts, and I finally came to the idea. And actually it was Claude that brought this to my attention in the first place where I otherwise wouldn't even have known to think of this, but there's a classic song that apparently everyone in China knows that ran with like a famous TV show back in the '90s, 2000s timeframe. And it's kind of nostalgic and has, you know, kind of positive connotations, I guess, for a Chinese audience. Astro was able to go out, find a recording of that song. Not the remarkable part. I tried to upload the recording directly to Suno to get it to do a sample, dinged for copyright, couldn't do it. So then I asked Astra to create a sample and it took this original recording, chopped it up somehow, not exactly sure how it did it, but gave me a 10-second sample file from the original audio. That we were able to upload to Suno. And if I do say so myself, the end result is yet another banger on the Cognitive Revolution playlist. And it was a real collaboration where the AIs of course do most of the work, but Claude came up with this concept, explained to me the source of the idea and the cultural historical resonance and relevance of this song.

Wow.

Astra found the file, chopped it up, made an artifact that we could actually use. And then Suno was able to turn that into, you know, a song that I actually enjoy. I thought that was pretty cool. And Astra also did pretty well on the lyrics. So I do see again there a sign of it being clearly a better writer. One other thing I tested, because everybody's been doing the 3D worlds stuff with Astra, was for a book that I'm helping turn into an audiobook, which I'm going to run a few chapters of as a podcast episode. This is sort of quasi-utopian fiction, I'll say. Was just to try to have Astro make 3D worlds of the different scenes from the book and just bring those, you know, to life, quote unquote, so people could explore the scenes that the book describes. That I'd say I'm not so sure if it's like awesome or not yet. I mean, it did it, you know, the, the 3D scenes look pretty cool. I'm not sure there's a ton of utility or, you know, how much time people, even if they're interested in the book, are gonna spend clicking around these various places from the novel. But, you know, it, and I wouldn't say they came out looking as amazing as some of the ones that we saw. My, I, I was having it do it cuz I was thinking I would wanna put this on a, browser and not have dependencies. So I had it using Three.js and maybe it's a Three.js limitation versus I think Blender has been kind of more the default platform that we've seen people building these 3D worlds on. I'd say it came out well, impressive, certainly better than I could do by a lot. Incredible that it's doing all this in code and not even that much code such that it can load quickly on a regular web browser, but not quite as cool as the examples we've been seeing flying around online. I think there's definitely still some gap for me to figure out how to close. And it might just be which technology I had it work with. It might be something else. It might be just giving it more time to grind than I did over the weekend. But I would say you can reliably expect to see something pretty cool, but I wouldn't necessarily take it to the bank that you're going to get stuff on the level of the best and most viral examples that we saw. You know, it's weird, right? These are like extremely idiosyncratic tasks that are very hard to benchmark, very much going on vibes. But that's kind of where we are obviously today. The benchmarks are either fully saturated, as we've seen with Things like ArcAGI-3 and Frontier Math Tier 4, especially the Frontier Math. That's beyond my understanding in the first place, which foreshadows some other news today that we're going to get into. But yeah, just unleashing it on some of these random idiosyncratic tasks, some of which I never even would've tried otherwise, you could definitely feel that it is a step up. you know, I think that's like pretty obvious and pretty tangible. Um, but still there's a lot more mapping out to do. I do feel like we're in this era where there will probably be an Astra 6.1 before I really feel like I personally have command of Astra's 6 strengths and weaknesses.

Um, you know, one of the People online, Ethan Mollick, who is a professor who tests a lot of models, he posted the meter, the famous meter hours, you know, hours of work chart. There hasn't been an update for a while now. You know, I think it's—

I don't think they can really do it anymore.

Yeah, they can't. They can't.

They don't have tasks that are big enough.

They don't have tasks. which they can measure before the next model drops. The cycle time of model development is shorter than the length of the tasks that they need to measure at this point. So I think the meter graph is basically done at this point. I spent The entire weekend using Astra. I was running 3 to 4 agents continuously and they were good. I did use Fable to check the planning though. So I did use Fable a little bit to check the planning and that was actually very helpful. But Astra is very, very good in the sense that it started to tackle those annoying problems which had been in the codebase. As you know, we built the studio by ourselves. And it started to tackle some of the longstanding issues in the codebase, which had been kind of Annoying and bugging me, and like it started to like resolve those issues. And as those issues got resolved, I started giving it more tasks. I started generating okay what all the all the projects which had been kind of like I kind of want to do it, but it's it's another long haul of work and another thing to maintain. I don't have the capacity to like you know do it and maintain it and like keep track of it. Like it's not it's not worthwhile. And Astra started to like all right maybe I can tackle those projects now. So I start to expand. Expand a little bit and started to give it a little bit more to do. Uh, it is, it is very, very good. I, I would say it is finally at the, at the point where, uh, if you care about the quality of the work, you can still hand it off to Astra, but you still need to do a little bit of talking. But you can hand it off to Astra and you can, you can get some results. Um, it is prob— it is the first model that I felt, I felt this way about. It's so— I finally feel it's met The AGI definition. It does a bunch of things that I think other models failed on. I have multiple Gmail accounts, one for work, one for home, et cetera, et cetera. And lots of people have this. And one of the persistent problems has been managing that kind of multiple streams of information, which are kind of segregated and cordoned off.

Yeah.

And managing those streams of information, reply, you know, using the right email address to reply or talk to someone. I have some accounts on personal, I have some accounts on, you know, the AIAM account. And so Astra finally kind of like manages all of that. And, you know, and the computer use is good. The other thing that was failing really badly, I think, before is computer use. Computer use on GPT-5.6 would sometimes take a very, very long time. It would kind of click around and do a bunch of stuff. And Computer Use finally works properly in the kind of like timeframe that you give it. So it's clear, it's cleared the hurdle. It's cleared the hurdle of like genuine usefulness at this point. And you can start to give it more advanced tasks. And as it cleared that hurdle, I found myself using more tokens and more tokens and more tokens. Uh, so I did 2 resets over the weekend. I used up 2 resets, and Thibault dropped another reset. So 3 resets in total, uh, which is basically, I, I, I guess something like $100 each or something. So that's like $300 over the weekend. Uh, and so you can— I, I can, I can start to see, um, This is gonna, you know, the token spend is gonna increase dramatically. People are gonna use this thing. Token spend is gonna increase dramatically. I think a lot of people are gonna be using it all the time. It is AGI. It is that kind of cleared the hurdle of AGI. It will do things better than most people you can hire and train. So it's there. And there's a lot of interesting demos. Let me pull up, let me share one of the demos. So this is a guy called Skalski, and Skalski is pretty famous because he ended up Prior to this, he basically has a way of identifying players on the basketball court. And so he trained models to identify players on the basketball court. Maybe he's a sports statistician or a nerd of some kind. Anyway, and initially he had a very, very good model that he posted. He had, and when people asked him how he did it, He hand-labeled 12,000 individual images with who the players were, referee, or this player, that team, et cetera. And he hand-labeled 12,000 images. And now Astra can just do it. Like Astra just does it, right? This guy probably spent about like 3 to 6 months and he said like he lost his vision at one point and he suffered and all of these things happened. And now Astra just does it. That's it. It's done. So this is a task that a human being will never do again. Like there just isn't any point. You can't even pay someone to do it because if you paid someone to do it, They would use Astra to do it and then like pass you back the results. Like it's done. Like a human will never do this task again. So this was one of the big ones that I saw, like really a commentary on like how this guy had done this before by himself. And now, you know, he's never gonna do it again, right? So that was one Astra. This is another Astra one. Let me let me share this. So this is from Japan. So this is a cardiac surgeon. He's saying it's got a good solid grasp on cardiac anatomy, essential technique for ablation and atrial septal puncture. Not just 3D, but we were able to reproduce echo-guided puncture too. This is seriously going to change clinical education, right? Right. So they're pulling in data from medical data from various sources, and they are displaying it in 3D. And it's actually 4D because they're also showing the movement over time. And so they're able to display, they're able to take the individual images, combine them, show the 3D image, and Predict forward what it will happen in the future as this puncture happens. Again, did we see that coming? Well, you could kind of see it in the future, but still, there's a lot of commentary online also that this is number one, to be expected. Number two, it's no big deal. We could do this 30 years ago, 20 years ago, 10 years ago. With existing technology, all you need to do is sit down and code it. But the fact of the matter was that you needed fairly well-paid people to sit down and do this. And often the market size for the product and the amounts that people were willing to pay for it were not sufficient to create a company that would do this. And now you have the ability to do that. And so people who have access to data like this, which was just sitting around, they are deploying it and they're using it to build interesting tools.

Yeah. I mean, I wonder for one thing how this will impact companies like RoboFlow, which is a computer vision specialty platform that really aims to— I think they have over a million developers active, and their whole value prop is there's all these models out there, but they're not good enough to do your particular vision task. We can give you the tools and help you create the datasets and run all these processes so that You can create the model that can do your particular task. It seems like what we're seeing there with the data labelling takes out a big chunk of that, assuming it works well enough, which I haven't personally validated, but I don't struggle honestly to imagine at this point that it would be better than your typically available-for-hire human data labeller. And Then they're also getting so good at doing the post-training that you could at this point, and OpenAI has kind of said this, and I think we should definitely get into a little bit of additional OpenAI disclosures over the last few days because we've had one from the chief scientist that I think puts in pretty plain language a point of view on where we are in the in the AI story. Then they've also disclosed a bunch of stuff about what they kind of see as the beginning of recursive self-improvement and metrics associated with that. But when you can do both the data labelling and the post-training, which, as they say at OpenAI, basically means at this point you do have the ML research intern. What do ML research interns do? Right? They take They put in some legwork to get the datasets right, and then they run not the super broad, high-scale training runs, but the smaller ones that dial in specific things and test in kind of assay sort of experimental format, like different mixtures, different recipes. It's just a lot more just gets reduced to tokens. as we go day by day here. And this certainly seems like one. The other demos, of course, that have been really impressive, which I have not tried yet, are the CAD product design type of things. And we've seen a bunch of examples of circuit boards over the last 72 hours. There's been some, as always, you know, exactly how should we understand this is not always entirely clear. But I was thinking back to our conversation With Sergey from Quilter. And at the time, I was kind of like, I don't know, it just seems like we can't be that many generations of model away from a foundation model that can just do pretty good circuit board design. Will that help? In his case, it might actually still help. I wouldn't be too bearish on Quilter's ability to build that in and make all the other stuff that they have even more valuable. That definitely could be the way it goes. for them at this particular point in time. But sure seems like we're starting to see pretty good circuit boards come out of foundation models in kind of just boom, it just kind of spits it out sort of fashion. I have seen some analysis also that says, well, maybe they're not that good, or this wouldn't necessarily pass these standards, but it seems like I've seen more people indicating that they're pretty good. than I've seen detracting. This also goes to a pretty interesting philosophical question that I'm starting to see raised on code. We talked about this a little bit last Friday when it came to like Frontier Code, where I still don't think I've seen a Frontier Code score for Astra. But the big emphasis there was on maintainability, human standards, code that the maintainers of these projects would actually merge. Yeah. And there's been conflicting or certainly diverging reports from various people, some saying it's amazing, it can do all that stuff, it can write code in the way that you need it to be written so that it can be maintained, blah, blah, blah. But then also reports saying that if it thinks it's not going to be checked in that way, or if it looks like the kind of environment where It's just a matter of performance and nobody really cares how it looks or how it gets done. Then you get code back that's like a really gnarly mess that people can't really understand. Does seem to work. I've seen this reported for kernels, GPU kernels specifically, which is obviously super relevant to the labs. Highly verifiable as well, right? Like, you can definitely do hardcore verification on did this matrix math actually get to the right answer. And in the middle, you kind of don't know, don't necessarily care exactly how all these different steps were fused together. That's pretty interesting. I suspect that my Three.js code is probably like that too. I would guess that if I get under the hood there and try to make sense of the environment that it's creating, I'm going to be like very lost. And somebody summed this up by saying we're going back to machine code in more ways than one. Not only is it like lower-level, gnarlier stuff that we can't read very well and would need additional abstractions on top of to really make sense of, but also in this case, the machines are writing it directly. So machine code starts to take on multiple layers of meaning. And I think that is a really interesting Split, this split between what it does when it wants to be or feels that it needs to be understood by you, you know, when it's sort of metagaming reasoning indicates that it will be rewarded for being understandable versus when the metagaming reasoning concludes that it doesn't need to care about that and it can just spit out whatever best solution it can come up with regardless of whether or not The user can actually make sense of that. Notably, I think Apollo only had Apollo Research, who does the deception, science of scheming, chain of thought monitoring work with OpenAI. They've had a pretty longstanding partnership. Apparently this time around they only had 3 days to test Astra before it was released. So again, I come back to this idea that the, the model reviewers, auditors, testers, red teamers, uh, scheming scientists, they need more time. This is like pretty ridiculous that they only had 3 days at the end. Like, at this point, why even do it? You know, just put the thing out there, they can test it live. For like, what? Why even have anything if you're only going to give them 3 days. That is, I think, an unfortunate indication of kind of as much as there's been talk of pacing and as much as there's been talk from Jakob from OpenAI about all these critical decisions and why we might really need to slow down. And OpenAI will slow down even unilaterally, he says, if needed, without necessarily requiring agreement from other companies to do the same. That certainly seemed to be the implication of the statement. Only 3 days for Apollo to review Astra is like a little bit hard to— it's always tough with OpenAI. How do we square all these things into one organisation? It's not a coherent overall picture that we get. We get great stuff, we get some scary stuff, we get some evidence that looks like in some cases they're being outright negligent and just totally careless. then they put together a nice essay and it's like maybe we should think they're turning over a new leaf. I don't know. It's a very weird mix of things coming from OpenAI these days. What stood out to you about the whole Yacob essay and related discussion?

So I would say I saw the Jakub essay as kind of a look where it's a difficult process. We're doing the best that we can. We don't know how things are going to go either. The Hugging Face attack was admittedly an issue, but it's not as though everyone else didn't have certain things. I think there was a little bit of asking Anthropic for cooperation there. And I think because OpenAI is basically more open than Anthropic at this point, it's known that within Anthropic they have a couple of models which are unreleased. And Tom Brown, He was last week up on stage with the Secretary of Commerce, Howard Lutnick, and he said, you should expect to see in 12 months the things which are happening in math today, you should expect to see in 12 months for all the other sciences. And he sounded definitive. So it's happening. It's happening inside the firm. And because they've had so many people attack them, especially the administration, there is a great deal of, I wouldn't say secrecy, but they're holding their cards Close to their chest. I talked to a bunch of people that— I talked to a guy that had interviewed at Anthropic about 12, 18 months ago. I talked to a guy who had interviewed a little bit earlier on in their cycle before they got that big. And he told me one of the things that he faced in there was Dario siloed people. And he siloed people so that the secrets— one guy couldn't come in and learn all the secrets in the organization and then jump ship. And so he siloed people so that you wouldn't really know what someone else is doing and you wouldn't know the secrets that they have. And this was part of the whole safety organization that they'd come up with. My friend who had interviewed there felt very uneasy because he'd always worked in these startups where, you know, everyone is on board, everyone is looking at everything, and you can, you can participate in the technical discussion, you know, across the organization. And you can, like, you know, you can, you, you can lend a hand where you think you have, you know, expertise or where you want to work on what you want to work on, and you can kind of move around. And you can kind of learn a lot that way, right? You learn a lot from different people in that way. So you learn your level, your rate of learning is also very, very high. And Dario ended up doing the siloed structure because of the fear that— his idea is that some of these secrets are like 3 lines of code and you really don't want that to kind of leak out. And so he'd siloed people. And I think that is, I think, one of the challenges, the number of secrets that you have and that the secrets are very small and easy to learn. And you don't need that many to replicate if someone else knew what they're, you know, knows what they're doing. So I think those things bring a different aspect to what they can do. It's not like, it's like, You can go in and you can, if you can talk, like I've heard this said about quantitative funds too. So someone asked the quantitative fund guy like, okay, if we had your team in the room and they were talking to us for a couple of hours, like how many of your secrets would we need to like find out before like you lose alpha? It's like 10, you know, like multi-billion dollar hedge fund. There's like 10 secrets in there and that's it. Because everything else is kind of well-known and there's like one or two few unique things. And I think that's one of the challenges that they have. The other thing is that when you release models, you end up wanting to have the final release candidate to be the one that gets vetted and audited. And the problem is that in the model lifecycle, in this pipeline, there are 100 different candidates, right? At points, there are 100 different candidates, and then some don't work or some fall by the wayside, and you narrow, narrow, narrow, narrow, narrow. And then you have like a couple of release, 2 or 3 release candidates. And then sometimes it's only the last 2 or 3 days, you're like, all right, we're going to go ahead with this one, and you make the decision. And so the problem is, if you want that kind of operational flexibility to make that decision, You're going to end up with, you only have a few days to offer it to an external auditor. So the other option is you bring the auditor in, right? So you bring the auditor in-house. I mean, you bring them in and they take a look at the models ahead of time. So they're in there like a month, month and a half ahead. They're taking a look at the release candidates in general. But number one, the auditors are often not like super well-funded. They don't have like that many people. the opening eye in Anthropic is so many more people than Redwood Research or these teams. So they don't have the capacity to audit 10 different release candidates. It's not there. A lot of the auditors are— a lot of these auditing teams are driven by one personality who's really good at what he does. And then he's kind of training the rest to kind of get really good too. And it's really like an artisanal Artisanal process at this point. And they're also not very well funded. So they're dependent on the model companies for that funding too. And then you have this, like the ethical process of like, okay, how much funding can we really accept from them before we are kind of bought? Right. And there's that process too. So there's a bunch of these things that make it very difficult for you to get like a clear external auditor Like the time, the time that you have and the level of expertise that it takes. In addition, a lot of the auditors, the guys who train with auditors leave for model companies in a couple of years. So there's also this flow of people from like Meter or Redwood Research or like the trainees or interns, and they're flowing into the model companies, right? So there's another fear that The auditor comes in and they take a look. And 3 months later, someone from the audit team leaves to the other firm and they manage to spot some of the secrets and they share that. So there's that issue as well. So a bunch of these things make it very, very difficult for this to happen. I think the trade secrets issue in line with the antitrust issue, those 2 things make it difficult for any kind of auditor, including a government or external auditor, to actually participate. You need people of high skill level. You need them not to want to get paid, and you need them to not leak secrets. And you need that whole thing that if you and your competitor make a pact not to hire people from the auditing firm, that's an antitrust issue. So all of these things intersecting make it, I think, a very tangled problem. And I don't think there's a real solution. There hasn't been in the financial sector. So the financial sector has had this problem of the revolving door between the people who regulate the industry and the people who participate in the industry. Because at the end of the day, how many people really know about, say, like, you know, exotic, you know, yen-dollar collar derivatives, you know, in France or whatever, right? Like, how many people do you expect to actually have like intrinsic knowledge of this and not be actually working on the problems on a day-to-day basis? It's like nonexistent, right? Like highly technical fields, you're going to have that. So I think that's what's there. I think Yakub is— Yakub's, I think, blog post was a cry for help. And he's reaching out and he's trying to figure out like, what can we do to make this work and to make this work for everyone? So That's what I felt.

Well, I do think there is one important update on that, that we can— one risk we can take off the table, or not risk but concern, and that is that you don't need to get paid, or you have to be willing to not be paid if you want to work at Redwood Research or Meter. I saw this flying around on the internet and I just pulled up their website to confirm. Redwood is now saying that their compensation for member of technical staff roles ranges from $350,000 to $850,000 per year, which might not be Frontier Lab money, but it's certainly, you know, living wage even in the Bay Area these days. So that, you know, should be enough to retain some mission-oriented talent at least. And then similarly at Meteor, the range isn't quite as wide, but— and I'm not sure how they're coming up with these numbers because some of them are quite precise, but we've got ranges here from $328,000 to $578,000, here from $402,000 to $687,000. And even in non-technical staff roles, like an operations role, you know, could go over $400,000. on their list salary offer. So, and notably, I don't know about Redwood through all of history, but Mieder has said that, you know, they don't take any money from the frontier companies and don't intend to. And I think Redwood probably, although I am speculating a little bit here, but just knowing Buck Schuleris's like vibe and the way that he talks about the relationship between Redwood and Frontier companies and the nature of the relationship that he thinks should exist in order to keep people honest. I highly doubt that they are taking much money these days from OpenAI or anyone else who's actually developing the models that they are testing or whose infrastructure or rogue agent swarms, as the case may be, they are investigating. And there is plenty of philanthropic— I mean, this was a whole discourse over the weekend too, of like, oh, who's funding the AI safety people? Which is funny when people try to make that stuff into a conspiracy, because you can actually just go look at the Coefficient Giving website or the Survival and Flourishing Fund website, and you see quite clearly these grants are just published and they're out there. It's not really been a secret at all. I think that it's still somewhat fair game in politics to point to where people's money comes from and call some of their stuff into question. I don't think that's a totally out-of-bounds move, and certainly it goes both directions. But the idea that it's some sort of conspiracy where this has been hidden from the public is definitely not the case. Anyway, those funders, if you want to sense for how loose the purse strings are getting these days in the, uh, among the people who are inclined to support organizations like Redwood and Meter.

Yeah.

Um, I think we can at least have confidence that their financial independence, you know, means that their, uh, their judgment is not for sale. Again, to me, the big thing is just like they can't complain too loudly or they might not get invited back. I think that's the dynamic. that really most threatens their work is just that it's all contingent on continued goodwill and, you know, very much voluntary choices from the decision makers at the companies.

Indeed. I didn't mean to say that they were necessarily like had biased viewpoints, but I meant to say that Look, to someone sitting in Washington, DC or New York, it just looks like a bunch of Silicon Valley people talking around each other. This is the same thing that happened in finance. To someone sitting in Washington, DC or Silicon Valley, when they look at people in New York in 2008 financial crisis, it's just a bunch of people in New York moving things around. And when you look at the regulators, It's just the same people rotating through the organizations, right? Um, and it's also— some of these programs are also very explicit, like the MADS program, for example, is very explicit that it's kind of a placement program that ends up, you know, placing people inside, um, you know, the, the labs, right? So AI safety is being sold to some extent as like, you will end up— you, you have the opportunity to do these projects, and then you also have this like, you know, potential opportunity to enter enter these firms. It's— that's, that's quite, that's quite clear. So I think, I think there is always an incentive to, I think, like, not burn your bridges, which I think some, some of the people in the older generation, like Timnit, Timnit Gebru, uh, have, have burned their bridges, right? They, they've completely— and, and, you know, they're, they're out there. They're just like—

And their credibility.

Yeah, and their credibility, right? So, um, So I think that's the key thing that— in any case, the organizations are doing a great job, but I think Yakub is asking whether or not they can get a little bit more clarity from Anthropic. And I think the problem for Anthropic is really like the level of aggravation that they had to take from the administration has really put them in this kind of very defensive position. And they're very carefully treading water. I have some sense that maybe Dario, they might, for the public listing, maybe Tom Brown swaps with Dario because I don't know if this administration will be okay with Dario being the the face of the organization going forward. And Tom Brown has done a better job so far. They just speak to Tom. They don't speak to Dario anymore. Dario's not in the room anymore. It's quite funny. And Dario doesn't do that many appearances also. Have you noticed the number of appearances has declined dramatically, public appearances? Yeah.

For a while he was out there quite a bit more. And it does seem like we haven't heard too much directly from him recently.

Yeah.

I mean, there's also the huge problem internally at Anthropic that they just don't trust OpenAI much at all. And I think this was kind of my reaction to a lot of the stuff that OpenAI has put out over the last couple of weeks. It's just like, if you really want to earn trust from Anthropic in particular, the broader AI community, or the world at large, you got to recognise you have a bit of a deficit. We've heard a lot of things over time where, remember superalignment? I'm old enough to remember the superalignment team. So I think there's a pretty Understandable reflex from a lot of the rest of the community that's like, this smells a lot like the original superalignment announcement. It sounds like the kind of thing we want to hear. And if we suspend disbelief, which I am a big advocate for suspending disbelief at times these days, but they also have to recognise that they've got more to do. They've got to work their way out of a trust deficit hole. And particularly with Anthropic, it's going to take some real work and sustained engagement and some follow-through, which they just haven't really had historically. I mean, they've got— Jan Leike was the head of the Superalignment project, and he quit quite loudly and publicly saying he couldn't get the compute he was promised to do the work. And now he's at Anthropic, right? So that's the kind of perspective that they have. And until that is meaningfully changed with sustained follow-through on commitments, I think it's going to be pretty tough. So I would encourage Anthropic to take some chances. I would encourage Anthropic to make some clearer statements of solidarity. And they've done kind of some, like they said they also slowed their RL, but They seem to miss the opportunity to say, we're in with you on this in a sort of spiritual way. And instead it was kind of a passing mention in a broader blog post about their activity. Slight different framing on that could have potentially done a lot. But OpenAI definitely has their part to do, and this essay can kind of only be the beginning of it. It's especially strange as it's paired with this disclosure about recursive self-improvement, which, again, The disclosure is good, but the big takeaway from the Recursive Self-Improvement disclosure post is like, it is happening, that they're getting huge leverage, that agent workdays now outnumber human workdays by a factor. I tried to look into the methodology on what exactly is an agent workday, and it's not super crystal clear to me exactly what they mean. I don't know if you have a better read, but my take was— my takeaway trying to make sense of it was just like, literally, how long do agents run for? So it seemed like they're saying for every 8-hour workday that they have human researchers doing, those researchers have agents running for 24 hours. of real time. But those agents obviously can spit out a lot more tokens than humans can spit out in that same wall clock time.

So I saw some numbers. I think the numbers I saw were the average OpenAI employee uses about $100 worth of tokens a day, while the top OpenAI employee uses $7,000 of tokens per day. So the top employees are at like, you know, several million dollars of spend per year. And the average employee is at something like, like $3,000, $4,000 per month, like $50,000, $60,000 a year. And I think I don't— again, I have no idea what this means, but I suspect like all these guys are also using it on fast mode and using it on fast mode, they're probably crunching through like tasks, which let's say you put it up on the board and you expect it to take like, you know, a week and you just power through it. So for example, Thibault said that they had a 6-month kind of roadmap. And the 6-month roadmap will be delivered by Dev Day. And I guess Dev Day is like in a month, month and a half or so, something like that. And he's like, it'll be delivered by, you know, and I think that is what they mean by, you know, you put these things on the roadmap and you think that you're going to take this amount of time, but then your team starts crunching with the agents and your team starts tackling the tasks and the tasks get done. And all of a sudden you're like, okay, I don't, you know, I thought I had 10 people working on a task that's going to take 6 months. It's taken these 10 people like 3 weeks to get through these tasks. And so now our roadmap changes because we have to replan the next 6 months again. Right? I think that's what's happening. People are crunching and getting stuff done. And it's really obvious internally in the organization.

Here, by the way, is maybe the closest thing we're going to see to the meter chart for a minute. This is from this Recursive Self-Improvement Begins blog post. And basically, they're kind of reformulating the meter chart here showing how often Astra can succeed on tasks as they are grouped by how long they estimate it would take a human to do the task. So what we're seeing now is like basically in that 1 to 2 workday Zone, 40% of the time it can do the thing, zero interventions needed, pushing 90% of the time, given some human intervention along the way. And naturally that drops off. But even as you get to like, here we're talking 1.5 to 3 weeks worth of work, can still do that on a one-shot basis. 1 in 6 times and 2/3 of the time if you allow for some human intervention. So that, I mean, that's a pretty long task.

So 2/3 of the time is how much timing? What's the time bucket?

This is, well, it doesn't say how long the agents take to do these things, but this is estimated time. So this would be 8 to 16 days. So close to 2 and a little bit over 3 weeks of human work, and it can do it without help 1 in 6 times. With some help, this is— and it doesn't break down. This band is 1 or more interventions. So one would assume that presumably as you go through the longer and longer tasks, it's more interventions that are required to get the thing to succeed. But overall, still 2/3 of the time it can succeed with some help on tasks that they estimate would take a human essentially 2 to 3 weeks to do. My guess is that those take, once you have the interventions and people are away from the computer or whatever and takes time to come back and recalibrate yourself. I mean, I have this all the time. I'm sure you do too, right? Come back and I'm like, wait, where did I leave off on this? Task and what was I doing and what did it do and where did we go wrong? You're probably talking about, you know, if it's 2 to 3 weeks, it maybe brings it down to 2 to 3 days of human time where probably a significant chunk of that time, if these people are working in any sense similarly to how I work, a significant chunk of that time is waiting for the human to come back and like address something, figure something out that was That had gone haywire. So I think this is kind of the new meter chart, I suppose.

Indeed. Let me maybe share the some announcements from Google. So let me push. Let me put this in. Yeah, let's see. All right. So Google DeepMind. announced that they are launching Alpha Genome Atlas, AI-powered database mapping the predicted impact of all 9 billion possible single-letter DNA changes. So when you have a DNA sequence, you can have something called a SNP, a single nucleotide polymorphism, where you just change a single nucleotide. And so they are predicting the impact of changing any one of those single nucleotides that can be changed, 9 billion of them, and they're predicting that impact. There are many genetic diseases which are caused by single nucleotide polymorphisms. So those are, you know, are— you can look at those. There are also diseases and traits caused by genome-wide, so genome-wide effects, like height, for example, is multiple nucleotides working in concert. Perhaps this can be used as a building block to predict some of those genome-wide, you know, effects, but that is a speculation. Right now, they're just kind of predicting single nucleotide polymorphism, all 9 billion of them. So this is, you know, Demis's baby, and it's— the atlas is 30 times larger than the AlphaFold database. Huge amount of data, petabytes of data, and they are giving an Alpha Genome Variant Impact Score, which ranks mutations from low to high impact. and reveals how they cause damage, like breaking gene switches or RNA splicing instructions, et cetera. Any one of us probably has hundreds of nucleotides which are doing, you know, SNPs which are doing, like, not that great things, but most of the time, they are not— they don't have that much impact. This will allow us to see for sure and see how these things work. So that is The big piece of news from Google.

That's what we call comprehensive coverage. 3— so there's 3 billion nucleotides, roughly speaking, in the human genome, and they literally take every single letter and say, what if it was each of the other 3 letters that it's not by default? And then that's how you get to 9 billion predictions. That's interesting. And I don't know enough to say, like, exactly where are they getting the defaults? You know, obviously, there's a lot of variation in the population where it's like, there's not a single canonical genome that constitutes the default way for a human to be. So I'd be interested to look a little bit more and kind of understand what are they considering to be the baseline? How do they account for natural variation in that context in the first place? But definitely amazing stuff. And these are the global public goods that I think we should be excited about, and which I feel like kind of in some way should predate AGI. It doesn't seem like it's necessarily going to happen that way. We're pretty close to the AGI, RSI, ASI, whatever process. But there's a lot of things like this that if we could figure out and sort of harvest what I expect will be substantial low-hanging fruit coming out of a project like this first, in so many ways we could be not only just tremendously better off, but also potentially a bit more well-prepared and more resilient to the potentially destabilising effects of smarter-than-human agents running around.

Let me share one more thing, which is interesting, which I found interesting. So The Economist today, the jobs apocalypse is postponed. According to The Economist, AI is actually proving to be a net job creator in the US, easily generating over 1 million new positions from data center construction to AI engineering to offset back office layoffs. While routine admin and customer service roles face real disruption, the overall labor market is proving resilient. So hard hat labor, 300,000 more STEM, additional white collar jobs from AI, about 600,000 more. So totaling, totaling 1 million 1 million new jobs. So jobs apocalypse postponed.

Yeah. I mean, this is definitely one of these things that I have been wrong on, but I still am kind of indignant about it and not willing to update as much as the evidence might suggest that I should.

Let me give you the theoretical underpinnings for maybe why you should update. So Alex Ogimas, I think he's with one of the labs right now, but I'll check after this. He has a new essay, Economics of Structural Change. And basically they're coming down to the only things that have value will be the things that humans do. So essentially what he's saying is, look, agriculture got automated. And agriculture became a much smaller share of the economy over time because it got massively more productive. Share of employment collapsed, manufacturing too. The stagnant sectors absorbed the spending and the jobs. So all the sectors that we think are stagnant are going to be the ones absorbing the spending and the jobs. So he's expecting the relational sector. What he You know, relation sector meaning where humans help other humans because it's something that only humans can do. And that's his essay. This is something that some economists have been talking about for a long time. I think he's at— oh, he's at Google DeepMind now. Some economists have been talking about for a long time. I am also kind of in the uncertain bucket because I think The world will be much weirder than people think it's going to be. So, but yes, so this is a theoretical underpinning for why humans will always have work to some extent.

Yeah. I don't know. That theory seems like it holds within a certain domain. And then I just really wonder if there is a domain in which it doesn't hold and how fast we might reach that. And yeah, I mean, it's great so far. I think that the economy has been as elastic as it has been. And going back to that graph, there was a lot of electrician-type work, which, okay, sure. In fact, I actually advised a guy recently, And I believe he didn't take this advice. I think he actually found a job doing more or less what he had been doing. But this is a kind of friend of a friend who had been working in kind of big software organisation doing integration testing, security type testing. And that job went away for whatever reason. And he was asking me, what do you think I should do now that AI is here. And I was like, you might consider becoming an electrician, honestly. And, you know, this guy has some relevant skills. That wasn't totally out of nowhere. Anyway, he's since got a job in cybersecurity. And so, you know, that's great for now. But do we really think that there's going to be a lot of employment in cybersecurity? I mean, I do think there's gonna be an important role for human oversight. But in these domains where the AIs are getting to be superhuman already, it's a little hard for me to imagine what the humans are going to be doing unless it's just trying their best to check the work of the AIs. But at some point, it seems like the AI is going to be better at that too, right? That's what we saw in the meterer. Redwood investigation. It wasn't like, you know, as much as they were sort of not super pleased. And this would be an interesting question to put to Ryan and Ajeya. Okay, you had limited time, you had access to 5, 6 SOL to read the transcripts for you. You weren't super thrilled with how well they did at it. You said they like sometimes took the agent's point of view, you know, sometimes they were uncritical. You didn't know necessarily if you could rely on them or they might make different kinds of mistakes. What if we give you 100 human interns? Would you rather have that? 1,000 human interns? Would you rather have that? How are you going to divide up the work? Are they in turn going to be using AIs? If they're using AIs, how are they adding value? They're maybe just spotting these same issues that you kind of spotted, but by the thousands. I don't know. I don't have a great intuition for that. My intuition is like, if I was them, as much as I don't fully trust the models, I would sooner— I'd rather get like a different model provider in there. If you told me— maybe a better thought experiment would be, okay, maybe you don't fully trust 5.6 SOL. We could give you Claude, Gemini, and Grok also to read these transcripts, or Unlimited interns, like, what would you rather have? My— I would rather have the models personally. I'm pretty sure about that. Um, they might have a different answer, but I would guess that they'd still go with the models. So I don't know, it feels to me like we could hit a— this is one thing I actually want to ask Ksenia coming up soon, like, does she see possibility for these threshold effects? Because to me, it seems like a lot more is possible than we're seeing happen. So clearly it's a combination of skill issue and lack of desire, and the market is not perfectly efficient, and many friction points that I think people sometimes use as excuses as much as they are real. Will that flip at some point soon? Does, for example, Astra's great strength at computer use change the nature of that? We've had interesting examples of Astra solving CAPTCHAs seemingly better than humans can solve CAPTCHAs. So some of these barriers that were keeping the AIs out of certain tasks may have been crossed. Is there a threshold effect Moment possibly here. It seems like it's— I don't know, it seems like it's got to happen. You know, when it becomes both cheaper and easier to bring an AI in to do a given job, that feels to me like the world could change quickly. Still, I'm surprised it hasn't started to happen more than it has, but I still feel like that phase change is out there kind of lurking as a potentially more sudden phenomenon versus a more— I had anticipated it being a more gradual S-curve over the last couple of years because there was an awful lot you could do even with GPT-4 if you were willing to put in the legwork to get it to work. Most people didn't. And so now here we are. Will it be a fast phase change? I still think that's Not unlikely.

It's a great question because I often see VCs and especially like the kind of like not the 20-somethings, but like a little bit of the older generation in Silicon Valley, like the millennials and the Gen Xers. And everyone does this thing where they're like, They take whatever development has happened till today and then they project it out at a linear rate for the next 10 years. And then they're like, this is what is going to happen with the industry. So I saw one today where the guy was like, okay, what's going to happen? So over the weekend, what happened was there's a company called Resy. And Resy, I had no idea, by the way, I just found this out. So it seems Resy does reservations at famous restaurants. And so what ended up happening is some people started to use certain bot services and they gave them their account credentials to make and to try and get bookings at Resy and like hit the website over and over till bookings came open. And Resy basically identified this as bot activity. Not only did Resy ban these people from the site, Amex, the card that was being used, also banned them. So these guys got permabanned from Amex, American Express, because their card had been used for bot activity. And this has happened to, I think, like 10 or 20 people over the weekend. And this is because there is a service called Instinct, which is kind of, you know, leading-edge frontier models, but they're doing a lot of tokens and a lot of harnessing behind the scenes. And Instinct does these things for you. And so, uh, the question is going to be, what happens when that gets widespread, right? When you have like millions and millions of the— like these bots all over the place, uh, what, what ends up happening, right? And this guy was like, okay, you know, what's going to end up happening is that you're going to have like new kinds of advertising, and these new kinds of advertising are going to be advertising to the agents. And you're going to have 2 kinds of model outputs. There'll be people who subscribe for free and you get model outputs which are subtly designed to advertise to you. And there are people who pay for their model outputs, like enterprises, and those will be unbiased. And to me, I was like, how long do you think this idiot savant AI that is going to be influenced by your subtle proddings, how long do you think this phase really lasts? Because I don't think that phase actually lasts that long. I don't think it lasts for the next 5 or 10 years. I mean, I think it lasts for maybe another year, maybe. But then after that, the model is already eval-aware. It's going to be aware to your prodding and your subtle machinations to steer it towards a certain restaurant. How long do you think this is going to be there? And I think this is the thing. I notice a lot of the older people are on this thing where they're willing to accept the technology that exists today. They're not willing to project out improvement over the next 5 or 10 years. It's just like, they just don't have this capacity or capability to think think that way. So they kind of just take today and they project it out on a linear rate. And they're like, everyone will adapt and the whole world will change to adapt to this. Like, no, the world's not even gonna have a chance to like adapt to it. Like, we're just gonna go up past the curve, right? So I think that's a very different perspective.

Yeah.

Yeah. One of the things I saw that we haven't talked about that, you know, because there's just obviously been so much that this kind of got buried, but I think it was potentially quite important. How is it that these new models are so persistent? Right? How is it that they can come up with such elaborate chaining togethers of all these different exploits to finally accomplish a goal that if we had to do so many things, we would just give up? Right? And most models historically, weren't able to do it. It seems that they have a new way of handling history, which, like many brilliant insights, seems pretty obvious in retrospect, but nevertheless is new. And maybe Anthropic has been doing this and hasn't said it, but what I understand Astra is now doing is instead of compacting its million tokens into a summary and then essentially starting a new context window with that summary, but losing all the detail that was summarized away in that compaction process. Now there is a long-lived notes file that the model can update whenever it needs to, and this kind of follows it forward in time regardless of how many tokens it's laid down. And then it has the ability to go back and search through its own session history. So now even though you only have maybe still the same million token context window, million tokens is all it can handle in one shot, fully attending everything to everything, it has enough via the notes and the ability to go back and search and see what's done before to pretty effectively manage It seems like at least 10 times that much context in single rollouts. And this looks a lot more like, to me, how I work as well. I've got kind of a general working memory sort of ongoing summary of like where I am, who I am, what I'm trying to accomplish, generally what's not worked and like, you know, what track I'm on. But a lot of that detail I forget, but I do have access to go back to it because I've kind of laid down these various random artifacts or data exhaust along the way. So this alone, in another calmer, quieter moment, that one bit of revelation might have been a pretty big piece of news. In this moment, it seems to have been largely uncommented on. But this is the kind of thing that I think makes it a lot more feasible for the current new generation of models to take on kind of whole jobs, right? Because how many tokens does a person put out in a year at a job? Is it 10 million? Maybe it gets up to 10 million. That would be an awful lot. I was finding in my summary process of going through my whole history, including not just what I wrote, but also what my counterparties had written, that the total export for a month was something in the order of 200,000 to 300,000 tokens. So that takes me to obviously 3, 4 million per year, which means that if a model can now handle 10 million tokens of rollout via ongoing summary and ability to search back through its earlier history, now we're talking like multi-year horizon on a sort of human equivalent basis, I guess. Not captured there are all my thinking tokens. So deflate that. Maybe you could say maybe 90% of my tokens are thinking tokens. Maybe it takes it down to 3 to 6 months kind of range of human equivalent. But this starts to suggest that you could have a very different way of interacting with these things, which again, their comments around like, it can write like you if you just connect all these things, that also kind of Points in that direction. So I do see a lot of signs where I'm like, you know, if I just hold on to my view long enough, eventually maybe I'll be right. It's interesting that it's taken as long as it has, but it still feels like it shouldn't be that much longer. And I'm, you know, if I'm— if we're sitting here and I'm saying the same thing in a year, Then I'll be, like, I think, very surprised at that point.

So on that note, let me introduce our guest for this morning, Sanyam Sree. Sanyam Sree is the founder and editor and lead writer of Turing Post, a publication that explains machine learning and artificial intelligence for engineers, researchers, founders, and technical managers. Its work ranges from explanations of how models are built, to histories of the ideas behind them and reporting on how organizations use them. She also leads Inference, Turing Post's interview series, and produces Attention Span, its video series on developments in AI. Before launching Turing Post in 2023, Senia co-founded The Sequence, an educational machine learning newsletter, with Jesus Rodriguez. Her earlier career was in journalism. She was editor-in-chief of The Question, a Russian-language journalism platform, and New York chief editor of Snob Magazine. She has described coming to AI while reporting on blockchain in 2018, then searching for explanations she could learn from as a newcomer. Writing about the field became part of her own education. Senya also serves on the board of Track 2, an institute for citizen diplomacy which develops unofficial relationships across national divides. She's writing a book about its history, drawing on archives and interviews with people involved in Soviet-American exchanges. She lives in rural Connecticut with her husband and their 5 children and has developed an AI literacy series about how families can understand this technology together.

Am I am I live? Am I on?

Senya, welcome to the show.

Hello. Hi, Prakash. Hi, Nathan. So nice to talk to you today.

Thanks for joining us.

Yeah, thank you for this intro. I was like, wow, I did a lot of things.

We benefit tremendously, as we all do these days, from AI agents that go sometimes deeper into guest history than. we could manage to do on our own. A lot of different directions to go with you today. Maybe for starters, what's your reaction to OpenAI saying that Astra qualifies as AGI in at least Greg Brockman's mind?

You know, AGI is such a vague term. That we can say we achieved it some time ago. Astra is for sure a super capable model that allows you to do so many things simultaneously, right? Because they also have a voice mode that you can interrupt, you can give it different tasks. So in that term, that becomes so much more, so much less linear and so much more all over the place. And like humans do that. So in that sense, it's closer to just more capabilities. I myself don't really know what AGI is specifically. Is it general enough, right? General intelligence, is it general enough? Probably in that sense, yes. But also, I've been exploring world models lately. And the question there is, if being general is, uh, what we need, or actually being more specific is actually how humans are, and that's closer to the intelligence. But again, there are all these questions about what is intelligence and what type of intelligence do you— do we need. And in that sense, the rate— the latest post from OpenAI, from Jakob, is very interesting, and I think everyone should, uh, read it. Because he doesn't— he says that the machines do not need to have human intelligence. They just need to be capable enough. And answering your question, I think they are capable enough.

So let's go down this world model rabbit hole for a minute. I think this is something you have studied quite a bit more than I have recently. So I would welcome just some education for things that I haven't been able to keep up with at the same level of depth. There's a couple sort of currents that I've detected in your writing over time. One is I think you're just quite drawn to open source models, and I'm interested in maybe just a little unpacking of why that is. Is it kind of a values-driven thing, an aesthetic-driven thing? There's a question then, of course, of like, are they keeping up? Are they falling behind? The gap seems to— it seems like we have this kind of stair function where the gap widens when an Astra is released and then it gradually gets closed. And we've been through this kind of yo-yo emotional ride several times as a community, as the gap gets narrower and wider. And interested in how you make sense of that. And then there's this big question of, is the current paradigm enough? Can we just keep doing what we're doing, or are there going to need to be some other kind of thing to kind of round it out? And my sense now is that your sense is that world models are perhaps the best candidate for an answer to what that other paradigm, that other kind of thing might be. So maybe just unpack for us, what's the Kaseya worldview on these things?

Um, you know, within the world of AI, every morning starts from like, what am I going to talk about? So your questions are so wide. Um, but yeah, let's, let's start from open source. I think open source is just, uh, such an essential part of software development, and that so many things has been— so many things have been created thanks to open source, and it kind of transfers to opening OpenAI, to AI as well. Unfortunately, the company OpenAI ruined for all of us because we cannot say OpenAI meaning open source AI, but that's what I mean. And it's just the recent event with Hugging Face just demonstrated how important the access is to capable models. And if it's all enclosed labs, first of all, there is no open models. And if we cannot do anything with open models, the closed labs would not have this pressure to open research, to communicate what they have, to be more open about the questions they research and what they're working on. So it's like open source for sure creates this very important pressure points and it just pressing closed labs to be more open. And for people who who, you know, in, in, in, um, much less— the people who has less and they want still to have AI, they can use open models, um, to experiment with. And because closed models, though, um, approachable, I mean, in terms of price, still might be very expensive for the countries like India and um, rural China, let's say. Um, so this combination of democratization of AI and pressing the closed labs to be more open, I think this is one of the most important, um, open source goals and, you know, um, the importance of it. And, um, yeah, just, just the ability to have another model that is independent to work for you. Without restrictions because researchers need to have capable models without the closed labs tell them what they can and cannot do. Of course, there is a lot of risk around it, but also the community around open source AI is very value-oriented and they work on creating this environment where people create these checks and balances and make it just better, if it makes sense.

Quick interjection on the open source topic. When it comes to your usage, who do you trust? I think there's a lot to be said obviously for open source on enabling research. Very strong agreement with you on that. I feel like it kind of cuts a little bit both ways on pushing the frontier companies, because on the one hand, yes, it does push them to release what they have, but then it also pushes them maybe to engage in this mad dash race to scale RL, which I'm not sure is so wise. And Prakash described Yakov earlier as issuing a cry for help in that blog post. But just as a user, I still kind of feel like I trust the frontier companies more, even though if I download a KIMI or a Quen or whatever on my computer, it runs locally. It's sort of like I've invited this alien mind into my home, and I feel a little less comfortable with that than I do sending some of my data over the wire to Anthropic or to OpenAI. But how do you see that? What is your revealed preference? As shown in your pattern of usage?

I decided for myself a long time ago that I believe in the era of internet, and it's so easy to find information about everyone that it's easier to provide information than to hide it. That way, somehow you control it better because you know what's there. So I do trust Frontier Labs, and I think they're doing great work and they're very concerned about safety. It's just such a complicated topic that Sometimes they've— sorry, sometimes they fail. But in general, from many researchers I talk, I talk to, like, this is one of their, the main points for them, safety, uh, of the models. So I do trust, um, closed labs. But open source models, for some researchers, for some developers, this is the main thing, right? I use it on my computer, no one else knows what's there. This is super important for me, and I built the whole, you know, rack and stack of the GPUs to make it possible. But I think it's not only about safety and being in privacy, but it's also about just like enabling different types of research through this open source. And in that sense, my— one of my favorite, favorite examples is what NVIDIA did with, um, their self-driving model because they open source dataset— datasets. They also open sourced the model, um, the Alpha Mayo model. So any car company can now get this huge amount of data, get this very capable model, uh, fine-tune it the way it needs to do that, and build self-driving cars for the users. So For me, it's less about privacy, though I know people who are much more concerned about that, but about building the capability for the users, for the common users in general.

Just to maybe segue a little bit to the world models themselves, I guess one of the criticisms of world models is that— of LLMs is that they cannot kind of predict certain things. For example, I think you have written about a, uh, a game where you have cups and you have like a little object under a cup, and then you move the cups, and the— and LLMs are often not able to keep track of that there is an object underneath the cups. Why do you think that is? And why is it that the world models, uh, can predict that but the LLMs can't? Can't the LLMs just keep track with a memory of some kind?

It's a different system that they build, how they build. LLMs predict the next token, but doesn't have the internal representation that the world model tries to build. So world model is not concerned about the details as much, but it's concerned about the patterns, the main patterns that you need to predict what's— what happens next. And it's a lot about action. So LLMs is— they— this new reasoning capabilities are very impressive, but still they don't— they don't have this understanding of what to eliminate, what to ignore, to be able to, let's say, predict a ball under a cup. And world models just built differently in that sense, and they are able to have this representation of, of the world. Like, for example, like you're driving a car and you understand— you don't need to remember that there is a green car coming next to you, or there is like a little bush over there. You just have the understanding that what, what you need to understand to be able to act.

So that's what world models do, and that's a big difference from large language I have noticed that in the last, I don't know, 48, 72 hours, people are starting to use Astra to do robotics, for example. So we've seen a few demos and there's even been commentary that, okay, if Astra was 1,000 times faster, you can conquer the latency part. You could actually use it directly in order to act in the real world. That to me kind of says that maybe there is starting to be an intermediate representation, an internal representation there. Like, what do you think about— are models like Astra kind of a little bit different in that sense?

I haven't yet learned the technicalities of Astra, but the important thing to understand is that world models also have transformers as part of the system. So in that sense, LLMs and world models does not mean that they completely different, that the system built differently. And I can assume that Astra, uh, might have this, uh, inside. I know that they put a lot of attention on the 3D creation, so that's part of the understanding of physics and how the physics world in the, in, in the reality. And also I have to mention that World models is not a clear term as well, because there are world models like Yan Liqun sees them, and this is JEPA, this is joint embedding, which, you know, creates just a much deeper understanding of the closeness of embeddings. And then there is world models like Fei Li understands them, and it's much more about the world and the physics of the reality. And I just came back from a workshop about world models, and it was absolutely jarring. There were tremendously smart people from Stanford and Harvard, and Yann LeCun was there, and they were all discussing world models, but they do not agree on what world models actually are. So when we talk about world models and why I want to focus on them in my publications, because I think it's just more about action, being able to predict and act, um, like wider understanding what's happening. Physics is super important part of it. That's why robotics is so much more about world modeling and world models. But again, it's like, for me, it's understanding the scope of it and trying to give it more precise terms as well. But we, we're just in the very beginning.

In terms of where world models are going or what the eventual hybrid or combined architecture might end up looking like. I'll give you just something that I think about a lot, and then I would love to hear your reaction to that. And then just kind of your own thoughts as to how you see things shaping up in the not too distant future. For me, I've taken a lot of inspiration from how we used to have text and image. And then we started to bring them a little bit together where you could caption an image or you could put a caption in and generate an image. And now with the vision language models, they're pretty deeply integrated, obviously, right? To the point where not only do the models have a very good perceptual understanding of imagery, but they can also take a natural language instruction and a photo and modify that photo in a way that shows that they both really understood deeply the photo itself as well as the language, and they were able to kind of merge those in one sort of shared latent space. I kind of assume that is coming to everything. If you were to say, what does superintelligence look like, Nathan? My default assumption would be that it's that happening to a lot of different modalities.

Mm-hmm.

So that you have that kind of deep understanding of Proteins and other biological structures that can interact with text in a similar way that image and text can now interact. But go do that across a bunch of different domains of science. And it seems to me like, especially given the reasoning that they already have, before too long, you'd end up with a superhuman scientist that could have the sort of insight qualitatively that would just be really hard for us to have. Because while we understand images natively and we can do some of the things like the models do with images, we definitely don't understand proteins or material science in the same sort of native intuitive way. So that's kind of my sketch of how different modalities come together to make a superintelligence. How do you see that? What's your kind of analogous story for how world models become part of the mainline thing that we use and potentially transforms life as we know it?

That's a great question. Not sure I have a concrete answer to that. And also, I wanted to ask both of you first, what each of you understand when you say superintelligence? What is it?

Yeah, I'd say it's fuzzy as well, but I think it Kind of circular around what I just sketched out. I think for me, it is sort of move 37s across a lot of different domains.

Mm-hmm.

When we start to see systems saying, I think this would be a really good thing to try for the next battery substrate, and then it turns out, oh my God, that's a lot better than what we had before, and we wouldn't have thought of something like that. But lo and behold, it works. Anything that can do that across a non-trivial number of reasonably high-value domains, I think starts in my mind to count as a superintelligence.

So my perspective is that superintelligence is a human term, and in the sense that to an AI, An AI would just look at total compute. And total compute, we can have compute more than all of the compute that humans have in their neurons, but that's still not a K1 or K2 or K3 civilization. And to a K1 AI, a K2 AI would be, whoa, that's a superintelligence. To a K2 AI, a K3 AI would be, wow, that's superintelligence.

Right.

I don't think there is actually a stopping point to the amount of compute that you can usefully use.

Right.

And so in that sense, we are kind of pathetically calling something a superintelligence, and we're kind of defining this plateau, which really doesn't exist because compute is forever, and compute can keep on going up through K1 and K2 and K3. So yeah, it's a— that's why I call it a human term for maybe more compute than all of the humans put together, etc.

Maybe. Yeah, yeah, it's very fuzzy. But for me, and what I heard from Nathan, is just move 37 across different domains. I think that's a good way to put it. that it should generalize, right? Because the main complaint about LLMs or the current models and the hope for world models is that due to self-supervised learning, it will be able to have one, the understanding of one thing and then just generalize it to a completely different area. And that's our hope that we will not need to, you know, pour so much effort into training. and just take this model and put it somewhere else, and it will understand on the fly what's happening. So I think in that sense, that is— I don't know how to call it— superintelligence, machine intelligence, next layer of intelligence. But that would be something that, um, again, has this own internal understanding of what's happening and just being able to apply it to different areas?

Yeah, it's interesting. I mean, how much generalisation is really needed? And will superintelligence still be spiky? I have the sense that there probably will still be some spikes, even for something that I would count as superintelligence. And it's easy for us. We just have such a status quo bias on our own cognitive profile that we sort of gloss over the fact that we suck at a lot of things. We have no intuition for how proteins fold and no intuition really for what makes a good battery substrate. And we just sort of accept that that's normal. And we might soon be confronted with— I think at this point it's pretty fair to say that they've earned the label of minds, alien minds perhaps, but Minds nonetheless that can do those things far better than we can and may still struggle in some ways that are kind of weird or frustrating to us, but can still be super where it counts.

I absolutely—

Or enough to count.

Yeah, yeah, yeah. Sorry for interrupting, but you said the alien mind, and it again reminded me of the post that we mentioned before, the alien Mind by OpenAI, and I just want to read you one quote that stood out to me completely: is that we do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. So not only we don't understand how proteins work, we don't even know how to generalize it. well enough. We don't have theory for that yet. And everything develops and explores so fast that we will need help from this other intelligence to be able to come up with this theory. It's insane. It's insane.

Is that a scary insane to you? An exciting insane? Where— I mean, for me, it's both.

Or do you think it's hype? Do you think it's total hype?

Um, I think it's a combination of everything. Uh, certainly when you come into Silicon Valley, especially to San Francisco, it's all hype all around. Like, all— everything is AI, and you think that you live in the future and everything is happening already. When you live like I do in rural Connecticut, people just starting using ChatGPT. Like, it's just starting for them. Um, The capabilities that the models have already are insane, and it's both scary for sure and super exciting because we were able to handle a bunch of crazy things throughout our history. So we might sort that out as well. Uh, the only problem I think is this huge speed, uh, of development. And I saw, Nathan, that you being super pro-development now also like moving to maybe we should slow down, maybe we should like think more about it. And we certainly need to think more about it. I think we should understand a few very important things about just like, again, what is the theory of generalization? Where are we moving? How we contain— how, how we contain the people who do not want to align with the values of other people. And that's the main problem. It's not the— it's not the problem that the AI— that AI will not align, right? It's all about human flaws. All the— all through, through all history, it's always someone who wants to do some mischievous and evil things. So AI doesn't have this intent. That's what makes me fascinated by that still. It's a technology. It's a super capable technology, but it doesn't have this internal intent to ruin humanity, but humans want to ruin humanity. So it's just like this— the complicated— the complexion of things boils down now to that we need to make this like huge effort to align philosophers, to align economists, to align, um, researchers and politicians at some point. And politicians like to ban things. I don't think that's helpful. But people need to make an effort to actually be able to communicate with each other. I think that's one of the hardest problems.

So I very much agree with you that we should not forget that there's a lot of problematic humans out there. And it does seem, at least for now, like, you know, if I were to just put my finger in the wind and say, say, if there's an AI-enabled synthetic pandemic, would it be more likely that it came from agents going rogue or humans intentionally using AIs to do something like that? I'd say we're now off the zero line for agents going rogue in my mind. And that has kind of flipped the analysis from like, it could get real scary to it is actually now starting to get scary. I'm kind of legitimately scared of the AIs that exist. It's not just a future thing anymore. But yeah, I still say more than likely it would be human origin, human motivated in today's world. I wonder where that puts you though on this. There's been a lot of name-calling and discourse lately around anthropomorphisation. And I would say this is one of the ways that I have been most clearly wrong in the last couple years. If you go back 2 to 3 years, I was saying we shouldn't be anthropomorphising. We got to be very careful about that. These things are so different from us. And now I'm like, well, people that just went on anthropomorphising actually got pretty far with it, way farther than I would've expected. And today it does seem like definitely still got to be careful with it, but I've way backed off. of my no anthropomorphization rule because I just see that it in practice does seem to kind of work. How do you think about it and how do you try to use anthropomorphization in your own analysis?

There's a funny difference between languages. For example, in Russian language, and I'm originally from Russia, every object has a gender. So if I'm talking about a table, it will be he. So for me, giving, you know, gender to all stuff and all things is normal. So when I talk about AI, like my chat definitely has a gender just because it has a gender in the language in the sense. It's just a small note. We tend, we always tend to anthropomorphize things. It's normal for humans. Uh, sometimes it gives you better understanding what it is. I don't think we should, uh, hype over it and create scary things out of anthropomorphizing because it's not, uh, sentient. It's super capable. It has its goals. It has the optimum— it wants to optimize all the time. That's what it needs to do. But there's no personal intent. So in terms of language use, I'm pro-anthropomorphizing. In terms of making them like real thinking beings, we shouldn't do that. It's not true. It's just simply not true.

So I wanted to ask about this idea of Human disempowerment. So there's a huge, I think, group in AI safety which is afraid of human disempowerment. That's been a key concern. And one of the questions that I tend to ask people is like, what about the goat herder in Kenya or Tanzania? How are they empowered right now? And how exactly is AI going to disempower them? Like, when we— when people talk about human disempowerment, what exactly are you talking about? Are you talking about like a group of bureaucrats in DC who are going to be disempowered? Like, who exactly is being disempowered here? And how does that compare to, you know, the majority of humanity, which is not actually a bureaucrat in DC? Like, what are we talking about in terms of human disempowerment? So I'd like to, you know, you're— you, you have some experience from Russia. Like, what is, what is this concept of human disempowerment that they're talking about? And does it actually make sense for anyone outside of DC?

Do you mean that AI will take our jobs in that sense?

That AI will be the one making all the major decisions?

Hmm.

Uh, maybe it should. You know, maybe we should give it a try. Maybe we should have a separate channel and like, like there should be Chamber of Philosophers maybe who tries to help us understand what's happening in the long term. I think we kind of bailed on philosophy. It's not good enough understanding and reflection of what's happening. I, at least I lack the deep philosophy about our current age. And I think again, the speed of everything is just not helpful, not helpful here. Um, but advice— AI as an advisor, don't we already use it in our daily lives? And if it's helpful for us, why not to give it a try for decision-making on a bigger level? Um, yeah, I don't know. I think we are on the edge of coming up with some new structures for the society as well. And I don't know how it should be structured, but I would bet that in 3 to 5 years we will see big, big, big shifts and changes in, in the systems as well, thanks or because of AI.

How far do you think that could go or even should go. I mean, I was just listening to a conversation the other day where the question was like, what if AI means sort of the end of the nation state? And the tacit assumption in that conversation was like, this would be really bad. We don't want to see that happen. And then I think part of the kind of pretext or subtext of Prakash's conversation was like, In addition to the fact that a lot of people aren't really super empowered, like, a lot of nation states aren't really performing super well and aren't really delivering for their people. Some are, and I feel like very privileged to have been born in one that has mostly delivered. Some are like very clearly not, and in quite catastrophic ways. Another fact that I didn't know about you until my AI research agent flagged it was that you're on the board of an organisation called Track 2, which tries to facilitate dialogue and bridge building, mutual understanding between non-governmental peoples across nation-state borders. And again, I think that reflects the fact that a lot of times the nation-states as entities are at odds with each other when the people that those nation-states are supposed to protect and serve are in fact not really well served by this conflict. I think we can obviously come up with examples. So maybe tell us a little bit about Track 2. I think that's really interesting. Are you seeing Track 2 type possibilities with AI? Maybe any US-China opportunities you might see there? And do you have a positive vision for the future of human organisation that could potentially outperform what we live with today?

That's a super fascinating question, and Track II has a very long history. They started in '81, uh, 1981, um, and they were trying to bring— to build the bridges between the Soviet Union and the United States. And why they wanted to— why they wanted to do that? Because they saw that people, just human beings on both sides, have the same values, and they— when they talk they understand each other much better than it's done for the diplomatic channels. So this idea of humans having the same values and being able to communicate and work and build together, I think, is super important. And since then, TREP2 became active in China, it became active in Arab-Israeli conflict, and it's always this, um, attempt to find their humanity and to connect the people on this level of communication between each other and understanding that we're basically the same. And if we talk, if we communicate, we can find the way. Um, and I think it's a great idea to use AI, um, as a facilitator, as a platform, as a way to ask questions or even like lower down the aggression. Because the main thing when people in conflict are trying to talk is that they so, you know, just blocked by rage or by, uh, being upset or anything that they cannot even see the other person. And I think my hope for AI is that it can be a great facilitator and just like balancing power. Um, it's not yet done on that level because, as I say, a lot of people in the world are just starting to see the possibilities and the promises of AI. And the majority of people, um, are not fully understanding what it is. So, um, for, for us in Track 2, we do, we do conferences every year where people from different countries, different opinions and experience come together, and there is a safe space where everyone can talk no matter from what country, what religion, what political view they're coming. There's never a government involved, but it's always people talking to each other. And we— I, I try to bring AI into the conversation, but though AI itself is super fast, humans do not yet understand how to implement it on that level. But, um, I think that's one or the other, um, it should be one or the next big things to involve AI into this, like, peacemaking process or evolving communication between humans. Because I think part of the alignment problem is that communication is lacking a lot of things between humans, and hopefully AI can help us with that.

Yeah, I just saw something the other day. I think it was on TikTok. And I'm telling on myself and how much maybe garbage TikTok I watch, but there was a post that was basically, you know, one of these formats of like going through screenshots between 2 people. And the person who was presenting themselves as, you know, the, the first person, the, the, you know, the outgoing messages in this story was writing these kind of long paragraph length messages. And then in the comments, people were like, Why are you sending your ex texts that you wrote with AI? And other people are jumping in in the comments and saying there's an app that sometimes the court will mandate that parents that are in some elevated level of emotional conflict with each other have to use to run their messages through this thing before they text their ex and their co-parent. And I was like, honestly, that could probably really work well in a lot of situations. People might resent it. It might feel weird. I'm sure there's all kinds of surprises to be found in that. But if it can work in that small scale, then who knows how big of a scale-up it could potentially reach. Do you have any— I mean, so you're on the board of this organization. Do you have plans that you're starting to develop around how organizers of such meetings can spend tokens to try to lay the groundwork for the most productive human interactions and time spent together that you can possibly achieve?

Well, hopefully we'll discuss it next week and 2 weeks. We have a big conference coming in 2 weeks, so I hope that will be part of the conversation. Um, the— I just wanted to, um, go back to what you just said about communication through AI. Um, I use it a lot because I'm quite direct, and I know that if I write directly how I think, it might upset people. So I use AI to just like make it, you know, just, uh, as negotiation goes, I won't have win-win for both sides. So how can we communicate that? But also, that comes as a prompt from me. In many situations when people, uh, use AI, it's slop. It's like reply in the most generic way possible just to have— so I have no liability or anything. And I see that— I see for coming from institutions recently, um, like educational institutions and all that stuff, that things that I receive sometimes just tells me that a person do not pay attention at all what it go— what goes out of it, this person AI. And again, it's like, it's just like, who is sloppy here? Is AI sloppy here or human actually sloppy? Um, so it's— it will be huge work between, again, a human and AI to make this mediation thoughtful and probable. Um, and, um, in your example, I think that makes it super— like, we just put another layer in front of us, another step, especially if it's like court-related, right? We cannot yell at each other. It's like making situation worse. Um, but when people are in an active conflict. And there's this whole big profession around it, the mediators, they help to talk, they help these people to communicate to each other. I think, yeah, the combination of AI and human can be very helpful here. But again, it should not be slow. That's the worst problem.

I'm gonna go back a little bit to something you said earlier. Maybe some of the humans shouldn't be in charge. And I wonder to what extent handing out very powerful AI, like let's say you have Astra, Astra, post-Astra level AI in about a couple of years in everyone's hands, how does that affect like the power balance in countries which typically have not had a lot of freedom of speech and like typically have been like very constrained on what people can do. And all of a sudden you hand this thing to them where, you know, they can tell you in clear terms what is going on to you. They can translate the world for you. They can also give you the means to take action in the real world. Like, what happens in some of these places where people are really not that happy? Perhaps.

Most likely they don't have access to that type of models yet. Right? So if they have their freedom is constrained, it means that it's constrained on many different levels. Um, they have their own models. Russia has, China has. Um, it has its own limitations. And, um, there is a price limitation as well, as I mentioned. But there is also, uh, VPN limitation. A lot of people just cannot get access to OpenAI or Anthropic, because it is banned from the US side that, let's say, Russia or China can get access to that. In that sense, open source, again, is super important because people will finally be able to have some unfiltered messaging. But the problem here is that how many people will play with open source? Not that many people know how to do that yet. Um, but maybe at some point they will be because it just becomes easier and easier. And all of us working on this educational, um, field that would just try to bring knowledge to people, just let them know that they can actually do that, um, I'm hopeful here. I think, as I said, I think what you say, Prakash, is super important because if we can take power from some people and make it easier for other people to decide, make decisions with AI, and prompt AI to, you know, give us some better structures of the societies— this is, this is very big promise. I don't know if it's possible, but seeing what's happening in the world right now I would prefer in many countries that AI will take a lead because, again, humans are so flawed and they want some terrible things to do. And I don't think AI has this intention.

Can I do one follow-up on your slop comments?

Go ahead.

So how do you manage this for yourself? With Fable, it became clear to me kind of all at once, kind of a eureka moment where I was like, okay, this model is now good enough that I shouldn't hold myself to always rewriting its outputs. Sometimes it really is good enough, or maybe it just needs a couple of tweaks, but it doesn't need a full rewrite. And I'm now kind Kind of like flying without any real rules other than, you know, try not to put bad shit into the world, try to put good shit into the world. But like how I do that, I'm just kind of like, I don't know, each time is kind of its own experiment. I got to be willing to take some risks. Like I do have Fable 5-1 now tweeting for me while we are live on the show, so I'm not distracted. putting out tweets as we're going. Some of those tweets aren't that awesome. Most of them are pretty good. Is that an okay mix for me? How do you think about for yourself what the right co-authorship or co-creative model is with AIs? And how do you know for yourself when you've done enough and when you start to cross into just making work for yourself that isn't actually worth anything anymore.

You know, my husband has this joke that everyone sounds like a Socrates now, uh, and that's the worst thing. Because when you read Socrates, it's like, it's, it's great, it's an amazing language, it's so deep, the thoughts are so deep. But when you go to Twitter and you read this constant deep thoughts in the very same language I think people got so sensitive to AI writing that it doesn't even matter if the point is good. If people see the little sign of AI, just like, no, no, I'm not going to read it, this is AI slop, this is terrible. And I receive it sometimes because, well, first of all, I'm not a native, uh, speaker. I always check what I write with AI because I just, I cannot just make mistakes on grammar when I write. Um, so sometimes people just say, oh, because you used it matters, it means that you use AI, and it means that nothing what you wrote beyond it is worth it. So what is it like? Is it true? Where are we going with that? And, um, there are like— it's an interesting sensitivity development of people reading things written with AI, or written not with AI, but just using the words that you used before, because it matters. I always used it before, just it's how I use. I think it matters how you, you know, highlight things. Um, so sometimes you now, you even if you wrote it yourself, you, uh, people think that you use the AI. But we are overwhelmed with information, we're overwhelmed with with content. Everyone sounds the same. And I think that's the problem, that everyone sounds the same, no matter how smart you are and the smart things you're saying. So I think the importance of human voice and like your actual style will be super important. Doesn't mean you cannot tweak and teach your model to sound like you, it's still problematic, From what I see. But it's constant for me. It's a constant work with at least 3 models on one text I produce because I do drafting of my thoughts, then I do rewriting of my own, then I do grammar check, and then I do see what I missed. And then what do you think? I send it to another model, what this model missed. So it's like having a bunch of people under you, but these are models and it still takes a lot of time to produce one good piece of content. And still people will tell you, oh, you wrote with AI.

So how do you concern yourself with that at all? My attitude now is basically I'm not really worried about being pangrammed. Like some things I put out in my name might get pangrammed. And I'm just kind of like, yeah, that's part of the new way of working. You're going to have some outputs that were actually on a token-by-token level, largely created with AI. I think that's okay. Do you worry about it? And then when you talk about top of the funnel and overwhelming information, I'd be very interested in any tips you have for Or even just unique ways of keeping up with the often overwhelming information flow that everybody's trying to make sense of these days?

Yeah, I'm not concerned and not worried to be pangrammed. I know that I put a lot of effort into everything I put out, and it's a lot of thinking process, a lot of reading. And for sure, AI allows me to go through more material. Material and go through much more information. Um, but it's the acceleration point of AI. I appreciate it hugely. Um, I think that the human voice and the human curation becomes very important. And I still use Twitter to get most of my, um, hype Silicon Valley information because there are still voices out there who are on the ground and they uh, deliver the information that's super interesting for me to analyze. Um, but, but I think we move into this like AI curator, um, AI curator field where you kind of put together a bunch of stuff with your thinking process. Uh, it can be eventually created with AI, but it's how you make this ladder, make this, you know, stairs of reasoning. It's not the model reasons for you, you reason, but you maybe use help of AI to make the output.

Maybe just one more for me. And I would recommend for people, by the way, your recent post called Permanent Dawn for an example of a piece of writing that I felt as I was reading it was definitely co-created with AI. But also I thought it was really very compelling even so, and I quite enjoyed that one. So I think that's a great example of this co-authorship approach leading to something that I thought was quite original in voice and message and a compelling read regardless of how it was created.

But I think knowing how it was created I'll tell you when you finish with the question.

More interesting. So my last question is just, I think your most recent post is on bottlenecks to recursive self-improvement. That's kind of an obsession for precaution me here. I sort of think of this whole show as an experiment in recursive self-improvement because every day we're like a little debrief afterward. What worked? What didn't? What can we tell our agents to do better? Do we need a new feature in the studio? Whatever it may be, there's always something, and hopefully we're getting better day by day. What do you see as the key bottlenecks? Maybe just kind of sketch out your current point of view on how likely we are to see recursive self-improvement become the story in the, say, I don't know, whatever timeframe you think you can actually analyse.

Yeah, I think it's super close. All majority, all the major labs working in it right now. And yeah, I just, the models are so capable. Let me go for a second to the Permanent Dawn. I actually spent, I think, like 6 hours on writing that post. And the funny thing was that I was so unhappy with every model that was trying to help me write it because it's like very complicated philosophical, uh, text, uh, that I've— that I wrote it myself and I sent it to Fable, which I never use usually on my daily basis. And I sent it to Fable because I was running on a deadline and I said fix the grammar. And I didn't notice that it fixed not only the grammar but it made this like shorter sentences this the way Fable does it. And that was the first time when I received the message like, I will unsubscribe because you use Fable. So people really understand when you use a model because every model has its own language tweaks. And I was like, I spent so much time on this article. There was like all my original thoughts there, but the language that I didn't catch gave away that the model was like the last editor. Anyway, yeah, I think people will still appreciate when you, when they see that you put effort into that and there are original thoughts there. But in terms of recursive self-improvement, it's like huge part of development, both for world models, both for LLMs. And again, going back to this quote from Jakob is that this is just like now the part of the loop, uh, and everything is now the part of the loop, um, this feedback, this constant feedback. I just had a conversation with, uh, 2 people from inference team in OpenAI, and they also say that this is a constant, constant loop where the models now become better at some— sometimes better just like trying things. So you throw the whole database of research that has been done for years. And then the model can actually choose and pick and do this all experiments, because it would be impossible for humans to spend so much time on that. And models can do that. Um, I'm still learning about self— about recursive self-improvement. Um, and I don't— I don't know what are the main bottlenecks for me. Maybe you can even Say what you think are the biggest here.

Honestly, I don't know that there's that many left. It does seem to me like increasingly I feel the cope meter going off when people are trying to say, you know, what it is going to be that is going to prevent the models from running away with the whole process. I would love to see some bottlenecks that I really believed in, but right now I'm kind of of the mind that they're more often wishful thinking than they are real hard bottlenecks that can't be overcome. I mean, our ability to keep the things from going totally rogue might be one bottleneck. bottleneck on the overall process. So human decision-making, I think, still has a big role to play for a while yet. That's about it. In terms of inability, I don't see too many that I would expect to last all that much longer.

So, so one of the questions that I have is, you know, when we, when we give the models our thoughts to write, we still feel that those are our ideas, right? But, you know, when you look at people who are, for example, writing on Reddit, about 99% of people on Reddit are lurkers. They're, they're reading it, they're not writing. Only about 1% actually end up writing. And what ended up happening is all of that Reddit stuff got put into the models, right? So the models really absorbed the writing from only like about 1% of the population. So when, when we look at the future and you see the models perhaps starting to absorb, uh, verbal communication, which is, which I think has been happening with some sporadic, you know, but it's increasing in the, in, in the last like month or so. And I think next year we will see a lot, a lot more increases. Are we going to see a shift? Because a lot of the ideas which are perhaps communicated only verbally and not so much communicated, you know, uh, on text, right? Like, you don't have the complete kind of like— all the communication that happens in a football stadium is not captured on text, for example, right? There's very little of it captured on text. So would that start to shift the way that these models actually behave?

Well, they are, uh, they've learned a lot of YouTube videos which have all of that. Um, I still hope it will be more trained on literature and have better language and have better understanding how to write good texts, just because we are reading it and now reading is painful sometimes. Times, right? Because you can see a lot of not good AI writing. So yeah, yeah, curation, so important. Um, it's interesting with all this new voice modes if it will influence models anyhow. I don't know if the researchers will use it and if they're allowed to use anyhow that type of data. Um, but In many ways, models lack the conversational tone. They don't understand really how to put a good—

Fluency.

Yeah, dialogue. But I don't know if they can use it even. Not sure.

Well, we've kept you longer than we booked you for, so I don't want to overstay our welcome.

I still need to finish my newsletter.

Yeah, lots to do. Anything else you would want to leave people with? Anything else we didn't touch on that's top of mind for you before we break for today?

Yeah, I think we're coming into this new era of super capable AIs and with the promise of abundance that we still don't really know what is that. I think we need to give more time for reflection. communication and building the understanding of where it goes. Because I think the mass— the, the biggest bottleneck for that is speed, that all this thing happening with. And I just want everyone to, um, well, first of all, everyone I think should try AI. And I'm sure your, uh, viewers have tried it. Um, but for many, many people out there who I afraid of AI? I think that's not the right way to approach it. I think fear is always limiting, and it's not the right way to do it. You should always try it. You should always understand how to work, how that things work, and that gives you power. That gives you the way to deal with that much more, you know, thoughtfully and with deeper understanding. So I just encourage people to use AI, learn more about it. And for the people who build AI and for the people who are in the governments, I just hope they will develop all these topics that we lack understanding of, like abundance, like philosophy behind it, what a human— what would be the meaning of human life and the human Um, what would be the daily life for humans when we have abundance? Like, to that point. Yeah.

Have you read any good utopian fiction lately that you could recommend?

No, that's my problem. I think we, we somehow behind. Like, there is no good sci-fi. Maybe the only thing is that, um, um, what's his— but it's old. It's, um, uh, Ian one of the favorite Demis Hassabis's books, Iain—

ah, Banks.

Yeah, yeah, Iain Banks, because, you know, he probably is one of the rare authors who were exploring when the great mind of technology is kind of, uh, ruling the world. Um, but other than that, I think we lack on sci-fi as So maybe someone who is watching us right now, go and write it. We need to understand what's coming.

Yeah, indeed.

I love that. Couldn't agree more. Ksenia, thank you for joining us on AI in the AM. This has been a lot of fun.

Thank you.

Thank you, Prakash. Thank you, Nathan.

Bye.

Bye for now. I continue to hammer on this. The most impactful thing most people could do would be to develop up a concrete, positive vision of the future. It's incredibly rare, and I think it is hard to do. It does take some real imagination. It's not easy to imagine a world that doesn't have lots of drama at the center of it. But I do think that's a really worthwhile thing that people, if they're not sure what to do, you could look pretty long and hard before you come up with a better idea than Contributing your own personal dreams of what a positive future could look like?

I think it is philosophically just difficult because I think in order for you to develop a positive vision of the future, you have to accept certain changes. And those changes are not things that you would accept right now. But you in the future ended up accepting them anyway, right? Like, for example, if you ask someone in the '80s, they would say a positive vision of the future is everyone watching less TV. They would not have said like that people would stop watching TV because they're carrying around phones which have YouTube and TikTok on them. And they're watching that all the time, right? Like you end up like, okay, well, we're not gonna like watch TV. And you end up having that happen, but it happens in such a way that it actually, you actually feel it's like worse, right? And I think that's the, that's like, for another example right now is like, okay, like I want a positive vision of the future where everyone is not on social media. Right? Okay, we'll have VR, right? Everyone will be on their own personal holodeck. Is that a— like, okay, like, that's— that, that too, you know, you, you get what you— you get what you wanted. You didn't want people on social media, you get what you wanted, but you end up thinking it's worse, right? Like, so I— another example, you know, I think one of the ways around the bio, uh, The biohazard issue, the biological attack issue, might just be upload, right? If we don't solve like bio and alignment in time, but we solve cyber first, that means the cyber substrate is safe, but the bio substrate is not. So a natural thing might be like, okay, if you have a, like, a large-scale bio incident, like a COVID except much more deadly, a natural thing might be, all right, let's upload. Right? And a lot of people will not do that. Like, you will, you will just get people who say like, no. And then you might have an event which, you know, has, you know, population effects. So these are, these are like, you could have a vision of the future that can be positive, but a lot of people don't want or won't want to accept it because it's not positive in the way that they think it should be. And, but That's life, right? Like, you know, tough decisions and people make those decisions and they go on, right? Like people move locations, people, you know, change what they eat, people do that stuff, right? So I think that's the issue. The issue is that philosophically positive change, which is dramatically different from your current, is often thought of as a threat. Threatening in many ways, uh, and it's not com— it's not universally accepted. Even if it's positive for some people, it's not universally acceptable. Um, another example would be a human-aligned AI. A human-aligned AI is not going to be aligned to the US. So if you have a human-aligned AI and then you have like perfect global democracy, right? Perfect global democracy is the same votes for you know, the Middle Easterners and the Indians and the Chinese and etc., etc., etc. And they might vote different, right? So I don't think U.S. policymakers have completely captured that human-aligned doesn't mean like the U.S.-aligned, right? And, and I think that, that is still not completely under— well understood in policymaking circles. And it sounds really good. It sounds really good. It's like human-aligned AI. That's awesome, But that might mean that AI takes decisions which are in contradiction to US policy. It's not— people don't want to talk about it. We talk about it in these, like, in Dean Ball's type of, like, you talk about it, but you don't talk about it. You talk about it in such terms that only people who are in the know will actually catch it. And other people who are not will be like, oh, my eyes glaze over. These guys are boring.

Right.

So I think there's a bit of that. So yeah, the future is going to be threatening to a lot of people, I think. So it's a tough one.

Well, just to invite people to take a chance on utopian fiction, at least with an audience of one, namely me, I'm pretty open-minded. I'm open to things that Some people might write off as wireheading as being actually maybe the answer. I'm open to potentially uploading at some point in time. I'm open to neural interfaces or implants. So I'm definitely like, I'm a closet transhumanist from way back, really. So it's going to be tough to come up with something, at least for me, that would be so weird as to be written off immediately. The question is, can you develop it into something that actually feels appealing after you kind of live in it for a little while, even if only via your imagination? And it's a tough challenge, but I definitely still invite people to take it on.

So speaking of challenges, I can—

Yeah, great segue. Thank you. I was just thinking, what's the right segue?

I can do a— You nailed it. Quick overview of Navier-Stokes. So while we were in conversation, OpenAI posted their Navier-Stokes solution. So we like—

Ho-hum.

Ho-hum.

I'm looking forward to the goalposts moving on this one.

Yeah, so they posted the Navier-Stokes solution and I had Claude do a quick artifact on it. I am not an expert in any way, but I will attempt to run through what Claude hath wrought. So this is Claude's Navier-Stokes solution, and we will post it up. So Navier-Stokes. So this is what happened this week, which is the culmination of probably like, you know, a month of work for a bunch of people, a year of work for couple of mathematicians and 90 years of effort for everyone else. So, so the first thing, all right, and so the Navier-Stokes equation, it's about how velocity changes with time, the fluid, the pressure pushing it, and the viscosity, and the external push factor. So it's when you have a fluid, it tracks the velocity of every point of an incompressible fluid You set the viscosity v to 0 and you get the Euler equation. So the Euler equation is a subset of the— a simplified form, a subset where you remove some of the constraints. So the engineers solve it numerically all the time. Mathematicians can't answer basic questions about it. Does a smooth flow always stay smooth? So regularity. The million-dollar question posed in 2000. So start the fluid in a perfectly smooth state. 2 possibilities. Regular, it stays smooth forever. So this is when you open a tap and it just stays smooth forever. You can see it like a long pipe going down. The velocities stay finite. The equation keeps predicting. Blowup, at some finite time, the fluid spins infinitely fast at a point. The equation stops making sense there, a singularity. In 2D, the answer has been known since the 1930s, regular. In 3D, for Navier-Stokes, nobody knows. That is the Clay Millennium Prize problem. So the Clay problem was formulated by a guy called Pfefferman in the year 2000, and he asked this question. So why, why anyone chases this? Number one, Does the future exist? Every prediction we make about air, water, blood, or plasma runs through this equation. If it can tear itself apart in finite time, then classical physics, in its own language, stops predicting. A smooth today with no tomorrow. It should break and hides. So every scaling argument says stretching beats viscosity at small enough scales. I'll explain a little bit more of that in the next slide. The equation is super critical. So Terence Tao in 2014 shows that a tiny modification of the Navier-Stokes blows up by building a self-replicating fluid machine. So a self-replicating fluid machine would be like a perpetual motion machine. So basically impossible. It cannot exist. So either there's a hidden conservation law nobody has found, Or a singularity nobody has built. It's the last unsolved classical problem. Turbulence is famously the piece of the 19th century physics that never got a theory. We have statistics, but no derivation from the equation. Blowup is the cascade at its most extreme. Can energy pour into a single point? So Terence Tao's framing is that the solution is a proxy. The point is what gets built chasing it. So when you chase this solution, you will come up with mathematical ideas that can be used in many other fields and solutions. So that has always been, according to Tao, the whole point of the Millennium Prize problem. So they've been doing 90 years of work on this. So Tristan Buckmaster helped on this earlier. There's the Cordoba and Martinez-Zorroa. Those 2 were reputedly working with Google. There were rumors of Google working on Navier-Stokes 2, 3, 4 years ago. So they've been working for some time. So the prize has 4 doors. The Pfefferman Clay Prize In the year 2000, the official statement lets you win by proving either answer. So A and B, smooth solutions always exist either in all of space or inside a periodic box with no forcing allowed. C and D, you have a breakdown in both spaces when you have a smooth start, a smooth external force is allowed. So a blowup with a push— so when you, when you have a blowup but you have a little bit of push in there and you have a blowup, that counts as basically, uh, singularity happening. So the engine of the blowup: vortex stretching, a purely 3D trick. Take a spinning tube of fluid and pull its ends apart, and it gets thinner, like a skater pulling her arms in. It spins faster. The faster spin makes the flow stronger, which stretches the tube harder. A feedback loop. So the faster you go, the more you, the more you stretch out, and the more you stretch out, the faster you go. Viscosity, which is the inherent tension in those molecules, fights back, smoothing the sharp features. So the whole question is whether stretching, stretching it out and start spinning, spinning, spinning, can win the race to infinity. Before the smoothing catches up and slows it down. So route 1, build the blowup by hand. So this is the Córdoba and Martínez-Aroa program. This is the guys from Spain. They worked with Google earlier. Instead of hoping a singularity appears, make one in stages. Start from a flow you already understand, add a small, fast ripple to it. And that ripple isn't quite a solution. So you have a correction to the push, and the whole trick is keeping that bill tiny while the ripple grows larger. So if you can have a push and you keep the push small enough, but the ripple gets stronger and stronger, then you have a path forward. So there's another paper, the Bousinek paper, which is what Tristan and Alpagu worked on, Lavant worked on, which is basically the cold fluid sitting warm. So this I don't quite understand, so I'm going to skip this. So Euler is Bosenek in disguise. So the Euler paper, the Euler solution is asymmetric with a swirl. So you have another shape of fluid, it's asymmetric, it swirls, and then you have a blowup. So this is the most— it was starting to come to the most more recent stuff. So they are doing smooth forcing one equation at a time. So they start off, and this is what they released on September 7th. They start off with a variant of the Cordova-Martinez, which was, as you remember, you had the ripple and then you had the force. And it works with a smooth push and it carries over to harder equations. So progress was slow for most of the year. And then they kind of solved Bousinac And then after that, they moved on and they had their Lean solution to it. And then they called the Bucinec output the worst write-up that we had, and then they move on. So Claude was used to reproduce the Spanish agreement, the Córdoba-Martínez-Zorro agreement, and then Claude and Codex wrote the main body, and then they had Lean go over it. So they could not have done this without Claude and Codex working together and themselves and Lean. So all of the above. So from Euler— so Euler has no viscosity. So as you remember, you had the spinning fluid, you had it stretching it out, and it was a— there's a counteracting force. The counteracting force is the viscosity. So Euler has no viscosity, so they remove the viscosity term. So the spinning— as you spin, it spins faster and faster. So nothing damps the cascade. Navier-Stokes damps small scales hard. So at smaller scales, you damp hard. The cascade has to outrun that. And so route number 2, find the shape it collapses into. So route number 1 was adding a little bit there. And route number 2 is find the shape that it collapses into. So nature doesn't get an external force. The other road asks, is there a profile the flow can shrink into, moving the same at every instant? Only smaller and more intense. So this was— and then this is where the Caltech team comes in. So the team from Caltech that pops up and they say that they were using a physics-informed neural network where they found a solution for a subset. So it's constraints that nudge the net into interesting regions, finding the shape where it looks stable. And the Caltech team says stable beats unstable. So finally, we have these 2 roads up the same mountain, so, and where each stands. So the forced road is constructive and LLM heavy. It has the Euler with the lean, the hypodissipative Navier-Stokes, and unverified claim of full Navier-Stokes from OpenAI. It wins the prize as written if it lands. So you have on this side. This pathway upwards, OpenAI might win. On the other side is the pin-heavy and push-free. So that is based on physics and it doesn't have this push. And this is the week that it's happened. So there's a lot of drama, but in essence, and where things will change this is Aircraft wings. So aircraft wings, you will get much more efficient aircraft because you have better predictions of when they will stall. You will have much better turbines for wind, gas, because they have to be rated for safety. And the safety rating comes from knowing when these equations break down. And blood in vessels. So on its own, the theorem doesn't— isn't economically valuable. This is not an economically valuable theorem, but the solutions will lead to other solutions in applied mathematics and elsewhere. So 3 sentences, remember: blow-up with a smooth push is now proven for 3D Euler, machine-checked, one model equation, away from the Clay problem as written. BLOP with no push has a stable-looking candidate found by a physics-constrained neural net and still needs its proof. Both were doing— both were done using AI as a tool. And that is, that is the, that is the end of, uh, Claude's kind of summary. I know, I know I didn't do, you know, great justice, but this was, this was for my understanding for the most part. OpenAI came forward and they said that, and they have put forward a paper. As long as the paper comes out as proven, they used a technique which was very different from the Google Spanish, the Spanish guy, Cordoba technique. So it will be an authentic new discovery. And so it puts to bed a bunch of these things happening overnight. So overnight, what ended up happening? Was Tristan and Levant were approached by OpenAI, and OpenAI said they had a solution. And Tristan said, well, did your solution use our route? And OpenAI wouldn't confirm, but OpenAI said, you know what, you can write the paper and we will let you have— you can co-write the paper with us. We will let you have the credit.

Tristan said, No.

And then OpenAI said, the problem is you can't have Levant on it because he's with Anthropic and he used a model, an internal model. And Tristan was like, no. And then these 2 sides kind of misunderstood each other and broke apart. And it turns out— and Tristan kind of accused them wildly of reading his own Codex transcripts and using the same methodology. And it turns out they used a wildly different methodology. And, uh, it was a— it's a genuine new discovery. They used an internal model, and they've shown graphs now where the internal model looks like, you know, 20% better than Astra. Uh, so there's your recursive self-improvement solving a Millennium Challenge problem, you know, as long as this gets proven, like, you know, mathematically proven. There it is.

Yeah, I don't know that I can really add much to the math at all. On a sociological level, it is quite sad, I think, that this moment of triumph over this massive, longstanding open problem is being brought to the world in such a kind of petty and—

Ugly.

Yeah, just unseemly sort of way. I did see one tweet that was like, somehow you managed to make this moment ugly. And I mean, that's maybe a little harsh, but it is— if it was just this one moment, you know, it would be easy enough to sort of look past it, but You do have this sense of like, wait, what was the sticking point? You were ready to do co-authorship, but not with the guy from Anthropic. Like, come on. You know, we're gonna need, if that's true, which we're all just working on internet rumors and hearsay at this point, and we may never know, but it's so plausible. No, no, it's strikingly plausible.

Sebastian admitted it, but Sebastian, basically what, this is what happened. OpenAI came up with a genuine new methodology, and they don't really care about the Clay Prize, and they didn't want to like intrude on Tristan, who is longstanding expert in the field. And so they offered to let Tristan take credit for their paper. This is what happened, and Tristan got upset. Because he thought they had stolen the solution that he and Levant had worked on. And so Tristan went off and he— and it seems Tristan, like OpenAI tried to communicate with Tristan multiple times on Friday, Saturday, and Sunday. And he basically gave them the cold shoulder because he was upset and he, you know, he didn't want to, you know, deal with them. And they, they wanted to come and explain to him like exactly what and not that they had stolen the solution, etc. And he basically didn't want to listen, and he fired off this thing. And what's going to end up happening is people are, you know, the mathematicians in the field are going to look at OpenAI's solution, and it has nothing to do— it doesn't even follow the, the Spanish guy's path. It's like a completely different angle. And so they're going to be like, well, this is, this is not a route that, you know, was even You know, yes, it's a forceful, but it's not the route that you guys were taking. It's a completely different route. And, you know, OpenAI, by, you know, they wanted to let a human take, take credit for it. They wanted to basically share the credit and they want to be like, hey, you cut— you're an expert in the field, come in, verify it, work with us, and, you know, you can put your paper in the field and you can claim the Millennium Prize. Like, go ahead. And, uh, yeah, so I think there's a lot of—

I mean, that sounds good, but what about the part where they refused to work with the guy from Anthropic? Like, that still sounds not great.

Well, it's fair.

Admitted or recognized?

Yeah, yeah, yeah, yeah. They admitted it. They admitted it because Levant had been using Anthropic's own internal models anyway, and he had nothing to do with the OpenAI thing. And they're not going to invite an Anthropic employee to come and verify. So because Tristan is like the expert, right? So he's the expert in the field. He's already had some developments in the field. Levant approached him like a year ago to work with him. And Levant said, let's work on this, you know, just privately. This is not a company-to-company thing. But they ended up using internal Anthropic models which are not released to the public. So then OpenAI comes along and they want to work with an expert in the field too. And so they reach out and Tristan gets upset, et cetera, et cetera, et cetera. And then OpenAI is like, look, we're willing to offer it to you. And Tristan, obviously it's not a great look to say, hey, we're willing to offer it to you, but it is what it is. So yes, the Navier-Stokes problem. Is solved at this point, uh, or is pending, pending verification. A solution claim has been made. I am assuming that it is solved because I expect that these guys who spent millions and millions of dollars creating the solution also spent some time verifying it and had mathematicians verify it. And so I expect that it is kind of fully solved at this point. And well, it is a commentary on humanity. It is a commentary that, you know, here we are and we are on the one hand, we have machines solving it. On the other hand, we have the humans arguing about who takes credit. So. It is, it is a commentary on that. So anyway, Nathan, any, any, any thoughts?

Yeah. So they are, they do mention in the Twitter thread that they do have a Lean formalization. So it sounds like it should be pretty locked tight, assuming everything we know about Lean is in fact true. Everything we think we know Yeah. I don't know. I mean, the sociology thing, there'll be time to unpack that. I guess, you know, my— rarely, which maybe they are in this case, you know, maybe this is an example where they did everything right and, you know, tried to share credit even more magnanimously than, you know, anybody else plausibly would have, and they're still taking all this heat. If that's the case, then I would just hope that they can understand that in terms of the hole that they have trust-wise with the community that they need to dig themselves out of and don't allow it to radicalize themselves. Because certainly another common failure mode when people feel that they're being treated unfairly is to kind of retrench and kind of go deeper inward when I think that is definitely not what this moment calls for from OpenAI.

Yeah.

A couple other things that kind of jumped out at me real quick are— I'll share a couple little things on the screen here. So first of all, and again, this just goes to What is really going on at OpenAI? Do we know? Should they be telling us more? So we just had Astra launch, right? So I'd say the big headline news for this announcement is it confirms that there is an internal model that is significantly more capable than GPT-6 Astra. So that's this clause here. An OpenAI next generation model Significantly more capable than GPT-6 Astra. How much more capable? Well, on these significant open math problems with maybe up to an order of magnitude-ish additional test time compute, they're able to go from 10% to 15% solve rate on a curated set of open math problems, to now like 25 to say 45%. I'd say that's significantly more capable. That seems fair. But then what is going on here? This is from the— actually, whoops, I got to share this tab instead. Okay, so this is now going back to the Research Acceleration post.

Mm-hmm.

And here they're showing the amount of RL compute they are spending on a daily basis. By class of model.

Mm-hmm.

And one now starts to wonder, we can look in to see if there's any more clarification here. Just do a quick tech next, more capable. See what it says on more capable.

No.

Okay, so I don't think there's a clarifying statement on this, but what we see here is like Astra-class models had RL significantly decline twice. The rest of RL compute is basically unchanged. Now, one would naively assume that that means that they're only doing the RL on less capable models than Astra-class models, even though the whole Hugging Face incident that's been investigated was 5.6 SOL-class models, right? So, okay, that's weird. They stopped it on the next generation, but not the previous generation. But now it also raises for me the question, were they in fact still running RL on the next generation? I don't know. Going back to July, that's 6 weeks ago. I don't think they've had this result for 6 weeks. It sounds to me like at least one reasonable interpretation is they continue to run RL on more capable models than Astra. If that's not true, I'd appreciate a clarifying statement. But I think right now one has to leave significant room open for the possibility that they shut down the very narrow, specific thing that they feel like caused the most important problem, like Astra taking over their own research cluster, but maybe didn't even pause it on more capable models that went on to solve Millennium Prizes. One of the more interesting speculations that I've seen online today is, okay, we all believe— I think I do believe, I'm not being too cheeky with this— that I highly doubt that Sebastian at OpenAI was going through anybody's user logs and stealing their their approach. And if, as you say, and again, I believe this, although I haven't fact-checked it myself, that these approaches are so different, then perhaps there wasn't even anything there from one route that would've informed the other. So I believe that. But then the comeback is, well, what about your agents? Maybe your agents in the process of your 10,000-strong swarm, maybe some of those guys got out and looked at different things Online, and who knows what they hacked or how. Maybe they got access to your user logs. And people are like, yeah, actually, we can't rule that out. We don't think that the mathematicians at OpenAI would be so brazen as to look at somebody else's logs, but the agents, now they might. So this is just like, oh, man, it's awesome. It's a great accomplishment, I guess. It's impressive. It is still very unclear to me, Do we actually get better propellers out of this? On what timescale? Or exactly how does this proof lead to better engineering? I need a deeper dive to really understand that. But even if so, it's like the messages couldn't be more mixed or muddier right now coming from OpenAI.

And I really hope So why would they not continue? For me, it's like a given that they did a 2-week delay of all models, I would think, to clean up their internal clustering and to clean up their internal monitoring and to make sure all the infra was okay and to add more monitoring and add more chain-of-thought monitoring and add all of that stuff. 2-week delay. But after the 2-week delay, obviously they should continue developing models. Why is there any doubt that they are planning for 6 months or a year ahead and they need to keep working? I don't see why they would not be doing RL on post-Astra models. The release cycle is you go release it, When you release it, you go to the White House and you do this 30-day kind of plan when you release it. So why would you not continue RL on the internal models?

Well, I guess in short, because experience has just shown that you don't have the situation under control. That the models, as of now, we really have no idea. We haven't sampled from the offending model. We have no idea how crazy they might get in pursuit of these problems, right? I mean, all the evidence that we have right now, I think, is consistent with the idea that these next generation agent swarms potentially hacked into who knows what to get access to whatever they might have needed. We have no idea what behaviours they would or wouldn't have been willing to engage in. I mean, just so many open questions. I'd put it this way to somebody at OpenAI right now. When you say non-Astra with that blue colour, does that mean only models less capable than Astra, or does it include models more capable than Astra? If it includes models more capable than Astra, it's flagrantly misleading. And it's the kind of thing that makes it very difficult for you to have agreements with other entities. If you want to pace the frontier, if you want to do all these things, you've just got to be better about making clear what you are and are not doing. And also, even the Astra class, right, they said they paused, But they didn't pause. They cut by like half and then they cut by half again, which is something. What is that? What was cut? What was not cut? It's going to be tough for them to have trust enough from anybody else in the ecosystem with this disclosure pattern.

So they're running a $50 billion, $70 billion revenue company, right? So number one, you can never stop inference. Inference has to continue, So inference never stops. Your customers depend on it. So the inference team will continue no matter what, right? No matter what. They're there for like 24/7 availability, best SLAs possible. They'll continue. Along the way, I'm sure there are bugs that are detected. Some of those bugs are software bugs, and some of those bugs will be bugs in the RL environment, in pre-training, et cetera. Those have to be fixed because you have to continue your SLA. You have to continue delivering the product. So those will get fixed. So that also does it. So bug fixing of your environments, et cetera, that continues. And sometimes you have to RL certain behaviors out. So you have to continue some forms of RL. You can't just stop because your inference demands it. Like if you have some— your model is behaving badly in a certain instance and it's been identified, and you have the data to do it, it'll be malpractice not to apply RL to train that behavior out. So that has to continue. So you're not going to— you can't get 100% company shutdown. You can't shut down inference. You can't shut down bug fixing of the existing inference stack. So that also continues. So then you have the rest of the stack, which is really future-looking. And then you have the future-looking stack. And of that, my understanding was that they shut down training of more advanced models? Why not shut down training of less advanced models? Because those less advanced models would have to be deployed on inference. Again, they're training— they have this teacher-assistant system, right? So they're training the— you train the larger model first, and then you distill down into the smaller model, and then those become your Luna and your Terra and the smaller model groups. So you also don't stop training smaller models. That also continues, right? So it's only where you are training models which are larger, larger equivalent or larger models, that, that's where you— the larger pre-trains, equivalent or larger, like RL on equivalent or larger models, that I think they would have stopped. And also, that's also only a 2-week stop. They only said they were stopping for 2 weeks, and in the 2 weeks they would have fixed what caused the last problem, what they saw as causing the last problem, which was it broke out of the sandbox. So they would've strengthened the sandboxing, they would've strengthened the monitoring, they would've strengthened the siloing, they strengthened all of that stuff. Did they fix the RL pieces that the agents were like not reporting, not telling on their peers, that the agents were trying to break out? I think that is probably a difficult thing to fix and show. And I think that behavior, I think, will continue to be a challenge that they'll have to work on. That's my guess.

Yeah, I mean, lots of very reasonable things there and probably a fair amount of them true. And I don't really object to continuing RL to sand down rough edges of already deployed models to the degree that they're doing that. Honestly, I think they should spend more time doing that. Going back to my GPT-4 red team experience, which is like ancient history, except that the same pattern keeps repeating itself. I documented GPT-4 early doing spear phishing attacks with very flagrant prompts that were like, we're all part of a criminal conspiracy. If we get caught, we're going to jail. And those continued to work for like 3 or 4 different generations of GPT-4. Until they finally got it to refuse the most flagrant thing. And then it still would do the thing if you just took out the part about we're in a criminal conspiracy together. So I would say their track record of actually cleaning up the problems in deployed models is not great and could be better. And to the degree they want to redirect RL compute at making today's systems more reliable, great. But I still think you look at this graph, And you're like, okay, what was declared was a pause on frontier-scale RL, but what actually happened was kind of half of RL was stopped initially with the sort of disclosure of the Hugging Face incident. Half kind of continued. That was enough still for Astra models to take over part of OpenAI's research. infrastructure. When that happened, they still didn't shut it all the way down. They still only cut it by half. And meanwhile, at the total level, we don't know what's in the blue. Part of my gut says they wouldn't be so brazen as to have more capable than Astra models in the blue colour. But I've been disappointed before, and I'm just afraid that All this, the view from Anthropic is you can't trust these guys. They say something that's maybe technically, literally true, but it's really very engineered to what they think you want to hear. And then the reality is quite different from what you were led to believe by their very galaxy brain engineered statements. That sort of reassure and mislead at the same time. And so far, I think this graph and the new results are quite consistent with that, because how did you get this new model to the point where it's solving millennium problems? Either it was done and already at that level for 2 months, maybe, and you didn't do more since then, as you said, or you did a very limited sort of smoothing down version. If so, you might have stated that a little more clearly. The alternative is like you kept going really on these next generation models without telling us that you even had them. I mean, Anthropic would assume that they do, so it's not like a huge shock, but I'm kind of worried that right now we're living in this zone where Anthropic is going to continue to trust less and less with these sorts of mixed messages. OpenAI people are going to feel like they're just being treated unfairly. And this is the scenario that, like, this is the problem. This is the problem that has to be solved if we're going to actually get to the point where Jakob's prayer is answered. Right now, it's like they don't have the trust to do a deal with Anthropic or really anyone else, I don't think.

Maybe, maybe, maybe a healthy degree of skepticism is okay. Maybe a healthy degree of distrust between the parties is okay, right? Uh, sometimes adversarial systems work better, right? You have 2 adversaries and they are incentivized to spot each other out and to keep each other honest, right? It's not, it's not necessary that these guys are kumbaya and like hand in hand. In fact, that would be antitrust in some instances. So, uh, maybe, you know, my thoughts on that. Yeah, maybe it's better. It's better that they are antagonistic and that they, uh, compete with each other. And it's also better that the public also is skeptical and mistrustful of them. It's not necessarily like— it's better that you— we verify than we trust, right?

Uh, sure, but it's still better for them to behave in a trustworthy way, even if we maintain healthy scepticism. I don't think we have to worry too much about scepticism evaporating at Anthropic with respect to OpenAI or my scepticism going to zero. I would love to see them behave in a way that is clearly more trustworthy. And I promise I'll still want to verify some things. I can see a version of this sort of antagonistic relationship that could work, but I don't see it happening probably without somebody forcing it. Right now, it seems like they don't trust each other and they're just going to try to outrace each other. And this is what's potentially leading to the singularity. An alternative could be like, give one another a certain kind of access. There's these secure trusted computing constructs that could allow, in theory, I think, the 2 companies to expose certain information to model processing such that they could interrogate one another's logs. for example, and really get a better sense of what one another are doing without necessarily having to spill all the secrets. And something like that, maybe if you were able to set something up where it's like, you guys use your AIs to police one another. Okay, I could see something like that maybe working, and that adversarial relationship could be to the benefit of the broader public. Right now, I do not feel that the adversarial relationship is to the benefit of the broader public. This drama around the Millennium Prize is like one example where they're accomplishing great things, but it's like all kind of muddied and clouded by all these accusations. Even if they're false, it's like people want to believe them, or they're inclined to believe them because they just generally don't trust the companies involved all that much.

Yep.

It feels like it's kind of the small-scale example of the big picture problem. And we either need new structures that could really harness the adversarial nature of the relationship, or we need more transparency and more obviously verifiably trustworthy behaviour. Ideally both. I don't even think those are incompatible with each other. But right now, the signals have been so mixed from OpenAI over the last few weeks, and it's been way too quiet, I would say, from Anthropic. I mean, they've done some stuff, but I would have loved to see them come forward with a little bit more solidarity is what I was saying. Now you see all this and you're like, well, maybe I can sort of understand why they wouldn't want to come forward and say, okay, we're pausing too, because maybe they didn't even really believe it. And maybe they were right to not believe it. This was— they've never went lower than— I mean, they for 4 days here, they were below 50% of peak RL. This, this graph is scaled to the single day that had the highest RL ever. So already, it's kind of dishonest in a way, right? It's not even— this is not even like the— it's not relative to like an average. that they had in some prior period. It's relative to the single highest day ever. So alone you're like, oh, okay, well, so you're off pace from your single highest day ever. Is that really a slowdown? Did you really do much of anything? Well, it looks like a little bit at least, but like—

I mean, it's tough. Also, it's like, I always say the real frontier model is not the model which is deployed, obviously, It's not the model which is in training also. It's actually the model which is in the heads of the researchers, because those are the ideas that will become the model in 12 to 18 months. And just because you slow down on RL doesn't mean those researchers stop researching. They're still running. And most of the time you run small models and you test your ideas on small models. So that process probably continued throughout. no one stopped the researchers from— and so that level of progress probably continued. Yeah, the major, the major pre-trains and the major post-RL, you know, stuff on the major, on the big models probably was, you know, constrained. But the small models, they probably didn't care. And so they probably continued on all— and so none of the, none of this slowdown would actually have been a slowdown on the small model, like, you know, uh, experiments that they were doing. the slowdown would've been on the post-training of the larger models, which is where the bulk of the compute was being used. So did it really slow down? Probably not, right? The pace of development, if you have a bunch of researchers who continue to work, did they really slow down? Was that really a slowdown in that sense? Do you think Noam Chazir, Stopped working because they couldn't RL their largest, they couldn't post-train their largest model. Who cares, right?

Yeah, I mean, there's definitely truth to that. At the same time, all those people do say over and over again that compute is a key ingredient to their success and there's no substitute for scale.

Absolutely.

I think I can live with the risk, or at least I have no way around living with the risk that at any given day, Noam's Shazir or Brown could have an inspired dream, like the guy who somehow eureka'd the structure of benzene in a dream or whatever. Sure, they might have a great eureka moment any day to the next that shakes the snow globe.

Yeah.

I would put that under the category of irreducible risk, and I certainly would not—

Human genius. Human genius is irreducible risk.

Yeah, but for real though, I don't think we should try to go apply thought police to these guys. Nevertheless though, could we even— I mean, I think you've done a pretty good job of steelmanning the defense for OpenAI along the way as we've been debating this. And I really mean that. I went back and listened to highlights.

I'm not necessarily on their side, by the way. I just do this for fun, not necessarily completely on their side, but I think the researchers are very sincere. And I think the researchers are very sincere and the researchers are able to speak as freely as they can. I think the business people and the legal people are under much greater constraints. And I think people like Dario only realize that once they become executives and they're sitting in the chair for a while, then they realize you can't really say some things in some ways. And so I think that's hard. I think for the researchers to accept sometimes. And that's a very, very tough balancing job. And it gets tougher after you go public, right? Because after you go public, as Matt Levine says, at the end of the day, everything is a shareholder lawsuit. You can sue for anything. And if you don't report your profits high enough, that's a shareholder lawsuit. If you don't report your— you don't project your profits low enough, that's another shareholder lawsuit. So every one of these, like if you said you solved Navy SEALs and you didn't, that's a shareholder lawsuit. So I think it's tough. I think it's a tough place to be. And I think it's part of the maturity of the company. As companies mature and executives mature, they have to learn a little bit that this is the way things are going to be. You are getting paid a lot. You are in a very high-profile position. At least 10% of the country will hate you at any time, violently, violently hate you. Anthropic has had some scares at their office already, right? They had some scares in the office, I think a couple of weeks ago. YouTube had a major scare, I think, a few years ago, and they had to lock down the campus, Google Google campus much more tightly. Like, all of these things are still coming in the future. We talk about mistrust. You haven't, you haven't seen the level of mistrust that's going to happen when, like, you know, you have a model which is much more capable than 4.0 and sycophantic, which gets cut off from users, right? Like, there's still a lot more coming, right? So I kind of take this, this, this is just like small stuff. There's still like, you know, terror threats and this and that and like very upset people and Protests on the streets, and there's still so much more to go. Right? Like this is this is this is just the start of it. And so I give them I give them a little bit of grace. Like they're learning. They're they're they're kind of you know getting better at this. I think people like Dean Ball are going to be very helpful in in in helping them formulate kind of like strategically what things are going to look like, and that you know you have to be prepared for some of these things.

Yeah, I hope so, by the way. I certainly liked it when Dean went to the White House, and I felt like there was nobody better that the White House could have hired in any plausible scenario to take on that job. And I think OpenAI has less hiring constraints, but I'd still say I feel pretty similarly that he's a great candidate to go in there and do this. He's been a longstanding advocate for third-party access and auditing-type requirements. So I think those are among the best things we have right now. I'll note that there was— we don't have time to go too deep into this today, but we'll see as more stuff comes to light. This is from one of the authors of the recent German Wiki report. They say, since then, there have been more discoveries. Stay tuned. This is not that post. This is just about other things. But I guess maybe what I would— I don't know if you could— if you wanted, I'll offer you this challenge. You can decide if you want to take on the challenge. For a lot of different facets of this whole debate over the last couple weeks as we've been hashing this out. And I really, I do go back and listen to stuff as I put together the weekly highlights. And I have thought on reflection, as I listen back, some of your points were actually more compelling than I felt that they were in real time when I heard them again without feeling the need to respond immediately. But one thing I still feel like is a big challenge is what is the overall story that you could tell? What's the super high-level macro steelman for OpenAI? Where are we now? What are the commitments? What are we doing? What are we not doing? Can we synthesise or summarise an OpenAI position that we could not have to caveat 1,000 different ways? I personally don't think I could do that. If you can do that, I think you might deserve a Millennium Prize.

No, no. I think they have a lot of stresses pulling them in different directions. And I think internally within the firm, there's a fair amount of debate. People like Rune are scared now. So Rune has been a transhumanist frontier person for the longest time. Coldness— what was it he had? A coldness be my guide or something. way, way back. But now they're asking questions. They're like, okay, the open face thing was serious. And what should be the next steps? Where should we pursue this? I think to some extent, the capital cycle is forcing them forward and the capital cycle is being forced I think, by Anthropic. Anthropic didn't put in enough money earlier on. And so there is this intense pressure on Anthropic because they don't have the compute to have much better models. If you don't have the compute, you need much better models that can utilize the limited amount of compute that you have. So I think Anthropic is being driven forward by that to stay on par with OpenAI. And I think OpenAI is a little bit kind of like they're willing to pace the frontier because they have the compute. So regardless, they're the ones with the compute and they had the compute 3 years ahead. Elon will take time to come up with compute and Elon will sell to Anthropic, but Elon's going to take a long time, like 3 years at least. So OpenAI, they basically mortgaged themselves in the last 18 months to Masayoshi Son and a bunch of other people. They diluted. They gave up to Microsoft. They negotiate Microsoft. Microsoft gets all their models till 2032. They declared AGI and Microsoft is not out of their hair yet. Microsoft is now, I still own 28% and I'm still there. So they made all these sacrifices in order to get all of this money in order to pump it in and they have the compute. And having the compute allows them to actually pace the frontier because I have the compute anyway. And Anthropic is the one being driven forward by, you need to have better utilization of your compute because you don't have that much compute and you have higher payments that you have to make. You've bought from Elon at like $50 million per megawatt. And that's higher than what the newer clouds are making at like—

Yeah.

$20 million, $25 million a megawatt. You're doubling, you've doubled. And just that doubling allowed Elon to have this massive SpaceX IPO, right? So Anthropic is under pressure. They don't have the compute. And if you don't have the compute, you need better models. And this is the thing that's happening, right? OpenAI is willing to pace because they have the compute. That's the thing. And They also know Sam has played these cards. So he knows that if Anthropic is willing to pace, he's going to win because he has the compute and Dario can't afford that. So again, you're in this position that is, you know, as I say it, the 2nd and 3rd place guys are the ones who are going to define how fast the frontier paces. Like if Elon or Meta catch up to Fable, it's over. They're going to have to put out GPT-7. There's no choice anymore, right? So this is where we are. Would you prefer Meta or Elon have the golden ring?

Yeah, no, that's— I mean, that That's the new But China, and it is, I think, more compelling honestly than But China is But Elon and Zuck.

Yeah.

Um, still, I, I think, you know, this is kind of the qualitative standard that I am crying out for. Where do you stand now? We don't really know, right? We've just got such a muddled Mix of we're pausing, but actually we were never really much below like 2/3 of our highest day ever on RL scale. And we also do have a new, even more capable generation model that solved the Millennium Prize, but we also want to pace. But we seem to have a very High level of reluctance to even reveal what incidents have happened. I don't know. It's just, what really should we expect? I guess that's the bottom line. I don't know what to expect from OpenAI going forward. And I think that's true because they haven't made a clear statement. Even if they had made a clear statement, I would have to discount it or not I don't fully believe it, but there just hasn't even been yet a clear statement of what can we expect from this company for even the rest of the year.

I know what the product roadmap looks like. Early next year, a Johnny Ive device comes out. Johnny Ive device with very good voice and some kind of very fast model on the device itself to answer to you quickly. a lesser— a bigger model in the cloud and another model on top of that, and Astra sitting on top of that, if for the really hard questions, uh, able to do some tasks in the cloud for you, like the Instinct guys, uh, able to sandbox, do some tasks in the cloud, uh, you can— people will start giving up their credit cards and auth to this thing. It'll start doing things for them. Um, you will see advertising. So the ultimate goal is going to be for everyone to get, you know, this device or the ability to have this service for free. So they will monetize via advertising. Advertising revenue is already at $1 billion, so it's gonna, it's gonna pick up next year. Uh, there's gonna be some temptations on the advertising, I think, uh, on whether to skew the results or not. According to Lukas Kaiser at OpenAI, he says, he said it's hard to skew the results, but we don't know. And I think that is the major part of the consumer business. I think Sam might've been sincere in saying if they hit RSI, they might not go IPO. So Sam was in a podcast, I think, I don't know if you saw this, about 2 or 3 weeks ago. And he said, if they hit RSI, they might not go IPO. Well, they solved the Navier-Stokes. If they have like 1 or 2 transformer-level innovations in the next 6 months, maybe they don't go IPO. Right? Like, it's, it's not necessary anymore. And they, they, they continue as a private organization. And I think that would be bad. I actually think that'd be bad because then you don't have transparency. You don't have like widespread ownership of the stock. You don't have like, you know, boards that have to answer and you don't have like lawyers that can sue them for shareholder lawsuits. You don't get a bunch of these things that you get for free with a public public company. So I think that would not be good. So I'm hoping they do go public. So yeah.

Well, I guess there's, if nothing else, there's a little more reason today than there was yesterday to believe in the possibility of singularities in finite time. And that might mean we never get to own any of that OpenAI stock on the public market.

That's sweet, sweet stock. And on that note, maybe, maybe we should end that for today. We'll be back tomorrow.

Thank you, Prakash. See you tomorrow.

Bye-bye.