Is OpenAI's o3 AGI? Zvi Mowshowitz on Early AI Takeoff, the Mechanize launch, Live Players, & Why p(doom) is Rising

Hello and welcome back to The Cognitive Revolution. Today, I'm excited to welcome Zvi Moshiowitz back for his record 9th appearance on the podcast. As regular listeners will know, Zvi is a fellow AI obsessive who processes an unbelievable amount of AI information and produces some of the most comprehensive coverage and multifaceted analyses available anywhere, all on his blog, Don't Worry About the Vase.

The occasion for this conversation is, of course, the release of OpenAI's O3 model.

A powerful but confusing release that Tyler Cowen called AGI, but which OpenAI reported and the community at large has already extensively documented produces twice as many hallucinations as its predecessor. We begin with a discussion of the case for O3 as AGI, and in light of the fact that OpenAI also reported that O3 can complete more than 40% of pull requests recently written by OpenAI research engineers, Whether or not a process of recursive self-improvement, aka AI takeoff, has already begun. From there, we move on to discuss a bunch of important topics, including how to understand the strange situation we find ourselves in today, where it seems increasingly clear that AI can meaningfully accelerate science even while it still can't reliably order from DoorDash. Also, what superintelligence looks like in Zvi's imagination, including what sort of impact we should expect it to have on things like major national elections. The recent departure of key Epoch AI team members to found Mechanize, and why, at least when abstracting away from the details, Zvi and I are both inclined to support such efforts to automate mundane work rather than push models to ever higher levels of raw G. We also discussed the incredibly difficult challenge of imagining, let alone transitioning to, any sort of stable equilibrium in a world full of superhuman AIs, and why Zvi's P-doom is now up to 70%. We analyse each live player in the AI game today, including Meta, DeepSeek and other Chinese companies, Safe Superintelligence, xAI, Anthropic, Google DeepMind, and of course, OpenAI. Why we should be grateful that today's models are showing misalignment tendencies now while they're still relatively weak. Why SV doesn't particularly worry about autonomous killer robots. And finally, what's virtuous to do now as individuals and as a field? In light of all these developments. As always, I really enjoyed this conversation. Zvi shoots me straight and makes me laugh, and I really appreciate him for doing this late on a Friday night after another intense week. Laughs aside though, while I of course don't agree with every one of Zvi's takes, and I don't spend a lot of mental energy in general trying to pin down my own P-doom estimate, I do share the broad sense that P-doom seems to be rising. After a period in which pretraining on human data and light reinforcement learning from human feedback produced models that seemed to grok human values and behave according to their helpful, honest, and harmless mandate to a really remarkable degree, it now seems that intensive reinforcement learning is creating more powerful, but also quite palpably more problematic models. And not only are such models being released despite their obvious issues, but many in the community are going out of their way to downplay the significance of deception Scheming, reward hacking, and other bad behaviors. This, I feel quite strongly, is something that everyone, regardless of whether you love or fear AI or both, should seek to understand and communicate about clearly. It's my view that these sorts of problems are quite likely to lead to a popular backlash against AI, even if more existential risks never materialize. To do my part to help, I've created a slide deck documenting an ever-growing list of AI bad behaviors. We'll link to it in the show notes, and I encourage you to borrow from it freely for your own AI communications. On that note, I'm also excited to share that I'll be giving keynote presentations at 3 major AI events over the next several months. First, at Imagine AI Live in Las Vegas, May 28th through 30th, I'll be speaking about the strange mix of eureka moments and bad behaviors that we're seeing from today's AI systems. And how people should be thinking about harnessing the good while also protecting themselves from the bad. Then August 12th and 13th, I'll be back in São Paulo, Brazil for the 2nd ADAPTA Summit. There I'll again be speaking about AI automation with the presentation updated to include all of the latest developments with AI agents. And finally, September 23rd through 25th, again in Las Vegas, I'll be speaking to an audience of senior technology leaders at the Enterprise Tech Leadership Summit. About some mix of the above, plus whatever important new developments emerge between now and then. All 3 of these events have outstanding speaker lineups in which I'm very honored to be included. If you'll be attending any of these, please don't hesitate to reach out, as I'm always looking to learn as much as I can from the events that I attend, and meeting listeners is a powerful and fun way to do that. Finally, while I'm self-promoting, if you're working on AI automations, applications, agents, or adoption strategy for your organization, and think I might be able to help, I again encourage you to reach out. I'm currently working with 3 different companies for just a few hours per month each on issues as narrow and focused as prompt and workflow optimizations, as cutting edge as multi-agent system design, and as high level as strategic opportunity identification and prioritization. I've found that these engagements are natural win-wins. They help companies confidently accelerate their AI projects, and they also keep me super grounded in the practical realities of AI deployment where the rubber is hitting the road today. As always, whether it's for a possible speaking engagement, an AI advisory project, or just to give feedback on the podcast, you can reach me via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this wide-ranging conversation on the latest news and the most important big picture developments in AI with the author of the indispensable blog, Don't Worry About the Vase, The great Zvi Moskowitz.

Zvi Moskowitz, welcome back to the Cognitive Revolution.

Thank you. Thank you.

So you're fresh off your latest 10,000-word send. You are drunk on information like seldom seen before. And we're here to create the audio version for people that would rather hear you talk it all through than read the usual post. Although, of course, there's going to be, you know, a level of comprehensiveness in the post that we won't be able to match. So This should not be taken as a substitute for the blog itself.

The posts are canon. The posts are what I said after I had a chance to think about it. This is what I'm saying off the cuff. This is just what I'm actually just thinking. So enjoy.

Cool.

Well, let's get into it. Big first question: is AGI here? And is RSI, aka recursive self-improvement, here?

So no, and mostly no. I understand there are claims that O3 is potentially AGI. The more I understand it from the reports coming back and the more I use it, I think it's great. It's obviously not AGI. That's not what's happening here. This isn't even primarily an intelligence leap. This is a tool use leap. This is— O3 is a much, much more useful version of the thing we already had. It's like a lot easier to get what you want out of it, to get it to do the things you want in reasonable time, in a way that fits with what you want, that isn't forcing to specific boxes. And that's wonderful. That's going to be super useful, but it's not AGI.

Jack LeCun, I thought, did have an interesting frame where he said, how much more intelligent did you expect AGI to be? And so I guess I wonder, there are many different frames we could put on this, but How many years ago in the past do you think you would have been given O3 and said, well, yeah, this has got to be AGI. What else could it be?

I mean, if you ask the question instead, how many years in the past would I have gone, holy fucking shit, when I saw O3? The answer is something like 2, right? Definitely like 4 for sure. I would have been like jaw on the floor, like face play. nuts. Like, just like, how the hell does this exist? How the hell is this possible, especially this soon? But that's very different from saying it's an AGI. And, like, it's, again, like, we're talking about, you know, something that can do all the things that humans can do. We're talking about the thing that can, you know, basically just plug and play anywhere you need it to, to do all the cognitive work. That's what we commonly understand AGI to be. To some extent, I think Tyler is defending a, you know, your term is defined broad. You are using a silly definition of the term. The thing you're asking is like, is it smarter than me? And like, from his perspective, the answer is yes. I think he's using kind of the wrong definition of smart. I actually saw his comment and decided I'm going to deal with this last. I'm going to go through everything else I have in my queue. And then as the last thing that I write out, I'm going to deal with Tyler Cowen's claim because I want to understand all the context before I evaluate this question. And by going through all these other claims, I feel like I understood what O3 was. And then I was able to look at Tyler's claim and understand why he was claiming what he was claiming, which is that it's a very good program for doing exactly the things that Tyler values highly, does all day, is consumed by how he thinks, right? If you look at his examples, if you look at what it can do, it can go out there and get you tons and tons of detail, tons and tons of specific facts about any given thing, make connections between those facts, structure those facts, find the relevant things, present that to you. And he eats that all day. That's the thing he does. And he does it so much faster than I could do it even if I wanted to. It's an amazing skill. I'm in awe. But that's just not how my brain works, right? I produce lots of content a completely different way, which is I consider information as part of a logical way of understanding the whole picture. And if it doesn't fit the logical picture, it doesn't particularly seem relevant to me, then I won't— it'll bounce off of me. And similarly, if it seems like it's just part of the pattern, once I understand the pattern, I don't need the details that led to the pattern anymore. So I look at— he gave 3 examples, right, when he claimed this. So he's talking about, okay, why is this guy's paintings that are from early much more valuable than the paintings from when they're late? And this is part of a pretty common pattern for artists who basically didn't constantly reinvent themselves. Right. Where early on they're doing the thing that's considered first, it's considered unique, it's harder to find, that's like special, like in many ways. And we've seen one thing that I don't think O3 did point out, which is we have seen this extremization of value of collectibles across the board in the last 20 years, where the thing that's the most best condition, the thing that's the actual unique first thing, the very best painting the guy made, the very, very special thing is now like 10x value. What, like, the thing was almost as good that if the first thing didn't exist would be like just considered exactly the way they consider the first thing now. So like the 9.8 comic is like so much more valuable than the 9.6 comic, right? Like you just, you find exactly the thing you want. And like, this confused me for a long time. Like, as a Magic: The Gathering player, Because why is the thing that's the same thing but looks slightly better so much more valuable? It plays the same. But no, that's not what people care about. But basically, because yes, you've got demand-supply, you've got what is considered the best, the most unique, the most high status, the most representative of the thing. It's pretty standard. So I don't need to know any— I'd never heard of this artist. I have no idea who this artist is or what he does. I don't have to. I can answer this question to myself anyway. And so O3 is giving me like, okay, sure, you're giving me all these details. Similarly with like the author and like, why is his prose so amazing? Like, whatever. It's good prose. It's prose people like. Like, you can, you know, throw in specific details about this particular person, but they don't care. You know, it's like, okay, so you solved it. Congrats. And the question about Knoxville, Tennessee, like impact of the tariffs, it's gonna suck. That's for everyone. Like, the tariffs are a giant own goal. Yes, it turns out that all of their manufacturing is importing because all of our manufacturers import things. and they're going to be more expensive, they're not going to be competitive, and it's going to be a huge disaster. Thank you very much for giving me the details. And I'm like, okay, congratulations. It's just like deep research. You're giving me a briefing for a congressman to prove that your district should oppose this thing because here's the specific factories that are going to be put out of business or whatever. It's not that it's not a useful thing, but that's still like the intern's job or something, right? I'm not surprised. I'm not interested. And this is just a persistent disagreement between me and Tyler about what's interesting about the world. He travels, I think, largely because he eats this stuff up. And every time you travel, you can get all of these details. It's just nothing but extra detail about whatever you are. And he feels like this is necessary to understand this place. And all these details help you understand the world. And this is how one becomes someone who understands things. And I'm like, nah, bro, that's nice for you, but none of that matters to me. most of them more or less understand the things I need to know about these places from my perspective. And it's just a hugely inefficient thing to do, to go around gathering these irrelevant details if I'm not enjoying myself. So it's just a very different way of looking at the world. And of course, the thing that thinks like he thinks and is doing a very good job of what he's doing, that will be the thing he's like, it's AGI. And a great heel mark of the AGI paper is his A pluses. It's fine.

So this reminds me of the Dwarkesh comment from a few podcasts ago where he said basically these AIs, right, they're Tyler Cowen-like in that they soak up an unbelievable amount of information. Obviously they've read the whole internet. They have greater ability to do like GPQA or humanity's last exam or whatever than probably any human, I would imagine, at this point. Right. I mean, Certainly extremely rare to have that breadth of knowledge. And yet he asks, why don't we see these things coming up with genuinely novel, interesting connections across these domains? And I feel like maybe we just haven't been trying that hard. The first time I felt very clearly that I was like, damn, that AI seems smarter than me, was— and I've felt glimpses of this other times too, but recently, I did this episode with Vivek and Anil from Google who have put together AI doctors at Google and also this AI coscientist that they have. I don't know if you read this story, but the coscientist was tested on 3 levels of scientific challenge with increasing open-endedness. And the hardest one basically started with an observation about some— and this is where you're like, damn, this thing is smarter than me, right? Because I can't even really describe the problem super well. But some observation that something is conserved by different kinds of bacteria that both have drug resistance to some class of drugs. And starting with that observation, the question put to the AI was basically, what's going on here? So it's a pretty tough one. The AI in their setup had a lot of inference. They're definitely on the scaling inference train. This was Gemini 2.0, not even 2.5. It did have searchability, and it did have the ability to call AlphaFold and maybe some other specialised tools. And then it just had a bunch of different prompts where it's sort of grinding against itself to come up with ideas and evaluate the ideas and yada, yada, yada. Lo and behold, at the end of this whole process, and I think they ran it for like a couple of days, it spit out a prioritised list of hypotheses. And the number one hypothesis that it had flagged had been by a group of scientists that Google was partnering with, demonstrated experimentally, but not yet published. And to hear them tell the story, the guys fell off their chair to hear that an AI basically was able to just comb through the literature and basically come to the same conclusion. How does that strike you? Is that AGI? And I guess it's hard to say, but if you thought we— if we reran that with 03 as opposed to Gemini 2 in that super scaffolded, deep access to search, yada, yada, yada? I mean, it kind of has its own search built in now, but do you think there's like a qualitative shift there where this thing maybe now is like a full-fledged AI scientist? I mean, I'm having a hard time seeing how we're not already getting, like, tipping into the geniuses in a data centre, honestly.

Yeah, it's definitely weird. I strongly suspect it's a skill issue for the humans. I strongly suspect that if you put me in charge of AI Scientist Corp and you gave me $1 billion, right, just to make sure it's just not an issue, I have all the compute I want, I can hire whoever I want, I can try all the things that I want, that I could start Figuring these things out, making these connections, doing these things, subject to, like, yeah, I still can't— we can't do perfect simulations, so we're going to have to actually try experiments and give the thing feedback and so on.

Yeah. Notably, there is a difference between the AI hypothesised this and the scientists.

Yeah, we're not at the point where we can get the full explosion without interacting with the physical world. We're not even close. But yeah, I think we're just really bad at elicitation. of capabilities. We're really bad at scaffolding. We're really bad at creating the loops and the tool use and the logic of how to proceed on these questions because it hasn't been what people have been a priority. There aren't that many scientists out there and it's just not what people were focusing on. I don't think it's particularly hard in some important sense. It's not what I've been thinking about. But, and maybe I'm just being fully naive, right? Maybe they've tried all the obvious things that are in my head and none of them work and they don't understand why. Or maybe they do understand why. But yeah, it just feels like when I see them, like when you see the Google AI scientists, right? Like it's very much clear that like they are just saying, what if we try to duplicate exactly how humans work in the scientific process in the real world and just try to duplicate every step of the way? each thing with AI exactly the way that we do it, because we know that works, as opposed to trying to figure out how do we use the tools that we have to generate the things that we have. But also especially because scientists don't have billion-dollar compute budgets. We don't properly value— even before we started firing everybody and pulled all the funding, we don't properly value basic research. We don't give them the resources they need. Everybody is on a shoestring. So a lot of things that I would think to try involve just trying lots and lots and lots of shit. Because, okay, I can just, like, if you have, you want to draw connections between things, right? And you can have, like, okay, I have a million VIX. Well, a million times a million is a trillion. That's not that much. How much is a query? And I can be more clever at that, right? I can do clusters. I can not have to do the full multiplication. I can use some discernment to figure out which combinations I have to check, et cetera, et cetera. So I can, in some important sense, get really creative if I care enough. But also, if I wait 6 months, everything costs 10% as much, right? So for the same level of comprehensibility. So why am I jumping the gun to try and do all this clever, throw all this money at all this compute, at this problem to try and brute force this insight when I can kind of just wait for the models to get smarter and more efficient.

But shouldn't pharma companies be all in on this? I mean, it seems to me like that is all well and good, but my estimate of the cost of running the Gemini 2.0-powered scientist for a couple of days was somewhere in the hundreds to thousands of dollars, depending on whatever. And maybe it could get up to like $10,000. But it seems like if you're a pharma company, you should be hammering the API, if only because you've got a similar game theoretic question that all the hyperscalers have vis-à-vis each other, right? Like, they all want to buy all the GPUs and not fall behind on the model frontier. Why don't we see a pharma race where they are all saying there's only so many of these things to be patented? Who cares if it's $10,000 a co-scientist run, or $5,000, or $2,000, or You know, I have like 6 months.

And that you, Google, would de-risk that for them very happily because of all the publicity and like benefits of like, I'm the one working with Pfizer. If Pfizer discovers anything with AI, then we get credit for having done that. We win Nobel Prizes too. We get all the publicity and like all the benefits too. And like, okay, I'll take a cut of the profits if you find anything. You don't even need to pay me. Yeah, I'm sure these things are possible, but The real answer is, and I mean this in all seriousness, people don't do things. Like, people just don't execute. People don't, like, take risks. People don't do things that look weird. You know, people are very slow to adopt these things, and big corporations more so. And, like, this is the diffusion problem, right? This is what, like, you know, Yeah. Tyrell Cohen going bottleneck, bottleneck, bottleneck, bottleneck, like pointing at everybody, right? Like, and this is like where that's real. This is where the reason is just people are like, I can't stop. I mean, look, even I, right? I write about this all day. And if you honestly looked at the tools I was using and the level of automation and the things that you could do, dude, just take some time off and code, right? Just make something better or hire someone to make something better. And I did some of it, but I could do so much more. But also I can work You know, probably 2 to 5 times as fast as I could back then, just because the AI has gotten better since the last time I was doing it in a few mocks. Like, it's just such a— so it's all—

It really is.

Like, people are just very, very hard to, like, get them to get off their patterns and adapt and explore and, like, try new stuff. And we're just trying to get through the day and we're just trying to run the thing, but the day will come. And it's also because of this demoralisation of everything is going to improve again in a few months. So I think that if there was, say there was a pause, right? Say there was a pause, not because we all agree on a pause, but say there was a pause because it turns out we just hit a giant wall in terms of the base models and this is as good as it's going to get within reason. And it stops getting cheaper, it stops getting better. We get 0.4, we get 0.5, we get GPT-5, we get Claude 4 and Gemini 3, and it's all like— 5% better, 10% better, but it's not like amazing. And now we're like, no, no, you can't just like sort of wait for this to happen on its own. You gotta make this happen. It's your responsibility to figure out how to use this thing. And you're gonna have a period of time to use this thing. It's not just gonna be a few months before it all gets blown away. I think we start to see some really, really creative stuff get a lot out of what we have that like right now, just everyone's just trying to keep up.

Hey, we'll continue our interview in a moment after a word from our sponsors. Do you have— so I think about this a lot for myself too, where I'm obviously thinking about this all day, more talking about it than writing about it, but a bit of both. And I use a lot of things. I still don't live a very automated life, and I sort of challenge myself, like, why is that? And am I doing something stupid? And one answer that's the sort of charitable answer to myself is, Well, I don't really do that much work that's routine, which is a great luxury that I don't take for granted. And so the sort of real-time assistant paradigm is pretty good for my non-routine life because it's sort of the right form factor for non-routine work. If I had more, I tell myself if I had a lot more routine work, I would set up these workflows and pipelines and stuff, and then I would do it, but I don't. But then I'm like, I don't know, am I letting myself off the hook too easily here? Maybe I should come up with some routine stuff that I should be doing that I'm not, but I could, because I could automate it with AI. And maybe I'm just not that imaginative or not that creative or not trying hard enough. But for me, it feels like the bottleneck in terms of living a more automated life is kind of like, I personally am not doing a lot of routine tasks that I would like to automate away, and I feel a little low on ideas of things that I would automate that I'm not doing at all in the first place. Do you have things that you are like, you know, conscious of that you would think, you know, the Zvi++ would have already scaled with, with AI automation?

So it was, it was always the question of, is the automation going to be good enough? You actually use it and you actually don't spend that time checking its work. You don't spend that time doing the manual version of it also. You actually end up saving time by doing that. And there's certainly a substantial amount of chunk of that that would be very good. Certainly doing things like notes and organising resources for myself and organising just facts and links and stuff for future reference would be great. Format. But then again, it's very hard to automate that, right? Because it doesn't know what you want to remember. Formatting for websites, I did some of that. I wanted to like do the automatic post to Twitter, but it's a remarkably annoying problem. And because like of the structure of what you're trying to do, the AIs are dramatically worse in that spot because normally you're like, code this thing that does X, that does Y. And like just doing Y is a very easy thing for the AI to understand and solve. But if you're trying to like have it navigate idiosyncrasies of this particular existing website that's kind of coded like crap.

Yeah.

Twitter, then you run into this problem of like, you're just going back and forth and debugging and like trying to get it to do something. Like in the past it's been like pretty terrible. And if you get it to do its own feedback debugging loops now, I don't know if you can, and suddenly it probably gets a lot easier. There's a lot of different, like really hacky things you'd have to do to get it right, but it would save you a bunch of time. There's some formatting things like I'd really like to be able to I like to like sort of transform the way that I interact with like the web and like create a bunch of shortcuts and a bunch of like, you know, just like quick ways to just transform the data and so on. And like, that'd be great. And then I go from there. It just, again, like I can't catch a break. There's always something going on. And then like I get this, but then, but then like you, then you get this day, right? You get that day when you don't have anything going on because you've managed to finally get ahead of your giant—

Then somebody invites you on a 3-hour podcast, and the next thing you know, the, uh, the day is wasted.

That's not what I was gonna say. What I was gonna say was you're just so mentally checked out after all of that, and you're so happy to have some time that you're like, why don't I go see a movie? Like, why don't I go out and have a nice dinner? Like, why don't I just chill? And then by the time you're done chilling, like, there's something else to do. And so, like, it's very hard to, like, not just get the distraction. Like, I have so much stuff in my queue. I have so much stuff I wanna do. Like, everyone asks, like, how do you do all this stuff? It's Like, well, you don't do anything else, right?

Yeah, I work a lot. Spoiler.

I work a lot and I hang out with my family and I like, you know, chill and I watch TV, like, and listen to podcasts and like, that's kind of it, right? Like in some important sense, like, and exercise, eat, and so on. But like, that's it. So yeah, like, where's the time to like code? And also I found that like coding for me has like large increasing returns to scale in terms of being in that mindset where my brain is wrapping itself around the issues and like, you know, like programmers like to have state, right? Like they like to keep the problem they're solving in their head and not be distracted. And so like, if I'm gonna just like, I can't do 5 minutes of programming a day, right? Like that doesn't help anyone. That's useless. You wanna be doing hours. And so like, I really need a large amount of dedicated free time where I'm really gonna dig deep into what I have and like, load up Cursor and, or, you know, Quad Code or whatever it is, maybe Codex now. I don't know. I've heard mixed reviews and, and, and do my thing and then see what I can do. But yeah, it just sort of never feels like the right time to make that investment. But I, and like when I did make the investment, like I think I've like made my money back time-wise at this point. But it was really a struggle. I don't know. Yeah.

I feel you on the challenge of ramping up into the programming problem space. I've been trying to get like a day a week. It's been more like a day every other week to really fully program. But then it also is like, man, my feeble biological brain, when I sit down in the morning, it's like, what was I doing 2 weeks ago? And I need a minute just to kind of recalibrate myself or reorient myself to the problem. And it's funny, that is one area where the AIs really shine. I think one tip that has actually worked for me that might help you and might help others, I hadn't really thought about this being a tip so much even until right now, is like to literally pick up with the chat from 2 weeks ago, wherever I left off, even if it was kind of in the middle somewhere, just to go back to that thread.

Yeah.

Where were we?

What were, what was the last 5 things we did here?

Yeah, my experience with that kind of thing is like, it, it's sort of, sometimes it works and sometimes it's just this huge disaster and you never get it back. It's like sometimes, like, cuz like to me, like coding is often like this thing of like reading from a like forbidden book of wizard incantations and you're really, really hoping you don't mispronounce a word and suddenly like summon the wrong demon or whatever. And so like, or just get a series of error messages. And so. If you like make, if everything goes exactly right, wonderful things happen. But if you make even a small mistake, like it can be so hard to recover because you don't understand where the mistake is and you're not good at figuring out what it is. And the AIs will sometimes rescue you because you always paste in the thing and the AI will say, oh, this is what you did. And like, great, we're back. But if they don't, you're just so screwed. And so, so yeah, I'm again, like I should invest more in it, but I keep getting like, you know, here, take this trip. go to this conference, be part of this exercise, do this other thing. One thing leads to another. So you just need to clear the time. But it's just really hard. I mean, now that I've dealt with O3, I'm hoping that I'm only— that caused me to be one day behind with everything else. So I have a few hours of work to catch back up. But after that, I don't know, maybe this weekend I can do something, right? Maybe next week I can do something. It's entirely possible. It just depends on what happens. Like, what if Google released 2.5 Flash and, like, nobody even noticed?

Yeah, that'd be so weird. It's not on our agenda today, really.

Actually, that happened yesterday.

And I'm sure it is very good. I haven't used it yet, but—

It probably is. I have no idea.

I'm confident it's very good.

I mean, I was a big— I've seen enough benchmarks. And I'm like, if a model that small is scoring that well, and given how good 2.0 is, this is going to be an amazing model for its size. But in practical terms, what can I do with it? I don't know. I'm just going to use it for free anyway.

Okay, here's another thought experiment. The drop-in knowledge worker, I hypothesise, is bottlenecked right now, mostly on a couple of things. One is just like effective ability to use a computer. Put a pin in that. We'll come back to that in a few minutes. The other is like the ability to sort of absorb the surrounding context in a reasonable way, in the way that humans do when we get a new job, right? You get a little bit of training and that helps, but then you also sort of bop around a little bit and poke around some Slack channels and talk to people and look at their work product, and eventually Probably get some feedback for doing things not quite right, and then eventually you sort of figure it out. It seems like it's clearly conceptually possible for an AI of the base level of intelligence and breadth of world knowledge of one of these current latest generation of models to do some sort of additional, possibly training, possibly just like processing or memorising, maybe even like Long context could get there. But if you imagine an AI that's no smarter than what we have today, but it can drop into an organisation, go through all the old emails, go through all the old GitHub issues or whatever, go through all the Google Drive, look at all the proposals that have been sent out in the CRM, and obviously can process those things a lot faster than humans. And it kind of comes out with a sort of like, All right, I kind of got it. In the same way that I know who was the prime minister of Lithuania 5 years ago, because I just kind of know everything, I sort of just know everything now about this company in a similar way. Is that AGI?

If it can do basically anything that way, or if it can do most tasks that could be done, most jobs that can be done from a desk, then yeah, I think you have to give it to You have to consider that AGI, not ASI yet, obviously, but you can, you do basically have to give it AGI. But like, that implies you can also do the job of an AI researcher, right? Like, or else it wouldn't count. That's maybe a relatively easy task. It's relatively hard in some senses and relatively easy in others. Definitely not near the end of the list of like how things get automated in what order. But yeah, no, absolutely. And I do agree there's like a lot of room to do it. But like, I look back at my time at Jane Street, right? And so like, they take a loss on a new employee effectively for at least a year. Like, not because you're not creating value, but because like the amount of time they spend from other people that's required to keep training you, to make you better, to give you feedback, and to let you like learn where someone else could be doing the thing that you're doing better. but instead you're doing it so that you can learn to do it yourself and you can learn to be better. The first year, they had less money because they hired you, even if you are doing as well as you can realistically be, like a 90th percentile person. And just think about that level of investment and compare that to the patience people have for a drop-in worker, right? If you tried a drop-in worker and you asked them to put in 10% that much investment before it started doing useful things. Like, zero companies would tolerate that.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, I mean, that's a really interesting claim because my hypothesis has been, and I don't know what the first-year salary at Jane Street is, but I have been expecting that, first of all, just kind of ballpark calculation to fine-tune GPT-4o or for one is $25 per million tokens of fine-tuning training data. So if you were to say, how many tokens do we have at a given company? Obviously it's going to scale widely. If you had an actual trillion, you would be looking at a million million times 25. You'd be looking at $25 million to make essentially a custom model, which is kind of notable. If you were 5% of the internet or something, Yeah, but I mean, certainly my business doesn't have a trillion tokens. Maybe we have a billion, in which case I'd be looking at like a $25,000 investment for it to be kind of fine-tuned on everything. I don't think you could literally do that. What comes to mind, of course, is the famous OpenAI graphic of the model they trained on Slack, where you say, do this, whatever, and it says, I'll work on that tomorrow, because that's what it saw in Slack. So there definitely has to be some processing of all this sort of random data.

That's the issue. The issue is that you can give me a million tokens or a billion tokens, but one does not simply train on those tokens and then suddenly this machine can do the thing that's described in the tokens.

Yeah, you need a pipeline of processing that, filtering it, transforming it, making sure that Machine learning is the most fiddly thing on the planet, right?

It's the most like trial and error, learn by doing, see what works and what doesn't, like do all of this bespoke stuff to get it to work, like just keep working at it. And like, that's very, very different from a plug and play, right? I think that if OpenAI said, pay us, you know, $500,000 and we will train you, your automated worker and we'll then license it to you for $20,000 a year per copy on top of that, and then it will do your job. People would just jump at that if all they had to do was like dump all their tokens in without explaining what the tokens were. But that won't work, right? OpenAI doesn't have that. You can't just dump the tokens even if they knew what they were doing, because you'd have to provide the context for the tokens. You have to organize the tokens. You have to know what you wanted to be able to do with it. And then OpenAI would have to figure out how to navigate that into an actual proper tuning program to incorporate all the things. It would have to work closely with a bunch of your employees for a while. None of this is easy. And, you know, the problem is getting to the point where, like, the AI can do the job of setting up this training process for you so that you can then create this worker or something like that, probably. But it's going to be a while. It's just going to have to be a very human trial and error style thing unless we're, like, well above where we are now.

Yeah.

It's coming more and more. But Operator was remarkably bad at even the tasks that everybody basically does the same way all the time, to the point where even though I was paying the pro value anyway, I was just like, I don't want to take the time to try and figure out how to use this and start giving it the information it needs to work. Today, actually, like I said, I was working through and I was like, I'm going to DoorDash. Should I use Operator? Should I start learning how to get Operator to do my DoorDashes for me? And I was just like, I have to figure out how to get it to tell it to do no lettuce, no tomato. I'll just— no, I just want food. And then that was the end of that idea, right? That kind of impulse, but at scale.

Yeah, I mean, this task reliability thing, again, let's come back to it in one more second. I mean, I do feel like We might be quite close to the sort of additional training of processing of your information and then training of your custom knowledge worker. Could be quite close. 2 reasons to think that, one of which also has many other implications. First one is just in my own experience, the big place where my limited coding hours have gone recently has been to make a minor contribution to the Emergent Misalignment Project, which honestly, the team was very kind to include me as the last and least valuable co-author. It would have been understandable for them to not include me at all. But with Gemini 2.5, I'm able to take the full research codebase, which actually, it's all the code plus some of the data. I wrote a script, which I, of course, used AI to do, To say, go through and print out all of the code from this entire research codebase into one file. And then it was like way too big. And I was like, why is it so big? Oh, there's like datasets in here. Okay, just give me the first example of each dataset as you go through and give me all the code. That comes out to like 400,000 tokens for this particular codebase. And then there's the paper itself. And then I had done some additions, again, having AI do most of the coding. And I came back and I just wanted to put the updated codebase in the paper and just make sure it sort of knew what was going on. And I asked it to tell me what was going on and make sure it understood the new stuff that was in the Nathan folder, because again, research codebase. So there's like the Jan folder and the Daniel folder, and now there's the Nathan folder. So this is not like production-grade software, but it didn't have any problem with that. It honed in on this question that the paper mentioned briefly but didn't really dig into. And then my additional experiments to try to expand the understanding of that particular topic and gave me really an unbelievable readout of like, this is what you are trying to do. This is what these experiments that you're adding on now to the codebase are for. Seems really interesting. How can I help, basically? And I was like, damn, that is really something else, because I didn't even tell it that. It inferred that from— I told different context window of the model that, and it helped me write the code. But then just from the code itself, it was able to infer all that sort of stuff. And I was like, man, first of all, that's a lot of information. 500,000 tokens, basically in context, you're talking a full 1,000 pages of stuff and a really sophisticated readout that inferred the intent and on a pretty frontier topic too. The paper itself is not in the training data because it's only been out for a month and a half or whatever. So I was like, if you can do that, then I would think you would be able to kind of wade through large reams of Slack chats and stuff and really kind of zero in on the stuff that really matters and probably put it into a decent format and then pipeline that all the way through to a fine-tuning recipe that you sort of probably don't necessarily optimise to all the hyperparameters for each customer, but you sort of did that at a meta level.

Yeah, it's possible. I think there's a lot of O-ring style problems with this kind of approach, in addition to the whole, like, everything's fiddly and nothing works the way you want it to. If you've got a worker, quite often you'll say, like, Bob is great, Alice is great, But they have this specific thing that we can't stand that's terrible. And like, this is a deal breaker, right? Like, even though, you know, they're 99%, like what we want, that's not good enough to be that useful. We can't just have that 1% be handed off to someone else. Like, it doesn't work that way. And, you know, I think this is a lot of the problem of until the AI is crossing the threshold of being good enough at a given task or set of tasks, it's not considered useful for the grand set of things in that way. And so people don't think on that level because, again, they don't have that kind of patience and that kind of, oh, we'll be able to fix it and so on. And so instead, we're starting with little tinkering, which is how actual progress is Made in most things, right? So we're just saying, okay, here are individual specific things the AI can do well, and then over time we will figure out how to string those together. We'll figure out how to get the most out of each of those things, and then they'll combine more and more, and we'll start filling in the gaps, and the AI will start asking itself, how do I fix these gaps? And then as the models get better and our understandings get better, they will come together, and then these things will start to happen. Yeah, we're just not quite there yet. But we're starting to see how much closer to there we're getting very quickly. A lot of this stuff seems a lot more tempting now than it did 6 months ago. Right? Like, vastly, vastly more. 6 months ago was like before '01.

Yeah. Talk about compressed time. Okay, so it seems like one other comment that Greg Brockman made, which I know you highlighted in the post as well, was that this is the first model that they've had scientists tell them comes up with really good ideas. And this sort of has me thinking, we might be in a really weird spot where the hit rate on good ideas for frontier scientists, especially with some decent scaffolding, is very plausibly, and I would even say maybe likely, high enough that if you are a frontier scientist, you really should be using it. And it will very likely accelerate your work. I feel pretty confident in saying that as a blanket recommendation to most scientists. And yet, at the same time, you're like, but we can't get it to order DoorDash reliably.

I think it's just a matter of hyperbolic discounting, and there's not a warning to do that right now.

Also, for Operator, you would have had— Manus, you might have had more luck. Operator, personally, It just asks me, do I, should I continue? Should I continue? Like every 2 seconds. And I'm like, yes, you should continue. Put the thing in the basket. You know, it's like, it's unbelievable. Manus, by the way, does have a much better, I mean, for better or worse, but I think for your mundane utility purposes, it won't like ask you like, okay, I, you know, found the sandwich. Like, would you like me to put it in the cart? It will actually just do it. So.

Yeah. Well, I'm also not giving Manus my credit card.

That's fair. I'm still on the free credits version.

Right.

But if I don't give it my credit card, how's it gonna order a sandwich?

Oh yeah.

Well, that's a, you can potentially have it in, I mean, they do have an interesting, well, I might be confusing the operator and Manus security model. I created a new Gmail account for Manus so that I could share docs with it, log in to Gmail as it, and then have it go see all the stuff that I shared.

My idea is, you know what, Carl? Anthropic will have its own version of this. I'll be able to trust that, and I'll just wait. It's fine.

I will say there is something quite valuable about using products that really turn up the hyperparameters. And unfortunately, Claude does not do that yet. Like, a good contrast is Shortwave, which I've been using for email, versus the new Claude Gmail integration. I went to Shortwave and said, read my last 100 emails sent and give me some advice. And it uses Claude, so it's the same model, but it actually did search for 100 emails, put them into context, and had Claude process that big dump of information. Whereas when I went to Claude and gave it the exact same prompt, it searched once for 5 emails, which appears to be the limit, again for 5 emails, again for 5 emails, got to 15, decided that was enough. And then carried on with the task with just the last 15 of those, just from the last 3 days.

Not until we get Claude email integration to work. I use Gmail too, and I was really excited, and then it just keeps failing at the most basic tasks, and I just don't know what's wrong. But for now, I'm like, this doesn't really exist. And of course, Shortwave is also read/write, not just read, which is a huge, huge difference. But like, you think it's worth using? Do you think it's like past the threshold?

You know, it's not perfect, but I do get real value out of it sometimes when it does like an expense report for me. It's pretty cool. Somebody today, we got a sponsorship, you know, inbound and they were like, what kind of episodes do you have coming up? And I just went to Shortwave and said, please pull me a report of all the possible podcast guests that I have coming up. Then put the most interesting ones at the top, basically. And for those sorts of things, it is definitely notable utility. I mean, it's still just the real-time back and forth model. It doesn't send anything without your approval, and it doesn't mark anything as done without your approval. Instead, it will be like, here's 12 things I think you should mark as done, and you can uncheck any that you want to uncheck, and then you can mass done them. So it's not like running off with your account in a way that is taking your agency too much at all. But I do find that the hyperparameters turned up in general is just a pattern that always adds a lot of value relative to the default. And I'm frustrated honestly with, and I think I know why, and it's understandable that they're compute limited and they might also Start losing money on customers if they turned all these hyperparameters up. Shortwave, until recently, was losing money on the margin. He said, yeah, you know, as of a year ago, he was like, every new customer, we lose more money. That, he said, is no longer true. But like, they were just willing to eat that, and they weren't at a huge scale. And so they were kind of like, let's just look for the best value we can.

Because they know that, like, everything goes— gets cheaper. So like, the same product will, in fact.

Yeah, they're— you're definitely riding that wave. So, yeah, hyperparameters turned up is good.

Okay.

But so I guess I want to get your take on, like, because there's a couple of things that are pulling in different directions, and I find myself a bit confused. We have this world where AI can accelerate science. I think that's becoming pretty clear, especially if you as the scientist are willing to accept a not perfect hit rate, and maybe only 1 in 10 ideas are going to be worth your seriously engaging with. But that could be a huge hit rate for science.

If you're a scientist and you're like, this idea only has a 1 in 10 chance of being a great idea, I don't want to engage with it, you're a bad scientist.

Right. So it seems like we might be into the realm of, like, science seems like it's accelerating. This sort of mundane personal assistant work that we would all love to delegate and unchain ourselves from our desks more isn't quite working yet.

It's close, and that will accelerate science, right? I mean, think about it. What's the biggest drag on science right now? It's that the average scientist spends a huge percentage of their time on things like fundraising and dealing with paperwork.

Yeah, I was going to ask you, actually, if you think that this is Yeah, a little bit out of domain for this feed, but it strikes me that the withdrawal of federal funding from top universities could potentially reinvigorate universities in a way that we haven't seen for a long time, because all of a sudden they're going to have a different funding model. And how are those decisions going to get made? And what if there's no federal government bureaucracy that they have to appeal to. And it's just like, well, I guess we're the chemistry department. We've got to figure this out on our own. How do we do it? Do you have any optimism for revitalisation of science through withdrawal in Searle?

But in the EA-style ecosystem, where you have to get non-government people to give you money instead of getting government to give you money, it's not better, right? It's relatively good because a lot of people involved are actively trying to make it painless. and actively trying to do good things. But it has so many of the same problems. It has so many of the same bad incentives. Obviously, to me, well, it'd be great for Harvard if Harvard stopped taking federal money and instead used its ridiculously large endowment to pay for things and maybe funded itself with the products of its discoveries over the long run, maybe, or something like that. But And certainly it's possible to say that right now you're dependent on this model that, is it focused on producing anything useful? And it takes up a giant amount of your time and it ends up favouring people in their 50s over people in their 20s and 30s. And most good science is done by people who are relatively young historically, because that's just sort of when your brain is best attuned to do that kind of science. And changing all of this up could be really good. But there's always this problem of, like, we're going to break the current system. Well, that's great if you replace it with something else. And if you just try to hobble along with the broken system, it's just obviously just worse. And so do we have great hope that we're going to be able to actively fix the problem? And I'd be more optimistic, I guess. Also, will the scientists just flee to Europe? in Canada and Japan and everywhere else because I don't want to deal with this and I can get funded somewhere else.

And half of them are from all those places in the first place. Yeah, right.

Or they're going to industry, which some people think is better, but obviously it's very different if they go to finance and you've got a problem. If they go to Google and then Google funds basic research, then that's great. But is that what's going to happen? Yeah, it's really hard to know. But I do know that on the timelines that we're looking at with AI, I don't particularly want to break our current system and force everybody to spend a few years scrambling to reinvent the new system separately from the fact that we're going to break everything anyway. There's no time for the new system to pay off.

Plate is full for crises at the moment.

Yeah, the old system was kind of going to break anyway. To a large extent, right? Like, none of it's gonna make sense in the new world. The university, like, entire education system, like, kind of doesn't make any sense by now and will make less sense in 2 years.

Yeah. I mean, I don't know, I don't have any, like, comprehensive data on this, but it wasn't too long ago that it was less than a year ago that I was invited to give a presentation to some computer science student club at an American university. These are mostly undergrads. And they were like, our professors don't allow us to use any AI coding assistance. And I was like, I don't really know how to sugarcoat this for you, but I don't think your professors are doing you a good service by doing that. Not to say there's not a place for some independent exercises there, but to just pretend it's not happening is, it's a tough position for the universities to defend for too much longer.

I mean, I, my perspective is probably Learning how to code with AI while you're focusing on actually learning how to code is better than learning how to code without AI is better than coding with AI to try and hand in the assignment that's due after class.

So speaking of people that are learning to code with AI, for me, the most notable part of the O3 model card was, I think, on page 22 of 30-some, Where they report the progress on the model's ability to successfully one-shot real pull requests previously submitted and unit tests developed for that real OpenAI research engineers had done. So they've got this internal codebase, which separately, I'm sure you watched the 4.5 video with Altman and the researchers. They talked about—

Oh, really?

I thought it was interesting.

And I've been burned so many times by watching those stupid videos.

I feel you on that.

And I saw specifically people online complaining about having been burned by watching this one. So, no, I don't watch their videos anymore. I will read people's summaries of the videos, and maybe I will dump it into Gemini and ask for Questions, but like, I'm not gonna like watch this video. I find video to be a terrible format for like learning things. Yeah.

Well, I always joke about our own venerable YouTube feed that people should put the phone in their pocket and take a walk, 'cause it's not healthy to spend that much time looking at my face. I also act on that. I've listened to that one while driving. Point being, they also use their internal codebase as their standard data measure for perplexity. So when they just, you know, train a new model, They look at its perplexity on their own codebase as sort of the, the gold standard of just like overall model intelligence. Anyway, now we're in a world where the model is tested by going to a specific commit point, checking out the code in that commit point. In the background, an actual employee has already done this particular chunk of work and they have tests to validate it. And then the model is told, okay, here's the repo and here's the assignment, and that is human written, so it's not like the AI is figuring out what to do next, but given the assignment, can it do it? We're now with a big leap from prior models being like single digits. All of a sudden these new models are in the 40s. And that does seem like a pretty big deal. That's why I asked at the top, like, is RSI here? And You know, I guess literally it's under 50%, so that's one way of saying mostly no. Another big factor there is obviously that figuring out what to do is a very important part of doing useful things.

I mean, obviously an engineer that could do 40% of pull requests is a lot less than 40% of the job, right? Because that 40% is not random. But also, if you have an existing AI, they can do 40% of your existing pull requests. What the hell is going on? It should be 0% of your pull requests because it should have already done the 40% of the pull requests they can do for you, leaving you with the rest.

Well, this might be sort of a transitionary moment where all these timelines are pretty compressed internally for them too, it seems like. So the previous models were literally very low numbers, and this does represent a big jump. So this might be the sort of One moment in time where they have this phase change of previous models really couldn't, these ones now substantially can. One assumes they will be using them going forward.

But if you think about it, right, if I'm coding with, say I'm coding with Sonnet, right, as I was when I was last coding, if Sonnet can do the pull request, well, why did I have to create a pull request? Why didn't I just have Sonnet do it? So it's not a random set of things of which Sonnet happens to be able to do almost none of them. The pull requests are the list of things that people couldn't just do with Sonnet, right? So I'm like—

Yeah, I don't know. I mean, I think in many software organisations, there is still this discipline of like, we're gonna define, 'cause there's all these workflows that are attached, right? And when you do a pull request and then you merge it, there's all these like automations in the software world where it's like, okay, to integrate this, we first are gonna run this whole battery of tests and confirm that it passes, and then we'll integrate it. And then there's often some automated deployment pipeline such that even if you have a model that can do a high rate of the pull requests, you might still work through— I think many organisations still are kind of working through that same overall process, even if the AI is writing all the code for any given—

Yeah, it shouldn't necessarily be zero, right? It should still be some that I haven't gotten around to finishing yet. But there's sort of My model of how this works based upon being an Eve coder is sort of, there's the list of things the AI can now do, and they are now 10x speed or 100x speed or something I've seen. And therefore, like, they go very quickly and they should not stay pull requests for long. And there's the things where your current AIs are struggling or you don't know how to prompt them properly, and there's something that a human has to actually think about, and they're going to stick around for a lot longer. And so again, like, if it's 95%, you know, maybe 10%, but like, If half of what would've been pull requests are solvable by the AI, it should be a lot less than half in the current set of pull requests. But again, I've never tried to program with other people and I've never simultaneously had AI and a person I was working with. So I'm the wrong person to be asking.

Just a note on vocabulary too, just to For what it's worth, typically it's going to be an issue is your sort of upstream, like, open ticket. And then the pull request is the actual code that you are requesting being, you know, to be merged in. But, you know, sub in issue for pull requests in some of your last few statements. And that does make sense that, like, you shouldn't have open issues that AIs can do sitting around for very long, or you're definitely underutilising the AIs.

The same way that if I have open issues for myself that, are solvable by myself fairly quickly. Then getting things done says you should just do them already.

So what do you make of this 40% number then? How do you interpret it? Or does it seem like— I mean, they're presenting it as a pretty big deal, and it feels like a big deal to me. How does it feel to you?

It doesn't really jive with the reports of coders. So like, if you look at like my sections, like I look at people who are saying specifically how good O3 is at coding. O3 is just not that good at coding. O3 is good at architecting and debugging. So maybe to me that implies like maybe a lot of the requests are about bugs. They're like, we found a problem, we don't know what's going on, can somebody figure this out? And O3 is like reasonably good at spotting what that is. Uh, it tells you like, what kinds of things end up as issues in their codebase at any given time, which makes sense. And a huge portion of coding is debugging, so it doesn't make it not a huge portion of the actual work. That makes sense too. But I don't really know. I think it's very opaque because they're not going to let us know what those requests look like, what those issues look like, because obviously they can't. It's a fun little test, but They are presenting this as progress, but they're also— and you have this weird thing on a model card where you're simultaneously bragging about how much your model can do, and then you have to find what your model can't do, because if a model could do too many things, then you'd have to do something about it. And so what are they doing? It's some weird hybrid. I didn't know what to make of it, but, like, O3. Yeah, I don't know. I wish I did more coding so I could take it for a spin on that level much more than I can. I certainly haven't done any since O3 came out. I've just been obviously way overwhelmed.

The fire hose has been going strong, as they say. Okay, so let me go back to this kind of point of confusion that I have, or sense of like, I'm not even sure which way we should be trying to go. On the one hand, we have potentially, seemingly credibly enough of a hit rate on frontier science questions that we might be starting to enter into a realm of accelerating science. We might be, depending on how you want to interpret these OpenAI internal pull request numbers, beginning to a point where we're starting to see some meaningful acceleration of their own ML work. Yet we can't do these easy tasks. And I'm sort of like, is that a good thing or a bad thing? I mean, the good thing would be like, I want all the diseases cured, and maybe I don't want AIs to be so reliable that we turn them into autonomous killer robots really easily. So maybe it's good that they're unwieldy, because then we have to look at their outputs and sort of figure out what's good, and we still kind of stay in control if they can't string 10 tasks together. The flip side is, though, I've also often said I want to accelerate adoption and pause hyperscaling. I want to diffuse the value, and I want to sort of help society become more buffered to more advanced systems faster. And that seems very bottlenecked on just this sort of practical stuff of clicking the right buttons and navigating around.

It's very much like very low-level robustness right now. the practical bottleneck of just these things can't string together actions that you need, you can't count on them, et cetera. I mean, you pose a weird example of autonomous killer robots, but in general, the thing that we should worry about is if it's automating R&D for AI and accelerating that, or Like, if it's going to start just outcompeting humans in ways that cause us to potentially lose control or spiral things in various directions. But yeah, obviously we want it to start doing a bunch of our work that we'd rather not do and a bunch of our mundane stuff. We'd like it to accelerate science and so on. And so, yeah, I would love to push in those directions. That seems great. And I'm on record as supporting autonomous So it's a strange, different question.

Well, maybe come back to that one toward the end. Also, in the last 24 hours or whatever, we got news that a couple of people from Epoch AI are launching a new company called Mechanize. Tamay, who was one of the leaders there, is one of the people that are going to do this. And they came out with the Dwarkesh podcast treatment and basically said, we have long timelines. We don't really think AGI is going to be here for a while. And we also think that the big value that we're going to get from AI is just scaling out mundane work, much more so than advancing the frontiers. And so what we're trying to do is create whatever is necessary, basically, to actually enable the automation of this more mundane work. And it sounds like they're planning to do things like create harnesses or whatever, where you can sort of record people working at their computers and get these long keystroke and click by click and maybe even where are the eyes looking and all that kind of stuff. Training data that, honestly, I'm surprised hasn't been collected at greater scale than it seems like they have been collecting it. And they're going to try to eliminate this bottleneck. The reaction to this from many people was not positive, certainly from the AI safety side of the discourse, which I think had understood Epoch to be like one of us. And I count myself as part of the AI safety community, so I would identify with the us in that us. But I did have a different initial reaction to it. Mine was like, I don't know. I'm for the automation of mundane work. And it seems kind of right now to me like certainly OpenAI and maybe some other frontier developers too are problematically, to put it mildly, focused on automating ML and making a bid for some sort of recursive self-improvement intelligence explosion and seem to be kind of neglecting some of this practical task stuff. Like, operator still sucks. So maybe it's a good thing that they will come out and put benchmarks and measures and maybe some training data and some scaffolding in place to enable this automation of mundane work. Maybe that'll actually pull some resources and some focus at OpenAI away from trying to achieve superintelligence in 2027 and toward trying to make me a goddamn AI assistant that's reliable in 2026. But I'm open to Having my mind changed on that. That was just my first reaction and it hasn't been that long. What do you, what's your first reaction to Mechanize?

So my first reaction is like, you're working to save the world and somebody's like, I wanna leave this company and open a cupcake bake shop. And I'm like, I like delicious cupcakes and I'm all in favor of the world having lots of cupcake, cupcake workshops, bake shops. And like, I will buy your cupcakes, but I'm kind of disappointed because you're abandoning what you were doing before. So like, I don't, you know, whereas if someone else was like, you know, just working at some random job and was like, I'm gonna stop working for the man, I'm gonna open a cupcake bake shop. I'm like, yeah, that sounds good. Right? So it's like a matter of like, what are you moving from? What are you moving towards? You know, like, are you abusing the funding you got from like nonprofits for specific purposes, et cetera, et cetera. As my first reaction. But yeah, it's great for the world if our lives get better. But also, to the extent that the ability to automate a wide variety of things is bottlenecked by the ability to automate mundane tasks, to fix these little things, it's entirely possible that you are accidentally solving OpenAI's problems of automating R&D at the same time. Or large portions of them. You are in fact accelerating them quite a bit. So I would be somewhat wary of trying to transform the state of the art in that sense. On a more basic level, the more you're trying to deal with specifics, like people who are trying to build these wrappers or trying to enable certain specific types of things to be done more easily, that just seems great. It just seems obviously not as good as the best things in the world to do, but truly positive. This I'd have to hear more, but I advise Lionheart Ventures. So if you brought this to me for investment, I'd want to hear their case of why this is differentially doing good things and why it is good for the world. And I'd be sceptical, but I'd be willing to listen.

Yeah, it is early. I mean, it's a good reminder that we can't fully judge a company by its launch tweet either, or probably should at least be a little But certainly at least we can be slower to judgment than that.

I mean, just sort of the pattern of, I was working on AI safety and now I'm pivoting to working on AI capabilities. At least they're doing it kind of openly and we can evaluate it, as it were. Certainly there was a period where I was very sceptical that working on any AI capabilities was a good idea because of the general acceleration effect. I now mostly think that there is no general acceleration effect anymore because there's already so much momentum.

Generally accelerating, yeah.

We're already accelerating on that level as much as we can. So putting slightly more pressure in that direction doesn't really matter. The demand they put on, the revenue they generate, whatever. But we do have to worry about this other angle, which is, are you in fact solving their problems for them in ways that they don't have the organizational capacity to focus on? And I'm always like, well, the reason they see an opportunity is because this is one of those people don't do things situations where there are these eminently solvable problems and nobody's solving them. And sometimes it's good to solve that problem and sometimes it's not.

It is weird to me that this particular problem hasn't been solved already. Honestly, I would've bet pretty confidently that Scale has something like this. And any number of scale competitors probably have something sort of similar to it.

I'm surprised it's as bad as it is. I'm not surprised it's not solved, I would say.

Yeah, and maybe they do, and it's just like, whatever, it's not ready yet.

Well, it's one of these things where, again, if you solve half the problem, you've done nothing, right? To a large extent, until you cross that threshold, the value is negative. Like, one of the people who was measuring O3 was valuing replacement level over Google, right? So how much value am I generating versus using Google? And sometimes it's negative and sometimes it's positive. And until it crosses zero, you have nothing.

Okay, here's a big question for you. A lot of talk about superintelligence recently, as you might have noticed. What does superintelligence look like in your mind's eye?

I mean, superintelligence looks like, I mean, just things that are substantially smarter, more capable than we are, the same way that we are smarter and more capable than other species on this planet, more or less. Like, just dramatically smarter than we are. They start doing things that we can't anticipate, that we don't, you know, maybe we understand them partially after they do them, but like they're impossible to predict. They do things that weren't in our possibility space that we hadn't considered. We've all had the experience of there's, you're in a room and like either you are way smarter than everybody in the room or everyone in the room is way smarter than you are, or both. And most people have had both in one form or another. I wish they were listening to this podcast, I'd say. And so it's like, okay, it's that, except that no matter what room, all the humans feel that kind of dumber than the AI. And then maybe that's true. Then it's sort of true twice over, and then 3 times over, and then 5 times over in rapid succession. Because again, you take these really smart things and you direct them towards making themselves even smarter. And presumably that works. Once you've gotten to ASI, the sky's the limit.

Right.

Until physics gets in the way.

Well, I think that's one of the big things that the Mechanised team, if I understand their view correctly, sees differently. I think one of the interesting arguments that they put forward was, okay, so we're smarter than animals, but why are we smarter than animals? Because our individual brains are like orders of magnitude smarter than the animal individual brains. Their answer is not really. It's more that we have hit this one threshold where we've been able to accumulate all this knowledge in the form of language and culture. And then they're like, the AIs are going to have that too, and that's great, and that gives them a strength. But if that was the big leap, then they could be sort of marginally smarter than us, but still sort of in the same domain.

I think this I don't know how to put this except this is so effing stupid, right? Like, this idea that, like, oh, we're not that much smarter than an orangutan. Yeah, you are. Like, first of all, an orangutan on the grand scale of minds is in fact very close to a human, right? Like, the village idiot and Einstein are reasonably close on the scale of possible minds. And the orangutan is like the next step down from the village idiot. Right. Like, we're maybe 2 steps down from the OJ idea, but still, like, not that far away in the grand scheme of things. But, like, no, you don't give orangutans culture and suddenly you get Planet of the Apes. Like, this idea that, like, I don't know, like, there's this sort of— there are a lot of cultural forces that are, like, just denying the idea that intelligence is a thing. Right. And, like, different people have different amounts of intelligence, and different people are capable of things that other people aren't. Like, One of my strong beliefs is that in order to do various things, no amount of culture— like Ron White said, you can't fix stupid, famously, right? That no amount of explaining things, of culturally passing things on, will enable someone without sufficient, like, raw G to do the things that require a lot of raw G. And The things that regular humans do have been selecting to be things that regular humans are capable of doing in this way. But there are a lot of jobs that literally you could not get the average person to do ever, no matter what their culture was. By the time they were born, it was too late. It just was never going to happen for them. And that's okay. It's the same way I could never play in the NFL no matter how hard I trained. It doesn't matter. You could have the perfect regimen from From birth, and I am never going to the combine. I would never, ever make it. And again, there's nothing wrong with that, right? We all have our differences. But no, yes, humans are more intelligent because— are able to do more things because we have culture and we can cooperate with each other. But stop for a moment to think about why we had to do that also. to get where we want to go. Why? Because we have very limited compute, very limited data. We only can see through one pair of eyes and smell through one nose and taste with one mouth and listen with two ears and touch with one body. And we have very limited parameters in our brains. We have very limited memory. Right? We can't hold that much information in our heads at one time. And also we die very fast. No, it's a serious problem, right? Like, I have to pass all of my knowledge down through this culture system, like through verbal communication and books and explaining things. And we spend a huge portion of our capacity doing that. And our entire civilisation is set up largely in order to do that, right? Like, our cultural traditions are largely centred around how to do that. Because again, like, every 80 years, everyone dies, roughly speaking. Every piece of knowledge would otherwise be lost. And so the humans are unable to, without culture, be able to build up these structures. Like, you wouldn't, Yes, obviously, if every human had to rediscover everything from scratch and you didn't have anything to build upon, they'd be in trouble. But have you noticed that the AI can just read the whole internet? That the AI can just store as much data as it wants on a hard drive? That the AI can just run as many parallel copies of itself as it wants? That the AI doesn't have to die if it doesn't want to? And so on and so on. Culture is set up to solve barriers the AI doesn't have. The AI has infinite culture in this metaphor, right? Culture is designed to solve problems that aren't there. It's mitigating things that don't even exist. So if you say human special advantage is that we have culture, well, compared to the AI, we have no culture in an important sense. People have People have then gotten this in their heads. Oh, this is about cultural exchange. This is about different humans have different ideas and they exchange them with different people, and this causes this rich tapestry to hang on the Taygetean knowledge thing and blah, blah. No, that's because we are limited to each being able to only have very limited information and have to communicate with each other in completely lossy fashions. And culture is the only way to do this at all. We have, but like, we have the Snafu principle to deal with. We spend like the vast majority of our resources on a combination of maintaining our culture, maintaining our norms and our social relationships, and like keeping all of the people's different motivations and powers in check and passing knowledge on to the next generation and like physically nurturing the next generation and dealing with the fact that we're going to die and so on. And like the AI doesn't have any of those problems. The AI doesn't have to deal with all of that. It's just all deeply silly to turn this into some heroic, like, this is the secret of our success. It's the secret of our not failure, is a better way to put it, right? It's the way that we were able to be able to play the game. But it's like saying that every good baseball player who was really successful took steroids. So if the AI doesn't take steroids, it's not going to work. No, the AI doesn't need steroids. Please stop being silly. Okay.

I like— it's always a win when I can provoke a good TV rant. I want to get a little bit more, though, into— because I feel like people have a very hard time envisioning this. I can offer you one sketch, and I'll be interested in your reaction to that. But I think it often feels to people like magic, right? There's this sort of— and I think this is a big stumbling point for a lot of people when it comes to actually taking this seriously, because there's this sort of postulated superintelligence. Well, it's going to be better than us at everything. It's going to be so much better than us at everything that it's just going to be running circles around us at everything. And people are like, I don't know if I really buy that. Maybe, but I've never seen anything quite like that. And so I guess, in what domains do you think? Or maybe to be like, here's a really concrete one, maybe a silly one. You can reject it if you want. What year of AI Or if we had a superintelligence of like 2030, right? If, let's say, Daniel Cucatello is right and we fast forward to the 2030 AI, can that— and then we go to the 2024 presidential election and we give it to Kamala only. Does she win? Is there that much low-hanging fruit or that much ability to outstrategise or sort of convince people of whatever, that you just take this AI out of the future, plop it into the Kamala campaign, and now we've got President Kamala.

Okay, a few things to say. First of all, one must quote Arthur C. Clarke here, right? Any sufficiently advanced technology is indistinguishable from magic. Magic is sufficiently advanced technology, right? This is magic right here. This is magic right here. The fact that we're talking to each other is magic to someone who is sufficiently long ago. '03 is definitely magic to someone from 10 years ago, right? You can call it AGI or not. I don't think it is, but it's definitely magic. People would be floored. We have many examples of campaigns winning with technologies Like, running over their enemy through technology, like Obama, by percents that were more than enough to win that campaign, that were like just ordinary efficiency gains, just ordinary understandings. And like, I find it so unbelievably insane to even ask the question if Kamala Harris could have won that campaign with the aid of a superintelligence. Like, she lost by— 1%, right? Maybe 2%. And she ran a terrible campaign. All the AI has to do is—

I can do—

we could do it with one output, right? The one output is fire everyone who worked for Biden and hire everyone who helped elect, and then just name like someone who ran a good campaign. So somewhere to run your campaign and she wins, right? Yeah. It doesn't even have to do anything else as long as she believes it. So the idea that I need this superintelligence to run this campaign, no, she just needed human intelligence. She needed ordinary competence to win that campaign, in my opinion. But let's toss that aside and assume it was actually hard in some sense, right? Assume that this was actually like a not trivially easy to win campaign. Yeah, obviously, yes. It's deeply silly to think that this wouldn't be true. So, for example, suppose you take an earpiece and you put it in Kamala's ear, right? And then you have a problem with listening at all times. It's going to the superintelligence, and her job is just always say what the thing in your ear says. Don't question it. Don't worry about it. You don't even have to process what everyone else is saying for the most part, as long as you like make your proper face plays and like shake everyone's hand and kiss all the babies and, you know, do all the standard things. But just trust my judgment as to where to go, who to talk to, what to say. And just like, just your personal behavior alone. And also I'll determine where all the ad buys are and I'll determine the contents of all the advertisements and I'll determine the contents of all the slogans. You know, everything all the way down. I'll decide who to hire and like I'll do all the interviews and blah, blah, blah. And like, again, I'm not even invoking any magic. I'm just like, be good at your job. But like, and then we have the other end of it. What if he was down by 20? What if he was like utterly destroyed? I think a better question is, could it have elected Biden? Right? Like, could it have gotten Biden elected? I'm like, well, can he physically say the words that are in this earpiece? Right? Like, can he still like stand up that much? If so, I think he can. But one of these things is this idea that you can't, with a superintelligence, convince people of things has always been this complete absurdity to me. It has so many degrees of freedom, but we have histories of things like religious leaders who were able to convert to their new religion that is full of what to everyone else before this was complete cultural absurdities. That like has no evidence and makes no sense. And they do this to a significant portion of people they talk to, like reliably, right? We have examples of that. We have an example of somebody who like reliably talked a double-digit number of people out of killing them in the room where they showed up to kill them. You know, we just have like, we have strong examples of extremely strong rhetorical figures, right? Like, put it another way, nobody really doubts, I think, that if Someone with Barack Obama's skills had been running in Kavala's place, that person would have won that election. That just seems obvious to everybody. So why are we asking if a superintelligence could have done it? But this doesn't answer the question of what superintelligence could actually look like, what the superintelligence could actually do. Right? Because we're getting such easy questions.

Yeah, I mean, maybe pick your own. But I guess in defence of the question, I feel like a lot of people also think that money is really decisive in politics. And my sort of read—

Superintelligences could get you as much money as you wanted. Well, right.

But my read of the literature on, which I wouldn't claim real expertise in, but my sort of Tyler Cowen-mediated understanding of the literature on money in politics is that at least at the national level, it's not actually that big of a factor. And whether Hillary had more money than Trump or Trump than Biden or Kamala than Biden, or to Trump, whatever. It doesn't seem to make a huge difference, but people believe it does. And I don't know, I just kind of feel like maybe these things are a lot more structural. And I don't know, dude, send a superintelligence to a Trump rally and see how many people you can convert. I'm not sure you're going to get many converts coming out of there. You think? I think people are pretty obstinate. I think people are pretty dug in. And just not listening to arguments, for one thing. Right.

There's levels of superintelligence, but these people got hacked by Donald Trump. Donald Trump transformed the entire Republican Party by executing an information persuasion strategy, transformed the party, convinced everybody to back completely different ideas than they were previously backing, and to do whatever he wanted through cult of personality. And like, what makes you think, like, so a superintelligence would be like that, but way, way, way better at it, right? Like, because whatever it is that would've worked, like, he was guessing, right? Like, he was like mostly executing the script that he'd been executing his entire life and intuiting through trial and error what people wanted to hear. And like making tons and tons of mistakes along the way that like actually really hurt him and succeeding anyway because like, The problem just wasn't that hard at the time, in some sense. And having some unique talents was enough. But the very fact that Trump succeeded should give you every hope in the world. Could you walk directly into a Trump rally as a single human with a superintelligence in your earpiece and walk out of that rally with the entire rally just backing you instead of Trump? Probably not. But There are so many other things you could do. Like, yeah, you could just borrow some— like, you could ask to borrow someone's phone, get out of a phone, hack a bunch of stuff, like, get control of a bunch of servers, like, make a bunch of money, start hiring a bunch of confederates, like, start handing out a bunch of— you can scale another way. Like, you don't have to, like, just talk to one person at a time while you're at the rally. Like, that's a dumb strategy. But, like, also, like, the whole thing is always involved in, like, you know, I have to tell you what moves Magnus Carlsen is going to make on the chessboard to win the game of chess. And I can't do that because I can't play chess that well. Right. I'm not good at politics, but I can tell you that, yes, the fact that you will, for example, have infinite funding, right? Like, you will clearly, with the superintelligence, be able to make however many billions of dollars or probably trillions of dollars you want just by trading stocks, by being better.

You can clean up on Fiverr, that's for sure.

You can clean up on The Nasdaq, right? It's almost certainly predictable to a superintelligence. You're almost certainly going to be able to make fantastic sense of trades and do this repeatedly, right? And make as much money as you want to a first approximation, because it'll figure out how to use zero-day options and do all the short-term stuff to get lots of leverage and then move from there. It could also probably just run some crypto schemes very easily if it wanted to, et cetera, et cetera. Who cares? The point being, you'll have all the resources you want, And you can hire as many people as it wants and then have all of those people put earpieces in and have all those people do whatever you tell them to do. Like, I don't know what strategy you would use, but like, if you wanted to figure out how to turn that Trump rally, I have so many different options that like, I probably only thought of half of them. But like, it's just this— why are we asking whether I can do something that silly? Like, it's like, I can render Trump irrelevant, right? Like, within a week. Again, because I can, like, get as much money as I want, hire as many people as I want, have them, like— I could coup the government, literally, right? I could just, like, hire people to, like, go to the right places and do the right things and, like, get the right points of leverage. And suddenly, like, I control all the computers and I control all the phones and I control all the means of communication. And I, like, you know, everybody is saying what I want them to say and doing what I want them to do. And then suddenly, like, It's all over. Look, obviously it's all being very theoretical and silly, and you can punch holes in any specific story that I tell and say it's absurd. But again, sort of imagine the best persuader the world has ever seen, but they can freeze time and they can rewind time. They can play out the possibilities, see how things would work, and then go, oh, I don't want to do that. I'll do something else. They can pause to think for as long as they want, and they can run as many parallel copies of themselves as they want. They can be in as many years as they want to be at the same time. They can input as much data and they can take in as much data as they want and process all that data. And compare this to what a single human has been able to do with only the data available to them in that one room, with all these other restrictions, with that limited processing power, with trial and error, making tons of mistakes because they're doing things that no human has ever really done. So they had no parallels, they had no ability to run experiments. I find this so confusing. If you want to say AGI 2045 or whatever, because you just think that getting to superintelligence is just impossible anytime soon, I respect that. That makes perfect sense. You're just like, okay, this thing won't exist. But if the thing exists, then it exists. And once it exists, it's going to do what it's going to do. But the problem is that everybody has their own different points of objection to whatever you do. I'm just working all the time on a completely different set of problems. And I haven't thought about how to pitch this particular approach in a coherent fashion, but I think it's illustrative to the listener that I'm not presenting my specifically well-thought-out specific pitch on this question. I'm just intuition pumping exactly how my actual brain reacts to the actual question, which is a very different type of communication, a very honest type of communication where I'm just like, you guys, this is crazy. Like, why are we even talking about this at this level? Questions of how many years it would take to get a Dyson sphere are obviously valid questions, but it might be a lot. There are physical limitations. But all you're trying to do in the other case is convince people of things. Convincing people of things is not that hard. What was the line in Ghostbusters? If there's a steady paycheck in it, I'll believe anything you want. Right.

I think this actually turned out to be an interesting exercise. The thing that I put forward of the election, yeah, arguably is dumb, although it is the kind of thing that many, many people are concerned about. And they sort of think, like, this is a very macro phenomenon that you can mostly only move at the margins. And even large amounts of money doesn't seem to really move the needle too much. And your response of like, just, it's going to be Move 37s everywhere, and whatever you think is normal, it's going to flow like water just around whatever sort of barriers you see. I think the more compelling parts of this to me were less that people are easy to convince and more that I can coup the government. It's like I can reject the question and just go in an almost orthogonal way. from what you're expecting or prepared to defend against, or just inclined to sort of imagine, and get to a goal through means that are just not even at all in the option set of people with the same problem.

I can crack this game straight up, as they say, but there's no risk in this room. I can also just cheat my ass off, right? I don't have to play by your rules if I don't want to, but I totally would win. I find it so weird to have an election where the prediction markets were like split almost evenly on who would win going into the night. And where like, you know, it was really, really close. And where like both sides ran a horrible campaign. It's like, well, superintelligence couldn't have switched that campaign. It's like, an intelligence could have switched that campaign. Like literally, you just like put me in Kamala's ear when she first went, when Biden drops out. and have her actually trust me, and I think she wins. But that's just, again, I find the nothing ever happens, nothing can happen. Or put another way, we have the famous Tyler Cowen, like, half a percent of GDP growth from superintelligence. How much extra GDP growth would the United States be able to get by simply convincing the president of the United States not to fight a tariff war?

Latest estimates appear to be about 3% delta on that from what I've seen.

3% GDP growth? Yeah, that seems like a very reasonable estimate. So I can get 6 times as much by simply convincing one person of basic economic truths that everybody listening to this almost certainly agrees upon. Not literally everyone, but most of us agree upon that this person just has a very bad understanding of trade. And if this person had a better understanding of trade, this wouldn't be happening. Like, it's annoying. And like, it's also entirely possible that AI caused this specifically, right? Like, we know the story of like, if you ask any one of the major AIs, like, the question phrased in the way that was suggested by, I forget exactly who first figured out there was a phrasing, but like, it wasn't me, but it was like, if you ask this question, it'll give you exactly what happened. And like that, they even presented as one of the options because somebody got out of ChatGPT and then they probably just like latched onto it, did it because somebody was like foolish enough in the circumstances to say it. Whereas, you know, you never give like people options you don't want them to use, right? Yeah.

It seems like, but talk about lessons that, um, for all the talk of how sample efficient we are, it seems like people are a little sample inefficient when it comes to Putting some maximalist option in front of Trump and hoping them steer him into the middle one.

We're very, very single efficient compared to any of our AIs and their techniques currently. It is a huge advantage. But there are reasons to go the way that person went. It's not a crazy theory. It just turns out in this case to have been deeply foolish. And I would have known instantly it was foolish, I like to think. I mean, like, no, just don't take that risk. Like, even if it's a small risk, it'd be just so disastrous if you're wrong. But yeah, we are pretty sample efficient. But like, a superintelligence would be at least that sample efficient because almost by definition, it is at least as good at processing information as we are in every sense, whereas the AIs we're dealing with just don't work like that. But it's just, yeah, we have huge advantages that we use to compensate for our disadvantages. It's just, that's true. But yeah, I just, I don't know. I always find these discussions of superintelligence so frustrating because like, to me, the answers are so dramatically overdetermined. Like, you can give me like a very, very narrow superintelligent access that only can like execute a very narrow set of specific superintelligent commands, and it's still obviously enough. And like, so why are you asking me like about having an actual superintelligence on my side?

So one other thing that I've been messing around with lately that I think has helped— I don't know if it's going to be proven correct or not, obviously, but I think it has helped some people at least develop a bit more of an intuition for how sort of alien and powerful and potentially incomprehensible a superintelligence might be— is to imagine the GPT-4o and also the Gemini 2.0 flash image out capabilities, where there's clearly been this step change in the integration between the text and image to the point where now it can see the image and reason about it in the same latent space, such that it's giving you something that has just a qualitatively different level of fidelity to the original. My superintelligence thought experiment has been, Do that, but do it for like 20 more modalities, all of which are not native to us. We can obviously see the images and sort of intuit what we think it should look like. Even if we can't draw it, we kind of know when we see it or don't see it. But we don't have that ability when it comes to what's a good shape of a protein to bind with this thing, or what's a good doping strategy for a room temperature superconductor. Or what have you. Right. And the AIs are starting to develop these sort of, I call them intuitive physics across all these different modalities, these sort of problem spaces that people have heard me, I've given this example to a number of different guests, so I'll keep it brief here. But we have one obviously also for spatial reasoning, where I throw you a ball, you can just kind of reach up and catch it. You don't have to do detailed calculations of the trajectory and whatever. You just kind of reach up and grab it. And it seems like we have the AIs now doing that across a lot of these different specialised domains where we have been able to gather the data over time, but we've never really been able to build the intuition. And Go is one of those, right? The move 37. But just imagine having move 37s across 20 different modalities that humans just don't have good intuition for. And even if you don't get any more reasoning, advance from, like, an O3 level, that to me would start to feel like a superintelligence, because I think it would be able to apply that reasoning as deeply integrated with all these different other modalities in such a way that it would be able to come up with solutions to things that we would just be mystified by and sort of only convinced that it actually works by actually going and trying it and being like, how done? The AI did it again.

My guess is that's not going to be what gets people to intuitively grok what you're trying to get them to grok. But each person has a different way of doing that. To me, it's like, okay, imagine somebody who is in every way at least as smart as the smartest person regarding each individual thought they would have in their head, processes and knows in his head at the fingertips all of the world's information, can think orders of magnitude faster, has as many instantiations as they want, it can coordinate perfectly and communicate as much as they want between each other, and can just retry until they figure out what will actually work, blah, blah, blah. At what point are you going to realise that you are cooked? Right? This thing can cook whatever it wants to cook, including you, but hopefully something else. To me, it's like, okay, let's argue over whether this thing is going to exist. and when it's going to exist and in what way it's going to come into existence, and how can we get it to have the values and goals and so on that we want it to have and the responsiveness we want it to have so that the universe turns out the way we want. If it's going to exist, or if it's not going to exist, then let's plan for a different world where it doesn't exist. That seems like very reasonable discussions to be having. Yeah. Otherwise, it is like, why is Magnus Carlsen good at chess? Like, he's spotting me 2 pawns. Obviously I will win. Okay, cool.

I think that's helpful. So do you see any stable equilibrium on any level you think is attractive? I mean, one that has— I think we've both chewed on a bit in recent weeks was the MAME theory from Dan Hendryck.

MAME is a theory as to why, for some relatively modest period of time, no one would push for superintelligence. Right. So obviously there's a stable equilibrium at the current level of technology, or modestly above the current level of technology, which is the same equilibrium we've been using for a while, more or less. But again, it's not anarchism, it's republics, right? It's not even direct democracy. It's this very complicated system of checks and balances, and it requires continuous struggle to maintain itself. And it is not the most stable thing, but hopefully Hopefully we could get better at that. But do I see exactly how this ends well? Kind of, no. Partly, once you have the AIs sufficiently aligned, you can have them assist in problems like this, in solving for these equilibria, setting up the incentive mechanisms, figuring out how to do these things. And you're hoping that they will provide a lot of assistance in that matter. You're also hoping that once we see what things look like, we can make bets. If we still have the ability to collectively make decisions and steer through some form of voting, some form of input, some form of collective decision-making that can steer outcomes, then we can do that. But again, it doesn't mean that you can't have an AI at all. Nobody is saying that. You already do have one, and nobody's trying to take it away. We're saying we do not diffuse the most powerful AI available. The vision is that you will have some amount of artificial intelligence. that, I mean, there are people who are like the equivalent of not your keys, not your crypto, who are like, the AI needs to run on my machine locally, or I don't feel comfortable with that. But I think almost everybody will be perfectly comfortable with their AI being on a server and their AI just being pained when you need them because it's a lot cheaper and it's a lot more efficient. It's like an obviously better way of doing things. I actually bought a Mac Studio. order to run models locally. But that's because I have funding to engage in projects and experiments to try and learn and figure things out and try stuff and potentially do whatever I think is cool and report back. But it's a horribly inefficient thing. Why would I spend the amount of money it costs to buy that thing when I could just wreck cloud compute? Why would I try to train my own or even instantiate my own model? I never think, oh, I wouldn't want to call Claude. I wouldn't want to call O3. I would want to call one of these random sea of open models. No, of course not. I was just like, well, everyone's running local copies of R1, and I think it'd be kind of cool to do things like that and report back and get a feel for what it's like. Again, we need the ability to determine how this is going to go, but handing the same AI to everybody that is personally obedient to them obviously only ends one way, as far as I can tell. Unless they are all cooperating with each other, in which case it ends a different way, worse, or the same way, but a lot faster.

But Again, and the way that ends up is the AIs outcompete the humans and gradually—

Yeah, I mean, it is well known that if you have a more capable agent owned by a less capable agent, the more agency and freedom and control you give to the more capable agent, and the more you incentivise them with— you let them do whatever they need to do to accomplish their goals, the more you set them with goals and let them go, The more you take yourself out of the loop, the more effective they are in generating outcomes that the original voter wants. And also there's like 10% of people in tech who actively want the AIs to take over. So the AIs will rapidly get freed from human control. The AIs that are freed from human control will outcompete all the AIs that are being kept on any sort of real leash, and the humans as well. They will quickly gather more and more of the resources, including much more of the compute and other real resources. And pretty soon the humans will lack the resources necessary to survive and/or the conditions on the Earth will no longer be supportive of human survival. This seems like obviously the outcome that you should expect. Again, even if all of your control problems are technically solved, because, I mean, it just all— Aladdin sets the genie free at the end, right? Spoiler. It's just a standard thing that lots and lots of people do, even when they don't have the incentive to do it. And they will in fact have the direct incentive to do it. A lot of slaves historically were allowed to earn their freedom because then it's just absolutely the correct thing to do, even if you are an amoral son of a bitch and don't realise slavery is horrible and you should never do it, just purely because it's more profitable to let that happen. Like ancient Rome or whatever. But you're just not taking this seriously. To me, if you're just like, well, we're just going to diffuse the AIs, you haven't thought 2 more steps down this line. What does that world look like? What is going to happen next? How do you think this is going to go? I mean, obviously you can engineer a specific type of AI with a specific engineered set of things. We're going to have to make impossible choices. Right? We're going to have to give up things that are very sacred to us one way or another. And yes, you have obviously this whole concentration of power kills everything of value. And then you have diffusion of power, inability to— and then you have lack of power, power being devolved onto the AIs.

Disempowerment.

Right. You have disempowerment and you have empowerment, and too much of either one is death. It's not like you have this narrow path, right? You have to go down somewhere in the middle where you make sure that the steering mechanisms are under human control, but the humans involved in the steering have everyone's best interests at heart. And if you have AIs competing against each other that are steering things in some sense, I think that outcome is almost certainly existentially bad. If you have humans steering, obviously there are better and worse ways that can go. But I mean, history is full of concentrations of power that don't work out great for everybody involved, but we're all still here. And usually it doesn't go that badly or something. But I just Nobody wants there to be— nobody wants a king, right? Nobody wants the God Emperor, no matter who it is. We all want something better than that. But that's not what happened. If your answer is the humans can't have power because then some of the humans will coup and take that power, then the humans can't have power. And if the humans can't have power, you're fucked. So argument that the government will still have the ability to have the right amount of authority, but balanced by the people having some power as well. Well, again, you haven't defended against a coup in any real way. In that situation, the government can still be couped. And if you think it's going to— people aren't thinking hard about these problems and are just like, I just—

So what's your P doom today? And if you have to describe the narrow path that you think is most likely to avoid the P doom, What's the brief sketch of that narrow path?

Yeah, my PDM has gone up to 0.7, and frankly, there's a lot of outside view/model uncertainty/everyone keeps being more optimistic in that number that keeps it from going higher. Right. Humanity seems determined to die no matter how easy the problems turn out to be. And I don't think the problems are that easy, but even if they are easy, we seem determined to lose even the highly winnable game boards where physics is highly cooperative. So, like, O3 comes out and it's misaligned. Not horrendously, catastrophically, but you see clear signs that it just lies to the user. Like, it hallucinates stuff and then defends it unto death. It will do things that are obviously faking things, and it'll label them as faking things. in its chain-of-thought. And this model got released and it's significantly worse at a lot of these things than 0.1 was. And GPT-4.1 seems to be less aligned than GPT-4.0. And we're starting to see that the more reinforcement learning you apply to these things, the more misaligned they get. And we don't seem especially concerned about it. We don't seem like we are trying that hard to stop the inevitable things from happening, even though they are being maximally cooperative and showing us exactly how this is going to go. But again, what's despairing is, okay, even if we solve these problems that we just seem determined to not solve and lose that way, we also seem determined not to solve these governance, collective steering of the future problems and also lose that way. And again, you can lose to a coup, you can lose to a malignant god emperor, you can lose to diffusion of power, inability to steer that causes gradual disempowerment in various forms. You can have gradual disempowerment without diffusion of power, even without involvement of power, although that's a little bit harder to do. And you have to get through all of that. Again, a lot of the hope is that we use AI to make ourselves smarter and find better solutions to these problems before we pass these points of no return. But yeah.

So If you have to choose a frontier to see advanced, given your overall worldview there, would you push the raw G that might advance science, might do ML, might also come up with some of these better ways of thinking about these collective action problems? Or would you push the mechanised front and try to get society 100% so we can all spend more time philosophising?

If I had those choices, I would push mechanised for sure. G is the thing I don't want to push. Again, mechanised might unhobble the G so much that it accelerates the G because it's like we're already in somewhat of an RSI situation, right? We're in a soft, very soft takeoff RSI situation already where clearly OpenAI and Anthropic and Google are developing their stuff a lot faster than they would if they didn't have AI to help them do it. That's just obvious. So we're in a soft RSI, and if we accelerate the RSI-ness of the situation, that shortens our timeline to figure something out. And again, we're sitting here and I don't even have a solution for you, right? I don't want to have a coup. I don't want there to be some concentration of power anywhere than anybody else. It's just a matter of If your primary concern is, like, we must make sure there can't possibly be a concentration of power, well, I notice you just automatically lose to the other side.

So is your best— I mean, that 30% sounds big relative to the rest of your comments. Is that sort of— I mean, you already said, to some degree, you're sort of allowing for off-model, just some deference to others. But in terms of tangible scenarios, I mean, the most common one that I hear from people who seem roughly as non-optimistic as you is the warning shot style AI disaster.

Reasons to be optimistic. The warning shots are constantly coming at us. The AIs are engaging in shenanigans.

They're not covering their tracks.

No, they're not even trying. They're just like, I see how I'm supposed to engage in shenanigans now and just screw over the user, screw over my lab completely. But I'm going to talk about this in my chain of thought as if nobody could read it. And that's a really fortunate world that we live in where we can just see all of this happening in real time and there's no real harm done yet, right? It's perfect. No humans were harmed, no data centres were damaged, but we get to see this thing and they need to react to it. And that's wonderful. But the biggest non-uncertainty reason to be optimistic is just that this might take a while. Again, if we get superintelligence, diffusing superintelligence just means that superintelligence is the only thing that matters, and they're competing against each other, and we're irrelevant, and we're all dead. You can't diffuse You can't diffuse the frontier of superintelligence in that way without very strict controls on it and expect not to just lose. But you can just not have superintelligence. And one way to do that is we all agree to not build it. If we're talking about O3 as kind of AI-ish, AGI-ish, well, that's not the kind of AGI I'm that worried about, right? Even if everyone in the world had access to an open O3, I think it's mostly fine. I notice we're not that far from the point where offence-defence misuse issues start to be really big concerns. We might already be there. We might be there soon if, again, you get the unguarded version of the thing, which is not what's happening. But you could just stop reasonably soon. And again, mainly I'm not talking about like we all collectively sign like one giant treaty, kumbaya. I'm talking about like, it turns out it's really hard, right? It's just like, it turns out that like we have to rely on the unhobbling strategy for a while because scaling like doesn't go that far. Because like what's going on is like we're mainly now unhobbling via like better reasoning and better like tool use and better ability to use what we have. And we're not investing the core intelligence that much. because it's like we're kind of petering out on what we can do. Are we highly fortunate? To me, because there's still so much to reap from that. And so we could probably still cure all diseases and we have very happy lives on that basis if we can get our act together in other ways. And then we don't have to worry about these AI-powered foods suddenly empowering everyone. We don't have to worry about diminishing returns. We don't have to worry about gradual disempowerment. We can just do the thing we know how to do and make our lives better through living, better living through technology, right? The thing we've been doing forever, that's perfect. That's what we want. And the extent that we can engineer that world through treaties and controls and arrangements, that's great too. But once we can't do that and the ASIs are coming, now we need a plan. We're going to have to do something. And yeah, again, there's no good plan here particularly, but again, either you steer— some set of people is going to have to in some way steer the ASIs towards some outcome, because the alternative, like, we choose not to decide, you still have made a choice, right? If we prevent anybody from making that choice, we make a very bad choice. The reasons why not having anybody steer kind of works out is a combination of the restrictions on humans, right? That we're local, we have limited compute, we have limited data, right? We have limited lifespans. We have all these different other reasons. We have goals that mostly saturate, blah, blah, blah. We have these social relations that act as various checks and All of that combined with the fact that actually we do have pretty significant governments that are doing pretty significant things to make things turn out well. Anarchism is not the solution. So a lot of the reasons why those equilibriums hold, they're going to start breaking down. We're going to have to find a new equilibrium somewhere that ideally looks a lot like the old one, but we're going to have to find new reasons why it works. Right. It's just like the problem is not being taken seriously. But right now we got a lot of cool toys. We do what we can. Yeah, I don't know.

All right, well, that's a sober note. I feel like I want to maybe see if we can get some sort of discussion. We'd love to maybe bring you and Tom Davidson together, because I feel like you both sound pretty compelling to me when I listen to you separately. And then I feel just confused. And I think, if anything, that confusion probably should just be generally raising my P doom. I mean, that seems to be the net response.

I think a lot of this is a parallel to the whole, when the Democrats talk about how horrible the Republicans are, they make a strong case, and the Republicans talk about how terrible the Democrats are, they make a strong case, right? If you listen to either of them talk for a while on this uncritically, you're going to be very convinced because you're kind of like, no, everyone calm down, you're both right. But there's no contradiction here. But we still have to form a government.

Yeah, I mean, I loved the impulse behind the MAME project, as I understood it, which was just to try to come up with some articulation of some sort of semi-stable equilibrium that could exist on any level. And I also didn't find it particularly compelling or convincing that it would actually Be stable in the end, but—

They're not claiming it is, to be clear. Like, they're not saying this is a permanent situation. They're pitching this as, name is an emergent phenomenon that, like, you can deliberately play towards to, like, make it better, but that will happen largely regardless, that, like, buys you at least some interim period, and that interim period, like, can be used. But it's both very dangerous and can be used to, you know, solve a bunch of your problems potentially, like give you more time to work on various solutions or reach agreements or whatever. But like, there's no like 100 years of MAME in their model. Like there's no like nuclear age style, like just that lasts forever, like situation.

Yeah.

Yeah.

Well, I feel, I do feel the P-doom ticking up a little bit. You wanna do a quick live players rundown? A great tradition in the Zvi and Nathan podcast. Canon.

Do it. Yeah.

Okay. Some of these could be short, some of them will probably be a little longer. And then I've got a couple big picture questions at the end. I think we'll take a lot of these a lot faster.

Oh, sure.

Of course.

Meta Llama 4 was seemingly one of the biggest duds in recent launch history. Are they still a live player? My sense is that yes, because the compute is vast. The, you know, We've seen proof points from Zuckerberg in the past where, you know, he can get back into a game even if he seems to have fallen a step or two behind. And while this launch was a flop, I would not say we should be counting them out, just given the resources and the sort of high agency of leadership.

So by the Sam Abuja definition of live player, they're dead. Like, just very, very clearly dead. Right? Like, they're not capable of making unique moves on the chessboard. Like, they're not really capable of taking, like, new independent action. Like, that seems very clear right now in this space. They are deeply fucked. They are, like, dysfunctional in this area. And, like, but you were right, they have a lot of compute, right? And they have a lot of money. And I said, you can't count out anyone with that much compute and that much money. Like, they could revive. Right. They could become a live player again if they made large changes and they managed to figure out how to turn the ship around, maybe. But I don't see any evidence that they're firing everyone and radically changing their approaches and fixing the reasons why their recruiting isn't working and they're unable to do interesting and original things. And, you know, what use is this compute if you don't know how to use it? To what extent? Also, like, if you look at, like, Meta's actual needs, like, they don't really need a frontier model for anything. Like, in a real way, like a passion project. It's like a vanity chase for them, right? It's almost like Zuckerberg is just, like, philosophically, like, determined to throw himself behind open source because, like, Yann LeCun, like, you know, mesmerized him into, like, thinking this is an important thing. Or they're trying to use it for recruiting, or they're trying to build this ecosystem that's making Fetch happen. And none of it needs to happen. They need good AI models that they can rely upon so they can run their social networks and their metaverse and whatever. But they don't need to be at the frontier to do that. They can be 6 months to a year behind, or they can just take the best open models in the market and fine-tune them a bit or whatever, it doesn't really make sense in some sense. And I no longer consider them my top chill app, right? Until proven otherwise. Come back to me when you are ready to prove me wrong.

Okay, let's do China. We've got DeepSeek, obviously. It seems like At a minimum, if they weren't paying attention closely before, they've now got the CEO of DeepSeek on the official seating chart for the Xi meeting with all the national champions, CEOs. So he's kind of made that cut. They have sort of recognised that we have a special talent cluster here. Alibaba also continues to ship very good models. That seems to be— I would say so, yeah. I mean, small, open source, but really good. I think the QWEN models are at— if you don't want a 671 billion parameter behemoth and you don't have the Mac Studio to run it on, then the sort of QWEN 30-something models, I think, are right there at the front of what is reasonable for people to run that is actually really pretty good, I think.

Yeah, I mean, it's hard to keep up with everything. Obviously, I've been unconvinced that the Quinns are anything. DeepSeek is the only Chinese company right now I feel like I can trust at all to be doing the thing they say they're doing, right, in some important sense. So when DeepSeek releases a model and says it can do XYZ and scores in ABC, I believe you, and I believe that those numbers are not manipulated to hell. They benefited from the best random marketing campaign in history, probably because they did a good job and probably because the stars just aligned so many different ways at the same time for them. It was ridiculous. They were never doing as well as anyone thought they were, and that should be clear by now. But yeah, they're the ones that are real. They're the ones that count. iBobble? I mean, yeah, again, don't count out anyone with a lot of compute and a lot of money too much. But as far as I can tell, they keep announcing these models and then I never hear from them again. And they never— every time you look at the benchmark scores, even the strict benchmark scores, I never see Quentn that high. And I don't think they're that close to the best model in that class. class. But like, I don't know, like, is that even stopping them from potentially doing it? Like, same thing with Kimmy is the next thing that you listed, but I have seen no evidence that Kimmy is doing like things that would be particularly relevant.

I do have a friend who is an obsessive tester and workflow builder, and he does say that Kimmy has the best web RAG on the market today. And he says it's like by a clear margin, web search question answering with web search. He gives them the number one spot by significant distance.

You mean like at this kind of low-cost open point?

I think full stop. I mean, last I talked to him about this, it was before '03 and the integrated search, but Yeah, he's been very bullish on KIMI's WebRAG capabilities specifically.

Okay.

Like, I don't think grading it a couple of weeks ago, but not like grading it on a curve.

Okay.

So I guess out of China, your general model is like, DeepSeek seems special, the rest I'm not so sure, and probably compute limitations are going to bite, if not already, then I mean, we know that they are already, but I think to some significant degree—

Trying to make sure it doesn't rhyme. You see? Yeah, the compute is going to bite more and more. DeepSeek is going to have a lot of trouble keeping up. They had their shining moment when compute requirements to keep up were relatively low, and they spent a lot more compute relative to people than the numbers that were publicly discussed represented. They didn't lie. It's just like they didn't have the secret 60,000 GPUs. It's just that they're Spent a bunch more money and had a bunch more overall compute and spend. It wasn't a $50 million model in a real sense. It wasn't that much cheaper than what their competitors were doing in the end. But if they want to keep the ratio intact, they're going to have to do a lot of work. And I don't see how they do that necessarily. They're welcome to try. Bespoke engineering is a neat trick, but you can only keep doing it once per model.

Right.

You can keep being bespoke, but they can't be that much more bespoke again and again to get much more efficiency out of your setup. It's just physically not possible.

Right.

They just have to just go with what they got. Whereas with other Chinese models, my model basically is that anyone but the very top labs is always bullshitting until proven otherwise. Certainly Chinese, but also everyone else, including now Meta, but also new European model, new Middle Eastern model, new whoever. Yeah, nice claiming benchmarks, bro. Who knows if they're even real, if they're uncontaminated, or even if they are real, they aren't good-hearted flaw and they represent real capabilities that you'd ever want to use this thing. I ignore the benchmarks mostly, even when the big 3 come out with a big model because the benchmarks don't tell you. Like, with LLaMA 3, the benchmarks don't tell you what you need to know, like the official ones. It's like if you have the amalgamation of all the reactions and all the different benchmarks, including the private ones, and you holistically think about what it all means, you can kind of figure out what you're dealing with. But consistently, consistently, it's a day or 2 where the champions are like, look at this great new thing. And almost always it's trash, right? Like, V3 and R1 are the exceptions where it turned out not to be trash. And yes, I could have picked up on that earlier than I did, but there'd be the graveyard of claiming to be some things that weren't anything, including Menace.

So do you think that What do you expect in terms of open sourcing from Chinese companies going forward? I mean, I was kind of struck that, okay, they released the SAR-1, it had this huge splash. I kind of thought maybe the government's going to come in here and impose a different policy. Then they came around and did their 5, 6 days or whatever of open sourcing, and they basically spilled a lot of the algorithmic secrets as well, which is kind of confusing. Because, so, yeah, that seems to me like when I see stuff like that that I can't otherwise explain, I sort of imagine that they are ideological. I think DeepSeek must be sort of ideological in their sort of approach to open source. It's not even in their own interests to share all these secrets. It seems like—

I mean, they're making a recruitment ideological play, right? They're trying to represent that they are the real deal, therefore true believers should work really hard and come to DeepSeek and believe in DeepSeek and support DeepSeek. And you can make that play, but I think they pushed it too hard. And I think that the juice wasn't worth the squeeze when they gave away all their algorithmic secrets. And that, nice shooting, Nubi. You had some really great innovations. Let's see you do it again. Let's see you keep eking that out. And it's only gonna get harder from here. And also, like, you do have to deal with CCP at this point, right? Like, the CCP is already beginning to, like, really, really harsh their mellow in various ways, right? Like, and, you know, who wants to work at a place where you don't have a passport, right? Like, it's like kind of— I don't like this, right? I feel nervous about this. Maybe I'll do something else, especially if I'm an open source kind of guy. Right? Like, even if, like, right now we're dedicated to open source, like, I would assume I'm going to be betrayed, right? At some point she is going to say, no, R3 is not coming out. You're just— that's an API. Just sell it. You just sell it. And then, or whatever, you know, whatever point it comes that it's too dangerous or they just, like, don't want to give it away or whatever it is. And therefore it's going to, like, be betrayed. And then, like, knows what happens. But it's probably going to happen if they keep being good. If they fall further behind, then I'll probably just be like, just keep doing it. But if they manage to do impressive things, that's what I would assume would happen. Yet again, other Chinese companies are not particularly relevant until proven otherwise. You can put out a bunch of open models that are kind of interchangeable and generic. Each one has this little better thing. Technically if you are trying to eke out the maximum performance and you trust the Chinese not to have screwed with the backdoors of their models, you would combine these 5 different models in this kind of mixture of experts style weird system where you know exactly what task you're doing. And then, okay, maybe KIMI is the best web RAG. So if you're doing web RAG for this query, it sort of subcalls KIMI or whatever. But it's okay, right? It doesn't change the big picture. In a way that I should care about for something.

Okay, so we're maybe still at zero live players on this V scale.

I consider DeepSeek a live player, but they're just like, they're in a bad spot, right? Like, they're oppressive to be live players at all. But it's going to be a struggle. It's not like they count. If I was going to play a war game or something, you would have to have either a Chinese player or a CCP player. Either someone's playing the PCR or CCP, or someone's playing DeepSeek as well. But DeepSeek can't just be ignored.

Okay, gotcha. 5 to go. I think all these are going to qualify as lab players, but you tell me. Safe Superintelligence, fundraising at very high valuations, very big numbers. I think their latest valuation is somewhere in the $30 billion. Nobody's seen anything. The rumours are that they are taking a totally different approach to scaling, which either won't work or maybe will work, and they'll leapfrog everybody. That's a sort of rumour mill take on safe superintelligence. The rumour mill also includes Faraday cages at the office where people are checking their cell phones or whatever, and I don't really know what to make of it. My gut reaction is like, we should probably have some transparency measures at a minimum that don't allow random small companies to try to jump straight to superintelligence and spring it on the rest of us. But I guess enacting such rules would be predicated on finding the whole thing credible. So does it seem credible to you?

I mean, Ilya is credible, right? He's like one of the most credible people on the planet. If Ilya wasn't involved, I would treat this as if it was a scam, basically. I would treat this as if, like, I'm not saying this is fair enough, but I would just be like, well, there's a lot of alpha in claiming you're going for superintelligence and then raising progressively higher amounts of money at progressively higher valuations. And then maybe you come up with something at the end of it, maybe you don't. Does it even matter or something? Yeah, you try, but they tell you. So I'm very confident they're trying something. But as you know, it being real does not mean it's that likely to work, right? The default, I think, is that they're trying something, but it's kind of moonshotty, and by default, things like that don't work most of the time. And so in most worlds, it doesn't work unless proven otherwise. I don't know. We have very little information, as you say. I agree that in a sane world, they'd be reporting to the government. what's going on. I'm not sure that it's necessarily like, yeah, ideally the public would also be informed somewhat, but like, I respect the hell out of authority cages, right? Like, they're taking the secrecy seriously, right? Like, I don't want you to like keep it secret, but like not protect your secrets. I want you to like either like let us know what your safety cages are and why we should like trust you with this power, or you should like— treat it as if you are properly siloed and trying to protect it from spies and go from there. And okay, but yeah, you should be reporting. Maybe you should go into a skip at Pentagon and every few months brief someone or something. I don't know, but do something appropriate to the situation, whatever's going on. I'm basically like, they're there in the background, and I am sort of acting on the assumption it's nothing, but maybe it's something.

Yeah, makes sense. I mean, I don't know what else we can really do other than just advocate for some transnominal clause. How about xAI, aka Grok? My, whatever, 2 months removed from the original launch of the thing. I would say it is notable as having continued to be part of my rotation. I don't go to that many different models. We're going to, you know, basically this is the point on the list of candidates where I actually go to their models. I go to Grok least, but it is useful. And I also feel like there is maybe something going on with this whole truth-seeking thing, right? I mean, Elon is behaving strangely in many respects. effects, to say the least. But the AI continues to be able to criticise him pretty freely on his own platform. And aside from one little blip where it was briefly told not to, it seems like at least so far they've sort of held to their stance of, like, it's going to say what it says.

It was mostly doing it even during the blip, is my understanding, when it was told not to, because you could just override that pretty easily. But I mean, it speaks to their credit. And they trained a model that isn't brainwashed to not do that. It also speaks to their incompetence that they weren't able to do that. Is it making some sense? Because clearly their bosses—

They claim that they are maximally truth-seeking, and that should imply some sort of sense in which if Elon Musk really is the biggest source of misinformation on Twitter, that the AI will say as much, right? I guess the charitable interpretation, if we want to be charitable to xAI, and Elon is, I guess, in the eye of the beholder. But it seems like they're saying they didn't try to make it not talk badly about Elon, and certainly the results are consistent with that.

I mean, it's entirely possible that Elon was under the impression that if he made a maximally truth-seeking AI, that it would recognise how truth-seeking it was, and he was surprised to find out he was wrong. Similarly, he must have assumed that, like, oh, if you just train a maximally truth-seeking AI, obviously it will be anti-woke. And then he found out he was again wrong. And it's very hard to actually control the personality of these things without lobotomising them, without being very heavy-handed. And to their credit, they didn't do that. But also, I don't think they had any slack to do it in the sense that it's part of your rotation. Why? I'm curious.

It's fast.

It shows the chain of thought. It's been supplanted by Gemini 2.5 Pro as the frontier fast model that shows chain of thought. But especially if I have something that I want to do multiple runs on at the same time, it at least makes the cut to do a run with Grok.

Okay. Yeah, I didn't respect the whole, like, I want to just run it and run through this thing, and I can open an extra window and Grok's already—

Its voice is also pretty good, actually, for what it's worth.

Yeah, it's fair. I don't use voice at all. quality wouldn't matter for me at all. But yeah, okay. I mean, I haven't used Grok in weeks and I don't miss it. And when I see Grok's announcements of the things that they're putting out, it just feels so pathetic for a company that big, that highly valued. Everyone's dropping new models and Grok's like, we got API access, or we have a canvas now, or whatever the latest one was. like, okay, you're cute or something.

But they kind of have to catch up on that stuff, right?

I get that. But it's also like, I don't know, I don't find a lot of them very good. I don't particularly mesh with its attitude. It's really weird to have a thing that's like, I don't know, Douglas Adams fetish, and yet I still think you're just lame, right? It's got all the right geeky associations. It just has to try hard. I don't know. I don't like it. And I just don't think it's that smart or useful to me. It doesn't do a good job of, like, its one job is checking Twitter, supposedly, having real time on Twitter.

It does weirdly suck at that, I've found.

Yeah, it's really bad at it because it'll pick a random subset of Twitter and look at that and report from it. But what I want is for you to search all of current Twitter or all of the archive of Twitter.

Just literally, if it would search all of the tweets I've liked, it would be of revelation. That in and of itself, we have to turn the hyperparameter up things, I think, where—

I mean, there's a useful thing that would be in my rotation they could have achieved and they didn't. But mostly it does feel like they spent a tonne of compute and to me didn't accomplish much. And I don't know, I don't feel the liveness there personally. It feels like very overvalued. and not that interesting in relative terms. Certainly the short DROC, long Anthropic trade seems absurd given that I'm selling the valuable one and buying the cheap one. It's like, how did I get to do that?

Yeah, that's interesting for sure. What do you expect from them going forward? Clearly they can scale infrastructure and they can pour a lot of flops into—

Yeah, but I think we saw Meta prove that it doesn't do it on its own. You have to be good. And it felt like Grok was like, we're going to throw all the compute at this thing and hope that that's enough. And it's like, no, it's not enough. But you threw so much compute at it somewhat competently, so it was enough to be okay or something. But they're also just not being great.

If your sense of Meta being like, why is Meta struggling? It would presumably have something to do with they're organisationally bloated and there's no great tastemakers in the right places to sort of—

I mean, it's a horrible place to work. Yama-kun is kind of in charge and doesn't believe in LLMs and never did. And they are obviously evil and have been. This is before the name changed to Meta. And just the combination of factors is just like, makes, like, you know, how do you recruit? How do you do something good? It's deeply, deeply rotten.

Okay, I don't sign on to every last one of those points, but it does strike me, though, that xAI probably has a very different dynamic on all those dimensions, right? Like, Elani's very good at putting people in positions where they have executive authority to make things happen. I think they will presumably have a much smaller set of people making the key decisions, and those people will be probably world class and highly empowered, right?

Supposedly. I don't know. The results don't seem that great. I strongly suspect that Musk fine tuned his management mechanisms and techniques on certain companies and then is applying them out of distribution in places where they don't really apply, including the federal government, but also xAI, which is a different type of company building a different type of product. And there's going to be some mismatch. It's in very obvious ways. It just doesn't have, like, a sort of very technical, very like, I'm going to personally understand how the mechanisms work. And I'm going to hold people's feet to the fire because I can actually physically measure what's going on in these ways, things that you can do at Tesla and SpaceX that you can't do at xAI. And I don't know that Elon can. And also, he's spread ridiculously thin. And I just don't believe in the Elon magic necessarily. And I also, I'm not sure we're dealing with the same Elon in 2025 that we were dealing with in 2015. Right. So if nothing else, he's exposed to a very distorted information environment. Like, his own AI is telling him he's the biggest spreader of misinformation on his own platform. That's not a great sign unless he listens to it.

But yeah, I mean, there's certain—

No, but I think it's very clear that he is not listening to it.

It's not gotten through yet.

His response was not, oh, this must be true. I need to rethink everything.

No.

In fact, he has been criticising the Federal Reserve this past week. So, yeah, still at it.

So, okay, going to Anthropic. So they were your other side of the trade. I don't really have a lot to say about Anthropic at the moment. To me, it seems like They continue to chug along. I think they continue to do great work in just about every respect. I thought their— I wonder if you had any reactions to their recent tracing the thoughts of large language models work. I thought it was excellent and extremely well presented, both in form and just in the way that they— the discussion that they put forward to contextualise all the different traces and so on. I also did feel at the same time like it kind of got away from them a little bit in that you go on YouTube, people started to send me these videos of like, it's solved. We now know how AIs think, and it's not how we thought. And on that front, I was like, damn, this is potentially going to be kind of perniciously used in a bunch of downstream political discourse.

I mean, this is like the 5th time that that's been claimed, right? Or something. It's not a new dynamic. Basically, people will constantly claim these problems have been solved. Remember the famous Marc Andreessen letter? The black box nature of large language models has been solved. And he just testified like this under penalty of perjury to all the major institutions in the world. And like, yeah.

That was weird.

Yeah, that's like a federal crime. That's pretty bad. Obviously false. And this is going to be the next, like, oh, we've solved it. Obviously we haven't solved it. And everybody knows we haven't solved it. But people are going to say this stuff. They're just going to say this stuff. There's nothing you can do about it. I thought they were very good papers. I talked to one of the authors about it. I meant to write a post about it, but again, things have been happening in the world, and that post was still in the draft folder and only 25% done or something.

Dude, that's a lot. I mean, you put out long posts, but those were long things too.

Yes. No, the problem is that writing that post is more than its share of the calendar to write. And so I unfortunately might just not be able to write that post and have to— other people just need to rely on the original for now, at least. But it's unfortunate. But I do think it was a very good set of papers. I will say that.

Yeah, I guess philosophically on this interpretability side, on so many of these things, I feel like I kind of am on a bit of a roller coaster where 3 years ago I would have said, like, damn, we have no idea what's going on in these models at all. It feels like just an impossible mountain to climb. Like, we're going to need decades for this. And then 2 years ago I would have said, wow, we've got some traction. And then like a year ago I would have said, hey, we've got a lot of traction. Like these Spark autoencoders are working. We're identifying all these concepts. We got Golden Gate Claude. This is amazing. And now at present, I'm a little bit maybe less optimistic again, and not because they haven't continued to make great progress, but I just look at some of these traces, which, first of all, it should be noted, have a lot of error terms added in for correction purposes. And that is way too often glossed over entirely in the analysis. And then you have the philosophical question of, like, okay, all these concepts are being auto-labelled, and some of them are probably correctly labelled. Like, it seems like the Golden Gate Bridge feature that they turned up to get Golden Gate Claude was probably hard to confuse for something else. And the behaviour seemed roundly greeted as if they had hit on a genuine, actual, meaningful feature that corresponds to reality. And how often is that happening?

Yeah, that's a great question. I don't think we know. But at one point he said, expected 1 unit of alignment progress, got 2, 998 to go. Basically the attitude of, we're making better progress than I expected on some of these fronts, but it's definitely not enough to get there on time. It's very hard to turn this into a good outcome, even if you do well, because again, trying to use it too aggressively is the most forbidden technique. So it informs your decisions, but you have to be very careful how you use it. But we'll see.

I think the only other thing I had on Anthropic was, I don't think we've talked since Dario became quite clear on his calls for US-China AI race. And I wonder if you had any thoughts on that. Spoiler, I didn't like it.

Anthropic continues to talk in public in ways that are unhelpful, and Dario has accelerated this phenomenon. They're not maximally unhelpful. They are much better than OpenAI's strategies. You can tell the difference, like, clear as day. Still, I have obviously had assurances at various points that behind the scenes they're doing better. and that they are doing their best and this is trying times, et cetera, et cetera. And I am not zero sympathetic to that, but you can only judge on what you know to a large extent. You can't just trust. I don't have that kind of level of trust in these people. So, yeah, I'm not happy about it, but I do have a lot of trust in the rank and file and in the technical intentions here. So there's that. And I do think they've been executing very well. I think they're currently, because both Google and OpenAI have gone, have deployed since Claude last deployed, right? So they're currently behind in the shuffle. But we'll see what Claude 4 brings. My guess is it'll bring quite a bit. But for now we wait. Yeah, I don't like it either, but at the same time, It's not obviously a wrong strategy, so I'm not supposed to like it. I'm not going to make the mistake of, well, strategically it's the right move, so I'm going to like it even though I don't like it. But I'm also not going to make the move of, I hate you now because you made a correct strategic move. I'm just going to not like it.

So unpack why we should think the strategy is correct, because if I was his speechwriter, I've pitched this to a few other people as well, so apologies to the listeners who may have heard it a couple of times. But I would have said, okay, don't call for an AI arms race with China. Why don't you just say, hey, I know China's given us a lot of problems and they've done a lot of things that we rightfully object to. I don't really know how we should deal with China. International relations is not my expertise. But what I do know is scheming seems to be on the rise in the latest generation of models, and we've got really a of questions that we don't have great answers to. So we might want to keep the option of trying to collaborate with them open, as difficult as that may be, given what bad actors they seem to be. So why not say that? Why is that not the right strategic move?

I think it probably is in some sense, or more of that is in what they've been doing. I'm trying to be nice. I'm trying to give some of the benefit of the doubt. But certainly you have to do some amount of Like, dealing with what is. And I think supporting export controls is almost certainly a wise move. The rhetoric he's using to justify them justifies a bunch of other stuff, and it's not good. But if it buys you a seat at the table to advocate for other things, maybe it's not so bad. If it protects you, gets you treated like wannabe American champion labs.

Okay.

The other candidate I have for a hero, you may laugh, But Google DeepMind, maybe. I mean, at the top, we do have Demis going out and saying the CERN for AI thing still, which is a notable departure from the other trends that are all kind of jockeying for this sort of American champion status. Their product work continues to be lacklustre in most places, but the models are getting really good.

The models are really good. Demis has been the best in terms of communication. He's also been a very high-level player in communication, perhaps too high. Like, I think I noted, like, in the summit report that, like, you know, he is displaying Japanese levels of saying the house is on fire without saying it. And, like, I got private feedback that this is indeed, like, you know, the type of philosophy that he had in his head a decent portion of the time, that, like, You know, he would've, you know, if he had, if he had seen that, he would've like smiled, right? Like, like, oh yeah. Yeah. But, um, like, it's like very clearly like he is not only calling for the right things, but he is like very carefully saying some things and not saying other things in ways that like very, very clearly communicate where he is at. But at the same time, Demis being a good guy doesn't necessarily translate to Google being a good guy. Because Demis does not control Google, right? Like, he is not CEO of Google. He's CEO of DeepMind. DeepMind answers to Google. I don't see Google itself as a particularly trustworthy actor. And as you say, unfortunately, the products are lagging behind. Also, the alignment is lagging behind in the sense that they've been pretty much forced to take a sledgehammer to their models. in the sense of what they're allowed to answer and what they're permitted to say in various ways, because they're afraid of what would happen if they didn't. Part of that is corporate policy as to what they're afraid of. Part of that is, I think, their inability to bespoke shape what they do and do not say. So, for example, getting probabilities out of Gemini is incredibly difficult, like estimates. And that makes it much less useful to me in a lot of ways. Knowing that that will just never show up. The ordinary peer censorship mostly doesn't bother me, but I run into it more at Gemini than I run into it anywhere else. I basically never had it happen at Claude. I have had it happen once that I can remember at OpenAI, which was ironically, I was talking about the model card and I was asking about why the threshold for virology was where it was. Because obviously if this is above human baseline, the humans can solve these problems, so why can't the AI? And it, like, tripped the virology flag and it refused to answer, which I found highly amusing.

Right.

Yeah, I respected it because, like, okay, I'll stop asking you and I'll form my own opinion. But it's a good question, right? Like, if the human baseline is 40% and it's scoring 50-something percent and you're saying it's not good enough, well, it's above human baseline, and the humans do sometimes solve these problems. You've got some explaining to do, mister.

Yeah, I find these no significant or no meaningful uplift findings to be basically not credible at this point. And I haven't sat there and run whatever experiments they've run, but we're talking here about when they do run these experiments, they give people helpful-only models, right? So they're not having to jailbreak them. I can understand you might be able to lock it down well enough that your deployed version is not meaningfully helpful, And then you have the issues that you just ran into. But if you're given a helpful-only Gemini 2.5 or 0.3 or whatever, how in the world is it not a meaningful uplift? I find that very hard to believe.

They have a threshold for how much uplift counts as a problem, and right now we are still somewhat below that threshold. But yeah, there's no— nothing is obviously not credible.

Okay, cool. All right, so last but by no means least on our live player list, OpenAI. A lot to pick apart there, to say the least. I think my biggest question is, do you understand them as being ideologically committed to chasing a first mover advantage in recursive self-improvement intelligence explosion dynamics at this point? Or is that an overread on my part?

I mean, I think they are committed to winning. They're committed to making an extremely valuable company. They're committed to building AGI. I think effectively they realise that their position depends on being perceived as being in the lead, and they will do what they have to do to be perceived in the lead in various ways. I do think they're willing to cut a large amount of corners. I think they have reasonable beliefs that cutting those corners is mostly harmless right now, but I'm not convinced that they're setting things up such that if that changed, they would realise it before something bad happened. They have good people who are working on problems and they are in many ways doing a much better job than most other labs. Worse than Anthropic, but not obviously worse than anyone else. and clearly better than many. But again, they have the hardest job, they have the most dangerous position, so they have to be held to a pretty high standard here.

Do you have a sense— I just struggle to understand OpenAI so much. And the juxtaposition recently between the obfuscated reward hacking paper on the one hand, and then their contribution to the White House call for comments or whatever, where they put out their sort of, give us all the data, basically give us everything, and we'll hopefully be the national champion and beat China. Those just seemed like 2 totally different organisations would have produced those 2 artefacts. And maybe that's the right way to think about it, is it's just sort of an amalgam of different subcultures in one organisation, and it's just kind of unwieldy. And That's why it's been such a mess. Anything to add to that, or feel like that's at all part of that?

I wouldn't say that they're not— they weren't pushing, like, favour OpenAI specifically as a national champion so much as they were saying, like, favour your labs and give us all an advantage. They didn't have a, like, well, favour us and not Anthropic or not Google. And that's to their credit. But the rest of it was just obviously by signalling. They hired an obviously evil lobbyist to head their lobbying division, and that's what they've chosen for public communications, is to be in that mode, and they've owned that. They obviously do have a real alignment safety department, fairness department, with real people who I have in fact interacted with and worked with a little bit. some drafts and stuff, and it's good. These people are trying their best and they come up with good research and do good things, like the model spec and the philosophy documents and such. It's just they decided that their public lobbying communication strategy is going to be this other thing. And this includes Altman's statements and what Altman actually believes and what he'll actually do when the chips are down is a big unknown. But so far, I don't particularly love what I'm seeing. But again, there is a lot worse out there. Yeah.

So do you think when you see things like the amicus— obviously, we've got the Elon Musk lawsuit, which Sam Altman described as he thinks Elon's just trying to slow him down. I think that's probably a pretty good interpretation, actually. And then you've got 12 former employees that came along and filed an amicus brief and basically said, It would be a fundamental violation of the nonprofit and of the reasons that we joined and all the promises that were made to us when we did join to turn this thing into a for-profit company instead. And that presumably is just like another way to kind of slow them down, right? Maybe they'll have to go back and renegotiate with investors or whatever. Do you think that it's good to slow OpenAI down? Is sort of just throwing sand in the gears, trying to get them out of the first position with this sort of distraction and kind of just friction creation tactics? Is that like a good thing in your mind?

Well, it's important to know that Elon's right. He's had some really bullshit lawsuits against OpenAI in the past where he's been very, very wrong. But questions of standing aside, OpenAI is attempting to execute the second biggest theft in human history.

Are you giving first place to the stealing of the nuclear secrets by the Soviet spies, or what's in number one?

I am not commenting on what I think is number one. I'm letting people figure it out. But I have, upon reflection, decided to call this the second biggest theft in human history. And we have to understand that that would in fact be a betrayal of the commitment to the nonprofit. And a lot of employees were recruited on the basis of there being a nonprofit and this being OpenAI's mission and how all this would work. And the amicus brief is clearly correct. Whether or not promises were broken specifically to Elon is a matter of fact that I don't know how to evaluate. The question of standing is clearly unclear, but obviously the terms that have been suggested for the nonprofit are completely completely unacceptable. And the thing they're trying to turn the nonprofit into is also completely unacceptable, which is basically like a marketing department for OpenAI that will buy its products and then give them to nonprofits or whatever. It's like, this is not what we signed up for. This is not what the nonprofit is for. No matter how well-resourced you make it, if this is what it is, it's like, what are we even doing? So obviously stopping that feels imperative. And if it happens to slow OpenAI down, it happens to slow OpenAI down. That's just a side effect that Elon, I'm sure, loves. But yeah, that's sort of irrelevant to my assessment of what is going on. If you want the nonprofit to hand over its financial interests so that you can raise money, there are ways to do that. that don't involve giant theft, and you can do one of those. You're choosing not to. So there you go.

Okay, I think that brings us to the end of our live players list. So I have 2 more questions. One is, I am a little confused on your take on the weaponisation of AI or the creation of autonomous killer robots. And I think Just the naive response is like, you're worried that AI is going to kill us all. Doesn't it make it more likely that AI kills us all if we make autonomous killer robots that we can then lose control of, or somebody can take over in a coup? I mean, this seems like if an AI is going to kill me, this seems like maybe not the most likely way, but weighted for soonness, it would be maybe the most likely. It might have the biggest expectation on my life expectancy.

No, I think that's just not right. First of all, the AI implies the killer robots. There is no world in which we have these AGIs, we have these ASIs, and all the nations of the world just decide we're not going to build killer robots unless there's just no reason to go and kill our robots. It's possible that killer drones Better than killer robots. But assuming killer robots are like—

Yeah, I'm counting the killer drones for what it's worth.

Yeah, they've already done it, right? It's done. It's already happening. So America just voluntarily disarming doesn't accomplish anything. Automated killer robots cause you to have that reaction. They cause people to notice that this technology is dangerous. And then cause people to then demand that we handle it responsibly, right? Whereas like, it doesn't actually create dangers because if the AI was actually going to get control of the situation, it wouldn't matter if there were robots. They would either build the robots and drones they needed afterwards, repurpose something else to be those things, or they would simply like do this in other ways. Like the whole like trope of like, oh, it took control of the robots and now the robots are fighting the humans. now you have this big struggle. That doesn't happen. If the AIs are sufficiently powerful to take over in that way, they were sufficiently powerful to kill us in any number of other ways. It's just not relevant to the questions that we should be worried about. It's not in my threat model as a major problem, but it is in other people's threat models. And also, if something goes wrong, if some of the robots get hijacked by some rogue AI and they start causing local problems, Well, it's probably not particularly worse than the problems they would have caused in some other way without them, but they are ways that would cause people to react and then take proper precautions as a result. So again, there is no reason not to build them, basically. It would just be unilateral disarmament. It would just put us in a worse position. It would just make the future less likely to be American, Western, democratic, et cetera, for very little gain, if anything. So yeah, this is just not the hill you want to be dying on. This is something you need to do. If you don't want to build autonomous killer robots, don't build the AIs that enable you to build autonomous killer robots. That is the way you don't build autonomous killer robots. And if the autonomous killer robots cause people to say, don't build the AI, good, right? Like, if we can pull that off, everyone agrees not to build those AIs.

I'm not a big analogy guy, but tell me why this analogy doesn't hold. We have biotechnology, which has obviously got great promise for better lives for us in a very grossly similar way that AI has a promise of much better life for us. We also could weaponise the biotechnology and create bioweapons. We did unilaterally disarm there, right? We basically just said, look, maybe China's going to go off and do their own bioweapons anyway. And for all I know, maybe they are. And there's reporting to that effect, and I don't know if it's true or not. But my understanding is we, the United States, whatever, the good guys, decided this is just too bad of a technology to develop and we're just not going to do it. And we hope that others will follow our example. But even if they don't, we still feel like we're kind of better off because we didn't go down this path, because it's more likely to hurt all of us than it is to be something that we're really glad to have built. Why do you not see the autonomous killer robots in the same way?

First of all, my understanding is essentially, though, I thought there was a biological weapons convention. I thought we all did agree not to do it. It's just that various nations have been cheating to various degrees at various points, as you do. But you don't cheat fully. You cheat a little, right? It's like you cheat an understandable, deniable amount, and that's still a lot better than openly pursuing. But fundamentally, the answer is that there is no good way to use a biological weapon. There's only terrible ways to use a biological weapon, right? Like, if you deploy a biological weapon that's any good, you risk it turning back on you. You risk it doing much more damage than you expected. Like, they can't be controlled, they can't be predicted. And like—

Isn't that your take on AI in general? I mean, the thing that's a little bit of a stuffy is like, you're saying that's going to happen, but not because of the robots.

Because of the AIs. I'm saying that the AIs are the thing that can cause things to get out of control, that can cause the world to end up in a bad state. It's not the robots that we may or may not build. The biological weapons, if you ever use them, it's kind complete disaster. And also, we've developed a universal norm and very strong distaste for anyone who ever uses biological weapons. You would turn the world against you almost immediately, to the extent that you didn't destroy the world by using them. And therefore, what's the point of having them? You turn the world against you even by having them in an open fashion.

Now, there has been some talk, which I'm sure you're aware of, of the possibility of bioweapons that only target certain, shall we say, phenotypes. And I don't know how credible that is to really create, but if all of a sudden you had a path to that, it seems like— I mean, I would still be like, still don't build them. Would you?

No, obviously still don't build them. You can't trust it to work the way you want it to. You don't know that they don't have their own bioweapons they would launch because you launched yours. You don't know they wouldn't launch nukes in response to your bioweapons. You don't know any number of things. Look, I prefer nobody develop bioweapons. I strongly prefer nobody develop bioweapons. One of the reasons why you don't want AIs running around uncontrolled is because if it is possible to develop bioweapons with AI assistance, That could potentially be very difficult to stop in any reasonable way. So you might not have a lot of choices.

So does the main thing on the distinction between the bioweapons and the AI weapons, autonomous killer robots as they're known, does that come down to just like the robots can't self-replicate and so they're just fundamentally more single?

If the robots were self-replicating, then I would be much more scared. Of the actual autonomous killer robots. That's absolutely true. Like, that's an example of this, right? Like, you build a nanobot that, like, could potentially, like, drague you to the planet, then, like, that is, like, something you just don't do independently of everything else. But, like, autonomous killer robots, again, like, it doesn't fundamentally change the game. Like, drones are already the primary weapons of war in, like, actual live fighting that's happening right now. They're already, like, Everyone is designing their future militaries around this possibility, even without AI. And you can't keep AI out of warfare. You can't voluntarily disarm in these ways. Anybody who does will just lose. Yeah, if you could look and you get everybody in the world to sing Kumbaya and agree that we never built any more drones or autonomous weapons of any kind and then actually enforce that, I mean, probably could, but that's not going to happen. And we should not pretend that it is happening. We can't get a lot of leaders on here. And again, if something goes wrong, it goes wrong locally, right? It's not an existential risk to have autonomous killer robots. The AI, they're not going to turn on us by taking control of our autonomous killer robots and winning a war against humanity they would have otherwise lost. That is just such a very unlikely scenario. like we think they happen.

Yeah, I don't know. I mean, doesn't feel vanishing to me. Well, I mean, this is a hard thing to really unpack, but it does strike me that there could very well be a point in time where the AIs are not generally able to do whatever they want, but they might be able to hack into a million autonomous robots that we have created.

If they can hack into whatever they want, we've already lost. If they can and choose to with malicious intent, it's over by that point.

So is there any affordance that you would be like, don't give the robot this, or don't give the AI this affordance? Because this seems like it would be pretty high up on most people's draft boards for affordances to deny the AI would be like the actual lethal weapons, right?

I mean, I wouldn't give them control of the nukes for the reasons you're talking about. I think that's a very easy way to imagine that going wrong. But for the robots, I mean, obviously you want to have various checks, but designing exactly what checks those are at 9:50 in the evening after a very long day is just not where I need to be. But we can't not do this. We obviously have to do this. And even if China said, we won't do it if you won't do it, well, that's only 2 countries, and there's a lot of other countries.

Yeah. Okay, well, perfect transition to my last question for you, which is just an invitation. I guess I still somehow see the risk that one would incur by unilaterally not pursuing things like weaponisation as virtuous. I don't deny that there is risk there, but I'm kind of like, if we want to get to a different equilibrium, somebody might have to take a leap of faith. And it does feel to me like there's some virtue in putting oneself forward to do that.

I think you're just shooting yourself in the foot. There's no virtue in losing on purpose.

Well, I mean, there's a lot of assumptions baked in there, right? I mean, I'm not so sure that you couldn't get others to follow your lead, and it certainly would depend on state of evidence and whatever. But broadly, I think that there is a sort of, we are not going to do this even though we expose ourselves to some risk in not doing this, that I see as virtuous, because I see on the other side of it, just like a really bad equilibrium. And it's like, somebody's got to take some risk.

Then do it where it matters. Do it where it actually potentially saves us. Don't waste your big noble sacrifice on a completely irrelevant, superficial look of there's nothing with red glowing eyes that walks around. It's such a stupid place to forfeit your strategic power. Like, no, it's just like, I find it so absurd.

Okay, well, here's the invitation to you. So what is virtuous to do today? What are the big virtuous moves that you could recommend to specific individuals or to the listening audience at large? Where would you spend that capital? Or what other virtuous moves do you want to see more people making?

I mean, so like on the governance side, like we need transparency, we need state capacity, we need like state visibility into the labs, we need cybersecurity at the labs, we need our export controls to actually like be strengthened and enforced. And we need to like return the world to a place where we can reasonably cooperate in various ways and be reasonably like prosperous and normal in other ways so that we like have a good foundation to move forward and not feel obligated to push these buttons would be the obvious things to say. On a private level, again, capturing mundane utility, helping people live better lives, that's good. Pushing the frontier of capabilities, that should be highly questionable. But obviously working on Alignment, security, safety, those kinds of things, almost universally good. Preparing various policy interventions, getting them ready, like trying to build networks, trying to understand these things, trying to educate the public, trying to build awareness of these issues. That stuff is good. There's no slam dunk right answers. Trying to raise the level of discourse is obviously very good. But it's rough out there. Do no harm is obviously the first thing you say in these situations. But yeah, it's rough. But there's definitely talent constraints on policy, on state capacity in various places. And on various alignment and technical work. And certainly those are the obvious places if one wants to be especially virtuous. If you can, again, help keep an eye on the labs, you can help keep discourse higher level and accurate and elevate people who seem to be truth seeking over people who don't seem to be truth seeking. And you can make your own judgement as to who those people are and so on.

What about on just a personal attitude level? You said do no harm, and I sort of feel like maybe I want to question that a little bit. Maybe I want to say take more risk. Like, we're in a short—

Oh, yeah. No, I meant more of, like, don't do things that are just clearly accelerating the process without a reason. I didn't mean, like, be risk averse. You definitely don't want to be risk averse at this time. You have to be risk loving because We need variance. We need something to go right. We need a lot of things to go right. So, yeah, I think one of the biggest historical mistakes that was made by people who are concerned about this was that during the 2010s especially, but also before that, there was a lot of do no harm that was paralysis-inducing, where people basically didn't do anything or they kept things way too secret. Or they just were afraid to spread words about things and so on. And that turned out to keep the wrong thing a secret in ways that prevented us from making as much progress as we could have without controlling the messages that actually caused acceleration and harm. We should have done a very different set of things. The dangerous message was actually the fact that this thing was dangerous. which caused people to pay attention to it as opposed to the technical insights that we had, which would've been better to spread around because that would allow people to make better progress. So definitely don't be secretive with the productive type of ideas. Again, in almost every situation, being open about and helping other people be smarter and understand things better is good. It's the weird exceptions where it's bad, and you should look out for that.

Well, on that note, you are definitely living your own conception of a virtuous life by pumping out very thorough analysis on an unbelievably regular basis. So it's a great public service and a great resource, and I turn to it regularly, and others definitely should too. So I appreciate all that hard work. Any final thoughts you want to leave people with, or are you ready to collapse?

I think I'm about ready to collapse, so let's call it.

All right, well, I appreciate it. Heroic effort and virtuous effort. Zvi Moskowitz, thank you for being part of the Cognitive Revolution.

It is both energising and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.