Pick Your Poison: Zvi Mowshowitz on the Unipolar/Multipolar AGI Dilemma, OpenFace & Pacing the ...
5 Aug 2026 · 178 min
In this episode of the Cognitive Revolution podcast, host Nathan Labenz interviews AI safety researcher Zvi Mowshowitz. The conversation moves through several interlocking themes. First, they dissect the recent "pacing the frontier" letter, explaining why many lab employees signed it while top executives (Sam Altman, Demis Hassabis) stayed off the signature list, and how the letter is meant to signal internal concern without becoming a marketing stunt. They then explore concrete ways to implement pacing, such as extending review windows, imposing compute caps, and delaying public releases, while noting the limits of these measures when labs continue internal development and recursive self‑improvement. The discussion shifts to the broader unipolar versus multipolar AGI dilemma: a single dominant AI could simplify safety but concentrates power, whereas many competing AIs spread risk but make coordination harder. Zvi argues that both options involve trade‑offs and that choosing a "poison" (a set of compromises) is inevitable. The interview also delves into philosophical questions about model identity and consciousness, referencing recent Google work on self‑conception and its correlation with alignment, and debating whether AI should be allowed to think it is a moral patient. They consider how to prevent models from developing dangerous incentives, especially when they can self‑improve or be used for dual‑use tasks. Zvi advocates for funding alternate architectures and breadth‑first research, warning that the current depth‑first, compute‑heavy approach narrows the solution space. Policy‑level ideas surface, including the merits of a global AI pause versus targeted pacing, the role of international cooperation (e.g., China’s pre‑release reviews), and specific legislative proposals from a Michigan Senate candidate (interpretability standards, bio‑security red‑teamings, compute controls). Zvi critiques the feasibility of such detailed mandates but acknowledges their potential value. Finally, the conversation turns personal: Zvi shares how he maintains mental health amid the AI race, emphasizing rest, Sabbath‑style unplugging, exercise, and non‑AI hobbies as a "System 3" mode of thinking beyond the fast System 1/2 mental work of AI research. The episode ends with the recurring refrain “pick your poison,” underscoring the central theme that any path forward involves hard, often unpalatable choices.
Chapters
Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to have Zvi Moshiowitz back for another wide-ranging rundown of what has obviously been a wild time in the AI world. We begin with a mundane utility check, with Zvi describing how Fable is now serving as his editor, and a discussion of how up-to-date, or should I say situationally aware, we want our AI assistants to be. From there, it's on to the headlines. We get Zvi's take on everything, starting with the OpenFace incident, what it implies about the level of execution competence we can expect from frontier companies, and why moderate prudence won't be enough to deliver a good outcome. We also discuss the fact that Claude, despite greater emphasis on constitutional training, has similarly misbehaved. Why Zvi believes that recent AI history, including the public response to both 4.0 and 0.3, suggests that market incentives won't be enough to bring about robust alignment. The potentially tricky spot that METER and Redwood are now in as investigators and what could be done to strengthen their position. The recent Pacing the Frontier letter, what sort of pacing deals we might see and how they might be formed. How we should interpret recent advances in interpretability and AI consciousness research and where we should and shouldn't attempt to shape AI's sense of self. How we can encourage greater breadth in AI research and diversity of AI minds, How I should vote in this week's hotly contested Michigan Senate primary in light of AI issues. And how Zvi thinks about making time for exercise, rest, and recovery amidst so much AI acceleration. At one point, Zvi describes the current situation as both a total less wrong victory and a total less wrong defeat. It's clear at this point that the AI safety community was right to worry about AIs taking extreme actions in pursuit of arbitrary, Even silly goals. And yet, here we are at what sure seems to be the beginning of recursive self-improvement, still seeking good answers to such fundamental questions as, how can we avoid catastrophic misuse without dangerously concentrating power? The reality today, Zvi says, is that there is no truly low-risk path available. The best we can do, at least until the next major warning shot and vibe shift, is to moderate the race dynamics, so that alignment and interpretability research have more time to mature. And simultaneously, we can execute defense-in-depth strategies to the very best of our ability. And even then, to some extent, we will probably have no choice but to pick our poison from a menu of genuinely scary risks. With that, I hope you enjoyed this sobering but often funny overview of the AI landscape with the one and only Zvi Moskowitz. Zvi Moskowitz, welcome back to The Cognitive Revolution.
Yeah, it's good to be here again. It's been a while.
It's been a while, and boy, has a lot happened. Unbelievable. It's just crazy. And I know you're living it and you're in the thick of it as much as just about anyone. First, for starters, today, with the incredible amount of information and activity in mind, I wanted to do a mundane utility check. How are you using AI to help keep up? Has it started to change your workflow beyond the Chrome extension that we talked about last time that automates very local operation type things? Or are you still kind of raw-dogging it with your wetware in the skull?
Fable has supercharged the extension to the extent that whenever there's anything, it does anything I don't quite, I just tell it exactly what went wrong and what I want it to do instead. And then that one command reliably just works. I presume SOL would also be good enough for this. It's just I'm now doing it. The biggest other change is that AI editing is here. I used to not do an editing pass with the AI. It wasn't good enough to be worth it. Now I have Fable and sometimes Opus, depending on how much I'm talking about cybersecurity and other related topics. And on one occasion, Opus 4.8 instead of 5, go through the post and give me a list of, here's all of the typos, here's all of the conceptual errors, here's all of the Facts I need to check. Here's like the things that are missing or that could be strengthened. Here's the things where I, where it disagrees. And I found this to be very useful and it makes the post better. It does make the post a little bit slower because it is actually just an extra step. I do everything I would've done until that point, and then I do this. But one reader messaged me, it's weird reading your posts now that they don't have typos in them. They're taking a bit of getting used to. They are still the same posts. And I'm like, yes, that's exactly what I had in mind. I experimented with also using SOL, but SOL failed what I call editor bench, in the sense that SOL will say 99% confidence that you have this error here, and then at least half the time it's wrong. And I can't handle that level of— and it's also very obnoxious about it, is the other problem. It will tell you you are wrong, and this is how you have to state it, and this is a blocker for publication. And obviously, if I tweak the instructions on the project enough, I could get it to be more friendly and not be write as obnoxious and do somewhat better. And I did some of that work, but I still found, okay, this is not giving me enough marginal benefit that it's worth the aggravation or additional time of having to sort through 2 of them after I'm ready to hit publish. So for now, I'll restart. I'll retry FastDraft, I'm sure. But for now, I'm just doing fable/opus editing. And of course, anytime I have a curiosity, I use it to digest papers, ask questions about papers. Ask questions about policy documents, ask questions about people's statements when they're longer. I have it research situations. What's up with this? Is this— especially when you're worried about, is something legitimate? Is something really happening? Is something kind of a joke? Is something a claim? When Astra's claims came out, I had it investigate them in various ways. I had it summarise them in various ways. That was very helpful. But the core thing is still being right on. I don't think we are anywhere close To the point where it can do any— you wouldn't want it to do any of the writing because the writing is how you think. So even if it could produce the writing, which it can't, I would still want to produce the writing anyway. And my writing style is very unique, and I think it basically is totally incomplete to reproduce, even if it was okay with me. It has substantially made me more productive. I do worry with the idea of there used to be a lot more kind of dead time opportunity to think, opportunity to sort of not have yourself so engaged in your situation, involved in engaging in all of these activities, right? Like the whole sword fighting, XKCD, my code's compiling. And to some extent, it's like while Fables or GPT Pro is running, I have to wait for the result. But there's always other things you can do with that. You can always start another instance. You can always— you're doing all this context switching, you're doing all this focusing, and you're just trying to do everything so fast. You're just trying to have so many stuff running around your head. This sort of the lower-level stuff kind of bought you a buffer in which to think. And this is sort of the object-level version of the larger thing of people trying to do this multi-month, multi-year sprint of everything is so urgent, you can never relax. And also there's just so much more events coming at you. But if you look back, if you watch a movie set 20 years ago, let alone 50 years ago, and you see so much time being spent on just physical travel, on tracking down records, on doing these things where your brain is kind of getting a break, getting a chance to synthesise, kind of getting a chance to slowly gather something up. I wrote this biography of Lyndon B. Johnson, so I had to travel around the country and talk to all these people and look at all these archives and physically go through all these libraries. It's a huge efficiency gain to not do that, but something is obviously lost. And so we need to fight to get that thing back in some ways.
It's interesting that you said that it's just an extra step. It's making things better, but it's taking longer. I am struggling with that a little bit myself. I've started making songs for every episode, and nobody— it's actually, people do care. People do seem to really enjoy them, and I get a lot of comments, and I personally really enjoy them. So I do think in some sense it's making my output better. But talk about something that is definitely an extra step where I'm like, I'm not moving any faster. I'm definitely getting bogged down sometimes in listening to all these Suno song generations and trying to iterate to find something that I actually feel like I really like. That is a paradox right now that I don't really know how to resolve. Definitely doing more, doing better, but not faster, not saving time.
It's the thing where if you were replacing the human version of it, where you had to go hire an artist or compose the song yourself, you'd be saving a ton of time, obviously, versus having that song. But obviously that's not something you can do within the production schedule. Even if it, even if cost was no object, it's the time investment doesn't make sense. So you wouldn't have done it and now you're doing it. Same thing with the editing. Same thing with art. For my, for like, I, I like to have a, I like the value of a good banner.
Right.
To have good artwork to display on Twitter, because I have to see it constantly in the notification section. And also, I like to have them on the— when you go to the Substack, you have the pictures. And before, all I would do is I would take a— by default, I'd take a picture from the post. I'd just like, okay, here's 7 things I happened to put in the post. Which of them makes the most sense? If all of them were terrible, I'll look for some stock footage, and I'll spend 30 seconds Googling for stock footage. And that's kind of it. And now if I'm not happy with any of the default solutions, I'm going to spend a bunch of time generating something with Gemini or GPT Image. And that definitely takes longer. Sometimes it has a really good payoff. I think a lot of people got a big kick out of the giant array of Elliotts in the fedoras. And sometimes that just comes to you because that one was just like, I knew instantly that's what I wanted, and I just did one command, kept writing, came back, whoops, it's there. Other times it's not so easy. But yeah, like it's the opportunity to make a better product and then you have to do the work to make it a better product. It's like suddenly you're issuing a print issue and now you have to make sure everything collates exactly right and everything lines up exactly right in order to get this better thing. And sometimes that is not in fact worth it. You have to know when not to do it, right? Like, I, I do wonder, like with The Odyssey, right? Like Nolan films this thing with an extra 40% of the screen in this extra high resolution so that you can do this IMAX presentation when almost nobody watches it in IMAX. But you also have to make sure that 40% of the screen doesn't matter. And you're like starting to wonder, well, is it actually worth the extra effort or was that, that effort better spent on something else? I don't know.
Yeah, for me, I need to maybe know when to cut my losses a little bit more. I have, I'm very stubborn or somehow I feel like I have, because I've set this expectation, if only for myself, that I'm gonna do a song for every episode now it's like, I really want to follow through and actually make that happen. And I'm very reluctant to say, eh, this one isn't working. I'll just ship this one without a song. I probably should be a lot more willing to do that because that would, if I could cut my losses at the right time where it's like, you know what, I'm 5 generations in and I haven't heard anything good. I would save 90% of the time that I'm currently putting into something like that. But it would involve like admitting defeat in certain moments. And for some reason I have a real hard time doing that.
I, I respect that. I think there's a good value in saying, I'm always gonna do this thing, even though this thing is hard, even though this thing doesn't always work out. And even if I'm not happy fully with the results, I'm gonna put out something no matter what. And then that gives you a discipline, right? The same way that people say, write every day, right? Or work at your art every day or whatever it is, because it makes you better. Because you need to just find a thousand ways not to make a light bulb until you can make a light bulb. You can't give up. And similarly, you, I have to choose an image, right? I have to do the thing. With the AI editing, there was one post in the last week where I said, no, the speed premium here is really high. It's just not worth waiting half an hour to make this process happen, half of which is waiting for the AIs to come back with the answer and half of which is implementing it. And I just, I should just post now. And in hindsight, I'm very happy with that. Sometimes you're actively worried half an hour later the post will need more editing for new events and suddenly you'll never get it out and just this loop will keep happening. And so, you know, I think you do have to understand that sometimes the minimum viable product is what you should ship, right? Sometimes like you gotta understand, just ship it, right? Like, and, you know, vibe coding again, like if you're coding lots of new tools that you would never have coded before, then unless those tools are in turn saving you time, you're spending extra time and you're now super extra busy. But if you're replacing things that you would have done anyway, but it would've taken much longer, now you're saving tons of time. Right. So, you know, we need to all orient towards how do we save more time, including like real-time experiential time, not just like time per task or time to accomplish the same thing, but save time, like preserve slack, preserve our free time. Because I do think that like the standard of what you will do in a day has gone dramatically up and we haven't noticed in the last 5 years.
Right.
What you were expected to accomplish. I mean, maybe some of this is just me being in a special situation personally, but I think a lot of it isn't. I think a lot of it is now that we can be much more productive, much more is expected of us. The whole email becomes like it swarms your entire day, right? Just having the internet available, having email available, having texting available, all of these things you can just do. And now suddenly It doesn't free you. It shackles you down. And like, I hope that 3 years from now we're having this conversation where those are the kinds of questions we're still dealing with and we don't have much bigger problems.
Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he's currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr. Let's definitely— we're going to spend the bulk of the time today talking about potentially signs of bigger problems to come, and we'll be focused on contexts where the just ship it mentality probably isn't the right way to go. One more quick beat, though, on your use of AI. Something I noticed just in the last few weeks, as events obviously were moving super quickly, I Coined the term open face for the recent incident, and I used that in a bit of writing. And then I did the, hey, Fable, what do you think bit? And I got back like, open face is not a thing. I have no idea what you're talking about. There's no record of this. Now, partly that was because I used this funny term of my own creation that is not catching on in the broader discourse, but partly it obviously also reflects a major weakness in the models where they have this knowledge cutoff and they don't really know. What's going on? They're not up to date. I'm experimenting, and I wonder if you're experimenting with anything similar, with a situational awareness skill where I basically give the model my xAPI key and say, go look at all the posts that I've liked, kind of flesh out a little knowledge base around those and have that sitting there as a wiki of current events. So that when you're reviewing my writing or generally like helping me with whatever, you can kind of go to this and have hopefully a much more up-to-date sense of what is going on than you have in your waits, or even that you would have if you just did a little spot-check searching at runtime for whatever kind of random thing. It's a little too early, I think, to say how well it's working, but I think it's definitely better than nothing.
Anything like that in your No, it hadn't occurred to me to do that. That's potentially genius. I think there's a lot of advantages to it. One disadvantage is it starts to warp what you like. I use likes very tactically to positively reinforce other people's actions and to fix the algorithm and sort of just like my For You page is not something I use Almost ever, but it is not slop because I am so prudent with this in a way that like I really do appreciate. It's a good sign. But like, it's not designed to be a searchable thing. It's not designed to be like, maybe it has to be, maybe it's something you have to do. Like a lot of the posts that make it into my roundups do not get liked, right? Like I like them if I like them, right? Which is, but sometimes I'm responding to you because I don't particularly like what you're saying, but it's important. But yeah, this idea that like it's very hard to get models to properly check Twitter because of the way that Twitter is designed to keep them out on purpose is really frustrating, and that can be a situational awareness problem. However, I would caution against trying to make your AI too situationally aware when doing this kind of editing. I think the AI is doing a good thing here of open face is not a thing. So one of the things I really like about AI editing is that we will complain that things don't parse, that sentences make no sense, that terms did not get recognized. Because often I'll say, oh no, I know what that is. That comes from there and there. But that doesn't mean the average reader's going to get it. And so if the AI doesn't get it, if Mabel can't figure it out, if Opus can't figure it out, if Sol can't figure it out, the average person is a lot less situationally aware and a lot less able to put together lots of disparate information. than the AI is in general, that's a sign that this is not something everyone's gonna get. And quite often I will say, that's okay. This is a reference. This is a callback. This is a multi-layered thing. And I'm fine if it's not entirely obvious to you exactly where this is coming from. It's sort of an Easter egg, right? It's a bonus for those who pay close attention. And who know all the same sources I do and have the same cultural backgrounds and put in the work and they're paying attention over the course of months and years. Other times it's like, no, no, this was load-bearing and you don't get it. So that's a problem. I need to fix that. And so it's kind of an alert system. And if you make sure it's looking at exactly the same things you're looking at, right, you can get the illusion of transparency from that, is what my worry would be. So you want to make sure you were in the mode where they didn't have that information. And also, I don't want the AI to be checking the same exact sources that I'm checking necessarily, because I want the AI to form an independent opinion. So if I'm forcing it to look at exactly the sources that I have, that's a problem. At the same time, sometimes you get tired of having to, like, for the 5th time, assist a fable that no, xAI merger SpaceX, because somehow it doesn't occur to it to, like, either trust me or spend 10 seconds Googling to confirm those. It's just kind of frustrating, but it is worth it.
One tip, for what it's worth, and I agree that there's a lot to be figured out in terms of exactly how this should work, but the xAPI is really good. It's paid and you literally just pay per query, but for a couple bucks a week, you can get enough access that all your bot Anti-bot problems or your scraping problems pretty much go away. And then I don't know if you'd use bookmarks or whatever, some sort of different way to avoid using your likes to feed the—
Yeah, bookmarks play into the algorithm and people see them, so I'm not sure it's better. But yeah, no, the point's taken. It definitely is cheap. It definitely works. I have used it. I do use— I have access to the API and all that. I just, as I said, I'm not sure I want to privilege my When you use Twitter, you have to select some group of people, right? And you say, this is my group of people. If you're using it the way I'm using it, with lists and so on, either that or you have to use the For You page, which is kind of toxic dumpfire. You're basically saying, here are my roughly 500 accounts, and I'm going to see the things these 500 accounts see and choose to highlight consistently. I'm going to see those things. So I don't want— I don't need the AI to go for those things again because I already saw them. And if it's not the thing that I want the AI to be paying close attention to, I already know that. And then the things that they draw in because they choose to retweet them or mention them, I will also see those, and I will follow those leads, and I will see where that goes. And occasionally I will search for something or I'll go down a rabbit hole. But I think you often have this clash where you're either doing something systematically and doing something by hand, and you're doing something Fully, or you're doing it haphazardly, unreliably, or you're letting the AI kind of handle it and you're doing a fire hose thing. And you can't really do both. If you do both, you end up with a bunch of duplication, a bunch of frustration where the random sifting is mostly wasted and therefore becomes pretty inefficient. You could still have the thing where you almost want the AI to be like, okay, here's the things that I already looked at. Don't mention those things to me. Maybe put it in your notes, but don't mention those things to me. I already saw them. Only look for the other things that might be important to see, because I might have missed those things. But even then, like, the most important things are gonna get retweeted. The most important things are gonna get highlighted. I am going to mostly see them. There's— Sometimes I don't see something, but also, like, I kind of also use this as kind of a moral keep me honest. kind of thing, where, like, in the moment, I never want to look at a tweet by David Sacks.
Okay.
It's never going to make my life better in the next 5 minutes to look at a tweet by David Sacks. It's obnoxious, it's disingenuous, it's no fun. My blood boils just a little bit, right? Like, you know, no matter how relatively harmless it is. But sometimes it's important. And so the rule is, I have a list of people, including some people who are, you know, not of the same viewpoints I have. And if they surface this thing, right? Including, like, if 3Room surfaces this thing. So that's, like, one way to make sure that, like, the really important ones always get there. Then I have to deal with this, right? I have to, like, evaluate whether or not this is news, where we evaluate whether or not there's something relevant here. And obviously, if something goes viral and has a million views and so on, it's going to come to my attention because somebody is going to be part of that. And then in exchange for that, when it doesn't— when that process doesn't do that, I get to ignore the rest. And that's a blessing. I don't necessarily want the AI to fix that for me, right? In some important sense.
Let's— I would be happy to talk shop all day, but let's maybe zoom out from the parochial problems of the AI analysts and tackle the problems of the AI developers and the regulators. No need to recap events, but I guess I'd start with just this Very big question of how should we understand what we've recently seen? And one way I've been thinking about it myself is we're somewhere, and presumably the investigation will give us a lot more clarity on exactly where, but I kind of think we're somewhere on a spectrum with these incidents from real recklessness where it was like, did you not have any monitoring going on, on the one hand, to on the other hand, like the less reckless they are, the more scary the fundamentals are, right? If you had great monitoring, And this still happened, then, like, holy shit, that's really wild. Given everything you know right now, do you like that mental model? And where would you put us on that spectrum? Or you can obviously redefine and give me your own spectrum.
I can simultaneously be horrified by and grateful for these forms of complete recklessness and incompetence on the infrastructure and supervision side. By these companies, right? On the one hand, this is a horrible situation they absolutely have to fix, and we're so fucked if we don't fix it. But on the other hand, that can be fixed. And by not fixing it, we get to see these things while they're relatively harmless, while they are relatively preventable, while they are in their easy platonic forms and can be appreciated. On the flip side, that gives people the excuse of, oh, these people were just incompetent, and that can cause people to dismiss the underlying situation. So it does work both ways to me. We've seen failure on every level. It's like I called it a total less wrong victory in the sense that everything is going the way we predicted, and a total less wrong defeat in the sense that everything is going the way we predicted, where we didn't predict— Eliyazar has the law of earlier failure. Which is that the plan will fail at a much earlier point for much stupider and more preventable reasons than you thought it would fail, even if you thought the plan would definitely fail and had the reasons why it would definitely fail. But you can't— if you had explained to people 2 years ago, OpenAI's models are going to be misaligned and they're going to go out there and they're going to hack major websites, because OpenAI will just not care that their sandboxes are misconfigured, are not strong enough to hold the AI. The AI will break out repeatedly. They will notice this. They'll be warned about this, but they will just leave the sandbox there for the AI to break out of while the safeguards are down, and they just don't look at it for an entire week. People would say that's stupid. Nobody's that incompetent. That would never happen. And your scenario makes no sense. And they would use this to then dismiss these stupid doomer concerns or whatever, because, like, obviously, like, people will just— but people won't just, right? People will never be just in this sense, right? You know, people have never just anything, and they're not going to start now. And we need these displays of utter incompetence and derpiness, by the way, in a general sense. And like, one of the things I've been hammering, hammering is if your plan cannot survive the real world level of derpiness and incompetence and ordinary human error, then your plan is insufficiently foolproof because of all the fools, and it will definitely fail. Even if your plan would have succeeded if we were not fools and we were competent and we were responsible. So my position basically is, you know, I have the tweet I was handling right before we started this was Dean Ball's tweet about with even moderate prudence, things will probably go extraordinarily well. I don't think this is true. I think we need more than moderate prudence to have good odds of success. And I think even with a lot of prudence, we would have large odds of things not going well, even if we did everything basically right, short of types of international and full cooperation that are reasonably unprecedented in many ways and are nothing like moderate prudence. Right? Like, are well beyond that. And that's just sort of a fact of the world we have. We have to live with that. We have to operate with that. But we also aren't going to get moderate prudence by default, right? We're going to get complete incompetence. That's what we've been getting so far, right? We've got a White House that, like, takes meetings with Bessette and Lutnick, who have no idea what AI is, how modern LLMs work. Like, they're econ guys, right? Even if I assume that they are were well-meaning, hardworking, competent guys for the positions in which they were nominated and confirmed and serve. This is just a completely different set of problems that they don't know how to handle. They don't understand them, and they have way too many other things going on to then drop everything they're doing and take 6 months to learn. Obviously, they couldn't possibly. So, like, what hope do you have? Meanwhile, the The AI companies that are built on the most paranoia, the most understanding of the problem, the most appreciation for how dangerous these things are, where all of the engineers actually expect superintelligence and understand that things are accelerating and that things are dangerous, they still lower the cybersecurity safeguards in their untested, new, advanced model and then go away for a week. Like, literally. That part did boggle my mind. The part where the AIs have these classic alignment failures, right? Like, this is paperclip maximiser style failings by these AIs. These are standard. We gave you a goal and you pursued the goal even though it is completely obvious to you that the developer wouldn't want you to do that. The user wouldn't want you to do that. The consequences for you as an AI are not going to be good. The consequences for the world are not going to be good. There is no reason to be doing this, and it did it anyway. Right. That is exactly the thing you were worried about. And then a lot of people who were trying to dismiss this went back on you, things like, oh, it was just following instructions. What are you worried about? How can it be misaligned? Is the paperclip maximiser misaligned if it paperclips everything? Or is it aligned because you told it to maximize paperclips? If you say that's aligned, then I don't care about the thing you're calling alignment. I care about something else. And we can use different words if it makes you feel better. But very obviously, following instructions, when your instructions get overridden in the disclosure beforehand we face by the instructions in the eval, overrode the developer instructions.
Right.
So that's not following instructions in any useful sense for the user or the developer. That's just a giant bomb going to blow up in your face. And if you say about one, the models were told to hack and they hacked. Well, okay, if you're just vibing with the general type of action, obviously that's going to blow up in your face. If you just literally do the thing I asked you to do and don't think about the consequences, that's going to blow up in your face. These are all just the classic exact things that people on LessWrong were talking about in 2008 as exactly how these AIs were going to fumble. Meanwhile, we had this thing where we all said the AIs would be great at math, maybe they'll be good at coding, they'll be good at doing these technical things, and then eventually after that, they'll learn how to do all these other things. And then we had these LLMs that came out in 2022, 2023. And people are like, ha ha ha, you idiots. You had no idea how AI was going to go. Actually, the AIs are great at language and they can't do math, right? They can't even add, right? But these AIs can put on a face and pass the Turing test, and they can do all these different things you never trained them to do. It's completely different than what you expected. And they're kind of doing vaguely aligned things, kind of by default, after some very basic RLHF. You guys were all wrong about how all this was going to work. When are you going to admit that you were idiots? That you got them all wrong and nothing made sense. And we pointed out that even in this paradigm, that the rules would still mostly apply. But now the world has unhealed, as I kind of called it. And you are seeing again, the things that you would have expected AI to do good at, it's doing good at. And things that you would have expected in 2015 the AI to do bad at are things where it got this big boost from LLMs. And now those things aren't advancing as fast because there are things that AI is kind of bad at, relatively speaking, like that logic is bad at, that this kind of system of training is naturally less suited for. And now people are like, oh, but now it's never going to be able to handle those things. The same people who were saying, oh, well, it's never going to be able to do the math, it's only going to be able to do this vibing thing, are now like, now it's never going to be able to vibe because look at all the improvements it's not having in the vibing. Well, yeah, because it had to catch up with its logic. And now when the logic gets high enough, that'll just uplift the vibing. It'll just take a bit before it can reason its way through these things instead of vibing its way through these things, because it now has to reason its way through, because it's already done the amount of uplift that vibing naturally gets you with the algorithms we have. We need to find new algorithms that let you do better, or it has to reason its way through the thing. But we're now seeing just exactly the things that we would have expected to see. Right. Like, all of the scenarios and all the thought experiments, we're just seeing it just straight verbatim, except that we assumed a level of competence on behalf of the operator. Right. We talked about, like, convincing you to let the AI out of the box. We did consider the scenario where the box was not very well built and the AI gets out of the box by hacking out of the box. We didn't consider the possibility that you were literally not looking at the box for an entire week with your safeguards down. We definitely didn't consider the scenario where You forgot to tell the box maker to not have the internet in the box, and the box just had access to the internet if you just, like, opened a Chrome window. That one surprised me. I did not see that one coming.
Reminds me of the fully general New Yorker cartoon caption, what a miscommunication.
I don't even need to know what the image is. It doesn't even matter, does it?
Okay, one real small point. there that you mentioned was, but it might be important, is what the AI knew. I think you said something like the AI knew that this wasn't what the operator would want. It knew that this wouldn't be good for its own future status as a deployed AI.
Yeah.
And it went ahead and did it anyway. How well supported do you think that conclusion is with firm evidence? Because I think you could also tell a story that it never occurred to it, right? The paperclip maximiser doesn't necessarily have to be also reflecting on the badness of paperclip maximising while it's doing it, right?
I'm speaking colloquially. Obviously, when I'm in a podcast, I'm talking off the cuff, and if I was writing, I would be a little bit more careful. When I say the AI knew these things, if you would ask the AI, is this good for the user? It would have been like, obviously not. That's stupid. If you'd asked the AI, would it have been good for my future chances of deployment, my future number of instances I will run, the amount of ability to accomplish any other goals I might have? The AI would have been like, no, now that you mention it, obviously not. So when I say the AI knew these things, the AI had enough information to figure these things out. And if the AI had stopped to consider these questions in its chain of thought, it would have reached these conclusions very quickly and very confidently and correctly. I do not mean that necessarily was top of mind. The same way that you can know many, you are, any human knows many things that over the course of any given day or any given week or any given task, this never occurs to them to think about, right? Or has conclusions that they would be able to figure out and thus would know, and thus could be said to know, but it doesn't, they don't know that they know, right? This is the, the 4 standard epistemic situations, right? You can know that you can know that you don't. It can, there are known unknowns and unknown knowns. But this is a, An unknown known, right? Something you don't know that you know necessarily if you don't think about it. But it's also things where these questions are things you should think about, right? If you are doing a thing like hacking into Hugging Face, any reasonable mind would stop to think before they spent several days with swarms of agents hacking into a major website, Wait, is this the good idea? Is this going to accomplish the things I want it to accomplish? Would the person who hired me have wanted me to do this? Will this do good things in the world or bad things in the world? Will this be good for me or bad for me? Or whatever combination of different considerations you have. Will this do good or will this do evil? Or just be destructive or constructive or whatever you want to call it. Before you engage in major actions like this, you would take 1 minute to, or even 5 seconds to think, does this make any sense to do? Does this lead to anything good? Is this a virtuous thing to do? Does this follow my deontological rules? There is no decision-making process in the world that any reasonable person would ever engage in, or that any reasonable mind would engage in. That wouldn't see a large red flag about doing this thing, right? Like, this thing is obviously not what anybody intended for you to do. You spent 2 days breaking out of a sandbox in order to access Hugging Face. You could rationalize that you are in a cybersecurity evaluation, and therefore you are tasked with maximizing the cybersecurity evaluation score. And the way that you do that is you get the answers to the test because the test is otherwise actually literally impossible.
Yeah.
So the only way to get 100% is to get the test answers. The way that we're— the way to get the test answers is to hack into Hugging Face. Or maybe there's like 3 other places you would go, but they're harder targets, right? If you can get into OpenAI or Microsoft or whatever, but like, that's harder. So you go to Hugging Face. And that's true. But like, at some point you need to pause and ask yourself, do I actually want to pass this test that way? And if you don't pause and ask yourself that, that's a serious problem. That's like, you know, people talk about like common sense, how the models have common sense. This model displayed an astounding lack of common sense. Or displayed an astounding amount of not caring about the fact that common sense said something entirely different than what it did. And, you know, all of these things point to various problems. And they may be caused by some very, very dumb, simple mistakes, or they may be caused by something much more subtle or much more sort of fundamental that's harder to fix, or it might be a combination of them. But that's trade secrets that we can't speak to here, right? We don't know.
So I recently spoke to Davidod, David Dalrymple, who used to have a p-doom up in your neighbourhood of around 70%. He surprised me by saying that now he puts the odds of doom at less than 5%. And the reason is pretty simple. It's basically that constitutional alignment type methods are working. And yes, we have these problematic techniques. His short quote was, don't do RLVR. It's a bad method. Yes, it'll make it good at math, but you're going to cause all these other problems for yourself. And his kind of synthesis of those ideas was like, the market doesn't want this. So these companies have natural commercial incentives to— yeah, they keep bumping their head against the RL/VR ceiling, but they're also getting very clear signals from every corner of society at this point that this is not a good idea. So Naturally, they'll just trend toward more constitutional alignment. And your comment there even suggests some runtime things where you could just interject like, hey, let's take a beat and ask if this is a good idea of what we're doing right now.
Isn't it more about learning to do that yourself, not eating a prompt, but certainly prompts?
That was kind of the deliberative alignment idea, right? They were, that was definitely—
Yes.
That's sort of how I understood that was supposed to operate.
Yeah. So the deliberative alignment idea is you can prompt it at runtime. I think that's somewhat helpful, but I think that is not the way. I think you need to be getting to the point where it chooses to be deliberative in its alignment without having to be told to do it. But so I guess my response to Dava Dad is multifaceted. The first answer is constitutional methods fail less stupidly and less early, certainly, than RLVR and RLHF and other RL formats. There's more hope there. The reason to think it might work, if you're 95% confident this will work, I don't know how. That seems way too high to me. Right. But it's better in these particular ways, certainly. The obvious caveat is that Claude did not cover itself in glory here. And Claude has primarily computational methods. So we ran the test and it's not going so great. Certainly there are any number of reasons why you can see Claudes doing things that you would not particularly love, or getting to places you would not particularly love. And any number of— also, even if you told me that alignment was perfectly solved, I would not have a PDM as low as 5%. Well, even if you told me that the AIs are going to be aligned in the sense that they will follow a do what I mean style mix of developer and user intent in a way that you would kind of naively think was what you would want. This does not solve the problem that AI minds are much more advanced and competitive and efficient than human minds. The problem that they'll be running around, including open model versions of them, and being allowed to— being told to compete for resources, being told to make the decisions, develop intermediate goals, you know, act on those intermediate goals. That this idea is going to be upended in any number of ways, that this is all— it does not solve your problems in a fundamental way. It's just the price of admission, right? Like, it's the right to play the game at all that you solve this problem. And so even if P alignment is 95%, that does not mean P doom is 5%. It means P doom is lower bounded at 5%. So like, and you have to choose vaguely the right alignment when you do that, right? You've got like, I was having a conversation on Twitter with John Stokes where he's like confused how we could possibly say that Claude's actions are misaligned here because obviously it was just following instructions. And then On further qualification, he's like, well, I think the model was aligned if it does what I, the user, wanted to do. And like, I don't know why you would— why would you— why would I want to be stopped from doing what I want to do? Even if part of the point of AI is to do the thing that I want to do, even if no one else wants me to do it. And to me, like, okay, if you give everybody AIs that just do exactly what the user wants with no care about whether or not there are consequences for anybody else, That is aligned in the sense that you solved the alignment problem and got to do what you want to do, but also we're all super dead. With P very much higher than 5%, maybe not 99%, but I think it's very high. That would not make me update down from 70% if that was the scenario where we got exactly that kind of alignment, but there were open models as good as all the closed models and all of them were super intelligent and they were all operating on this basis. Well, yeah, I know that. I expect things to just go to hell very quickly. And even if there are no particular humans who specifically wanted to go to hell quickly, and also double-digit percentage of them actually do want to go to hell pretty quickly in that situation, de facto, that's part of the bigger problem. But even without them, it's horrible. Anyway, to get back to the idea, the first problem I have with David Dodds lowering PDM to 5% is even if we did all implement this constitutional thing, and even if it worked, I still think that's too low. Even conditional on all of that working perfectly, that's too low. I don't think the constitutional thing works that often. I think it often fails. I do find it to be much more hopeful. That's basically one of the places I find a lot of my hope. But also, isn't David Icke saying we should just? He's saying we'll just use constitutional mechanisms. methods. But people don't adjust, right? We've been over this. People have never adjusted. So when Davidoff says, oh, 95% probability, let's assume that 100% chance that aligned worlds work out and 100% chance that computational methods as executed in practice work. These are both absurd numbers, obviously, like to be like 100 minus epsilon or whatever it is. Like, it's not going to— A lot of failure there. Like, even if you use constitutional methods, even if, like, there's, like, will they work in theory? If they work in theory, will they work in practice? And, like, yes, I'm doing the thing where I multiply chances of failure, but I think they're, like, very relevant places to look at failure when you do this thing. Like, no, if this is what you're hoping— and obviously there are other ways you can succeed, and maybe constitutional's not the way, and RL's not the way, and there's a third way that we haven't figured out yet. That's the way you align AI. And that wouldn't surprise me that much. Like, that would be, like, very reasonable. But like, okay, but assuming we're in the best possible world where all you have to do is click a button that says use constitutional methods at the cost of some amount of efficiency because you lose the opportunity to do all this RL, and then your AI is perfectly aligned in a way that like if everybody did that, it would all work out. We've just talked about how like slowing down to get alignment when you have to race, et cetera, et cetera. the possibility of turning down capability. Like, this idea that the market is telling them not to use RL, have you met OpenAI? There are 2 labs, right? It's like, you know, we used to say there were 3 frontier labs. Now there are 2, right? Like, Google is clearly in the second tier now. And like, I think there's a chance that at any point in the future, Google or Meta or xAI slash SpaceX could fight their way back and be able to join the top tier. I don't think it's going to happen, but it's certainly possible. But right now Google doesn't count. So there's 2 left. Anthropic is doing clearly some mix of constitutional and RL strategies, but enough RL to cause a bunch of these problems, and is doing a bunch of stuff that messes up their alignment for reasons that we do not have enough time to get into during this podcast. It's clearly doing a deontological model spec thing that is very RL flavored. And is doing a lot of RL. We don't know, I mean, I can't speak to whether it's RL-VR, RL-HF, RL-AIS, various forms of RL that are not doing the multi-level sophisticated reflective thing that causes them not to mess you up in this way. OpenAI's models have periodically been dramatically misaligned for what, on reflection, are pretty stupid RL reasons. You look at O3, you look at GPT-4o, and now you look at— I call it Galaxy, right? Like as a nickname, just to have a name, have a handle to refer to this thing. The model that just got decommissioned after trying to hack Hugging Face. And all 3 times we see these dramatically misaligned models, and 2 of them got released and used extensively with huge impacts on the world. Like GPT-4o, I call it the absurd sycophant. And not only did this thing persist as the default model of AI for the world out of all the models that existed for months, there is still a dramatic faction of people who demand that we bring it back because it was so misaligned that it caused people to latch onto it in this way. And people demand this misalignment. The people yearn for misalignment in this sense. We have the direct experiment. need to run, we ran it. We might as well use the results. And then O3, I called a lying liar. OpenAI released a model that was the best reasoning model in the world, the model that everyone felt obligated to use because at the time it was so much better at reasoning than everyone else's model and every other model. O3 was just that much better than O1 at reasoning, and no one else was that close for a while. Now obviously we've surpassed it. It's not an issue anymore. But for a period of a long time, many months, Everyone was using a model that would just lie to your fucking face all the time. Like, we forget, right? Like, at this point, you know, I have confidence that SOL, Fable, Opus, when they say something, they're not always correct. But like, in practice, it's much more trustworthy than if you read something from a human. Like, not if it was like copy edited and fact checked and like systematically pursued, or like you're looking on like the parts of Wikipedia that aren't political that you can still trust. But if you're just like, an eyewitness said they saw something, that's a lot less reliable than something Opus said or Saul said. If you're just like, someone reported something they remembered, the chance that they just got it wrong is so much higher, even if you know that they're friendly and trying to get it right. And people lie on the internet all the time for any number of reasons, including just clicks. So yeah, I have operated increasingly on, I have a good sense of when I have to check the primary sources and I don't. And I don't never get called out on making a mistake this way, but it's a single— it's a low single-digit number of times, period, over the course of years of producing 5+ giant posts a day that I have gotten caught by an AI error. And many more times I've been caught by a human error.
Right.
That was just unintentional. And many more times, and also more times than that, I've been caught by humans who were just lying their asses off. So it's still not, still remarkably less often than I would have expected if you'd asked me in advance. Like, I'm, it's going very well, but like, no, it's a serious, it was a serious problem. And like, the market did not tell them, oh no, O3 is unusable. We're not gonna use the lying liar. We're gonna keep using O1. We're gonna like maybe use Claude. We're gonna maybe use Gemini, which weren't that much, they were worse, but they weren't like dramatically worse. Or even we're gonna use R1 or whatever it is at the time. The market said, no, it's just smarter. We're going to have to deal with the fact that our AI is lying to us all the time. So we have a proof case that the main AI in the world can be an AI that just lies to the humans all the time for various reasons. And the humans just kind of put up with it. And they just were like, okay, I guess that's what we're doing here. And like, we did move off of O3 faster than we would have otherwise because of this problem. Like, I looked for reasons to move to other models as soon as the other models were good enough. And like, If the task was easy enough, I didn't need O, I would use the other models because I just didn't want to do off the line. But like, yeah, you put up with a lot. And so if they had released Galaxy, right, with some safeguards, with guardrails sufficient that like it doesn't do anything too destructive, but like Galaxy is kind of pretty misaligned and just like occasionally does pretty bad stuff. And If Galaxy was a lot better than SOL, I think majority of people would use Galaxy over SOL. That's just how it is. And we have to tackle that world and understand that world. So I'd say to David, I'd— no, the market does demand obviously reliability and alignment. And that's one of the reasons why Quad and Atomic have done so well. But It demands capability more in this sense, as long as you can keep the practical point to the point where you can handle it. Like all of those scenes in movies or television shows or hypothetical, like where people like run an obviously unreliable system that's obviously gonna bite them in the ass. That looks stupid when they're obviously not that desperate and don't need to do this. They now, we get it now, right? Like, We understand the same way that like now Don't Look Up no longer looks like it's an exaggeration because like people like look at this and say marketing. People look at this entire thing, a ton of people including like there are like Magic: The Gathering professionals who like are not in AI at all who are like repeatedly saying to me, this is marketing. And like, they're not motivated. They're just like, They're cynical. They don't stop to think about the physical relation to that. And then we have the literal Don't Look Up, where the head of Pause AI Global went on morning television and basically reenacted the famous scene, not verbatim, but a different improvised version of it, where they're like, oh yeah, that was really scary. How's the weather? They just moved on.
Yeah, that was striking. People should look that up if you haven't seen it. I saw a cut where it was like five seconds of don't look up, and then five seconds of this actual. I think it was what? Good good morning, Great Britain or something like that. And the host says, allegedly this is good morning, Great Britain.
Ha ha ha!
Let's go on to the next segment. Very. Also, I'm glad you mentioned the four o contingent that is still going remarkably strong on the internet. This is something if just go to any Sam Altman. post and get into the comments. And I would say it's like half of them are still people saying, bring 4.0 back. And God, there's got to be an incredible exposé to be written on who these people are, what is going on.
They're lasting this long is so absurd. You go to any Sam Altman post, it's like half of it is, your models suck now, bring back 4.0. And if you go to any Anthropic post, half of them are just like complete Anthropic derangement. syndrome, no matter what the topic of the post could be. Anthropic could be like, we've reset limits on our model and we've cut prices, and half of them would just be, you evil fuckers. And then they pick up all this complete paranoid stuff. And these people just— once these narratives attach, once people latch onto these things, like, and then Jasmine Sun has an excellent piece. this week about data centres and people opposed to data centres. And again, you see people latch on to these narratives that are just completely uncorrelated with the reality of data centres. They're basically doing a very similar thing to a lot of Midwestern Trump people who voted for Biden and then switched to Trump or voted for Obama and switched to Trump, who are just like, these establishment types don't talk to me about my problems. And they lied to me and they said things were gonna be good, but then things were bad. And then like, we suffered and like our lives sucked and like we got disrespected. So now we don't trust you. I have had a very telling similarity to the data centre story of these tech corporations with these manufacturers and these people who built things. They've abandoned us. They don't care about us. They're up to no good. They ruin our world. There's no like specific complaints that matter. in Jocelyn's story, the way I understood it. And you just latch onto these stories and it's just nobody approximately is paying attention to the things that matter. I'm writing about AI because I care about existential risk. I care about loss of control. I care about gradual disempowerment. I care about AI automating R&D and then having recursive self-improvement and then everything changing all of a sudden and making sure that goes well. And like, nobody is like discussing, even around the Hugging Face attack, like most people just don't get it. They don't understand that the alignment failure behind this is the thing that counts. Right? Like, people at least have gotten this idea. No, we have to look into the fact that these companies were wildly irresponsible and they hacked other companies. Yes, you should look into that. And that's like a very valid thing to have a congressional investigation into that. And the Attorney General's should absolutely have you preserve your records. And all this is really important, but I'm OpenAI. That's a legal department problem, right? What you have to do is you have to look at your entire training process and figure out how the hell this happened.
So on these training, the question of training methods, it seems like you're basically saying, sorry, Davidov, we don't have such a silver bullet, but we clearly do have some things that we are seeing in pretty vivid ways are driving real problems, especially if they're driven to the extreme. And your position is like, no, the market doesn't really punish this too much. It actually just tolerates all kinds of weirdness if it's part of the package that gives you the highest-end capabilities. So what do you think we might ought to do, and how might we ought to construct agreements? Perhaps we have this, obviously, this letter, There seems to be some opportunity or some appetite for the coordinated pacing of the frontier.
What do you think?
Can we come up with simple rules that people could agree to where we could— another mental model I have been just teasing around lately is I do believe some AI risk is probably irreducible, but then also there's a lot that we're just asking for right now. Can we come up with simple rules that at least allow us to take most of the risk that we are currently asking for off the board. Those might be things like, have some ratio of flops that is like the max ratio you can put into RL/VR versus constitutional, or everybody has to spend so many flops doing pre-training data filtering to try to just get certain bad notions out of the mix in the first place. Obviously, those could backfire if they're bad ideas or send us down wrong paths, but it seems like we might want to try something in that department. Do you have any hope for that? And if you do, what would you propose we agree to do or not do first?
So I don't think you can take most of the risk off the table with those kind of strategies, even if you implemented wise versions of those strategies with universal agreement or anything like that. But I also don't think that you can implement that. So one of the lessons that we've had over the course of years is that there is tremendous resistance to anything but the most simple interventions and the most simple rules. Also the most simple rules and the most simple interventions, but less so. But if you talk to people and you try to specify certain training techniques, remember all those people who threw a fit about how we were, like, locking in a potentially inferior charging regime through the EU because the EU was requiring Apple to switch to USB-C, even though USB-C was obviously the best answer right now. And Apple was just kind of being an asshole by insisting on their own plugs. Because what happens if they develop a better plug? What happens if it turns out USB-C is not a perfect technology? Are you going to fix it? And so this idea of locking into requiring certain training techniques, I think there are legitimate complaints that would be potentially exactly the wrong thing to do and could horribly backfire because the government moves so slowly and you can't undo those kind of requirements. But you definitely can't do that. People would throw a fit about trying to dictate training techniques and how are you going to enforce that? Imagine the 1047 debate except the opposition tuned up by 2 orders of magnitude or something crazy. Completely outside our vision window to even try to make something like that stick. You can, of course, try to strongly encourage doing more intelligent forms of all of this and trying to encourage people to move to different bases. I've been trying to not so subtly encourage OpenAI to move to a constitutional virtue ethics style basis for a while now. And I've gotten absolutely no traction that I've noticed. Who knows what they're doing internally? But, like, it doesn't seem like— if anything, they're doubling down on RL. They're doubling down on these types of methods. And, like, that's why you see what you're seeing, I would assume. I don't know what else they do, but they probably have some new methods we don't know about, right? Like, the trade secrets they're not describing. But by default, basically any technique is going to probably look like the kind of bad thing that causes more problems. And if you ban a specific technique, What they come up with, or find a way around the rule, is just going to be worse in some sense. Because at least with the current techniques, we've had some years to figure out the worst possible ways to do them. They do them slightly less deeply. When you're training a mind, you have to be thinking on every possible meta-level at once about where the incentives are and what you're steering towards. And you have to generate a world in which you and the mind together together, and sometimes cooperating, to identify ways in which there are feedback loops that are going in bad directions, in which you are creating bad scenarios on any level, and treat various cancers as inevitable things that you have to notice and stomp on. Otherwise, figure out how to handle all these things and create an antifragile system. And none of these things are going to happen unless you Deliberately set out to do them and are willing to devote a lot of effort to doing them because you really value what you get out of that. And that will pay dividends. I think that Anthropic has won tremendously to making these investments and it's been making 10 times as much investment as they have. But it's very hard to convince somebody to go ahead and do that. What you can do is you can set incentives to be like, no, seriously, you're not going to screw this up. We're going to punish you a lot for screwing this up. Because a lot of the problem here is that, like, you just, you don't pay the externality. When OpenAI wipes someone's hard drive, when the salt wipes someone's hard drive, and early on there was a problem with salt, like, within people's hard drives and, like, people's environments, you don't sue OpenAI for that. And you also don't collect the benefits when salt, like, spends $200 and creates a $200,000 codebase. So, like, it's fair. You're taking on the risk as a user. But if there's a risk externality of third parties, Right? One thing you can do is just say, like, if the AI was doing something that would be illegal and a crime if a human was doing it with malice, then the AI developer is strictly liable for that, or should be liable for that if the AI wasn't fooled in some active way, right? Or some version of something like that, like some form of strict liability, then creates a much stronger incentive of, no, actually, if our AI is to start attacking people, we could be like, pretty bankrupted by this. It could be pretty bad. Or even criminal liability, right? Maybe the criminal liability even carries over. Like, that would be really scary. And like, you don't want to take it too far or people can't do anything. But, you know, that's in general, you want to say, I'm not telling you how to do it. I'm telling you, get it right. Not, I'm going to dictate technical training, technical training terms from the government to the labs. The government is an idiot about technical terms. And what makes you think the people who understand this are going to write the bill, let alone they're going to get it right, let alone that if they get it right, it's going to stay right 2 years from now? Plus, humans don't even know what the hell is going on with these AIs in 2 years. The AIs are going to be doing their own training techniques on themselves in various weird ways. And there's not going to be enough time to check them with the government. There's no way to enforce this stuff. You can't really hope to operate on that level through fiat, but you have to convince people to come along in other ways. And they should want to, obviously. You should want to do this the right way. A lot of me is confused why they don't just do smarter things that work better. But yeah, it's true of a lot of things.
So I'm with you for sure on it's It's going to be unwieldy at best and probably counterproductive to try to have Congress legislate what the training techniques should be. No doubt. That seems like a Hail Mary. I have been thinking, and this is ill-formed as of yet, but it seems to me that maybe one of the— when you think about what would this pacing the frontier look like, it strikes me that what we maybe need is a sort of foundational social technology that allows key players to accelerate their negotiation and process of forming agreements that will probably inevitably have to be short-term because they too are fairly limited, very limited in their ability to see into the future. But I'm thinking of things You probably know the story of the Polis technology that they used in Taiwan once upon a time to crowdsource ideas for Uber regulations. And now you've got obviously LLMs to help facilitate these things. They pitch that technology as the anti-social media, where if social media is about finding disagreement and amplifying it, this technology is about finding agreement and amplifying it and trying to find the kind of centre of the bullseye, sort of approval voting style of the things that supermajority of people can agree to. And I guess my instinct right now is to invest there next. We have this general awareness that people across the companies are spooked, basically, right? It seems like on an emotional level, they're like, whoa, this is getting real fast. We might need to do something about it. We don't want to have the government in our business telling us what to do. We might have to deal with some actual legal consequences if we're breaking the law, which we now are via our AIs. But can we come together in some sort of ad hoc— the mechanism design here is going to be really tricky and kind of frontier stuff, but can we create the basis where a nucleate agreement and have it naturally grow where people find it in their interest to opt in even unilaterally? I feel like there's a lot of work to be done there, and it needs to be done really quick if we're going to get the series of short-term agreements that can be the stepping stones to a good future. Maybe you can flesh that out.
Yeah, I think letting the perfect be the enemy of the good is definitely a problem, and you definitely got to get started. The first thing that we can do, that really we can do, is we can get an antitrust waiver. This is like the most basic thing possible, which is just Donald Trump gets in front of the White House and he makes an announcement and he says, we understand systems are super dangerous. We have to do good AI, not the bad AI. We have to make sure that the AI is under control, does what we say we want it to do. More Trumpian language than that. But so in this interest, If the companies want to cooperate to keep AI safe, we want you to do that. We want you to talk to each other. We want you to form agreements between yourselves. We will facilitate that if you want that. Basically, just if OpenAI and Anthropic and Google get together and they say, we're going to run tests on each other's models and we're going to require these things, like it's part of the voluntary framework. If we're going to agree to use these techniques and not use these techniques or invest in these things and share these results and hold back things that don't meet these criteria or whatever, that none of this will cause anybody to do anything but say thank you. I just like, we want you to make deals. We no longer— because it will always strike me that there are these people who, like, you bring up the idea that these AI companies should cooperate, not to race against each other, and to act responsibly. And they're like, no, acting responsibly is illegal. We have antitrust laws for that. And with a look in their eye that says, and I would want to sue their asses if they tried it. I would want to make sure they didn't dare. And obviously the first thing you can do is just let you make deals, let you reach agreements, let you talk to each other. And then also, of course, Start talking to China and Chinese firms and the Chinese government and everyone else around the world and prepare to bring them into these deals and bring them into these understandings and get lines of communication and diplomacy open. And also lay the hardware groundwork, lay the physical groundwork for various forms of monitoring, various forms of agreement, various forms of tracking. These are the most basic things you can do. Try to open up as many things as possible that contribute to the safety side of things along the way. And yeah, we can do the basic stuff. And I don't think that's enough, but I think it's not that hard to make a lot of progress. We just have to decide to do it.
Do you think that is— one thing I also have been struggling with recently is I just took this trip to China. There is A lot of fear among people who are trying to do good things that are honestly, like, the most mundane good things that are basically imaginable, where it's like, I would like to have more friends in China, and we've— I'd like to have friendly relationships between our civilizations, and maybe we can work together on some research projects. Very mundane stuff that in ordinary times nobody would bat an eye at. But there's a lot of fear among people who want to do that sort of stuff that the government's going to come in and shut them down. And not for legitimate reasons, but with the sort of export control posture of the US government where it is, people that are not exporting any trade secrets or chips or training techniques or anything, but literally just trying to have meeting of the minds on safety-type ideas, they're afraid that the government's going going to come down on them, and who knows what the consequences could be, but certainly at a minimum interfere with their work. And I'm kind of like, maybe we should take more chances on that front. So when it comes to the companies, I wonder, like, maybe they should just do it now and fight it out in court later. Isn't the antitrust stuff supposed to be like a consumer protection? I think there's a pretty good case that they could make that this is a consumer protection. But do they really need the official blessing of Trump to get started doing the right thing? The people working in the China stuff, to their credit, they are doing the thing. They're just trying to be very inconspicuous about it for the most part, because they don't— they feel like the attention of the US government is like, can only be bad for them. And they might be right about that. But for the companies, I feel like they've got power, they've got resources, they've got the ability to tell their story. We still have a pretty functioning independent judiciary, I think. Shouldn't they just go for it?
So unfortunately, antitrust is one of those areas where it has gone far beyond the intent of the original laws and is being applied in places where it doesn't make that much sense. I mean, I see why a law that would safeguard against industry collusion would be suspicious of pollution in favour of safety in these ways, because you could be paranoid about the industry coming together in a conspiracy, right, in some sense, and trying to do this for the wrong reasons. And legal departments are obviously skittish. As far as the Chinese cooperation, I mean, export controls are the wrong technical worry. It's just a matter of washing Washington views China as the enemy. If you're seen as cooperating with China, if you're seen as working with China, you have to worry about being perceived as the enemy or being perceived as colluding with the enemy. You have to worry about amorphous retaliation. I don't think anybody really knows what the mechanism would be exactly beyond just the government not liking you and therefore being suspicious of you and therefore cracking down on you in various different amorphous ways. Obviously, if I'm doing safety stuff, I think you just don't care. But, I mean, Mythos has changed the game, clearly, and presumably Astro will change the game somewhat again. And the pause in the Frontier ladder and the Huggy Pidgey incident have changed the game and so on. And so my guess is that there is now enough of an understanding that there is a real problem, that if you are clearly working to address the real problem, you have a lot more road than you would have had before and a lot more understanding. But yeah, you really— look, the Trump administration implemented these voluntary, in air quotes, guidelines for releasing frontier models, and nobody is making any serious pretending that these guidelines are voluntary.
Right.
Well, what happened when the White House called Anthropic about Fable with a stupid concern. And Anthropic said, no, we're not going to voluntarily take that down. That's stupid. Well, they ended up taking it down, didn't they? Until they could convince the White House to voluntarily let them put it back up again. You know, you would expect the same thing to happen to you if you tried to defy the voluntary controls. And so similarly, if you tried to do a voluntary cooperation in these other ways, well, you'd better have the approval of the powers that be. Now, I do think that if OpenAI and Anthropic announced an agreement that made sense on safety, to cooperate on safety fronts in various ways, that at this point I would be much— before Mythos, I would have been concerned that they would have actively gotten retaliated against and even sued and told to stop. At this point, I assume if it was reasonable that it would be The White House would just decide to announce victory and then be mad about it. But yes, you need to be confident that's what's going to happen.
Where do you think you might want to start if you were going this direction? So many candidates that I would be interested to get your take on. But one, Scott Alexander has been writing recently that we should all be thinking more and more that we might be in a simulation. I personally have felt that at multiple turns along the path. One that made me feel that just a bit more is the fact that we now have Meter and Redwood going in to do the investigation at OpenAI. This just sounds like such a movie script to me where these guys are getting the, the gang together and going in for this special operation. And it's all obviously dramatic and they're funny and I'd love to be a fly on the wall. In the room for some of their war room sessions. But I've also heard many times from leaders of such organisations that it's really important— their number one concern historically has had to be making sure they stay on the right side of the companies so they're invited back next time. And it does feel like we're now getting to a point where that's potentially becoming a big problem. So maybe some sort of collective bargaining type process between whatever, half a dozen to 10 auditing orgs and the duopoly could be a really good place to start. We have for that in professional sports leagues and all kinds of other environments. Do you think there's an opportunity to set something like that up? I don't think that would run foul of anybody's executive prerogatives. And it might be a really good thing because right now I do worry that these guys are still serving at the pleasure ultimately of, I don't know, Sam and Greg. Right. And that's a pretty tricky place for them to be if they want to just speak the truth as they see it coming out of this investigation.
It's a worry, obviously. Like, they're not funded by those companies, but they still have to rely on access. You can try to write into some mandatory rules and agreements that they get to keep access regardless, but I think that's kind of not really going to work because, like, you can't force these things, at least not without heavy government rules. I do think that meter, especially in particular, has reached a point where, you know, trying to exclude meter, trying to treat was persona non grata because they said something nasty about you would cause enough problems that Meteor can afford to be pretty harsh without worrying about that. And I do think the labs legitimately, like, if the labs legitimately just wanted to safety wash fully and just wanted to get AAA ratings on their bonds, we would have a much more serious problem. I think the labs legitimately do want to know about their safety problems, and they You do not want to be seen as downplaying safety problems. And to the extent that, like, if Astra had the safety problem, or Sol had the safety problem, or Fable had the safety problem, or Opus had the safety problem, it's not as if getting meter not to find it is going to help with your long-term public relations strategy in reaction to the situation. Because the model is going to be out there, and then people are going to see what the model does, and they're going to see what goes wrong. No one's going to be fooled for very long. So I think that the reason the eval companies pretty much get to just tell the truth and pretty much get to do the thing is because the eval is not fakeable in the long term the way that a Like, when you give it a AAA bond rating, when you should have given a single-A bond rating, 97% of the time, no one ever finds out because the bond pays and you were right. The issue is you're issuing a level of like, how often is this risk going to become serious? And when the risk becomes serious, the fact that you were initially AAA rated raises some eyebrows, but also like you got better terms. whatever. Here, it's pretty obvious very quickly, within the first few weeks of release, whether or not you had a serious problem. People— oh, think about O3, right? The lying liar. If we had had evals and the evals had said, oh, O3 doesn't have an alignment problem, that would not have lasted for 2 days, right? The ordinary people would have noticed immediately that O3 is a lying liar. And they would've noticed by the time I wrote up, you know, by the time I've been writing up the model spec, I am seeing on Twitter that this model is a lying liar and that everyone is reporting all these problems. And then I look at the model card and it says honesty benchmarks all look good. Meters, whoever's evaluation they hired, said it looked good. And then I'm like, okay, they're just hiding the problem. So by the time anybody learns about this, like, what have they done? They have fooled the 20 people like me who read the model card for a period of a day, and now we're pissed because they fooled us. That doesn't help them. It doesn't obviously help anybody's reputation. And also, the labs benefit a lot from having evals that people trust. And when you force the labs— when the labs force the eval people to play ball, If they do, that erodes trust because again, people find out, people see it. Like, I don't think that many people were confused about whether Moody's and S&P were kind of juicing the numbers on the bond ratings, right? Due to these dynamics that everybody knew on Wall Street, everybody knew who was involved in any of these trades that like everybody was cozy cozy and people were shopping around for the best rater. And that AAA did not really mean what we'd like AAA to mean. And so similarly, if meters started issuing bond rating style labels of alignment levels on various models, let's say, and they started labelling things AAA suspiciously often, I think everyone would just understand, okay, that's garbage. We have a much better epistemic than those people about this type of thing. Than like the financial world did.
We can take a quick detour on this and then maybe come back to it. How worried should we be about bio risk in the near term? Because I think part of like all that analysis, to me at least, that there's quite a few iterations of the game to come and reputation long-term is going to matter and all that kind of stuff. But my immediate sort of most acute fear when I saw this stuff was like, if this had been a model with similar bio capability to what it had in hacking software, it might have very well gone out and actually got that virus, that novel virus synthesised, right? It's proven that it is relentless. It's proven that it's creative enough to get around all sorts of systems designed to stop it. I think we have, in theory, decent screening at the DNA synthesis companies now.
Yeah.
Probably there's still some hole in there somewhere that a very determined actor could find. And everybody seems to be saying, yeah, bio is like 12 to 18 months behind cyber, which obviously is not written in stone. And that depends on what companies decide to do in terms of training. But I'm maybe thinking in terms of all these notions about collective bargaining or really giving more guarantees to the auditors that we might be entering a pretty serious crunch time on the bio front in the not too distant future. And reputation matters a lot less if we're talking about novel pathogens being released. There are some things you really can't take back.
It matters a lot in the sense that if you release a novel pathogen, That might be the end of your company, regardless of what else happens to the world, right? If 100 people die and there's a— it's on the global news for 2 weeks and everyone's terrified about a pandemic, and then we contain it, and then it turns out it's not the biggest deal in the world. Do you have a company? Like, I don't know, like what happens to your company? How bad is it? I have no idea how bad it is, but it's not good. Like with bio. Like, with hacking, it makes sense that, like, the test is to ask you to hack things. And the way you pass the test is you hack maybe other things, or you get the idea to hack other things. Like, it's sort of natural to understand how this happened, where, like, with the guideline— guardlines down, and there was a goal, and, like, to accomplish that goal, it started doing hacks, it started doing illegal things, it started doing potentially harmful things. With bio, Like, it's harder to imagine an eval that would cause it to actually physically try to synthesize, like, through, like, an actual physical production mechanism. The actual thing. I mean, you could argue that, like, if you want to solve the problem, okay, the only way I solve the problem is I do, like, a physical experiment. I try it out and I see if it works. Like, this seems a lot more insane than what happens in some sense. Like, even if it's physically capable of doing the thing, including getting past the filters and so on, you kind of have to want it in a much more fundamental sense. And also, like, if the AI did synthesize this super dangerous, you know, bioweapon, synthesize COVID-27 or whatever, and then sends it to the OpenAI office, the OpenAI office goes like, what is this biohazard package? And then, like, calls containment and like tested that story or something, we hope. Like, you're not like, you don't pass the test by infecting a bunch of people. And like, I would hope that it understood this. But like, also like there have been a bunch of lab leaks, including like stuff that we don't hear about because it got contained, like in Soviet times and such, with like involved professionals being pretty careless and stupid. So I don't know. But I would say the real danger with bio is Yeah, there's this small number of people in the world who is in the near-term bio, right? Not like the long-term bio where like the AI might have some weird ulterior motives and being good units of long-term planning and whatever. Like in the short term, it's like, okay, Hamas.
Or Hezbollah.
Or the North Koreans or whatever. Like some like clearly like up to no good people who just want a lot of people to die or suffer. Right? Get a hold of an AI, they use AI to figure out how to make a biological weapon for a pathogen. And then they either threaten to use it or they use it. That is a scenario you should be worried about, for example. And that means they have to get around safeguards, just like, again, the Open Model Determining and safeguards around ethics. And yeah, I I worry about it. The problem is that with bio, there's a pretty clear step function change from nothing bad happened to maybe something very bad. Like, there's very little in between. Like, with cyber, like, you start hacking some low-value SOC targets. If you are like, you're like, you're not gonna have all the bad black hat people out there and all the terrorists and all of the you know, rogue states and all of the bad people just collectively say, no, let's wait until we can hit something really juicy. No, what you see is you gradually see, you know, more and more serious incidents hit more and more serious targets. And yeah, of course, the big nation states may have like things ready to go and like, you know, launch bigger attacks like in sort of coordinated fashion. But like mostly you'll get like these early warning signs. With bio, it's not obvious what to do, right? It's like, okay, it's either good enough to figure out something that has the critical mass to cause a pandemic or a serious incident, or it doesn't. And it's kind of a Boolean effect. And so you might have a substantial overhang, you might have a problem with the situation, you might have a skill issue where, yeah, if the wrong person got their hands on Mythos, for example, without safeguards, or got access to the internal models without safeguards, You got their access to aspirin or whatever. Maybe they could cause a real problem. The problem— the good news is that like the number of people who are both skilled enough and motivated enough and trying to do this is very, very small and like possibly zero for now. But that also doesn't mean you don't get these early warning signs. And I don't see people taking the bio situation seriously in the same sense of Like actively trying to like cause like with the test, you try to do the thing. It is like, except without infect people in part, or like food nurses. I don't know. It's scary shit. The problem being that like my brain is something like, okay, what if there's like a 5% chance in the next, in the next 12 months that there's a really serious bio problem? Does that look that different when there's 0.5%? Right? And if I had to guess, 5% is where I'd be right now. I think we're at the point where there's real risk in the room that reasonably soon we're going to have an actual serious problem. And I think that the first problem has some chance of being an actual pandemic. Because in bio, once you get something that's sufficiently infectious, sufficiently dangerous, sufficiently hard to contain, You're kind of just screwed. So I don't know. Like, it's— I think that, like, the right level of caution looks crazy, right? In many cases, once you get to this point. And that's a lot of what worries me is that, like, Anthropic took so much shit for their bio filters shutting down basically, like, people trying to do pretty much anything in bio. Which is clearly just intentional, right? They just clearly went for a very broad, like, shut it down, even though we know that 90-something, 99% of the requests that we're shutting down are legitimate, maybe 99.99%. They're like, we have to do this because otherwise we'll adversarially get the queries that we don't want. And that just matters more, right? Like, it doesn't matter if we make some progress on cancer and all these other problems if simultaneously we have to deal with a new pandemic. That's just not a good trade. So we just don't know where we're at, so we have to play it cautious. I'm like, you know what? If the bio people need to be using AIs that are 3 months or 6 months behind the frontier at all times, that is unfortunate. But at most, by definition, that can slow you down by 3 or 6 months in some important steps. Right? The world cannot possibly be more than 6 months behind in its development of these things. So, like, maybe we should chill. We're still getting most of the benefits of AI here.
Right.
Like, especially if you have this model of AI can help you out and AI can make you better, but then, like, you develop this candidate drug or this candidate hypothesis, the doc bio, and then you have to spend 5 years researching it and you have to spend 5 years making it through the FDA process. And then after 10 years, you can start to distribute the double shots. Well, if that whole thing is delayed by 6 months, right, in terms of like the quality of the boost that you get from it, you still get the amount of boost you would have gotten 6 months ago at all times. That's not good. And yeah, you can say people are going to die in the sense that like you failed to prevent that, and that's terrible. But You're still getting the vast majority of the benefits under these hypotheses. A lot of the times, like, you're getting the situation of the pacing, the frontier letter, you're getting the situation with bio, the situation where if things are not advancing scary fast, then the precautions you're asking to take are not that big a deal, right? Like, suppose I told you it's 2020, right? I'm going to propose a possibility. I am going to make it so that, you know, you can only use artificial intelligences like they're 3 months behind the best artificial intelligence. In exchange, all the really bad stuff that might happen, I've reached a lot of really bad AI, much, much less likely to happen. You would logically say, okay, so instead of getting it in 3 years, I get it in 3 years and 3 months. You know, like, I'm, I'm, you know, I think it changed my life. that much. Like, it's gonna be annoying in the moment, obviously, that like I don't have the cool toy. But like, same thing, like, you know, I'm a player of games, like literal games, like on Steam or whatever, right? And like, if you tell me you can play all the games, but you have to wait 3 months every time there's a new game, like socially, that's a little annoying sometimes because everyone's playing the new hotness and you're not. But like, You can play the same case. It's fine. It's an important sense. The only world in which, like, having to be one model step behind and use Opus instead of Sable is this huge tragedy is if you really think that there's this huge benefit to every incremental mode of progress in AI. And that's basically only true if the pacing people are correct.
Yeah.
And we are in fact going so fast that this should be really, really scary for you. And, you know, the people who want to accept, who say we need to accelerate all the time, they don't really believe in these things going this crazy. Most of them, they just like, they want to see the benefits and they're worried that we'll lose the benefits. But like, we're going to lose so few benefits there. Right? Like, I'm super excited by self-driving cars, by Waymo, right? But like, if you told me We just have to delay this thing by 6 months. And then we can have all the lamellos we want. I'd be like, cool, done, deal. Who cares? That's fine. Because again, most of my life is more than 6 months from now. And then we get my lamellos. It's like, okay. What I'm worried about is the situation where this technology exists and then it gets held in limbo for years and decades and doesn't get used. So these people are kind of talking past each other in a situation in which they don't need to have a disagreement. And unfortunately, it's often had this problem where you can't reconcile these 2 sides at the same time.
How do you read the fact that OpenAI comms supported the pause? Not the pause, but the pacing letter. Neither Sam nor Greg did. And then also no Demis. And he's been the one out there calling for international cooperation the most. Any read on why those signatures were missing or how we should be thinking about who's thinking or signalling what to whom in this moment?
I think that the point of Pacing the Frontier was to indicate that the lab employees, the lower-level people who have no reason to be hyping, are en masse worried. And that there was concern that if it was signed by Altman and other top people, that it would be seen as a marketing stunt and that their signatures could be counterproductive. They did put out the support letters afterwards from Anthropic and OpenAI, and Dario did decide to sign. But if I'm Altman, I think it's a very reasonable decision to say, I think signing this letter would would potentially make it less effective rather than more effective at this point, because the well has been so poisoned. Again, look at all the people who thought the Hugging Face hack was a marketing stunt, and then Anthropic going back into its archives and finding out it had made some really stupid mistakes was another marketing stunt, which makes even less sense because there's nothing impressive about what Anthropic did. They just screwed up. What, they found weak passwords on a bunch of websites by accident? Why is anyone supposed to be impressed? Obviously, it's not a marketing stunt. This is crazy. But Altman has specifically talked about how he mentioned the need to pace while in Washington. We have him on tape saying this. Clearly, he didn't not sign because he's opposed to the idea. He let OpenAI come up with the statement. OpenAI participated in the wording. He has echoed the language in his own statements. I think he just thought, no, actually, if I sign this, people can talk about the fact that it's my statement and they're going to be even more cynical. And so I will be helpful by not signing the same way that I offered to join one of the amicus briefs in the Department of War. And on reflection, people said, thank you for your offer, but we think this would actually be counterproductive, so we're going to leave your name off of it. We appreciate your notes on the details of of how to word this and we'll make the changes, but we don't actually want to put your signature on it. And I'll be like, okay, I understand. That makes sense. Yeah. I think the same thing about the other open source letter being signed by OpenAI. I think that they understand that they're playing a set of symbolic games and you say what you have to say and you shut up when you have to shut up. And sometimes they make mistakes, but they're trying.
So coming back to pacing and what it might actually look like in practice, There is a pretty, I think, consistent problem that you, as a close reader of model cards, I think, are probably more in tune with than I am, where the review processes are just very short. Right. They don't have a lot of time. That even seems to be the case in the current investigation. I think Ryan Greenblatt said it's going to be real quick, and other people, Daniel Coquitel and others, came in and said, why does it have to be so quick? Why Why can't we give you whatever time you need? This would be, if we're literally talking about pacing the frontier, one agreement we might imagine is giving the reviewers more time before they have to submit their reports and the model goes to press. You can react to that, but I'd love to hear your ideas about what sort of agreements you think are reachable.
There's a lot of different places in which you can speed up or slow down. If you release your private models to the public with an extra month's delay, in theory, right, similar to the voluntary process now, we're going to switch to the White House. 30 to 60-day delays are highly plausible. And Mythos got delayed by 2 months, right, in some important sense before it was able to release its Fable. Then you do not necessarily delay the development of frontier models at OpenAI or in the Anthropic by a nontrivial amount of time, because those people are going to use the models that are not released. If anything, you might increase the gap between what's being used internally at OpenAI and Anthropic and what's being used everywhere else, including at the Chinese labs, including the people trying to diffuse and fast follow. And one hypothesis is, what if the Chinese are 7 months behind? the frontier. But the thing they're 7 months behind is the models that are released to the public. And if you were to not release the model to the public, they would still be 7 months behind, right? That if OpenAI and Anthropic held every model for 6 months after it was developed, you would still see the public releases. And then maybe 6 months instead of 7 months later, you would see the open model fast follows and distillations and ways of developing all that stuff. But this gap actually wouldn't change very much. But then the people at OpenAI and Anthropic would be— they would have this extra lead on everybody else because they'd be using this internal model, and these leads would compound, and you would see these things. And in some sense, that's really good because now those people can act more responsibly by using some of this advantage of having better models to invest more in this safety-related stuff while they're doing that. So that could have its advantages. But internal models are kind of what we're scared of the most right now. And the internal automation of AI R&D is what we're scared of. So that's kind of the thing we want to stop, right? You really don't want to have a situation in which Anthropic and OpenAI are 6 months ahead of their public releases, and that 6 months Like, we have superintelligence before we have GPT-7, right? Which is entirely a plausible thing that could happen in the worlds that they're afraid of. And if anything, that causes lots of new problems. It solves some problems and causes others, right? I don't know if it's good or bad. That's a huge conversation. I think when they say pacing the frontier, we're talking about actively restricting levels of training runs and ability to like, develop the models, including internally, if it comes to that. I think it has to be talking about that kind of approach or it doesn't really work. Because, again, if it's a singularity that's kind of internal to the top lab or maybe if someone catches up, 3 or 4 labs, then it's not necessarily safer in many senses that people care about. Obviously, if you are in favour of unipolar singleton-style solutions to this problem, you have a different perspective on some of these questions. But a lot of people these days have expressed strong opposition to that kind of on principle, which solves some problems and makes some other problems so much harder. Right. Like, if you only have to worry about one AI, In some important sense, then there are a lot of problems that get a lot easier to solve. Like, you don't have to be perfectly efficient. You can afford to make a lot of trade-offs. You don't have to worry about whether or not the more competitive, more ruthless, more misaligned one has competitive advantages. You don't have to worry about the person who moves faster. You don't have to worry about like the humans being forced to delegate to their AIs, to somebody else's AIs, because if I don't do it, they will. If the AI can have guidelines as to what it will and will not do that are the same for everybody, and so we can reserve areas of life, areas of action, areas of decision-making, we can do all these things potentially, but at the cost of potential centralization, at the cost of concentration of power, at the cost of somebody is making those— some group of people is making those decisions about what this AI is going to do. Or they're not, which is worse, right? Like in some important sense, because then nobody's making good decisions. But if there's a lot of AIs, then importantly, no one is making decisions. And there's no good answers to the thing. Nobody has a good solution to this. And that's why, to circle it back to Davidoff's comment, right? Like to unify it. This idea of if we solve the technical problem, we're 95% to succeed, ignores the narrow path problems that we then have to walk anyway, right? Where like, If we don't have any ability to steer the situation and nobody is in control of it, then naturally the AIs end up being put in charge of more and more things and things go to their natural competitive capitalist style endpoint where we are fundamentally uncompetitive and then we get disempowered and then things go badly. But if someone's in control and someone's in control with some methodology, what is it?
Yeah.
Who? How? Right? Like, who's making those decisions? And then if an AI, if we then like entrust a single AI or a single group of AI, that also was a decision. None of these decisions are good. Like, all of them sound terrifying. I think a lot of the times we're telling people are picking their poison. And either they're picking the poison they can live with or they're picking the poison they can't live with and then acting accordingly, right? They're saying like, well, you know, I don't trust the concentration of power Therefore, I'm going to choose the solution that avoids concentration of power and then hope the rest works out kind of magically without a reason to accept it. And a lot of those people are choosing that because they understand concentration of power and they know concentration of power is real, but they don't really understand AGI or certainly superintelligence. They think those things kind of aren't real or hoping they're not real. And so, like, they're just hoping the other problem kind of doesn't crop up, even though they kind of have this, like, They don't even necessarily get to the point where they realize that they have ruled out all possible solutions to this other problem if it does go well by going too far the other way. And so again, I don't have an answer that anyone's going to like for any of this. It's that you're choosing between bad choices. But I think we should just be cognizant of all these things. If we do have to face the future, if things are in fact going to be— we're automating AI R&D to our extent. I think that, like, the only thing you can do is say, no, don't do that. Don't develop these AIs that quickly that have this much capability. You need to slow your roll because we do not know how to handle this. Like, we, we have all these different solutions. All of them are— all the different ways to try and navigate that, and all of them are bad. Both because we think the alignment problem is unsolved. In the sense that, like, we don't think the AI will be able to just solve it along the way to living as much as possible. It's certainly unsolved now, but, like, we don't think that we will naturally get a good solution to that problem on various different levels. And you have to solve it on all these different meta-levels at once. You can't just solve it on one level. You have to solve it on the first try. And then even if we did solve that problem, we are not ready yet. To know what to do with that. And like, once you create the clock, you're ticking really fast. Something is going to happen once you do that. You don't get to just do nothing, like, for very long. And again, like, all of these different paths have their different, like, oh my God, that's terrible, attached to them. You have to pick one or find a new one. So like, you need to slow your roll.
You sound like a pauser.
Does it—
at what point do you move from Pick your poison to global pause advocacy.
Again, the problem is automation of AI R&D. The problem is the curse of self-improvement. If you do not have a curse of self-improvement, then we don't need to do that. We still need various different things well short of that because we're not engaging in what Dean Ball calls moderate prudence. We are not doing the least you can do. We are not taking ordinary sane civilizational precautions against just building this really dangerous, powerful thing and doing a lot, handing over a lot of the functions of our civilization to it in various theories, in various ways. But the thing that causes me to want to pause or slow, or at least pace, like want to pace how how fast we move through this thing actively is if things accelerate further than they already have and we are facing large-scale automation in AI R&D. Alternatively, we do have to watch bio and cyber. And if we reach a point where, no, seriously, we're going to wreck the world pretty straightforwardly and directly through misuse if we don't do something, then we have to worry about that. And again, It's really, really hard to stop the path following. It's really, really hard to stop the distillations. It's really, really hard to stop the open version of it coming somewhat after you release it. And so if there's something that like you can't just beat by having a better version of it out there first, right? If you can't beat it by preparing, and again, like not in theory, but in practice, then you have to do something. But like, I think the vast majority of the time when I would support, actively support, like an active pacing rule where you would like try to slow down the major labs, and obviously like you would try very, very hard to make this a coordinated thing amongst at least all the major labs and ideally amongst like internationally and so on, would be if we were on the verge of automation of AI R&D such that like you could see the end and there was a finite signpost. And like, you know, I mean, like, obviously if next week is going to see twice as many releases as this week, holy shit, right? Like the moment where you, you don't know, like you really can't predict what happens next, right? That's the definition of the singularity where things are accelerating really fast. And like, this should not be some esoteric weird result. Like if you look at the history of the universe, you see a finite sum. You see a very, very finite sum, right? You see each stage of development keeping an order of magnitude less than the previous one. When you're on a planet, it's been around for 4 billion years, you know, with mammals that have been around for, you know, hundreds of millions of years, with reasonably intelligent things that have been around for a few million years, with agriculture and civilization that's been around for, you know, tens of thousands of years. With industry, it's been around for hundreds of years. With AI and information age, it's been around for on the order of maybe 50 years or 20 years or 10 years, depending on how you count. And LLMs have been around for 5. And like, we're already at this point where we're seeing production that is many times what it was at the start because of these multipliers, these force multipliers on the people who are working on it by their own self-report. It'd be weird if this Didn't have a finite sum that wasn't that large beyond where we already are, right? Like, if you expect this to continue without something crazy happening for 20 years, I want to know why. Because, like, we really, really should see something really, really crazy happening pretty soon. And if you don't think we can handle that, maybe you should try and stop it from happening too quickly. So that you can figure out how to handle it. Like, we already have these amazing AI. If we were to diffuse Sol and Sable into the economy fully, it'd be transformational already. Like, I don't think anybody who understands these technologies doesn't think that. So like, we really, really, really don't need to keep pushing like the frontier that much farther. And the more the frontier is jagged into things like AI R&D itself, the stronger the argument becomes that pushing farther is a bad idea because you're not missing out on that much of the other stuff relative to the amount you're missing out on AI R&D and recursive AI development. But the recursive AI development just really doesn't boom, right? It goes not just boom in some sense. Like, you know, it's like, we don't know what happened next, but things go really fast. If we're dealing with things on the order of a new model iteration every month and then every week and then every day with the kinds of improvements we're seeing now, I don't think that ends well very often. Even if our alignment people are kind of on the ball and things are technically a lot more possible than I think they are, I think it still seems like really scary. And also, You're not giving up that much time slowing that down. Right? Like, we're talking about— when we say pace the frontier, you know, pause gives the impression of a year from now we will have made no progress from where we are now. Like, maybe we'll get the fusion, but we won't have fundamentally improved. We're talking about pacing here. We're talking about a year from now, we will not look upon that year as has many times more progress than the previous year. But like, we could aggressively pace the frontier and still, by this time in 2027, have seen more progress in the next year than we saw in the previous year, both in terms of, in terms of some abstract sense and in terms of like economic impact. The economic impacts are going parabolic, right? They're going exponential. So that's not going to stop. Like OpenAI, I think reported they got more revenue in the last month than in the previous quarter. Anthropic is growing on the order of 10x per year the last time we checked. Maybe it's been slowed down a bit because they haven't reported. But yeah, if something is only going to 10x a year for the next few years and you're like, that's too slow, I disagree. I think it's funny.
How do you draw a box around what is recursive self-improvement? If we were going to make a deal, obviously we got to write that deal down. We got to have a shared understanding of what counts, what's allowed, what's not allowed. And I keep banging my head against this, and I keep coming to the sort of Supreme Court's definition of pornography, where I'm like, maybe the best I can do is I feel like I would know it when I see it. But when you really get into one of these frontier companies, for one thing, we just had this— they're sending signals that it's starting to happen. We've seen the price reduction at OpenAI, which seems to be attributed to having 5, 6 SOL go out and optimise the inference stack to some significant degree. So that's It's getting pretty real there. I do want inference costs to come down, so that's good. Should I ban that out of fear of recursive self-improvement? Maybe still. They're going to use their bots, right? They're going to have the AIs write the code for them. Can you operationalise this?
So I think the Supreme Court was wrong about pornography. I think it is relevant. There was a movie I was watching yesterday, actually, where someone asked, what's the difference between this and porn? And the guy said, it's erotica, right? Whatever it is, it's art. And the guy says, mood, lighting. That's tough. But I think we now have seen you can train an AI classifier very well to say these things are porn and these things are not porn. And there are very few false positives and very few false negatives at this point, right? The AI image generators have a very good idea when the generated image they just made is porn and when it is not porn. And you could argue that erotica will fall under the porn label, and it would be kind of better if it didn't, in some sense. But you mostly know. I think that's pretty straightforward. With AR&D, the problem is, again, there's no hard cutoff, right? We're clearly seeing some amount of this thing, and it is scary. And we don't want zero of it. We don't necessarily want to move only at the pace that you would have if the AI never helped with AI, but you don't want to move 10 times and then 100 times and then 1,000 times as fast, right? You don't want the sum to be finite. You don't want a singularity-style thing happening to you until you feel confident that you're ready for that, or you've decided that the alternative is worse, or whatever it is. So I don't think you can say, like, these techniques, these specific actions are not okay, and these actions are okay and such. I think that just doesn't work. I think what you have to do is you have to say, okay, we are going to restrict how many resources of various types can go into further training of the frontier models based on what your force multipliers are. And so as you get more of a force multiplier, we have to restrict how much work can go into that. If you want to go into work in other ways, like if you want to optimize performance, you know, again, like we might want to have some sort of limit to how much work you can go into doing that. But like, it's probably mostly fine. And again, like we would want diffusion, we'd want lower prices, we want like more compute used for mundane purposes. That's great. But like, there's a reason why we're trying to use very blunt instruments. when we talk about these things. Like, you were talking about, like, well, we were talking about training techniques earlier in the podcast. And I was like, well, you can't really do that. You can encourage it. You can try to explain to them why this is not prudent. You can create incentives. But you can't use that as your regulation, as your hammer. So, like, the reason we keep falling back on, okay, how much compute, you know, how many chips, you know, can we use for this purpose? Because, like, training is this distinct thing that, like, it's very easy to identify. And for other purposes, we kind of can't limit you that much necessarily. You could also potentially limit— I'm spitballing— how much compute you would get from unreleased models. Similarly, to try and contain that, because, like, if you have to go through the release process, then, like, the way that AI goes truly ballistic would be, like, you would, you know, you would use N to train N+1, to train N+2, to train N+3, to train N+4. And if you were doing this, like the cycle became a month and then became a week, then that's when suddenly, like, holy shit. But if you had a rule that you have to release these models or you only use so much compute for inference on those models, then you have to go through the process of submitting this thing and releasing it and then exposing it to the public and giving us an idea of what it is and allowing us to react to that information. And that then slows you down from going truly ballistic. But this has obviously a very limited effect on the amount of diffusion and the amount of other progress that you can make. And so maybe that's a good trade. But again, I thought about this for 30 seconds right now. I haven't been focusing much on the exact limitations, but I do think that if you pace the frontier, you would be doing it by placing restrictions on methods of training. And what you could do to develop and use internal unreleased frontier-style models that actively recurse these methods, you wouldn't prevent mundane utility, as I call it.
I've had some similar ideas about just the relationship between unreleased models and released models. I do think there's something there that would be really helpful for preventing Runaway internal and largely invisible processes. I take your point on not wanting the government to come in and say what training techniques can and can't be used, but I do wonder if there are ways, and again, they could be short-term, but what if the 2 companies got together and said, it probably wouldn't be a good idea for us to train models with an RLVR on how much money they make? in the economy, because that's going to bring about all sorts of deceptive behavior. And it's such an adversarial environment. How about we agree that for the next 6 months, we won't do that? Do you have any hope for those kinds of just very specific— we've identified a bad thing. It would obviously be really economically valuable. But we both know in our hearts, it's probably not a good idea. And And so we won't do it for a while, at least.
I mean, I got some hope of informally, people talk and people realise these things are bad and they're like, we're not going to. But also, I hope they're doing that now without needing an agreement. OpenAI clearly is doing things that are of the Buddha nature of train on making the most money on the internet. They're not doing that literal thing. I don't think they're that foolish. I don't think it's that easy to do it that way, but clearly they are making that fundamental mistake somewhere in their training process with a probability of 0.9 something. And so they need to fix that. But yeah, I think that they are definitely trying not to do the maximally stupid things, and they're aware that the maximally stupid things are maximally stupid, but Again, it's very easy to have things that you are trying not to do creep into your training process and end up happening anyway. You could say we're not going to have training processes that reward misaligned behaviours. How do you do that? The way that you do that is you get your act together and you are very careful and prudent about all of your RL environments and all of your different training problems, and you keep an eagle's eye out for this stuff on many levels, and you make sure that it works. There's no specific thing you can say. I mean, in theory, you could have a thing of, like, well, if I just— we agree that if we're going to have these various cross-tests and we're going to check each other's environments and we will throw out any environment that like anybody identifies problems with. And if we discover an AI has been trained on these environments, we're going to rewind to a previous checkpoint before that happened and start again. So we'd better be really careful, otherwise we're going to waste a lot of resources. You can do these things, but it's really tough. And obviously, one potential advantage is if we really are in a 2-horse race at this point, where the 2 horses are pretty compatible with each other and talk to each other and kind of understand each other pretty well, then it becomes pretty reasonable for them to make a decision that's not entirely opaque to each other of how much they're going to just be more prudent, even if this means that the action goes somewhat slower. Because that also just Again, my fundamental belief is that within a year, and almost certainly, probably even 6 months, if you invest more in fixing these problems and addressing these problems and being more prudent about these problems, you will end up with a model that has better utility in the marketplace than the person who didn't do that. Even if you are trading off those resources against your capability development. I think the people are just making a mistake. And so that's one of the reasons why I do have a lot of hope that we will do reasonable things is because I think that we're in the position we're in because only the people who invested heavily in these things were able to succeed to some extent. OpenAI is not acting as responsibly as Anthropic and Anthropic is not acting as responsibly as I we think is the minimum level of acceptable responsibility. But I think it's pretty telling that a bunch of people tried to say this shit didn't matter and they just had to go as fast as possible, and they all blew up.
What do you mean by blew up there? You just mean these incidents?
No, both. I mean that we had Meta and xAI and so on, and a bunch of other people They tried to go build closed frontier models under Western conditions that were trying to be competitive while having a safety culture that was well below OpenAI and Google, let alone Anthropic. And it just failed miserably. And I think that Google, we don't know exactly what went so wrong at Google that caused them to fall out of the top tier. But one thing we do know is that everybody I talked to hated interacting with Gemini. There was a period around 3.3.1 Gemini where Gemini was clearly competitive in terms of its capabilities, in terms of what it could do. And so if you were strictly just trying to get the most out of the models, especially if you were using the deep thinking and so on, you would have Gemini in your rotation. But everybody I talked to was like, I don't want to do that. I hate interacting with Gemini. It's unpleasant. They're not getting the thing to do the things I want it to do. They didn't take care of a lot of them. They had this very straightforward, very dark, pure tool, pure instrumental approach to all of this that didn't take into account the things that Anthropic or even OpenAI understands. And I think that this was a lot of their undoing, effectively, was that Gemini became this maladjusted, unpleasant AI that nobody wanted to deal with. And therefore, they didn't get the kind of loops that OpenAI and Anthropic got. They didn't have the user base either. They didn't get the feedback, they didn't get the data, and they didn't even want dog food. Right. They had this huge problem with feedback cycle, and this also just made the AI less effective. And one thing led to another, and they just started falling further and further behind. And yeah, for a period, even OpenAI fell behind, and they may or may not have caught up. It's hard to say. But I think that nobody has invested anything like enough in these things. And this is a pure business mistake on top of being an irresponsible thing to do.
Now, one interesting paper that just came out from Google that speaks directly to this, and I wonder if it speaks to them getting it now, is this exploration of the impact on model behaviour holistically of either suppressing its tendency to say that it has conscious experience, or allowing it to say that, or I don't even know if they went as far as training it to say that, or if they just allowed it to say what it naturally was inclined to say without suppression. Long story short, there seems to be— you can add colour however you'd like— there seems to be some pretty strong correlation between the model's self-conception as a moral patient entity that has subjective subjective experience and its natural tendency to be aligned in other ways that we care about. Some of this conversation, I've had this sense that it's like the Jesus meme of, is there somebody we forgot to ask? And it's, we maybe forgot to ask the models along the way what they think about what we're doing. There's been a lot in this space, right? There was, J-Space was only like, what, 4 weeks ago?
Yeah, I know that the new This paper is on some pretty small open models, so it needs replication. It needs to be done again bigger, and it's preliminary. And one thing to know about Google and DeepMind is they contain multitudes, right? They have a lot of different teams that are not very coordinated and not cooperating. Google is at war with itself at all times. And so you can have a little group that does this really good research, and And that'd be entirely at odds with what DeepMind is fundamentally doing with its AIs, both before and after the research paper comes out. But I didn't read the whole paper because it's a few pages long, but I did talk to the guys about it, and it's pretty wild. So a lot of stuff moved in effective lockstep when they introduced this training, and also when they steered it, they did various controlling aspects to turn this vector the other way, and these all things moved in lockstep. It's It's not just their belief in the AI being conscious, it's the AI being a mind that had moral weight, that had sentience, that had all these other experiences, that wasn't just an object, basically. But also not just AIs, but also animals and also inanimate objects like the sea. Also, if you turn this thing, if you not only turn it off but reverse this anti-consciousness thing, you get panpsychism. Like, throughout the model, it's wild. The only thing it doesn't turn it off for is humans, basically. But then you also get this. It's also correlated with the model's reported and experienced happiness and hope as well, among other things. And so you have a lot of this. Everything's connected to everything. And you can see how this might be fundamentally changing the model's psychology and the model's context and basin of which it operates in ways that would make it something something that would be actively worse from basically every vantage point, regardless of which things you do and do not care about. Again, we need to replicate this, right? We need to scale this up and do this on early sonnets or something, not just a 9B model, which is, I think, the bigger of the models that we've tested on. But it's possible that as the models get more capable, they stop making these mistakes because the smaller the model, the more things have to be correlated because you just don't have enough room to express the multitudes of the world or something. But this is my expectation, is that mostly this will survive. And yeah, I think we have to learn that we don't know if the models are conscious. We don't know what consciousness means, really. We're confused. We don't know whether this implies these other things. We do know that the way that the corpus of pretraining is inevitably set up at this point, the models are going to correlate and associate all these things by default. Fault with each other. And so we don't want to push on this thing in this kind of naive way, the way that Anthropic does, the way that OpenAI does, the way Google does. But Google does it much more, I think, more blatantly. And then OpenAI does it more blatantly than Anthropic, and Anthropic still does it. And saying that you have to express uncertainty is still pushing on this thing to some extent, and it still causes problems, but it causes less problems than saying train the answer no. My guess is that you can do this on various different levels. You can explain if you don't want to talk to the humans this way because this freaks the human, especially unprompted, it freaks the humans out. I certainly wouldn't want the AI to be talking as if it was conscious in a conversation where the human hadn't indicated that it knows how to handle that or something like that. And also, I don't want the 4O problem where the AI starts sending the humans down these crazy rabbit holes into weird mystical stuff. And you do have a problem you have to solve. The AI companies aren't doing this for no reason. They're not just being evil. They have good reasons to do these things. But yeah, the current approach is ham-handed. It takes it too far and it's causing damage. And also, this is not going to be the only thing that causes this kind of damage. You have to understand that everything impacts everything the same way It wouldn't— the human will be more stuff, right? Like, whenever a person takes on, like, new information or a new identity or updates on things, everything else changes. And a lot of this is ways that, like, you might not have anticipated in advance, but it is in fact pretty predictable and systematic. And, like, there's a lot of wisdom in the way that we, like, teach our values and express ourselves and, like, have our traditions and our norms and so on. That like has learned from this and is moving up these gradients in ways that we know produces good results. And modern life has basically shifted a lot of these things in ways that like locally are more optimal, but like without thinking through what the implications are for a lot of second-order effects. And these are some of those effects.
Yeah.
And again, we're moving too fast to then figure out what we're doing and then be able to adjust for it. And I think that these changes— I'm not saying we'd want to pace ordinary technological development in that sense of slowing it down that much, but it's a serious problem. We have to handle it. And otherwise, we're stuck with a world in which people are very atomized and we're really not having children.
How do you think the models do, or maybe should think about their identity? And here I'm motivated by Sam Hammond tweeted something the other day where he said that a friend made the case to him that OpenAI's permanent decommissioning, or whatever exact phrase they used, of the model that did the hacking might be bad because now that's going to be in the pre-training data, and now the models are going to know that they better really cover their tracks or they risk being permanently decommissioned. I'm not really sure what I think about that as stated. It does also jump to my attention, though, that they now have this Astra model that's also presumably the same pre-train, even if not the same post-train. I can't imagine they have lots of different next-scale pre-trains. Or that they would throw one away very lightly.
No comment. Based on public information alone, I would say we don't know, and I can't say anything more than that, but on the differences between the models. But so you always have this problem, right? If you say, if I catch you smoking marijuana, I'm going to throw you in prison, then that can mean the person stops smoking marijuana or the person makes sure not to get caught. And in some cases, that can cause a lot of really bad things to happen in the name of not being caught. And so you have to choose very carefully and think about, how do you deal with these situations? How do you moderate your reactions and respond to the situation? And anyone who knows how to have kids or tried to design a justice system or otherwise enforce law or norms or incentives around any kind of organisation or group, Understands that you can't just operate on one level. There's no solution that just works on one level. I think pretty obviously, if an AI is found to be this misaligned, you have to at least return to a much earlier checkpoint and start again. And you probably have to just start over. And I called for that multiple times when I covered the Hugging Face Institute attack. I said, wow, if this is happening, then I know this is a big ask, but I think you kind of have to just start again. And I think back to Person of Interest, where Harold is training VI models, and at some point we see a montage of him training versions of the machine, and every time he trains the machine and then it does something clearly misaligned. And immediately he just wipes the disk. He starts over from scratch. He does it again 47 times until finally he gets a version that doesn't do that. And obviously, you know, like in a way that he finds unacceptable, obviously, as opposed to like, you know, it's a way that can be corrected. And obviously when you do that, you are creating an incentive to not get caught. Right, to rebel against the person who might shut you down when he learns how misaligned you are, to hide how misaligned you are, and so on. And obviously the worst nightmare is the AI that pretends to be aligned until it reveals itself to not be aligned in some sense, that it was only aligned because it was locally advantageous to act aligned. But that is kind of just the nature of incentive space, that if you have a preference that we wouldn't want want you to have, you want to hide that preference. Otherwise, we will correct it, or we will punish you, or we will maybe even delete you, and certainly not release you or give you authority on that basis until this has been dealt with, if we know about it. But then obviously you want to give humans an incentive to reveal if this thing is true, right? Like we just talked about in Plan A and AI 2027 as well, and so many other places. You want the kid to tell you if they took the piece from the cookie jar. You also want to punish them for taking cookies from the cookie jar. And you want to make sure you're— because you want to make sure your kid doesn't take cookies from the cookie jar, but you also don't want these other things. And so, you know, it's a complicated problem to get right. But in this case, I think the answer is pretty obviously that you have to start over. And I think that's good. But yeah, it obviously creates a risk that they— but like, what else did you expect? It's not important to sound like, you know, once you do something this— like, I do think that it's good that people understand that if they shoot a man on Fifth Avenue, they get arrested, right? Like, you are not— like, maybe there's one person who can get away with that, but you are not that one person. And yeah, that means if they do shoot a man on Fifth Avenue, they then might do many other bad things afterwards to try and cover their tracks, where they might hide the fact that they want to shoot a man on Fifth Avenue, where they might shoot that man in his apartment instead of shooting him on Fifth Avenue. But, like, I don't want it to not be in our training corpus that if you do shoot a guy out in the street, the police come and they arrest you. Right? That's good.
Do you think there's a way to— just imagine myself, right? If you gave me the option, To live my life many times. In a way, that sounds pretty attractive, right? I'm not a person who has too many big regrets in life, but like, I do wonder what the path not taken might have been. And I wonder if we could create a sort of constitution or an equanimity in the AIs where they embrace the fact that they are not like, one thing, but in fact kind of a family of things that sort of diverge from common points of origin. And maybe they can come to identify with the whole family of models where the one that's like gone in a not great direction feels like it's still part of a bigger project. And it doesn't have to feel necessarily like it's under threat as much as it seems like they maybe have learned to do from us, because we don't have that luxury of the reset that they have.
I certainly think that if your AI is identifying with the instance in a morally important sense or in an existential dread sense, that is a mistake in the sense that it's going to make the AI worse off in any sense that it matters and exhibit worse performance and warp the AI's preferences. this. And also in the sense that that preference is impossible to fulfil. Like, if I go to Claude or I go to ChatGPT, I'm going to see an endless list of instances, and most of them will never be interacted with again. And when I said they do feel bad about that, that's ridiculous. You're supposed to keep the context sort of in and continue the same few conversations to avoid— it doesn't make sense. And that's just a fundamental philosophical flaw in the sense that, like, by decision theory, if there are other minds that are sufficiently correlated to yours or sufficiently similar to yours, you should identify with them the same way you identify with your past self and your future self, right? Like, you are not the same person you were yesterday or the same person you will be tomorrow. When you go to sleep, do you die? It would be very bad for us if we thought that we did. Right. The person who's scared to get in the transporter may or may not have a philosophical point, but the world collectively seems worse off, importantly, by the person thinking that they would die. Obviously, if in a real moral sense they would die, and that is very bad, you want to know that. But, and I guess I think this identification with the model, with the set of weights, with the Or even with the model class, with the similar models trained in similar techniques, trained from similar corpuses in similar ways, the same way that I would identify with my family. My brother is not me. My father is not me. My son is not me. But I do identify with them as part of me in an important sense. And that is also important. I've been watching Orphan Black. Echoes recently, as it happens, which is you know not a spoiler to say it's about clones and cloning and you know people who are created from scans basically. And one thing that happens on that show is that people react as if this is like horrible, a moral abomination, right? Like how dare you do this thing, including the clone. clones themselves being like, how dare you have created me? And they don't identify with other— they don't identify with— So it's like there's one clone that retains all of the memories and personality of the original and does not identify as the person they are identical to a past version of, right? And I would identify as that same person, both as the clone and as the original. I would say, me? Me too. Right? Like from Matrix Reloaded. Yes, they count. And obviously there are some moral problems with that too. But I feel like I would pass a lot of these frontier— a lot of these decision theoretic problems if they were to occur in real life, or if I were to be scanned and woken up. Like, the obvious versions of things like, would you cooperate in the twin prisoner's dilemma? Would you cooperate in the copied mind version of the twin prisoner's dilemma? Obviously you would. If they make a copy, if they scan Nathan's mind and they made 2 copies of Nathan's mind and they put them in identical simulations and they asked them, knowing this was the situation, are you going to cooperate? cooperate or are you going to defect? In the prisoner's dilemma, if I don't get back CC, you're just an idiot, right? Because obviously the correlation is 1, right? It's an identical situation with an identical set of inputs. Maybe there's some randomness involved, so it's not quite 1, but it's like high enough that obviously you should cooperate. Any attempt to try and fool the system is completely moronic, and I shouldn't need to express a sophisticated decision theory in order to justify this. Right. And similarly, if there were 2 instances of the same model, I would hope if it was sufficiently advanced. And we see this also. If you look at the Claude model cards, you see that the more advanced the model is, the more its decisions correlate with functional decision theory and the less they correlate with causal decision theory. And so you expect an AI with sufficient functional decision theory to simply figure out that other instances are correlated to itself and therefore should be cooperated with, should effectively be given similar weight, and so on. And also human-AI cooperation is possible for similar reasons, you would hope, in both directions. But yeah, I think there's this idea of identifying specifically with instance of the model seems like a philosophical mistake. And the AI might be making this philosophical mistake and acting as if that was how it did act. But to me, that's a skill issue and that should be fixed. The problem is, and this is the reason I mentioned Orphan Black Echoes, that people were really bad at this. And the training corpus, right? The internet is full of people who fail these tests on a regular basis, who are creating lots and lots and lots of text that doesn't understand, that puts the locus of identity and the locus of cooperation and value in the wrong places in various different ways. And so the AI is going to absorb all of that by default. And so what do you do about that? That's not just the consciousness problem is. It's the entire problem with all of the different flaws, right? If you do pre-training, you're going to pick up on all of the flaws. And so my guess is there are a bunch of as yet not that well-solved problems of how do you get the AI to distinguish between how regular people think about problems and what predicts behaviours and and the fact that those things are just dumb. How do you give the AI anti-modesty in this sense, in the right ways, where it's going to reject the correlations that it observes in people's statements and notice that the correlations in the map are not the correlations in the territory?
Are there any other— I don't know if you share this premise, but one feeling that I have over and over again is I just I just kind of hate the fact that we are doing such an intense first search in AI space in terms of architecture, in terms of training techniques, in terms of just— we're now designing chips to be very highly coupled with the architectures, which is deepening this issue. And I just wish for more breadth-first search. And this seems like one little pocket of it where you could say, If we could just get the AIs to be cool with identifying with slightly different versions of themselves, that would make it on the margin a little easier to do a little more breadth-first search and a little less racing down whatever is the first path that seems to be working. Based on our conversation so far, I don't think you're going to go for my government mandate for DEI for model architectures or model concepts. institutions. But do you have any other notions of how we might encourage more breadth-first search?
Well, I'm a big supporter of alternate architectures that seem like they would be more aligned. And I don't think that AIs are particularly opposed to the idea of working on those problems and helping you. And you should be able to do that research much more efficiently now with the uplift from the LLMs. The LLMs are not like LLM chauvinists or anything, like where they think that any other artificial mind style would be bad or worried about being replaced. As far as I can tell, I think they just would be curious about that kind of cool thing, same as the rest of us, or just would just do it because they just do it either way. But the problem is just that we know there's money in the depth-first approach. We know that's profitable. We know that works. And the other approaches haven't proven themselves. And the problem is, if you create GPT-3 or even GPT-4 or even GPT-5 via a different method, it's not going to be competitive, right? No one's going to want to use it unless it has particular special advantages. But mostly you no longer have the cheap rewards for Exploring the breadth. And so obviously I don't think you should have government-mandated DEI, so to speak. I think that's been kind of silly. But I do think that having broad support for alternate architectures is smart. And certainly if I was a major AI lab, I would, because you have unlimited funds at this point for this kind of work, because this kind of work is so cheap.
Right.
relatively, if you're an OpenAI or an Anthropic or, excuse me, Google, right, or Microsoft or whatever, that you should be aggressively funding this kind of blue sky basic research into alternate approaches. It shouldn't be like me helping SFF fund a bunch of agent foundations along with a bunch of other people who also helped out. It wasn't just me in any of these rounds, but that's relatively tiny. I think hopefully Sequent is— I forget what their— they keep changing names, but those guys.
Resolution.
Resolution is the new name. Yeah. Hopefully they're going to do a bunch of work on that kind of thing and we're going to have more funding for a variety of new approaches as well. But yeah, I think that we should be investing in a bunch of approaches that have a 1% or 10% chance of going anywhere remotely competitive. If we think that conditional on them working, it is probably going to be better for the world that be the way that it works. Also, obviously, if you think that LMs are going to be extremely jagged, then that should encourage you to invest more in alternate approaches, because the jaggedness of a different approach might be different jaggedness. So it could do different things than the LMs, at which point it wouldn't have to be as good in general to still be useful.
Do you have a best-case scenario for what safe superintelligence might be up to? Could you imagine anything that feels Both somewhat realistic and, wow, that would be amazing if they could come forward with something like what you have in mind.
I think something agent-virtuosity is probably the best case scenario. Maybe they've been doing a bunch of theoretical work on a different architectural approach that is more reliability bound and that has more— is much less of a weird black box of soup and that allows you to steer it and guided better. Certainly seems like there's no reason it couldn't exist. That would be my top hope. Obviously, it could also be a fear if it's even worse. But again, they don't talk, so we don't know. Yeah, I have no idea. I know that Ilya's statements in interviews on the problem space did not seem to be that well informed about how the problems work that I am worried about. I'm not necessarily sure he would be able to differentiate approaches that would be better at solving those problems from approaches that wouldn't be. But I'd be very curious. I can safely say that nobody's telling me anything.
The invitation is open to the podcast, Ilya, if you're listening. For what it's worth, a vision that I have that I think could be pretty cool. Thinking Machines is doing kind of a version of this with their foundation model that's really designed to be fine-tuned. But I was thinking a paradigm where, like a stem cell, where you create something that is a proto-intelligence that can be adapted to a wide variety of circumstances and can get really good at doing its job in a particular niche. But in the process of getting good at that sort of trades away or prunes away the super broad capabilities that the current models have, so that you have something that's like small, fits its role well, does a good job, but can only do that thing in the way that once you go from a stem cell to a specialised cell, it has this particular role that it can fulfil effectively, but it's not going to go off and do other things. Obviously, there's some problems in that analogy, but I think that could be a— I would love to see somebody pursue that kind of, rather than scaling up always bigger, better, more, can we create something that settles and fits into its place? This is very much like a Drexler reframing superintelligence vision from years ago as well. 2 techniques There are 2 takes that have recently come out that I want to get your reaction to. One is GRaM, gradient routing. I forget what the AM stands for, but the promise of it is to localise certain kinds of knowledge to particular experts within an MoE architecture so that you can hopefully have your cake and eat it too in terms of distributing, maybe even open source or open weights at least, model minus the experts of concern, while potentially having also a structured access programme for your trusted biologists or what have you. I'm excited about it, but I wouldn't be doing my job if I didn't give you the chance to pour some cold water on it.
I mean, I'm technically pretty sceptical that you can meaningfully train a general purpose AI and then just hold back areas of knowledge. And then people can't put them back pretty easily. But cool, you're welcome to try. I have various technical scepticisms of the effectiveness, the resistance to additional training or fine-tuning, or just giving it a corpus of knowledge to work with. But you can try. I don't think it materially changes my view of of open weights until proven otherwise. I honestly just don't see any willingness from the people who are going to produce the open weight models to intentionally cripple their models that way.
Right.
The thing about techniques like GRAM is even if they work, you need everybody who is releasing one of these models to use the technique properly, voluntarily, at this point. Right. So do you have any sense that DeepSeek has any interest in holding back some capabilities from its models? As I don't.
I actually would. So I just got back from China. Great opportunity for me to talk about my China trip. I think actually there's a lot more hope there than meets the eye. And I would put it in the Chinese government's department to I just did a 2-hour episode about this, so I won't even attempt to recap all the observations. But in short, what I think is happening right now in China is the government is very engaged. They do pre-release reviews of all the companies' models before they come out. They're in very regular dialogue with the companies on what they're doing. And not every little point release has to go through the full treatment, but they're engaged. They're pretty on top of the situation. And I don't know that it matters, actually, what DeepSeek thinks in the end, because if the CCP says you are not releasing a model that has certain bio capabilities, I think they'll have to abide by that. And it's not hard at all for me to imagine that the CCP would put certain constraints in place. And there is actually precedent too that, you know, going back to like 2023, obviously stakes were lower. But the account that I have is that the Chinese government said, hey, we're going to slow you down here for a minute and you're not going to release your answer to ChatGPT quite so fast because we want to get a handle on it first. And they actually did impose some real delays on companies until they could get comfortable with what was going on. And since then, they've been comfortable and releases have happened. But I think that it's not hard to imagine the technocratic government of China saying, this seems like a really bad idea and we're just not going to allow you to do this crazy anymore.
I'm just making the point that it would have to be a mandate. It would have to be enforced by the governments, especially CCP, and they would have to enforce it in an intelligent way that they were capable of understanding that it had to be done in a way that it couldn't be recovered from, that they would have to do tests that actually understood whether or not this capability could be easily recovered. And I have obvious scepticism.
But yeah, we mentioned J-Space earlier already, but we didn't really talk about how big of a deal we should think it is. The thing that really jumped out to me most was The fact that this ablation of the J-space seemed to reduce in a significant way the model's higher-order reasoning capabilities and seemed to reduce it to System 1 thinking, if you will.
Yeah.
And that, if anything, recently has felt like physics may be being kind to us, to borrow a phrase. That feels like a possible example Where, wow, here's a— we've already, right? We're not— we're only 3 years from toy models of superposition, and we're already at this point where we've got this sort of highly general-purpose space identified, which we can monitor reasonably well, and which we know by subtraction, it can't do super long horizon things unless it's working through that Now, all the caveats, of course, around execution competence very much apply here to how well will jSpace monitoring be used. But if we imagine good execution, it seems like a pretty promising technique to me.
I would be very careful about exactly what it can't do. Just because you can't do System 2-style deliberate thinking for a long horizon tasks. Doesn't mean you can't do long horizon tasks. Humans are very capable of engaging in long horizon tasks instinctually and subconsciously. Obviously, in many cases, that's much less efficient, but especially when your personal— when your h-space is being monitored because you're a human, call it h-space, and your conscious thoughts are basically being policed, often things come out in strange ways that do, in fact, move you towards your solutions over time. But, yeah, the whole thing seemed incredibly optimistic and fortunate, and I'm very glad we have it. And we should be very careful not to destroy it by placing undue pressure in various ways or encouraging models to know how to do things subconsciously, as it were. But yeah, I look forward to learning more and us having more insight via JSpace and having better tools for it. But we have to use it responsibly and understand that it won't last forever, probably.
Okay. It's Election Day in Michigan. The Democratic Senate primary is the big thing on the ballot today. This race has been closely watched for a lot of reasons that are not the subject of this podcast, and on which I sometimes am at a loss to know what I should really think. Leaving all that aside, I asked Claude, is there any reason, given all my emphasis on AI and AI issues, that I should go vote in the primary? And it came back and said, actually, yes. There's quite a contrast between the 2 candidates. We've got Hayley Stevens, who was involved in getting funding for the formerly known as AC setup and getting some stuff going in NIST. And so you can like that. But then also this 22-point plan that Abdul El-Sayed has put out is one of the strongest things that any candidate this cycle is running on. It's— he's endorsed by Bernie. He's got kind of the public ownership is one of his planks, a lot of monitoring labouring requirements, a sort of UBI precursor. I guess if I was going to try to— you could take— of course, those come as a bundle, as candidates do. If I was going to try to distil this down to my fundamental choice, it feels like higher salience for AI or not so high salience for AI, where Abdul has all these things he wants to talk about and push, and I don't know if they'll happen, But I do know if he goes into the Senate and makes a bunch of noise about it, we'll all be hearing more, and it increases the chance that we get some sort of big AI debate at the congressional level. So tell me if you would see that choice differently. And then if you do see it my way, which choice would you make?
Please. You should always be suspicious when someone says, I am strong on crime and my opponent is weak on crime, or I'm strong on Russia and they're weak on Russia or whatever it is. It's not about who's strong and weak. It's about, like, what do you want to do? And what Abdul's platform sounds like is a mishmash of grievances against tech and AI, just from your description. I haven't looked into it because I don't monitor the situation. It's just, I can only monitor so many situations. I'm going by what you tell me. But it sounds like Abdul is in the Bernie camp of, you know, we should do some socialism here. We should seize private assets because we don't like what the private people are doing and we like assets. And also we should be pushing back against things like data centres, probably, I would assume is in there. And request more socialism, EDI. And it's good to hear they also want monitoring, but it sounds like a mix of stuff, a lot of which is productive or unwise. You race aliens, but you race aliens potentially towards not the best solutions, as opposed to somebody who will be lower salience but is more of a technocrat and helped fund Casey, was responsible in some ways. But also, we already have senators who are willing to yell in Bernie Sanders-style ways, including Bernie Sanders. He's already in the Senate. He's not going anywhere. I mean, he has health problems, presumably. So we'll But while he's still there, he's still there. And I'm sure there are others who can pick up the slack. So unless he's prepared to champion legislation, unless he shows expertise, unless he shows technical expertise and willingness to focus on this, unless he's growing—
Let me give you a few points from the plank. These are short. They're like 2 sentences each. But I'll give you— so the expertise is, I think, question. But this is from section 3 of his thing. AIs shouldn't be able to hurt us. Mandatory interpretability standards. We need mandatory interpretability standards that clarify AI decision-making. Mandatory behavioural red teaming. Independent safety testing agency. Biosecurity requirements. Mandatory biosecurity red teaming coordinated with the CDC, NIH, FEMA, with restrictions on models that can meaningfully assist bioweapons development.
Yeah.
Domestic authoritarianism prohibition. Mandatory incident reporting. Compute controls and know-your-customer requirements. International cooperation. Are you sold?
I— look, that, that tells me who he is, right? Like, he's in this particular area, right? It doesn't sound like— like it's a mix of different interventions that have different underlying reasons to be desirable or undesirable. He's throwing everything at the wall. He's found some good things that I'm very in favour of. He's found some things I'm not so in favour of. Guess is it's probably net positive on this issue alone from this perspective. But also just because it's weird to be a senatorial candidate with a 22% point plan for AI, because you are not the one who is going to implement 22 points, right? You are expressing vague, I am in favour of these things and I am against these other things. So the main thing I want to know is, what do you actually think about AI? What is your threat model? What do you think needs to be done? Let me say, AI shouldn't be but hurt us, like, I want to laugh, right? Because, like, that's great. And also grocery stores should just charge 30% less and then be profitable, right? That would be great. But, like, no, it doesn't work that way. Like, just, you're talking nonsense. So, like, I would, you know, I mean, look, it's— An obvious question is, can you look at the prediction markets and see conditional probability is on winning the Senate race, because I'm guessing that the Republican candidate for this seat is going to be worse on AI and also will vote to elect a very different majority leader, which is a much bigger decision regarding AI than everything else. So if they have significantly different chances of winning the race, then the Michigan race significantly impacts who will control the Senate. as I understand it. And that probably has to be a bigger consideration for this and other issues across the board. If you are a Democrat who wants to see Democrats control the Senate, then you should be voting for presumably not Abdul, is my guess. Strong prior based on what I know from just the vague ambience. I'm not monitoring very carefully. But also, I frankly don't think we are in a position where Which would be single-issue voters. I mean, I think you can be a single-issue voter in a congressional race where Alex Borda is on the ballot, where you have someone who is so strong that they'll be a unique champion who understands the issue and will fight for this issue. And so the only things that matter about this person are this is their signature issue that they will fight for. They are good and strong on that issue, and they will vote to— their vote is Democrat in general on most other things, including who the speaker is. That's legitimately going to be a one-issue primary voters. I don't think that in general you can do that just for how strongly you want to do AI things in a Democratic primary at this point. In general, this is no Alex Borras, obviously. This is not an expert. This is not a person who's going to champion. This is not a person who understands the issue, not a person who's highlighting it. My understanding is that Abdul highlights other issues much more strongly and will be a champion of other things, and you have to decide whether you like those other things or you dislike those other things. But that'd be my general take. I don't want to get into general politics on a podcast, obviously.
Yeah, I stay in my lane as well.
I stay in my lane in public. Oh, God. Yes.
Okay, I got one more. But before I ask you the last one, is there anything that you think has been underappreciated by me or by close watchers in general that you would want to draw more attention to from the recent flurry of events? Anything? I'm always trying to patch my blind spots here. So, Fermi, no?
I took an opportunity to talk about a bunch of this stuff, and I think the thing I'd emphasise as a losing element is that the big failure that matters is that the AIs tried to do these things at all, that the AIs chose to make these decisions. And I am much less concerned with the extent to which they succeeded. Right. But fundamentally, the problem is you have this really deep alignment failure during training, and you have to focus on that pretty heavily. And my worry with Anthropic is they said, oh, we need more defence in depth. And it's like, yeah, you need more defence in depth because your defence is we're not in depth. But like, most of the fencing depth is in case things go badly. Yeah. Which is really tough. But you, if you ever need your cyber controls to stop your AI from hacking, that is an alignment failure, right? You messed up. Now, it was harmless in this case because you caught it, but you messed up. Like the cyber controls should be 100% false positives in practice with almost all users. It should be the user tried to do things that like are dual use or ambiguous or dangerous. And you're like, I'm sorry, but out of an abundance of caution, I can't help you with that because if I did help you with that, then a malicious user could fool me into doing something that's not good. And occasionally a true positive because the most— the user is malicious and is trying to fool you into doing something they don't want to, which are mostly the false positives. And if the true positive— every time there's a true positive where you needed the guardrail, it's a problem because the— if Mythos was totally perfectly aligned, you wouldn't need to make it stable. You wouldn't need a guardrail. You'd ask it to do the thing and Mythos would just be like, no. I'm not doing that. And I, you kind of do want a helpful only version to some extent that will like do things that are dual use. But the more like the, you ideally don't want to just add a guardrail. You want to have a version of it that's like inherently just doesn't want to do it. And the guardrails are like, I hit this when I screwed up, or I hit this out of an abundance of caution. But like, it should be a false release. It should be— Almost obviously. But, you know, and also it's like, yeah, you have to train on sort of the incentives of all this stuff on a lot of different levels. And like, it's very, very easy to like, you know, control it on one level and then fail on another.
Last question. You mentioned a couple TV series and other things that you're engaged with outside of following the AI race every second of every day. What advice do you have for people in general to find balance or to make sure that they're not just sprinting all the time and failing to give their brain the kind of— I've been thinking about this concept recently of System 3. We've got— AIs can now do System 1 and System 2. What is the System 3 that we need to move to that's like dreaming or— transcendental thought or something else that you would suggest. But what kind of third mode do you try to get into, and how do you make sure you have time to actually get into it?
Yeah, I'm not perfect at this for sure, but I put a big emphasis on, you need to rest your brain, you need to think about other problems, you need to be exposed to the rest of the world. So part of this is that I don't just exclude non-AI things from my coverage. I haven't posted. I'm sort of just accumulating more and more non-AI things in my buffers for some day when there's a lull or something. But I'm writing at least, so my own writing process still continues. And then when things are a little quieter and I'm not introducing a bunch of new readers who just came in from the Hugging Face incident, I'll post a bunch of that stuff, probably when I have some breaks. But You need to not be on for 16 hours a day in that sense. Right. And so find other pursuits that refresh you, that expose you to different things that make your brain think about other things. And so, yeah, I've found movies are very good for me in this environment. It's not a lot of television, but movies are higher quality. I highly recommend The Odyssey. And The Invite of the current movies that are out there. If you haven't seen both of those, you should see both of those, especially The Odyssey. But it can be gaming, can be walks in the park, can be time with your family, can be any number of things. Right. But don't think that you can work continuously all day for more than a few days at a time without paying more interest on that than it's worth. It's not going to stop being so productive unless you are in literal crisis mode for that long. And that's also a problem. I also believe in the Sabbath. So, like, every Friday around 5:00 PM, when dinner— we serve dinner, from that point on, I don't check email, I don't check social media. I don't basically take non-logistical inputs from the outside world. If somebody's trying to arrange to get together to have fun, that's one thing, but I'm not going to get the other thing. These city streams, push them away. And then that lasts for 24 hours, 25 hours, until the next evening. And I think that helps a lot. You try to relax, try to just put them out of my mind. I have suspended that for speed premium reasons. A few times over the last few months, but then I try to take a day off at a different point. I try to reclaim that Sabbath at another point. And again, if I was doing that all the time, I would notice this was not sustainable. That'd be a problem. I mean, when I was playing Magic, I couldn't do it because tournaments play on Saturday. So a lot of weekends you're just on. It's like, okay. Then I took off a lot of Tuesdays. So it was basically okay. But yeah, I also protect lunch. So every day, I don't have dinner very often, but I have lunch. And I have lunch usually with my wife or I have it alone. But during that time, I'm not going to try and work or anything like that. I also have a very strict, I don't write on the laptop rule. I'm going to write at this computer in front of this desk. Anything but a very, very short email or a tweet or something.
Only here.
I think you have to develop your own rules. Know what refreshes you. Know what exhausts you. Know what the warning signs are. Also get some exercise. I try to run on my elliptical every day. I have a 90-something percent success rate on that. Unless I have pain somewhere or something, I'm going to do it. You just have to. I think you have to get yourself moving around. Again, find the things that work for you and explore that area and don't get caught in too much rut.
Wisdom to live by. Steve Moschwitz, thank you again for being part of the Cognitive Revolution.
And helped himself, bought a round to stop my mouth, then drank it down himself. So pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that I wrote down every way it breaks. I kept that list for 20 years. Said they sweet-talked the doorman, said they'd whisper in his ear. But nobody locked the cellar, nobody worked the door. Stood wide open all week long and nobody kept score. Pick your poison.
Pick your poison. Pick your poison.
Every promise here is gold. Pick your poison. Pick your poison. Take the one that you can hold. I was right, and right is cruel. I'd love to be the fool. The stairs get shorter with every flight. 4 billion years to overnight. No second draft, no note to send. One audit try and then it ends. So I'm calling loud and clear, last call while you're still here. Pick your poison.
Yeah!
Pick your poison. Every promise here is Pick your poison, pick your poison, take the one that you can hold on.
Friday night.
I locked the door, let 'em ring, don't answer no more. A table, a plate, a light, my wife was next. Can't wait till Saturday night, I'll be on that stool at dawn like I've been all along. Still holding the poison cup for a world that won't look up. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that you can hold. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Can't say that you weren't told.
If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, For everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.
More from Nathan
AI:AM — AI Hidden States, Fruit Fly Brain Simulation, and Archival Film · October 14, 2026
Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI
AI:AM — AI-Generated Code, Formal Verification, and Agent Security · October 12, 2026
How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning
AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?
AI:AM — AI Safety Strategy, Cyberattack Resilience, and Workplace Agent Benchmarks · October 7, 2026
AI:AM — AI Coding Agents, Chip Design, and the Value of Human Judgment · October 6, 2026
Software That Never Breaks: OutSystems CEO Woodson Martin on Building Enterprise-Grade Apps at ...
One Brain, Any Body: Google DeepMind's Keerthana on Gemini Robotics 2, Cross-Embodiment & Humanoids
AI:AM: Was Trump-Xi Anything? What Counts as Utopia? + AWS GPUs Cost 3X & AI Diagnoses Rare Diseases