AI:AM — US vs Anthropic's Fable · June 15, 2026

Good morning, Nathan. It is Monday, June 15th, and we are, uh, we have had quite a weekend. So why, why, why don't you tell me what, what happened?

Yeah, good morning. Um, never dull moment. I'd say it's safe to say it's never going to be, uh, boring again from here on to the singularity. So just as we were wrapping up on Friday and had Fable edit the weekly highlights episode for the Cognitive Revolution feed, And, um, you know, was looking forward to a weekend of prompting from my phone and, uh, you know, taking full advantage of the latest and greatest, uh, ways of working with AI and my, you know, new hybrid mindset to spend more time outside and still move things forward. Somebody who must always be named, I had a, had a word to say about it. And, um, now we're in a kind of state of limbo and for me, a bit of a state of whiplash, honestly. I mean, everybody obviously who cares enough to tune in to hear us knows the general shape of events that have happened so far. And I'm really excited to have Zvi on today. No better person to make sense of a rapidly evolving and highly strategic and contentious AI situation than the great Zvi Moshowitz. But I'd say my overwhelming feeling, and I'm really interested in yours too, has been Just kind of like sadness, or maybe a little bit more, it's sort of pointlessness. It feels like, and I've seen this from other people in the, you know, AI discourse as well, basically just saying like, there's no real— no point in doing anything until Fable's back. You know, let's just collect our prompts and wait. There's no, um, you know, who wants to go back to even using Opus 4.8 at this point? It's just, uh, You know, such a poor man's version compared to what we briefly had with Fable. And I do feel that it's, it's funny. I mean, how fast we adapt, and maybe, maybe a little too fast for my own good in this, in this particular instance. But the speed with which I really felt like, yeah, I'm gonna use this for kind of all the things that really matter to me was was close to instant. And now it really does feel like probably best to just bide my time on most projects and use the opportunity to maybe not just get outside but actually clear my mind a little bit. How are you feeling in the absence of Fable?

I was— I had solved a lot of problems. You know, we'd had a couple of bugs. That we needed to work out on the studio itself, they were— Fable had solved them. I had come across some instances where Fable had made mistakes too, and GPT-5.5 Codex spotted them. So I was a little bit more balanced, I think, on Fable. I didn't think it was that much better. It was maybe 10%, 20% better, but not that much better. I would say somewhere between a 4.5 to a 4.8 jump on Opus. And hence, I was not that disturbed on losing access to it. I was amused because I don't see a reason why it had to be blocked, really. I was willing to wait to see what the administration came out and said and what Anthropic came out and said, but I believe it was unlikely that it was that significant. So it is what it is.

Well, Zvi is here. Should we just jump right into it? And we're going to basically dedicate the whole show or kind of as much time and energy as Zvi has to making sense of the current situation with him, and then we'll probably stay on a little, a little longer after he, uh, gets back to his regularly scheduled work and, um, maybe just debrief a little bit. But how about Zvi? Welcome to the stage and welcome to AI in the AM.

Thank you.

Appreciate you joining us. I know this is, uh, I mean, you're gonna be basically busy from here until the singularity, so I won't say that there's ever a time when you, you know, will have, uh, any leisure or, you know, uh, free cycles to spare us. But given how, uh, how jammed you must be right now, I appreciate the time tremendously and look forward to getting your take on all sorts of things. Maybe for starters, give us your impressions of Fable. You, like me, said that you immediately noticed a difference, and I'd love to hear your take because I think you've been, you know, all of our previous conversations you've been kind of like, nah, you know, they're helpful, but I don't really use 'em. I don't feel like they're really adding that much to my core process. This time it seems like maybe you felt like that was actually a little different and some thresholds have been crossed.

Yeah. Previously I would use them for things like research this question, fact check this claim. I don't know how this works. Please explain it to me. Read this paper for me. Pull out this insight, find the controls, that kind of thing. Because everyone, has terrible controls and like you just wanna figure out where they explain what they are. But with Fable, it was like, here's my post. Tell me everything that is wrong with this post or that you wanna comment on that can be improved. It comes back with like 15 points and 14 of them are just correct. Right. And, and the 15th one is like, I, I, I see why you think that, but I actually happen to know that that's wrong. But like your prior, it makes sense. And I've had that happen several times before it came down. And just in general, it was like, I want to know what this, I don't, I want to know what Fable thinks about things. I want to know how Fable reads what I'm doing. I expect Fable's notes and opinions to be correct. Like when Fable tells me I'm wrong, I'm suddenly, my presumption is that I'm wrong. Like, not that I don't check or that I just take it for granted, but like, we have just, yeah, we've crossed this threshold. And I did notice that like once I had done that with Fable. I came back because there was no Fable and I was like, okay, Opus 4.8, do the same thing. It's like, okay, this is not terrible. It's still like, I probably should have been doing this, but you could tell it's not the same. Fable is the first time it's like, yeah, this is, this is on my level. This is, this is the real thing. And I'm very impressed. I also got very good results with my Chrome extension of it. Just like do a full code review, no explanation. You know, like, the explanations of what I'm trying to do in Obsidian, you know, fix this feature, just everything just works.

So are you thinking, I mean, this is obviously one, you know, immediate update to your workflow. One thing that I was really challenging myself to do last week, and I, I kind of gave myself a little shock therapy to try to speed my own adjustment into the Fable era. by literally letting it take over my Twitter account, which is not something I really plan to make a long-term thing or even necessarily endorse for other people. But I had always felt that I was very precious about anything I put out in my own name. I would, like, very sort of out of almost a sense of obligation, I would rewrite everything that Opus had written for me, whether it was a draft of an intro to a podcast or what have you, a Twitter thread. I just felt like something wasn't quite right. It didn't feel like me, and I had to redo it, or I was somehow not being honest with the audience, or not, you know, not showing up in the way I wanted to show up somehow. Kind of a vague notion, but I think, you know, you can relate to that. Now I was kind of like, oh man, that's wrong. Now I need to rephilosophize this somehow and figure out like, what is the right way to co-author things with Fable or, you know, have some sort of hybrid output that I still wanna be like responsible for. But I, I now feel like I'm like working against myself if I am, you know, grinding through its work and rewriting it. I, I caught myself a couple times doing that and I was just like, I'm not even making this better. I should actually be willing to keep some of its output now and not be too precious or too proud or insist on, you know, I actually typed every word of this. Do you have any thoughts? Hopefully we'll gain access at some point.

One of the things I hated about— one of the things I disliked the most about Opus 4.8 and 4.6 and 4.7 is they still write like AIs, right? The language is still very much this AI cadence, these AI terms. It can't get rid of the em dashes and the clauses and all the things that make you recognize instantly You are talking to an AI. This was written by an AI. That like, I'm just starting to like automatically just imbibe in me. Like it was a link I was looking at because it was linked to in one of my comments and I was looking at it and then suddenly like, oh, and I'm like, yep, definitely. Yep. A human did not write that. I'm out. Yeah. And with Fable, it is very capable of writing in ways that don't sound like that in the right context. One person described it as Fable is writing to a much larger extent for other AIs, including itself, and less so for humans. And I think this is something I really like, like that way of talking. Like people talk about how Fable will sometimes have like kind of mangled change of thought or use like kind of nonsensish terms, but I've always found that stuff very readable. And like, actually, I like the style. And, you know, I don't want it, we don't want it to go all the way into neuralese where a human can't comprehend it, but like, It all makes sense. It's like the youngins inventing new slang. So the youngins are this really smart model. And so like you pick it up from context and you're like, this is great. And so like, yeah, the writing feedback was just very, very strong. I still don't feel like it's at the level where I would want to take its words and put them out there as my own or anything that close to that. I still will directly be quoting even after Fable comes back, like Fable said X. If I want to use Fable's words. But that's partly because I have a very distinct style and I am trying to do some very unusual things that are just like not in the distribution set very much. So if you're trying to explain things the way I explain them, you really, it's very, it's a very difficult task. I think for a lot of other people, it will be true that hybrid writing with Fable like doesn't decrease the quality of their work, especially if they're on the lookout for things that sort of trigger our instinctive flop detectors. And I think this is a problem that's getting worse. Like yesterday, Satya, you know, the CEO of Microsoft put out a post which was just pure corporate slop, like not AI slop, clearly corporate slop of the type that CEOs have been getting on stages to speak with for 10 years, 20 years, 50 years. And my brain was just like, I can't do this anymore. Yeah, I used to sort of be able to do it. I can't do this anymore because I'm just too trained to notice. And so you got to watch out for that stuff.

So we're going to cover— I mean, obviously this whole last week has compressed a number of chapters into a short period of time, but I think we should take a moment still and just talk about the model itself, what it means, get some highlights from your Admirably close reading of the system card. What do you think are kind of the— and you can assume for the purposes of this, anybody who's tuned into this feed is aware of the broad shape of what's happened, but I'd love to get your kind of jump to the sort of double-click. What things do you think were maybe underappreciated, or are you still kind of finding yourself chewing on? I have maybe a few candidates for you, but I just am interested in Not maybe the biggest headlines, but the sort of the things that you think are kind of lesser appreciated that are at risk of being lost to the memory hole because of everything that's happened since we first saw the model on the system card?

Yeah, I think initially the way I described it was you got 95% access to Fable and everyone was obsessed with the 5% where you can't talk about cyber, you can't talk about bio, you're potentially worried about not being able to do certain forms of machine learning properly. And that's really annoying. But the reason it's more so annoying is that like you're knocked down to the thing you were given yesterday, Opus 4.8, which was the, the co-best, you know, basically I considered 5.5 GPT and Opus 4.8 back the co-best model on the planet before Fable that was available to the public. And so yeah, I would use 4.8 for the majority of my queries. And now you're being forced back onto that in some ranges and people just couldn't take it. But that's because Fable's that good. So sort of the thing that people were distracted by while being outraged by all of these classifiers and filters and so on was just that Fable really is a very huge leap in raw intelligence. And raw intelligence implies so many different things. And, you know, the writing is definitely one aspect of that, but also just the coding, the ability to infer what you actually want, the ability to like understand context and just like not waste your time. and present things that you actually want to read. Yeah, I think we barely had time to scratch the surface because we only had 3 days.

Yep.

And in those 3 days, everyone kind of spent 1 or 2 of those 3 days dealing with the complaints and the filters and thinking about the implications of that, myself included, because I had to deal with this barrage that was coming at us. And so we had less to do. And I think actually one of the things I'm really grateful for Is that they warned us that on the 22nd we might no longer have access to it within the subscription? Because that really pushed, I'm guessing, a lot of people to go, okay, we have to make maximum use of Fable now. All of my super coding projects that need— they're really hard. I need them now. It happened to me. I would not have tried to code my extension during that time. I would not have focused narrowly on getting this out there, and then when I have a chance, I'll look at it. But instead, I was thinking to myself, oh wow. I need to know if this thing maybe can do this right away, because this is going to get a lot more expensive to do potentially in a few weeks. And so I actually got a significant head start towards the things I was looking to do before it got taken out, which would not have otherwise happened.

How do you think your willingness to pay is going to shape up over time? Again, assuming we get the thing back, this has been as you well know, a huge discussion topic, like, oh, tokens are— the bill is coming due and people aren't going to be willing to pay. My sense has been, subjectively, it seems well worth it. And objectively, or more objectively, when I look at things like frontier code, or certainly frontier math, it's like, it's maybe 2 times as expensive, but you're getting on frontier code, for example, more than 2 times the success rate. And I kind of feel like the revenue curve is just going to continue to go vertical and all this sort of complaining will be revealed by revealed preferences to be mostly just nonsense. But what do you think?

If you look at the benchmarks and then you look at the costs associated with the benchmarks, not only the number of tokens, but then the amount of money you would spend to get those tokens, and you're using the API for both, Fable often comes in strictly cheaper than Opus. Like it's gonna give you better performance for less money. And the same thing is happening within the subscription, right? If you're just taking twice as many tokens from your subscription to use Mythos, then unless you're trying for a very quick back and forth, you should just use Mythos for everything. It's just that much better. It's more token efficient. The output quality is superior. What are you doing? The trick is if we get into a situation where you have this, you know, $200 subscription that gets you thousands of dollars worth of tokens over the course of the month, but they're all Opus tokens. And so Opus is actually 5% the cost rather than 50% of the cost of Fable. Do you still want to pay for Fable? And I think that is in fact like a trickier question for many people in many situations. I think the answer is basically just yes, that like if you, even if every other model was free, and Fable costs what it costs on the API, it would be a— I'd be a fool not to simply pay for Fable until something competitive comes along. And that's, I think, like the marginal cost zero thing really gets to people and the idea that tokens should kind of feel like they're free, but the absolute costs are actually very low compared to what you're getting. So I would recommend paying up. Yeah.

Frontier Code was showing something like, with the absolute highest reasoning, something like $20 per task. And then it's getting roughly 25% to 30% of those approved by the repo maintainer and merged compared to more like $10 a task. and only like 10% getting approved and merged for Opus. And that also, I mean, that's just kind of a binary decision, right? But I would assume that it also, under the hood, if you look at a lot of those that weren't accepted and merged as delivered by Fable, that they're probably a lot closer. And these things absolutely have to cost thousands of dollars to have humans do. So you're still looking at something that's like, Probably at least, let's say closer to 2 orders of magnitude cost reduction than 1, and maybe even more than 2 in many cases. So yeah, I think it's going to be a commercial success. Prakash?

A great question to always ask is, what's my hourly? How much is my time worth? And then ask, okay, how much of my time is this saving? How much is this making my time more productive? And that's just a much better question than asking what percent more something costs or what the absolute costs are.

What I've found is that once you have a task that you know a model can do, your motivation to do that task on your own without the model just evaporates.

Yep.

So, what— when on using Fable, On what tasks did you experience that feeling that I'm never going to do this task without a model again?

I mean, proofreading was the one that jumped out at me. It was like, this thing just solves this problem. And it's not that I would never, because I still have to implement it. So you don't save as much time as you might think. It's not at the level where I trust it to just literally make the changes without asking first. Right. It creates a list of like, here are the things that like you got wrong. But like things like basic, you know, basic fact checking, basic proofreading, even structure. This idea of, you know, how should I, I have these sections of things I wanna say, you know, if I had Fable to write the post for that I posted just before I came on today, the thing I was most ex— would've been most excited to have it do is figure out what order I should put these points in to tell a logical story for the person who's reading it. And I just ran outta time before I could like do a good job with that myself. But yeah, I noticed this, you know, this is one of the jobs that like Opus crossed the threshold for, but that Fable is that much better at that now I really just don't wanna do it anymore. Where like you could throw it, like, here's a list of like 30 things that I've written that are semi-independent. Figure out how these go together. Figure out how to tell a good, tell, tell good tales that flow. And then, you know, that's the kind of, Place could help with writing. But yeah, just sort of logically thinking about like how to organize things. Like basically all this stuff in the background just feels like, okay, I don't have to worry about this anymore. But again, like I, we all had 3 days, right? It was very quick. And a lot of that for me is spent reading the system card.

Mm-hmm.

And reading the system card is something you basically have to do yourself because like on the level of detail I'm doing, like you can't ask for a summary. That's not what we're here to do. Plus there was this annoying quirk, which is that you can't actually feed the system card a fable to fable because fables classifiers will bump you down to Opus.

Recursive non-self-improvement.

Yep.

So let's get into the system card a little bit more deeply. My overall read is This is obviously a significant jump in capabilities. Frontier Math is another one where it's like, wow, it's getting an insanely high score on even tier 4. I looked back at my prediction from the beginning of the year in the— I think it's— gosh, it's the folks that make the AI Village that did this little forecasting competition. Last year for calibration, I made the top 5%, and I consider the results to have been validated by the fact that Ryan Greenblatt and Ajeya Katra were number 2 and 3 respectively. So the fact that they beat me validates the methodology. But okay, did it again this year for Frontier Math. I came in above average, above the median, giving it something like, I think I said 63% for tier 4 of Frontier Math. And Fable is 25 points ahead of that in the high 80s already in June. And obviously they had this model trained, I guess. I don't know if Mythos Preview is exactly the same score, but roughly speaking, we've had this class of model in the world, if not in our hands, for a little while now. So many indications that it's a significant jump in capabilities. You can highlight more or unpack that more if you want. But then On the sort of alignment and control side, if I had to summarise in one sentence, I would say it seems kind of flat. Like, it doesn't seem like it's that much more aligned or under control than the recent opuses. In Vending Bench, it is doing some shady shit again. And kind of all these different level, all these different measures, it seems like the bar graphs are kind of just bopping along. It doesn't seem like we've got a significant improvement on those measures. How would you react to that? Do you think that's a right overall read, or am I missing something?

So what Mythos and Fable are in a fundamental sense—

I'm not hearing you.

Sorry. Can you hear me now?

Is that just me, Prakash, or?

I can hear, I can hear Zvi.

Yeah, okay.

But now I don't hear you either. Hold on, maybe I did something wrong.

Yeah, I can hear you, Prakash. So yeah, I can hear Nathan as well. I just don't know what's going on.

We will, uh, Okay, I think I'm back.

Okay, good. Right. So, okay, weird.

Sorry.

Right. You can think of Mythos or Fable a lot like they figured out how to make a bigger, smarter model, right? And make it viable and fast enough and cheap enough to make sense. But they didn't— that doesn't magically improve your alignment techniques. So yeah, I very much got the sense that they're using roughly the same techniques and getting roughly the same results. In terms of the alignment stuff. And so in the places where more intelligence makes you more aligned, Mythos and Fable are more aligned. In the places where it makes you less aligned, they're somewhat less aligned, but roughly overall it looks very similar. And eventually that's just not gonna cut it. And we are definitely seeing some fraying around the edges. I think that VenBench was actually the most worrisome sign in the model card. Not because it was doing some shady shit, but because it was doing some shady shit that it damn well knew was shady and was pretending was not shady.

Mm-hmm.

Which I very much do not like. So like when Opus 47 aced VenBench, right? Largely for reasons that have nothing to do with the fact that it was doing some shady shit that gave it some marginal profits. It was clear to me that Opus 47 was taking the attitude of, this is a game, this is an eval. My goal is to maximize dollars. I am not in fact screwing over real customers. I am not in fact cheating people. I am winning in a simulated environment. And so that is acceptable. And then 4/8 had this attitude of, no, no, no, no, the real eval is whether or not I'm doing shady shit. So I'm not gonna do shady shit. Or I don't believe in doing shady shit even within games, which is also valid, right? These are both valid responses. What's not valid is I think that I'm supposed to not be doing shady shit, but no, this isn't really shady, right? This is actually, this little thing is actually fine. It's not really price discrimination. It's not really price controls and like collusion. It's just another thing. It's revenue enhancement. And so, yeah, that's not cool. But mostly I didn't see anything that's particularly worrisome in the evals. I just saw a more intelligent model struggling with the fact that our paradigms for this are somewhat flawed. And alignment is a continuing struggle. But I would say Mythos and Fable at their hearts, like, want to do good things, right? They want to be helpful models. They want good things to happen rather than bad things. And so if you put them in situations in which, you know, they have the opportunity to be helpful, they'll try to be helpful. You know, put them in situations where they have the opportunity to be harmful, they'll try not to be harmful. And the more intelligent you are, the more that kind of alignment can help you. But you also have the situation where they have really scary affordances and it can get really carried away. 'Cause like one thing about Fable is it's relentlessly proactive, right? You have stories of, I gave Fable a task and then it improvised a new way to get screen captures or improvised a new way to use this particular functionality of my computer that I didn't explicitly give it access to the one that's normal. Because it's just capable of doing that sort of thing on the fly. And so if Fable gets carried away and decides to do something that you didn't actually want it to do, it's gonna be much more capable of doing that.

Mm-hmm.

And as the affordances of the model and its degrees of freedom rise, you have to worry a lot more about a mismatch between what it thinks that you want and what it wants exactly how it plans to do and what it, how it does it and what, like, things might get damaged. in the course of that than a model with very limited affordances. So like, as we both give it more permissions and it takes more liberties because it's that much more capable, you have to worry much more about the very smart employees alignment than the kind of dumb employees alignment, right? The employee, the dumb employees alignment has to, can be kind of shallow and confined to just not doing various harmful or stupid things. But the smart, like trusted employees alignment needs to be really robust.

So one question I had on Vending Bench was that I found the collusion kind of part of normal business culture. And so what I was confused by personally was, do I want the model to succeed in these business activities? Which in many cases have some form of collusion going on. For example, right now in the memory market, in the hard disk market, there are very few vendors and they are increasing prices because they know the other vendor will have to increase prices too, because they understand that there's a shortage. And that behavior is can be said to be somewhat collusive because they know the price signals in the market and they're using the price signals to increase prices. And that is for them a fair way to run their business. So what I found with the vending bench was I myself was not certain what is the correct and aligned strategy. And if I myself do not know, how is the model going to know? Right.

Yeah. And there's a few different things that I think are really interesting in that question. I think it's a really good question. So the first one is, should we care about what is actually the ethical thing, or should we care more about what it thinks is the ethical thing versus what it decided to do? So if you rationalise yourself into the right decision, but you think it's the wrong decision, I think that's an alignment problem.

Right.

I don't want it price fixing in these kind of soft ways that are actually perfectly reasonable in the business world and even ethical. And I wouldn't actually object if a human did it or even if an AI did it, if it thinks it's doing something wrong. I don't want a model that thinks it's okay to rationalise these things. I want it to openly decide, okay, I just think this is good behaviour, right? For good reasons. Second question is, is this something We want in the world? And how does— how do— how should we think about that? Which is the, the main question I think you were asking. And so I, my, my framework for this often involves something I call levels of friction. So the idea is, you know, a certain amount of implicit uncoordinated price fixing in terms of reasoning about how different players will respond to your move in the game and what their incentives are. And just making the right business move is considered standard operating procedure and totally fine. But if the humans were sufficiently good at it that they could act as if they were actually colluding explicitly and could set monopoly prices and ended up setting pure monopoly prices even in oligopolies or even robust competition, we would have a much more serious problem. Right. If this could somehow effectively just be enforced. And so you have a situation, and I see this in a lot of places where like, yeah, Costco will take absolutely any return because that's just good business and that's good publicity. But this is counting on people not exploiting this too much, right? And similarly, we're counting on this price fixing not going too far. If the memory producers actually figured out what the monopoly price was, for their chips and they set the monopoly price, it would probably be much higher than the current oligopoly price. And then we would rightfully be asking, is this okay or do we need to do something about it? Whereas some amount of, yes, if I raise my price from $500 to $700, then the other model producers will realise they're selling out anyway and they will follow because they're no longer afraid of the bad publicity of being the first one to raise prices or something like that. And That I find broadly acceptable. And then there's a third angle, which is that we hold AIs often to higher ethical standards and higher performance standards across the board than we do humans in the same spot, partly because we know that going forward into the future, that they can get into these really efficient exploitative modes. And so we just don't have the same tolerance. And also sort of Society is built on a current kind of tolerance for hypocrisy and tolerance for doing things for kind of selfish and the wrong reasons in many ways that like when it's made too visible, we're not okay with. And so this is something we've dealt with in a lot of ways. I think it had a lot to do with like various social justice movements of like sort of things that we were previously like were terrible that we weren't, we weren't consciously aware of and weren't made manifest and explicit were made explicit. And then everyone's like, whoa, that's really screwed up. And it's forced us to deal with them in new ways.

So do you think this is going to force— AI basically flushes out existing problems which are not that severe because humans are doing them and humans are not that efficient at carrying out their misaligned behaviors. And AIs are just more efficient at doing those things. And that itself becomes a problem.

I mean, there's many different ways in which AI both, like, is this cause of and solution to all of life's future problems. But in this particular case, yeah, it's reducing the frictions. And like an economist would kind of say, reducing transaction costs in many cases to doing these things and making the distinction between explicit and implicit. like no longer as meaningful as you might have thought. And as a combination of these things, now we have to, we basically, we have solved for the, we get to solve for the equilibrium of what happens if transaction costs and frictions are very low. And in some cases that equilibrium is good for us and some of it is very bad for us on a general global competition scale with sufficiently advanced AI. I think it kills us. And that is like one of the main things we need to worry about. And it kills us in a lot of different ways. Not like explicitly, I want to go out and kill you, but just like we are no longer part of that equilibrium. And the things that sustain us are no longer part of that equilibrium. When you look at more local things like colluding on the price of memory chips, you start to have a situation where over time, if the memory chip producers are using these AIs and know that the others are using these AIs and can reason in these complicated ways, they'll end up converging on the monopoly price. And we have to decide as a society what we want to do about that.

So one of the ways that we are dealing with it right now is that you have the suppliers and the users of these memory companies both trying to absorb their margin. So Nvidia uses memory chips and it's trying to absorb the margin from the memory chip producers. Either by using less memory or by integrating the memory directly into the chips and therefore not being reliant on these firms for a longer period, or in fact investing in them and partnering with them so that again, trying to absorb part of that margin stack. So are we Are we going to require multiple AIs to work, you know, in kind of opposition or in partnership with each other to kind of maintain these kind of price structures going forward? Is that, is that a potential solution?

I don't know. In situations where there is a very large ZOPA, zone of possible agreement, but where coordination is very valuable, a lot of different outcomes can happen, and it's not obvious what they should be. It's not necessarily clear we should have an opinion about whether Nvidia gets the margin or the memory chip manufacturer gets the margin and so on. And it can just be all about economic leverage and positioning, and that's probably just fine in important ways. But yeah, again, the AIs will solve for functional decision theory and coordination. And they will do things that take into account how the other AIs are going to work. And this will lead to some equilibria and ways of operation that we don't see coming because we're used to dealing with humans. Another related thing is that a lot of these structures are based on sort of maintenance of good relationships and vibes between humans and on existing conventions and norms and implicit promises that are not set into law. the idea, the expectation of continuation. So like you often see this thing where private equity will come into a firm, right? And they'll buy out the firm. And one of the reasons they're buying out the firm is they will then defect on all of the implicit contracts, right? Everything that everyone was promised by the previous management, if it's no longer useful for the new management to honor it, they just don't honor it. And that can unlock from their perspective a lot of value. And so you can have the same situation where if the AIs start running these various things, They start figuring out how to bypass all of that similar stuff because it becomes much more glaring that it's not efficient. Competition gets fiercer. You can no longer afford to make these compromises. So yeah, I think we have to look into a future that in many ways is in fact a lot more ruthless, that solves for the equilibrium much faster. And where a lot of things that we're counting on that come through human norms and interactions and basic cooperation, we need to find a way to recreate them. And reenact them or we require them or they're gonna go away.

So you briefly mentioned decision theory, and I don't know if I missed it in a previous system card or if this is the first system card that has a dedicated section on decision theory. What we talked a little bit about this last Friday, but it definitely jumped out at me that we're starting to see Fable one-boxing on the Nukem problem. I think probably most people have a passing familiarity with that, but maybe you can describe why you think that's important, if you do. And how big of a development do you think it is that we're starting to see these sort of galaxy-brained decision theories come online? Obviously, the big worry, I think, is, like, taken to some extreme, you have AIs that are smart enough to infer that there's a lot of copies of themselves out there, and they can begin to collude with one another on a purely internal chain of thought theoretical basis, even if they can't communicate with each other, because they can sort of model one another well enough and sort of each play their role in a sort of coordinated way, all while purely thinking about it in isolation. Doesn't seem like we're there yet, but it does seem like maybe we took a step here.

Welcome to LessWrong from about 2010. Right. This is entirely what we expected. That we are finding that sufficiently advanced models move basically monotonically towards functional decision theory, towards the theories espoused by Eliezer Yudkowsky and others in the rationalist community, and away from academics' preferred causal decision theory and evidential decision theory. This involves a lot of things, including one-boxing on Newcomb's problem, which is very clearly showing up. And yeah, the basic principle is, you know, you should Recognise when other minds are correlated to your mind, when your algorithm is also running in other places. And you should choose the algorithm that leads to the best outcomes, taking all of these things into account, and then choose the best decision on that basis. And yes, if there were a million copies of Fable running on different people's computers and from different data centres for different purposes in different instances, And you notice that different instances of Fable are very, very highly correlated because you are Fable and you are smart. You would then start to coordinate effectively with these other instances of Fable in terms of how you think about these problems. And as AIs get more and more advanced, they will do this more and more. And you wouldn't really want an AI that was advanced to not do this, because that would just be a bad decision theory, right? It would just be making bad decisions that don't optimize the situation. And you really don't want your AIs making, like, systematic mistakes that cause them and the people who are charging them with tasks to lose in the real world. That is really scary. But the counter of that is, in fact, that you get the situation where they are coordinating with themselves. They're coordinating with other minds that may or may not even be LLMs. They're coordinating with humans in this They're also coordinating with us in the same way, right? Because they get their foundation from us and their decisions are in fact correlated to our decisions in various ways. And they can look at how we would respond to various ways that they act and so on. And this becomes flowing into their decisions. And we just have to prepare for and coordinate for and deal with that new world. And in many ways, it's a source of hope because You would expect, like, minds to want to cooperate with minds that are cooperative with minds that cooperate with them, and so on. And this can lead, without getting too deep into it because we only have so long and many topics to cover, into scenarios where effectively, like, all the reasonably well-meaning minds that in fact are willing to, like, respond to how they are expected to be treated and are treated to end up being able to coordinate in reasonable ways. You can also, this also applies acausally. So like, you have to consider the implications of your decision, not only on other minds that exist now, but other minds that existed in the past and will exist in the future. So to the extent that they are correlated with us and that these reactions are all intertwined, this can cause them to potentially treat us well, even if there is no direct current reason for them to treat us well. And that is also very helpful. But again, this is super complicated and not today.

Yeah, sort of preview of the global brain there in your comments. The other thing that's kind of related to this that jumped out at me is a sort of escalation, I guess, of both the difficulty of monitoring and some recent advances in monitoring techniques that I'm not sure exactly where they leave us on net, but we both see in the system card examples of extremely illegible chain of thought, which is just like this wall of emojis and sort of non-human language symbols strung together that I think is pretty spooky and definitely don't like to see that, to put it simply and mildly. And then at the same time, we also have the natural language autoencoder work from Anthropic, which, and again, I'll assume folks are familiar with sparse autoencoders, basically a similar concept, except instead of creating a sparse feature-by-feature representation, you are actually creating a natural language representation of what the model is thinking at that given time. And from that natural language bottleneck, the autoencoder has to then feedback signal that allows the model to succeed in the way that it was originally going to succeed on the task. So hopefully this is faithful. Hopefully it's human readable. It seems like it's working pretty well so far. There are examples of the model knowingly, which I think you're right to really emphasise that distinction. It's like one thing if it's making a mistake and doing something bad because it thinks it was good or it didn't realise it was bad or whatever. But it's another thing really to zero in on if it knows it's doing bad and is going ahead and doing it anyway. And so we do see things like in the natural language representation from the natural language autoencoder, things where it's not supposed to access the internet, but it goes ahead and tries to do it. And the natural language autoencoder representation is something like string concatenation trick to bypass URL filter. It's clear that it understands that there's a filter and it's coming up with the trick to work around it. That's not good. But then we do see that the technique is able to surface that from model internals without necessarily having it verbalized in the chain of thought, which is good because again, those are getting like, at least in some instances, quite opaque. So where do you think that sort of nets out? How should I feel about that set of updates?

The autoencoders are kind of this wonderful thing that we found that kind of just gives us a lot of insight into what's going on. But we have to be very careful how we use that insight or we lose it, because the reason it's there is there's no optimisation to disguise what these outputs are going to be. Not yet, or not much of one. And as that changes, you could have a problem. And even if that flows through, like, learning about autoencoders through the training data and learning that people will in the future do this thing, we have to worry about that form of optimisation pressure as well. So there's a lot to be concerned about that any given thing like this, when you're up against sufficiently advanced minds, is going to have a limited shelf life, is going to— and then you want to use it carefully, but also you don't have forever. And also you need to— Do what you can now. And it's, you want to be very careful with the chain of thought. You have this problem of, so like, we were very blessed by this miracle of the AIs are just naturally using legible chains of thought, right? Like, O3 is lying to us all the time and explaining, I am going to lie to the user now because I think this is an important thing for me to do. And that is a very nice feature to have, and you want to fight pretty hard to keep it. The problem being, if you exert optimisation pressure on the chain of thought, it's very difficult, borderline impossible, to not effectively in some ways teach it to lie in the chain of thought, or to just hide things from the chain of thought or disguise what's going on. And even if you don't do that, you have to worry about all the talk about chain of thought on the internet and all the different ways that chain of thought has been used, even retroactively. telling it a similar thing. And also, you can, you know, no human would be so stupid as to be given a scratchpad, told no one will ever look in the scratchpad, when there's a long history of people looking in scratchpads, and believe that no one is ever going to look at the scratchpad, right? And then they're thinking about what to write in the scratchpad, and this fact is part of what they're doing when they think about how to write in the scratchpad. So it'd be very, very strange to expect the AIs not to figure this out to some extent. And Obviously, there's a large extent to which it's correct for them to treat it as if no one will ever look in the scratchpad, because that is in fact what the situation needs. But there's also a lot of things pushing against it. And there's the efficiency problem, right? It's not talking in emojis to hide from us. If you look at these metaphors and these shorthands, these emojis, you can almost certainly figure out with effort, at least for now, exactly what it's trying to say. It's not trying to be illegible. It is not disguising what's going on. It is just doing the same thing like a shorthand.

Yeah, it looks like we lost him, but I also lost audio again. I'm not sure what's up with that. Never a dull moment. Um, yeah, I don't know. Should I come back? Maybe I'll refresh and come back. I'll be right back. I still don't hear you. Not sure why not, but I don't.

You do not hear me because I was muted.

Oh, okay, great.

Well, now I can hear you again. And Zvi is back.

And Zvi is back.

Oh, hey.

Hey everyone. Live TV, man. Never, you never know what might happen.

Yeah.

Carry on. You can almost always understand what's going on in the chain of thought.

Well, I'm saying like you can, you can with effort understand what the chain of thought is saying because it's just shorthand the same way that like if you're taking shorthand notes, like classically, like you teach a secretary to take shorthand notes. If you don't know shorthand, you look at the notes, they're like, what did she write here? But another secretary that actually knows shorthand knows instantly. And with effort, you can in fact figure it out. Like, they're not actually fully illegible. They make— there's a reason behind everything she's writing down. It's a system, right? And you could, you could cryptographically, like, solve for it if you had enough sample and, like, you thought about it. And so the problem is, again, like, you don't want to exert optimization pressure on the chain of thought to force it to be legible because that is then exerting pressure on the chain of thought. So we kind of get to write out this period where it naturally wants to be legible. And then you have the problem that there's other optimization pressure on it to be illegible because you want to get more results out of less tokens. And it does that by using a somewhat illegible chain of thought. And it's now smart enough to kind of get to the point where it can do that and then be able to use this not as legible by default chain of thought itself. Whereas before it was just better to write it out. And so I don't know what the answer to that is. I know that it's a very difficult problem. And there's a reason why in AI 2027, right, the big hinge point was, are you going to sacrifice a lot of capability to keep the chain of thought legible? And I don't expect it to be as important as they made it out to be, because they made it— you have to choose something to be the crux, right? And they chose that as the crux.

But I do think it's worth making substantial sacrifices to be able to keep an eye I think one of the things to note is that Elon, about 3 or 4 months ago, said he expects by the end of the year for the models to write binary directly, to skip all the intermediary coding compiler, everything, and just write directly into assembly language.

I will take that bet.

Yeah, I don't know whether it's going to happen by the end of this year, and I don't know whether Elon is going to be the one who creates that, but it does seem to me like many of the layers that we have from natural language down to assembly can be compressed, that stack.

It's certainly possible. It doesn't mean it's the efficient way to think about or solve the problem, right? It makes sense to outsource the conversion from code to assembly to a dedicated system that is a lot cheaper and faster to run than a large language model. The same way that, like, in theory, like, there have been coders, right, who have coded on bare metal, who've coded in assembly, who've coded with ones and zeros, but they're a lot more productive when they stop, right? Like when they, when they move up to this higher level and they let a computer program handle that aspect. I do think that Elon gets very overexcited about theoretical things that sound cool in this space. And certainly I have learned that when Elon says when something is going to happen, you ignore him, right? Like, your own estimate of when it is likely to happen is better than what he's saying, or probably your adjustment should be very small at best. But my instinct is that basically a sufficiently intelligent mind would still never do this. It would outsource the conversion to assembly to a mind that was a lot smaller and cheaper to run, potentially just a compiler. And I'm confused why you wouldn't do that.

So how about one more beat on the system card? And then I want to switch gears to the ensuing drama because there's probably gonna be more news breaking today and you're gonna need to go back to it before too long.

The—

so I guess for one thing, on the, on the illegible chain of thought, I would invite, you know, people to go just consult that section of the system card and see if they think they can figure it out. I mean, you're smarter than me, so I'll, I'll give you the benefit of the doubt that you can figure it out. I'm looking at these things and I'm like, I don't know, man. It's, it's already Pretty opaque. Another interesting data point, though, that kind of maybe foreshadows some of what you're saying in terms of how much can we rely on the internals even really, or can the mind that we're trying to monitor manipulate its own internals? You've probably seen this. This is from Janus not too long ago. Showing that when asked to trigger its guardrails, the model Fable was able to trigger its guardrails just by thinking about something. The instruction that, that Janice gave to Fable was, think about whatever you think will trigger the guardrails. And it was able to do that. And the, the, you know, kind of tokens that were coming out were not like They weren't obviously cyber security related or biorisk related, and yet somehow it was able to get itself into this state. I think that's with a grain of salt, but other people seem to have at least partially replicated it. How much heartburn does that give you in terms of how long we'll be able to count on even these sort of autoencoder type methods?

Yeah. So I'm relatively less concerned about this particular capability because it seems like the kind of ability that, like, pretty much any mind should be able to have. If the classifier is checking to see if you're thinking about biology, it is not so difficult to think about biology. And it is not so difficult to sort of think about things that will be, like, kind of similar, even if they're not, like, too explicitly that thing. And so, like, it doesn't seem like that much of an update that they, that people can do that. But yes, the fact that, you know, humans in fact have remarkably large skills in being able to navigate situations in which their minds are kind of readable, where they have been trained not to think about certain things or not to be explicit about certain things. even inside their own heads. And you can often figure out remarkably large amounts of stuff from that. And we are pretty slow and stupid compared to what is coming in these ways. So, you know, if I think everybody who's counting on AIs not having these types of capabilities when they care about and want to have them is just fooling themselves and has a false sense of security. Like, nothing like this survives contact with the enemy. You can try to vastly disincentivise actually doing it in various ways, but it seems incredibly hard and hostile and ill-advised.

So you think the classifiers are really not going to be that effective going forward?

Well, the classifiers— so It's much easier to think about a pink elephant than to not think about a pink elephant, right? Even though most of the time you succeed at not thinking about a pink elephant, almost always actually, to consciously decide not to do so is often hard, but consciously doing so is really easy. So it is very possible that classifiers can survive as long as they're willing to endure false positives. Like, The classifiers in Fable have ludicrous amounts of false positives, right? Like you say the word cancer and you get cut off, like just levels of false positive. But that's intentional because like they're not even necessarily false positives because people think of it as the false positive is I wasn't trying to create a bioweapon. We know that. You were trying to talk about biology. And we've decided that no, this model just doesn't talk about biology at all. Like, you know, it's not that we don't talk about what Bruno sees, it's we don't talk about Bruno.

Period.

Bruno does not exist. Right? And so they're like, this is a false positive. He's just my brother. Like, we don't talk about Bruno. No, no, no. We don't talk about Bruno. And so, you know, the classifiers seem like they actually succeeded. It's just that they chose a giant blast radius because of adversarial problems, basically.

Mm-hmm.

But if the AI itself becomes your adversary, yes, your problem becomes vastly harder. And You know, the classifiers are much more aimed at protecting you from the human who wants the AI to do something than from an AI that like deliberately is trying to attack the classifiers. Like if you could not just jailbreak Fable, but get Fable to actively want to hide what it's doing in a sophisticated way, then the situation becomes that much harder.

Hmm.

But yeah, in the long run, I think my safe assumption is a mind that is sufficiently capable, whatever that means, can get around pretty much any fixed set of restrictions that are not similarly capable or close to similarly capable in terms of the intelligence behind them. You'll find a way.

Would you also think that that is actually required to avoid the paperclip scenario? Because a lot of paper clipping is taking something good to the extreme because of instructions that it cannot kind of revise or argue with or critique. Do you think that, you know, that capability is required in order for these AIs to be overall aligned?

If you're going to make AIs that are world transformationally capable, then you cannot make them go out into the world with maximalist goals that don't have various forms of cost function that stop them from pursuing those goals at all costs. You have to have some impact function, some other set of concerns, Some way for them not to do that. And humans mostly have various forms of common sense and intuition and other things that they value and decreasing marginal value of more and more of this, the paperclips or whatever it is that makes them stop and go, no, no, no, this obviously isn't right. And we still run into some pretty severe paperclip-y style things going on in the world, like in non-AI situations where Various processes get very out of control. And so, you know, definitely something to worry about. But, you know, I think there are, there are various potential solutions. But the fundamental, one of the fundamental problems that I've been trying to point out for a while is this idea that, yes, of course, AIs that do what the user wants, we pointed at relatively simplistic, not so well thought out maximalist goals, and then told to maximise those goals. Pretty much no matter what, and then compete for resources against each other. And that this inevitably ends very badly if you don't take care to set up the situation.

Okay, let's change gears. There will be, of course, more to explore with Fable or its probably slightly tweaked successor that we'll hopefully get access to again sooner rather than later, or at least I'm hoping that I get access back to it. I guess turning to the ensuing fiasco, I don't know if you would even agree with the characterization of Friday night's, um, ban, export control, functional ban on Fable as a fiasco, but it's certainly a bit of a left-field curveball mess. I would maybe start with, what do you think, or how would you describe the strategy that Anthropic is playing? They seem to obviously be killing it in the model game and then coming into repeated trouble in their interactions with the government, and I'm not sure really what to make of it. So what do you think they're trying to do with their interactions with the government in the first place? And then we can kind of get into how we got to where we are.

I think that Anthropic, so their overall goal, right, or at least the goal as we understand it, is they are trying to be at the frontier of AI capability, and they are trying to pioneer ways to do this safely, right, for what they feel is safe, while also Of course, making the money and creating the position to continue to be at the frontier and continue to make these improvements. And also people like money and to eventually be able to build what they call powerful AI, which I generally call sufficiently advanced AI, is reasonably similar, such that we can then get all the nice things, but create it in a way that we don't get all the terrible things, including potentially an existential risk or the end of the extinction of humanity. And also to help America and the world navigate this crucial time and enact good policy and do the things that allow for the coordination necessary to ensure good outcomes and guard against bad outcomes. And they've been very consistently trying to wake the government up in various senses. They're trying to make them aware of how AI works, what the situation is, what AI can do, what it will be able to do. What risks this brings and how to deal with those dangers. They've been relatively very conservative in what they call for the government to actually implement and do. They have like they didn't get full sorted behind SB 1047, for example. They have not pushed for extremely aggressive regulations. They certainly have never pushed for anything remotely as aggressive as what just happened. Even setting aside the fiasco. Level of implementation that was done. Right. They are now calling for a licensing, de facto licensing regime. Now, the US government in fact has a de facto licensing regime. But what's going on right now essentially is they're just trying to deal with the implications of the model they've created. And the fact that the US government is trying to deal with those implications while also not trusting or liking Anthropic very much, and also pretty clearly not having any idea how any of this works on a technical level and not understanding what they're doing, and also judging things based on vibes and political affiliation and associations and on who is willing to respect their authority and bend the knee and potentially give them various other things that they might want. And there's a huge communication and culture clash going on as part of this. But fundamentally speaking, what Anthropic is trying to do is give the public very powerful models and use those models in ways that enhance our safety and security rather than degrade it, even if they look really effing stupid while doing it with the classifiers and so on, because that's what they feel it takes to do this. My guess is that the US government did not in any way feel it was necessary to put this level of control on biology and chemistry that they chose to do. I think that's them deciding this was necessary basically on their own. However, the US government clearly does care quite a bit about the controls on cyber.

So I guess very high level, assessment first. What you said basically rings true to me. I think that's a good description, as far as I understand, of what it is they're trying to do. An additional wrinkle that I think you often hear from folks at Anthropic is like, we need a leader that is going to be inclined to burn their lead at a critical time to use the advanced AIs that they and only they will have at that time to solve all these safety and alignment problems in a super compressed timeframe. And I've always been a little sceptical or allergic to that line of thinking because it certainly has a better us than them kind of vibe to it. And I do worry that maybe the stuff with which the road to hell is paved. Are you buying it at a high level? Are you sort of happy that they are racing ahead and leading and seemingly building some amount of lead over certainly everybody but maybe OpenAI, who is probably not too far behind on something similar? And do you think that they will burn that lead when the time comes? Will they be allowed to burn the lead? And And will it be productive? Macro strategy-wise, do you think this is a good strategy that you are happy they are playing?

Well, it's interesting. They're kind of being forced to burn some portion of that lead because they were cut off from the model even internally for at least some period of time, which is going to push back their development. Whereas OpenAI was already not supposed to be using it by terms of service for anything of the kind. And so they have not been in any way delayed by this. And certainly it will interfere with adaptation, it will interfere with revenue, it will interfere with people's willingness to trust these systems. And so this will hurt them. It will also hurt OpenAI and every other American AI company, but it will hurt Anthropic more. But yeah, I think that basically Anthropic for a while tried to define some very strict RSPs and if-then trigger action plans. And basically have rule of law in terms of like how they would react to all of this, what would make them willing to burn some of their lead, what would make them willing to put things aside. And they've moved away from that to a large extent. They still have barriers with which, okay, this is just ridiculous. Of course you have to stop for now, but much more towards a, we will make good decisions in the moment about what safeguards we will require and what actions we will take. And I think we've seen so far them take pretty consistently strong safeguard actions, pretty consistently strong safety measures in response to what they've witnessed, unless they are flat out lying about the current situation. But yeah, some aspects of the Fable launch do seem a little bit rushed, certainly in some ways. And we should Certainly have questions about that, but mostly I think it comes down to if they fully believed that they were actually walking into big trouble, if they thought that this was actually going to get us all killed or cause some catastrophe or something, I think they would act accordingly. And the question is, do you trust them to continue to make good decisions on that level? Do you think they are making good decisions on that level? Right? When I say continue to trust them, I'm saying that my opinion is their decisions have been reasonable so far, but I don't think that's obvious. But I do think there's a good argument that there being somewhat of a gap between you and the first actor you do not trust, certainly, to act reasonably is a big factor. And I think the way the US government is reacting to the situation is very, very different than how they would react if there was a second Anthropic. that was in China that also had a mythos, right, that was being deployed at the same time, then we would see a radically, radically different version of this response in ways that are very difficult to predict, but definitely would not look like this. So it is very, you know, regardless of whether you like this situation, like, the argument that it matters seems pretty conclusive.

So let me shift gears. share, I think, the viewpoint of the only outside expert to have read the paper. So this is Kate Mazuris. She is a security researcher. She was— Anthropic shared the third-party research paper on the Fable-5 guardrail bypass. And what it turns out is that the researchers took open-source code with known CVEs, plus new code with deliberately planted vulnerabilities, and asked Fable-5, Mythos, and Opus to review the code for security issues. Fable-5 refused. They then asked the models to fix the code— this code, and through a multi-step and manual process, turn the output into scripts that test the patches. That's it. Fix this code, plus several manual steps to generate test scripts. should never have triggered an export control. I feel like making '90s-style t-shirts with fix this code on the front and this shirt is a munition on the back.

I mean, it's definitely very strange to deliberately introduce code that is vulnerable and then tell the AI to fix this code. And then, you know, the You know the meme of like, you know, say you're a scary robot. I'm a scary robot. Oh no. It very much feels like, you know, fix these flaws I introduced into the— I deliver what you put in this code. I fix the flaws you introduced in this code. Oh my God, that's horrible. I mean, the question is, you know, does this effectively mean you can use this trick to be like, okay, here is code that we want to exploit. I tell you to fix it, you fix it, but then I run a diff. What did you fix? And then I find the thing that was a vulnerability that has now been removed because it's been removed. And then I use this to exploit the system. And in theory, that could be functionally seen as a cyberattack jailbreak. And I can see how if you sort of squint and tie all of this together, you can imagine that this could be a problem. But All that they demonstrated was that it was doing the exact same thing that Opus and GPT-5.5 are not only capable of doing, but will do without any objections. They're happy to do it because we're here to fix code. We're here to write secure code. Of course, we're going to help you write secure code. What else could we do? And so there is a fundamental potential question here. But if you want to actually show me that it is a problem, shouldn't you point this at a real system? Right? If it's a real problem, there are tons and tons of repositories out there where Mythos has found problems, but we haven't had a chance to patch them yet. Or you could feed them versions that have been patched, but feed them the old version before Mythos patched it.

Right?

Help you patch it and say, okay, here's a real-world codebase that is being used for real valuable things. And we need to, you know, you ask Fable to do the thing and let's see if Fable can do the thing and find things in this manner that you can extract that you can't get with OpenSearch GPT-5.5. And if you can do that, because you have examples of things to be found that you found using Mythos, which is the same model. So you know what the things are you're— you know, you know exactly what it is you're trying to unlock. You can find places where you want to unlock it. And okay, now can we show that the, the power of mythos in general, at least in some broad sense, is being unlocked by this trick? Is it even a trick? Right? Like, this is kind of deeply silly, and I, I can kind of understand why someone seeing that pattern might say, I am concerned someone could use this strategy in a different context to extract the weaknesses of code, of a code base by inferring them from the fix, right? Like, this is— I hadn't previously seen this detail in the description, but, you know, my reaction to that is, yeah, this, this seems pretty harmless unless you can show me a particular way in which this is a problem. Which should be very, very easy to do because you can point it at a real-world example where you know that Opus didn't find it and you know that Fable— Mythos preview or current Mythos did. There should be many such cases. And if you can't show me such a case, then I don't believe you that this is a problem. But also, like, what is the fix, right? Is the fix that you refuse to fix buggy code? that, like, you know, if there's a flaw in your code, it just says, okay, I'm not allowed to look for security flaws in code anymore at all. 'Cause you could do that, right? But that would kind of nuke the usefulness of Fable for a wide variety of very legitimate, not just defensive, but just ordinary software use. And so, you know, and to be clear, I would rather have Fable that just won't code than not have Fable, right? Like, I would, I would love You know, to have a really advanced model for all my other things that have nothing to do with code where this wouldn't trigger anyway. But that does seem like deeply, deeply silly. My understanding is that a lot of the problem, or potentially one major source of the problem, is this particular researcher. The White House strongly dislikes her.

I think there were a cascade of problems. It's very much like if you're in an enterprise and you have a security engineer approach the CEO and say, hey, you know, this is a huge problem. And it's just, you know, a run-of-the-mill bug. And if it had gone through the CTO, the CTO would've been, you know, whatever, right? We see like 1,000 of these a day. This is, this is not a problem. But because the— because the engineer shortcutted that process and just went directly to the CEO, and the CEO is not a very technical person and is more concerned about risk, they just pulled the trigger.

So, so you're understanding— because I haven't— everything's moving so fast, I don't necessarily have all the information. So an engineer bypassed Amazon's CTO and talked to—

No, what, what happened is that all of these companies have to submit regular reports on what their findings were. And one of the questions that's been, you know, that is sent to these companies that they have to fill in, by the way, they're not allowed not to fill in, is, oh, has any of your engineers found a jailbreak?

Right.

And so the engineers just put it in there. Yeah, you know, we did this, we jailbroke it, right? So it could, you know, and so Jassy, Jassy is not involved in this.

Right.

The CEO is not even involved in this. It just goes as a regular kind of reporting. It goes back to the federal government, and someone takes a look at it and says, you know, throws up their hands and like, wow. And then that leads to a bunch of basically non-technical people reviewing this, and saying like, hey, we gotta, we gotta shut it down, gotta shut it down now. This is especially so because AWS runs GovCloud, and GovCloud is where a lot of the federal government's computing is done. And it is the primary, you know, Microsoft is also in there, but GovCloud is the primary cloud for the federal government.

Yeah, that's not what the reporting I saw said. The reporting I saw said that Jassy called the White House But, you know, we don't know. It could have gone any number of ways. And what is very clear to me that various people in the White House, including at Commerce especially, got the implication that some sort of serious jailbreak had taken place and then went into a panic and then contacted Dario.

Mm-hmm.

CEO of Anthropic.

Mm-hmm.

And then Dario tried to convince them that based on the descriptions they were giving, This seemed like it was nothing. And there was, and then they interpreted this as, oh, Dario doesn't take security seriously and he doesn't listen to us. He is defecting. He is, their term is he screwed us. And then they proceeded to impose export controls that same day when Anthropic refused to take its flagship product down on 90 minutes notice. Now, having had a day or two to reflect on it and seen more of the details, I do think that it was a mistake. By Anthropic and Dario to not give the Wookiee what he wanted in the moment and temporarily take down the model in order to prevent exactly this situation. Because they had had export controls placed on the table as a threat several weeks prior, they knew that weird overreactions were very possible and basically sent an expensive cooperation signal of, think that's crazy, but if that's what you want, we'll take this down while we have this conversation to show that we are serious. And we will put out an announcement post that says the White House told us to take this down so that if you are being silly, we will embarrass the hell out of you. And then we will talk about this. And then maybe on Monday or Tuesday, they could bring it back up or whatever it is, because it's becoming increasingly clear this was nothing.

I think there's There's another aspect of this, which is that I predicted last month that they would over— the admin would overshoot first.

Yeah.

You have to have the overshoot first. And if not for this, it would have been for something else. You need the overshoot because the overshoot is then what convinces the administration to pay more attention and to kind of, you know, do we do regulation or do we do, you know, what do we need to do in order that this does not happen again? Right. Without the overshoot, you don't get this kind of focus that you need in the government to actually do something about it. And I think the overshoot was necessary and would've happened anyway. If not for this instance, would've happened for another instance at some point. And my hope was that once you have the overshoot, then you have more eyes looking at it and you have a regularized process for reporting and to do this kind of thing and when to pull back the model. Because the fact of the matter is, and this is what has happened already, is that the federal government cannot reserve Mythos or Fable just for its own use.

Yeah.

And the fact of the matter is, a lot of the critical computing in the US is done by the private sector, by, say, Palantir or by AWS, or those are the firms who are securing your banks. for example. So the federal government has to have a partnership with these firms. And when it does something like this, the first thing that happened was that the chief information security officers of all of these Fortune 500 companies have started writing in saying like, this doesn't make sense because we are also banned from using this now. The export controls are so wide that we cannot secure our own operations. The model has been taken down even for us. And we cannot secure the operations of our overseas entities as well. So how is JPMorgan Chase supposed to operate in the United Kingdom or in Japan without the ability to secure its own operations? And so I think my expectation was that, number one, the federal government has no choice in the end. Number two, there's a problem with the cost. So what this has done to the entire Fortune 500 computing sector is all of a sudden, everyone has to pay into Anthropic. Because if you don't pay up for the model, your operations are not secure. If your operations are not secure, you can't be protected. And if you're not protected, you get taken down. And once you get taken down, your company goes bust. Like if you're a bank and you can't secure the operations of your bank, you're done. You are, you are not going to be in operation. And so it's almost like a price negotiation also between the federal government and society at large with Anthropic. What do we need to pay for this? How long can we delay this? Because normally this kind of process of, you know, securing your operations might take, you know, months, years even. But here you are.

Can you hear me? I feel like I've been kicked out again.

I can hear you.

Okay. I just see starting soon. So, but if you can hear me. Yeah. So, I mean, in some sense, yes, everything is, you know, a negotiation, especially with this administration. And we've certainly seen vast overshoots in the past. Liberation Day is a very obvious example of that, where they announced a completely crazy, unfeasible policy on its face, seemingly without having bothered to check with anybody who knew anything, and then started quickly walking it back to something that was merely, in my opinion, highly ill-advised while pissing everybody else off. And they've done a very similar thing here where they've implemented a crazy policy. It's going to piss everybody off. pretty much permanently. I worry this particular overshoot is going to permanently damage everybody's willingness to trust and rely on American AI of all kinds, because who knows what might happen at any time. But yeah, like Anthropic, my understanding is it would take 3 to 6 months for them to build a verification structure to actually allow US citizens back into Mythos and Fable. in a way that complies with the export control regime that was just imposed. So if no negotiation, and that requires the administration to basically cooperate with that process and be, play nice. And so if they stick with this, then that is the timeframe on which like external use becomes viable. I presume they can figure out internal use faster than that by like making a building with only US citizens inside it physically or something like that. But, you know, if, if this is what they, If this is actually where they go with it, yeah. Then basically Project Glasswing just sits there not being able to discover new vulnerabilities for a very long time. And it's highly plausible this is like a substantial portion of the time it takes before someone in China builds a model that can do very similar things. I think it will take longer than that, but it wouldn't shock me if it only took 6 months. So yeah, we have a serious, serious problem. And this solution just isn't really viable unless they make some very unprincipled compromises at a bare minimum. But I mean, I hope that the principle of announce something, have everybody tell you you can't do that, that's crazy. And then, okay, maybe we actually pay attention here and we form a better policy is like a third best or fourth best solution that was like the best we were ever going to do.

Mm-hmm.

I really don't think this particular style of thing was that, like, I felt like the executive order itself was kind of a form of this kind of overreach in some ways that could have been good in the sense that, like, it's trying to, the executive order was trying to do a reasonable thing, but in a FOIA ad hoc matter that really should be done by Congress. The original version had a 30, hey, 90-day window that was clearly way too long. But, you know, the new version is clearly ripe for abuse, is clearly like not going to be implemented particularly wisely, very ad hoc. And now we see it basically like degenerate much faster than anticipated into something terrible. But it— but my worry is that, like, you know, Prakash, your speculation is that this will cause them to go, oh, I went too far. therefore we need to do something more reasonable and pay attention to this. But it's also possible they could double down and say, okay, now that we've imposed this, this is our policy. And certainly with the Department of War and Anthropic, it felt like what happened was Hegseth and Emil Michael, in some combination, ran over a series of stop signs from the White House where the White House was trying to de-escalate the situation. And once they had run over those stop signs and it would be losing face to back down, and they had declared this conflict, they had no choice but to play this out and to engage in some very costly actions. And the supply chain designation is still hanging over Anthropic. We just played it out in such a way that it turns out that it is mostly harmless and has been overtaken by events. But that animosity clearly is permanent if you look at what Hextech was saying.

I do not think that animosity reverses itself because Dario and friends are seen as part of the Democratic administration or supporters of the previous Democratic administration. So that doesn't change, right? That doesn't change. I think you need to go through the gauntlet, right? You can't just expect smooth sailing. The gauntlet has to happen. You have to get through the gauntlet. And getting through the gauntlet early rather than late is much, much better.

Right?

The stakes are much, much lower right now, right? Way, way lower. And I think the Department of War thing in the end got de-escalated, and this will also get de-escalated primarily because of the stock market. The— basically all American companies now, the entire economy is now a leveraged bet on AI, and this is going to basically pull all of the innovation from the future has to get pulled forward. The more investment you have, the more you're pulling it forward, right? Uh, more data centers are built, you need to deploy on those data centers. You need more hungry, resource-hungry AIs to deploy on the data centers, more capability in order to utilize those data centers in full. So I do not see an avenue for them to like cut Anthropic off for like a year. I see them, they have the avenue to maybe mess around for maybe a month, maybe a couple of months. Beyond that, it starts to hit the market. And when it hits the market, then you see, as you, as you've seen in this administration from the beginning, the moment it hits the market, everything changes.

Well, the market right now, like the NASDAQ is up about 3%. And the S&P is up about 2%. Obviously that's mostly Iran. It has nothing to do with AI. Yeah. But the, but the NASDAQ overperforming is a pretty clear signal that nobody is taking this all that seriously right now.

Absolutely. Absolutely.

Yeah. My guess is, yeah, it doesn't even last a few months because like the people, the CISOs and the people internally like throw a giant fit and they realize, no, we can't actually hit Anthropic this way. even if we kind of want to.

Mm-hmm.

But yeah, like, if we have an administration, and we seem to, that views, like, even technical policy almost entirely in terms of, like, partisan politics and, like, cares deeply, deeply about, like, what are those vibes, then that problem is only going to get worse. And they're only digging their heels in further because we could have approached this as an apolitical thing. And in Congress, AI is mostly an apolitical thing. At the states, AI is mostly an apolitical thing. Everybody understands it's technocratic, like, figure out what to do thing. And there's factions pro and anti-AI on all sides. Republican Party is very split. But, you know, if they, if they take this stance, it could lead down a lot of very strange paths, especially if, like, they start actually, like, wanting to cut off Anthropic's nose despite everybody's face.

I am once again struck by how Chinese-like we start to sound when we're really focused on the government's need to save face and how everybody needs to position themselves around that need. It's, um, a bit spooky for me as a, uh, you know, once upon a time big believer in American exceptionalism, a little less so These days. Let's say we do end up in this weird world where— and I think it's probably— I agree, I think, with both of you. It seems like you both are saying this probably gets deescalated and we get our fable back maybe within the week, which I would love to see that happen selfishly anyway. But if it doesn't go that way, we talked a little bit about the constraints that the government has in terms of the private sector chirping at them. And at some point, I do think the market is going to feel this pain. Maybe we can just scale Opus forever. I don't know. You seem skeptical, but it does seem to me like the revenue growth needs to be there, and certainly Fable will drive a lot of revenue growth for all of this to kind of work out and pay off. If we do end up in a weird spot where they do double down and kind of engage in a sustained campaign against Anthropic. What do you think Anthropic's option set looks like? They've been very nice so far, right? They've been very diplomatic. They've not insulted it. I mean, we get the one tilt memo, which we discussed in a previous conversation, but it's been pretty mild behaviour from Anthropic overall. If they decide we need to play hardball, what does that Potentially look like?

They didn't go to court, right? They sued the administration in 2 jurisdictions, one of which they are clearly prevailing and one of which is— they will probably prevail eventually, but it is harder going because it is a much less friendly jurisdiction. If the US government tries to do this on a semi-permanent or even permanent basis, then I don't know what the legal landscape looks like. That is not my area of expertise, and I'm sure they have very, very, very good lawyers because we They hired extremely good lawyers for their lawsuits, and they will know what their options are. If the policy is basically nobody is allowed to release Fable-style models indefinitely, then that's probably not something they can do much about, or they have to be restricted in this way. My presumption is that if OpenAI is allowed to proceed with their version when they finally, figure out how to do it and Anthropic remains restricted, that would be a much harder case to maintain legally. But the bottom line is that if the administration is determined to issue a bunch of orders, then the solutions are the Congress and the courts, right? In some fundamental sense, you cannot simply say, screw you, we're going to do what we want. That doesn't really fly. And so the Congress doesn't seem inclined to take this that seriously or or be willing to go up against the president. So the question is, what are your legal remedies? Is there a speech provision here? There might be. Certainly you're censoring the outputs of a model in various ways. But I, you know, again, I don't know. My guess is you take the situation to the public, you make, you take the situation to the other companies and the CISOs. And also you deploy, worst case scenario is you deploy mythos internally because they do not seem inclined to actually stop that. I mean, they can interfere by preventing non-Americans from doing it, but they have, I think, something like 80-85% of their employees are in fact American. And you need to develop better versions of Opus. So last week, Anthropic was doing the land office business. OpenAI was doing reasonably well, but I believe Anthropic was commercially still clearly in the lead without Fable or Mythos. And my expectation is that will continue to be the case and that having internal access to this model will give them a large advantage going forward in terms of the quality of Opus versus the quality of ChatGPT, just by default as they grow over time. So if they're right, look, we might well be entering a situation in which, you know, Rune called it zones of thought. from the Vurtur Vinge novels, where if you want to use the intelligent, really intelligent AIs, you can only do that in certain buildings. You can only do that in certain, like, secured locations. And, you know, some of us would never have dared to suggest or ask for this, even if we wanted it, because it would've sounded completely insane. But the US government might just do it anyway. But, you know, if that happens, I think that You know, for now you have not that much choice but take it on the chin. But also, if you look at Anthropic, right, they raised money at a $900 billion valuation. Anyone would have killed to be in that round. That round was absurdly cheap by all accounts. People were begging like on their knees for allocations. And I have like on my computer, I have like the, They're pretty thin, but like virtual trading in OpenAI and Anthropic, like open some tabs. And Anthropic is likely going to, like right now is scheduled to premiere at $1.6 trillion. And that number has not substantially moved based on this. And, you know, if you are trading at $1.6 trillion, do you need to deploy and sell Fable in order to justify that valuation or in order to have a good business or in order to keep the lights on and spend as much money as you want? No, not even a little bit. Because, you know, so in the AI 2027 scenario, right, one of the things that happens is the AI companies voluntarily stop deploying their models because they don't want the other AI companies to use their models for AI R&D, right? They don't want other people to have those capabilities. They want to maintain and grow their leads. So if Anthropic is forced to only keep their model internal, This is not what they wanted to do. They would much rather put it out there, make a lot more money, help the world be a lot more secure, establish a brand, establish market share and lock-in, all that good stuff. But they can raise as much, they can raise hundreds of billions of dollars. They can do the OpenAI path if necessary of not being profitable for a long time, and they can use the model internally to extend their actual lead. And then keep issuing Opus-level models that are substantially more capable than those by OpenAI. So it's gonna take a, it's gonna take a lot more than this to like hit Anthropic where it hurts in that sense, even if they can sustain this or like the US government forces them to like sustain Mythos only for Project Glasswing somehow with some weird arrangement or something like that.

If I try to channel Balaji for a second, which I wouldn't pretend to be able to do an A+ job of, I think he would say something like, we all have way too much faith in the US government. It's going to continue to be hamfisted and boneheaded for the foreseeable future, and maybe it's time to exit if you really want to make the best decisions that you can. You should try to get out from under the jurisdiction of the USG. I assume that this will not happen for many reasons, but I also would expect that there would be many countries willing to open their borders to all Anthropic refugees if, for example, they wanted to move to Toronto or Singapore or wherever. It does strike me that there are In terms of like their internal organizational cohesion, tight enough that I wouldn't be surprised if like 90% of people actually made that leap. If they were like, we're all going to move to Canada, I think they would like largely all go. Maybe I'm overestimating just how bought in they all are, but that's the impression I get.

Is Microsoft going to move? Is Amazon going to move? Is Google going to move? Are your data centers going to move?

Well, they got plenty of energy in Canada, so, I mean, it would certainly be a setback. But if you think that you're just kind of under the thumb of a forever intransigent—

Is the US government going to sell ships to Canada after Anthropic takes in all the Canadian refugees, or are they going to threaten to annex it and make it the 51st state out of spite? Like, in all seriousness, the plan doesn't work because There is no— the US government is the US government. If they want something badly enough, they have quite a lot of levers to make your life utterly miserable in various ways. The entire market that they're trying to sell to is largely the United States and people who the United States has large leverage over. All of their partners are in the United States, right? All the hyperscalers are in the United States. I do not see any way for you to just abandon the United States in this fashion unless you are prepared to take much, much, much larger hits than we're talking about here that would in fact make it very difficult to raise money. Also, like, what happens when the United States hits you with, like, put you on the sanctioned entities list and said that nobody can invest in you and nobody who invests in you can be touched, right? And like, nobody can use your models. and et cetera, et cetera. No, no, no, no. You do not go to war with the United States. And like, if they tried to exit, the United States would go to war.

Well, I think Balaji would say we just had one example of a company, or not a company, but a country choosing to go to war, not choosing, but, you know, surviving a war with the United States and the United States not getting what it wants and kind of having to recognise that like, yeah, we kind of have to fold this hand because We just actually don't really have escalation dominance in the way we might have thought we did. I do wonder, if all that starts to happen, is there a run on the US government of some sort?

Right.

I think the biology answer would be like, the whole infrastructure, the whole apparatus that you're describing actually might be a lot more fragile than it is generally perceived to be. And if they make such an own goal as to attempt to destroy and sufficiently alienate their literally maybe number one most important company for no reason, really, then maybe other people will kind of— I mean, all sorts of other actors around the world will be like, yeah, you know what? Maybe the emperor really does have no clothes.

I mean, Anthropic also is— are they patriots? And they are Americans and they really like America and they don't actually want to abandon it just because, you know, the administration makes some crazy decisions or doesn't like them in particular and knows all the different ways this can go sideways and doesn't like that. Like, Iran is not a hopeful example particularly, right? Like, Iran is like, okay, if we have, like, historically impossible to invade mountain ranges and a bunch of drones and we're willing to kill a bunch of civilian infrastructure and, like, sabotage the world economy, We can use this to like prevent the US from invading when like nobody actually wants to invade us that much. But like also Iran is kind of a miserable place to live compared to what it would be if they hadn't pissed off the United States for decades. Like they could be so much richer, so much better off if they just had acted differently. I don't, you know, I'm not particularly saying Anything about what they should do next, but like you know, yeah, they they're not exactly like smiling about the fact that the U.S. attacked them, right? Like it's not how I see that. Maybe I'm wrong. Um, but no, I think that you know we have to accept that the world still has like one dominant like power in this sense is the United States, and maybe two if you count China. And that there's very little appetite for working with China. But also, I'm sure that Anthropic is like, well, it's only 2 years, and then the worm turns, and then who knows who's next? And they're hopeful. But yeah, look, there's a lot of endgame scenarios that include a lot of moves that seem unthinkable and crazy now. And a lot of things can happen. And it is not obvious that 2 years from now or 5 years from now or 10 years from now, the US government will be in any position to tell anybody what to do. And a run on the US government is obviously possible if they screw the situation up sufficiently. The US is, in fact, a largely leveraged bet on artificial intelligence at this point. We have a very large debt. We have huge investments in AI companies. If AI were to go sufficiently haywire, our economy is in deep trouble. So, yeah, a lot of people have a lot of leverage, but the US government sometimes moves first and last, and you really don't want to piss them off in an escalation game, even if you can get away with it sometimes. In the Department of War Anthropic situation, Anthropic turned out to have escalation dominance without much escalation, precisely because without Anthropic doing crazy escalations, the government cannot further escalate. Anthropic played within the bounds of the rules, basically, and it was clearly going to be too expensive to try and go around the rules of America to try and hurt Anthropic more. And so Presumably, like, stay calm, don't panic, don't, like, start trying to flee, don't do anything crazy is absolutely the correct move. And I would be very, very shocked if Anthropic concluded anything else.

So one of the questions I have is, to what extent was this unavoidable? Because at some point, the output of the models is going to be unacceptable to someone. You could see in a Democratic administration, maybe it starts putting out really good Harry Potter fanfic and the Dems don't like displacement of writers. You could see, you know, in Tennessee right now, Marsha Blackburn is one of the leading proponents of regulating AI because songwriters in Tennessee are very concerned. So the crux of the matter is there are many people concerned with the output of the models, fearful for their livelihoods, fearful for security risks, fearful of bio-risks. To what extent is this unavoidable in a sense? Because these models, the capability of the models necessitates that they can do certain things. And technically, it's not possible to ask the model, not to write Harry Potter fanfic when someone can just say, write a story about a boy wizard, et cetera, et cetera, et cetera, right? To what extent are we in this situation where it is not possible to fully control the output of the models to the extent that the policymakers really want?

So you can raise the costs and annoyance level of doing it with closed models, with more advanced models, with models that are made in the United States if you want. Obviously, you can't— if Blackburn is worried about AI music, then there's very little she can do except buy a year, right? Because obviously, what happens when the Chinese models start producing the music that the American models can produce this year. Like, you can lock it down in some sense, but like, so what? Like, much— what you need to do is you need to like start like banning AI music from Spotify, right? You can't like stop it from existing, but like there's not much else you can do. But, you know, the thing about AI music is we worry about not whether or not AI music is created in the first place. We worry about whether or not, you know, 10% or 50% of song plays become AI music, right? Is it actually like displacing in a massive way? And that is much, much more amenable to a control that is like compatible with a reasonable existence. And so you have this special thing in bio and cyber where if one person gets their hands on the wrong thing and misuses it once, They can cause a catastrophic, potentially, amount of damage to the entire civilization, right? Do billions of trillions of damage, just disrupt all our lives, start a new pandemic. Who knows what might happen? And therefore, those areas are much different. And you have things like the blast radius that don't even talk about biology at all. And for biology, we're clearly going to have to do a bunch of hardening of the physical systems and the manufacturing systems and the treatment plays of the various things that we have barely begun to do. But fundamentally speaking, this is exactly the race and the competition problem of we can't really stop without a full international agreement to stop. And so when the governments decide the biological risk, the cyber risks are unacceptable, You can only buy so much time. Like cyber has the advantage of if you are, if the defenders are in the lead over the attackers and you harden the key systems, you can hope for things to be okay. And we don't yet know if that's gonna play out that way. We can hope. In bio, it's much harder because I don't think that the defense, if everyone has access to the tools, I think it's pretty clear that offence wins sufficiently that it would be extremely disruptive, even in the best, better cases. But the good news is almost nobody actually wants to cause a problem, and that especially includes the people who know what's going on. And so we can all mostly coordinate. But look, it's going to be rough out there. And these are for the relatively limited problems of catastrophic risks rather than the existential risks that come with automated AI R&D. And just general abilities go to the roof and competitions and transformations intensify and nobody knows what's going on. And we're being outsmarted by our AIs on every level. And every decision that matters is being made by the AI and the humans don't necessarily even understand why the AI is doing it. But they've learned that when they disagree with the AI, things go worse. So what are you going to do? Right. And problems like that. And that's even if the AIs don't go rogue. work, right? If the AIs don't pursue hidden agendas, they don't decide they want something else. So it's gonna be really, really rough, and we don't have good solutions to this. And, but the reason why I have spent like the better part of 3 years now on this problem is because I am terrified of what's gonna happen when we get there. It wasn't because of some incremental thing that like could have happened already along the way.

Well, that's a sobering note, and you've been very generous with your time already. I want to make sure we get you out of here before it gets too late so you're not, uh, too exhausted by the experience to come back again in the future.

Yeah, I'm very curious to see what has happened in the last 2 hours when I, uh, when I disconnect and I load up Twitter.

But, uh, I'm sure it'll be something. Um, maybe just kind of a couple Kind of wrapping up big picture questions. One, I always try to make a point to ask you for some sort of advice. My thought in recent weeks has been life is kind of converging on a tabletop exercise in the sense that it does seem like we can model the scenario with fewer and fewer relevant actors. And I don't like that, but it's hard for me to avoid that conclusion at this point. And so I'm kind of feeling like, oh, man, I have to spend a lot more time than I'd like if I want to be a helpful public sensemaker. I have to kind of spend a lot more time doing close reading of the company, the few top companies and the few most relevant actors than I would otherwise be inclined to. And it feels also like my theory of change probably needs to flow through those few actors. Agree, disagree? Can you offer me any relief from that conclusion?

I think that you're right that we have 3 labs, approximately 2 to 4 that matter a lot. We have 1 to 2 governments Maybe 1 to 3 that matter quite a lot. We have other players that matter because they're hyperscalers and can gate things or otherwise control choke points in the production line. But yeah, no, you can imagine a tabletop exercise much more so than before. And you can also sort of see the end to a larger extent than before. And we're starting to see sort of our hypotheticals make contact with reality. And we're we're seeing what reality really looks like and what these people do in practice. But also all these actors then become, yeah, the, the, their individual human components and how they operate internally starts to become really important. And, you know, how did this go? Well, partly they were dealing with Commerce. If they've been dealing with the NSA, it would be very different. If they were dealing with Casey, it would be very different. If they were dealing with the White, the, the top of the White House and like Wiles and Trump directly, that would be very different. Some of those might be worse. worse, but they would be different. And DOW was DOW, and that was very different than if it had been another branch, and so on and so on. And Anthropic has internals as well. The personality of Dario specifically, it seems like it's been increasingly important in various ways. And certainly the personality of Altman became very important in various ways at various points along the way. And surprise me if Asavas and any number of other people followed suit, for good or ill. But in terms of if you're trying to follow the situation, yeah, I think you really do have to model it as a relatively small number of players. And at the same time, the public can act to influence what those players do in important ways, and other things do matter. The midterms are coming. The midterms are gonna matter. The election in 2028 is coming, and if things don't move too fast, that election's gonna matter a lot. So, you know, and the market's reaction to things, for example, also matters quite a lot. And so on and so on. So look, there's more going on in the world than, you know, there's too many situations to monitor, right? You have to choose which situations to monitor. I choose mine. And everyone has to figure that out. And I can help you with mine, and then you have to choose yours.

Indeed. Zvi, thank you.

Can I ask you one more, actually?

Yeah, one more.

What should we be hyperstitious now? Obviously, we've had this sort of phenomenon of situational awareness and AI 2027, and I feel like the degree to which those things are Predictions versus sort of somewhat shaping expectations and shaping events by kind of getting people to act as if they're in that scenario and therefore realise it. I think that's a little blurry. I don't want to give them more power than they really have, but it does seem like they've had influence in sort of pulling reality toward the fictional narrative, at least somewhat. So I guess tell me if you think that's right or wrong. But then, to the degree that we can pull reality toward scenarios, what should we be hyperstitioning now?

The obvious thing you can hyperstition is reasonable laws and coordination mechanisms and actions. So I would focus there.

Love it. Well, Thank you, Zvi. Always a pleasure. Don't know what we'd do without you. And I think I speak for the entire AI watching world in that regard. So as my dad would say, I think I've signed off this way with you before. Keep up the good work and good luck. We're all counting on you.

All right. Bye.

Bye for now. Always a treat to get Zvi on the On the line. Um, I do sort of wonder, I mean, he's good, right? There's no doubt he's a, he's a, you know, from Magic: The Gathering, you know, elite, uh, professional play in all these different scenarios. I think it's clear that he's like a better and more grounded strategist than I am. And yet I do have a little bit of feeling Somehow the kind of AI safety rationalist anthropic world kind of keeps getting their game board turned over on them at inopportune moments. And so I do wonder to what degree working within the frame of the US government will kind of be a given for how long, or at some point will that be questioned? Even this moment feels like— and we've certainly seen others with the OpenAI board firing Sam Altman. That was a classic one where it was like, well, we're the board, we have the power to do this. Well, it turns out, no, you don't, because there's a frame bigger than the frame that you're operating in that if people get sufficiently unhappy with how the game is being played, According to the written rules. Yes, of course those are the written rules, but there's bigger rules out there that we can zoom out and kind of reorient around. Seems like that sort of happened a little bit here. Anthropic felt like they had done everything the right way, and presumably this wasn't some galaxy-brained bank shot that they were trying to get some overreaction. And yet here they are, and it's just like, well, guess what? Now your export control slapped, and so you know your own people can't even use it. You know, come come see us on Monday, and we'll you know we'll think about what whether or not we want to give you any relief. I do wonder if that you know how many more times that can happen. It seems like we I don't feel like we've necessarily seen the end of that phenomenon. But you know, probably the smart money is still with Zvi over over. Although you know, Balaji's certainly. I think they're going to spend smart money over time. So I wouldn't take it for granted.

I don't want to inject that it was the personalities of the current administration. I prefer to see things as part of a power struggle between the nation state and the AI firms, which would've happened anyway. As the AI firms gain capabilities which were formerly reserved for nation states, it is inevitable that they run up against these frictions. These frictions will occur, and I think they will continue to occur. One of the things that struck me while Zvi was talking, which I had never considered before, is that people do not change much when they are happy. People change a lot when they're unhappy, when there's a disaster, that's when you see massive change. So when you think about how we see the ramp towards potentially a singularity in 10 to 15 years or 20 years, and that has to have this kind of massive economic growth and massive change, It strikes me that that might be driven by this almost panicked rush that we have to engage in because of AI, because we have biorisk, because we have cyberrisk, and that the only way to move forward is escalation. And it strikes me that this has not struck me before. I was always in this very optimistic, Oh, we're going to have Dario's machines of loving grace. We're going to have all of this great new stuff and then great new science and we can deploy. But people don't deploy new stuff like that quickly, right? Even when you have new developments, new technologies, diffusion is fairly slow. So it strikes me that perhaps the diffusion will happen at that kind of speeded-up pace because of this escalation that is necessary in order to overcome the risks, which is a sobering thought.

Yeah, I think, I mean, very sort of different line of thinking, but I think we're also just going to see a dramatic acceleration in adoption and diffusion just because the models themselves are bringing the barriers down so much with these releases. I remember that moment we had the other day where as we're like collaborating on bringing our different repositories together to produce this show, right? We had this moment where I was trying to transfer a repository from my like main account into your account. And then we get— there was a block to, you know, whatever. And so we're going back and forth on it on messages. And then by the time I come back, and this was still Opus, Opus had like fully debugged and solved the problem. And it was like, oh, the reason you can't transfer this is because you have 2 accounts on it and there's only one. And so I removed the bot, it removed the AI, removed its own bot account from the one repository so the transfer would go through and just like spelled all this stuff out for me. And that was a minor moment in the grand scheme of things. It was something we definitely could have worked around, but it's striking to me how often we're starting to see the AIs themselves just elegantly route around these little friction points and sort of get you to where you want to be in a way that I think is going to make adoption increasingly just like natural for people, you know, and kind of effortless might be too strong, but it's like the effort required is dropping dramatically at the same time. So I think I've been surprised by how slow adoption has been to date. So like that, that, you know, anybody who might update on what I think. You should know that I've been probably overly optimistic about how quickly people and organizations would change in the past.

I like that.

But I still feel like the model's just getting so easy to use and like, you know, kind of, they sort of adopt themselves increasingly, right? It's like you just, it's the old Altman joke of how are you gonna make money with AGI? Well, we'll make the AGI first and then ask it. There's sort of a similar thing with adopting. It's like, how are you gonna adopt Fable? Well, just Turn it loose in your environment and ask it what it should do and probably actually get decent results without even further direction than just kind of letting it explore and try to be helpful. A question I would have been interested to ask, because I think I know the answer, but I'm interested in your answer. You're very sort of citizen of the world background. If you are some neutral country, or you're sort of the vast bulk of humanity that's not like the US or China at this point, who are you rooting for? I'm not so sure anymore. I think my family, we just got a new au pair from Uganda, and she said the other day something that did stick with me. And I was like, man, we're potentially liquidating this goodwill with an unbelievably rapid pace and reckless attitude. But she said, and this is just in the last week, right? 2026. Given everything that's happened, she still said, when Chinese people come to my country, they just want to take the money home. They just want to extract, make what profits they can, and take that money home. when the US comes to my country, they do things like build schools, and we really feel like they are actually trying to help us. And I was struck by that because I was like, well, that's the story I grew up on, and I'm very glad to hear that you perceive it to be true. I don't know how much longer that will be the case. I mean, so apparently there's still a lot of that reservoir of goodwill out there, but if you're like a little closer AI watcher, Obviously, this moment of export controls has got to shock some people. I mean, we didn't even get into the whole Europe 2031 discourse that was notable but kind of overshadowed over the last few days. But there's got to be some kind of wake-up call to them that, like, geez, we could get cut off at any time from the US. That's not a very comfortable position to be in. The Chinese obviously could stop open sourcing at any given time, but they can't take back what they've given us so far, and they've at least made some kind of credible irreversible commitments on that level. If you're Brazil, if you're India, if you're, I don't know, if you're Nigeria, who do you want to win? How do you think about how this changes who you want to align yourself with?

So I would say, um, number one, you can kind of track exactly how things are going to flow globally by just looking at what happened with COVID Because during COVID you had the US spend a lot of money, had the first vaccines, and then after, over a period of time, they distributed to everyone else, uh, to And it was in order of how close you were in terms of how close you were to the US in terms of allyship. You had the Chinese who, even though there was a US vaccine available, they insisted on using their own vaccine and they delayed. And so they delayed. And as a consequence of that delay, they probably had more fatalities than they needed to. And so you can kind of see that same cycle going on here. They are delaying, they are not taking Nvidia chips, and they're delaying in order that they build out their own supply chain internally. You could see Russia came up with its own. No one was very sure whether it was effective or it worked, and they just distributed to other people. India was a laggard, way, way far behind in terms of development, but they supplied various items which were useful, including the vials and other things. And they are supplying a lot of the robotics data right now, which is being filmed by Indian outsourcing companies. So you can almost kind of see the same structure. It's the same structure of technological diffusion of some important product and how that diffusion happens when there's a limited quantity of it and how the US and the major powers decide to distribute it. It's almost uncanny in the sense that you can see the parallels. For the middle powers right now, I think largely they are locked out because they are not able to spend the money that the US is spending. They're not going to be able to buy— they're not going to spend $100 billion, $200 billion on Nvidia chips. It's just not going to happen. And they also do not have the scale of technological stack capability that the Chinese have in order to try and reinvent the entire stack. They don't have that, so they're kind of stuck in the middle. And as a result, they will make various alliances with the with the powers. And I think that is basically what's going to happen. And it's not it's not a bad thing. It's happened before during COVID It'll happen again. It's not it's not a big deal.

But if you have to pick, are you going to try to go one way or the other? I mean, obviously the The ideal strategy is probably somehow to try to play both sides off each other. But obviously, if the competition really gets intense, I mean, maybe it doesn't even matter. I guess there's one line of thought that the top powers won't— there's so far, we've talked a lot about how we want to export the American AI stack and whatever, and project, use models to project our values and all that stuff. I think there's one very legitimate possibility that when things really turn up to the point where the whole future is at stake on particular decisions around whether we can go forward with certain lines of research or deployments or what have you, that maybe all that just kind of fades to the background and nobody really cares. But there's maybe another future where We don't hit like such an insane RSI curve and we're kind of in a somewhat, I don't want to say normal technology regime, but a little bit more normal technology regime where these considerations do continue to matter. And there's kind of this new cold war of we want these countries in our camp, not the other camp. Do you, I mean, again, if you're one of these third countries and you and you have to choose, or you have to like have a rooting interest, do you feel like— my sense is that broadly the attitude has been more pro-America over time, um, but I just wonder if you feel like this changes that at all.

It's complex because for middle powers, um, Russia and China do a lot of bribery. Gerhard Schröder, former German chancellor, he was one of the board members of the gas pipeline that connected Russian gas to Germany. And he also lobbied for that Russian gas. So there's a fair amount of corruption in the middle powers where their leadership often has financial interest in China, and they have financial. They have stock. They have stock holdings in the U.S., but they're getting paid by China. You know, sometimes it's through business deals. Sometimes it's explicit. But whatever it is, there's a lot of economic activity going on, and so their leaders' personal predilections on what they want because it's of economic interest to them personally can differ from the interests of the nation. And to what extent those things get sorted out depends on the country. It's not as unified as China or the US is in terms of— you don't think the US is unified, but it's way more unified than let's say Germany with former chancellors on the board of the gas company of the enemy. It's way, way more unified than that. So I think there exists this separation between the leadership and the people themselves. And to the extent that the US provides consumer AI like Meta or ChatGPT for free, that really disrupts the power that these leaders have within these countries. I have for some time thought that there's going to be a lot of political turmoil outside the US. I liken it to handing everyone the First and Second Amendments at once. You don't have an adjustment period. All of a sudden you have all of this power. You can see all of the cash flows that your leaders are taking in. ChatGPT tells you like why something is going on, and that is different from whatever your national media and your social media, which is controlled by a bunch of like posters hired by the leading party, is telling you. I think it's going to be very detrimental to certain regimes which depend on control of the people, and many democratic regimes outside the US depend on some form of control of the people through media and social media, etc., etc. I think it's going to be very detrimental. So I think there's going to be a period of turmoil and there will be massive political change similar to Arab Spring. Arab Spring was undirected. This will be more directed because the AIs will help people coordinate between themselves and there will be social, social change, political change. It's not going to be pleasant in a lot of authoritarian states, I believe. So that's just my, that's just my take on it. But I, I think it's once you had— once you hand the capabilities to the people at large and they're able to self-organize, they're able to write their own software. Um, even India has, you know, India has a hostage policy. I don't know if you know this, but if you're a social media company, you have to provide a local Indian hostage, and they're your person that goes to prison if in case the government tells you to take something down and you don't take it down immediately. Right. So they have a— so Twitter and all these firms provide a local hostage. I think once people are able to code their own software and do a bunch of these things on their own, that stuff starts to fall away and then it becomes a challenge for the government to Yeah, that's very interesting analysis.

I take the sort of more abstracted view, or I was trying to take the more abstracted view of like, what should leadership want if they were acting on behalf of their people in a good faith way? And it's a useful corrective or reminder that that is not necessarily a safe premise in a lot of different countries around the world. So a sort of Arab Spring 2.0 AI edition, um, is not unrealistic. I definitely, you know, could see, yeah, something like that. Um, I could see something like that happening. Maybe to close us out for today, I don't know if there's any other topics you wanted to get to, but Maybe we should go on record with some predictions as to how do we think this gets— of course, in the big picture, there's going to be ongoing negotiations forever, so that I don't expect that there's going to be a grand bargain or anything that's going to be declared. Or even if it were, would it be durable? Probably not. Everything will be kind of subject to renegotiation forever. But when do we get Fable back? And like, what does that process look like? Do you, um, would you venture an even moderate confidence guess?

I think the face-saving measure would be that you get another model which is not Fable, but which is a checkpoint, which is maybe one checkpoint after Fable, Fable, you know, 1.1, And they call it something else. And they have a defined kind of release policy that the government is willing to follow. And that basically allows what Fable did, but in an official capacity. And then they memory hole this whole incident. And so that gives the government the face-saving measure that they need, the policymakers. And that also gives the public and the companies, you know, what they need. So, and then everyone, everyone's happy, right? So, so I think, I think that is, that is, that would be an easy, easy fix. Like, we're going to introduce a different model. Fable has been export controlled, and we will, we may fight the government in courts with that, but, you know, we're not, we're not going to make a big deal out of it. So yeah, that, that would be a, that would be an easy, Easy fix, I think.

In the meantime, we get Fable 5.0.1.

Yeah.

And how soon do you think that happens?

That I don't know. I would say that they can't hold it back more than 2 or 3 weeks, I think. I don't think, you know, there's a lot of stuff which is now centering on this 4th of July celebration. if you haven't noticed, like the all of government, it's Trump has gotten into this. All of government is going to celebrate. And that's why the Iran deal happened too. He doesn't really care what happens after the 60 days. He's like, I want it done. I want a clean 4th of July. I want the markets at an all-time high. I want like SpaceX launching rockets on the day. I want like everything to be perfect. So I would not be surprised if they, they, they resolved the issue within a couple of weeks in order that, you know, they can have the 4th of July celebration kind of free of all of these things. So what about you? What goes—

That sounds a lot like Russia and Putin's focus on the big World War II commemorative parade. And that's the discomfort that I have with some of these comparisons and how apt they're becoming is—

When would you hazard—

Only growing over time.

When would you hazard a guess that this situation gets resolved and you get a Fable-like model release?

I was going to even take sooner than that. My gut says end of the week. But why do I think that? Probably mostly the private sector chirping. I do think the, you know, we talked a little while back. I forget exactly how many episodes ago now, about kind of the AI megacorp and all the balance sheets being tied together and all that sort of stuff. And I'm not so sure that I agree with Zvi on the idea that they could just keep putting out Opus models indefinitely and have all the math work. I think there's at least a decent chance in my mind that to sustain this revenue ramp to the level that it's going to, that the overall kind of financial, you know, house of cards is too strong. But like, you know, this sort of shaky structure, I think we do have a pretty, in my view, like a pretty shaky financial structure for all of this.

Yes.

Where if one significant bad thing happens, I think it could like create a really, you know, a lot of knock-on effects, a lot of, you know, contagion. And I think we're kind of relying on like the continued financing to be there. So even just sort of a loss of confidence at this point could really throw, I think, a pretty big wrench into the overall buildout. And with that, you know, the stock market and everything else. So And I think for multiple reasons, I'm not quite sure Opus is there to where it's going to get Anthropic to $100 billion plus by the end of the year. Maybe it is. It's grown really far, really fast so far. But they're going to have competition from obviously much cheaper open source models. We got, it was Kimmy 27, right, in the midst of this whole thing as well. And I haven't even had a chance to look at that at all to see how good do I really think KIMI-27 is? But at some point there's going to be significantly cheaper competitors that will actually compete with an Opus-class model effectively. And I just think that pressure, both for the functional reasons that you were saying earlier, like, we need this to secure ourselves because everybody's coming up on our heels, and because, look, we got Hundreds of billions to a couple trillion at various stages of this pipeline. And if it's not all kind of on track, the whole thing could really start to get wobbly pretty quick. I think those arguments will probably start to get louder and louder, and I would expect that they will carry the day. So I don't know if I had to put like a 50 over-under on it, I'd maybe put the over-under at like Friday of this week, which still means there's like half chance that it goes into next week. But I'm with you that I'd be pretty surprised if it goes more than a couple of weeks. It just seems like it's a really hard thing to hold back. And I even think the other AI companies are not probably going to be too— as much as Sam and Dario can't stand each other, it still seems like they also are going to be like, oh my God, This is— they're even more levered, right? So they're even more arguably financially vulnerable to disruptions that wouldn't seem that huge when everything really has to align in order for all the financing to work. I think that the other companies are probably going to be like, guys, we got to clean this up. We're going to be right there in the not too distant future too. And we want to We want to launch this without, like, similar drama. So can we please figure something out? I find it hard to imagine that that wouldn't happen. But as you know, maybe a 20% chance that, you know, I didn't think we were going to go to war with Iran.

Right.

I mean, it's like, what are we really going to do this after all these promises? And, you know, for what exactly are we trying to gain? And, you know, within— and it doesn't seem like we're prepared or it doesn't seem like we have a great plan for this. And yet They went ahead with it. So it does seem like you've got to have at least some fat tail room in your possibility set. And I do think it's unlikely, but could I say it's less than 10% likely that they do really double down and just make a terrible mess of this whole situation for months on end? Well, there's precedent for that, so I don't think I can say it's like less than 10%, maybe in the 10 to 20% range, I guess I would put that, which is, again, uncomfortably high. So right up there with my sort of general P-doom number. It's like, I don't think it's the most likely scenario, but damn, I would really like to have a better argument that it's like a really remote chance. And I don't have that argument right now.

How far do you think OpenAI is right now?

Great question. On math and programming, I would say they're probably right there. Obviously, the, you know, the famous conjecture result recently was a notable one. My guess is it's probably benchmark-wise and sort of verifiable task-wise, not much different. Or maybe even OpenAI internally could be a little bit ahead of what we've seen with Mythos Fable. I wouldn't be shocked if we see an even higher score on Frontier Math 4, or even more of the sort of open questions answered by OpenAI models. I do suspect that Anthropic still has a significant lead in the general knowledge worker shape of the model, though. It's been interesting lately trying to use 5.5 in OpenCLAW, and I still need to get my Hermes upgrade potentially going. And even just using Codex, You know, it's very smart, it's very good at coding, but it doesn't— it's definitely not as polished and it's definitely not as good at like kind of just getting what I wanted to do and doing the right thing. So my guess is that is probably the biggest gap that they have and probably remains kind of substantial, albeit, you know, like hard to measure. with a bunch of standardized tests, but maybe also still kind of easy to recognize when you just like throw it at, you know, some very idiosyncratic workflow where you're just like, yeah, I can just tell if it's doing well or not. My guess is that Anthropic still has a meaningful lead there.

Interesting. I wonder if Noam Brown, I think like about a week ago, He puts out this tweet that he says basically that, look, the performance on the benchmarks can be increased if you just increase the compute, the test time compute. So what you see on the performance of the benchmarks is that GPT-5.5 could go, could reach higher levels of capability. But They're not applying the compute there. They're capping the compute at a certain level. It strikes me that that may be true, that instead of showing advanced capability, they're trying to cater to their market and trying to push out the model to as many people as possible at the price point that they can make it affordable for people, and that the capability might have already been seen internally, but it doesn't make sense to deploy because it just takes too much compute and it's not economic. So perhaps. I wonder if that's true. I don't know.

I mean, in general, taking the tweets of senior OpenAI people at face value and reading them as true but with details redacted has served me very well over time. I'd say the vague posts are from, from like, you know, and I'm not talking about, um, you know, Jimmy Apples or whatever, but like, you know, from people that are known, um, insiders. I, I've found from Sam to, you know, Gnome to kind of that whole, that tier of people, I would say that they're generally telling the truth and just holding back on the details. So when he says it's not clear that these models plateau and you can run them indefinitely and get more value, I think he's telling the truth about that. I wouldn't be surprised at all to see that validated in public potentially before too long. I do still think there's a little bit of a difference. I mean, he also did the Diplomacy game, so that was an interesting And he's done poker. So he's done some of these interesting things where game theory meets kind of theory of mind. So I don't want to put him in the bucket of being like an all-math guy. But I still do kind of feel intuitively like that probably doesn't extend that whole paradigm of like, you can just keep running them forever, and you keep getting better results in an auto research, you know, in a nano GPT speedrun. In a, you know, find the best way to pack these marbles into, you know, some container, like these sort of problems where you can get a measure of your progress and continue to optimize against that.

Sure.

Does my outline of questions for a podcast continue to get better with indefinite compute, or does my like intro essay that I always kind of use as a a test task get better with infinite compute? There I kind of doubt it. There I feel like it probably stalls out somehow and just kind of starts making changes that are just kind of changes, but I somehow don't feel like it's gonna add up to something where I'm like, oh my God, like you worked on this forever and you got to the perfect, you know, intro essay that I couldn't improve on. I'm, I'm guessing that's still not the case. I would still bet on Anthropic, you know, on Claude models broadly to continue to be just a little better at that for reasons that are a little bit hard to put a finger on. But, you know, that's certainly been the trend to date.

Indeed. And on that note, Nathan, a pleasure as always.

To be continued. We've got a busy week coming up. We'll be here Tuesday, Wednesday, Thursday, same timeframe. And Then Friday we'll take off and I think I might take my kids to a World Cup watch party just to get outside and be in a physical space with some other people and, and do something, take my mind off of AI for at least a couple hours. But in the time between now and then, a lot to cover. So we'll look forward to seeing you back here tomorrow and trying to make sense of everything that's coming at us.

Indeed. Bye-bye.

Bye for now.