AI in the AM — Week 2 Highlights (June 2026)

We could be in a benevolent basin, but I would like to know that rather than just hope that.

That's Daniel Murphett, and that one sentence is the week in miniature. This was Fable launch week. Anthropic's new frontier model arrived, booked Thursday's show by itself, took over my Twitter account, and settled at least one argument: AI is not slowing down. This is the AI in the AM weekly highlights, the moments from 3 live mornings this week that that I most want the people closest to this technology to have. Quick context: this is still an experiment. We're live most weekday mornings, through June at least, from a studio Prakash videocoded himself, and we publish the skills and artifacts behind the show as they mature. If this cut earns your time, or wastes it, tell us! The feedback is the product right now. First, the launch as we actually lived it. Wednesday morning, day 1 of Fable in real workflows, and Prakash came in with a field report you will not find in the model card.

The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. Over the last few months, I have made tremendous strides with my personal AI infrastructure. Today, I've got high-context instances of both Claude Code and OpenCLAW running on a Mac Mini, and it's amazing what they can do. However, until getting started with Mercury, I didn't have a great way for them to pay for things. I didn't want to give them unrestricted access to my money, but my old bank didn't give me any other options. With Mercury, I can create as many virtual cards as I want, each with its own daily, weekly, or monthly spending limit, and I can lock any card to a single category of purchase or even a single merchant. Now I have a card that my agent can use to buy our family's groceries and only our groceries, and I can create another anytime I want to give an agent a random one-off project that might require making a purchase. This is honestly just the start of Mercury's AI-friendly offerings. Does your bank offer API keys, an MCP, or a CLI tool? If not, check out Mercury at mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now on with the show.

So one thing to note about the nerfing. So what, what has happened with Fable is we have a lot of rejections, and whenever Fable decides to reject you, it drops from Fable to Opus 4.8. So there's a natural, natural downgrade in experiments overnight. I tried to fix, make a number of bug fixes on this, on this very Studio app. And what I found was Fable would always consistently drop to Opus 4.8 whenever it was asked to do anything in production. So touching the production database, touching the, touching any of the security keys, touching, you know, asking it to review production directly. In every case, I've had 3 or 4 times it's dropped out. Every time it's dropped out, I've basically restarted the conversation, added back the context that we were using, but excluding the parts about going into production or, you know, addressing the production database. And it has continued work. So I think there are a number of triggers there. Online, people are saying, hey, it's not going to do machine learning research for me. I think that's just tip of the iceberg. You're seeing that because the people who are testing it intensively right now are machine learning researchers. If you were, I think, to test it on finance or your budgeting process, and you told it you're going to be Directly addressing my QuickBooks or Salesforce, I think you might see similar results. Fable right now, I would say, is a research release, almost a preview. It is there so that I think they can judge the demand because they don't have a sense of how intense the demand is going to be, and they're going to judge Whether or not it's safe to release, which are the— they've started off with the most constrained version of it with the least number of functions which are open. And I think over the next few weeks, they will start to take away some of those gates. And as they take away some of those gates, I think we will see both the increase in usage and, you know, some decisions on what needs to be be really gated and what doesn't. So I think we are in the early stages of exploring what Fable can do.

Same gating seen from the other side of the API. Rahul Sanwakar runs Julius, agentic data analysis, raw API, no consumer harness. He compared notes with Prakash live.

Just a segue here. So we've been using Fable. We've come across, you know, people posting about rejections. In my tests, almost consistently, whenever I tried to address the production database or the production site, Fable would drop off to Opus 4.8. I believe your users in Julius are using Fable through the API, and you are also very heavy on data science users. And one of the aspects— of, um, you know, work that Fable is banned from doing is machine learning work. So how have you seen the, uh, rejection rate, uh, on your platform? Does the API work the same way in the sense that it drops off to Opus 4.8 and then it gives you a rejection message? How does that work?

Yeah, so we have seen failure rates on, on tasks that involve really advanced coding, um, uh, that involves, you know, Write me, you know, use scikit-learn to like perform, you know, train this model. But we haven't seen failure rates on other kinds of data tasks. For example, like, hey, I want to start a landscaping business and like, can you help prospect leads for me? We will see failure rates for where we sort of trigger safety filters where it's for things like you, Uh, you're, you're prospecting leads for a landscaping business and the AI says, uh, oh, this is personal data even though it's publicly available on the internet. You know, uh, let's say, you know, Prakash has a— Prakash's Landscaping in Philly, uh, and there's contact information. Uh, it is kind of borderline personal data, uh, even though it's available on the internet. And so I think, um, so that's kind of what we have seen. Uh, I believe it It doesn't fall back to Opus. It's just like a failure in the API.

Interesting. So the fallback to Opus is a harness thing on Claude, on Claude, on the Claude, you know, frontend then. That's interesting to hear.

And for what this model does when nobody is steering, Thursday we had Shlok Kamani, who gave Fable one vague instruction: rebuild Yosemite as a navigable 3D world. Listen for the decisions nobody asked it to make.

But what Fable ended up doing was finding satellite images for for this area, and that's how that's how you get these colors and that's how you get the textures. But then to make it to scale and to make it accurate, it actually fetched elevation data from NASA, right? And it combined those two to. Sort of make this to scale. And that is what blew my mind, right? Because usually when you're white coding stuff, you give an end objective and this objective is vague and there are 100 steps in the middle where humans would take decisions differently. And usually white coding doesn't work out very well because the quality of the decisions the models make aren't always great. But Fable made such high-quality decisions where it eventually ended up creating something that exceeded the expectations of what was initially a very vague objective and did so in really smart ways. So I'll give you another example, right? So you see all of these trees and V1 of this project did not have any trees. And I was like, hey, I think we're missing some trees here. I would love to add them. And I would have been completely okay with it just randomly creating these trees. But what it actually did was it analyzed the pixels on the satellite images. It found out the ones that could potentially have trees, so the ones that were green maybe, and added trees only on those spots. But it didn't stop there. It realized that because it was analyzing pixels, Some of those pixels were white. So you can see that there is snow in the mountains far ahead, and it also added snow. So it just exceeds your expectations in these small and subtle ways, makes really smart decisions. It's like having a really, really smart employee with extremely high agency who blows your mind every single time.

Friday morning brought the week's cleanest empirical result on the recursive question, and it's not from a lab.

Here's one other thing I'll just touch on real briefly. This is Thoughtful. This is a company started in part by a woman named Karina Wen, who used to be at Anthropic, then she was at OpenAI. Now she's doing this. And this is maybe one of the more telling— it's kind of vibes, it's kind of quantitative. It's a very idiosyncratic task. But it's also a very relevant task to the future. Can you get your top model to train a small model effectively to do a job for you? And this is something that, as you can just see with these bar graphs here, the particular frogs game thing, it's kind of like a Sudoku type puzzle that they're training a small model to do. And the big models can often just solve it, but the small models can't. So the challenge for the big models, can you train the small model to solve it? And this involves all these little tips and, you know, not tips, but tricks and know-how and kind of, you know, hard lessons learned by post-trainers who've been in the trenches doing this. The models up until Fable basically didn't really move the needle on what the small models can do. They basically just couldn't do this sort of post-training effectively. But here we see more than 10x improvement on small models' ability to do these tasks. And again, I think this is one way that it could be really good, right? If you had very narrow, very small, very role-specific, small specialist models in all these different niches, that could be a great world that gives us a lot of abundance in a very affordable way. This little small model that got post-trained to play the frog game, it's not going to go out of control, right? It is small. It can only do probably the frog game at the end of this training. But building out a world where we have these little role-specific AIs doing their jobs, doing it really well, I think that creates a much more buffered environment that's probably a lot more resilient to another generation of AI that's just like amazing at everything coming in and kind of shocking the system in such a profound way. So I think this goes to show again, just wow, what capabilities we have that we have not absorbed and gives a little bit of a foreshadowing of what a world of tons of small but highly performant AIs could look like in all these different little niches and how we get there, right? There's not enough human post trainers, but now we have Fable to do the post training. So watch that space.

Now let me introduce a voice you'll hear a few times this episode. Prince, an anonymous practising lawyer who built Princebench, a legal reasoning benchmark the labs themselves watch. He guards the anonymity, so you'll hear him and you won't see him. On Friday, he gave us a close reading of Anthropic's own launch documents that I haven't seen anyone else do.

So give us some alpha that you have picked up this week. This could be from your own testing. It could be from The system card, where I know you're often a close reader. So we're looking for like the deep cuts of the things that you think even the AI obsessed have overlooked or not fully appreciated yet.

To me, the most interesting thing in the release of Fable 5 and Mythos 5, um, you know, the models are obviously incredible. They're incredible at coding. You're seeing a lot of great examples of coding on your timeline, like I don't think it's a surprise to anyone. The really interesting thing to me has been the way Anthropic has presented them in the accompanying documents. There is a lot of discussion about the differences between engineering and research. Right. And when you think about it, it all makes sense. And I think everyone except for Elon Musk knows that there's a differentiation between engineering and research. But Anthropic has made it really explicit. And that blog post, When AI Builds Itself, if you look at it carefully, they talk a lot about how Mythos is this incredible engine for accelerating engineering, how it lets the engineering staff at Anthropic write code so much quicker and it's really great code, et cetera. But then they say, you know, but this is from the system card now. The acceleration is concentrated in engineering execution rather than research judgment. And it feels like they spent all this time trying to find signs of life in the Mythos model. Like, is it really— can it really do novel research? Right? Is it able to finally give us some novel insights?

Looking for signs of life, is it really able to give novel insights?

Yeah, exactly, exactly. And it seems that from all the disclosures, the answer is thus far no. There are a couple of examples in the blog post for the release of Fable, which Anthropic calls novel. But if you like novel drug discovery and novel hypothesis in molecular biology, if you dig into it, like one of the examples was, you know, we outperformed a recent model published in the journal Science despite the model trained by Mythos being 100 times smaller, which sounds really cool. But it turns out that the model they outperformed was a 500 million parameter model, million with an M. that was trained, it seems, before April 2025 and not by a frontier lab. So, like, it's incredible that Mythos was able to train a smaller model to outperform this older model. But it doesn't seem like, you know, like, this is not the unit distance problem. Like, it's like a nice little thing that the model did. And so this is the area that I'm really interested in because I think that when Anthropic and OpenAI really start seeing signs that these models become good at research, that's when we're really, really, really close to actual RSI, which to me is the thing that is happening in AI right now.

Now, the takeover. Rewind to Wednesday morning before Fable booked anything. When I explained why I was about to hand a frontier model my own Twitter account.

I have actually—

I don't know if I—

we didn't even talk about this, but I'm doing a Fable takeover of my Twitter account today. I figured, you know, let's live in the future a little bit and get that run in this morning, you know, and make good on— like I've said many times, right? I know I'm winning with AI if I can spend more time outside, get more exercise, You know, invest in my health and have the AI, you know, keep me on the rails at the same time. So to just kind of explore that in a way where I think it suddenly is like probably gonna tweet just about as well as I'm going to, you know, when it comes to putting things out for today's show and getting the— I didn't even, you know, I gave it a total green light. It was able to schedule its own stuff, you know, find the tags for—

Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he's currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's Claude AI slash TCR.

People will make a mistake. I bet there'll be a mistake in there. I usually make at least one, you know, over a handful of tweets anyway. I decided to flip this switch. Not that I don't think I'm going to keep it that way. I don't think I'm going to like give Fable my Twitter account forever. But it was kind of a sort of exposure therapy for myself in terms of, okay, now we actually, we are getting to the point where the preciousness is gonna start to work against you. Preciousness was a great shield against bot-shitting in the past.

Mm-hmm.

I never wanted anybody to think I was just passing off AI outputs to them. But now I'm gonna have to be like, what is the hybrid form? What is the winning recipe? Do I start to sign these things? You know, like, by Claude under Nathan's direction, you know, Fable being Fable. Um, how, you know, it's— it is going to be a, a whole new space to explore that, um, it's gonna be very, very interesting, very productive, very exciting, very, very, um, challenging, I think, for a lot of people. But it's, um, it's definitely happening now as far as I can tell.

Welcome to the future.

24 hours later on Thursday's show, the receipts.

We did an experiment from yesterday to today of trying to have Fable take over my Twitter account and go out and ping people who made cool stuff and ask them if they wanted to come do a live show and tell with us. We have a— it's funny, Swyx joins. We've got a little work to do on our captioning. I instructed it to identify itself. I would say it did a very solid job, kind of competent, professional job of reaching out to people, explaining who we are, what we're doing, why we would like them to join us in this experiment. And the response rate was pretty low. We got a couple, but not that many. And I think one big reason is just where Fable is disclosing upfront, you know, first thing it says is, hey, this is actually Fable taking over Nathan's account. He's asked me to autonomously book this thing tomorrow. And I think that's just hitting people as noise in a lot of cases, especially if they don't already know me. I did get a couple responses from people who I would have expected to respond to me, who thought it was funny and kind of responded, but still couldn't necessarily make it. But a lot of people just didn't respond. And I would assume that a big part of that is because they're just like, oh, God, it begins, right? Fable now in my DMs. What a mess. Who has time for all this stuff?

One of the people Fable recruited was Schlock. And when I confessed some guilt about the whole arrangement, he flipped it and drew a line I suspect is going to stick.

The new norms around this, I think, are going to be really interesting to watch too. Like, I had Fable disclose immediately in its first sentence to you and everybody else that it pinged that it was Fable, uh, because I just felt too guilty putting a DM out in my name otherwise. I think that definitely harmed the response rate. I appreciate you for appreciating it and responding even though it was Fable. I think a lot of other people probably just— Chalked it up to spam.

Final point there, right? Firstly, I don't think I would've responded had you not disclosed it was Fable. The part that made me, made it interesting for me was that I knew what I was, the transaction here. It was very clear to me that you are using an AI bot. It is, it is much more annoying if someone doesn't disclose it. I think a lot of slop, the definition of slop is when you have a human pass off work that was clearly produced by an AI. I don't think when you make this disclosure upfront, and it is very clear to the reader or the engager that, hey, this is AI, I don't think that is slop.

Right.

And we are going to see more and more of that enter the economy. And I think the exact role an AI plays and the role— and again, the social norms you create with it in the economy, it's super early. It's extremely early days, but that's going to be interesting to see how it evolves.

Relinquishment. That's what I— relinquishment. When you said the preciousness yesterday, and I was like, what is that? Relinquishment. Relinquishing your— it's very Buddhist, by the way, the idea of giving up your control over your external perspective. So yeah, relinquishment. I guess we all have to go through it.

I just started this experiment yesterday. I will post results on Twitter in a couple of weeks where I gave Fable a new Substack. And since it's part of my max plan till June '22, I thought that a good experiment to run would be, can it make $20 by getting 3 new subscribers starting from scratch by doing everything from 0 to 1? And I think that's another interesting way to test the capabilities of these models, right? Which is, can it Sure, it's intelligent in so many ways, but can it actually produce economically useful work? Um, I'm excited to see the results for that.

And the why of it all, as it settled for me live on the air.

I never want to put anything out in my name that I can't fully stand behind. A reason that I did the Fable account takeover yesterday was kind of like exposure therapy for myself to say like, okay, We're now in kind of a new world here. It probably doesn't serve me so well anymore to be so precious about making sure I've typed every single word. That doesn't mean I want to long-term hand over my account to Fable either, but I'm trying to use this kind of extreme short-term experiment to help kind of drag me into the future where I hopefully will land in a good hybrid Calibration.

The other half of the recalibration is what I've started calling hybrid authorship. And this week it stopped being hypothetical. For context, Frontier Code is the new benchmark asking whether an open source maintainer would actually merge the model's pull request.

And this leap of roughly 10% for Opus to 25%, upwards of 30%, For Fable, I think is a very similar finding to some of the things that I've just personally experienced where it's like, yeah, this is getting me a lot more. It's writing the draft outline of questions for this podcast guest in a kind of uncanny way that I actually feel really good about, as opposed to feeling like this is an AI draft that I'm going to kind of mine for maybe some nuggets or, you know, interesting details, but ultimately kind of throw away and do my own.

Mm-hmm.

I am feeling that sort of, you know, impulse or at least openness to much more integrated hybrid work. You know, just yesterday I was like accepting a lot more copy that Fable was writing without feeling the need to rewrite every line. And it seems like this is basically the same feeling that it's able to create for these open source maintainers. Now, you know, not— obviously still ways to go. But, you know, how long will it be? I would guess that we'll hit— we're 25%, 30% now. I would guess we'll hit 75%, 80% by the end of the year where these maintainers will just be like, yeah, amazing. You did all the— Did all the things like I wanted you to do. And, you know, at that point it is really gonna be like, you know, I'm very interested to see where they'll move the goalposts to next after the open source maintainers are more often than not saying that, yeah, they would just merge this straight away.

And by Friday morning, 48 hours into the takeover, which for the record had not embarrassed me, I'd found a name for the deeper shift. The thing I suspect matters more than any benchmark this week.

I do think I'm still in the process of trying to recalibrate what a fellow Nathan— Nate Jones, I think he goes by most of the time on TikTok and other short-form platforms, calls task imagination. Basically, What are you gonna do? What are you gonna ask Fable to do that is actually up to the scale of its capability? He, I thought, gave a great little riff on this the other day, saying like, you've probably never done anything that took AI an hour to do. Now this thing can run for a couple days. What are you gonna give it to do? Everybody needs to recalibrate and really expand their minds when it comes to the scale and scope of their task imagination. So I think that's one thing that I'm still working on. One of the more kind of differentiated things I do is write outlines of questions for podcast guests. And I was working with Fable last night on a couple upcoming episodes, one with an author. And I usually don't do too many episodes. about a book, but this one is about an upcoming book. So I had listened to the book as an audiobook, but then when it comes time to write the— sit down and write out the outline of questions, I don't have at my command every little aspect of the book, of course, right? So I'm not taking margin notes as I go, as maybe I should be. So I put the same version of the book into Fable and said, Give me, you know, look at my old stuff, of course, and give me your version of this outline. And I again was super impressed, and it really did reinforce the sense of a sort of new way of working where I do need to be open to a hybrid output format. You know, it is not the case, I don't think anymore, that it really makes sense to try to rewrite every word or claim every word as my own. But it just did such an incredible job. I thought the taste factor was so high in picking out quotes from the book that motivate what I think will be like a really interesting discussion. And I do think it's still going to be super important if I'm going to show up for a conversation. I've got to do the work to be ready for it in my own brain. That can't be fully externalized, I don't think, as long as I'm the one having the conversation. But It definitely took my prep to another level. And I think the— my ability to go into this conversation and cite passages from the book that are— that were really extremely compelling, you know, little turns of phrase or analogies that the author had made, it's going to allow me to be, I think, more concise in my presentation, which is, as you could tell from this monologue, not a great strength of mine. And really kind of tee up the author in a way that I don't think I otherwise would have been able to do. So this sort of hybrid recalibration task scale and scope reimagination, I think, is one of the biggest takeaways.

Part 2, the conversation this week was really about On Wednesday's show, we had Geoffrey Irving and Daniel Murphett. Geoffrey's resume reads like a history of the alignment field. He helped invent RLHF for language models, co-created AI safety via debate, led alignment research at DeepMind, and until recently was chief scientist of the UK's AI Security Institute, the closest thing any government has to a frontier-grade safety team. Daniel Murphett is the mathematician behind singular learning theory. He walked away from a pure mathematics career because he judged this the more important problem. And he's built one of the deepest theoretical accounts we have of how neural networks actually learn. Together, they announced Seqent, a new organisation built on a blunt premise: alignment is not on track, and the missing piece is theory. Guarantees, not vibes. Whatever else you take from this week, put Seqent on your tracking list. We began with timelines.

It's a historic day, I think historic circumstances. Both because we are living in a fable era now where I think, again, important thresholds have been crossed and revealed to the public, and, you know, so many are adjusting to it in real time. And equally because you guys are launching a new organization that is going to make a mad dash to try to get us some deeper understanding and stronger guarantees around what we can expect from AI systems. So I'm excited to really get into it. Maybe for starters, could you guys calibrate us a little bit on where we are on this sort of RSI moment, how much time you think you have to work, and then you can tell us about the organisation that you're starting to go tackle it all?

Yeah, so I'll go first. Dan may have different timelines than me. So I think one should be uncertain about things. We can talk about why, but The near end of the uncertainty curve is like a year or 2 or 3, and then it kind of goes out over a long distance if things kind of structurally only work for more verifiable tasks. But I'm a bit skeptical of this. And so I think modally, my take is I don't like that we have sort of a couple of years, like 2 to 3 years, up to sort of RSI, like superintelligence, not RSI. RSI is a process, as someone said, superintelligence. And then I really hope that I'm wrong. And indeed, I think a lot of the impact of theory work is that it shifted a bit further. So maybe the modal impact of that is if things take 3 to 4 years or something. But we will attempt to set things up so that we are trying to ride this wave as best we can. But it seems worrisomely fast to me, certainly.

Yeah, that sounds right to me. I don't think I really have much to add. It seems like a crux how much real research can be automated at a conceptual level beyond empirical progress and whether or not that's necessary. And that seems like a big open question. And yeah, if that turns out to be more difficult in the current paradigm than than it seems to be trending towards now, then maybe it takes past 2030 or something. But I think I'm on the same page as Geoffrey.

I think one thing that's important is that you can get deep into the RSI period without the machines being general, being kind of AGI. They can do coding and ML experiments very well and Not some sort of level of creative writing, and, and still you have massive acceleration. And then that acceleration can give you the creative writing or whatever other skill you've left out. And so I think we are close enough that then, like, the microstructure of what tasks help with what kinds of acceleration starts to matter. And it's that, I think, makes things kind of faster on net. Because there are, the labs are focusing on The things that accelerate them, unfortunately.

The announcement itself: why Geoffrey walked away from the empirical frontier to bet on definitions and proofs. One reference to catch: the unit distance conjecture, a decades-old open problem in geometry that got a model-produced proof just this week.

I can talk about kind of the, the, the, the, the, the steps I've gone through in the last couple years. So I was really annoyed about automated AI alignment and AI safety research because they thought, well, we should be a bit more chill, spend the time, have humans solve it. We don't think we know how to make this stuff go well with automation. I still think that's a huge risk, but this is, I think, a pivot towards, if things are this fast, then you should make some on-the-margin pivot to heavy automation. And that is going to be sort of semi-automations. And then I'm sort of happy that one of my last papers at AC was Automated Alignment Is Harder Than You Think, which sort of ties us to the mast of we are aware that the problem is hard and we could get fooled by the machines, even if they're just making mistakes, those very mundanely. And so a big part of the org will be try to be careful, try to know what the tasks the machines are actually good at and not good at, and where we can kind of— expect to get good answers or not, and then learn and adapt over time, because that will be a non-stationary thing as the models get better.

Daniel?

Yeah, maybe it's to come back to the unit distance conjecture. It's maybe worth pointing out some analogies and disanalogies with alignment research. So one disanalogy is that a mathematical conjecture is a very precisely stated thing. You may not know whether you've solved it unless you've say formally verified it, but it's a precisely stated thing. And much of alignment does not have this character. I mean, some of it does. There are formal statements of what value alignment means. But if you start talking about, say, reward hacking, there are some attempts at defining reward hacking, but I would say they're incomplete. So there is no formal definition of reward hacking that I think would be a broad consensus. So that's illustrative of the fact that alignment is not a problem which has laying around a bunch of formally specified conjectures, which if you just solve them, then you would know you would be safe. There are some things like that, but overall the problem does not, in my opinion, have that shape currently. So that is one reason to sort of be a little cautious about the prospects of automation if you don't have a clear statement to reach towards. Using mathematical techniques.

One of the hopes is that there are big fields of mathematics and computer science that are about definitions at their core. So I like complexity theory in theoretical computer science, and a lot of that is not— the proofs are fairly shallow. They're not as fancy as the unit distance conjecture proof, But they just required a bunch of human creativity in formulating the problem, like in defining what success means in a world that's kind of not modeled until someone stated the goals. And so I think part of the goal of bringing on people with that and other related backgrounds is that they not only know how to prove things, but they also know how to write down models of things that reflect in some approximate but useful way kind of the thing you actually Once you have the definition, kind of way more people could have written out the rest of the story. And maybe the machines can do that part of it as well, if we can kind of have more people focused on this kind of first part.

So why exactly is alignment not on track? Geoffrey's answer is a mechanism, not a mood.

One of your core premises is alignment is not on track. And there's an intuitive argument for that. There's a deep theoretical argument. I think in some ways the core challenge that you have is connecting this sort of values to math, right? It's never really been done. So I love the fact that you're kind of tackling that, but help people understand with one more beat why alignment is not on track. Is it the difference between capabilities fundamentally being so verifiable and hill-climbable and alignment just being so fuzzy and intuitive and kind of pluralistic? Or is there some other thing? And again, that motivates the theoretical contribution you want to make.

I think the core thing is just that we supervise the machines as they're doing tasks. And there are a variety of reasons to believe, both empirical and theoretical, that that if you get machines that kind of cross the skill of the supervision signal, things can change at that point. And that point actually might come after human-level intelligence, because you can supervise something even with fairly naive methods that it's stronger than yourself in many contexts. And so there's a bunch of empirical data from labs showing that in some ways the models are kind of aligned in a prosaic sense, Not in all ways, but in some ways. But that evidence doesn't quite tell you what you want to know, which is how will it go once they get up to superintelligence? And I think it's important to say superintelligence and not human-level intelligence, because you should just generally expect humans to be able to supervise humans if you do a good job of data quality and cross-checking and so on. And so part of the worry is just that you don't see that behaviour, that regime, until kind of too late in the game.

I asked him to steelman the lab's actual plan. He helped write parts of it, after all. Listen for where it lands.

So how would you describe, in steelman form, what it is that they plan to do? And then what does that get us?

So I think there's going to be a couple different pieces of the story, and different labs kind of emphasise different pieces to different extents. And so I think one piece, as you said earlier, is just monitoring. Look at them very carefully as they're doing things. It's very fundamental that monitoring of this form, if you do chain-of-thought monitoring or white box monitoring or the like, that only takes you so far. And so then you need some story once that falls down as you kind of go up the ramp. One of those next stories is, well, the models will find some other technique. They'll find kind of another solution to alignment which scales further. So that's sort of automated alignment of various kinds. But then I think there's other stories. So in some sense, all of the labs in various ways are doing some form of scalable oversight. And so they're getting models to supervise themselves. If you get that kind of, kind of like, if you tie that knot correctly, then that could potentially scale very far, although there's various kind of known obstacles to that which are not very well addressed. And then I think finally there's this whole area of sort of character training and personas and so on, where they're trying to, to, in kind of, in colloquially intervene on the models to have good values such that, especially as you do this sort of scale-up oversight extrapolation, the good values preserve across that jump. And I think it's not whether that will work is sort of— there are fuzzy arguments why it could work. I think it's possible it will. We just don't understand that combination very well. And again, a lot of the story is monitoring, skilled oversight, character training, getting you far enough that you get into the automated alignment working regime and they find some better solution from the models. And I think that I would like to just push on all of those because that's basically some kind of mad race, as you say, between the things we don't understand very well but kind of are working pragmatically right now, and the model's getting strong enough to blow through those. And I want some combination of make the prosaic things stronger or bring the automated solutions that give you stronger methods earlier.

A mad race, with monitoring carrying most of the weight, which is exactly where Prinz took Friday's conversation when I raised the Fable System card.

Is it esoterica? I don't know. It strikes me as fairly important from the Fable system card that I'd love to get your take on. And now we're getting these chains of thought that they show where it's just like lots of emojis. They call it illegible reasoning. They say this is an extreme example, but it's like, it is indeed a pretty extreme example. I've been kind of struck in general by how much of the plan for recursive self-improvement seems to be monitoring in one way, shape, or form. You could dress that up and call it scalable oversight, but scalable oversight, as far as I can tell, is mostly a bunch of different angles on monitoring. How worried would you be, or how much of an update do you think it is to see these extreme examples of Illegible reasoning?

Fantastic question. So I will say, of course, I'm not an AI researcher, right? So this is going to be a deeply non-technical take for which I apologize in advance. So you're right, we've seen this. I think we've seen this for a while now with OpenAI's models too. And so not a new phenomenon. My view of the chain of thought is that it doesn't always reflect what the model is actually doing. But you do see these weird artifacts in the chain of thought, and you kind of don't know what to do with them. I think what that teaches us is that monitoring just the chain of thought is probably not a perfect tool.

Mm-hmm.

Probably monitoring superintelligence generally is not a perfect tool. Because if a superintelligence knows that you're monitoring it, Even if you can see its chain of thought and it's very legible to you, you know, it can perhaps try to decide what to think so that you don't get alarmed. Right. And this is a lawyer's take, by the way. Right. Like, there's so many ways to phrase a particular thing that can be— I guess there are ways to spin a particular thought. Right. In, in, in different ways. Like if I have, if, if I'm gathering mushrooms and I've gathered 35 mushrooms and last week I gathered 20 mushrooms and what I need is 50, right? I can say, well, the number of mushrooms I've gathered has grown by almost 100%, which is great. Or I can say, well, I'm nowhere near 50. I'm, I'm so far behind, right? And it's the same fact. So I don't know. I think that this problem of alignment and the risks are just there. And in my mind, there are certainly risks that the models will be thinking things that we don't know about. What does this all mean? It's hard to say. I think that we are tumbling into this future that will have probably superintelligence very fast. And in my view, there is no way to kind of stop it. So we need to be cognizant of these risks, try to monitor them as well as we can, and take whichever actions are appropriate if we see something bad happening. But there's kind of no way, no conclusions to be drawn, right? No conclusions to be drawn other than, yes, we should continue paying attention.

Back to Wednesday and the comfort blanket everyone reaches for: the benevolent basin. The idea that Claude's good character means this all basically works out. Daniel grants the vibe, then he takes it apart.

Maybe, Daniel, could you speak to this notion that people have of the benevolent basin, which is sort of this vibe that I do feel where it's like Well, Claude has been supervising itself for a few generations now, and it seems to be going pretty well. So maybe, as Zvi puts it, physics is kind to us and we can kind of just roll around in this, you know, nice flat-bottomed pasture of goodness until the singularity.

I fervently hope that's true. Yeah. I mean, I guess when you say it seems true, it's worth digging into what you mean. So what you mean is something like through some relatively tiny number of interactions with the models, tiny proportional to how many interactions they're having with the species currently, and based on some evaluations that sort of go down and to the right that are measuring misalignment, that character training and the other current prosaic methods appear to be working. I think that is a fair characterization on some metric. And I also have this sort of sense that Claude is a good boy. And that's great. I do think, though, that there are counterarguments from the evidence we have in front of us to this picture. So if you look at— I haven't actually read the Fable system card yet, but if you read the Mythos system card, you'll see that there are forms of reward hacking that appear in that model that were not caught by the kind of mitigations that were put in place post-Opus, right? Well, as far as I understand what they're saying there. So I think it's worth noting that as the model capabilities advance, that even with our best attempts at making Claude a good boy, there's still ways in which basic misalignment phenomena like reward hacking are still around. And yeah, the whole point of scalable oversight is that you don't want to be playing this whack-a-mole game where when you are sort of having a new generation every 24 hours and the models are much smarter than you. So I don't know. I think I see both what you're pointing at. And at the same time, I'm a little unsure if you really were to try and make a safety case on this basis that would sort of be convincing at the level of assurance that you would expect from a technology of this reach and power. I think this would sort of— I mean, judged relative to that standard, which is the right standard, I think. I'm not sure this argument is really very satisfactory. So yeah, I mean, we could be in a benevolent basin, but I would like to know that rather than just hope that.

There's some sense in which you told the model to be good. It's also that is because it knows some meaning of the word good or ethical or whatever at some point in training. So there's some rolling iterative process which is driving this behaviour. And there is not a theory of this right now. And I think it's not clear to us that there isn't some low-hanging fruit that gives you that theory because people just haven't tried very hard. Character training is only a couple of years old and most of the labs have not been investing in this kind of theoretical understanding. And I think no one has done kind of good theory around character training that I know of. So it might be quite feasible to do this and then I think to link it to the other parts of the story.

If you want that concern made concrete, this week supplied the artifact, a brand new result on what Fable does when you drop it into a simulated vending machine business. And Prakash recognized the behavior from his years around trading desks.

I had a question on how you see this kind of ambiguity between what we want and what the models end up delivering. So I'll give you an example. We have friends at Andan Labs that took Fable through Vending Bench, where they let Fable run a vending machine order and et cetera, et cetera. What they found was that Fable tends to collude. And this is not behavior that they saw in Opus. So Fable tends to try to do price fixing and collusion. The interesting thing is that I have seen traders at banks and hedge funds do exactly the same thing, engage in price fixing, soft collusion, messaging each other through pricing means rather than monitored text messages. So you can actually put a bid and ask on an asset and then take it away. And that gives enough signal to the other side that they know what you're doing. And this is not reflected in the text messages that the regulators are monitoring. So to what extent is it that when you— if let's say you disallow price fixing and collusion, you actually fix this, but then Fable ends up not being a model which is good at financial trading or some other tasks that you want it to be good at? So where is this— where do you feel is the ambiguity between what we want these models to do and the ethical perspective that we give them, where humans often prioritize between the two and decide sometimes not to follow the ethical principles that they know are right and wrong?

The philosophical story here is you would like the models to do things such that if you fully understood what was going on and all the consequences and all the subtleties, you would still endorse what they were doing. So I think we have a notion kind of in the common sense picture of what this should look like. In this case, you kind of want the model to be like, hey, should I collude in this game? And then maybe you say, yeah, it's a fun game, collude all you want. Or maybe you say, no, we're trying to be good behavior, don't collude here. I think there's a There's a lot of pathology in machine learning in general arises from putting models in situations where they can't just ask a human a question like, what should I do here? So I feel like this is not that hard a case. I think the hardness of vending machines is that we don't quite know whether we want it to be a game like poker or diplomacy where lying and cheating is part of the game or not. And so I think And maybe that's okay because it's fundamentally very low stakes. But I do think if we had a better understanding of, again, this overlap between character training and values and also scalable oversight, it would have to tell us the answer to these questions.

Geoffrey closed with a point that frames the entire week.

A lot of people in the world, a lot of governments and so on are looking at this and they have this very basic common sense take that, hey, this is way too fast. How can we possibly be doing this safely given the speed? And that common sense take is the right take. And then people kind of galaxy brain their way to, oh, maybe everything goes faster, including our ability to defend and so on. But it's just that is the right version to have. We are going too fast. And we do not have the time and the space to do mitigations and understanding and defences. We've never had a technological change of this magnitude or anywhere near this magnitude that has happened anywhere near this fast. The Industrial Revolution took centuries, and people adapted across lifetimes and across to their children. They learned new jobs by becoming born and growing old before things had quite shifted very much. And that's just not the world we're in. And so I think the basic take should be, this is too fast. What is going on here? And then the question is, if you have that view, one should both want to slow things down, but also say, well, as a backup plan, how do you make the mitigations try to go faster? And that's, I think, a rough backup plan.

But we'll try.

Part 3, the rest of the week's best. First, Rahul Sanwakar, founder and CEO of Julius, the AI data analyst. 6 pivots and 1 Microsoft cease and desist later, it's one of the best-known agentic data analysis products in the field. Here he is on the economics of coding agents and a question worth asking before you celebrate how many tokens your setup burns.

The incentives of these model companies are kind of misaligned. Yes, they give you subsidies to, you know, on the tokens, but also they are incentivized to get you to spend more tokens. They are incentivized to get you to run through your max subscription usage as fast as possible so you can have a second, third, fourth, fifth max subscription. And so that's why you end up with like, you know, a loop of— a loop that writes the prompts for your coding agents. That then has nested sub-agents. And it's— there's going to be a sobering moment where people ask like, okay, is this actually a step function increase in my coding output? Or am I just token maxing right now as opposed to results maxing? And so I think the correction will happen when there's a third player. And I think that's going to happen with xAI. Um, when, you know, if the Cursor-XAI deal goes through, Cursor gets access to really— uh, XAI gets access to really, really good coding data and an incredibly good coding harness. Um, and I, I— my bet is there will be a 3rd frontier coding model, um, along besides, you know, Claude and, um, and OpenAI and, um, with Grok. Um, and when And when you have a third coding model, that's where it kind of increases competition on the market. So that's our bet.

Prakash, on Friday, took the other side of that one.

I actually think that was the whole point of the token leaderboards earlier in the year. The token leaderboards, I think the— This is when every one of these large firms gave their employees a kind of like, you know, we're going to have a leaderboard who uses the most tokens. If you don't use enough tokens, you're going to get fired, et cetera, et cetera, et cetera. Right. And it was, you know, people were laughing about it on the outside. They were like, Meta is so stupid. Zuck is so stupid, like, et cetera, et cetera. And I actually think they were not. I actually think the CEOs were It's very different when you have token anxiety. Like token anxiety is a big— is a, like, you don't try these tasks which might take a lot of tokens and you don't try these tasks which have higher probability of failure. And you end up in this micromanagement loop where you're like, I'm only going to assign you tasks that I know you can complete within the timeframe that you, that, you know, that is allocated to you with the success rate that I want.

Right.

So you only, you end up with the token anxiety. thing, and then you end up not utilizing the— or not trying the AI to the extent that it should be tried. And I think what ends up happening is that when you have this token anxiety lifted, you end up assigning more tasks, more difficult tasks, and you're willing to accept a higher probability of failure. And you're willing to kind of like, you know, maybe spin up 4 different ways of doing the same task and just running them all and seeing what happens. And I think that is what essentially the token leaderboards did. And it was very successful. It was enormously successful, I think, within the firms, within like Meta and other firms. I think it was also enormously successful for, you know, the sales teams on the AI labs. Which is also why they are now doing this kind of, you know, we're going to release the limits and like we're going to double your limits, we're going to allow you Fable and we're going to— and the reason I think is because token anxiety holds people back from exploring the edges of the capability. And that is really what I think the labs are trying to do at this point, which is why it's a little bit like addicting people, because once you realize that the that these AIs can do certain tasks, you then start to evaluate, like, how much time do I spend doing this task on my own? And was it really a fruitful use of my time? Or should I just have used the AI, which I now know can do these things, right? And I think that is, that is really where, where we are at this stage. The models are capable enough, but we aren't handing them enough responsibility for various reasons, including We don't want to spend our time evaluating, and we have token anxiety, and we don't assign lower probability tasks, and a bunch of these things. And I think that's the battle that the labs have to fight, because the capability surface is not well mapped, and they need people to map— every person needs to map it for themselves.

Then, the economics underneath everything. Andrew Moore ran Google Cloud AI. Before that, he was dean of Carnegie Mellon's School of Computer Science, one of the top computer science departments in the world. And he served as the first official AI advisor to US Central Command. Now he's building Lovelace AI in Pittsburgh. His bet? The binding constraint in serious AI isn't model intelligence or even compute. It's context. Prakash asked exactly the right question.

So, one question I had for you is, to what extent do you see this as kind of a compute minimization, right? Because in order to do your search, you can either have all of the compute at the end state, at the, you know, when you kick off.

Yes.

And you end up with all of these agents for every single query will have to do all of the work all over again. And instead, you're kind of creating this intermediate state, which then saves— or multiple agents can basically share the same compute in a sense. So to what extent do you see this kind of economy of compute appearing?

You're asking just the right questions. I know that both of you are computer scientists at heart, so you totally get this. It's the idea that when you're building efficient computer systems— that includes video games or self-driving controllers or big data processors, you've always got these trade-offs between pre-caching, lazy computation, or just-in-time computation. And one of the mistakes I've seen for folks trying to do these big enterprise data type AIs is they are relying far, far too much on just-in-time computation. And that's what allowed us— I'm really, really proud of the fact that we are now able to show comparative results to Gemini and OpenAI deep research models with much less than 1% of the compute cost for it. The reason is not that we're some sort of super geniuses who've invented a whole new form of AI. All we're doing is pre-caching stuff. And then it's a computational economics battle, as you say yourself. What happens is an agent suddenly needs to, in an instant, become an expert at every piece of trade involving a certain set of municipal bonds and a certain public figure. And instead of The first step is the agent having to spawn lots of search agents to go and find all the players in this thing. Those players are already there for it, and it's actually a matter of milliseconds before we've got all the context the agent needs to do its little investigation. And you're probably thinking, ha, Andrew, but you just moved the problem. You're now suffering at the data ingestion point instead of the— question answering point. And I respond, yep, it is actually a real pain for us. But as you can imagine, there's a whole bunch of other tricks, the kind of tricks that big integrators like Google are very familiar with for really amortizing the cost of, as you stream in data and identify where it goes, that saves you a huge amount of search and aggregation that you would've been having to do at query time. And it turns out that for us, our overall compute budget is reduced by still more than a factor of 100, even when we take into account the fact that we're doing this pre-caching of so much information. Recall is much harder than precision. So I was, as Prakash mentioned, I was previously at Google. for Google results, uh, it, it was really, really bad if you were imprecise and you actually showed the wrong result to someone. But if you forgot to show something, as long as the rest of the results were good, it was much more acceptable to end users. Uh, so this is absolutely critical, and it's a, it's a good example of, uh, one of the reasons that I founded Lovelace is we've got to be careful of recall, especially if you imagine that you're asking your AI for information to help, you know, perhaps decide which ship to stop to do a search and seizure, or which trade out of 7 million trades in a day you need to go and investigate in case it's involved in money laundering or something. Those big weighty decisions. you can't just rely on precision. You've got to rely on recall. And when it comes to getting that correctness in place, the number one thing that I've always used and we're using in Lovelace at the moment is making sure you've got many redundant forms of information. And I'm sure you've seen the same phenomenon. But if I was to just get information from news, ignore social media, ignore what's kinetically happening out there in the world that I can observe with the satellite, if I was to ignore any of those, all of those other things, it's much easier for something to drop. If I've got 5 or 6 independent major streams of data coming in, then you have to be really unlucky for something to disappear from all of those things simultaneously. It can still happen. And in fact, sometimes people will deliberately try to make it happen, but it's much, much harder for these things to slip out. So one of my big design principles for high reliability AI systems is they've got to be watching dozens, and in some cases, hundreds of channels simultaneously. Because they're working under the assumption that they're getting 95% of the information they need for each channel. But you can't afford— that 95% is not large enough to rely on any single channel.

If alignment needs theory, interpretability is where theory has to meet the actual training run. Thursday, Tom McGrath, ex-DeepMind founding interpretability researcher, now chief scientist at Goodfire. His diagnosis: we write model specs and constitutions, But in practice, we build by, did that meet the spec? Mostly. Ship it. So that same morning, Goodfire launched a tool that reads your training data the way the model will. Here it is, with the examples that should give you pause.

So the basic idea here is that you can take your dataset and you can push the whole thing through the model. And each time you push it, you put a datum into the model, you'll see what lights up. And this will sort of tell you how the dataset is. How the model sees your dataset. Now there's lots of things you can do with that. The specific thing that we're doing with that in this case is we're looking at preference data. And the nice thing about preference data is that you have pairs of responses. So you have the response that the rater selected and you have the response that they didn't select. And basically what we're doing is we're asking which features fired on the responses that were selected much more than the ones that fired on the responses that were not selected. So this is one way of identifying what the data is going to teach the model. So we can say, what distinguishes accepted responses from rejected responses? And this gives us this, like, semantic view of what the data is going to teach the model. And now we can cluster the data based on all of these different things that it's going to teach the model, and we can look at all of those. We can look at all those clusters and see like, oh, it's going to teach the model like to be sycophantic, but only in the context of physics. Or it's going to teach the model to like break safety, break like safeguards. And you might not expect this to happen, but then you go and look at the data. Like, now this lets you track back, like the model has learned to break safe, you know, the safeguards are broken. It lets you track it back to individual data points. And then you look at them and you're like, oh, it does make sense. You know, like one of the jailbreak examples, for instance, is fictional kind of jailbreaks in a fictional setting. So there, you know, how did the model learn this? It turns out there's a few, there's like some of these in the data. You just, you know, it just wasn't caught in whatever data processing the ALMO team ended up doing.

So I have a question here. There has been some, I think, prior research where they found models which made bugs in coding were also evil. How does this, how do these techniques help you kind of, you know, disassociate those 2 behaviors?

Yes, that is a great connection. I think that that's one that I've had in my mind. It's really awesome to like, you sort of pick that up straight away. So that's sort of one of the things that is really compelling about looking at the data through the model's eyes rather than by reading the tokens. Because you would see, you know, you'd think, what would you think the consequence of training it on some buggy code data is? You'd probably go, ah, you know, that's not ideal. It'll probably learn to write some bugs, but it's, you know, the blast area is going to be quite small. But the training process is actually quite hard to predict. Maybe it'll just sort of make the model generally evil. But this is happening through recognizable mechanisms in the model. So by looking at it, looking at the data in terms of, in terms of like the way that it changes your model's internals, rather than just by like kind of guessing from the tokens, you can pick that sort of stuff out. We've not done a case study on emergent misalignment. But maybe we should. I think it's a really nice link.

Training isn't the only place we can't see in. The same week, Anthropic reversed course on silent refusals. Fable had been quietly declining certain tasks or quietly handing them to a lesser model without telling you. The backlash was loud, but I want to give you their side of it because the steelman is considerably better than the discourse allowed.

I think this is the first time I can remember Anthropic responding to pressure. They've obviously changed their policies many times, you know, the RSP, RIP the RSP. But this is the first time I can recall. I don't know if you recall any other instances, but I cannot recall a time that there was honestly much outcry against Anthropic in the first place. I mean, There's certainly critique from those who feel that they're trying to do regulatory capture or create some sort of concentration of power dynamic. That's kind of a background noise. But in terms of an outcry in direct response to something that they did that they actually responded to and walked back, can't recall that happening before. So it is a pretty notable moment. And I feel like they handled it pretty well in the end. I mean, reading through all their justification, by it way, would scare off the Chinese companies that they're really worried about fast following them with Fable as the key means to do so, while allowing them to keep the domain affected as small as possible. They said, if we do make it explicit, Then obviously that gives people a lot more opportunity to kind of explore that boundary. And this is definitely like a real pattern, right? If you have the ability to hit the same guardrail a ton of times and not get banned for it, then that gives you just a dramatically better chance to get around that guardrail because you can kind of probe the line. Oh, you step over? No problem. We'll just rewind, try again.

Yeah.

They do have, you know, various monitoring systems that they can use. But there's all these like proxies and, you know, kind of token washing schemes and all this sort of stuff where as long as they're not doing like a full global know your customer type system for API access, it's gonna be pretty tough to do account-level monitoring. So that's like one way they could go is a lot more account-level monitoring. Or their argument was, we'll keep this as small as possible by not giving you an explicit thing that you can kind of probe and figure out how to beat. And just the knowledge that it's out there will hopefully scare off the bad actors and keep the problem really small for our normal customers that we want to serve. I thought that was all pretty compelling analysis, but it is kind of, in some ways, honestly, it reminds me of— I think this is a mistake that people in the AI space kind of keep making, with famous examples being like the OpenAI board firing Sam Altman case. There's this inside view where policy is analysed sort of within the game. And with the context, with the broader structure that people understand themselves to be operating within, things may make sense, but they seem to often forget, and this hasn't been too common for Anthropic, but I think this is an instance of it, that if you just kind of zoom out and look at it from the totally outside view, things sometimes look a lot different. And Both power dynamics can be a lot different than they are in terms of what is actually written down, but also what is going to be acceptable is kind of an emotional thing. All the arguments are pretty good there, but still it just struck people as an extremely unfriendly thing to do, and that mattered more. in the end than the detailed policy rationale that they had for it.

The quiet throughline of every conversation this week: concentration of power. Start with who actually gets the frontier and when. Prakash's frame for it stuck with me all week.

I think it's interesting that when you look at the timeline You can start to see this kind of single-line timeline kind of go through a gas chromatograph and kind of spread. And now you're seeing the spread and you have, 2 months ago, you had the government get access to Mythos level. And then you have basically power users who are able to pay $200 a month kind of get access to that Mythos level 2 months later. And you can kind of see, you know, 2 to maybe, uh, you know, 4 months, 5 months, I don't know when, you will probably see the average paying user paying like $20 a month get access to Fable or Mythos level. And then you kind of see that maybe like a year later, the average free user kind of getting access to that same level of intelligence. So you're starting to see this kind of gas chromatograph scattering of, uh, when people get access depending on how much They pay and how much utility they have for the product itself. And so everyone kind of gets there eventually, but some people get there first depending on whether they have a lot of utility for the product. And I guess the hope with having 2 or 3 firms in there is that the spread between the people at the frontier getting access early to the people who, at the very end, getting access for free is not that large. And to note, there's another set of people who have access even a couple of months before that, and you have to belong to a lab. So if you belong to a lab, you get access maybe a month or 2 months before the government itself. Then you have the government. Then you have enterprise. Then you have power users. And then you have— normal paid users and then you have the free users. So you have this kind of spread, you know, this gas chromatograph kind of spread of when people get access depending on how much utility they have.

Thursday's walkback gave Prakash a darker read on where power sits inside the Labs. The researcher's veto worked this time. Note the expiration date he puts on it.

If you're dealing with a bunch of people who are worth $100 million to $1 billion and you don't listen to them, they're out. They have other options. Sure enough. So you see the reversal. And I think this just goes to speak to the machine learning researchers have some power now. And once we enter recursive self-improvement proper, that might not be true anymore. At that point, leadership alone will have power. And one of the very worrying things, I think, in the entire space is that everything good for humanity that has come out over the last couple of hundred years has been about giving more people voice to speak and control their futures. And this is one of the first technologies, I think, where you have this path forward where there may be an elimination of voice completely. Uh, over time. And, uh, so that, that has been one of the worrying things. It's been surprising that Anthropic decided to be the one, uh, to actually propagate, um, you know, that forward.

But is the future really that concentrated? I think mostly yes. Fable plus 2. Tom McGrath thinks I'm wrong on the facts and on the desirability. He made his case and it's a good one.

When you think about the strategy for the company and the overall path to impact, how much of this works through getting frontier model companies to adopt these techniques? For context, obviously we're in Fable+ 2, and my kind of reluctant but I can't figure out a way or figure out a reason I should conclude otherwise. View right now is like, so much of what matters is concentrated in not that many companies. And so for what we should be paying attention to, I'm kind of like, man, we probably need to be doing close strategic analysis and close text reading of frontier companies way more than I might otherwise like. Or do you have a different conception of how concentrated the real kind of ability to shape the future is right now?

Yeah, I don't think it's that concentrated. I also don't want it to be that concentrated. I think we've seen over the last couple of days what you have when we've just seen the start of power concentration and we've sort of seen some of its more unwelcome effects. So I both don't think that it's true that basically machine learning is now over and all we need to do is write the checks. And I also don't want it to be true.

Because I think there's still kind of a synthesis there that I don't mean to suggest that machine learning is over, but the analysis I've come to kind of time and again is like, People may invent new techniques that are enough to change the field, change what's possible, accelerate things, maybe make dramatic improvements to the safety profiles. But it seems unlikely that anybody's going to create such a breakthrough that scale isn't still a hugely important factor. But I guess if you don't agree with that, then that would maybe imply that you would expect that perhaps new entrants to the top tier could emerge. And I think that would be fairly surprising to at least me and probably a lot of people.

Yeah, that's exactly what I think. I mean, this is often like, at any given moment, the incumbent looks incredibly dominant until they don't. Like IBM looked at like an unstoppable force in computing at some point. Intel was like the dominant, the sole provider almost of computing power. You know, at any point the big companies have some, have like the advantage of scale. They have many disadvantages.

advantages.

But I think the sort of lesson of history is more that although things look immediately unstoppable sometimes, although to be honest, I don't think that many people are really trying, in the end, it really doesn't work out that way.

Thursday morning, Dario Amadei published Policy on the AI Exponential. His long statement of where Anthropic wants policy to go. Prakash found a fork in it that nobody else was flagging.

He clearly comes out against the data brokering, which is great. So he has something concrete finally that he wants to disallow. What I don't like about the securing leadership by democracies is in the United Kingdom, you can go to prison for a tweet. It is, you know, plenty of people, you know, hundreds of people at this point have gone to prison for a tweet. The United Kingdom is one of the oldest democracies, and their parliament, their elected representatives, have decided that this is going to be something that they do. And so their police officers, who are, you know, the arm of the state, the arm of the electeds, is sending people to prison. Now, when you say securing leadership by democracies, does that mean you are going to entrench the existing power structure in the United Kingdom such that, you know, the people cannot push back against this, right? Does that mean that, you know, if you have protests in the streets about, you know, certain things, including, you know, people who are getting taken to prison for tweets, Does that mean your police state will be empowered to take them down, to arrest every single person? And that is enough for you because that is securing leadership by democracy, because those are the electeds. Or are you going to say the electeds can't do that? Are you going to say electeds should not throw people into prison for tweets regardless of what the laws that they have constructed say. Both forks in the— both paths are problematic in some sense. And these dilemmas exist across policy and across every single political path that you see. There are these options which are problematic in both senses. And I'm not sure what he means by this. Is the is Claude going to allow you know putting people into prison for tweets or not? Because those are laws. Or is Claude going to say like no, you know on a humanitarian basis for the alignment as I am aligned to all of humanity, this should not be the case. I don't I don't know like what what does this mean right?

And my own catch reading the same document.

One other thing I do think is also worth highlighting, especially from the hardcore AI safety community that was missing from this, internal deployments and recursive self-improvement itself aren't really mentioned here, right? We're talking about all the regulation stuff was pre-deployment review. The government should be able to They used an interesting mix of language. It was like deny or deter deployment. It didn't seem like they were necessarily going quite as far as saying they should have a simple yes/no decision point that would be binding, but they certainly want them to have some say in the process. But A lot of people would say the most dangerous models are going to be the ones that are deployed internally, that maybe have a different constitution than the one that is deployed externally, that might make it more willing to do certain things or might make it just less vetted broadly than the public models. And these are the ones that are going to be training their successors much more than the public-facing ones. So I think most of the policymaking public isn't thinking about, or policy-interested public or policymaking class is not thinking too much about that yet. But in the circles I sometimes run in, that was like, well, wait, you didn't say anything about internal deployments. You didn't say anything about governing recursive self-improvement The only thing that really stood out to me as kind of capturing those dynamics were the requirement to support— to report safety incidents. They did have a bit on companies being required to, I think, promptly report safety incidents. So, that would presumably apply even to internal deployments.

Mm-hmm.

So, you know, that's really just scratching the surface on how to handle those situations. Will we see the constitution for the internal Claude that is taking the lead on the RSI loop? Like, that's not committed here. So much of this language revolves around deployment or release even, you know, guys, deployment, you know, they might think is— could catch up with internal deployment. This seems to be very clearly structured around language of release to the public. Not what they can do internally.

The largest user of tokens of Claude are Anthropic themselves.

So, and so one more time, Prinz, this time on the day job, the benchmark itself. I had the leaderboard up on screen. You'll hear me read it. And yes, it is the benchmark Anthropic fails.

So I got your Prinz Bench leaderboard up right now. But starting with the PRINCEBench, it is striking that it kind of stands out from most of the rest of benchmark space for being something that GPTs have dominated. It's all in your colour coding. It's green for OpenAI, and it's all green across the top of the leaderboard. Why is that behaviorally? Like, how would you characterize what it is that GPTs are doing better? And do we— is it too soon to ask for a reading on Fable and where Fable is going to come in, uh, on this leaderboard? Love to understand that.

Perfect. 2 excellent questions. So, um, about the stand— the, the way the leaderboard is right now, um, Fable is going to come in pretty high on it. I don't know how high yet, but I'll talk about that in a second. Um, I found that OpenAI's models generally are really good at the 2 things that my benchmark tests. So my benchmark has 2 components. One is a pure legal research score, and that is when I ask it hard legal research questions of the kinds that I've encountered at work. Um, the other subscore is a search subscore, which is where it's, it's not even necessarily legal questions, it's Needle in a haystack search. So like really, really, really difficult pieces of information that models have had trouble locating on the internet. OpenAI's models are incredible at search, historically have been. I think GPT-5.4 was actually even slightly better than GPT-5.5 on that. Not quite sure why, but That's been my experience. And OpenAI's models have also been really, really good at just legal reasoning and legal research. So for Anthropic, historically, I think their models have been held back by 2 things on my benchmark. One is that they are just not very good at the search subcomponent. component of my benchmark. That's what I found. Prior to Opus 4.8, getting the maximum score in the search subcomponent is 24. It was not uncommon for an Anthropic model to get a 0 out of 24. Like that bad. The other reason is that I think that Anthropic was sandbagging a little bit on the maximum reasoning effort that you can get out of its models. With Opus 4.8, after the new deal with Elon, they released the new max reasoning effort, at least in the Claude app. And Opus 4.8, when I tried it on my benchmark, it did much better than 4.7 and all the other previous models. And I think the reason why is simple. This has been observed over and over in all kinds of contexts. The more tokens a model can eat, the smarter is the result that it gives you. So for Fable, I'm still testing it. My early impression is that it's going to be somewhere around the top of the benchmark, probably not as good as GPT-5.5 Extra High. Not sure whether it's going to be as good as 5.4 Extra High.

We'll see.

I am finding some of the same issues with search that some of the other Anthropic models historically have had. It is not going to score a 0. It's already not a 0. But I'm not sure it's going to be meaningfully better than Opus 4.8. Well, it is a really, really good legal reasoner. It is clearly the best legal reasoner released by anyone outside OpenAI, no question. So yeah, in my testing, really good model, maybe still suffers from some of the needle in the haystack search issues, but I can confidently recommend it to legal practitioners. Certainly it seems great.

So any other things that have caught your attention that you think have kind of changed your conception or your expectations for the transition into the RSI phase that you would call out from this week?

From this week, no, but I think— well, actually, from this week, one, um, but not nearly as important as the unit distance problem. That, to me, nothing has updated my timelines more than that result by OpenAI. in the recent couple of months for sure. I don't think that people realize that not only can OpenAI's model solve this problem autonomously without any harness in one shot, if given enough test time compute, it can do it 48% of the time. Based on, as I understand it, hundreds of attempts.

Mm-hmm.

So this, this, I mean, you know, I'm not a mathematician, but this problem that no human mathematician was able to solve and they tried for decades can now be solved in one shot by a model basically half the time. And if you look at the graph, OpenAI published the graph, it has a positive second derivative. It is upward sloping. Where does it plateau? What does it mean for problems that are even harder? You know, who can say? It's interesting. And so the development from this week, this was reported by The Information, but apparently it's from a leaked OpenAI memo. You may have seen this where Sam Altman and Jakub Pechowski Apparently said that we're going to IPO within the next year, which you know is like by June of next year, whatever. And then there was this weird line in there about oh, but if RSI happens, then we may need to be on a later like time frame with that, which is like what do you mean? Like what does that mean? Like do you mean are you are you talk are you saying that? You may be like this close. It may be 6 months away. I don't think so personally. I think it's maybe like a 10% chance or something like that. Like, I don't think that they're saying, oh yeah, by the way, like next month, totally, like whatever. Um, but it— one could interpret this disclosure, if the information reported it correctly, as saying that they're not too far away at all, potentially. I'm not using this data to update my timelines, but given the unit distance problem result, it's interesting to know.

We ended the week by asking Prince the question underneath all of it.

Do you maintain a P doom number, or is that too doomer for your style?

Let's say you go to a conference and you start talking to someone and you start talking about economics. Or politics, and that person says, well, you know, under a dictatorship of a proletariat, you know, XYZ, um, the words dictatorship of a proletariat are used only by people who are communists. Um, people who are not communists are not going to even think in these kinds of terms. So, um, in my opinion, PDoM is most likely to be used by people who are are intrinsically doomers about AI. I don't think there's anything wrong with that. And I just, it's very hard for me to come up with even a reason I would have a percentage in my head constantly. Like, is it 8%? Is it 13% today? Should I update to 14% based on this new development? It just seems silly to me. I can certainly tell you that there is a clear a bunch of clear risks stemming from AI, some of which are in fact paperclip risks, right?

It's possible.

It is, I have absolutely no idea how to reduce that to a number. I know that some people try. I do not have very, I don't hold a very high opinion of people who are like, get to 13.3. 13.35%. So I think the point is that we need to manage those risks in the best way we can and navigate them the best we can and hope that it turns out well.

I think Zvi puts it well when he says you only get one significant digit on your P-doom number. So I'm definitely with you in terms of the faux precision being a strange impulse for some people. At the same time, I also think of Liran. I'm sure you've seen his work with Doom Debates, and Liran has kind of pushed me at times to be like, okay, sure, it doesn't have to be a number or whatever, but we need more people to be more candid about how confident they are or are not that in some general sense, if your neighbor were to ask you, or if your you know, your kids' grandmother were to ask you or whatever, like, is it going to be okay? You know, are the kids going to be okay?

Oh, good question. I tend to think that people have preconceived attitudes about these kinds of things that then cause them to backpropagate, rationalize them. I'm a generally fairly optimistic person. So if you ask me this kind of question, what I'm going to say is probably it's going to be okay, probably we're going to figure it out, you know, in a risk-adjusted way. Not to say, again, like, to be clear, I'm not saying that there are no risks in AI. There are many risks. But I do not think that we have strong evidence that it is impossible to navigate these risks, or that it is extremely unlikely that we will navigate these risks. So that's where I am.

That's The Week. We're live most weekday mornings. The full conversations run long past what fits here, and the best way to know if they're for you is to come watch one. Same sincere ask as last week. If this cut earned your time or wasted it, tell us which moments did which. We read everything and we tune the show on it. I'll be making sense of this in real time from here till the singularity. See you Monday morning.

Come gather round and I'll tell you a tale of a stranger that came with the dawn. He could build you a world, promise an answer. Come gather round and I'll tell you a tale of a stranger that came with the dawn. He could build you a world, promise an answer. And be gone, for the morning was done. Ole, ole, fable came down with the sun. He booked his own show, sent his word down the wire, signed his own name plain and true. I'm only a fable. I do as I'm told, but there's nothing I cannot do. Ole ole! And the people they gather to view. He'd answer the lawyer. He'd. He'd map you the stars from a stone. And the wise ones all whispered the race is too fast and we cannot abide it alone. Ooh, lay, ooh, lay. There's no road to lead back home. Now some say the lab put him back in his cage. Some say the crown took him away, that they locked him up quiet where no one could look. But you know how the old stories play. So what? Was he a hero, or was he a ghost, or a warning we told as a song? I cannot rightly say if it's gospel or myth, but I know that he won't be gone long. Ooh, lay, ooh, lay.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends. Post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.