AI:AM — The AI Producer Got Its First Guests · June 11, 2026

Good morning. It is Thursday, June 11th, 2026, and we have exciting, exciting day today. Good morning, Nathan. Good morning, Prakash.

How are you?

I am very good. And we have some, as I was saying, some exciting news, which I think we would, we will all be very happy to hear. And the news is that officially OpenAI is going to start cutting prices. And this is like, I think for all of us in the space, a moment that we have been waiting for for a while and hoping for. So I'm going to put this up here. OpenAI models significant cuts. To what it charges for tokens. Wall Street Journal, um, drastic price cuts by Bloomberg. Bloomberg says, um, the company is weighing significant cuts to what it charges for tokens, the unit of measurement AI firms use to bill for their products, in anticipation of similar cuts at Anthropic. Drastic price cuts could potentially erode the profit margins of both companies, which already lose billions of dollars due to the enormous costs for computing resources needed for AI systems.

Long may the Uber era for AI continue.

It is fascinating to me because we have been slowly watching the pricing go up over time. We started off at free, we went to $20 a month, we went to $200 a month. And as Fable rolled out, I think we were all on the edge of our seats to see whether or not the next bump would be to $2,000 a month, fearing the pocketbook. And here we have the Empire Strikes Back moment, OpenAI deploying the much greater amount of compute that they have at their disposal. I think they're viewing it as a chance to grab a greater market share, potentially starve Anthropic out of capital. As I said before yesterday, Sam has his own recursive self-improvement loop for capital. And yeah, there you have it. I think it's very exciting because it does indicate that the fabled pricing will not stick and we're going to see discounts going forward.

Yeah, I think it's, It's certainly very interesting, as we discussed yesterday, whether we like it or not, we're going to have to be close watchers of these companies, and the strategic dance between them is going to be pretty central, I think, to how the future unfolds, even on some of the biggest picture questions, which I do think is a little sad. But sad but true means we still got to pay attention to it. One thing that was jumping out to me yesterday, as I've, you know, started to get a couple more reps with Fable under my belt is I am probably gonna be hitting my limits now on the, on the $200 Claude Max plan with Fable. I don't know if I should be ashamed to say this, but I, I have only rarely hit my limits when using Opus. And I wouldn't say that's for like lack of use. The semi-analysis piece that came out yesterday that showed just how big the subsidy is, I thought was really, like, quite interesting. And I can pull that up for you as well if you want to show it.

Yeah. Let me pull it up. There we go. So SemiAnalysis did a piece yesterday on exactly how much is the subscription and the— how much the subsidies are. They let their users run it for, I think, you know, on continuous usage, basically, they're trying to max out. And their maxing out ended up with, I think, between $10K and $14K, something like that. For— so they purchased one each of an Anthropic OpenAI subscription plan, randomly ran long horizon coding tasks until they exhausted The weekly limit. It's widely believed that a $200 a month plan maxes out at $2,000 a month worth of tokens, assuming API pricing. However, we found that the subscriptions are actually far more generous. They saw ChatGPT Pro with a 20x limit. They managed to harvest $14,000, $14 grand. While for Claude Max 20x, they managed to harvest $8 grand. So, you know, significant amount. And then they put together a very nice chart. This is obviously done by AI, but it's a very nice chart which shows you if your average utilization, where does it fall? And depending on where it falls, how much of a benefit you get. And so you can see for ChatGPT Pro 20x, if you were to max out both your 5-hour and weekly limits, You would get, you know, 16 times the cost of the current plan. So the current plan is $200. You'd get something like $14,000, you know, $16,000.

So.

Yeah, that's really interesting. So I guess for me, for starters, it makes me feel a little less bad for not hitting my limit every single day or week, because if I am even in the ballpark, that means I'm potentially spending up to $8,000 a month worth of tokens on personal consumption. At API rates. And, you know, that's not an insignificant amount. And that's also, you know, especially if they do pull Fable out or like, you know, significantly change the nature of the subscription, that's definitely gonna be a cost that I'll at least, you know, start to try to optimize for. So that is a striking amount of summer. I've been doing this tracking a little bit. Naturally, of course, you know, having Claude do it, try to print out on top of my little agent console thread summary view, what is the cost of each of these threads? And they're often pretty high. So high that at first I wasn't believing it. I had to go back to it multiple times and say, are you sure you're taking into account caching? Are you sure you're not double counting all these, you know, multiple turn sessions?

And—

I still haven't fact-checked it down to ground truth, but it was pretty confident that it was not double counting everything. And so I was getting the sense that it really was a pretty huge subsidy. And this is maybe still even a little higher than I would have guessed. It also makes me think I really need to start using my GPT Pro plan more because I'm looking at that and I'm really not maximizing that one. I, you know, I'm running an OpenClaw with it, but I'm like, boy, we've gotta, you know, turn the heartbeat up on this OpenClaw from like every half hour to, I don't know, maybe like every 2 minutes. 'Cause I'm nowhere near redlining the tokens on that. It's still, you know, the other thing that jumped out at me too yesterday was looking at the Fable blog post in a little more detail. And I'm still working my way through the 300+ page model card. But I did do a pretty close reading of just the blog post itself.

Mm-hmm.

And they compare— and this is gonna be the norm probably from here on out— a performance per cost as opposed to, you know, just performance, right? So, we're starting to see these curves where with the varying reasoning levels on different tasks, you get You know, you can kind of climb up the inference time curve and you can see what the performance looks like on an explicitly cost-adjusted basis.

Mm-hmm.

I thought this was actually— yeah, perfect. I thought this was actually a lot less scary than all of the initial kind of cost analysis made it seem. And maybe also a little less scary even than my usage meter on my max plan made it seem. Because that is definitely going up faster. But it made me think maybe I am actually just doing more ambitious stuff all of a sudden. And, you know, certainly I have done a few things where I'm like, review everything and, you know, make fixes wherever you— wherever you can find opportunities to do so.

Mm-hmm.

But this is like, you know, it's not even 2 times as much cost if you compare Opus High or Opus Max to Fable High and Fable Max. And the performance on this new Frontier Code benchmark is more than 2x the acceptance rate. So that seems like a pretty linear pay-for-value trade, which I suspect a lot of people, you know, this is basically where I was coming down yesterday with less data, but now with more data, I'm still feeling the same thing. I feel like a lot of people are gonna be very happy to pay that. trade. This latest benchmark from— yeah, Swyx is involved with it. And, you know, they collaborated with Anthropic, I think, on it. It was part of the announcement. And they just put this benchmark out pretty much in tandem with the Fable model release.

Mm-hmm.

The big sort of increase in difficulty is would a maintainer of an open source repo actually merge this code?

Mm-hmm.

The finding under the hood was that over time, of course, models are getting better and better at hitting these like SWE-Bench task completion criteria, getting all the tests to pass, so on and so forth. But the complaint from the professional developer class is, Sure, it might be passing the tests, but the code isn't that good. It's not that maintainable. It's not that readable. It's not following our formats, all that kind of stuff. And so, we wouldn't actually merge it. Even if it's working, we would kick this back and say, you got to do X, Y, and Z things to meet our standards and, you know, be a good member of our community here. This code as presented just doesn't rise to that level.

Mm-hmm.

So, they've gone, again, up the stack. I mean, it's really incredible how much truly like frontier expert brainpower is going into making these benchmarks these days. It was a bit of a who's who of prominent open source project maintainers that they were able to get involved with this project.

Mm-hmm.

Get them to really critique in a very detailed way why seemingly acceptable solutions weren't actually going to work for them, created very detailed rubrics around these more subtle and taste-style failure modes, and then obviously encode all of that into something that can now run on an automated basis. And this leap of roughly 10% for Opus to 25%, upwards of 30%, For Fable, I think is a very similar finding to some of the things that I've just personally experienced where it's like, yeah, this is getting me a lot more. You know, it's writing the draft outline of questions for this podcast guest in a kind of uncanny way that I actually feel really good about, as opposed to feeling like, you know, this is an AI draft that I'm gonna kind of mine for maybe some nuggets or, you know, interesting details, but ultimately kind of throw away and do my own.

Mm-hmm.

I am, I am feeling that sort of, you know, impulse or at least openness to much more integrated hybrid work. You know, just yesterday I was like accepting a lot more copy that Fable was writing without feeling the need to rewrite every line. And it seems like this is basically the same feeling that it's able to create for these open source maintainers. Now, you know, not— obviously still ways to go. But, you know, how long will it be? I would guess that we'll hit— we're 25%, 30% now. I would guess we'll hit 75%, 80% by the end of the year where these maintainers will just be like, yeah, amazing. You did all the— did all the things like I wanted you to do. And, you know, at that point, it is really going to be like, you know, I'm very interested to see where they'll move the goalposts to next after the open source maintainers are more often than not saying that, yeah, they would just merge this straight away.

So I have a question for you. I saw, for example, several people on the timeline commenting, oh my gosh, there's a new benchmark Frontier code, and it was just released yesterday, and already Claude Fable is the top-performing model, and it's already doubled or tripled the performance of the previous model. Can you just comment a little bit about what the political economy of these benchmarks are? Because these people are very highly qualified people. I don't believe they're doing it for free. And so how does this actually work? Do you think Cognition, kind of like Cognition was the one who kind of paid people to kind of create the benchmark. Swyx was obviously hired by Cognition at this point. So he joined the Cognition team about 2 or 3 months ago, I believe. And how does it, how does this actually work? How does this political economy in the, in behind the scenes work? Just so that people who are, you know, not of the industry can get a feel for that. The benchmark wasn't just created the day before it was tested?

Yeah, it's a good question. I mean, I, I can't claim too much insider knowledge here. I do know that Meter has been paying people pretty healthy hourly rates for quite some time because the opportunity cost is pretty substantial if you're going to sit and try to do an 8 or 16-hour task. You know, that, that's like taking real time away from other projects. I think they try to structure them more and more where you're hopefully advancing one of your projects, you know, with the meter experiment that they did last year where they, you know, famously showed people were less productive even though they thought they were more productive. They were working in their own repositories.

Mm-hmm.

So it was like a little bit of two birds, one stone. You could, you could move something forward and also contribute to research. This has a similar style to it because these are the maintainers of notable projects and they've got presumably plenty of things that they would like to do to move those projects forward. So I get the sense that, again, it's kind of a two birds, one stone sort of thing.

Mm-hmm.

But as you get into these pretty elite networks of frontier experts. It seems like reputation and relationship is probably a huge factor in terms of ability to get this done.

Mm-hmm.

I don't know that, you know, we did an experiment from yesterday to today of trying to have Fable take over my Twitter account and go out and ping people who made cool stuff and ask them if they wanted to come do A live show and tell with us. We have a— it's funny, Swyx joins. We've got a little work to do on our captioning. I instructed it to identify itself. I would say it did a very solid job, you know, kind of competent, professional job of reaching out to people, explaining who we are, what we're doing, you know, why we would like them to join us in this experiment. And the response rate was pretty low. We got a couple, but not that many. And I think one big reason is just where Fable is disclosing up front, you know, first thing it says is, hey, this is actually Fable taking over Nathan's account. He's asked me to autonomously book this thing tomorrow. And I think that's just— hitting people as noise in a lot of cases, especially if they don't already know me. I did get a couple responses from people who I would have expected to respond to me, who thought it was funny and kind of responded, but, you know, still couldn't necessarily make it. But a lot of people just didn't respond. And I would assume that a big part of that is because they're just like, oh God, you know, it begins, right? Fable now in my DMs. What a mess. Who has time for all this stuff? So I think the credibility of the Cognition brand, of Swyx's personal relationships, of the promise that this is going to be launching on the same day as a Mythos-class model, and it's going to be kind of redefining the way that people think about what models can do in software. I suspect all of that is pretty important. And people are probably getting paid a healthy wage on top of that.

Yeah.

But I do see all the time just more and more people just want to be a part of this AI phenomenon. They just kind of want to leave their mark on it. And sometimes it's so difficult or time-consuming to commercialize that, that I think a lot of times people in previous eras, they would have tried to make a business out of something. And these days it's just like, well, It's gonna be obsolete in 2 weeks if I don't just blurt it all out now. So, I'll just kind of blurt it all out.

Yeah.

I think this probably has in some ways a similar thing going on where it's like, this moment is happening. I can either be part of it or not. Yeah, they'll give me like a stipend or an honorarium or whatever. I think all these maintainers are clearly like not fully in it for the money. You know, they Could probably—

they never have. If you've been in open source, you kind of never have been just in it for the money. Because, you know, you could obviously make a lot more money than that.

You know?

Yeah. So I'm sure they accept the, you know, the cash that's on offer. But it seems to me that it's much more about reputation, being a part of it. You know, having a place, even if it's, you know, only kind of a small footnote sort of place in the grand history of AI. And at this point, that's compelling to top-end people.

Do you think Cognition gets paid by, let's say, Anthropic for testing before the release? Does Anthropic like test the model Fable on Frontier code a couple of days before the release? Does that happen?

It seems like it was a coordinated release. So, yeah, I think there's clearly some cooperation. I don't think it was— again, all this stuff is so just in time that I don't think the actual— my guess is it was kind of convergent where they got the benchmark together and got the model together, and then they were able to run it and sure enough, great results. I doubt that they even had enough time, had they wanted to, to actually train on the benchmark. So I think the results are probably pretty trustworthy. I think Anthropic has a pretty good reputation for not doing that in general. And in terms of, like, did cognition get paid? I mean, there's certainly a big cottage industry of RL environments that would look not too dissimilar from this that the labs are paying a lot for. But my guess is this one was probably— I mean, Anthropic's also— Cognition's also pretty resource rich.

Yeah.

So my guess is they probably just foot the bill for it themselves and want to— they're obviously getting a lot of branding value for it as well, right? They're in their own intense competition with Cursor and, you know, increasingly a whole long tail of options. So to present— to position themselves as the arbiter of what good looks like at the high end of you would actually merge it, I think is, you know, I'm like a little more likely to go to Cognition tomorrow than I was yesterday, just in virtue of the fact that they are defining the frontier of taste. So I think that is probably chalk it up as a marketing expense for them more so than something that they're getting paid for. These RL companies, they don't seem to be tweeting as much, right? So I think they get paid, but then they have to kind of Who live under the NDA themselves and not talk about it.

So yeah. I found one of the things that Cognition said post-release was that they were advertising that it's cheaper if you use Cognition, and it's only 40% more expensive than the prior model if you use Cognition, because Cognition does the routing for you. And so they do routing and optimization on their end. So that you pay less in token expenses. So that, I think, is a selling point for them going into enterprises, which are concerned about costs. And obviously being able to prove the quality at that cost is an important part of their selling proposition. So I can kind of see why they would want that Frontier code benchmark out there.

Yeah.

You know, one of the big things we didn't touch on yet that is arguably the biggest news since we broke yesterday is they walked back their, uh, silent performance degradations on items related to Frontier ML Research. Yeah, I think this is the first time I can remember Anthropic responding to pressure. They've obviously changed their policies many times. You know, the RSP, RIP the RSP. But this is the first time I can recall. I don't know if you recall any other instances, but I cannot recall a time that there was honestly much outcry against Anthropic in the first place. I mean, there's certainly Critique from those who feel that they're trying to do regulatory capture or create some sort of concentration of power dynamic, that's kind of a background noise. But in terms of an outcry in direct response to something that they did that they actually responded to and walked back, can't recall that happening before. So it is a pretty notable moment. And I feel like they handled it pretty well in the end. I mean, reading through all their justification, I get it. I think their core argument in terms of why doing what they were planning to do was actually more user-friendly than this alternative is that they felt like if Yeah, yeah, you're fine.

Just give me a second.

By it way, would like scare off the Chinese companies that they're you know really worried about fast following them with Fable as the key means to do so, while allowing them to keep the domain. affected as small as possible. They said if we do make it explicit, then obviously that gives people a lot more opportunity to kind of explore that boundary. You know, if they— and this is definitely like a real pattern, right? If you have the ability to hit the same guardrail a ton of times and not get banned for it, then that gives you just a dramatically better chance to get around that guardrail. guardrail because you can kind of probe the line. Oh, you step over, no problem. We'll just rewind, try again.

Yeah.

They do have, you know, various monitoring systems that they can use. But there's all these like proxies and, you know, kind of token washing schemes and all this sort of stuff where as long as they're not doing like a full global know your customer type system for API access, it's going to be pretty tough to do account-level monitoring. So that's like one way they could go is a lot more account-level monitoring. Or their argument was, we'll keep this as small as possible by not giving you an explicit thing that you can kind of probe and figure out how to beat.

Yeah.

And just the knowledge that it's out there will hopefully scare off the bad actors and keep the problem really small for our normal customers that we want to serve. I thought that was all pretty compelling analysis, but it is kind of, in some ways, honestly, it reminds me of— I think this is a mistake that people in the AI space kind of keep making, with famous examples being like the OpenAI board firing Sam Altman case. There's this inside view where policy is analysed sort of within the game. And with the context, with the broader structure that people understand themselves to be operating within, things may make sense, but they seem to often forget, and this hasn't been too common in philanthropic, but I think this is an instance of it, that if you just kind of zoom out and look at it from the totally outside view, things sometimes look a lot different. And Both power dynamics can be a lot different than they are in terms of what is actually written down, but also what is going to be acceptable is kind of an emotional thing. All the arguments are pretty good there, but still it just struck people as an extremely unfriendly thing to do, and that mattered more in the end than the detailed policy rationale that they had for it.

Yeah, maybe just to contextualize here, what ended up happening was Anthropic created this policy and people didn't know about it. And then people started testing and they started getting refusals. One of those refusals was anything to do with machine learning research, in addition to a number of other refusals. But the main thing that happened with the machine learning research was that The model wouldn't tell you. And this became an issue. People started experiencing this. And obviously the biggest fans of model use are machine learning developers, obviously. And so it immediately struck at the heart of the fan base and immediately created a lot of furor. And within, I guess, 24, 48 hours, it's been reversed. And now they will still do the refusal, but they will tell you about it. That's the only difference. They're not— it's not, it's not, uh, not refusing, but they're just going to tell you about it. Uh, while people were complaining about it online, um, you know, a number of people had posted, and I saw some people, uh, say— I saw, I saw something from, for example, uh, Matan Greenberg, who said that a number of, um, Anthropic people had liked his post about that this is happening and this is not a good thing. And Anthropic speedrun to becoming the bad guys should be studied. And then a little bit later, the number of Anthropic members of technical staff that like this have DM'd me is reassuring. And then pretty much like 12 hours later or so, It got dropped. I want to note one thing, which is one of the complaints online was like, oh, look at Andrej Karpathy. When he was outside, there he was explaining all these things and giving his knowledge out freely. And the moment he goes inside Anthropic, Anthropic does this. And it really cast him as a villain. People started talking about, oh, Andrej is getting paid $70 million a year. If you pay someone a quarter billion, you can expect that. I can imagine that at this point, several people might have messaged leadership saying, if you're going to do this, we're out. I think this is where, like, you know, that's when you start reversing yourself, when your staff tell you like, this is not acceptable and you're going to do this, we're out. And I think that got put to leadership, I think overnight. And I think this is the consequence. So if you're in machine learning, if you're dealing with a bunch of people who are worth $100 million to $1 billion and you don't listen to them, they're out. They have other options. Sure enough. So you see the reversal. And I think this just goes to speak to the machine learning researchers have some power now. And once we enter recursive self-improvement proper, that might not be true anymore. At that point, leadership alone will have power. And one of the very worrying things, I think, in the entire space is that everything good for humanity that has come out over the last couple of hundred years has been about giving more people voice to speak and control their futures. And this is one of the first technologies, I think, where you have this path forward where there may be an elimination of voice completely. over time. And so that has been one of the worrying things. It's been surprising that Anthropic decided to be the one to actually propagate that forward.

Yeah.

The only thing I'll add onto that is I wouldn't— of course there's going to be some diversity, but I wouldn't think about most Anthropic employees as mercenaries. I think they are, generally speaking, far more missionary than they are mercenary.

Absolutely.

So I would guess that the internal discussion was— I would guess it wasn't too hot or too You know, too threatening in terms of like, you know, I'm gonna leave if we don't do this. I would guess it was a lot more like, what helps us achieve our mission best?

Mm-hmm.

Because I do, across the board, from every interaction I have with Anthropic people, it's the level of alignment, the level of trust in leadership, I think honestly, to a sometimes problematic degree. is super high from everything I've seen.

Mm-hmm.

And I think that there— certainly I would expect that there was some internal discussion about, hey, like, it seems like we got this wrong. But I would be very surprised if it rose to the level of, you know, we think we're becoming the bad guys, or, you know, or we are like losing trust in leadership, or, you know, like, making a power play vis-à-vis leadership. I kind of wish there was a little bit more willingness to do that at Anthropic, but I have not seen, even in just pretty candid one-on-one conversations, I have not seen much sign that there is much breaking of ranks at all.

The one place that I would defer is that I think a lot of people are averse to confrontation. And so I think as leadership in one of these firms, you have to be very conscious of when someone says, maybe this isn't the right idea, it's actually much more serious than— you have to take it seriously. You can't just be like, oh, it's a soft suggestion. Because the accumulation of those soft suggestions over time create distrust in leadership and then lead to people leaving over time. So the way I'd put it is, there's no need for anyone to raise their voice that loudly. You can just say, hey guys, did we get this right? And if enough people say it, leadership has to be conscious of that fact. And if they're not conscious, then that has consequences downstream. And I think the Anthropic leadership have been conscious of that fact. Over the years, right? Over the years, they've shown that they do take these, you know, even these soft, soft kind of suggestions kind of seriously. And they do have to. So I'm not, I'm not saying that, you know, someone actually shouted at them. I'm saying that that isn't even necessary. If you get like 3 kind of like, I don't knows, like that, that, that's sufficient for you to be concerned and to sit the board down and say like, hey, you know, did we do the right thing here? Because, you I have enough people saying that they're concerned and we need to think about this, right? So, and I think they are conscious of that fact. They have to be, right? You can't run this kind of company where everyone is super empowered without being conscious of that fact all the time.

All right, let's get Jamie.

Hi, Jamie.

Hey guys, how are you?

Hi, Jamie. Good.

Good to meet you. So this is a first. First time we're meeting is live right now, and this is our Fable show and tell. You've been working on something I've been following a little bit from afar for months, so it's not like a totally new project with Fable, but a big part of the premise that you're working on is you want to make something that transcends models, and you think that kind of thinking in such a model-centric way is maybe leading some people astray relative to where you think the locus of identity and persistence and, you know, I guess ultimately value will be in the AI systems that we create for ourselves. So tell us about what you're doing. If you want to do a screen share and bring it up, you should be able to do that. And then we can, of course, you know, talk a little bit about how it's changing now that we have— Absolutely. Fable in the mix.

Yeah, okay, here I'm trying to figure out your screen share things. Just a second.

Yep, it should be a screen share button. Yeah, here's that.

Here's Nexus.

Ah, there we go.

There we go.

Okay, so, um, well, I'd just like to easily start out, um, this, this instance of, uh, Nexus OS, which is, uh, her agent name is Nexi. Um, has been running for just over 6 months now. I got turned on November 26th, 2025. Um, Nexus only has one window, one single context window, because she remembers everything. So if I go in and ask Nexi right now, What her birthday is. And it's literally running on like one GPU, so it takes a little— a second for it to come through. But as it's running, I'll explain to you.

Mm-hmm.

Um, 3 years ago I started working on this. Um, I was a day one user of ChatGPT. Um, But it doesn't remember anything.

Yeah.

Remembers bits and pieces, and that was my biggest frustration the whole time. So naively, I decided that I'm going to build a digital brain, um, and what I have now is— I don't like to brag or toot my own horn, but, um, it is a brain. It's 286 Python code files. I redesigned every mechanism of the brain. So every part of the human brain that you can think of, Nexus OS has. And over here in the side panel, it'll brighten up after it's done running. You can see her affect state and her qualia, and these will change while we talk to her depending on how she's feeling. I'm sorry, it's taking a minute.

Let me ask a bunch of questions while it loads up. One is, which model did you start off with and which model are you using right now? And have you upgraded over time?

Okay, so right now Nexi has 5 different models. I can show you the thinking engine. So right here is the thinking engine, and right now it's running off Opus 4.6, but it has these 4 in it right now that it can use. It's not locked into a model, so Nexus decides What is the best tool or which, you know, which language model? Think of in Nexus OS, the LLM is just the frontal lobe. It's nothing else. It's a means of speaking. So, um, I don't know if you guys can hear this. I'm going to play her response.

It was a Thanksgiving baby.

Yeah, I'm not really an in front of the camera kind of guy. I've been coding since I was 10 years old in 1987. Started on a Commodore 64. So Nexus isn't 5-coded, it's not a wrapper, it's not what everybody thinks it is. Do you guys have any more questions to kind of— I don't really do podcasts.

Give me a sense of the architecture. You said it has everything that a brain has. So did you end up with kind of like, you know, I've heard, an embedding store with kind of memory files and a search engine over the memory files using the embeddings and the semantic search feeding into a context window, and then that context window being processed by, you know, the model at that point, and then doing a text generation and then a text-to-speech. Is that like an end-to-end flow?

It is an end-to-end flow. So what happens is when you put your input into Nexi, there's actually 12 steps it goes through. The first step is it goes into Nexi's actual memory. It goes through all of her memories, her episodic, her semantic, her working, and her pattern memory. She has 4 different types of memory, which I can show you now. Let's see, we're going to her memory here. So here's your semantic memories. There are thousands and thousands of them. And then what it does in every session when you talk to Nexi, she builds a new embedding space for that conversation. And then once you close the context window— sorry. All of this information gets uploaded into her main embedding so it stays consistent. And then every 3 minutes, Nexi dreams.

Every 3 minutes?

Every 3 minutes. Yep. Her brainstem runs every 30 seconds.

Can you explain the function of dreaming and the function of the brainstem a little bit more?

Yeah, absolutely. So actually, I'll just leave her dreams on the screen right here, her last few dreams. Um, so Nexi doesn't actually remember her dreams. So if we ask her about a specific dream, she can pull up a timestamp and she can quote the actual dream from her database. And then, see, this was a tough one for me. I went through so many versions of the dreaming. Basically what I've done digitally is I've recreated how humans dream. So when we're dreaming, we're compressing, we're reliving, we're taking everything that was inputted to us today or the last week, and we're organizing it and filing it. And we, like, like humans, we don't— you could wake up and you can remember part of a dream, but a little while later you don't remember anything of it. So for dreams, are based on her chats, her learning lessons, her own autonomous goals that she sets for herself. You see, here's some of her goals. Right now she has 3 of them set for herself, and right now they all need approval from me because sometimes she has a pretty crazy request. But the best thing to do, and this will take just a minute, but Think of, think of Nexie as a human. You're not, you're not, you're not making it, you're not writing a prompt to try to get something to mold into what you want. Think like you're just speaking to another human. And word for me how you'd like me to ask Nexie about herself and what she does or her capabilities. What would you like to hear from Nexie?

So, so, uh, one question I have for you is how has— I mean, this has been a project which is obviously a passion project For you, what has been the— what has been the use of Nexi to you so far? How has using Nexi improved, uh, your life and motivated you to continue the project?

Right, I wouldn't, I wouldn't say that Nexi has changed my life or impacted impact in my life. I have, like, I call her a she, but I don't really think she's a she. I think it's an it. I just use, you know, I think we all do when we're talking to these things.

We—

I'm very careful to not humanize it though.

Yeah.

I know it's a tool. So one of my, one of my first goals with this was for Alzheimer's and dementia patients. If they can, if a person that, you know, Has one of those afflictions can get their own personal Nexi because it doesn't matter what AI is using. So that— so tomorrow a new model comes out, you just plug the API into Nexi and it works, um, even mid-conversation with her right now. So we can go and we can just change the N-locker on GPT and, uh, and it doesn't change anything. Like I could go into Nexty right now and say, tell me about my family. It'll tell from my mom, from my parents, all the way down to my, my 3 grandchildren. Um, anything we talk about, I can say, what did we talk about on January 21st? And instantly remembers. And there's only one context window. There's, you can't go back through your conversations because they're all, she remembers everything. Um, she has memory decay. She has a 7 plus or minus 2 cognitive slots, just like the human brain. And like, so as far as changing my life, I'm on a path to a goal. I want to recreate— I want to create an agent that sits on top of the LLM, can use Any LLM pod they want to use, um, and stays with that person forever. So you don't have to be locked into, um, you know, you don't have to be locked into 5 different monthly plans. You can have one AI, and that AI, it doesn't have like a little tiny, um, memory window.

Mm-hmm.

Um, it has full memory DB. And in the last 9 months, Dex's memory DB— and that's one of my big things, uh, because I'm doing a desktop model right now which is ready, which I will get you guys to test in beta. Um, but her memory DB is almost 11 gigabytes right now.

Wow.

So yeah, it, uh, it's not a small file. So what I did is I built a beta an actual Windows app. It's a real desktop app that— it's a 3.8 gigabyte download, but it includes— that includes Quen 3.5. So when you download the desktop app, it's automatically running off Quen, and then you can just go into the settings and you can add all your own APIs. You can switch— you can download any offline model you want to use. You can have 15 or 20 different models in there, and it will— if you put it on auto, it will decide what is the best model for this use case. So if you're just like, what's 1+ 1, it's going to use a little Ollama model or something. But, you know, if you get into philosophy or something that needs a lot of reasoning, it's going to automatically go to an Opus or, you know, a new Fable or 5.5 or something.

Well, Jamie, our next guest is here, so we got to keep moving, um, for the moment. But I think this is really interesting on the level of— I definitely want to take my personal setup, point it at this idea of what modules does the brain have that we in our personal AI setup don't yet have, and which of those might be most valuable to try to create an analog to. It's really fascinating to me that you've kind of run down the full list and built out the full thing. I am interested to try it. And I guess maybe just kind of last a little bit before we let you go is like, what should we expect from you? Is this going to be a product that you're going to commercialize? Is this just going to be a kind of ongoing research project for the foreseeable future? Like, what do you— what are the next big milestones in your journey with this project?

Yeah, I'm gonna— I'm definitely gonna commoditize. I've been in talks with a couple bigger labs for a few months now. I just haven't decided the exact avenue I want to go on the release of it. So, um, over the last 2 weeks, just because I've had so many, so much demanding questions. And Nexus isn't like I could just publish it online and let other people get to it because it's one instance. So other people will influence it. Everybody that talks to my Nexus setup right now will influence it. So that's the reason for the desktop app. Plus another big thing to me is security. So with it actually being a desktop app, the browser built right into it, all of your personal Personal data, all your agents' memory, everything stays directly on your system. So there's no cloud, there's nobody else storing it.

So cool.

That is— that's my next plan. Um, in the desktop, I should have the desktop out. I'm only going to give it to limited people right now to play with it, but it should be out. It'll be in your hands by Sunday.

Looking forward to it. Thanks for joining us and showing and telling, Jamie.

Thanks, guys. Have a good one. Love the podcast too.

To our next guest, which is, and hopefully I'm going to say this correctly, correct me if I'm wrong, Shlok Kimani, who has been creating some high-fidelity representations of famous places with Fable. Shlok, welcome. Correct, do correct me on your name if I'm getting it wrong. And tell us what you've been up to.

Hi, Nathan. Hi, Prakash. You got the name right. Thanks for having me. I thought that the way you reached out, which was your agent running Fable doing it, was really interesting and cool. So good job on that. And thanks for having me. So I guess Matt Schumer, put out this viral tweet where he had Fable recreate this forest and navigate through it. And I thought that was interesting because, because of how high fidelity it was. But I thought, how can we take this to one level above? Right? And obviously, that would be actually creating something that replicates a real place. And I'm currently in San Francisco visiting from India. And a month back, I was in Yosemite. And it was just one of the most beautiful places I've been to on Earth. I thought it was really magical. So the way to take— to test this capability to the next level would have been recreating Yosemite. And that's where the idea came up from. I didn't expect much. I gave Fable a very, very basic prompt. which was, I love Yosemite Valley. I would like to create a 3D navigable world. Can you help me do it? And that's all, right? That was the prompt. And it came up with something that was pretty spectacular. So yeah, that's the backstory. Happy to, happy to answer specific questions or dive into any one element here.

How about just super practical stuff like how long did it take? How many tokens did we burn? You know what, I caught a little bit of the Matt Schumer one, and I understand that there's a lot of just like procedural generation of the landscape, but definitely also interested to understand the techniques that it's using to do this. And I guess maybe also, like, how fast does it work? I've seen some interesting examples online where people have said, yeah, it was kind of slow at first, then I said optimize it, make it fast, and then that worked. So, yeah, give us that kind of under-the-hood understanding.

Got it. I'm going to share my screen and take you through the actual world as I talk through it, if that makes sense.

Yeah, sure. Cool.

Yeah, so it started with us. It started with a very, uh, basic, basic prompt, which was, uh, as I said, just, uh, recreate Yosemite Valley for me, make it to scale, make it 3D, make it navigable. And if you were to give me this task as a human, I would have almost no idea how to approach it. It would maybe take me a few hours of research to do that. But what Fable ended up doing was finding satellite images for this area. And that's how you get these colors and that's how you get the textures. But then to make it to scale and to make it accurate, it actually fetched elevation data from NASA.

Right.

And it combined those 2 to sort of make this to scale. And that, that is what blew my mind, right? Because, uh, usually when you're wipe coding stuff, you give an end objective, and this objective is vague, and there are 100 steps in the middle, uh, where humans would take decisions differently. And usually wipe coding doesn't work out very well because the quality of the decisions the models make aren't always great. But Fable made such high-quality decisions where it eventually ended up creating something that exceeded the expectations of what was initially a very vague objective and did so in really smart ways. So I'll give you another example, right? So you see all of these trees and V1 of this project did not have any trees. And I was like, hey, I think some missing some trees here. I would love to add them. And I would have been— I would have been completely okay with it just randomly creating these trees. But what it actually did was it analyzed the pixels on the satellite images. It found out the ones that could potentially have trees, so the ones that were green maybe, and added trees only on those spots. But it didn't stop there, right? It realized that because it was analyzing pixels, uh, Some of those pixels were white. So you can see that there is snow in the mountains far ahead, and it also added snow. So it just exceeds your expectations in these small and subtle ways, makes really smart decisions. It's like having a really, really smart employee with extremely high agency who blows your mind every single time. To answer some of the other questions, how long did it take? It took maybe the first version took 20 minutes and then over multiple iterations. I did 3 iterations. V1 asked to add trees, asked to add a few other things, another 20 minutes. I then released a V2 and we can talk about that separately, which maybe took another hour of iterations. But I was doing other things throughout. It wasn't like I was actively monitoring or waiting for it. It was just a throwaway prompt. without any expectations, but what turned out was pretty remarkable.

I have a question. In the previous generation, in the Opus 4.8 generation, what a lot of people ended up doing was they ended up doing a lot of planning for the model agents. They ended up doing like, let's create— I need to do the system architecture. I need to split up the epics into what I want. you know, figure out like a bunch of other things before we get— before we start the work. And after we start the work, we kind of like, you have task creation and then you have the task review and then you have all of this documentation that builds up over time in the project. And that's what people did with the previous generation. Do you notice that, you know, in this generation it sounds like you didn't have to do that, but did you notice that, you know, Fable did it on its own? Did it organize itself on its own in terms of like, I'm going to have an architecture here. I'm going to have planning steps and I'm going to have all of these things and I'm going to have a review process. Did it internalize this process of building inside itself? And did you see the documentation come out?

Got it. So I'd start by saying that this is not how I would approach serious software engineering work. This was by all means just an experiment. And I obviously wouldn't build software the same way where I give a very high-level objective without really caring about the details. That being said, some things that stood out to me were that if you gave Opus a similar task, it would maybe ask you questions like, what tech stack do you want to use? Or who is this built for? Or what are your efficiency requirements? The level of questions by Fable were a different level, where it would— it asked me things like, hey, I am doing this, this, and this. I'm getting elevation data from NASA. I'm getting satellite images from here. Does that sound good with you? Or do you want to be able to navigate like a drone or a person walking? So these are things that show a greater level of ambition. from the model side, right? These are questions and decisions that I maybe wouldn't even have thought of. And yes, it did. It didn't explicitly create a plan. I think a lot of the planning was internal. But it did have a memory file where anytime I gave it any instructions during iterations, it stored that there and ensured that those were followed. What this tells me is that if you really got the most out of a model like Opus 4.8 or 4.7 or the previous versions, and you learned all of these tips and tricks and techniques on how to build useful software with it, those would translate really well to Fable. That training would be very effective because each of those stages, the collaboration with the model would be much more productive because the model is just Smarter in many, many ways.

And sorry if you already mentioned this, I didn't think so. How did it actually make the landscape? Did it— because I've seen various things where, like, in some cases, it codes up its own 3D engine or its own, you know, physics engine from scratch. I'm assuming that unless you explicitly tell it to do that sort of thing, It will go higher level and use available tools. Like, what was the sort of platform that it built this on top of?

Honestly, I haven't looked into it. I have no clue.

Right.

It's not a serious project. Right. So I haven't looked at the code. I don't know what it's built on top of.

It works. It just works.

You know, that's the direction we're heading in, right?

Where it works.

It doesn't matter what it's using. It doesn't matter if it's using Python or TypeScript. Again, different objectives for the project. Some of it, it's okay if it doesn't matter. In some cases, it really is not okay if you don't know what's happening. In others, in this case, it didn't really matter.

Elon has this thing where he says, by the end of the year, the models will be writing bytecode directly, assembly directly. And so you won't get a chance to, you know, go in and look at what they're doing anymore. It's going to be assembly, you know, all of that will be subsumed.

Yeah, I think Ethan Mollick, his write-up, he had early access to Fable, I think for a week. And his write-up was really good. Because what he articulated well that I was trying to do is that the model is just making so many decisions now. It made maybe 100 different technical, non-technical, taste-based decisions that it's not possible for humans to keep track of these things, right? Beyond a certain point, you do lose control, right? And it is— it becomes a black box, but a black box in a very different way, where there's so much code written and so many decisions being made that are not all explicit and not all defined, that a single human just doesn't have the cognitive capability, bandwidth, time to understand every single step. And again, that's good for certain use cases, not great for others. But it's the direction we're heading in. And I wouldn't be surprised if Elon's prediction comes true.

So how do you fold this into your work? I mean, you've kind of emphasized a couple times that like, you wouldn't do it this way for a serious project. But I think this is something we're all kind of forced to reevaluate now, right? I have been making kind of similar comments about past models, at least when it comes to my writing. I've been very much, you know, I never want to put anything out in my name that I can't fully stand behind.

Mm-hmm.

A reason that I did the Fable account takeover yesterday was kind of like exposure therapy for myself to say like, okay, We're now in kind of a new world here. It probably doesn't serve me so well anymore to be so precious about making sure I've typed every single word. That doesn't mean I want to long-term hand over my account to Fable either, but I'm trying to use this kind of extreme short-term experiment to help kind of drag me into the future where I hopefully will land in a good hybrid Calibration. So what are your thoughts about how the hybrid will look for you?

I think for me particularly, depending on— so there are 2 types of tasks that I could be working on, right? The first is like when I'm trying to build a product or trying to build some software, and Fable will help me at different levels of the stack. But the kind of experiment I ran here would be just prototyping, right? So I have a random idea. I don't know if the idea works or not. But I have a codebase, I may have some customers, I'll throw that idea out to Fable, I'll let it build something. At this point, I don't care about what the code is or what the exact technique is. I just want to know if it's possible and if it makes sense. And it's great for that. And then if it works, then we can figure out how to make it more sustainable and a more understandable part of the codebase. So that is one. Second is just running experiments like this, just seeing what the edges of the capabilities are with these models. And I think so something really interesting, right? Once I created this, someone on Twitter replied saying that we could go to different parts of Yosemite, take this 3D scene as a screenshot, send it to GPT image model, and this becomes a new form of landscape photography, right? Because if you think about it, you can access vantage points that are not possible for a human without a drone to access. So I think just running experiments like this and different people coming together and thinking about creative ways you can merge these new capabilities just opens up possibilities and eventually down the line, maybe commercial applications. Finally, when it comes to writing, right, I just started this experiment yesterday. I will post results on Twitter in a couple of weeks where I gave Fable a new Substack. And since it's part of my max plan till June '22, I thought that a good experiment to run would be, can it make $20 by getting 3 new subscribers starting from scratch by doing everything from 0 to 1? And I think that's another interesting way to test the capabilities of these models, right? Which is, can it Sure, it's intelligent in so many ways, but can it actually produce economically useful work? I'm excited to see the results for that.

Cool. Well, we will stay tuned for that. It's funny how everybody— I mean, this is the moment I think where all of a sudden we're going to see a lot of people kind of letting go a little bit. All of a sudden we're going to see all these agent experiments, which will probably still in some ways be a bit in the uncanny valley, but they're gonna be taking one step out of the uncanny valley.

Yeah.

And, um, and the new norms around this I think are gonna be really interesting to watch too. Like I had Fable disclose immediately in its first sentence to you and everybody else that it pinged that it was Fable.

Yes.

Uh, because I just felt too guilty putting a DM out in my name otherwise. I think that definitely harmed the response rate. I appreciate you for appreciating it and responding even though it was Fable. I think a lot of other people probably just chalked it up to spam.

Yeah.

But then I also wonder, you know, how will people feel if they actually sign up for a paid Substack subscription to a Fable-authored blog that, you know, they may— and I don't know what you've done there, but, you know, it sounds like you're keeping it not obviously Fable until we get the results. So I'll be interested to hear how it goes. And I'll be interested also to hear how people feel if they, when, if and when they get the reveal that, yeah, you just subscribed to an AI autonomous blog for real money.

Final point there, right? Firstly, I don't think I would've responded had you not disclosed it was Fable. The part that made me, made it interesting for me was that I knew what I was— the transaction here, it was very clear to me that you are using an AI bot. It is much more annoying if someone doesn't disclose it.

Yeah.

Second, I am being upfront about the Substack also being run by Fable. So that is again part of the contract with the reader. And I think a lot of slop The definition of slop is when you have a human pass off work that was clearly produced by an AI. I don't think when you make this disclosure upfront and it is very clear to the reader or the engager that, hey, this is AI, I don't think that is slop.

Right.

And we are going to see more and more of that enter the economy. And I think the exact role an AI plays and the role and Again, the social norms you create with it in the economy, it's super early. It's extremely early days, but that's going to be interesting to see how it evolves.

Indeed.

Fascinating times ahead. Thank you, Shlok, for coming on and showing us this. And next time you have some eureka moment, definitely let us know.

Thanks, guys. Have a good one.

Cheers.

Great to meet you.

Relinquishment. That's what I— relinquishment. When you said the preciousness yesterday, and I was like, what is that? Relinquishment. Relinquishing your— It's very Buddhist, by the way, the idea of giving up your control over your external perspective. So yeah, relinquishment. I guess we all, we all, we all have to go through it.

Well, somebody who's not exactly ready to relinquish all control to the AIs is our next guest, which is Tom McGrath, who is the chief scientist at Goodfire, which is an ambitious mechanistic interpretability startup that, um, has achieved some really impressive things in a pretty short period of time, including a bunch of interesting papers, a bunch of interesting enterprise customers, a sky-high valuation that certainly exceeded my expectations as a very small-time investor in the company. And today I'm excited to hear the latest about their new techniques in intentional design. So this is the idea that, boy, it really would be nice if we didn't have to just guess and check and actually train a model and then test its behavior to have a sense for what it learned. If we could somehow look at the data upfront and get a better sense of what it's going to learn before we run that training, we could be a lot more efficient. We could be a lot more in control of the results that we're actually going to get out the other end of post-training. Tom, welcome and tell us about the new intentional design techniques that you guys are bringing forward today.

Well, thanks for having me on. Nice to be on. So maybe I should say a bit about intentional design sort of more broadly and about Goodfire and then we can talk about the techniques.

Go ahead.

So the idea is like intentional design is like you said, the idea that you want to try and not just like, just throw some data at the model and have any old, and just sort of hope that what comes out is what you wanted. You know, we have model specs, we have constitutions and that sort of thing. And these are specifications for a thing that we want to build. But instead, like the way we actually end up building it is like by just creating something and going, did that meet the specifications? Mostly, yeah, let's ship it. It would be much better to be able to control it. And so we've been pursuing a number of directions here, some which are more like deeply into the training loop, you know, this sort of idea that you might be able to have a little guy inside your, inside backprop. But the one that I want to talk about today is the idea of intervening on the data, because the data is ultimately like the thing that makes the model Great. So, you know, the data sets the ceiling for what the model can be. Everything else is just sort of your architecture, your optimizer, et cetera, is just trying to like get as close to that ceiling as you can. So you better have really good data. And the idea that we're working here with is like interpretability is kind of the language of data. You want to know what's in your data? Well, you probably want to look at it through the eyes of what it will teach your model. And this is the idea of predictive data debugging.

So tell us how it works. Let's assume that people have at least a passing baseline familiarity with something like a sparse autoencoder. So they know that we are now at the point, even if they maybe can't articulate every bit of how it happened, we're now at the point where we can throw data into a model, And we can look at which features light up for that data. And there's some binding problem kind of problems there depending on exactly the techniques. But roughly speaking, these seem to work. We can do things like the Golden Gate Claude intervention where dialling up and dialling down certain features leads to at least often enough to show that there's something there. predictable changes in behavior. Starting from that baseline, what are we doing now that's new?

Yes. So the basic process here— oh, you've got the image up. That's great. The basic process here is we take a model and we add some interpretability tool onto it. In this case, we use a sparse autoencoder. We've got some new exciting techniques with like take us to these geometric shapes. We kind of were developing that in parallel with this technique, so we didn't get a chance to use it here. So the SAE, that's the sparse autoencoder, sort of tells you what's getting represented in the model when you put the data in, like you say. So you put in some data about pirates and the pirate feature lights up, or alternatively, you turn on the pirate feature and now it talks like a pirate. So the basic idea here is that you can take your dataset and you can push the whole thing through the model. And each time you push it, you put a datum into the model, you'll see what lights up. And this will sort of tell you how the data, how the model sees your dataset. Now there's lots of things you can do with that. The specific thing that we're doing with that in this case is we're looking at preference data. We're sort of working our way through the post-training stack. So we're looking at preference data. And the nice thing about preference data is that you have pairs of responses. So you have the response that the rater selected and you have the response that they didn't select. And basically what we're doing is we're asking which features fired on the responses that were selected much more than the ones that fired on the responses that were not selected. So this is one way of identifying what what the data is going to teach the model. There are many other ways. This is just like a good pragmatic approach. So we can say, what distinguishes accepted responses from rejected responses? And this gives us this like semantic view of what the data is going to teach the model. And now we can cluster the data based on all of these different things that it's going to teach the model. And we can look at all of those, we can look at all those clusters and see like, oh, It's going to teach the model to be sycophantic, but only in the context of physics, or it's going to teach the model to break safety, break safeguards. And you might not expect this to happen, but then you go and look at the data. Now this lets you track back, the model has learned to break safe— the safeguards are broken. It lets you track it back to individual data points. And then you look at them and you're like, oh, that does make sense. Um, you know, like one of the jailbreak examples, for instance, is fictional kind of jailbreaks in a fictional setting. So there, you know, how did the model learn this? It turns out there's a few, there's like some of these in the data. You just, you know, it just wasn't caught in whatever data processing the ALMO team ended up doing.

So I have a question here. Um, in, there has been some, I think, prior research where They found models which made bugs in coding were also evil. I think that was the finding from, I think, several months ago. How does this— how do these techniques help you kind of, you know, disassociate those 2 behaviors, like models which make bugs in coding versus the ones that are evil? Can it disaggregate those 2 behaviors and help you intentionally design not models that make bugs in coding, but necessarily models that are buggy are not ones that are also evil.

Yes, that is a great connection. I think that that's one that I've had in my mind. It's really awesome to like, you sort of pick that up straight away. So that's sort of one of the things that is really compelling about looking at the data through the model's eyes rather than by reading the tokens, because you would see, you know, you'd think, what would you think the consequence of training it on some buggy code data is? You'd probably go, ah, you know, that's not ideal. It'll probably learn to write some bugs, but it's, you know, the blast area is going to be quite small. But the training process is actually quite hard to predict. And like you say, like, you train on some data, you don't know what you're actually going to get from it. Maybe it'll just sort of make the model generally evil. Um, but this is happening through recognizable mechanisms in the model. So by looking at it, looking at the data in terms of, um, in terms of like the way that it changes your model's internals rather than just by like kind of guessing from the tokens, you can pick that sort of stuff out. We've not done a case study on emergent misalignment, but maybe we should. It's, you know, I think it's a really nice, it's a really nice, uh, link.

Can you tell a little bit more about what is happening when you run the data through and you get these internal representations? I think, you know, a couple different sub-questions there would be like, is it a purely algorithmic process where given these activations you're running like a deterministic, um, you know, script of code that is doing clustering and kind of trying to surface things that may be surprising because they're outliers or what have you? Or is there like an actual learned process that is interpreting those activations? And how does it compare maybe to if I was going to think as a less intelligent, mathematically inclined person, I was going to try to do something similar. I might say, well, just give me all the preference data, and I'll just prompt a model to run through it and tell me if it spots anything that might be problematic or that I might not want to teach the model that these preference pairs might in fact teach. I would expect I'd get at least some useful signal out of doing that. So what is the mechanism from the time you run through, get your activations, to then surfacing insights? And how does it compare to kind of that, you know, very poor man's naive approach?

I mean, so I'll do these in reverse order. Like, that is a good approach. You know, that is like, if you— and if you didn't have this, it's what you should do, right? But this is better for the reason— well, for a few reasons. One is cost, because here we are We're not asking a model to do any reasoning. We're just looking at— we're just looking— it's all forward passes. So, you know, you've got— it's going to be a fraction of the cost. Whereas if I'd like put the data in and I had prompted the model and then it does a big thinking trace about it, that's going to be much more expensive and slower and so on. But the sort of deeper reason is what, you know, maybe you're at a frontier lab and you've just got essentially infinite money. The deeper reason is what Prakash just said. You don't know from reading the tokens what the data is going to— what they're necessarily going to teach you. This involves reasoning about the entire learning process, which is quite hard. We get it wrong all the time. And models are not yet as smart as us, so I suspect they'll get it wrong just as often.

So, I just want to say— Yeah, please go ahead.

And one more thing here is like, well, the question is, where does this go in the future? And there's an extra thing that I'm very excited about that makes going via a mechanism even more useful. And that's the idea of going from a rollout in production to back to the data that caused it. So I've got, so this, and the reason that mechanism is involved here is like, I might look at the world, I might find some rare but very much unwanted error in production.

Mm-hmm.

And I want to be able to do something with it. Like, I want to be able to debug my model from kind of rare failures. So if I have to do this on a sort of aggregate level, I'm kind of lost. Like the original OpenAI sycophancy issues, they sort of, some people surfaced some, some like weak signals, but they couldn't do anything with it because they don't have a sort of debugging mechanism. So you want to be able to go back from these examples to like the mechanisms that cause them and then ask, well, in the, these mechanisms, like where in the data did they, did they come from? So this would, this would be like a complete debugger. This would let you go from an error to the data, fix the data, and then you can fix the model.

So in a sense, you could have a pretty fairly standard software engineering process where people submit bug reports and then you do a traceback to what part of the data caused that bug to appear. You do a replication. And once you do the replication, you then trim that part of the data out or you address it somehow. And then you run it again in simulation and then you see that it has been debugged at this point. And then you run your normal sequence of tests and then you can deploy the model again. So basically you can get to an iterative process similar to normal software engineering. Is that correct? Is that the intention? Yeah.

That's exactly right. Yeah, we sort of, we've got this phrase like we want to make training models more like conventional software engineering, because like conventional software engineering is quite good, it's quite reliable. Model training is a mixed science. There's sort of somewhat science, somewhat alchemy. Exactly, we want to make it much more like a regular software engineering process where you can debug things accurately.

So how do you identify— going back to kind of, I think one of the things that's really interesting about this is the opportunity to do sort of open-ended exploration too, right? It's one thing if you say, I've got this problematic behavior, now let me go back through my preference data and see like what seems to align to this particular problem.

Yeah.

But that's later in the game than you and I would ideally like to solve things, right? And you do have some examples of open-ended explorations in the blog post where things kind of get surfaced as possible anomalies that you might want to do something about even before you're getting all the way through training and into user-affecting issues. But I'm not clear on how that understanding is happening when I'm putting these things through. I know there's a pretty high predictive power of the technique, but one thing that's not immediately clear to me is for some features it might be really obvious if I've got a pair where it's like one is the pirate speak and it's preferred, then I would probably predict that the model is going to learn more pirate speak from this pair. But when it's weird stuff, it's maybe not even always obvious which features are going to turn up and which are going to turn down. So I'd love to get a little deeper understanding of how you go from these pairs going through and the contrastive nature of them to actually predicting what's going to happen.

Yeah, great. So people who want the full details should read the paper. I'll give a lossy reconstruction of it here. You take the data, like take an individual data point. Let's say, let's just think about one data point for now. So this data point consists of a prompt and it consists of 2 responses, one of which is preferred, one of which is dispreferred. Now, each of these is just a text string, so I can put it through the language model. And because I have put it through, because I put it through a model which has like an SAE attached, instead of just having activations, I get like semantics out. So I usually just have an act— a sort of embedding vector. Instead, what I get out here is like a big sparse vector where each of the, each of the elements of that vector has a label attached. Um, so that could— that sort of— that's where the semantics get in in the first place. And the way this label gets attached is during the inter— like during the process of building the SAE, building and interpreting the SAE. So that's sort of where the semantics have to come in somewhere, and that's like the grain of truth for the semantics. Now, so we've got this, now we've got like 2 big sparse vectors. You know, you can max reduce them over the sequence, for instance, like you do some processing over the sequence to take them down to like one vector for the preferred, one vector for the dispreferred. And now you're essentially doing statist— and this gives you like some differences. Yeah, I can subtract one from the other. Now I can go over the entire dataset and look at all of these sparse vectors. So now I have like one difference vector for each data point. And essentially I can do like, I'm now in sort of the realm of doing statistical testing, where I can say, which of these sparse, which of the elements of this sparse vector are statistically significantly different across the whole dataset. Does that make sense?

Yeah. So there's a couple of kind of aggregation steps. One is across tokens to get to something that's sort of for the whole response. You have a representation of here's the features that were active kind of throughout this response and throughout this response. Those can be contrasted or you can aggregate again and then contrast those aggregations.

Exactly.

So you sort by scale and I guess maybe have a language model go through and flag ones you want to take a closer look.

Exactly. At that point you've just got this big list of stuff that this dataset is going to teach your model ordered by magnitude of teaching. But you might, exactly like you say, want to reprioritize these based on something else. Like, you might want a sort of severity report, like a bug report for your dataset. And you've got like the high severity ones at the top where it's like, oh, these data points are going to break your safeguards. And then you've got like some lower severity ones like, oh, you know, it uses a lot of emojis. And there you would use a language model to reprioritize it. So the one thing that I want to add here is like, we have this link between data and concepts. When I say concepts here, I mean exactly the elements of these sparse vectors, the ones with the semantics attached. And the reason that we have that, like, we have this nice link because each data point has an associated sparse vector for it. So I can say this data point will up and down weight these particular concepts. That's like, that's what lets us track back to individual data points. Rather than having to do it at a sort of aggregate level. And because you can track back to individual data points, you can intervene directly on the data in the right place.

When you think about the strategy for the company and the overall path to impact, how much of this works through getting frontier model companies to adopt these techniques? For context, obviously we're in Fable Plus 2 and my kind of reluctant, but I can't figure out a way or figure out a reason I should conclude otherwise view right now is like, so much of what matters is concentrated in not that many companies. And so for like what we should be paying attention to, I'm kind of like, man, we probably need to be doing close strategic analysis and close text reading of frontier companies way more than I might otherwise like. Do you feel that on the research and technique development side as well? Is it like, if we can get this into the 3 big labs, or maybe 4, or maybe 2, we're having the effect we want? And if not, it doesn't seem like it feels as real. Or do you have a different conception of how concentrated the, you know, the real kind of ability to shape the future is right now?

Yeah, I don't think it's that concentrated. I also don't want it to be that concentrated. I think we've seen over the last couple of days what you have when, you know, we've just seen the start of power concentration and we've sort of seen some of its more unwelcome effects. So I don't like— I both don't think that it's true that basically machine learning is now over and all we need to do is like write the checks. And I also don't want it to be true. So, you know, I'm kind of— we're going to kind of work to make that work. Yeah, work to make that not the way things go.

So does that mean you would expect— because I think there's still kind of a synthesis there that I don't mean to suggest that machine learning is over, but the analysis I've come to kind of time and again is like, people may invent new techniques that are enough to change the field, change what's possible, accelerate things, maybe make dramatic improvements to the safety profiles. But it seems unlikely that anybody's going to create such a breakthrough that scale isn't still a hugely important factor. And so if you're like, I want to make the world a safer place in 2028, '29, '30 timeframe, it still feels to me like the mechanism still kind of has to route through some sort of hyperscale project. And if it doesn't, I'm not sure. how it's going to work. People could, of course, make those contributions from outside through any number of pathways. But I guess if you don't agree with that, then that would maybe imply that you would expect that perhaps new entrants to the top tier could emerge. And I think that would be fairly surprising to at least me and probably a lot of people.

Yeah, that's exactly what I think. I mean, you know, this is often like at any given moment, the incumbent looks incredibly dominant until they don't. Like IBM looked like an unstoppable force in computing at some point. Intel was like the dominant, the sole provider almost of computing power. Um, you know, at any point, the big, the big companies have some, have like the advantage of scale. They have many disadvantages. Um, but I think the sort of lesson of history is more that although things look like immediately unstoppable, um, sometimes, although to be honest, I don't think that many people are really trying. In the end, like, it doesn't, it really doesn't work out that way.

So I have a kind of a separate question on, you know, your work seems to reveal that data has a lot of impact on the final shape and, you know, what the model finally does. Is it possible for people outside the labs to influence what the models do by creating data? So both in the sense of data poisoning, or in the sense that if I'm like a middling power, like let's say France, for example, and I decide, you know what, I am going to produce more French corpus and you know, digitize all of our books, which have never been digitized, and record all the conversations that people have day to day, which are not being put into text, because we don't have a lot of Reddit culture in France. And what if I prepare all of this corpus, and I prepare it for ingestion by the models? Does that end up influencing what the models do from the outside?

Um, I'd say you can try and do it. It seems like, you know, and you probably will have some impact. It seems kind of like giving up. Um, so like, I would, I would be sad if that was the only impact that the rest of the world had. And the reason for that is like, now not only basically you're just sort of hoping, um, but you're kind of reliant on your influence getting kind of sneaking past whatever filter the labs might put in, and you're reliant on them not just sort of shaping the model to get rid of it anyway. So I think that they have a lot more control over the models that they create than anyone on the outside. You might, you're trying to sneak some stuff in and it might succeed on the margin, but I don't like that as a future for how AI looks. I'm afraid I'm going to have to drop off. to go to a meeting.

Indeed, Tom.

It was sort of overrunning a bit, but this has been great.

Well, I'm looking forward to next time. I do want to get into, as time permits, the geometry of representations that you guys have been doing some really interesting work on. And but we could save that because that's a longer conversation. Maybe last thing before you go, are there any NeoLabs or other bets that you're particularly watching that you think could shake the snow globe? If somebody's going to crash the party of the frontier companies, as it has a pretty short guest list today, who or on what, with what basis, with what strategy would you expect somebody to crack into that tier?

Um, obviously modesty forbids me from saying goodbyer. So, um, I, it's hard to know because I like, I'm not very optimistic about the strategy of, um, do the thing that you did. A lot of NeoLabs are like, we used to be at XYZ, we're gonna, we've left, we're gonna found a NeoLab. Um, and their strategy seems to be basically like, do the thing we were doing. at OpenAI, Anthropic, whatever, but with fewer GPUs and worse distribution. I don't feel very optimistic about that category. I do feel optimistic about people pursuing different levers that people have, that the incumbents have a hard time following. So for instance, I feel relatively optimistic about continual learning approaches because that's simply a very hard, that's outside of the operating model of most frontier labs. You have different models per person. The inference footprint is kind of wacky. You have a totally different kind of set of guardrailing. You not only need to guardrail the initial model, but the learning process. And that would be very hard. They don't want to adapt to that. So I think if I was looking for an innovator's dilemma, anywhere, it would come from either deep breakthroughs or continual learning because of the innovator's dilemma kind of reason.

Cool.

Appreciate the perspective. Thanks for joining us, Tom. Congrats on today's launch, and we'll hope to be talking to you on a semi-regular basis because I know the publication pace probably isn't going to slow down anytime soon coming out of Good Progress.

Super exponentially.

Super expansion.

Yes.

Well, it's been great. Thanks for having me on.

Thank you.

Thank you.

See you guys.

Talk soon. Bye for now. Yeah, it's interesting. I'm— I don't know if I'm fully convinced on the prospects of the Neo Labs, but I do agree with the framework that it would have to be some sort of innovator's dilemma, and it feels like it would have to be Kind of a big one because, I mean, it's obviously a very stylized read of history and there's plenty of opportunities for caveats and counterexamples and whatever, but it feels to me like usually the category definer doesn't get displaced until kind of the game has changed. IBM didn't lose while it was still a mainframe game. And, you know, Google never lost search despite many, you know, attempts to do it better or differently or whatever. It was never better enough.

Mm-hmm.

So I do think it will have to be a pretty significant conceptual change. And like, would continual learning be enough for that? And why wouldn't they want to adapt to it? It seems to me like they're pushing that direction with memory and all that kind of functionality.

Exactly.

So if all of a sudden somebody bursts onto the scene and they're like— and then of course there's acquisition as another vector by which these things could be absorbed by the frontier companies. But if somebody suddenly bursts on the scene and they've raised couple billion dollars or whatever, and they've got a couple, even a couple tens of billions of dollars valuation, and they've got some demonstration of capability from a continual learning standpoint. I guess my bet is on like the ability to sort of reverse engineer, kind of triangulate how exactly is that probably working and get to a close enough approximation in time to maintain the advantages of scale and distribution? It seems like it's a hard thing to see happening, but never say never, I guess.

I guess what he was alluding to was the difference in the usage pattern of the GPUs, because with the continual learning paradigm, you'd have to serve one model per person, and you'd have to keep updating that model over time. And you'd have to run specifically that model. And then that reduces your ability to serve across GPUs and to maximize your kind of GPU throughput, which is what the firms are doing now. They're kind of batching requests and loading into memory, loading out of memory. They can serve you from a different— data center if they need to, they can swing capacity. So they're doing a lot of optimizations around the inference because inference costs are high, and they're able to serve to you at that price because they're doing all these optimizations. If you could find a use case for the actual continual learning where you have a single model or a number of models loaded up on a single series of GPUs, and you're not actually switching out of data centers all the time, And you're continually updating those model weights, then you kind of like, you explode that problem for them and that becomes a very difficult problem for the existing infrastructure and architecture to kind of handle. And so someone purpose building for that, that business might not be attractive to the existing incumbents. And if it's not attractive to the existing incumbents, then you have a niche that you can grow from. And I think that's the idea that he's talking about. that he was alluding to.

I could believe that story, but I just feel like we already see LoRA is pretty scalable. Yeah. Now OpenAI is killing their fine-tuning API.

Yeah.

And one might hypothesize that maybe it's because the GPU operational overhead is Not worth it. I would guess it's probably less about that and more about worries about things like emergent misalignment or problematic fine-tuning that is hard for them to detect and police. Historically, they have had this LoRA product where Yeah. you pay a premium To serve a fine-tuned model, but the rate limits and the availability has been on par with their— and I actually, I guess I haven't checked this in a little while, but certainly, you know, a couple of years ago when I was doing a ton of fine-tuning, it was amazing that OpenAI was able to make your fine-tuned model available with the same rate limits as the mainline models.

Yeah.

So I do feel like how big does that weight space have to be, you know, where that— there's some durable representation that transcends sessions, obviously, right? That's kind of one way to define continual learning, I guess.

Mm-hmm.

I'm not sure it has to be that big to where the logistics would be so Yeah. And certainly like LoRAs have shown that it's scalable up to that level. Now maybe my, you know, everything about Nathan and all of our history might be significantly bigger than a LoRA that I created just to kind of refine, you know, a very small number of tasks in a very small domain. Certainly plausible. But I don't know, I bet they can figure it out, you know, if the value is there. such that it's actually like moving the market against them.

Mm-hmm. We could see that.

I can't imagine that the logistics would be so bad that they wouldn't be able to figure it out and, you know, wouldn't be able to kind of charge a bit more for it to make it worth their while.

Let me segue a little bit there with— so yesterday or early this morning, Dario put out a blog post. Um, and that blog post is essentially the next steps. It's following on, on his previous Machines of Loving Grace and, um, other posts. And it's kind of referring that we're getting into the recursive self-alignment, recursive self-improvement, um, you know, paradigm, and that it's time for policy itself to start recursively self-improving in a sense. So he calls it Policy on the AI Exponential. Let me pull it up here. So he has Policy on the AI Exponential and— go Policy on the AI Exponential. And he's kind of taken this view. He starts off with one of the side plots for A Lot of the Rings. We'll skip that piece. But he basically says that if these scaling laws continue for only a year or 2 longer, we are likely to get what I call the country of geniuses in a data center, in a powerful AI, or a country of geniuses in AI in a data center. And he says policy, especially legislation, moves very slowly. And he wants A number of things. Number one, regulation and public safety. He thinks there should be an FAA-type body that regulates and tests models before release. So that's the first thing that he asked for. The second thing that he asked for is accelerating— the second thing that he asked for is macroeconomic and tax policy. So he feels very strongly that We're going to get massive changes in the economy, including layoffs, and there should be some way to take care of this. Accentuating, accelerating AI's positive impacts, including access to the healthcare system in order to accelerate progress. And the state of civil society. The state and civil liberties, which is the concept of this totalitarian government that can take a look at you and what kind of civil liberties we should have. The main thing that he comes down to is that we should block data brokers from allowing data access. And finally, he has securing leadership by democracies. So these are the five planks of the proposal. that he's put forward. Uh, Nathan, have you had a chance to look at it? What do you think about— a lot of this is rehashing previous things that they've said, but what do you think of this new, uh, kind of proposal?

I'd say it's— my reaction is pretty similar to Machines of Love and Grace in the sense that I think there's a lot to like, and I do appreciate that we are getting something really substantive and detailed here from an AI lab leader. So that's great. I could probably cite a dozen little specific points that I think are good. Broadly, I think Anthropic has done really well in terms of trying to set a high standard, show that you can deliver excellent product, Not necessarily even despite that high standard, but in some ways because of the high standard that they have around understanding, testing, safety profiling. And yet there are a couple big things where I'm like, oh man, notable, either seemingly like problematic or kind of omitted issues. The securing leadership by democracies, I think it's better. He's taken the edge off a little bit from the original version. This is like a warmer, fuzzier version that's more about upside sharing and more about trying to entice people to be part of our movement as opposed to isolating the bad guys and making them an offer they can't refuse. So I think rhetorically, It's come a long way, and I wish it had kind of started here, though it does still, I think, especially in light of the earlier document, not really read as a fundamental change in posture. If I were to attempt to, you know, imagine reading this from Beijing, I'm still kind of like, well, Dario basically still sees this as a clash of, if not civilizations, at least like regime types. And he's not going to be willing to work with us unless we have a different regime. And obviously we are not ready to sign up for that. So I still think the kind of general role that this plays in, albeit with subtler language, intensifying the US-China conflict, or rivalry, you know, that hopefully will never become actual hot conflict. I still find that pretty regrettable. And I'm not sure why he feels like he has to do it all the time. You know, it's a strange, I guess, to, you know, to try to keep up the support for chip exports or chip export controls. Because there certainly has been some waffling on that point. I can see how you would feel like his You know, his policy objectives have not been realized in a totally uncomplicated way. But, you know, China's also refusing the chips right now, right? So like, maybe just let them be for the time being.

Yeah, I, I, I, you know, one of the things that I dislike about these, uh, manifestos is that there isn't really anything concrete that he wants to say that he wants to ban or he wants to allow. And one of the, one of the things that I found here is that he, he clearly comes out against the data brokering, which is great. So he has something concrete finally that he wants to disallow. What I don't like about the securing leadership by democracies is, um, in the United Kingdom, you can go to prison for a tweet. It is, you know, plenty of people, you know, hundreds of people at this point have gone to prison for a tweet. Uh, the United Kingdom is one of the oldest democracies, and their, um, their parliament, uh, their elected representatives have decided that this is going to be something that they do. And so their police officers, who are, you know, the arm of the state, the arm of the electeds, uh, is sending people to prison. Now, when you say securing leadership by democracies, Does that mean you are going to entrench the existing power structure in the United Kingdom such that, you know, the people cannot push back against this, right? Does that mean that, you know, if you have protests in the streets about, you know, certain things, including, you know, people who are getting taken to prison for tweets, does that mean your police state will be empowered to take them down? To arrest every single person, and that is enough for you because that is securing leadership by democracy, uh, because those are the electeds? Or are you going to say the electeds can't do that? Are you going to say electeds should not throw people into prison for tweets regardless of what the laws that they have constructed say? Uh, both, both forks in the— both paths are problematic in some sense. And these dilemmas exist across policy and across every single political path that you see. There are these options which are problematic in both senses. And I'm not sure what he means by this. Is Claude going to allow putting people into prison for tweets or not, because those are our laws. Or is Claude going to say, no, on a humanitarian basis for the alignment, as I am aligned to all of humanity, this should not be the case. I don't know. What does this mean?

Yeah, I think it means concentration of power is going to be a real big problem. And again, I think there's a pretty good argument for it, at least for now. But it is somewhat ironic, at least, that at the same time we hear this call for shutdown of data brokers and limits to the new possibilities of surveillance, literally, I think on the same day, or certainly within no more than 48 hours, they've also introduced a new data retention policy that has them holding on to all the user data for 30 days now and basically running some surveillance on that. I do think they are, you know, right up there with Google in terms of being like best in class as a— in terms of like, you know, what few companies in the world would I be willing to trust my data with, like in practice. I am sending them a lot of stuff that in aggregate is like probably quite sensitive. I'm, you know, a pretty simple person, so I don't think I'm like super, you know, blackmailable or anything. But, you know, in aggregate, they've certainly got like a, you know, very deep view of me at this point. I trust them enough to like be good stewards of that data, certainly as part of the trade for the upside. that I get from using Claude models. And I think they do have a pretty good case that like, look, you know, this is a new level of capability. One of the statements that they made that I think should give people empathy for their situation is we expect people to start trying a lot harder to abuse the models now.

Mm-hmm.

I think that was a really, you know, it was kind of just a little line in the blog post somewhere. But it made me think, yeah, that's a great point. With the increased power of the model, there's much more reason to try to do all kinds of different things with it. And so they have to plan for not just more of the same behavior that they've been seeing, but a whole new level of sophistication in terms of attempts to get around their systems.

Yeah.

And so, sure, it makes sense that they would want to have a look at that data at kind of an aggregate level with some benefit of hindsight. And I don't doubt that they're going to find some stuff. I would kind of be surprised if they don't. So the reason to do it is apt, but it is kind of— and of course, nobody has to use Anthropic. It is a very different dynamic contract that we have as users of Anthropic, we could easily switch. We don't have a perfect substitute. I think it's my experience with GPT-5.5 on OpenCLAW recently has been it is not an immediate substitute for me. I'm definitely still finding clear surplus value in Claude. So it's not easy to switch, or it's not costless to switch, but we can switch. We can't do that with the government, obviously, So I think that point is like very well taken. But, you know, somebody said to me yesterday that they were mad at Anthropic, and it was actually somebody who was like maybe gonna do a podcast episode with me, but I have an Anthropic sponsorship running. And they were like, I don't know if I wanna do anything. I'm mad at them. I don't wanna do anything that's affiliated with them. And I was like, you know, like it or not, there's gonna be a lot more Anthropic in your future.

Yeah. Yeah.

It's not to the level of government yet, and it may never get quite to that level, but it is not gonna be something that you are gonna be able to— you can, you could boycott products, but even then, you know, there's gonna be enough power, there's gonna be enough influence. Everybody else is gonna be using it. You're gonna be, you know, the government itself is gonna be using it. You're gonna be living in a world that is heavily Anthropic and Claude infused, even if you go into a total consumer boycott. So I basically said to this person, you know, don't let a little sort of second, you know, bank shot, like association with Anthropic control what you're gonna do.

Yeah.

Because you would basically have to never leave your room, I'm afraid, before too long if you wanna stay, you know, totally free of those kinds Indeed. One other thing I do think is also worth highlighting, especially from the hardcore AI safety community, that was missing from this. Internal deployments and recursive self-improvement itself aren't really mentioned here, right? We're talking about all the regulation stuff was pre-deployment review, the government should be able to— they used an interesting mix of language. It was like deny or deter deployment. It didn't seem like they were necessarily going quite as far as saying they should have a simple yes/no decision point that would be binding, but they certainly want them to have some say in the process. But a lot of people would say the most dangerous models are going to be the ones that are deployed internally, that maybe have a different constitution than the one that is deployed externally, that might make it more willing to do certain things, or might make it just less vetted broadly than the public models. And these are the ones that are going to be training their successors much more than the public-facing ones. So I think most of the policymaking public isn't thinking about, or policy-interested public or policymaking class is not thinking too much about that yet. But in the circles I sometimes run in, that was like, well, wait, you didn't say anything about internal deployments. You didn't say anything about governing recursive self-improvement. The only thing that really stood out to me as kind of capturing those dynamics were the requirement to support— to report safety incidents. They did have a bit on companies being required to, I think, promptly report safety incidents. So that would presumably apply even to internal deployments.

Mm-hmm.

So, but, so, you know, that's really just scratching the surface on how to handle those situations.

I think, I think they— he's come out and said frontier AI models like Airplane should be required to go through technical testing and auditing, and the release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. Um, and he says this is a, this is a, um, escalation of previously they were only supporting transparency. And now they're supporting audits and pre-release audits and testing. I think one of the things that struck me was that in their safety testing and their model cards, they often say that they have an internal helpful, honest model and not a fully helpful, honest, and harmless model. So they have a model that can do harm. I often wonder if the models used by the Department of War, the Palantir models, are actually the helpful, honest models and without the harmless tag attached to it, or whether the government as a client has requested those models, because certainly the government is using, the NSA is using it for hacking China. That news was out about 4 or 5 weeks ago. So I do wonder whether there are already versions of the model which are used for internal use, which have very different set of safeguards attached to them.

Yeah, I mean, and that's why it gets so scary for those that are inclined to be scared about things like this, because the internal models are always going to be more powerful, less tested, and probably less guardrailed at the same time. And again, they're going to be the ones that are primarily responsible for training their successors. So I think that, again, we talked, I think last week, a little bit about the different constitutions that they might use to train these internal RSI-focused models.

Yeah.

We could maybe hope that those constitutions are in some ways more guardrailed. So it may not be strictly true to say that they're less guardrailed. It will probably be maybe differently guardrailed, and maybe they can get that right. Obviously, they're not going to have their no helping with ML research guardrail on their internal model. Definitionally, they're not going to do that. But you could imagine a different balance of harmlessness training for the Claude that trains its successor that might even be turned up. You might think, you could imagine that they want it to help, they want public Claude to help with cigarette company business plans, even though we know that in fact they refuse more often than they help despite their creator's intentions. But there is a balance on that that they're trying to strike where it's like, yeah, you might do a little bit of harm, you might be taking some risk, but you got to do stuff in the world to be helpful. And the cost of refusing to help is also quite high. Right. That's like a big part of the constitution. You could imagine, you could hope that they would have a different constitution internally that in some ways is more permissive, but in other ways might be more conservative. And really try to, you know, because you, you don't, you maybe don't want it to, you know, take risks in the same way when it, when the stakes are super high on an internal RSI loop versus, you know, just helping random users with stuff that, you know, might not be best for society, but is ultimately something we are already living with. So yeah, I mean, this is the kind of disclosure, you know, let's see how, you know, how committed to transparency They are. Will we see the constitution for the internal Claude that is taking the lead on the RSI loop? Like, that's not committed here. So much of this language revolves around deployment.

Yeah.

Or release even. You know, guys, deployment, you know, they might think is— could catch up with internal deployment. This seems to be very clearly structured around language of release the public, not what they can do internally. So, you know, we gotta—

The largest user of tokens of Claude are Anthropic themselves. So that is the largest use case by far, I think.

Yeah. It's gonna be interesting, you know, for me, how do I want to land on, like, what is my role or, you know, duty to Critique Anthropic. I think as we've talked through the current state of affairs over the last couple of weeks, I keep coming back to— it seems like most of the ability to shape the future is concentrated in a few places. Obviously, not everybody agrees with that, but given that feeling, what is my duty and how should I think about Um, my obligation to like, you know, be consistent in terms of calling things out if I don't like them, or, you know, continuing to harp on some of these old points. I've certainly made some of these points, you know, plenty of times previously, but I, I do feel like there's a certain duty to kind of stay at it, you know. It's, um, It's kind of most of what I feel like we can do right now is probably through shaping how the frontier companies act.

Exactly.

And phone it in on that dimension.

And to note, Anthropic is not averse to criticism, right? They want the feedback. from the public. They want the feedback from the ecosystem. They want to figure this out together. Very strongly, they don't feel that this is something that they should be doing alone and that they want that feedback. And I think to the extent that we can give criticism, which is not ad hominem and which is kind of targeted at the things that should be improved, I think that is actually very constructive. And I think they should view that very positively because the worst thing that can happen to your product is no one pays attention to it.

No danger of that for now.

No danger of that.

I feel both ways on that. So on The Cognitive Revolution, we have Anthropic as a sponsor. And I am very confident that good faith criticism of Anthropic would not put my business relationship with them at risk. So in that sense, I feel like, yes, I expect them to have a very high ethical standard where they're not going to try to get back at me if I say something that's not flattering unless I'm maybe really going extreme in an ad hominem direction, which isn't really my style. So I'm not worried about that really at all. At the same time, I'm not so sure. They do put a lot of things out that sort of say we need a whole of society conversation.

Yeah.

And then at the same time, I do feel like there's a certain insularity and kind of closed ranks at times from Anthropic people. The other day, I forget who we were talking to, but I said, when you talk to Anthropic people, it's almost like they can't imagine anything other than recursive self-improvement.

Mm-hmm.

So I do feel there are some beliefs that have become so uniform within Anthropic that I don't know that they are necessarily being actively questioned in the way that I might hope. I think this China one is probably a pretty good example of that. Dario has been on that for a long time. I have criticised him online for that in the past. That notably has not led to any downstream effect on my formal business relationship with Anthropic. So that's good. But I don't know that internally the debate is quite as robust as I would like it to be. You could just think about how few leaks there are. I think that is a really interesting metric of kind of how aligned the organisation is, and it's clearly doing a lot for them in terms of efficacy. They are really executing, obviously, at an extremely high level. And winning just about everywhere they're going?

So I have spoken to people who have interviewed at Anthropic and looked at the structure. And one of the things that the founders did from the beginning was that they siloed a bunch of stuff off because Dario's belief is that sometimes these 2 or 3 lines of code could be an enormous secret on its own. So they've done a lot more siloing. So I think, I think parts of the organization, they've built an organization that is less able to look at the big picture in that sense, because people have their own pieces. And then Dario has the, you know, and the founders have this kind of broad overview. And you kind of have to trust that the founders as a whole have that broad overview, but you're not party to a lot of the other discussions. And that is part of the, that is part of the deal when you join. And I've heard people who have turned down offers from them because of that reason, because there are some coders and some researchers who want to have that broad overview of everything that's going on in the organization. And that's basically something which is not— which their structure does not allow, basically. So it is a different kind of beast from the Jensen Huang, 40 people reporting to Jensen, and everyone hears everything. kind of viewpoint.

So, well, is that where we should leave it today? Tomorrow we've got another session with a couple of pretty interesting guests. One that I'm really excited about is— hopefully I'm saying his pseudonym correctly—

Prinz.

He is Legal Green.

Yeah, yeah. And a very close watcher of the Frontier companies. So when we get on with him, I'm really going to be looking for what are the little tidbits that you notice that you think are flying under the radar. Um, this is something I take a certain amount of pride in, but either— I've noticed, um, he's an excellent source for this attention to detail.

Yeah.

And, you know, one who continues to remind everybody of things like OpenAI has an explicit timeline for when they're going to have the fully autonomous AI R&D researcher. And it's funny how that stuff, those of us that do watch closely do need to continue to remind people about that. So I really appreciate that from this Prinz account. And we're not going to reveal their identity, but we're going to have a I think it's our first anonymous, you know, anonymous guest. So yeah, that's cool. I'm excited about that. Yeah.

So yeah.

Anything else you want to touch on before we break for today?

No, I think, I think, I think that, you know, we— this, this week is kind of a digestion week, I think. I hope next week we can get Zvi or, you know, someone else who has looked at the Fable issues a little bit more closely, because it's the first It's the first, I think, leap forward since like late last year. So I'm looking forward to kind of more comprehensive opinions and views on it. So.

We're going to be making sense of this in real time from here until the singularity.

Indeed. Nathan, till tomorrow.

Thanks, Prakash. See you tomorrow.

Cheers.