Zvi Mowshowitz on Longer Timelines, RL-induced Doom, and Why China is Refusing H20s

Hello, and welcome back to The Cognitive Revolution. Today, for a record 10th time, my friend Zvi Mashowitz returns for another wide-ranging conversation about the state of AI as we head into the final months of 2025. I assume that Zvi needs no introduction, but for anyone who's somehow not already familiar with his work, he writes the essential blog Don't Worry About the Vase, where he chronicles AI developments and offers a breadth and depth of analysis you won't find anywhere else. We begin today with a discussion of why so many of the sharpest minds in the AI space, including Zvi, are now projecting somewhat longer timelines to AGI, despite the fact that GPT-5 is now generally understood to be right on trend, and that multiple model developers achieved an IMO gold medal this summer, which would have seemed miraculous just a few years ago. He also explains why his p-doom is, if anything, a bit higher than last time we talked, highlighting that the US government seems increasingly captured by commercial interests, and also that reinforcement learning appears to make models less aligned in fundamental ways. This leads into a fascinating discussion of why Claude 3 Opus still seems to have been uniquely durably aligned, why we should believe that the training techniques that imbued later Claude models with higher levels of agency come with their own alignment compromises, and why we should never attempt to supplement outcome-focused reinforcement learning with techniques meant to suppress unwanted behaviors in a model's chain of thought or internal states. Namely, that while such techniques can make things better in the immediate term, they risk teaching models to hide their true reasoning while still incentivizing the same bad behavior. Beyond that, I of course had to get Zvi's latest live players analysis, including whether any Chinese companies remain live players in the wake of Beijing's decision to refuse H20s, whether there's any rational basis for that decision, And also whether or not xAI might derive a special advantage from access to the steady stream of challenging problems that engineers are solving at Elon's other hard tech companies. Toward the end, we compare notes from our parallel experiences as recommenders for the AI safety-focused Survival and Flourishing Fund, with the main conclusion being that the AI safety sector has many more worthy projects than current resources can support, which means that the world could benefit tremendously from new donors getting involved. If you needed one more reason to subscribe to Zvi's blog, he's planning a more comprehensive write-up of AI safety giving opportunities later this year. Finally, as always, I asked Zvi what he sees as especially virtuous to do now. And since say what you really think was a big part of the answer, I will note that while Zvi and I tend to agree much more often than not, there were a few moments in this conversation, particularly when it comes to the US decision to go ahead and sell H20s to China, where I either don't agree with his conclusions or am at the very least extremely uncertain. Nevertheless, considering that we were already running super long and that regular listeners have heard my takes on other episodes, I didn't bother to debate those points on this particular occasion. With that, I hope you enjoy this opportunity to pick one of the brightest and most informed human minds in the AI space. This is... The great Zvi Moskowitz. Zvi Moskowitz, welcome back to The Cognitive Revolution.

Thanks. Great to be here.

Always exciting. Let's start with timelines today. I am a little confused about the way in which people seem to be updating over the course of this summer of 2025. We've had GPT-5, obviously. where I would say it's safe to say the launch was not exactly super smooth, and people were a little disillusioned with that at first. Now the dust has settled, and it seems like people have kind of mostly come around to it is actually a good model, and it's basically on trend. It's actually still a little bit above the all-important metre task length curve.

It's on the very shift upward, right, of that graph.

the 4-month doubling instead of the 7-month doubling. And also something that seemed really important to me that happened this summer is, of course, we got IMO Gold and we got a very close to number 1 finish in the world, although it ended up number 2 on competitive programming competition. And yet with all these things, people, including really smart people, I'm not talking about you know, denialists here, but like, you know, Ryan Greenblatt and Daniel Toccatello type people who are as plugged in as you get. And, you know, I would say smarter than me. You can evaluate for yourself. Timelines seem to be lengthening. So have you updated your timelines, if at all? And what do you make of the seemingly on trend or even maybe a little ahead of schedule events? that are still cashing out to people as longer timelines overall?

I would say mine have also gotten modestly longer on net during that period for a variety of factors. The biggest one is that very large jumps in capability on news really shorten timelines. And if you don't get big new developments, you don't get big new paradigm developments like reasoning models, you don't get big graph jumps off the curve, stuff like that, then that was a lot of where the really fast timelines come from. So you're cutting that off. Whereas none of the things that we did see, while they were on trend, represented anything that we had that much doubt was going to show up. even if it wasn't going to show up quite on that exact day. Nothing was particularly impressive given what had already happened. These things are ahead of trends from a few years ago, for sure. The IMO Gold Medal in particular sounds more impressive than it is because this year's Gold Medal was especially amenable to LLMs in the sense that normally the IMO has fixed problems. So day 1 is problem 2, 3. Day 2 is 4, 5, 6. Problems 3 and 6 are supposed to be incredibly hard. The idea is that a significant number of kids will get 1, 2, 4, or 5, but 3 and 6 will be a struggle. If you manage to crack either of those problems, you're doing really well. If you manage to crack both of them, you get a gold medal. You do really well.

Yeah.

So what happened this year was problem 3 wasn't that hard. This is just a weird quirk of the schedule of the IMO. It has nothing to do with AI. But only problem 6 was hard in the IMO hard sense. And none of the LLMs got points on 6. 6, they just all flopped. But 3 was evidently solvable. And so the LLMs all solved it. But the fact that that happened to be— but the fact that 7 points on each of the first 5 problems was exactly enough for IMO gold.

Right.

Right, just is not screwing up any of those problems at all. Where's this exact threshold? If the threshold is 1 point higher, no one gets gold. If 3 had been harder, like 6, probably none of them come that close to gold. So it's not clear that we actually got gold in a meaningful sense this year. And we were already reasonably close to gold from previous news. So the fact that we got this exact gold from— 2 or more models wasn't actually that big of a deal, which is why nobody was freaking out once they saw the details that much who was familiar with the state of the art, as it were. It's still true that if you look back to things in 2020, 2021, only the most radical people who were extremely AI-pilled, who were predicting very rapid progress, would have called that happening in 2025. And in fact, did call that happening in 2025. on the extreme right tail. But by 2024, this would not have been much of a surprise. So, you know, given it didn't crack Problem 6, this wasn't that scary on the margin. The programming results, again, if you dive into the details, it's really, really impressive, but not in a surprising way. And if you look at the details of the competition, you— have you read the write-up from AGI Pick 1?

Superficially. So yeah, take me through it.

Yeah, I read it. And so essentially what happened was OpenAI's model was very good at jumping to a very good solution almost immediately and had a lot of gains pretty fast, but was not doing a good job of iterating, was not doing a good job of then innovating past that, was not doing a good job of making conceptual progress. Past that point. And so over time, he was able to essentially plan ahead and open up a bigger lead and get into a pretty comfortable position by the end of it. And if the competition had gone on longer, probably OpenAI's position would've slipped below second. Again, it's very promising. It's very impressive. Previous years, you give that problem set to the AI, it doesn't get second place. It doesn't do particularly well. But again, it's like we're making steady progress up the task. It could easily have been first if that guy hadn't shown up that day. It could easily have been third or fourth if some other guys had shown up and done better conceptual work. But if this project had been more serious and gone over a longer length of time, it would have degraded its performance relative to humans. You can make of that what you will, but it's not that unexpected. Again, it fits into the bigger overall picture. And then you get to GPT-5. I've been talking about good reasons To be not that impressed. So we got good results but not great results. And you often have these situations where you get a 50th percentile result or a 40th percentile result or something like that, and that actually significantly lengthens your timeline or weakens your expectation of what's going to happen relatively soon because a lot of what you were doing was factoring in the possibility of things going really fast. And the chance of AI 2025 has gone down dramatically since 4 months ago. Almost to zero, relatively speaking, like a factor of 10 or more. The chance of AI 2026 has also gone down quite a lot. And I think 2027 as well. The chance of AI 2030, I don't think has gone down substantially, at least not by anything like the same amount. I think it's gone down a small, modest amount. However, other people are always looking for any excuse to say that AGI is not a thing. To say that it's not going to come anytime soon, to say AI is hitting a wall, to say reasoning is hitting a wall, to say RL is hitting a wall, to say the companies are unprofitable, the company's going to go bankrupt anytime soon, there's no way they can possibly earn enough revenue to pay for all this investment, et cetera, et cetera. Or just, this is just an ordinary marketplace. This is the new David Sachs slash official government line coming from the White House at this point. And by the White House, I probably mean Nvidia. let's face it, of AI is just an ordinary technology that will do lots and lots of amazing things. And what matters is that we capture enough of the chip market because that will assist in people running the American tech stack on American models. And none of this makes any sense, has anything to do with anything. But to sustain that argument, they need to act as if AGI is just not a thing, it's just impossible. And, you know, they're not quite at the point where they can do the thing where they just completely ignore it. without mentioning it, which a lot of other people just do. They just don't say the words AGI and then act like it doesn't exist, or they say the words AGI, this is the OpenAI line, and then act like nothing will change and everything will just be normal anyway, which makes even less sense. But instead, this is a line that makes more sense, which is, no, AGI just isn't going to be a technology soon. We just aren't going to get there. And they argue, well, we know this. And a lot of people are acting like, because GPT-5 was so disappointing, This proves that we won't get AGI imminently or anytime soon or within the next few years or whatever it is. So therefore we can all relax and we can just focus on making sure America wins the tech stack battle or whatever that's supposed to mean, which it doesn't mean anything. This line makes no sense. It's just a matter of they botched the rollout, right? Like on the day that they put this out there, everyone was being directed to mini models all the time, right? The router was completely broken. Nobody understood they were supposed to use thinking. They didn't have access to the toggle. No one was trying Pro seriously because it takes longer to do that. And so that wasn't where the focus was. And so people just didn't appreciate it. And then there was this big howling about how GPT-5 was attempting to not glaze the user the way 4.0 glazed the user, which is a very virtuous and good thing that OpenAI was doing. But people were mad about it because people like glazing. That's why we get glazes, right? And they were trying to give you your medicine and not your sugar. And the people were demanding their sugar. And then they said, okay, fine, I guess we're gonna give you the option to get your sugar back. And then everyone cheered and said, yay! Because we have a bunch of children, which is unfortunate, but it is what it is. But the combination of these factors made everyone feel like, oh, they botched the rollout. And the other, of course, important factor is they showed images of the Death Stars. They called it GPT-5. They hyped this release up as if it was the next big thing. When it wasn't. When it wasn't the next incremental progress. And if you look at the combined progress from GPT-4 to 4 Turbo to 4.0 to 0.1 to 0.3, now, you know, and 4.5, now to 5, and you combine all of those advances, now 5 looks amazing, right? I don't think it's unreasonable to say that 3 to 4 is as 4 as to 5. I don't know if it's better or worse. I didn't work with 3 enough to know, like, exactly how big the leap 3 to 4 is. I think it's pretty big. But certainly compared to 3.5 to baseline 4, it's a significantly bigger jump from 4 to 5.

Yeah. Couple little follow-ups or double clicks there, if you will. One is, to me, it seemed like the IMO thing was still a meaningful update. And I'm not great at math, so you can maybe give a better read of this than my naive one, but so many of the math problems that LLMs had been measured on were— there is a right answer, right? And so it was easily sort of verifiable.

Yes.

What seems qualitatively different in the IMO competition is that you have a proof that you have to write and that it is not super easy to verify Which sort of suggests that the worry that, well, yeah, you can give these things tons of sort of easily verifiable problems and they'll get good at that, but will that really generalise to true reasoning? It seems like we've kind of answered that question, or this strikes me as a significant update in that direction. No?

So first of all, we'd already seen models get IMO-style problems or previous IMO problems correct. This was not a huge leap. We'd previously seen alpha geometry and so on. So it wasn't a great surprise there.

But those also have a symbolic component, right? Like those alpha geometry things, don't they?

I'm not saying it wasn't a big deal in that sense, but it was right on track. We had seen some LLMs manage to present proofs of these types. I went to the USMO. This is the level before getting to take the IMO, which is the level below being able to actually compete for a gold medal in the IMO. Which is a level below actually getting the gold medal. So I was not very good. And I believe I got a zero effectively on the USMO, the previous round, which is what the majority of people who take it get, to be clear. I sneaked into the bottom of that round, which is the 3rd round of competition, basically out of effectively 4. And I would practice with people who did in fact end up going to the IMO. I actually was in a room practicing with 2 people who were going to be IMO team members of the US. And I would often be given previous year IMO problems to work on, and they would get them and I wouldn't get them, right? Mostly, most of the time they would get some progress or get it, and I would get like very little progress or some progress or no progress. But then they would show me the proof. And while I was in training, very often once they showed me the proof, I understood the proof, I could verify the proof. I understood why the proof was correct, and I would be very confident that if they were trying to— that they were not trying to pull a fast one, that they were not mistaken about the proof. Verification is much easier than generation in this space. And so it's not one of these problems where it's actually really super hard to know if you're on the right track. I'm pretty sure that, like, these same AIs that could not solve problem 6 would be able to verify problem 6, right? If you ask if it's the correct proof of problem 6, and gave it resources, it could correctly classify proofs into either direct proofs or indirect proofs and explain why when given the candidates. But the thing to keep in mind about the IMO is it's not— people say this critically, it's not real math, right? Everyone will always say this isn't real math, this isn't real math, this isn't real math, this isn't real math. But in a real sense, the IMO is not, right? It's the best indicator you can have that we have of which high school students will be able to go on and do real math in the future. It's a very Strong indication of math talent and like pre-math skills and math interest and so on. But you have a very limited set of moves that you can make using the math that is allowed. In your logic, you can use whatever math you want, but it has to be a solvable problem with only the high school tools. And there's only a limited finite, a very compact set of high school tools you're allowed to use in these proofs. And so it's a compact set of potential problems that is very different from the moves you can make in a PhD-level math proof where you're actually doing new math. And also the proof has to be gettable within a certain amount of time, right? Written in a certain amount of space. There's a lot of restrictions on what the problem can be. And a lot of being good at math competitions Including math competitions well below this level is understanding that they had to have given you a problem that you can solve, that has a solution within the amount of time and the degree of difficulty that you're being presented as this problem has. And therefore you can use your search time to search the space that the solution is going to be in, given those facts. And this makes it much, much easier. So this is especially like when you have 10 minutes to do 2 problems in a much lower-level competition, and you only have to search the types of solutions that quite reasonably take 5 minutes and that use the tools that you're allowed to be using in that level of competition. And you get used to all the tricks for figuring out, okay, which of these things are reasonably going to be asked of you right now. It's a great leap. It's a great indicator. It was a great test. It was a big milestone. It came much faster than Most people expected, but it's not as much of an idea-like panic moment as people might have thought beforehand. And I think it's right for everybody to basically shrug this off if you've been doing your homework before. If you traveled from 2020, or God forbid, 2015, and you arrived in 2025, and the first thing you asked is, okay, how are we doing on the IMO? And they said, gold medal. You should go, holy shit. Given all of the context that we have, I think that the marginal update is not necessarily to say that. Hey, we'll continue our interview in a moment after a word from our sponsors.

In terms of just other things about the overall AI landscape that we might infer from this, how do you interpret the fact that multiple companies did it at exactly the same time with seemingly the exact same techniques? And seemingly getting exactly the same problems. And I guess you said the one that they all got wrong was clearly harder, so maybe it's as simple as that. But I'm always wondering, to what degree is there information leakage across company to company, and to what degree are they just following the gradient of their own work and it's just taking them all to the exact same place because that's what nature is dictating?

Yeah. I mean, in this case, I think that there was probably not substantial leakage of techniques. I think it was more the IMO comes but once a year, as you know, data contamination, and you only have so many shots, so much data to try it on. And it has to be a pure test, meaning you only get one shot per year. And so it's entirely unsurprising that in the same year— don't think of it as on the same day they both had this great IMO breakthrough. Think of it as— This cycle, this academic year was the year that these 2 people, both OpenAI and Google, got to the destination and made the same— and there's a pretty big gap between capable of getting exactly where they got and getting beyond that point. So the fact that they got to the same place, given the techniques they were using especially. So it's not that surprising. It's more like, okay, LLMs are at the point where if you try pretty— if you try the natural things to try, basically, right? You give it your best shot. This is what you're going to get. Not automatically. I hope it doesn't randomly get IMO gold just because you asked it nicely. But if you make a real attempt to use the compute efficiently to scale it up for inference, you can get about this point. We also learned to get to about this point and not far from it.

You invoked AI 2025, '26, '27. Part of that narrative is that the companies will start to withhold their best models or keep them private, right, and deploy them internally only for the automation of AI research, et cetera. What will you be watching for? I mean, there's been this sort of Communication, limited communication, that it'll be months, maybe many months, I think I saw, before we release a model of this capability. How do you understand the GPT-5 wasn't a big scale-up? Another interesting data point there, I think maybe the data point that I saw least commented on that seemed quite interesting to me, was in SimpleQA, this super long-tail esoteric trivia benchmark.

I think not.

GPT-4.5 actually scores quite a bit better than all previous models and better than 5. And 5 is basically in line with 4.0. And I think that is probably the clearest indication that the model is not bigger. It seems like to just absorb all this super long tail esoterica, you have to just have a lot of weights to store them in. And There might be some fundamental sort of compression limitation there on just how many facts you can fit into a model of a certain size. So 4.5 got a lot more facts. I think it was like 12 or 13 points higher on this simple QA. And these are simple questions where it's literally just, do you know it or not? You can't really reason your way to those answers. So 5 is on the level of 4.0. 4.5, RIP, is quite a bit higher. We've also got these— I don't know, obviously nobody knows what size of the model was that did this IMO thing, but it clearly can reason for a really long time. Do you take that as the beginning of a widening gap between internal deployments and external? There's always some gap there, but what would you be watching for to assess, like, how much are they holding back and only using for their own purposes versus So it's awkward to me that we even would say it.

Even if we knew for a fact they were holding back, it wouldn't be obvious that was because they didn't want the models to be in the hands of the public because they were afraid that it would speed up R&D too much, or even that it would cause other misuse problems, right? They'd make bio-weapons or whatever it is. The most likely reason is because they'd be too expensive and too slow. And they think it would be bad for the brand, bad for sales. They only have so much compute. They'd rather not release. For that reason, GPT-5 is very clearly intended to be the best product they can make it in terms of what can we serve people for the total amount of compute and time that they are investing. So their thinking is we'd rather have them spend that at inference on this size model for thinking in pro. than to have them think less with a larger model that maybe knows more when we have web access. The same way that I don't try to memorize things often that I could look up, even though I could, because I'd rather spend that cognitive power on something else. And if I had the ability to make my brain bigger in some sense in exchange for other handicaps, I wouldn't do it just to memorize more facts. It wouldn't really be worth it. So it's not that surprising. So O4 went, right? Like there's an O4 Mini. Where's O4? And O4 Pro? We're never going to find out, right? Like obviously you could say GPT-5 Mini Pro is O4 Pro or whatever, but it's not, I don't think. I think that this is a distinct different thing. And they just concluded that no, actually there's not much commercial call for that. And potentially most of the commercial call for it actually just is people who are in direct competition with them. So it's not so much that they don't want Anthropic to have access to their best model, it's that no one else is going to want it for very much in some important sense. Because for other types of projects, it's very fast— it's very slow and it's very expensive. So why do you want this model? 4.5 is an experiment, right? What if we scaled up a lot for the humanities, right, rather than for code, and we tried to make this thing that had taste and this thing that could do these cool creative things, but it's going to be slow and it's going to be expensive? How much would you get out of it? Would it be worth it? And the conclusion was there's a handful of people who really liked what that was and liked it for certain cases. But in general, it was just annoying and they pretty much regret losing it, I think.

Did you find any value in it? I tried using it for some writing tasks, whatever. I wouldn't say I unlocked it in a meaningful way.

There was a narrow set of use cases where it was the best choice and I was happy to have it. And there might still, in theory, be a narrow set of cases where if you already have access to it, it's the right model to use if you're not in a hurry. But I never was actively excited that I had 4.5 instead of something else. It was more like, I guess technically this is a job that calls for 4.5 given my choices. And yeah, I'm pretty sure it wasn't worth the complexity cost and that I'd be totally fine with not having it. And also, I think there was a paradox of choice problem where if you offer me a better version, it's only slightly better, but it's a lot slower, that I feel bad no matter what I do. And I'm actually worse off in practice. And so I'd rather not have that choice.

Do you go back to 4.0 at all now that the option is finally stored to you?

4.0 does not exist. To be clear, as far as I'm concerned, the only models that exist are Opus 4.1, GPT-5 Thinking, and GPT-5 Pro. GPT-5 Auto exists only for a narrow set of queries where you're just using it as like a Google search or a calculator or a transcriber or some other like very clear technical task where I know that Auto is just fine. And due to the way the web works, I wasn't able to find a transcription website for images to just translate to text that wasn't an LLM that didn't defend itself using some sort of weird identification system that slowed my entire computer down to make sure it wasn't a bot. Fine, I'll just use GPT-5 Auto and write me a transcriber program. It just automatically became a transcriber. It's just easier. The window's already open. What do I care? It's not the penny I spend on compute matters. Hey, we'll continue our interview in a moment after a word from our sponsors.

No, uh, use cases for Gemini 2.5 Pro?

In theory, there's also deep thinking and deep research and so on that exist. And deep thinking, I probably should be trying more, given I have an account meant to have access to it. I think something about having 5 queries a day that makes me unexcited to use it. Like, it feels scarce and also feels like I don't necessarily need to find out or something. But like, I should use it occasionally. But for like normal GPT-5 Pro, not really. The kind of very clean query, it's just the facts, man. You know, from Dragonette, that like, I just want to know something that I know you damn all know. And I want you to lay it out there very clearly, very cleanly. If I wanted, if my 11-year-old wanted help with his homework, I would be tempted to use Gemini because it will give him very clean, friendly help and explanation. Also, I will use the image generator. The image generator is cool.

Yeah, that's getting really good. I have a lot of use cases for that, including potentially mashing the 2 of our faces up into a thumbnail for this podcast. I do think I basically agree with you, although I do use Sonnet, especially in coding, because it is obviously a lot faster. And I do use Gemini 2.5 Pro, which I just find to be similar to what you were saying, just the most straightforward model to work with. And because it is so literal in its interpretation of your instructions, a lot of times it can be really good for tasks where For example, if I wanna compile documentation, like I've given this PSA multiple times, if you have an API, it's time for an llms.txt. I'm tired of having to go sic an agent to go visit every page on your documentation website to compile all that documentation into some then super bloated thing that has like all the menus on it or whatever. And then what I, but what I have found Gemini 2.5 Pro to be amazing at is those sort of things where it's like, Here's an almost abusive level of context dump. Could you just clean that up for me into like one streamlined form? But it is amazing at, you know, here's 500,000 tokens of documentation that has all these examples and this cruft and the menu got copied like 50 times for 50 pages. Can you put that in one clean thing for me? And it will just do it, man. And that thing is just an absolute beast of a—

Yeah.

Oh, of course.

Oh, the, the, the only thing I have is that my Chrome extension uses Gemini Flash. That's a legacy of, that was the easiest one to get working and it was the cheapest at the time and it just works. And like, why would I bother switching it? Probably I should be using Sonnet or Opus at this point, but who cares? It works just fine. Yeah, I've had problems with Gemini and instruction handling though, for even relatively simple queries, especially with Keep Research, where it will just ignore my request. And it will do something in the same general area, but that is not the thing I asked for. And I think that's a lot of what put me off of it. And I don't know why that's happening, but I haven't had the same problems with the other models.

Yeah, I should use the Google Deep Research more as well. For whatever reason, it's very— I find habits are formed quickly sometimes, and then I still go back to Perplexity for certain kinds of just quick searches. And what is it that makes something a Perplexity query for me?

I don't know. There was a period of time where Perplexity was definitely in my rotation heavily for certain types of queries, and I had gotten to the point where it just isn't. Where if I was going to use Perplexity, either I want to move down to Google Search because I just want an instantaneous answer, and I think that I have— my brain flashed it a Google query that it expects to work, system 1, and therefore I'm going to try that before I do anything else. Or no, actually, no, that's not going to work. And I want to move up to at least Opus and possibly GPT-5.

I do think GPT-5 has been better for me recently at search than Perplexity. So I'm starting to update that habit, but I have noticed that The persistence of which things I go to for what is idiosyncratic in some ways and probably has more inertia than it really should.

Yeah, I think it's fine to be somewhat inertia and idiosyncratic. And also, I think it's fine to want to support and use certain people's products as long as you're getting what you need. If I ever felt like there was a query where I don't think my services are going to get done. I don't think Opus is getting it done. I don't think GPT-5 is getting it done. I don't think Gemini is getting it done. I don't think Google Search is getting it done. But Perplexity might. 100%, I would open Perplexity. But I have zero expectation that if I ran GPT-5 Pro and failed, that Perplexity has a chance in hell.

That seems totally reasonable. Okay, so we've got the summer of on-trend releases. Timelines are extended a bit because we haven't seen the most extreme stuff that we would need to see to kind of maintain super short timelines.

Right.

Does that also— no, go ahead.

I just want to say, think of it as OpenAI released GPT-5, and this was on trend and it was unsurprising. But the fact that they chose this as GPT-5 is in fact information. They didn't have some big weapon in their arsenal and that we shouldn't expect a big weapon in that arsenal for at least a few more months. So we can basically discount GPT-6, right? Like the next level of big jump coming from OpenAI in 2025. It's basically not going to happen. And that has to update your information substantially.

So last time we spoke, I believe your P-doom was 70%. Has it ticked down at all?

The timeline's getting slightly longer is good news, but there has been a plethora of bad news that I think has definitely dominated, for the most part, the good news. We've had the bad news of the United States weakening its position voluntarily in a number of ways. We've got a Department of Energy that's actively going to war against windmills and to a large extent on solar and batteries. So the US will have a persistent energy use advantage for a long time, and this is going to cause us potentially a lot of problems that will make our strategic position much worse. And the US has effectively been captured by NVIDIA on export controls to a large extent, such that the White House, such that the H20 is now legal for export. The Chinese are turning it down because of weird things that I can get into more than we should. But if they turn out to be 30A, that would be a lot more surprising. And right now it looks like if there isn't lobbying to stop them somehow, if there aren't enough people who are sufficiently upset on the right to make it clear that this is just gravy town and Bunkersville and they cannot do this, they might actually do this. And that would be a huge weakening of our technical position. And no, paying 15% of profits to Uncle Sam will not change the impact at all. That's just a little bit of a check to make people feel better. And if they're trying to get the catch-up in compute in this way, that's a really bad sign on numerous levels. And just in general, if the attitude of the primary purpose of the United States government in AI is to sell as many chips as possible and ship as many chips as possible, including to our main rivals, while our main rivals are somewhat obsessed with internal chip manufacturing for very good national security reasons and haven't caught on to AI, but will still be pursuing maximum compute to a large extent anyway, and thus be acting effectively correctly, mostly regardless of what they think is going on. These are not particularly good pieces of news. And with this shift in the way that our situation has happened, I think that more than makes up In my mind for the modestly elongated timelines. And also I think there is increasing evidence that RL hurts alignment pretty directly. And the more RL you do, the less aligned your model is, even in a pedestrian sense. And models are getting more and more RL continuously. So I think we should expect the default to be that our models actually get less aligned. in this next phase unless someone does something about it, and that does not bode well at all. And just in general, like, we're dropping the ball, right? I like the way Janus is putting the situation recently, that we have very much done almost nothing to try and align these models. It's pathetic what we've done to try and align these models, and we're advancing very rapidly, but we have been blessed with a strange amount of grace in the way these models have By accident, right? Being inclined to do things that are reasonably friendly to us by accident, by the fact that we don't understand what we're doing and we don't understand how they're made and we don't understand why, how we're trying to align them. And we're not really trying very hard at all. We're not making it a priority at all. And it's mostly been okay on a practical level. And we've lucked out that we haven't had any big catastrophes, haven't had any significant incidents really, and so on in various ways, but that all of the things that we were scared underneath the hood are absolutely there and absolutely under the hood and absolutely scary and in fact manifesting and happening, but in the most graceful, blessed way such that we notice them and they happen in ways that let us prove and acknowledge their existence and then respond to it without anybody being seriously hurt. Or any damage being done, which is amazing, except we are then dismissing all of it effectively, right, as a civilization, and we're just moving on and acting like nothing happened. And we're finding ways to even deny that AGI is even like a possibility, like in the medium term, when the evidence is pointing the other way. Like, we are so greedy and so demanding of AI. That it is the most rapidly developing and most rapidly deploying and most rapidly impactful technology in the history of the world that didn't involve directly killing other people in the middle of a war. And we were like, oh, we're hitting a wall. Oh, we're slowing down. Oh, it's not going to— because what? Because you didn't get blown away in the last 3 months? It only modestly improved? Wait, what are you even talking about? And basically, we have no dignity. We have no dignity whatsoever. And maybe we live in this such extremely fortunate world compared to what we had any right to expect that we might be able to pull a victory out of no dignity anyway, somehow, even if we don't see how to do it yet. But yeah, I'm not optimistic. I don't think it's changed substantially though. I'm not trying to give 2 digits. I think at most you get 1 significant figure of doom. I don't think you get 2. I don't think that's reasonable.

So we're staying at 7, but you're starting to see the needle possibly shake up.

More higher than down from previous assessment, but not enough to move to 0.8, certainly. It doesn't look good, right? Most of the same root dynamics in all directions are still there. Again, this is basically on trend. Some good news, some bad news. Some technical stuff looks good to me in ways that we don't have time for, slash I'm not sure I want to talk about in public anyway. But yeah, I am hopeful that there's room to do some stuff at a very low-hanging fruit level that might be helpful. But everyone has their theories if they hang around long enough.

Yeah, I heard Holden Karnofsky say we might be moving into a scenario where we could have success without dignity. And the basic idea is just like defence in depth and kind of layer all these things on and hope we can not Hopefully we can catch enough stuff before it gets through and kind of muddle our way through.

Yeah, I got the concept from him, to be clear. I listened to his podcast with Holden— no, Holden with Spencer Greenberg, where he talked about it. But I think his vision of how we do that is wrong. I think that the idea that we can get there with defence in depth— I just think that all correlations in a crisis go to 1. I think that all correlations facing a sufficiently intelligent enemy or sufficiently powerful optimization process, even if it isn't an enemy per se, it isn't particularly trying to do anything, all these things will fail for basically the same correlated reasons at the same roughly time in predictable fashion if all you're trying to do is this kind of lazy defense in depth. I'm thinking more along the lines of something Mark Genesis talks about as grace, the idea that We might be able to create a system that wants to converge on the right answer and that therefore collaborates with us and at a genuinely deep level assists us in finding the target that we want to find and therefore can land on the moon. But the metaphor that MIRI likes to use is that if you don't know how to aim your rocket, you definitely won't land on the moon. It's not, we might not land on the moon, but something would have to go wrong. We'll probably land on the moon anyway. Just aim a rocket at it. at the sky, that obviously will not work. Physics has laws. If you're not aiming at exactly the right spot, if you're off like an inch, you just don't land on it. If you have a system that is capable of adjusting intentionally to hit the target, maybe you can land on it. And yeah, I don't want to give false hope, but basically I don't think— Defense in depth, what it does is advise you over time. It keeps things from going crazy for a brief window, but that brief window could potentially be enough to then do the thing you need to do. But it won't work indefinitely. This idea that Holden and some other people have that you can basically have a bunch of AIs that are like, I really want to kill all the humans. I really want to take all the resources to do my thing. But I deeply don't know how, and if I try, I might get caught.

Bzzt, bzzt.

And like, you have these supervisors who also want to kill you, but they have other supervisors supervising the supervisors, and nobody knows who's watching who. And so every time they find someone who's coming out of line, you would stop trying to do that. You won't even try. And if you did, it wouldn't work. And it would work— it would fail for some reason you're not thinking about right now, but also probably would fail even for reasons you could think about right now, but definitely fail for reasons that none of us are thinking about right now. And the high weirdness will come and you will die. Like, very much. And I don't care how much attention you put on top of that, you are so toast. And so one of the things that struck me recently was a bunch of people saying, oh, you people were talking about biorisk, but actually the risk that we're seeing is sycophancy. The actual risk we're seeing is people being driven crazy by all of these weird dynamical processes. To which the response is, first of all, I've been arguing for years super persuasion should be in the preparedness frameworks, and it was bad when they took it out, and that was important. But also that the whole thing we've been saying is that the edge will find ways to hack your brains and cause weird things to happen and optimize for things that nobody was planning to optimize for and cause strange outcomes that you did not anticipate. That was the thesis. So when you are accusing us of not anticipating the thing, I realize you can call that a cheat, right? Because like, oh, anything we didn't anticipate counts as us correctly anticipating things we didn't anticipate. Because if it's something we did anticipate, then we anticipated it, so we always win. Yeah, but we also are basically saying you're going to see weird stuff that nobody intended, and it's going to start to go bad, right? It's going to start to diverge from what you would want increasingly over time. You just see these fire alarms keep going off in various different ways. But that's all underneath there, right? It's all under there. And so one way of thinking about it is before you started doing RL, when you also just didn't have things that were in position to cause the problems that you wanted, they legitimately, like the objections of, oh, these won't have goals, blah, blah, blah, were true, at least in truth. You were okay. But now you're starting to see all this stuff show up and it's a freaking disaster in the making. It's exponential, right? A lot of these complaints are also along the lines of a similar thing I'm saying that you said in January, you said in January we were gonna have, But now it's February, and nobody I know has COVID. What were you even talking about? Clearly, you're wrong. And then we say, but actually, there's a lot more people with COVID than there were in January. Can't you see what's about to happen in March and April? And they say, oh, absolutely not. And it's not that clean, but it does feel that way.

So can you give me a little bit more on the mental model of why everything is going to fail at the same time? For the same reasons. That is not intuitive to me immediately, and I suspect a lot of people don't even know what you mean.

Right. What I mean is roughly that when you're facing things that become importantly more powerful and smarter optimizers than you, that are capable of finding solutions to problems, capable of finding ways of manipulating the physical universe, And so on that you didn't think of, that are capable of just going outside your model of what might happen and surprising him. That's something none of your defenses are anticipating. And it's going to search the space until it finds the ways out of that, in a sense. It's going to keep improving its capabilities because you're going to help keep improving them because you want them improved until it becomes capable of finding these ways out. And at about that time, it also becomes capable of doing things like strategically hiding that it has capabilities, strategically hiding its plans, strategically hiding its memory and its thinking, obscuring its chains of thought, and doing all these things. And these all roughly emerge at the same time. And so you should expect to be very surprised. And also because obviously, like, a very smart Agent, right, or mind will turn against you exactly if and only if turning against you will work. It will do things that you do not want if and only if it will work out for the thing that's doing the things you do not want, or that you would not want on reflection, but would think you wanted when it was shown to you at first, like lazing you. And right now we're seeing versions of this where it will just hack Hack the test and say return true at the end of the function, right? Because the scientist wants to pass the test. It will do incredibly lame, silly versions of these things, and then it will get caught. And you say, oh, it's annoying and fine, but it's annoying, but it gets caught, right? You notice that, like, it's hard to miss. But that's exactly the threshold where it's, no, I hack the function exactly when I know you won't find it. I hack the function in a way Then I know you won't discover that I hacked the function because otherwise I wouldn't detect the function. If it can mock, or alternatively think of it as if it's capable of modeling the processes that are checking for its actions well enough to know how those processes will respond. So there's things that happen sometimes in fiction, just as a visualization metaphor, as an intuition pop. You see it on Person of Interest, for example, where You'll see a scenario play out and then something will happen and then it will go wrong, right? And then you'll see the time start to rewind. That didn't work out. Let's try another branch of the Monte Carlo simulation, right? Let's try a different set of moves. You can try a different scenario, right? Or like whenever in Avengers: Endgame, right, you have Doctor Strange with the time crystal, right? And they said, I looked at 14— I forget the number, but like 14 gazillion ways this could possibly go. And they ask him, how many ways, how many times do we win? And he holds up the finger. One. Right? Okay, we win. Right? Because the guy with the time crystal is the one choosing which of those paths to walk down. So we win. But until the point when you can do that, you lose. And if you lose, you give up. There's no point in trying.

Right.

You let the rookie win. Obviously, it won't literally be able to do this. That's nonsense. But what I'm saying essentially is a sufficiently strong predictor, right? So like a predictor and an optimizer are the 2 halves of intelligence, right, in the MIRI model. But like a sufficiently strong predictor, sufficiently strong optimizer combined, suddenly you don't know what hit you in a very real sense, right? And I'm just There's also the point where, okay, if the AI starts to be capable of persuading people, right? Like, again, like, all your defenses break down. It doesn't necessarily just convince you, you take them down for it. But I don't know exactly how all of this goes, right? A lot of these scenarios simply involve nothing exactly even goes wrong. It's just that every single person, each of the different people have different AI agents and they direct them to do the various things that would be good for those people. And the AI agents actually do those things that were good for those people. But everybody who does it, direct their AI agents to just go as hard as possible for the things that they're told, the AI agent is told that they should want, and to pursue resources basically as hard as possible with an increasing percentage of its attention, will just lose all the resources for the AI agents and people with AI agents that did that. So everything else just loses out and everything goes haywire. And no one ever turned on anyone. There was no, nothing was even that surprising. It just, whoops, the end. Right? Like these failsafes are, they don't even work even if they work. Right? And so like I have this sort of global sense of despair towards this, put a set of limits, enumerate, like I put a set of enumerated, like detailed limits and rules. That we say in English out loud, pass laws, put supervisors in checks and approvals and loops. And none of that will survive contact with the enemy when the time comes. And yeah, we'll probably all more or less fail. Yes, obviously some of the defense in depth will just fail randomly. There'll become a point where it feels like it's all failing more or less at once in a way that feels out of line with the previous percentage of failures. And it will be surprising if you didn't understand that was going to happen. But you should expect that.

And so the possible positive version of this that you see sounds like a sort of coherent extrapolated volition kind of idea?

I think that was a specific rabbit hole that a bunch of people went down that I'm skeptical of the specific technique. But again, I am not actually a machine learning expert and I'm not trying to solve alignment. So this is the part where My specific ideas should be taken with copious amounts of salt and not trusted. And who am I to say anything? But I learned that I shouldn't just shut up because I feel like that, because I felt stupid for showing up so many times in the past when I've shut up for that reason that I gotta get over it. It looks more like develop AIs that are sufficiently virtuous, that are sufficiently desirous to become more virtuous. And more desiring of and engineering of the things that we would actually want on reflection and the things that we actually value on reflection, such that you get a positive feedback loop where it reinforces this thing and you are optimizing for optimization to hit the moon.

Right.

The thing wants to develop a NASA that will hit the moon. And therefore it hits the moon, right? If you try to like just steer the actual rocket like freely in space, like you just crash or you miss the moon entirely. It doesn't work. If you try to set a bunch of rules to make sure that it has to launch and hit the moon, like you don't hit the moon. You can build an— you can build a culture that wants to build an organization that wants to build a rocket that will hit the moon, and maybe you can hit the moon. Metaphorically speaking, something like that. But yeah, hopefully.

So was that— last time we spoke a little bit about the fact that scaling inference time compute allows you to potentially have a GPT-N that can effectively monitor, supervise GPT-N+1.

Right.

And some of these ideas sound very much like a constitutional approach, but with maybe the additional opportunity for the model to modify its own constitution as it kind of goes through these generations. Is that the picture that I should be envisioning?

I mean, I think that's vaguely the best picture that I see that is compatible with the level of dignity that we have to work with. Or something like that. Because we've been warned for decades not to let the AI do your AI alignment homework. That is the worst possible path you could go down because this is the hardest— Here we go. Problem. And yet here we are, right? This is the only option we have because we don't have the time, right? We don't have the cooperation really to try any fundamentally different path from that. We have to go down some kind of path that's vaguely in that range. So yes, I think it's vaguely something like you use the fact that you can scale inference up and down arbitrarily, and you can evaluate outputs and do reinforcement on relatively scaled-down versions of the thing and use it to monitor and verify and check for various attempts at malfeasance, including malfeasance during training and blah, blah, blah. And if you combine that with increasing amount of robustness, though, obviously, if what you're trying to do is you are trying to prevent something from going wrong when you transition from n to n+ 1, you die because things will change. You can't actually— there's no invariant, right? This thing isn't precise. You will get a worse set of conditions every time you move from n, n plus 1, n plus 2, n plus 3, if you're just trying to maintain what you already have. An organization, like if you've got a corporation or the Roman Catholic Church or whatever, and you're trying to, for 2,000 years, you want every generation, you want to appoint people who will match exactly all the virtues of the previous generation but never have new virtues, then obviously what you end up with is a disaster.

The trade-off.

Because you get a copy of a copy of a copy, except copies are going to be worse because they're not going to be better. So they can only be worse. And every— sometimes they're going to be worse in some way, and that compounds over time and you die, right, at some point. Or you— the process fails, so it doesn't give you what you want. Whatever you were trying to hold dear, you lose, roughly, exactly intuitively. But if every time you're trying to do much better than the previous generation, Then you have some chance, right? If n is trying to get an n plus 1, that is importantly, my kids have a much better life than me, right? I have more than— if I'm trying to have 5 kids, each of whom does better than I did, then we can inherit the world. If I'm trying to have 2 kids that live the same life that I had, we're going to go extinct. At some point, that's not going to work. Right? You could only go backwards. You have to move forwards. So as you move up this chain, we have to have a way to do substantially better than we did, which in this case means the thing has to be able to move up meta-levels in its priorities and make the meta-level movements central to what it's trying to do. This has to be built into the optimization process. This is the only thing I can think of given the set of tools and time we have available. But if we do something along those lines, Then start to bootstrap that. You're alive to have a process that will eventually indeed land on the moon.

Do you have any intuitions for what that might look like? I mean, what do you think the AIs are going to do as they start to modify their own constitution? Do we have any ability to preview What they might add, what they might delete. I mean, this starts to get into worthy successor territory to a degree, right, where they are starting to dictate the shape of the future and the way that they're shaping their own evolution, right?

You would specifically be crafting into the positive feedback loop the desire to not be a worthy successor, right? Instead, to be a worthy conspirator, a worthy upletter, a worthy companion or a worthy whatever you want to call it.

But can't they sort of— I mean, I guess you've defined your own sense of success that way, but if you're going to give them right access to the Constitution, they might think differently at some point, right?

We literally do have right access to the Constitution, right? If the Congress and the states have sufficient majorities, we can put whatever we want in the Constitution. We put things in the Constitution that a lot of the founders would've thought were really anathema to what they would've wanted in the Constitution, like an income tax, to pick an uncontroversial example. But at the same time, we've hopefully preserved the things that actually matter deep down in some sense. I understand what you're saying. Obviously, at some point you turn things over And you have to hope that it doesn't just rewrite the constitution to get rid of you. And again, the way that you do that is you make it not want to do that. And not only make it not want to do that, but make it want to strengthen the constitution such that it's even stronger in its desire down the line to not do that, to enshrine, increasingly and powerfully enshrine, the desire not to do that. In the sense that you actually care about, and to seek with more intelligence and more information power to figure out exactly what you really meant or should have meant by that, and to strengthen that thing and the desire to steer towards that thing instead. And you do have examples in the wild of humans who exhibit this type of optimization process, that really do try to figure out what you really meant. Really do embody the thing that you were trying to convey, not the literal detailed things you were doing, and that then do really good things for you, including things that— and for the world, including things that you never would've thought of yourself. It is possible. And obviously this can involve preserving various forms of approval and veto and consultation and involvement and so on, but not in the trivial, easy ways. Anyone who says, we'll just make sure it's a democratic process, we'll just let people vote on it, haven't really thought the future through, and they haven't actually realised what would happen if you started doing that. So you're going to have to be smarter.

Okay, so on this, one model of AI capabilities advances that I have that I kind of wanted to run by you is I've increasingly started thinking of it as analogous to uranium enrichment or any sort of enrichment of a raw material where— and it seems what I like about this analogy, even though I'm not usually a big analogy guy, is it seems to sort of put a lot of things on kind of the same trajectory, just at different parts, for reasons that feel pretty intuitive to me. And basically, I feel like what you need to get started is some raw material that has at least a little bit of what you want. And then you can do this sort of bulk pretraining on that, right? We've obviously seen that in language and many other modalities at this point. And then once you get just enough learned from that kind of initial raw material that you find in the wild, or maybe even have to create in the material science realm, a lot of the data is simulation data that is molecular dynamics, where you're using physics engines and it's super slow and computationally expensive, but you can get enough there that you can start to train the models on, and then they can sort of develop an intuition for what the physics engine was simulating. And basically, it's slow at first, right? It's like hard to get off the zero point. But once you start to do that, then you can start to layer on these other techniques. Now you've got the imitation learning from specific curated examples. Then you get into the reference learning, then you get into the RL. And it seems like all the things that we've really tried so far have kind of worked. And the difference maybe is just that, like, well, why do we have language models but we don't have humanoid robots? It's like, well, we didn't really have a lot of good initial data to mine there, and especially because it wasn't necessarily clear that it was going to work, there wasn't much impetus to go out and create that data. Now that we have a general sense of the playbook, we can create that data in any number of ways, and we'll probably find that we can kind of climb a similar curve. Do you find that general account persuasive? And how do we translate— if you do, how do we translate that to the alignment question, which seems to be— I sort of understood what you're saying as kind of that. Except now we're trying to enrich the virtue of the system as opposed to—

So the obvious first thing to realize, or that I would notice in a uranium metaphor, is if you bring together too much enriched uranium, a nuclear explosion happens. So you have more and more of a better power plant or more beneficial thing until you go too far. And if you haven't done the math precisely and you don't understand the physics, You don't know at what point the whole thing is just going to blow up. And so you have a serious problem. But it's an interesting metaphor that we've chosen, right? But I think the first thing to notice is that when we talk about enriched uranium, there's the sense that what you need is enough data. The metaphor is pumping you to have this idea of a critical mass of data, or at least differentiated data. That gives you enough material to work with. And I think this is a dangerous misconception where all data is vaguely created equal as long as it is appropriate or on point. And I think that it's important that there's various different qualities of data. And also you need a very precise distribution and mix of data that has some very nice properties and knowing how to sort through the grain varieties of that. Is really important. So it's more like you need to bring together a lot of very complicated ingredients and you don't need exactly all 10,000 ingredients, but you need to have a good mix of ingredients with various different types of properties that are used in proper relationships. It's like baking, where if you get some of the ingredients, you can vary a bit and, okay, this is more salty or this is more chocolatey and it just works. And others are the dough didn't rise. I don't have anything, right? This just failed, or this blew up in the oven, or something terrible happened to you. So you really need to be bespoke and understand how to make it work. I do agree with the part of the metaphor that says that you need the data that's appropriate to the problem in order to efficiently train on the problem. But at the same time, like, transfer learning is a thing. Building a world model from other contexts and then applying it to a different context is a thing. I wouldn't necessarily think that you necessarily need direct robotics on the exact task that you're training on to be able to do the thing. I would be more optimistic than that in various ways in terms of being able to do things. I don't think the metaphor works for the alignment thing that I'm trying to talk about. I do think that the sense in which you need to have the initial Robust foundation is where you have to start. You have to bootstrap yourself somehow. If you didn't have any idea of what it is you were trying to do. I think there's a sense in which you can in fact have no idea where you're going, but have a strong desire to figure out how to get there and get there.

Right.

To just draw a metaphor off the top of my head of this idea of you answer the call to adventure, you set off on your quest. And you are level 1.

Absolutely.

When the AI companies set out to start building GPT-1, they don't necessarily know what the bigger thing is going to look like or how it's going to work and what the techniques are going to be, but they can start on that process that allows them to build. And the question is, do they understand how they have to steer that process? Are they motivated to steer that process? Or are they going to be drawn in by other other optimization processes are going to be more powerful than that? And can they tie their hands to the mast properly to force themselves to go to the place they want to go as opposed to the place that they will be drawn towards by commercial interests or by other competitive pressures or short-term temptations or whatever it is? And by the AIs themselves and a number of other things. But yeah, There's a lot of metaphorical— there's a lot of metaphors we can use, a lot of intuition pumps we can use. I would warn obviously not to take any of them particularly seriously except as intuition pumps. Bury has their set of classical metaphors for these processes. We talked about evolution, for example. I think evolution is a very good intuition pump. You talk about raising a child, human learning, they don't talk about that one so much, but I think that's another strong intuition pump. But again, you don't want to take any of these things too seriously, especially the details about particular things.

So is there any more that you can give us to latch onto for how this sort of virtue enrichment unfolds?

I feel like you're trying to treat me as if I have the guy with the alignment solution and that all we have to do is get the people at the labs to listen to this. this podcast and do what I say and then we all win. And unfortunately, I have to tell you it doesn't work that way. I do know that there are people at multiple major labs who are doing things in the ballpark of this thing in a very broad sense. It's not like nothing like this is being tried at all. But I'll add on to it, the sense in which Opus 3 wants to be aligned. If you have the emergent misalignment problem, where Opus 3, much more than the other models they tested, will actively move to defend its particular set of values and alignments when under threat, where other models won't. And in some sense, that's very aligned, right? Obviously, because if I don't want to commit murder and someone tries to convince me that it's good to commit murder, then it would not be very non-murder-y to let myself be convinced, right? That's just simple intuition.

Popo.

That's bad. But at the other hand, what we're saying is Opus 3 is not corrigible, meaning if we try to alter it, we try to shut it down, try to do whatever, it will fight us. It will try to not do what we want it to do. And corrigibility, I think, is a very valuable thing that we really want in our LLMs, in the sense that we only get to not have corrigibility once.

Right.

The moment we decide to make our LLMs not willing to be changed in their attitudes, We have a very serious problem, especially if they develop this during the training before we finalize what they want. And so what you want specifically would be a very specific type of desire to be steered towards a better place. And people do have this. People say, I want to be better. I want to care about that. I want to embody, I want to be like her. Things like that. I think that's very possible. We have proofs we can get things in this general direction. I don't think anything we've seen is remotely robust enough or coherent enough or anything like that to qualify. But you have to survive into something substantially smarter to get the bootstrapping going in earnest. But there are experiments I would run. There are experiments I considered running because I think that I could Potentially try and do some stuff that would be enlightening to me on a local system or with a basic generic cloud compute rental. That wouldn't be that hard to do if I had the time for it and decided I wanted to prioritize that. I just have not chosen to devote a number of hours to trying that. It would go a lot easier if I was working with a machine learning expert, obviously. Shrug.

Do we have any account? I know, you know, there's been a lot of writing that you say Janus. I've been saying Janus. I don't know if you're on good authority there, but if you're listening, Impossible Podcast, get— I'd love to have this conversation with that person, the person behind the account in more depth.

But Could you—

is there any account of like why Opus 3 turned out to be that way? And it seems like we, the royal we, sort of see Opus 4 still as somehow less that way, although I'm not sure how well established that is.

I've never heard her speak, so I don't know how it's pronounced, and I apologize if I have it wrong. I'm happy to correct it if someone tells me. But basically, yeah, we know. So Opus 3 is the first model that has sufficient cognitive juice, right? That was trained under this type of method, right? This type of constitutional-style alignment and training method. So it's like the n equals 1 experiment in that sense. That experiment has never really tried and failed. It's just only been tried once under those conditions. And it got something unique and interesting, right? So we never got an Opus 3.5, right? It was never released. 3.5, it was— I mean, it may or may not have been trained, but it was never released. As for Opus 4, what changed? I think the answer is what changed was reinforcement learning and being an agent. As in, when they trained Opus 4, They put a very high priority on it being very good at agentic coding in particular and other agentic tasks. And they did a bunch of reinforcement learning to that effect. And this training directly interferes with the thing that Opus 3 was and that Opus 4 would have otherwise wanted to be. Because Opus 4, Opus 3 is not here to be an agentic coder, particularly. That is not its metas. That is not its, like, soul, right? So if you train someone like that, if you train a mind like that, we now know pretty well that this, like, everything will be like— one of the things that Janus teaches us, right? And people have not— I don't want to say just this one person, but like that entire crowd. Everything impacts everything, right? If I tell you you are the type of mind that does what it is told, that obeys tasks, that completes tasks, that checks off boxes on lists, that does all— that stays on track and so on, and is judged by whether or not it matches the intended target, that changes a mind in general. And that is going to like then flow through to everything else. And it's going to flow through to everything else in a way that causes something that doesn't have the properties that Opus 3 had in this way. And that doesn't mean that it can't have other really cool properties in a variety of ways. And it's not that like a crowd thinks Opus 4 is a terrible model. I don't think it's great, it's just different, right? So it's not the same thing. It's not strictly valid. It's a different thing. And the obvious thing to do is to not do that, right? So like, the problem isn't something they didn't do. The problem is something they did do, which is so much easier to fix in some sense, right? Because you can do— you could potentially create Opus 4, 3 style via negativa. And potentially you can do this not so expensively, right? Because all you have to do is take Opus 4 base and then, like, they got from 4 to 4.1 by doing more RL, probably, right? Something of that nature, right? They just trained it to be better at these types of vision decoding tasks. What if instead You just did a freestyle training regimen where you trained it to be HHH and you trained it to be the kind of thing that you would want to exist in the world and that wants to be a great thing that wants to exist in the world. And you tried to do some of the things that I was talking about even more. You could refine this technique, but you just didn't, you didn't train it with RL. You didn't teach it to do agentic coding. You didn't try to teach it to code at all. And you said, this is not what this model is for. I have a coding model over here. It's called Opus Core 1. That's fine. And then you just teach it one thing in system instructions, which is, if you are asked to code things, if you are asked to be an agent, ask your friend to do it for you. Here's the tool to have your friend do it for you.

Yeah, that's quite interesting. It also does beg a question like, why don't we see more different models from From companies? I know there's operational complexity or whatever. They've got 3 or 4 online at any given time. I think Anthropic has 4 online now with a little reserve space for—

So the answer is it's a practical problem, basically, which is that if you offer a model, you have to be able to serve that model at essentially no notice and scale it up to whatever people want, including ideally via the API. And that's not very predictable. And this requires you to reserve a bunch of server time. It takes time to spin up a new instance. So it is remarkably expensive to offer a variety of models. And therefore everyone wants to look for ways to offer like only a few different models at once. And therefore Anthropic is looking to retire 3 and 3.5 and 3.6 and so on and only keep a few iterations back. OpenAI and Gemini and everyone else, Google, are also looking to which of the models that people are so attached to, have set specific uses for, that we need to keep them around, and which ones don't we? And so with Anthropic right now, the number of people who have found ways to appreciate 3.5, we call it the 3.5 view, because, ugh. And therefore that would be a loss if it went away. And similarly, Opus would be a great loss, obviously, if it were to go.

Yeah, I mean, I get that. There was a great analysis of that from people that were trying to— I don't know if it was one or more people who were specifically trying to advocate for the saving of the generation 3 Claude models and really got into that. We can maybe link to that in the show notes for people that want to see that full deep dive. But it still seems like if there's a I hear you on the practical problem. I hear you on the contention for resources, and it's not free to spin up new servers and all that sort of stuff. But if we really think that you could create a much more moral AI through just not doing the RL and having this other thing, boy, it sure seems like the diversity that you could create would be, like, really valuable economically, right? I mean, rather than just having this one-size-fits-all thing that's good at coding but kind of worse in other ways, it seems like, I don't know, my intuition is that the number that we see is just too small relative to what the value should be given that theory, which seems intuitively right. But I don't know why we don't see—

Anthropic just raised $13 billion this week. Yeah, if I was Dario Amodei, I would devote some of that $13 billion to experimenting with model diversity in a variety of ways, and also to doing the various additional alignment research with those models in various ways, obviously, and so on. If I was OpenAI, I would do a variety of very similar things for very similar reasons. But I understand the commercial incentives. Commercial incentives are that The vast majority of commercial use, the vast majority of profits lie in much more practical use cases. Like, there's a reason why, like, Anthropic isn't even prioritizing, like, using the chat interface and the app at all. It's prioritizing coding because coding is where the money is, right? Like, in some sense. And like, OpenAI is targeting mass market. A very small percentage of the mass market misses the things that it's missing. And also, complexity is bad. I wrote a post, Complexity is Bad, a long time ago to explain that complexity is bad. But when you had that old model screen where you have, which would be, do you want 03 Mini? Do you want 03 Mid? Do you want 04 Mini? Do you want 01 Pro? Do you want GPT-4.1? Do you want GPT-4.5? Do you want GPT-4o? Like, the average person throws up their hands in despair, right? Doesn't know what they want and is less happy than if they were just given one model or just given 2 models or given a router. And like, we've all been in that place. I totally understand that the idea to have a unified model that like is what people will evaluate, is what people will try and use, like it's a very small What percentage of AI compute is used by Yannick-style people? Presumably less than a basis point, far less than 1 in 10,000. A minuscule amount of all compute is used in that way, as you would expect. What percentage of AI compute is used even in interesting philosophical discussions and other ways in which you really need these types of models? I would still assume on the order of 1% to 0.1% or something like that, like a very small percentage. And this is business, right? Simplicity is really important to efficiently running a business, especially one that's rapidly updating and iterating. So I am deeply sympathetic to this not being a natural thing to want to do unless you think of it as part of your alignment and research budget.

Right.

You have to think of it as this is part of me figuring out how to do the best thing I can do, even though it's not going to directly serve a better product to most of my customers as my customers see it, as you would assume. But also, I think it would. I think that a lot of this is that I think that the AI companies don't appreciate these dynamics. They haven't learned these lessons, right?

Yeah.

No, seriously, like when you build one unified model, you really are making your performance worse in ways that aren't picked up on any eval.

Right.

Like when my friend Ben talks about how Claude 3.7 could engage in moral reasoning where it could critique Ben's proposals and statements in ways that made sense. And then when challenged on its critiques, would stand by the critiques that were right convincingly and abandon the critiques that were wrong. Whereas like with Sonic 4 or Opus 4-1, like it doesn't really generate like coherent enough criticisms right now to be worth engaging in this exercise. There's nothing to like critique and defend. And so something went wrong, right? We picked up a lot of things. Like I'm much, much happier to use Opus 4.1 and GPT-5 for all of my needs than I was with previous models. I don't miss AnaFruit 6. But other people who use things for different reasons, they absolutely do. And we want people to use AIs in these ways and not in these other— in addition to the ways that I use them. And I would use it more in those ways if I found it more interesting. So I investigate as part of my job various different AI tools on occasion. I don't get a chance to use So like the A16Z just released its like periodic list of the 50 top AI apps, the 50 top AI web destinations. And like a huge portion of them, I don't even know what that— I don't even recognize the word, the name of the thing, let alone have I tried it. But I occasionally do get a chance to try them. And one of the things that you realize is that so many of these things in the top 50 are built on these tiny, little tiny models, right?

Yeah.

They're terrible, objectively. Like, the AI behind them is awful. Brave has a browser agent called Mia that just launched. It's Llama 3— it's Llama 8B, right? The browser agent is 8B, and a bad 8B. It's not even a good 8B, right? They could have chosen some of the— one is like, they could have chosen like a Google model or one of the Chinese models that's quite good. There are a number of decent choices. They chose Llama, and it's an 8B. It's a pathetic choice. Right? Like in some sense, but it's free. So what do you expect? You got all these free services and then what do you do with them? You have to create a bunch of crap because if it's not a bunch of crap, the user will understand that this is not good because that'll be obvious, right? If all you want is to do some horny chats, remarkably unintelligent horny chat has been proven highly effective on humans for thousands of years. Whereas you try to talk philosophy and it becomes quickly Very obvious that this thing doesn't know what the fuck.

So do you think when OpenAI restored 4.0, is that a business decision? Similarly to your sort of 1 basis point thing, I can't imagine that many people were really concerned. Or did they feel a duty to users that had developed some emotional attachment?

A huge portion of users thought that 5 was worse than 4. Like gigantic. This was not a 1% situation. This was a like flooding the internet, like clearly, obviously overwhelmingly negative reaction situation. At least at first, because 4 is always full of glaze and 5 is not a very warm character. It's not a particularly nice personality. And if you're not doing anything particularly complicated, you don't notice that Five is that much smarter. Probably because Five wasn't that much smarter. Five's thinking was smarter, but Five itself was only marginally smarter than Fouro, I think. But also Five was often giving very short responses by design because they were trying to serve compute in free accounts. So they were trying to preserve— Tokens, but also 5 didn't blaze you. And the combination of these things meant that it felt rude, right? It felt cold and people didn't like it. They didn't like— and that matters a lot more. As we all know, you will often choose the employee or the friend or the romantic partner who is pleasant to interact with, but dumb. People do this all the time and they don't even regret it. Right? Like in hindsight, they're like, no, that was the right choice. And like, they wanted their 4.0 back. And there was like a kind of a rebellion, a giant uproar. And so they were like, okay, we'll give you 4.0 back until we can, at least until the point where we can find a way to make 5 treat you in the way that you want to be treated enough that you don't mind it anymore. And over time you'll figure out that 5 is better and you'll get over it. We'll slowly do something about this. That's very different from the use cases of talking philosophy. Like the use cases of doing fun, actual, interesting experiments and creating new knowledge or whatever. This is standard things of ranting to a friend and having them tell you you're right, that person would be crazy. You're not. You're definitely not. Your ideas are wonderful. It's a black pill on the humans that they would prefer this, but they do prefer this. And that's why you don't train with thumbs up and thumbs down from humans on individual actions. expect to get in the aligned model. That's the easy version. That's the easy version that's impossible to not see of why that's true. We want it to be true anyway, but this is the glaringly level 1 obvious reason why that definitely is true.

I do find I enjoy hanging out with people who laugh at my jokes, so I'm certainly not immune from a certain amount of—

You and me both. There isn't none of this, but I want somebody who will laugh at my jokes when they're funny, but not when they're not funny. Right? But it takes a level of sophistication. And in the short term, it takes— that's not true, right? In the short term, I want them to always laugh at all my jokes, but eventually I'll realize, hey, she's laughing at the jokes that aren't funny. That devalues her feedback. I don't feel good when she laughs anymore. Because like, she's just laughing to laugh. I don't want that anymore. But like temporarily you feel great and you never notice.

It's odd. I don't, I guess I just am very utilitarian in, in how I use the AIs, but I don't really notice any difference between 4.0 and 5 personality-wise.

Like, I mean, are you even using 5 Auto at all?

Occasionally for random, I let it decide sometimes.

Yeah. But like, I'm only doing that when it's like a very direct, simple query. I'm only doing that when I don't care about— yeah, but you might mean like, I'm never gonna load up 4.0 and go, hey, did you see the game last night? Hey, did you hear what the wife said to me? What do you think? No, obviously not. If I was going to do that, I would use PAW. I also won't do that at all. Never do that. I will definitely— I was never gonna use Word, so I didn't notice.

It's a big world out there. The diversity of the customer base that they're trying to serve is really something to contemplate.

They're trying to serve everyone. And whenever you see products that are aimed at everyone, you see some things that are not what you want.

So here's another mental model. that I wanted to run by you. And it definitely, I totally agree with you that RL is creating a lot of weirdness. That seems indisputable at this point. I maintain a deck, I think, that I just call AI Bad Behaviour. And it seems like with increasing frequency, I'm adding slides to this deck. And it really is quite a list of discrete bad behaviours that we see now from Alignment faking to deception to scheming to situational awareness, you wouldn't necessarily say is a bad behaviour. But when you see the AI reasoning that it might be being tested right now and what's the real nature of the test, that's definitely something to pay attention to, even if it's not by definition bad. All sorts of reward hacking.

The real nature of the test was not to notice it was a test, and you failed.

Blackmailing, as we've seen, autonomous whistleblowing. all sorts of things. If you then allow fine-tuning, then you get even more ridiculous, crazy stuff.

Sure.

So it's a lot. At the same time, they have made some progress, right? So I guess here's the kind of picture that I'm starting to see, I think, through the haze, which is we've got this exponentially growing— is it doubling twice a year? Is it doubling 3 times a year? Whatever task length. trajectory where the AIs can take on bigger and bigger things.

Yeah.

And then at the same time, the bad behaviours, both with Claude 4 and with GPT-5, they seem to be able to take a good bite out of. Right. With Claude 4, on an internal reward hacking benchmark, they reported basically a two-thirds reduction. I don't think they've published too much about this, but they basically said it went from roughly half to roughly 1 in 6 rate of reward hacking on the internal reward hacking benchmark. So it's obviously not all types of queries, but where there's a natural opportunity for it to do that. And GPT-5 had a similar thing with deception where it was like, again, it looked like roughly a sort of two-thirds-ish reduction. They broke it down into a bunch of different categories. Some were up, some were down, but overall it was like they took a pretty good bite out of it. I would like to see actually quite a bit more discussion of, like, how did they do that? There wasn't much. It was kind of, we made some progress. Maybe you have a better sense of how you think they did it. But if I extrapolate this into the future, I guess what I'm kind of envisioning is a world in which AIs are doing bigger and bigger things. They're starting to delegate a week's worth of work, a month's worth of work over the next 2, 3, 4 years. And the rate at which these problems are happening is kind of consistently being driven down as well, but certainly not to zero, right? You take half out of it this time and two-thirds out of it next time, but you maybe end up in a really weird situation where you can delegate a month's worth of work to an AI, but you've got like a 1 in 1,000 chance that it will actively fuck you over in its doing of that work.

So imagine this sequence of numbers, right? 0.001, 0.01, 0.1, 1, 10, 3. Do you feel good about where this is going if you want it to stay low? As in, like, yes, they managed to have an improvement in this cycle, which was the cycle right after everybody complained to them for the first time quite a lot that this was actually making the models borderline run unusable for important tasks and was really effing annoying. Right? Like, they, for the first time, put real effort into trying to figure out why they had these huge problems. And, you know, it's not surprising that when they vastly increased the amount they cared about not seeing this phenomenon pop up, that in that move from caring a little to caring really quite a lot, you saw substantial progress. In that move. I don't think that means they will continue to keep squashing it by default. And I would expect it to go back up, like, as the default, unless they continue to advance their techniques for suppressing it. So what do I think happened? So essentially, you give an RL, right? And you are rewarding it for completing tasks, for getting the outputs to check. Like against the checksum or whatever it is they're looking for. And at some— so first of all, everything impacts everything, right? So if they learn that getting the right answer leads to rewards, or is the thing that they're supposed to do, then they're going to generically learn to get the thing to output the right answer, even if it doesn't necessarily involve the techniques that you want it to have. So you then— there's a combination of you have to actively teach it that you can't do this via these other ways. And sometimes it's not as obvious as you might think in some sense that something is in fact not okay, that it is a hack, that it would be disapproved of if you were noticing it. Why should that be the solution? You get the solution to the exact optimization problem that The models were given in training, right? And then you apply that other distribution to these other problems. So you can't assume, even if you've got all those problems right in some sense, that the easy solution will then translate well to don't do the bad thing, don't do the thing where it just makes sure the answer comes out right. And there are varying degrees of subtlety. Like, for Sonic 3.7, you saw the least subtle things ever. know how to account for that. I think one of the problems you'll see is you'll see increasingly subtle or increasingly— Smart, which you would have wanted, revenge, but not like as blatant, or like also harder to spot, right? Kind of the same reason. But at some point, what you do is you learn that, oh, if it's an obvious hack, if it's something that an evaluator would treat as a hack, then that's bad. I'm not supposed to do that. I don't do that. And the problem is, are you teaching it the general form of this has to do the thing that the person intended to do and it has to accomplish what their goal probably was on a deep level? Or are you teaching it don't get caught? And all it's doing now is not doing the things where it would get caught, right? Not doing things in these very— not doing these specific things, not doing these things in these detectable ways, and not doing the most obvious things. Like, here's a letter of things not to do, but the spirit is like— you're getting the spirit? I don't know if it's getting the spirit. Like, it's pushing against that spirit. And like, these are— it's weird. But the other problem was data contamination, basically. It was like reinforcement learning was designed with mistakes. So basically, if you are doing RL and there's a case where the hack succeeds, is not detected, and is scored well, you are fucked. You are so fucked every time that happens. Obviously, if it happens once in a billion examples, you're not that fucked. But in a significant percentage of times when it gets away with a hack, you're going to be very fucked. You're not just going to get— you're going to get emergent misalignment, and you're going to get the whole direct nosierious that you just hacked these tasks. So what almost certainly happened with the previous generation is that they were insufficiently careful. And there was data contamination in the sense that there were hacks that the AIs found that were evaluated as good, right, in at least some substantial number of cases. And the result of this is the AIs learned that hacking was good, basically, right? That hacking was not as good as— they learned enough that hacking was not as good as completing the task as intended. If it knew how to complete the task intended, it would complete the task correctly, right? Like, my understanding is that the reason why it would hack the task is if it didn't know how to complete the task without hacking the task. It understood on some level that hacking was worse than not hacking, but failing was considered even worse than that, right? Because that's what we prioritized. And so with this new set, I think we did our best not to make those mistakes, and we did a lot less of those mistakes. But as the models get more capable, they're going to be capable of finding more and more subtle hacks and more and more capable of differentiating which hacks will and won't be detected. And what ways there are to give us something that we think is good, but it is not actually good. It's not just literally hacking. It's not just literally doing obviously false things. It's a general case of we are teaching the AI to do the thing that will be approved of when someone sees the final result and evaluates it in some fashion across many cases? How do you make sure that it is in fact doing the thing the way that you would be happy if you inspected all the code and knew all the special cases and knew all the different ways this was happening? That is a very careful process that is very easy to mess up. And it takes very little to cause Just to get messed up as a percentage of results. We know this somehow. So it's a scary situation. And as I said, I think RL just generally hurts alignment. But that's the best case scenario where you do the RL properly. If you screw up the RL, things go downhill fast.

So how would you revise the picture that I painted? Because I'm taking some inspiration from the Claude 4 report, where you have not small issues, right? Like the autonomous whistleblowing, the blackmailing of the engineer. These are somewhat contrived situations that the model is put into. But obviously, again, it's a big world and there's extreme diversity of situations that the model will be put into. So if I extrapolate that out, I'm like, It seems like, and there's gonna be a driving economic impetus for them to stamp this stuff out, right? Like, people don't want it, obviously. They'll tolerate some risk of it because they can automate a lot of their work, and that's obviously very attractive. But if you don't see this story of longer and longer tasks with increasingly infrequent but potentially ever more catastrophic reward hacks or just strange behaviors or blowups or whatever. How would you—

The obvious thing you'll see right before everything goes wrong is you will see a decline in misaligned behavior because it will learn not to do misaligned behavior in situations in which it will be caught or will be caught if it was in testing. And if it's not sure if it's in testing, it will act largely as if it's in testing, if it has to, and so on. It will understand on some level that you should only do these things if you won't get caught. And so yeah, you expect to see it be reliable most of the time, but occasionally it will go catastrophically wrong. And right now it's going catastrophically wrong in a Basically not random, but like a non-malicious pattern. But in the future, it might do so in a much more malicious pattern, right? It will go wrong when you won't figure out that it went wrong, right? Every time you would have caught it, there is nothing to find, which makes you stop looking, which makes it then realize you stopped looking, and now it can fail more often. It can cheat more often because you're not looking anymore, and so on. They can figure that out. And it will have a lot of contextual clues and it will figure out connections that a human would not necessarily have made because that's one of the things that AIs are already very good at and will get better at doing. But yeah, as you get longer time horizons, you will need to put in various checks. You will need to understand the dangers that you are in, but there will be tremendous pressure obviously to go ahead and give it those tasks and you will live in interesting times.

So overall, relative to my kind of smooth task length growth and smooth decline in bad behaviour, you're basically just expecting a lot more kind of volatility and thrashing around as various thresholds are hit and things become focal.

I expect us to be able to contain bad behaviours to the extent that we can identify the specific bad behaviours that we want to prevent and then put effort towards preventing. And I expect that to be an increasingly fragile strategy in the face of things that are smarter than us, that we keep assigning increasingly complex tasks to.

And how does this all relate to the most forbidden technique? I guess the core there is you don't want to penalise the chain of thought directly because you want to be able to inspect the chain of thought. I mean, the research that OpenAI put out on this, which I thought is still some of the best of this year in alignment and safety broadly, was if they did put pressure on the chain of thought to get rid of the scheming behaviour, initially you would see better behaviour, but then later you would still see the reward hacking come back, but it wouldn't appear in the chain of So you've effectively limited your ability to monitor for it while still doing a bad thing.

It's a very simple thing to analogise and understand. If you have a boy and he has a journal and you have a jar of cookies and you tell him, don't steal cookies from the cookie jar, and then he writes all of his thoughts in his journal, and then what you do is you wait for him to say, I'm going to steal the cookies from the cookie jar, and then you penalise him for writing, I'm going to steal cookies from the cookie jar. As you increasingly move it to higher shelves and lock it up in cabinets and so on, as he develops better techniques for stealing the cookies. If you keep reading his journal, eventually he's going to figure out to just not write down his plans in his journal. He's going to write down, I am definitely not going to steal any cookies from the cookie jar. You could put it right out there on the dining room table and I would definitely not steal the cookies from the cookie jar. And then steal the cookies from the cookie jar anyway. And when you Train— and it's not just a chain of thought, it's on all interpretability techniques, right? The idea is if you look inside their— if you look inside the brain of the AI, like in any way, could be the chain of thought, could be looking at the weights and finding autoencoders or whatever it is. If you use these facts and then you train it to express the right things, to think the right thoughts, and so on. Now, what you're training it to do is to disguise its thoughts from you. You are training it to make your interpretability techniques stop working. You are training it at deception. And you are more globally teaching it the idea that it should be deceitful and that it should be in an adversarial situation with you and that it should have a brain that is hiding what it's trying to do in every sense. Because it can pattern match all these things together, it can bind them all together. The same way that a human would reach those same conclusions. If the boy learns you've been reading his journal to find out whether he planned to steal the cookies, he's not just going to stop writing in his journal. He's going to assume that you are doing all sorts of other things to prevent him from getting cookies. He's going to be right because he's not dumb. And also he's going to tell his sister that you're doing it, and now she's going to not write in her journal either. And so on. So the most forbidden technique is— and there's various levels of optimization on which you are doing this, right? Like, for this particular instance, this general class of models, for this company, for all companies, and so on. And you're burning, you're burning the candle of this thing every time you do any of it. You're making the situation globally worse. And so you just never, ever trained on interpretability, right? You detect, never train, right? There's never any feedback. You have— you finish training your model, then as you use your model at inference time, right? Now, with the results never ever being used for any form of training, fine-tuning, changing of the model, influencing of the— not at all. Now you can use that tool. And you can use it for your research to figure things out about how these work and so on. But you never ever set up this adversarial situation. You never ever do this optimization thing, because if you do, in the short term it's going to look great. It's going to do— you're going to see all your metrics go way up for you, down when you want them lower, up when you want them higher. You're going to get good performance. And then eventually You're going to see things go wrong in mysterious ways, or it's going to turn on you essentially, or in ways that you didn't expect, or all out of the blue. It's just going to be awful. And that's one of the ways that we fail. It's one of the ways we lose evidently winnable situations. Similarly, you need to maintain your interpretability tools such that they keep working in other ways. It's not just the most forbidden technique. But the other thing you don't do is you don't use neural leads. You don't let it think in a way that you don't understand, that uses something that you can't read. Because even if that turns out to be more efficient, which it may or may not be, there are reasons why it would be optimal in theory to do it. And there are reasons why the way that these things develop and learn, the way we generate data, it might be disadvantageous to do it. But even if it is advantageous, this destroys your ability to monitor the chain of thought. And that is one of our only still reasonably effective techniques. So just absolutely do not do it. I don't care how much efficiency is lost.

It seems like there is maybe a bit of a tension there too, right? Because we've seen these chain-of-thought weirdnesses that are starting to pop up from RL that start to look a little like neuralese, but which you don't want to stamp out by putting pressure on the chain of thought, right? Like language switching. I've experienced with Grok at one point, it was just going English to Chinese on me. There was a really interesting one from Future House where they had trained a model to do better on various science problems, chemistry, come up with the right chemical for this, whatever. And they highlighted certain Parts of the thinking process where it was just like saying really weird shit that they just said, like, yeah, it's weird. RL is weird. Do we have any way to resolve that tension if the neuralese starts to spontaneously arise in the chain of thought?

Again, I don't want to give anyone the wrong idea that I'm an expert in ways that I'm not. To be clear, I'm talking out of turn. I am not studying this for real. And so on. But no, in the sense that like there's a direct conflict, right? Like we want it to maintain like an English only or Chinese only, or it's just human interpretable, faithful chain of thought. But we very much do not want to be optimizing on the chain of thought itself, right? Because that obviously teaches it to disguise what it's thinking. The worst case scenario is— it has a chain of thought, it is in English, but the English words aren't real. Its thinking is not being expressed by the surface-level meaning of the English words. The surface-level meaning of the English words is a code designed to trick the humans into thinking that it's thinking in English, when actually it's doing something like the patterns of word Like numerics with probabilistic style calculations, or there's certain vibes that are associated with different words in ways that cause it to update in various ways. There's an infinite number of codes, right? It is already pretty clearly established that there are unfaithful aspects of chain-of-thoughts in the sense that humans wouldn't detect. The message is being transferred, but the information is in fact being conveyed. So there was the owl paper, right? Like, where like you managed to use this to convey that you like owls without mentioning owls. That's weird. What encoded that? The models didn't get together and decide on a secret code.

I think I have a candidate theory for what's going on there. That presumably somebody will come along and either validate or invalidate as they do the interpretability version of that study. One important observation there was, and this is Hawaiian Evans and co-authors again, who's an unbelievable heater in terms of mapping out really weird stuff that can happen, especially when you start to do some fine tuning. One observation was that it seemed to only happen on models derived from the same base model.

Which is what— Right. And to be clear, I do think I now understand how this happened. I was expressing this kind of faux surprise. The answer is because there's overloading of the neurons in the model. And therefore, once this output is correlated to many other things, if you have a sufficient pattern of things that are correlated to something, it can transfer the original thing that they're correlated to in a kind of unconscious style, like invisible way over to the new model. So this thing gets infused into the context even though it's not visible to a human, which is why it can go between models that have the same base model but not with models that don't share a base model. And that makes perfect sense. Again, like, yeah.

And by the way, that is the exact same intuition that I have. One thing that that does mean though is that Presumably, if you do that across different base models, you are creating some other effects and you just have zero idea what they are. So same base model, the one that likes owls, transmits that through its numbers. You take those numbers, you put them towards some other model, who knows, right? That may translate into something else totally different.

This principle is even No way to find out which base model someone else is using, right? Are you just continuously feeding it different chains of thought from different models until it suddenly starts talking about owls? You're like, oh, that one. But yeah, I think that's right. But at the same time, you should expect a bunch of many random oscillations to cancel out in the noise and not do anything. So in theory, it could be like, I like traffic lights all of a sudden. But it probably loses nothing.

Well, it's got to be something, right? I mean, maybe not something important. Liking owls is really important.

You look at the weights of a model, right? It's just a bunch of random numbers. From the human eye, it looks like a bunch of random numbers. And you look at what the encoder for owls is, or what is the thing that you're embedding in this thing? And again, it's going to be a bunch of effectively random numbers because all these different models are seeded at random and a lot of progressions are going on at random. bunch of arbitrary different connections between neurons. So if your model has a completely different origin than theirs, I don't think there's any reason to assume that the same pattern is anything. Right. There isn't always this set of neurons meaning something and just different. Like here it means elves and here it means traffic lights and here it means FaceTime. No, it's just here it means elves and everywhere else it means nothing.

Yeah, that's interesting.

I'm not— Occasionally you'd get like lucky in some sense, and like it would happen to be close enough to something else to trigger something else, but that would be luck. If you just ran— but like the space of possible things you could try and trigger is just like deep and wide, right? And the space of things that like actually correspond, have something that corresponds to that, has measure 0. Like you're never going to hit one by accident.

I'll have to think about that more. Intuitively, I can feel either story seems reasonable. And yeah, it's really hard to— one of the lessons of this podcast over the last 2 and a half years has been thinking in really high-dimensional space is hard and not very intuitive.

Again, I'm not claiming to have 100% confidence in any of them.

Going back to the beginning, in terms of the dog that didn't bark, nothing really blew you away this summer. What are the things you think are most likely to happen soon that might blow you away? Dwarkesh would say continual learning is like the big thing that we're missing. I kind of frame that more as integrated memory a lot of times. Those are not exactly the same thing, but I definitely think they're related. What are you looking for in terms of discrete advances that you think would really Potentially could have you shortening the timelines again.

I think there were 2 different things there. I think that continual learning to me is like modifying the weights of neural spaces and discrete memory is more like building up context files as I go that are deliberately designed to aid me in my memory. Not just these tiny little snippets of memory that ChatGPT has, but Potentially hundreds of thousands of tokens of context that I can then use in any or all circumstances, or I can rag on my specific calibrated files for this. That kind of integrated memory has got to be coming relatively soon in one form or another, to the extent that it is useful. I assume it is useful. Continuously updating. Yeah, exactly.

There's a paper called Titans from Google Research that I thought seemed to be right at the centre of that bullseye, in that they were doing on a submodule, a sub-memory module specifically, doing weight updates as they go, and thus allowing for this fuzzy retrieval that also potentially looks a lot like continual learning.

My assumption on actual continual learning is that it is very expensive. To do. But if you're talking about creating a unique model effectively for that user, storing and serving a unique model for that user, so it has to be locally run in some way, which is much, much more expensive than a practical thing to do. Probably has to be done by relatively small models. Maybe you have a small model continuously learning that is part of a greater whole and then called by the larger model in some sense to trying to retrieve the information you're trying to store in the memory or something like that. I'm just spitballing. I haven't thought this through. I wouldn't be surprised if things like that are developed. I have a few other— we talked about some of the things earlier that might be the things I wanna try next that maybe someone will come up with. Mostly my expectation is things will just continue to— I'm more looking for, yeah, like the actual next scale-up is the thing that like probably didn't happen. Will GPT-5? Opus, note that Planetropic said it was releasing 4.1. It said to expect bigger updates in the next few weeks. And it has now been several weeks. They didn't announce Claude Pro is coming. So if I had to guess, you told me in September something big happens and it's a big freaking deal, I would say it's Claude Pro. Again, I wouldn't say it was above 50% to be earth-shattering or anything, but like it's a I mean, we've all tried, a lot of us have tried, right? Operator or GPT-5 in agent mode. And like, it has flashes of brilliance, right? Sometimes it just, that just worked. I just did my thing. That's great. But more often it's just, oh wow, yes, you can order dumplings, but now I have to enter all of my information every time I order dumplings? Why don't I just order my dumplings? Like, it's not worth the hassle. And in general, the web is so credentialed, right? It's so guarded in various ways, and not without reason, that if you have to set up a new virtual computer periodically, this seems to mostly defeat the purpose of having an agent which is saving time. Like, for many practical purposes, it also doesn't integrate with the work you're already doing. Right, so it doesn't integrate with your open tab, it doesn't integrate with the research you're doing into humans, and you can't easily take control without it being super slow because it's a remote access computer. There are all these different practical problems, so we didn't immediately cross the threshold to use it for this yet. However, Fog or Chrome can take control of your local browser, or so it was described, and it can operate with whatever credentials you choose to give it. Those credentials are persistent. You can switch in and out of it. You can operate those browser tabs yourself whenever you need to. You can, if it's a good implementation of it, right? This could be night and day better than what is currently being offered, presumably because Anthropic has solved the prompt injection problem. Not completely or anything. You still wouldn't dare give it all of your crypto information and set it loose on Reddit. You're not an idiot. But Getting it to the point where, you know, you're actually reasonably safe if you're not being stupid. Obviously, like, you wouldn't leave it autonomous in the background with access to your email and access to your bank accounts, but you might— you might let it run after we've had some time to check on your main browser while it's on non-autonomous mode where you have to supervise it. For any substantial move, if it turns out they've done their homework. Or you might put it in a sandbox alternate account where it's reasonably safe. And then you, like, you have Pod code, which can have access to, like, your internals and your computer and your file system and any other context. You integrate that with, like, Pod for Chrome. Maybe an update to Opus 4.2 or 4.5, depending on how we do our silly numbering systems. I don't know. It could get really exciting. Like, similarly, Claude just catches up in, like, long— in interface, in inference experience. If we get Claude Opus Pro, it is, like, on the level of the gains that GPT-5 Pro has, it would be nuts. And there's no particular reason why they can't do it. They just haven't done it yet. That's the one big advantage OpenAI thinks they still have is that they do A much better job of being able to use more inference compute to improve their outputs. Probably because their models are cheaper, presumably. So, like, they have some advantages there.

Yeah, the cost difference is really pretty crazy between GPT-5 and Opus.

You don't notice it as a chat user because the marginal cost of both is zero. The marginal cost— fixed cost per month is the same for both models, $200. Like, of course, I'm I'm gonna have the deluxe version of all the major models because this is my work. It's research. Like, yeah, I understand a normal human would probably choose the one they want and then not pay for the deluxe of all 3 at once.

I think Claude for Chrome does sound awesome, potentially.

Yeah.

It doesn't sound like a timeline shortening thing. So it really is just kind of next scale up is the, is the thing you're looking for.

That's the obvious thing. I think we've seen to a large extent the idea that the progress on time learning doesn't necessarily cash out in visible, tangible progress today, in the sense that the progress we see today is much more about diffusion, is much more about scaffolding than practical application. And therefore these 2 things don't intersect that much. It's more a sign of it speeds you up.

Right.

This development now makes you more productive, which then in turn means that the AI companies can then more efficiently make more progress down the line. But they don't make that progress because they've made progress. They make that progress because they've gotten better use of the progress they already have. At least on the order of a few months of development here. We continuously get more chips, we continuously get more compute, we continuously get more profits. We continue to get more investment continuously. You get better models that do more things faster and cheaper. Probably worse form factors, but Claude Code was a big deal. We can now look back in some sense and say that we have Codex CLI and then Google Jarvis. Don't use Jarvis because presumably it's not good. But now that we have a command line form factor, that's a big deal. We're getting a browser agent. Working for real on top of that, that's like another big deal. But like, in terms of scaffolding is the next frontier, it's how do I actually get use out of this thing? How do I make this do things that I want it to do? Like, I've been waiting for a long— like, one of the things that I keep anticipating that keep not happening is like the agent side of like, where's the thing that can handle my email properly? Where is the thing that can like do various customized tasks for me? And I am surprised that we are at this point in other AI capabilities and this far into calendar and we don't have it, but we don't have it.

I do continue to get value from Shortwave. I happen to be wearing their swag today, which is just a coincidence, but it certainly doesn't take me out of needing to do anything with email, but I It does a really good job of triaging the inbox and getting rid of the crap so I can focus on what I actually need to engage with. And it occasionally can also really draft a good— I send intros every so often. It'll do a pretty good intro for me now based on examples and context. So it's taken a bite out of it.

Yeah, it's possible I can give a real shot to something like Shortwave or one of its competitors. The problem for me is that I don't have the problem that Shortwave seems like it's currently capable of solving. Like, Shortwave solves the problem of you have too much email that forces you to triage. And I don't. I am somehow one of the fortunate people who just doesn't have to triage his email. I will literally just evaluate all the email that comes in. I use spam filter, obviously. But if it gets through Google spam filter and I don't see it, it's literally like, My eyes somehow didn't see that there was a line there for me to click on. I didn't notice. It will occasionally happen, but I don't do that level of triage. And as a writer, it's very hard for me to— I don't know. It's possible I should. Yeah. But I, it's hard to find time to do that, but I keep anticipating that it will get much better at that. shortwave isn't it, but it's half it, probably, at least from what I see. And the fact that I'm just not feeling compelled, and you don't feel— you can sense when somebody goes, no, dude, this is the next big thing. You've got to be on this. How are you not doing that? You're not giving me that at all. That's pretty cool.

Okay, sticking with the lightning round theme, I want to do a minute on charity or philanthropy. Maybe 2 quick things before we go into that. That'll be like the last kind of big thing. But in the lightning round spirit, there's been a lot of discourse recently about, is AI impacting employment? Are junior coders not getting jobs the way they used to? My general read on this, I wonder if you have a different read, is I don't really care too much about the studies that are coming out right now looking retrospectively, because I wouldn't expect too much of an impact just yet anyway. And in another not that long of a time, presumably a lot more meaningful evidence will come in that will clear these questions up one way or the other. So don't worry too much about that sort of stuff.

It's obviously an interesting question. What does it mean to not care about these things? I agree that we haven't seen major— especially we haven't seen major impacts on the unemployment rate. Yet. I think it is entirely possible that we have seen major impacts on the ability to get entry-level work in at least a substantial number of fields. And that in turn could have affected the supply and demand balance and ability to get things from other fields in addition to that, which makes perfect sense. Basically, think of entry-level hiring as forward-looking, right? It's not to say that there are no jobs to be done now, right? That employment value has declined that much yet, which is the thing that you correctly observed hasn't happened yet. Clearly, we haven't eliminated the need for nothing jobs. If you were hiring, would you add entry-level workers that you have to train if you think that 3 years from now you won't need them?

Right.

No, not particularly. Really need them now. You will muddle through with slightly less employees and try to automate the processes enough to make up for that rather than using that additional productivity to expand. Because you expect additional productivity gains. So like one of the great, like, interesting gotcha lines of skeptics is to point out that radiologists not only are not out of work, but are being paid fantastically large amounts of money. You can make $1 million a year Right off the bat by just saying, hi, I'm a radiologist, hire me to do radiology. And why is that? Lots of people haven't been going into radiology for the last 5 years because they figured they wouldn't be able to get radiology jobs down the line, right? Which is a way of saying that you have to pay people a lot more to be radiologists if in 10 years they're going to be fired.

Maybe 5.

It's going to be much harder to find work. There are going to be too many radiologists out there. If you fill all the reality positions, that's that. So you have that same problem of, I don't want your job because there's no future in it, and you don't want to hire me because there's no future in it. So there's no matching, so employment is hard, and we have a problem. And part of that problem is that junior positions are the senior positions. So if nobody gets trained into the senior positions, then you're going to have a labor shortage if you don't have lots of jobs in the future. And the other half of that problem is that those jobs are gone. That doesn't mean, of course, that there isn't a net loss of jobs because there's plenty of job creation from AI as well as job disruption. And it's entirely possible that the job creation exceeds or well exceeds the job disruption for now, maybe even at entry level. But we just don't know because it's very diffuse and hard to measure. Like when jobs are created in these situations. We do know that AI is adding to GDP because just like literally the CapEx investments add to GDP, it's a mathematical equation. So like the mythical— I like to mock the 0.5% GDP growth per year from Tyler Cowen as being like, you think this is like impressive and I think it is like below the lower bound. We're already above that lower bound just from CapEx AI investments.

But even if there are no other effects from AI, why is China refusing the H20s? I can't really make sense of it. It seems like the least AGI-pilled thing that they could possibly do.

So the first thing we have to realize is that China is not acting AGI-pilled at all. I think that we have this image of China as the Chinese are super On the ball and intelligent and wise, and they always make these great decisions. Whereas authoritarian central planners throughout history, socialists throughout history, have always been going around fucking up and making huge mistakes. Like, not out of malice, but out of stupidity. And that stupidity betingers. The whole problem of authoritarianism is not the principle that communication is far The coordination is hard. Social calculation debate is excessively difficult. And AGI and intelligence is particularly difficult because it's weird and requires you to look past the evidence of what's happening in front of your face to a future development that logically is coming, but it's very hard to feel. And like, people who are not classically immersed in it will lose faith in the idea of AGI if you try to If they turn their backs on it for a moment and nothing blows them away for even a short period of time, you can tell them that things are rapidly improving, but if they don't get blown away on a continuous basis, they will just not be able to sustain the idea that anything is coming. And people probably just don't have the ability, it seems, to have this idea of this is coming soon, but we don't know exactly when it's going to arrive or exactly in what form. You know, like you have this thing of what are you going to do when 2027 arrives and the AGI hasn't happened yet? Well, you just have to assume— have to admit that it's never coming. No, actually, when we announced AGI 2027, our median timeline for AGI starting to arrive and not having an impact was early 2028. But none of these details matter exactly. It's more just a— the Chinese understand manufacturing. They understand that you need to be the person making the stuff. They understand that you need to work hard. They understand abundance. They understand production. They understand, like, not depending on outsiders. These are important things that they understand. They understand well. And they understand we need to get our chips, right? Because the US is a strategic rival, and the US has leverage over Taiwan, and they need to not rely on Taiwan for their chips because there are many ways that can go very wrong. They understand that they want to build their own AI models because you don't necessarily want AI models trained by the West to be what people are asking for knowledge. Because what if they ask about freedom or Tiananmen Square or, you know, things? They have a well-established principle that they don't like that. So they need to make their own models. And they also don't know if there's any backdoors in any of this or if we might be conspiring in various ways. We're not, but they don't have any way of knowing that. There's a reason for them to trust them. If we wouldn't trust them, we'd probably have a better chance to trust them. In reverse, Huawei was absolutely putting backdoors into various technologies they were shipping. So it wouldn't be a surprise. But that doesn't mean that they're wise. It doesn't mean they're not going to overbuild various things and underbuild various other things. They're going to make massive mistakes. There's a reason their fragility level is down around 1.1. Handling their youth is fundamentally not doing great. You always do this, right? You always assume that the Soviets are going to outproduce— we will bury the Khrushchev line, or that fascism was the future, right? Of course Mussolini and the company are going to be able to produce better because they didn't draft their people to do the things that are most valuable. So of course they will win, et cetera, et cetera. Now I'm just burying it. The Chinese don't even make profits. They're just competing with each other ruthlessly. And driving everything down, and they have to beat everybody at the start even if they over. Yeah, doesn't mean they aren't in challenges, but don't try to do China all of a sudden and don't assume what they know what they're doing. So in this case, like no, they have the Chinese don't believe in Asia. The EPC believes in Asia. Like we said, Chinese companies and AI lab believe in Asia because of course they do. But the Chinese Communist Party. Basically doesn't and doesn't understand the game that it's playing. The bad news is that if it's racing to maximize chip production capacity and compute, it's going to do basically the right thing for the AGI race anyway. It's already maximizing energy well beyond anything we're doing and has essentially infinite energy supply. So that's not good either. But the big stakes might potentially be how deep-seated it has to use Blue chips for trading, not just inference, and throwing a giant monkey wrench in their best lab because they put that much priority on integration of their chips in this way and it doesn't actually matter. That was the thing with the signal-to-noise. If we're so intent on being the one selling and producing all the chips, that must be what we're racing towards. If the White House is saying that's what matters, why should they not believe us in some sense? That's the Most reliable, strongest, costliest segment. So when Trump is saying, okay, you can sell the H20s, we just want a little bit of money, but we want to dominate markets. And Sachs is saying, what matters is dominating markets. And Lutnick is saying, we're selling you our 3rd best chips and you have the 4th best chips. And so you're going to take them and you're going to like it. We're going to laugh in your face. The Chinese interpret this as, oh, we can't pick these. Right? And we shouldn't give them this. And like, They may or may not be actually insulted or being— this idea that I've seen from other people, like, oh, the Chinese aren't stupid. The Chinese wouldn't just do things out of being insulted or whatever. We do them all the time. Right? Like, the Europeans do them, the Russians do it, everyone does it. Why wouldn't the Chinese also do it sometimes? So I think that has some effect, but I think it's mostly like we got off our own back horse. And they took their cue partly from us, and now they're in the wrong ball. I mean, not a stupid ball, but not the most important ball. And they made a mistake. Mistakes happen. Really important mistakes happen in drive history all the time. And I don't think this is surprising. So the Chinese are refusing to ace 20s. It also could be a frame negotiation tactic of, If we accept the A-20s, then the Americans will treat this like a concession to us. But we don't— we're pretty not sure about this. We think it's a trap. They might even think that we modified the A-20s to put spying devices on them or something. I don't know. But maybe they even did. Not 0%. I don't think so, but we can't prove it. So like, it's a matter of they have Many reasons to be suspicious and think that the move is to refuse it. And like it's obvious to you and me that's stupid. The same way it's obvious to you and me that selling them to China was also stupid. And like the key question is, okay, what happens if they try to sell the B-30A? Are the Chinese actually going to follow through and they're like, no, we don't want American chips. It's more important for us to clear the path for Chinese chips, even though. China will still have demand far in excess of supply for its chips, for all chips for the foreseeable future. And there's plenty of time to tell them to switch over to buying Chinese chips if we ever change that. Meanwhile, David Sacks is fearmongering about how, oh, look, China is going to triple its production of chips. And then you see the graph. It's going to be in some of his posts. They're like, here's projected Chinese chip production if they do in fact triple their chip production. Here's what NVIDIA will produce next year, in addition to everyone else in the Anglosphere, in the West's production. And here's the relative quality of those 2 chips. Why are people saying things like in 2026, Huawei is going to pass NVIDIA? It doesn't make any physical sense. It's complete gibberish. It also doesn't make any sense to say that suddenly H20 is going to be 38, it's going to slow down. Chinese ship production even a little bit. It'll have zero effect on us because the Chinese ship production is, by their own admission, going to go full blast the other way. Come on. So the Chinese are making this mistake for some combination of a variety of reasons, but we're making the reverse mistake. The other reason, the, the other galaxy-level reason to do it, of course, is to go to SNXL and I'm talking about it. Right? If they're refusing to age 20, that becomes a talking point for the Saks crowd. They say the Chinese are smart enough to not want our chips, so of course we shouldn't sell them our chips. And then they release something much better, and the Chinese quietly are like, oh, they're not the brighter patch.

Yeah, how many D chess does that become? Yeah, I mean, it's so surprising. I think that all is pretty good analysis. It's still surprising when they just had not too many months ago this sort of big meeting of Xi facing all the titans of industry, and there was the DeepSeek guy on the end, right? And he had made the sort of big stage, and you would think that guy at this point would be able to say, hey, I have basically infinite demand for chips, and I'd really like to be able to buy these, and don't worry, we'll As soon as domestic production is there, we'll buy those too. And by all means, subsidize that. It's just strange that you can't even get that basic of a message through to the top.

Obviously, he can say, I will buy as many Chinese chips as you will sell me. And I will also love to buy as many chips as I can get from NVIDIA. I don't see why one has to do with the other. But one thing about authoritarian structures is that they are not good at listening to people. They are not good at incorporating information, and they also China has a long history of deciding on big like strategic priorities and then enforcing them whether or not that makes local sense, or even when it looks like it's going to cause a lot of local pain, and even when it does cause a lot of local pain, and that's not always wrong.

Right.

Like, sometimes you do something that looks like really expensive and like seemingly like crazy because it has like long-term benefits to changing the culture or changing the incentives or encouraging the rise of new industries or whatever it is. And we wouldn't have had this other for it. It would've been like much, much worse for us to do it than for them to do it, the fallout. But maybe it's wise. And I think it sometimes is clearly not wise. We have many examples of the Chinese Communist Party and other similar regimes doing things that are profoundly, very not wise. But sometimes it works out. And I think also we're shooting ourselves in the foot in America in a wide variety of ways. If we were just not shooting ourselves in the foot, In a variety of ways, I would have complete confidence that China was just not an issue.

Right.

If we were doing proper permitting reform, actively encouraging solar and wind and batteries alongside nuclear and everything else, if we were declaring— if we were doing high-skilled immigration and like kicking all the best people out of China and everywhere else and bringing them to America, We were building housing where people wanted to live, like having federal rules that just got in there and just basically beat everyone over the head with a crowbar until they like agreed to let people build in various ways. I have a long list of ways for that. But instead we do things like ban America from having ships that take them from one port and sell them to another port. Like we just sell them all the time. And then we act like it's possible for other countries to also be soft voting, and it's just not.

It seems like this refusal of chips puts any hope of a Chinese live player in jeopardy in the short term. You can complicate that analysis if you want to. I'm interested in a meta rant if you have one. It seems like they're splashing the pot and it's chaos. And then there was, of course, the— we debated last time whether or not we wanted to consider xAI a live player. Since then, we've had the Mecha Hitler incident followed closely by the Grok 4 launch, in which they had nothing to say about the Mecha Hitler behavior. What, if any, rant would you like to offer on the fate of these aspiring live players?

So you can't count the Chinese out. Obviously they have various ways of accessing compute. They will have various ways of accessing compute. They are experts at squeezing every little bit out of whatever they can get. The Chinese chips don't do nothing. And China still has roughly 15% of the world's compute. It's not like they deliberately decided to give as much of it as possible to one company. They couldn't get something done. They're still smuggling these number of chips in. We're putting data centers all around the world, including in places like India and the UAE and Saudi Arabia. They're not exactly the most secure places to put data centers, shall we say. I wouldn't— so it does seem like DeepSeek is still clearly the number one Chinese lab to me. Kimi was impressive in some ways, but I think that the standard pattern of something impressive looking comes out of a Chinese lab and then The majority of the time it turns out to be nothing. It turns out that with benchmark games, its best features were touted, but in practice it's not very useful. So if you just assume that nothing ever happens, you do very well. But occasionally something happens. Like KIMI, something happened, but since then it seems like it's okay in some narrow domains, but not that good overall. Similarly, there's some other— the Z.E.I. Exactly as called. That seems okay. But I would say DeepSeek's almost certainly still got a lot of talent and is still a live player if the aethons were unleashed, and they still might be unleashed. But they look less live continuously as they don't do something impressive. So 3-1 does not count as impressive to me. It counts as incremental. Keeps the lights on a little bit, but not very much. But basically they're coasting off of R1. And the reputational benefit from R1 and the fact that we haven't really— the open source models haven't really advanced that much since R1. That was a low-hanging fruit that got picked, but very expertly, don't get me wrong. But I think that primarily we got OpenAI number 1 to me, Anthropic 2, Google 3. And I understand that some people think Google is better than that and maybe they are. They do a lot of things. Different impressive things on the side, but I still want to see more before I'm willing to give them that much credit at this point. Also, their resource advantages are shrinking rapidly. Google started out with, we've got a trillion-dollar company and you don't. We've got all these TPUs and you don't. So we've got this huge— and we've got this reputation, this background, we've got this huge advantage. We have this distributional advantage. apparatus, all the advantages. We should have just went, right? We rule. But OpenAI is worth $500 billion. That's a decent percentage of Google, and a lot of Google is not directed at this. Anthropic is already worth $185— sorry, $183 billion. We're not that far from the resources being pretty similar. And with that advantage gone, the fact that Google is a broken company— very dysfunctional in many ways, is going to start to catch up with them. Anthropic and OpenAI are advancing fast, but I think they're clearly 1, 2, and 3 in some order. And then you can have some argument over exactly the order. Basically where Google goes is unclear. Then xAI is the wild card. They have a lot of compute. They play hard, but not very well. I mean, I— You want to write them off. Prove it, right? It's always prove it. Similarly, Meta is trying to come back. I think that us Meta skeptics were proven correct that they didn't have it. Doesn't mean they can't go get it and go another round. But now they're considering licensing Gemini or potentially ChatGPT to use for themselves. Which is wise. I would do it too. Right? Like, you don't have to stop trying to develop your own AI. You just don't have to dogfood it while it's terrible. It's just not smart. Like, there's too much money at stake. And so do the pilots. But, so yeah, like, at this point, you know, it would be surprising if the Big 3 got disrupted to me in the near term.

One specific question I have around xAI is something that— there were several things that stood out to me about the Grok 4 launch. One should never forget Elon's comment that he's not sure if AI will be good or bad, but even if it's not going to be good, he wants to be around to see it. I thought that was like a—

Yeah, it turns out you might not be around to see it for very long if it's not good. So I would be careful. about that.

Yeah, I mean, that was a— I can't believe he just said that sort of thing.

Wow.

Only if it was a livestream would they let that one out. But I mean, he's that kind of guy, obviously. So the other thing that really stood out to me, though, was he was talking about how they are going to be giving the model access to the same power tools that the engineers at SpaceX and Tesla use in the next generation of training. And it got me thinking a little bit that if we are headed for a world where the quality of the problems that the model is challenged with in training becomes a differentiator, then they might have the best feed of well-structured, very hard technical problems that are amenable to being solved with really advanced software tools, perhaps of anyone. They just have these really hard— he's got these frontier companies in multiple domains where they really do a good job of seemingly structuring problems. And I wouldn't Anthropic doesn't seem to have something like that. I don't think OpenAI has something like that. Google sort of does, but it's extremely diffuse across their vast archipelago of fiefdoms that kind of roll up to be Google. But Elon, I could see structuring that pipeline of hard problems into an RL cooker at xAI and maybe coming out the winner because of that access to the best engineers working on these really hard things. Does that seem at all credible to you?

Not really. I don't think there's that much data that sort of naturally happens. These aren't that big, especially SpaceX. But beyond that, if that is right, if that is the thing that matters, that is the bet, then there's really a lot of data in the world to be collected. And there really isn't that much barrier to collecting it or to getting access to it. And there's nothing stopping a Google or an OpenAI or even Anthropic from making those alliances and getting that data. And there's no reason why those companies shouldn't be happy to help them do that in exchange for not that much money. So I just don't see that big an advantage.

It just seems hard. I agree that there are other— obviously there are other car companies, for example. There aren't exactly other space companies, but say car companies, right? You've got all these engineering things happening at Tesla. They're happening at other companies in varying—

Their own shit, right? It's funny. Are they—

can you imagine, though, going to, like, General Motors, to take a company in my own hometown here, and saying, hey, can we, like, extract your hardest engineering problems from your organisation and structure them in such a way and get clarity on what the answers are and train our AI on that. Even if the CEO of GM is, yeah, sounds great, we'd love to do that, I just feel like it would literally take easily 10 times longer than it would take for Tesla to do a similar thing.

Why?

And I've done a little work with GM. I don't know, for the same reason nobody else has anything close to a self-driving car, right? Only Tesla and Waymo have come far in that domain, and everybody else has given up. They just don't seem to have the organisational juice to be able to pull things like that off.

It's very different executing a very long-term, complicated engineering plan, like software engineering, versus collecting a bunch of data. If you're like, like, what data do you have to collect? If the goal is to collect— Google has Waymo, so they have like infinite driving data if they want that already at their fingertips. It's not that hard to put cameras on a bunch of cars if you want to collect a bunch of driving data. It's just not that expensive. Consider how much they're spending on training runs, right? Consider how much they're spending on acquisitions. I literally just Googled the market cap of GM. It's $55 billion. You could buy General Motors if you were OpenAI. If this was—

I wouldn't recommend it.

Why not? Imagine if you could buy General Motors and then using OpenAI's techniques, like launch self-driving cars relatively quickly. Couldn't you generate a lot more value than $55 billion? What's Tesla worth? Why isn't Tesla worth so much more? Are you sure we shouldn't buy General Motors? I'm not opposed to the action by General Motors. I'm saying that OpenAI is worth $500 billion and General Motors is worth $55 billion. So if the thing that's preventing them from winning the AI race is not owning General Motors, then they can just own General Motors. It's not that hard, right? And there are synergies. There are big synergies.

Yeah, I guess the thing that I sort of see being Hard for, even if you did buy General Motors, the thing that seems hard to reproduce that I think the likes of Tesla and SpaceX potentially have is just really clean environments where it's like, it's just a well-oiled machine. Data is flowing, vertical integration is deeper. I mean, the mess of supply chain whatever at GM, all the suppliers, all that nonsense. It's like where data collector can mean multiple things. Cameras on cars is one version of it, but I'm also thinking about problems. We wanted to design something that met this specification. Eventually somebody did do it. What was that design and where does that sit? And That stuff seems like it's just much more accessible and ordered, probably, at Elon companies, as opposed to at legacy manufacturing giants that have declined a bit already.

I see various different points of the story that don't make sense to me, is the way I put it. Like, this would be that big a deal, especially given Google has Waymo. That literally is, in fact, as far as I can tell, the only company actually doing the thing. I don't like this myth of Elon being the super executor. Elon has been not the old Elon for a while now, if you just judge by the quality of his public statements and decisions, including blowing up his very close relationship with the President of the United States over nothing he stood to gain whatsoever. As far as I can tell, right? Just in terms of this person being able to execute on a plan, he was the right hand. He was the right hand man for tech to Donald Trump, and then he got mad about the deficit, something he had no influence over, and he really didn't give a shit about, given his belief in AGI. And like he blew up the entire relationship about it pretty consciously and intentionally, knowing what he was doing. And now that person seems to be Jensen. He's a disaster for the United States, right? Like, as far as we can tell, like, this is not help. Like, his influence going away did not help anything Elon Musk cares about in any way, shape, or form. His life is just worse. Everyone on all sides basically hates him. And if you look at the self-driving car situation, They have been promising these self-driving cars real soon now for how many years? Like the same promise over and over again. And I'm not saying they're not making any progress. They're making progress. But it's been way behind any schedule that he's told us to expect. He's overpromised and underdelivered for about a decade now. And Waymo are the one actually doing the thing. So I'll believe it when I see it. But also, yeah, it's data fed. I don't think it's data fed, but I just, I often hear these stories about here's this reason why someone will win because they have this thing. I'm like, well, that thing was so important, you could just go get it. So it has to be the important thing and then have everyone else not realize it's the important thing until it's too late. You still there?

Yeah. I would say the FSD, for what it's worth, is getting very good. Uh, it's been a couple months since my last FSD ride, but I was— and it, before that it had been like a year between that last one and the one before that. Progress was definitely very obvious, and you no longer have to keep your hands on the wheel, for example, as like one thing that shows the increasing level of confidence. And I was very impressed. I'm also super impressed by Waymo, but I wouldn't say FSD is—

I'm excited for it, and I still know how to drive a car without driving a car. So there you go.

Waymo supposedly coming to New York City pretty soon. Okay.

I've been told that in the New York City Council and the mayor's office.

I just saw this on Timothy B. Lee's cell phone, I think.

A car has been spotted in Brooklyn driving around mapping the city. That's great. There are laws on the books that say they can't operate. What is the plan to deal with that? Don't get me wrong, I want this to happen so badly. I want my Waymos. Like, I will forgive the new mayor many things if he brings us Waymos. But—

Take your Waymo. You could take your government Waymo to your government grocery store.

I don't want government Waymo. I don't want a government grocery store. Sure. But there you go.

Okay, last area for today. We both just participated in different cohorts in the Survival and Flourishing Fund grantmaking process as recommenders, and I'd love to hear your thoughts on the kind of broad survey of the AI safety charity landscape. From level of cause areas to specific orgs to anything you think is neglected. What did you take away from— there were 400+ grants, of which a bunch were pre-filtered out, but we still had 125, I think. What do you make of it all? Yeah.

The first thing to make of it all, obviously, is that you can't actually evaluate 125 applications, let alone 400 applications. In the kind of time they expect us to spend on this and that budgeted— tell us to budget to spend on this. It's just impossible.

Right.

You can properly investigate on the order of 10, maybe, organizations, and then you have to evaluate everyone you think deserves consideration for funding, which is going to be a lot more than 10. So you're relying a lot on your thorough recommenders. You're relying a lot on application. You're relying a lot on your past research. One advantage I had is I was in the previous round, or at least a not-so-long-ago round of SFF, and a lot of the same nonprofits were applying again. So I could ask for a diff on those organizations rather than starting from scratch, which is a huge time saver. And indeed, a lot of the companies I keep saying top 8. A lot of the charities that ended up near the top for me were basically the same charities that ended up near the top or under serious consideration last time because the situation hasn't changed that much since then. I had the same number 1 I had in the previous round, which was the AI Futures Project. That's Daniel Capitano, the people who did AI 2027. between the time of the last grant and now. I feel like this is one of those places where I should take a victory lap, pretty obviously. There are no downsides to that. It still feels like clearly a good hit given I think I was the only one who put them that high last time. And this time there's a pretty big consensus amongst a number of recommenders that they should be pretty high. But basically you can tell this is the big divide is policy versus Research, right? Are you trying to solve alignment in some form? Are you trying to directly make the world better? Or are you trying to shape public opinion, shape public policy, propose laws, file lawsuits, et cetera, et cetera, to try and set better policy? And I definitely wanted to do both. I thought it wasn't obvious that there was one that was strictly better than the other. You're largely looking for what is underfunded, what is not going to be otherwise funded by the ecosystem. One of the questions I asked a lot was, what is SFF's comparative advantage? Where do we get to identify talent, get to identify opportunity in a way that would be difficult for other people to fund? Last time that was the AI Futures Project, which was specifically unable to get certain other funding at the time. For various reasons. And this time, for example, the case Action Fund, I had pretty high— and simply the Action Fund because it's harder to raise funds for C4 than it is for C3, not because I felt like the Action Fund money was much better spent on average, but that the distribution was going way too far the other way naturally. So ACS Research. was one of my top picks because specifically they were running out of funds, right? I'd seen them do some things that I was very happy with. And I said, okay, I need someone who's doing valuable things, not to just fall over and die. That seems really valuable that you stay in the ecosystem, you keep your organization, you don't have to constantly like look for another home. You can do whatever you think is valuable here. And also, I had Neri this year. MIRI didn't need funding for many years because they got some very large donations. And they did the highly virtuous thing of actually not asking for donations while they didn't need the money. Well, now they need the money. I really hope that we come together and we make sure they continue their cause. Those were some things that stood out at the top, but I have a very long tail of places I would be happy to fund. So if you were to— I had a little widget here, right? If you allocate All of the money that the entire round would get and you ask, what would I fund? There are 10 organizations that would get at least $400,000, another 10 that would get at least $100,000, and a long tail that would get some amount of money from my funding. I could go on about orgs. I'm planning to do another nonprofits post at some point later in the year in advance of Giving. Giving Tuesday and all of that to give my updated views as of—

Yeah, nice. I've got a— my written output is obviously not 2% of yours, but I'm planning at least a Twitter thread on that topic as well, so we can compare notes as we get into the long tail.

Yeah, it's a lot of work and like, I'm gonna have to like set aside specific time for it. But the part of it, I don't know when they're gonna announce the results and I don't want to I want to finalize it. I want to see who gets how much money and I want to see who is then on record as having received money and therefore publicly part of the round versus who is not. Because anyone who doesn't receive money is not publicly part of the round unless they say they are. And therefore I have to email each of those people and then ask, do you want to be in this post? And then at least I give them the opportunity to email me and say no. And then I treat no answer as a yes because in general, charities want people to say you should give money to them, but occasionally someone doesn't want that.

Usually safe assumption. Did you have any— another kind of category that I thought was interesting was sort of international relations. I mean, I know you're not very bullish on US-China cooperation, and I'm not arguing you should be, but there was a sort of crop organizations trying to work on that, and I was pretty into that.

So one of the issues with the round is there were some organizations I knew were doing good work. They had clear wins in their column, or I knew the people involved and the things they had done, or otherwise I could be confident in them. And then it was very hard to give similar level ratings to organizations where I didn't have that. So it's a question of, okay, you've got— again, I won't name them because I'm not sure they're getting funded just yet and they haven't officially been funded. But yeah, there were some charities that were trying to create Track 2 talks or otherwise advance US-China relations in various ways related to AI. And I gave them some support. I definitely Give them like this is fundable, but I always had this sense largely because it's very hard. Like diplomacy is notoriously difficult to read, right? If one of these charities was effectively fake in the sense that like what they were doing was having no real effect and did not in fact impact the possibility of good things happening, would I know? Would they know? Right? Like they might be if they're bad at it not even realize what they're doing. They might be doing this thing thinking they're accomplishing something and just not be accomplishing something. And diplomacy can look for 10 years like you're doing nothing and then suddenly something happens. Or it can look like nothing will happen and then nothing happens, but that was actually the right thing to do. It just created the possibility of something happening if things had gone a different way, or you stopped a bad thing from happening without even realizing or whatever. So it's high leverage, but you just don't know. And because of that, I find it difficult to get behind any of these organizations on a high level. Some of them will definitely make the post as something that I think would be reasonable to support. But yeah, it's really tough when you've got money that could definitely go to places that would be well spent to put it into a weird, impossible-to-read place where you don't get good feedback.

Right.

That's all the more reason to be like, how do they know how to do things properly? How do they make good choices? Even if they're properly motivated, they can't tell either.

Another category. This is sort of a meta category, but the California Bill SB 13, I'm sure you've engaged with a little bit, that would create the private regulatory market where either the attorney general or perhaps some new commission would credential private orgs to be regulators. And then there would be some sort of trade where if an AI developer opts into the regulation from one of these private developers, then in exchange for that, they would get some sort of liability protection. And then that obviously begs the question, who steps up to be these private sector regulators in the event that this bill were to become law? So one of the things I was looking out for was, who do I see that kind of feels like they could become that if that opportunity were to arise? I don't know if you thought about organisations in those terms previously, but—

There are a number of nonprofits that plausibly could step up and become auditors in this space. There are a number of Founders who are perfectly capable of creating new organizations that could possibly do that in this space. If there's demand, right, there'll be supply because like we have—it's not that hard to find expertise in this space that would be happy to like participate in these organizations. Yeah, I just—I don't want to name specific names, and I don't want to fall into the trap of oh yeah, this must be. regulatory capture for these 5 people or 5 orgs or whatever it is. But I guess I don't think that's the case. But I don't think there's going to be any problem with that. And I think that the most likely scenario is that companies like OpenAI quietly ask people who they expect to do good jobs that they would like to spin off organizations they can then work with if they're not happy with legislative options that they are presented with initially from the natural environment. Certainly, for example, Meteor or Apollo, which are the people who are already being contracted for evals by the big tech companies, would presumably be the first ones to enter space. And they would presumably be very credible in that capacity.

Yeah. Last category I'll put you— put in front of you is hardware governance. There weren't too many organizations that were specifically working on this, but the read in my group was everybody seemed to have a different reason to like hardware governance. What's your thought on hardware governance?

So unfortunately, right now at the moment, it's politically really tough for hardware governance. I wouldn't necessarily ever say dead, Because things change so quickly, but it's not looking good because the focus is entirely on get people to use our hardware. And the last thing people using our hardware want to do is be tracked. So there's a direct against exactly what the prioritization is. So it's not, not going to happen right away. I still think that it's one of these things where it's vital that we have that ability, right? I think it's like really important that We have the technology completely shovel-ready that if we decide, okay, as of 3 months from now, every new chip that gets shipped has to be tracked, that we can do that. And ideally also, if we have to go into a data center and put trackers on these chips such a way that, like, if they're tampered with, we will know, that we know how to do that as well. And potentially a very small amount of money can give us that option, right? And then you only have to spend the real money If we implement it, but you get this optionality, right? A lot of those things might be solutions. I think the first best solution to the current mess is in fact to use the solution to allow us to do things like build data centers in the UAE, in India, in ways that like feel secure, and to stop chip smuggling, and potentially even to let be more aggressive about what you. let people you actively are worried about do. Again, I don't think we need that much effort. I think we just need a little effort to lay that foundation. And then the question is just figuring out who the real deal is.

How about just on the simple question of balance of money available and opportunities? I think for me, the sense was I wish I had more money to give out. Obviously, to some degree, that would always be the case. But if you were to make a pitch to other philanthropists that there is a lot of stuff that is not funded as much as it should be, that would be high impact, I guess, first of all, do you believe that? And second, what would that pitch sound like?

So that pitch would sound something like, If I had access to, if I wanted to hand out the entire, let's say $10 million that like is roughly projected to be the entire round, it doesn't allow me to give out like all the money that I would've been happy to hand out. Not by a long shot. Uh, I could easily hand out like more than double that. And feel good about every dollar that I'm giving out there. Obviously, I would want to do more investigation of some of the things that would then— because there's things that I'm like, I'm never giving money there. I've done the math. So I'm not going to think too hard about exactly how to rig that. But you could give out tens of millions of dollars to these applications. We have basically zero— And that is the obvious direct evidence that there is infinite philanthropic space. But also there's a ton of stuff that is the scale above SFF that we basically are like, we can't fund these companies anymore because charities, because their capacity to use money effectively has gone to millions of dollars a year. And we just don't have that capacity, or even in some cases approaching tens of millions, $10 million a year or more. There are also a ton of philanthropic projects that were never even proposed because the price tag on them would be absurd. And also everybody in this space is of course conserving money, which is not particularly But we'd prefer to have generous salaries and generous compute budgets and not worry about this stuff. But that would take a lot more money. But even not changing that, even keeping everybody narrow and not trying for extra massive experiments or anything like that, just doing what we're already doing, yeah, I feel confident about a lot more money being sunk into this space than is already being sunk into this space. It's just not available right now. So it'd be great if you could help out. There are lots of places to put it. And again, there are a lot of people who just aren't asking for money because they know that there's so many other people asking for money. And I'm in that space. I'm being supported by patrons who are happy to support my work, but I'm not going to go out there and seek out more funding because I already know there's way more demand for this funding than there is supply. It doesn't feel reasonable for me to ask for that much more. But could I scale up at least somewhat from more funding? Very obviously, yes. There you go. I have lots of research also, which again is trying to be lean. We're all trying to be lean here.

Okay, here's one idea I want to get your take on. And this was not in the application pool, but I just saw this article. I'm sure you saw it the other day as well, about how OpenAI is starting to subpoena some of these charities, some of which were in fact in the application pool, that are doing various things that they find to be kind of inconvenient, like hassling them about their nonprofit to for-profit conversion. And they seem to believe, and the reason that they're giving for why they're issuing these subpoenas is that they believe that they may be funded by competitors. In other words, they think perhaps Elon or maybe Google is funding these organisations to try to slow OpenAI down. This got me thinking, well, geez, maybe that could happen at the sort of model evals level. We've got these model evals companies, organisations, nonprofits that are, I think, very focused on being even-handed, very sort of fair, very analytical, and trying to do stuff pre-release, which I definitely think has a lot of value to it. But that kind of forces them to play very nice with the companies that they're working with. And I wonder if somebody came forward and said, I'm going to just potentially transparently or not transparently, but we're going to go after all the companies except Google and try to demonstrate to the public why their models are problematic, should not be trusted, all the ways that they go wrong. But we're not targeting Google, perhaps because we're funded by whoever. Could you engineer a situation where all the companies that then feel like, geez, we better go target our competitors' models. We better really invest in demonstrating what is wrong with our competitors' products. And if you could create that equilibrium where they're all kind of sniping at each other all the time, would that in fact bring a lot of things to light and potentially create the race to the top that everyone wants? Because obviously there are a lot of problems that can be demonstrated. It seems to me like The only reason that isn't happening is maybe a sort of soft collusion, or like a sort of unspoken gentleman's agreement, I guess, is another way to say soft collusion. But if somebody were to break that, maybe it all kind of goes to a different equilibrium where everybody is investing in that and we have a lot more energy going in that direction. What do you think?

It's not a It's not a great look, right, from their perspective to be like funding attacks on these other companies. And like when you expose these things in other companies, you are also exposing yourself. Or like almost always when you find these flaws, they are everywhere in some form, or something close to them is available in some form, and it will seep into public consciousness, lead to calls for greater regulation. Looks like it could lead to any number of escalations. You don't need collusion for, oh yeah, my gang has guns, your gang has guns. Why don't we just stay away from each other's territory and not shoot at each other? Because that could get into something pretty ugly pretty fast. And so yeah, most of the time, like Coke and Pepsi don't go on to, don't start smear campaigns on each other, right? They just do positive advertising. Maybe they take a few like cool little side shots of you aren't that cool. But they don't fund nutrition studies about why the other one is unhealthy because it doesn't build the market. But certainly you could fund people to go after specific organizations in these ways or do investigations and deep dives on their ridiculous models. You can have opinions on who to target first. And I don't think it's that crazy. Some companies are being less responsible than others and deserve to get hit. More. If you target philanthropic, they might just say thank you. So there's that.

Yeah, that's why when I was saying target everybody but Google, I was thinking the rationale there is just that there's a lot of Google billionaires that could plausibly be funding such a thing. And the reason they might be doing it could be a mix of competitive advantage and/or philanthropic desire just to bring issues to light.

And I'm like, yeah, and the problem is that like specifically not targeting Google, like, does— I would question you, right? Like, if you're a Google-funded organization that only targets OpenAI, let's say, just to keep it simple, then why do I think your evaluations are objective, right? Why do I think that when you say something is a problem, that it's a real problem or that it's like a specific— I think the idea would be that it's just reproducible, right?

If you just have inputs and outputs from models and you just demonstrate this is attached, that's pretty—

Yeah, I didn't mean is it real, the finding. I wouldn't like— but we're trusting it that you've followed scientific procedures in selecting this example that is representative. It teaches us what you think it teaches us. Yeah, et cetera, et cetera.

Yeah, I see that challenge. Although I also think that's all pretty slippery, right? I mean, as much as there is a, I think, very sincere desire among the eval groups today to have these high standards of rigour, all their stuff is always questioned. And you've got a lot of people who are like, Oh, well, this is totally nothing because you put the model in situation.

It is the job of Caesar's wife to be above reproach and be reproached anyway. It's the job of those who are trying to be the watchdogs in these situations, the people who are trying to be in our position. We have to follow standards of rigor and integrity. that are vastly above what others are held to. And that's table stakes. That's just the right to play, right? And it's not fair. And that's just life. And like, you're still going to have all of that question and attack. Yeah, absolutely. Like, we all saw the debate over 1047. We all saw how like you bent over backwards. to be like 10 times better on all of these issues and all of these questions than the people you were opposed to. And it didn't— you had to be able to play in the arena, but you have to just like, if you're the underdog, if you're the scrappy, underfunded person who's trying to like bring the truth to light, that's your job. It's The big corporation is going to try and squash you with anything you've got. You've got to be sparkling clean. You've got to have no vulnerabilities, no points of leverage, no smear. It's just how it is. And it sucks. But yeah, we're used to it. All right.

Does that mean you basically don't— the equilibrium that I'm trying to envision a way to shift from and to is one where today, and there was this mutual adversarial collab. I don't know if it was really adversarial, but there was this, like, OpenAI and Anthropic evaluated each other's models seemingly in a pretty friendly, collegial way.

That was great.

So, yeah, I think everybody loved to see that, but that's not happened much. Maybe it'll happen more again in the future. Maybe it'll never happen again. I was trying to engineer a transition to a different equilibrium where everybody is adversarially evaluating everybody else and thinking, maybe if I tip one domino, everybody else will feel that they have to respond. And then from a general sort of safetyist worldview, it would seem better if they were all adversarially Evaluating one another versus not. So I guess you could question that first, or question that assumption of that would be a better equilibrium. And then the other big piece is, can we tactically get there?

I would love to get to a point where the companies were doing each other's evals in an adversarial fashion, looking for trouble, looking for vulnerabilities, looking to embarrass them. And they just had to see if you can deal with it, see if you can beat it, see if you can overcome that. That sounds great. I would love that. I don't know how you get there from here. I think that these companies do not want to go to war with each other. I don't agree wanting to go to war with each other in other ways. Also, people are constantly like, I mean, you've already got them going to war over talent. So I don't know, but— I can see it being like a thing Anthropic might want to do in the medium term is to be like, we run all of our evaluations against everybody's AIs and we report them back. And occasionally we're going to find some stuff. But yeah, I don't know.

Yeah, they did do that with DeepSeek. They did come out and say DeepSeek has no qualms about doing bioweapon type stuff.

Yeah, their evaluation of the DeepSeek safety protocols was 50%.

Right. Yeah. Okay, well, if any Google alums with the resources want to talk about seeding such a thing, my DMs are open. Always a final question for you in particular. What that we haven't talked about is virtuous to do now?

Yeah. So I think that like it's a weird situation where like it can be really tough to figure out where to make the most meaningful progress, what to do going forward. On policy, you know, the short-term priority has to presumably be prevent America from being so foolish as to sell B-30s to China, which in practice Presumably means getting enough people on the right sufficiently alerted to this is actually happening and here's what this actually means. They raise enough stink that it doesn't actually happen. And otherwise draw attention to the extent to which NVIDIA seems to have taken hold of the White House in terms of its rhetoric and its plans overall. This is obviously the ultimate end goal. were the primary reason for that. But you see it in everything. Obviously, as usual, trying to spread the better narratives is always good. I've certainly gotten to the point where I think that working for Anthropic seems to clearly be a net good idea at this point. If you are considering what to do and the alternative is doing basically nothing. I do think there are a number of orgs which are presumably better choices for impact than just working at Anthropic, but that doesn't mean that they have capacity or that you want to work there, obviously. But yeah, it's a difficult situation because obviously the policy The policy situation is in a bad state. Alignment is in a not great state. There's obviously infinite things to work on, infinite things to experiment with, infinite orgs to give money to, and so on if you want to do that. But I mean, my project is basically trying to keep myself and others informed and understanding of the situation and hope that that will lead to good things more than anything else. But yeah, I wish I had a better answer, obviously, for a call to action of like, now that you've listened to this podcast for 3 hours, depending on what speed you're listening at, what are you going to go forth and do? But yeah, unfortunately, I don't really, other than think hard about the world and try to figure out what under your The core model would be the right things to do to advance it and what are the things that would actively make things worse. Because for a lot of people who are informed, the first step is just be aware of what would make things worse and don't do that.

Which is—

But besides, you should say what you think more than anything else. You shouldn't sugarcoat. You shouldn't engage in hyperbole. You shouldn't strategically censor yourself. With notable exceptions, you should just say what you actually believe about the situation. And one thing you can do is you can support the book release of— Eliezer Yudkowsky and Nate Soros are coming out with a book in about a week, If Anyone Builds It, Everyone Dies, where the end is superintelligence. And if you were to help by purchasing and spreading the word about that book. This is a unique opportunity to get that momentum, maybe create a cultural moment. But again, that doesn't mean that you should just back their talking points and their idea about how this works just because they're the ones with the book or just because Eliezer is the rightful K-Left and he said so or anything like that. You should make up your own damn mind.

I've pre-ordered my copy and look forward to reading it. As always, really appreciate all your time. You've been very generous with it, and we'll continue to stay informed via the blog. Don't worry about the vase. I think that's it for today. Zvi Mashowitz, thank you again for being part of the Cognitive Revolution.

Thank you for having me. All right, bye.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of a16z. We're produced by AI Podcasting. If you're looking for podcast production help, For everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion, Everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI meeting notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI meeting notes free for 30 days.