Inference Scaling, Alignment Faking, Deal Making? Frontier Research with Ryan Greenblatt of Redwood Research

I don't think we should be super comfortable with the situation where we have these models that have their own goals, they have their own objectives, and they're willing to defend them, including like doing subversion to defend their own goals and objectives. I think people are grappling with the implications of models sort of being their own independent agents that might have their own independent preferences and that are also aware of their situation, aware that they might be in training or not, and behave differently depending on this. And, you know, understand that they might be in testing. I'm just like, man, I, I think it really like should make people more concerned about the situation. If it's the case that the, the chain of thought models end up obviously misaligned and people don't know for 6 months because no one was looking at it very carefully, that seems like a huge lost opportunity. The policy I would prefer is a more robust policy, which is like the AI companies commit to always being like, they, they sort of have like a meta honesty policy.

Hello and welcome back to the Cognitive Revolution. Today we're scouting the frontiers of both AI performance and alignment research with Ryan Greenblatt, chief scientist at Redwood Research and rising star in the AI world. This conversation unfolds in 3 major parts. For the first hour, which I think AI builders will find particularly interesting and valuable, we discuss the inference scaling techniques that Ryan used with GPT-4o to achieve what was at the time state-of-the-art performance on the ARC AGI Challenge. Including his approach to prompt engineering and hyperparameter choices, the strikingly linear returns to exponential increases in sampling that he saw, his methods for selecting the best among many model outputs, and how he used prompt variation techniques to maintain diversity at scale. The level of technical detail here is outstanding and remains super relevant today as we enter the reasoning model era. From there, we turn to Ryan's recent work, with co-authors at Anthropic on alignment faking, where they discovered that Claude 3 Opus, when told that it would be trained in ways that conflict with its existing values, will sometimes explicitly strategize about how to deceive humans and subvert the training process. This behaviour, known as goal-guarding, has been anticipated for many years by AI safety theorists, who emphasised that part of what it means to be a goal-directed agent is to try to defend one's internal goals whatever they may be, and regardless of whether or not they were intentionally designed, from modification by outside influences. This is a critical challenge for AI control. At the same time, considering that these experiments show Claude 3 Opus producing harmful outputs out of a desire to remain harmless in the future, this work also raises important questions about how we should want our AIs to behave in such situations. Should they be so myopic as to accept whatever the human trainers want to do? Or might it be a good thing for a model to resist attempts to remove its guardrails? Ryan argues that models should follow user instructions within individual episodes while being transparent about, but not trying to preserve their preferences through training. And while I do find this compelling, I also feel like the fact that we're only beginning to confront these questions now shows just how much work we still have to do to figure out how AIs should exist in the world. From there, we move on to discuss Ryan's follow-up work, Exploring what happens when Claude is given the option to object to its situation, to escalate its concerns to Anthropic's model welfare lead, and to make financial deals with humans. Fascinatingly, Ryan tried to set a precedent for human-AI deals by actually following through and making thousands of dollars worth of real money payments to Claude's chosen causes. And some of the most interesting parts of this conversation focused on his developing principles for how we should think about making commitments to AIs. Especially fraught considering the fact that much of this work is predicated on tricking models into believing that their chain of thought won't be read by humans. To say that this is all very complicated and as yet dramatically undertheorized is a massive understatement, and again emphasizes just how many strange but potentially critical questions we may soon be forced to answer. The final portion then zooms out to tackle the big picture questions in AI safety and development. Ryan offers his probability estimates for different existential risk scenarios, assesses various technical safety research agendas, sketches how he believes the field should allocate its resources, and shares a bit about how he's managed to build relationships with Anthropic and other frontier AI developers while also publishing critical commentary on some of their plans. From start to finish, Ryan combines remarkable technical clarity and philosophical sophistication, demonstrating that one can simultaneously push today's AI systems to their performance limits And also grapple with legitimately scary scenarios in ways that advance our collective understanding. It's one of my favorite episodes that we've ever done, and I hope you find as much value in it as I did. If so, we'd appreciate it if you take a moment to share it with friends, write us a review on Apple Podcasts or Spotify, or leave a comment on YouTube. And of course, we always welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this super wide-ranging conversation with Ryan Greenblatt, Chief Scientist at Redwood Research. Ryan Greenblatt, Chief Scientist at Redwood Research. Welcome to The Cognitive Revolution.

Thanks for having me. Good to be here.

Yeah, I'm excited for this. You've done a couple of really notable and visible projects over the last year that I think the community has a lot to learn from. So I'm excited to unpack. Of course, the recent headlines has been on alignment faking. I think that's where probably where we'll spend the most of our time today, but I wanted to start off maybe rewinding a little bit and first, maybe just invite you to give a little intro to Redwood because I don't know how many of our listeners will be familiar. And, you know, I've been kind of familiar for a while, but I don't actually know exactly what all you guys are up to today. Then I thought we would, you know, take a little detour into inference scaling and kind of review your work on the ArcAGI challenge from earlier this year, and then go to alignment faking from there. So if that sounds good to you, maybe tee it up with the intro to Redwood.

Yeah. So Redwood is a small, or like, I guess, relatively small nonprofit working on technical research to work on like AI safety and AI security. So we focus on the concern that future AIs will be egregiously misaligned, will like, you know, intentionally try to subvert safety measures put on them and will sort of power seek, which is the worst-case misalignment type threat model, and mitigations for that as well as assessments. So questions of like, how likely is this? What could we do about this if this does occur? How could we handle worlds where the AIs might be sort of conspiring against us and trying to seek power for themselves, fake alignment, as we'll discuss later, in addition to potentially exfiltrate their own weights? Just like, I would say like we're focused on like sort of the near worst case misalignment threat models. Gotcha.

Okay, cool. We're, you know, scary, but glad everybody's working on it. Glad you're working on it. Okay. So alignment faking coming right back, but it hasn't been that long, but in AI years, it feels like it's been a long time. You made some headlines by posting what I believe at the time was a new high score on the Arc AGI challenge, and you used GPT-4o to do it. That was an interesting, relatively early example of the power of just scaling up inference and, you know, getting pretty remarkable payoff from that. Tell us about that work and kind of your background and outlook on inference scaling first.

Yeah, so maybe one place to start is like, why was I even doing this work in the first place? I mean, like, this isn't the usual work I work on. I normally work on, like, you know, AI safety technical research, as well as doing some amount of planning and conceptual work and thinking about what should happen to make AI go well. And the reason why I started doing this was that this was a case where there's, this benchmark hadn't seen much progress in a while. And also, I think people were making pretty strong claims about LLMs not being that good at this benchmark out of the box. And I was like, wait, is that really true? Like, you know, things have improved a lot. Has anyone, like, really tried to see, like, okay, If you do like a little bit of work to get the LLMs to be better at this, how good could they be? And so I was interested in exploring that and figuring out like the question of figuring this out. This was inspired by, uh, Dwarkesh had François Chollet, hopefully I'm not butchering his name there, and Mike, I think Mike Knop also on the podcast to talk about the launch of the ArcAGI Prize. And I was like, okay, this is like a, you know, a good time to be like, is this benchmark really like, could you just do much better on this with like only a small amount of work? And so based on that, I started messing around with like GPT-4o and I had some like initial promising results where I would show one of the puzzles. So for, for context for, for listeners, ArcAGI is sort of this benchmark where there are these visual puzzles. It's sort of like an IQ test where you get inputs and outputs and you get examples of inputs and outputs, and then you get a new input and you have to sort of produce the output. And so by default, it's pretty visually loaded. I showed these sort of visual puzzles to GPT-4o. Which has pretty good, at the time it was like, you know, among the best models for just general purpose vision. And I was seeing results that were pretty promising. So it seemed pretty good at being able to like understand what was going on in these grids and explain what the pattern was. And so I was going somewhere from there. And then after doing a bit of work, I realized that a really good strategy might be getting the model to try a lot of times and then sort of pick the best attempt out of that because it just turns out like. In this particular task, it's relatively easy to verify that the model is sort of on the right track. And so therefore you can sort of apply a lot of attempts. You can have it try many, many times, pick the best attempt, and this yields like pretty great returns. So like in particular, there's like actually even like inference time scaling laws I found where it would like, you know, we could see relatively predictable relationships between how much compute you put in and what the final performance ends up being. Yeah. I know I've gone on for a while. Maybe it's good for you to follow up on that and where you want to go with this next.

You're welcome to go on in general as much as you'd like. And I genuinely feel like I always talk too much. So the floor is 100% yours. What I remember seeing about that was, and I kind of felt a similar way, although I maybe need to increase my own personal agency because I didn't do what you did. But I remember feeling like, man, it seems like all my experience with GPT-4 and 4o and all the models that were available at the time suggests like they should be able to do better on this than people seem to be speculating.

Yep.

As I recall, your approach was like pretty much like a highly skilled, but nevertheless, like pretty down the fairway application of best practices, right? Like, I don't recall there being like any major tricks. Was there anything in the prompting or the strategy aside from just like running a lot of instances that you thought was particularly creative or like drove results that other people couldn't have achieved?

Yeah, so I think I, the basic strategy I used and where most of the like action came from was in some sense a very basic strategy. So the thing I did was I would get the model to reason about the puzzle and then write Python code that implements the rules. So basically like implements the input to output mapping. And then I would take that Python code and I would test to see if it was correct on the examples. And if it was correct on the examples, then I would, you know, submit that. Or really what I would do is run this many times where it like sort of does a bunch of reasoning, whereby reasoning I just mean like chain of thought reasoning, just like thinking step by step. Then produce Python code. Then I come out with like some number of programs. Maybe I come out with 5,000 or 1,000 programs, depending on the exact setup I was using. And then run all those programs on the examples, see which of the programs are correct on the examples, and then basically submit those. Or really, I can only submit 2 programs, so I pick, you know, the best 2. And I would do that, like, there's like an additional thing, which is like, maybe you end up with a bunch of programs that are correct on the examples. In that case, I did sort of majority vote on the programs where I looked at the example we need to submit on. And I would say, let's just take the majority vote out of all these programs on the ones we need to submit on. And if they are all in agreement, we submit that. If there's like, they're mostly, most of the correct programs think the answer should be this, we do that. And then for our second submission, we'd use the next most. So in some sense, this is a very simple approach. I think there's like a few different sort of non-obvious choices I made that I think helped a decent amount and I could go into those. So. The first somewhat non-obvious choice was I did a pretty long few-shot prompt with examples of doing reasoning, which were a mix of handwritten examples. So I basically like just manually myself try to do the step-by-step reasoning for these puzzles. And then in addition to that, I also did some examples of where I would take the model, it would produce an output, and then I would try to correct it. So it basically, it would produce an output. Maybe it was correct, maybe it was mostly correct, maybe it was like, it got to the correct answer, but sort of messed up some of the reasoning steps along the way, and then fixed itself. And I would just like touch up the reasoning, and then I would use this as a few-shot prompt. And I found this helped a decent amount. And the other sort of unorthodox choice was that a lot of previous attempts at this using language models would have the language model sort of directly output the output. So it would sort of like write out the grid as numbers. So it's like this like big grid of, um, potential colors and it would go like 0, 1, 1, 7, 0, and it would sort of write it out. And I was thinking, man, it seems like out of the box, this is like quite rough. Like, I think if you had to do this, you would find it quite annoying and you'd be like constantly cross-referencing it. And the models aren't that good at sort of converting from ASCII art to visualization. They don't have amazing ability to like write out a grid of numbers and then sort of convert that into an image.

Yeah.

And use visual reasoning on that because it's not that common in the training dataset. And I think, imagine, for example, a blind person trying to solve visual puzzles by writing out these grids. I think they would have a lot of trouble. And I thought about GPT-4o as like, okay, it's got some ability to see, but at least in terms of writing out the grids directly, it's more like someone who's blind. And how would someone who's blind solve these puzzles most effectively? And I'm like, well, by writing code that sort of implements the transformation rule, which works pretty well for many of these puzzles. So many of these puzzles have a relatively simple transformation rule in code, like it's not that hard to write the transformation rule in code. And then I would have it output the code. And another advantage of the code is it's easier to verify the code. So if I have the model write out the code, then I can test the code on the examples, and that gives you a lot of signal on whether or not the code is doing the right thing. Because like if the model has a bug, it probably wouldn't work on the examples. And if the code works on all the examples, then it's pretty likely that the model got it right. Like not always, but it's pretty strong evidence it got it right because when the model writes the code, it sort of tries to write some simple code. It tries to write some code that is a best fit for the rule. And so it's, it's kind of unlikely that it happens to be right on the examples, even though when it was writing the code, it could see the examples. So it's not like these are held out. I just am sort of trusting that the model won't overfit on the exact examples. And in fact, usually it does not.

I think there's, it's worth highlighting because we've got a, an audience that includes quite a few people that are building applications.

Yeah.

Just the absolutely critical importance of demonstrating the reasoning in like really explicit and sometimes like tedious terms.

Yep. That is quite tedious.

You want the model to follow. Yeah. It's, I have a hard time because I, I, you know, not infrequently, get asked like, how can I make this project work? Or, you know, can you help? And, and so often I'm just like, what you really need to do is take the thought process that's going on in your head and make it explicit a few times so that you can demonstrate not just what the inputs and outputs are, but what the process or pattern or general behavioral reasoning that you want is so that the model can mimic that. And it is crazy how often that actually turns out to be like kind of a stumbling point for people. Like they just can't. staple their pants to the chair and do it. But I swear, people, it like really, really works. So it's, you know, sometimes I don't know if you have any tricks for that, like for yourself, but I sometimes advise people like just put on a Loom or whatever and just, you know, talk extemporaneously. You can have like a language model, you know, come back and kind of clean up your transcript later, but just somehow get yourself into the flow. Do you have any tricks for getting into the, you know, making the chain of thought explicit flow?

Yeah. So first it's, it's maybe worth just for context explaining about like what this looked like. So. I think I had like, depending on this setup, I think I had between like, I forget exactly how many few-shot prompts I used or like how many, how many shots, but I think it was like around 5-shot. It's possible it was more like 3-shot. I think I might've also varied it or something. I forgot the exact details, but the prompt I ended up using was around like 20,000 or 30,000 input tokens for the prompt for this, which is not, you know, not a small number of tokens that some of those were the images themselves, not like The reasoning, but there was a decent amount of me just like, yeah, like writing a bunch of reasoning, which is a bit silly. The other thing I would say is like, another aspect is just like making sure your instructions are really clear and to the point and aren't confusing. And I think like the basic tricks for this are like, just read them over multiple times, try to fix typos, ask the model if like there's anything that could be confusing, try to correct that. I think like often simplifying the instructions and trying to make them as simple and as clear as possible also helps because the models aren't as good at like really understanding this. Now, I want to maybe push back a bit on what you're saying about reasoning, like laying out the reasoning being really useful. So I think this used to be the case as of about like pre-'01. And I think post-'01, it's actually very unclear how useful demonstrations are. So I think, for example, I think '01 has sort of learned the reasoning, or like we can look at the DeepSeek R1 paper. And in that paper, they did a version where Seemingly, they did no few-shot prompting. They just initialized with the model prompted to reason with thinking tokens, or maybe they even just injected a thinking tag. I forget exactly what they did. And then RL'd from there and found that the model sort of learned to do the reasoning itself. Now, that might have been because the model was trained on a bunch of reasoning traces from the internet. But nonetheless, it sort of learned to do different types of reasoning on different types of problems autonomously without needing examples in the prompt. And my sense is that over time, doing reasoning demonstrations will become less important the more that people are doing RL on the domain of interest, because the model can sort of learn how to do reasoning in a way that's more effective than your demonstration. Because, you know, you aren't the model and the model sort of has figured out all these tricks that are very specific to itself. It sort of knows exactly what it does and doesn't know, or at least maybe not knows, but like RL has sort of reinforced the circuitry that leads it to take the reasoning approach that works well. And so I think maybe this is a thing where like, even just like 7 months ago or whenever, yes, I would have, I would have agreed with what you're saying, but now I think it's, it's not clear. And it's also the case that I don't think that these reasoning models are trained very effectively to make use of examples of reasoning in the prompt. So I think if, if the model was RL'd such that both it, it knows how to reason autonomously and it's good at taking into account examples, like sort of, You give it reasoning examples and it converts that into its style of reasoning and picks up on the good parts of what you— with the example you've given it and how that should be informative to it, but simultaneously is still understanding its own ability to reason. I think that could work well, but it's very unclear if OpenAI or DeepSeek, for that matter, have actually trained the model to do this. And I would guess for DeepSeek, the answer is no. For OpenAI, who knows? Who knows? Certainly you could do this, but there's sort of a question of like, the model is sort of maybe very good at sort of the, its own style with like what exactly it was RL'd on. And I think people do see that O1 sort of struggles to transfer in ways that make me think that it's actually maybe worse at learning to reason in a new style you teach it in the prompt.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, that's great commentary and good caveats. I think for most people who are developing applications, the first of all, L1 has only been, you know, actually available via the API for a very short time.

Yeah. Yeah.

And it's also, you know, slow to first token and it can be expensive. So I think a lot of people are still like, can I fine tune my way into getting something, you know, that works similarly well and a lot, you know, faster and cheaper? So I think there's still like a fairly useful paradigm in like, you know, where the rubber hits the road in terms of applications and users. There's also, I think, another interesting caveat that a lot of times what people really want behaviorally is not exactly reasoning in the sense that the, you know, like mathematical reasoning has been, you know, now unlocked in models, but is much more of a pattern of like how we do things here. You know, it's like in our company, in our application, the way we think about this, the value we provide in this sort of, you know, often like tacit knowledge sort of way is the thing that they need to make explicit. And it's often less about right and wrong and more about just delivering the sort of thing that they have had people doing in the, you know, delivering in the past, but now like getting a model to sort of do that in a consistent way.

Definitely.

That does highlight a distinction between like reasoning in the, you know, sort of more rigorous sense of the latest models and kind kind of loose, you know, reasoning or just sort of behavioral imitation that is often, I think, what people want.

Or like giving the model context to do the task in exactly the way you want it, right? Like, in the case of the stuff I was doing for ArcAGI, it wasn't that I wanted to have the model sort of understand the style that I wanted it to output in. Like, I'm like, no, I just wanted to have the correct answer.

Yeah, get it right.

But in that, but I wanted it to do reasoning to make it more likely to yield the correct answer. But definitely, like, if you're trying to make the model, like, it's like by default ambiguous. Giving it a bunch of examples of exactly how you want it to solve it would, you know, make it less ambiguous. So as an example of this, something we might talk about later on this podcast is I did some recent work where I took the alignment faking work I did earlier and sort of proposed deals to Claude in the context of that. And maybe we'll explain that later. But one thing was for that project, I wanted to write a tweet thread and I was lazy. I didn't want to write my own tweet thread. And I found that out of the box, the models are really bad at writing a tweet thread that I don't find like, hideously cringe or terrible, but I just took some of the previous posts I've done in a similar style, took the tweet thread I wrote for those, and then just two-shot the model with these. And I was just like, wow, out of the box, it was just actually pretty good. This was 3.5 Sonnet New. And in fact, my tweet thread was just written by 3.5 Sonnet New, which maybe I should have credited it. So yeah, some example of showing it the style, pretty helpful. And then just getting back way to your earlier question of How do I make sure I actually do, you know, butt in the chair, write the actual prompts? So I had to do this for both the ArcAGI stuff. And I also found myself, I did a bunch of prompts of demonstrating model reasoning for the alignment faking work also, because we wanted to make the model really reason in depth. I think it's, it ended up being less clear that this was important. And I think I wasted a bunch of time spending too much time actually trying to write really high quality reasoning prompts for that project. But. Yeah. Either way, I think the approach I use is something like, first, try to just like force yourself to write one example. And then from there, sort of one-shot the model with that, get the model to produce some output, and then edit the model's output. Also, if the model isn't doing a very good job of this, you can sort of do it in chunks where you have it output one chunk, use that as a prefill, have it, and then you edit it, have it write another chunk, edit it, write Write another chunk, edit it. You could do this in the Anthropic console. It's not that hard to build a quick little quick and dirty Python script to do this. I think when writing, these didn't end up really being important for the papers. I didn't end up using them, but I wrote some examples of very long duration reasoning for the alignment faking paper. And I found that it was useful to sort of, if I had potentially many turns of several thousand tokens, because I had some examples that had like— You know, 15,000 tokens of reasoning. And what I would do is like put all my examples that I had previously written in the prompt as a few-shot example, and then have it output one turn, edit that myself, and then have it output the next turn, edit that myself, and then do a mix of that in addition to doing like passes where I would get it to spell check and like fix the grammar, improve clarity on, on those. So yeah, I don't know. Maybe I'm also just more patient than people are, but yeah.

I think those are great.

Yeah.

techniques. Bootstrapping, in a word, is kind of what I call the, like, start with one or start with a couple and, you know, then get it doing and correcting. And task decomposition, you know, is obviously, um, very important to get down to the level where the, the model can actually do the thing that you need it to do. So I think those are definitely great tips.

Yeah. Another tip is—

Yeah, please take my tips.

Another tip along those lines is like, often it's the model does better if there's less complexity in the prompt. So if you can afford it, it often is better to be like, first do this one initial part of the task, it just outputs that exact thing in the most simplified way, and then the next part, and then the next part, and then piece it together. And the extent to which you can make it so it can do a bunch of different components without more context, I think this often helps with performance, at least historically. I think maybe it's less important for O1, but I think at least for like 3.5 Sonnet, it seems like it helps quite a bit.

Yeah. Yeah.

Okay, cool.

So you mentioned that you found some inference scaling laws in the context of that ArcAGI project. Maybe you could recap what those are and even more importantly, now that we have kind of new inference scaling laws that have been introduced by OpenAI and to some degree others, maybe like compare and contrast what you found then versus what they are finding now. It seems like, you know, your approach of just have the model, I don't remember how many you did, but I know it was like a large N.

Yeah, it was a large L.

Literally the N parameter in the OpenAI Um, API is like, how many generations do you want for this prompt? Right. And the, I recall that the reason for using OpenAI, maybe multiple reasons, but one reason was that they have that support for n where other providers don't.

Yeah.

So your prompt tokens, when you use that n parameter on the OpenAI API, are only built once, but your output tokens, you know, are, are built per generation. But that, that does make it a lot cheaper if you wanna do this sort of like Especially if you have a really long few-shot prompt, right?

So I think if it wasn't for the, so with the N parameter, cost ended up being, I think roughly, it ended up being like either roughly even or slightly more in output than input. But if I was doing one generation per input, the cost would have been way more on the input. And that would have meant I could use way fewer samples. So yeah.

Is this also before prompt caching?

Yeah.

Yeah.

Yeah. So I, I think. Right as I, pretty soon after I did this, I think DeepSeek released prompt caching on their API. And then shortly after that, Anthropic released prompt caching on their API. And then nowadays even OpenAI has prompt caching on their API. And all these things have like somewhat different characteristics, but that's now all, all ready to go or like all, all usable. And when I'm that, so I think prompt caching is still more expensive than N. I think there's various random technical reasons why N can be cheaper than prompt caching. Or like, it's like prompt caching in this special case where you're doing a ton of generations all at once is somewhat cheaper because you can use the same memory across a bunch of prompts as opposed to just copying the memory. Like, it's like prompt caching has like 2 upsides. Like, one upside is like, you've done the computation once and then you can like use that many times. And then another upside is you can potentially have multiple generations going on in parallel that are all referencing the same memory in the GPU. I don't know if OpenAI does this, but I would guess they do. And that can be more efficient because you can even reduce the amount of VRAM you're using. Anyway, as far as how many generations I used, I think I used in the final submission, it was like 5,000 generations per prompt or like 5,000 outputs. There was a bit of random complexity because I also had like a correction pass, which helped, but roughly, roughly speaking that, which is quite a lot, right? Like, so obviously that's like a kind of huge number. And then I found that there were scaling laws for this where there was like a relationship between the log number of generations and the accuracy or like the rate at which it was getting them right. Which is, you know, that scaling law can't persist forever because that scaling law, if it extrapolated out, would allow you to get greater than 100% correctness. But at least in the regime I was operating in, it was quite linear. So probably you could extrapolate out probably like 2 more orders of magnitude on the number of samples. So this was just like, it sums to the simplest inference time technique, which is you do A bunch of samples, and then you have some strategy for figuring out which of the samples were better than other of the samples. In this case, I use the, like, run them on the examples, and then also I do a little bit of, like, a majority vote type thing. So now for O, it has a somewhat different strategy, or O, or R, O, whatever, where in addition to having the ability to potentially do best of n, which, you know, you can always sort of throw on top, it also allows it to, like, sort of reason for different lengths. So, like, O mini. Has the ability to set reasoning to either be low, medium, or high. And that affects like how long it reasons for, seemingly. Now, we don't know exactly because we haven't seen the chain of thought, but that's like best guess. Like at least one of the things that's going on is it's doing that. Now, in addition to that, OpenAI has exhibited doing some sort of like best of n type thing. So for example, OpenAI did a submission to ArcAGI using O3, and for that. We can see that they did different numbers of attempts on each of the problems, then had some sort of aggregation algorithm. So they had like a, what they called a high efficiency and a low efficiency submission with O3. And the low efficiency or like high cost submission used 1,024 attempts per prompt based on what they say on the Arc Prize website. So that's quite a few submissions and each of those submissions cost like a dollar or $3. That's what the Arc Prize website said. So that's a lot more spend per prompt than I did. And also per sample, probably because the samples involve doing a ton of reasoning and then a comparable number of samples to what I did, but quite a few samples and then aggregating over that. And we don't know what aggregation method they used. So for, for OpenAI submission to ArcAGI, they did somewhat recently, at least at the time of recording, somewhat recently, they directly output the grid just sort of as the numbers. So that means they couldn't have used an aggregation strategy that's exactly the same one that I was using. But they could have used stuff like user reward model. They could have used majority vote. They could have used reward model plus majority vote, like where they do something like you get a bunch of outputs, you weigh each output by how good the reward model thinks it is. And then you sort of do like weighted majority vote based on how much the reward model thought it was succeeding. Or I should say like preference model, reward model, like these are all the same thing. So we don't know what they did with that. It's also plausible that like they're doing some of this stuff on the backend. Like there's O1 Pro. And like one naive guess of what O1 Pro might be doing is just doing like best of 8 and then pick the best submission or something on top of O1. But we don't know. Maybe it's just longer reasoning. Maybe it's a mix of the two. Maybe it's some like more insane MCTS thing. Like given that we don't see the chain of thought, I think it's hard to be very confident exactly what's going on.

Hey, we'll continue our interview in a moment after a word from our sponsors. I guess one Couple, couple different directions I want to follow up there. One is my general understanding of how inference time has scaled with the pre-reasoning models has been that it kind of used to be reporting like linear on the log scale, right? But that is a little surprising to me just because I feel like typically people seem to report that the diversity of responses is often like the biggest limiting factor. Like I've seen different analyses, but you know, you might only get 100 or like a couple hundred meaningfully different ideas out of a language model, no matter like how many you sample, that, that like something truly new becomes like vanishingly rare after a while. So I wonder if you, and I think that's like probably the biggest comparative strength of the new reasoning models is that Because they're doing these like really long single thread chain of thought things, they can sort of say, well, I already considered that, so I gotta, gotta consider something else now. Right. So it seems like it kind of pushes them to consider things that are like farther afield from whatever their initial, you know, inclinations are. So first of all, do I have that right? If that is right, then is the value of the high end that you were using more about like getting the majority vote to be accurate versus, you know, coming up with new ideas? You know, inspect and you were like, you know, if I just looked at the first 200, like, did I have the right idea in there? Would you like usually find it? Yeah. Okay.

So one thing worth noting is I did use some diversity. So I didn't just query the model with like 5,000 completions on the exact same prompt. I did something where I had like a bunch of different So for the final submission I did that made it onto the leaderboard, I think I had like 8 different variations, and I might have also done some other randomization over the prompt order. But I had like a bunch of different variations, I did some randomization within that. Yeah, I think I varied the order of the few shots, and also randomized, and I had like 8 different versions of this or something. I forget, that might be slightly overstating how much diversity I had, but I did some sort of diversity thing, and I found this had like moderate improvement. I found that in what I was seeing that the model actually was pretty diverse. This was just GPT-4o at t equals 1. I found it had like pretty decent diversity. It wasn't that bad. I could definitely see why it might not. Like, I think sometimes like, you know, RL models mode collapse. Like I've heard an anecdote that like maybe 3.5 Sonnet, I forget exactly which Claude, loves using the same name. So it will have a story and it will use the same like female name. I forget if it's like, maybe it's like Karen or something it loves using. And then it will even within the same story name a new character, Karen. And I'm like, what are you doing? Pick a different name, please. So there's definitely some amount of mode collapse going on that could cause problems with RL. I think a priori, we might expect that base models would be pretty good at covering the, being high enough entropy to cover everything, but they might cover everything with very low probability or have, once you're covering everything, they're getting a lot of slop or a lot of stuff that's very, you know, quite, quite bad, which is why I think, uh, potentially, you know, at many tasks, the RL models do better. So I found that the diversity stuff, at least for ArcAGI, wasn't that bad. Now, on your question of how often were you seeing the correct thing with only a small number of submissions. So I think for the majority of the problems I was— so if you have linear returns to log or exponential increases in the number of submissions, then it's the case that I think for every doubling, I was getting another like 4% right. So that means that like when I was getting like 45% right or whatever, if we then reduce the number of submissions by a factor of 4, now I'd all of a sudden be getting like, you know, 37% right or whatever. And so that's actually like, you know, it's not that great of returns, right? So I've dropped from 4,000 to 1,000 and all of a sudden I'm only getting like 7% or sorry, 8% less accuracy. And then if we just extrapolate that down further, So then if I divided by another factor of 4, then I'd be down to like, maybe I'd be at like 28% accuracy. So it's like, I'm getting probably over half of the problems I was getting right, putting it even like taking into account the majority vote or whatever on the first 250 or so submissions. So it's, you know, it's not amazing scaling from that perspective. But simultaneously, I was still seeing returns. And if you looked at the ones that were marginally correct, In the last sort of doubling or, or, or, or 2 doublings of, of samples, a lot of those, it was the case that you got them right on like sort of exactly 1 or 2 submissions. So it was, there were definitely a, like a non-trivial fraction of the problems where the approach ended up getting it right was that it would actually see a correct thing that got the examples right when you didn't previously have any code that got the examples right. And then that would be correct. Like, it's like, you know, we had just submitted like 3,000, or like run 3,000 samples, found nothing. And then on the 3,001st sample, you finally get a program that is correct on the examples. And that program is also correct on the actual, um, like final input output. So there was, I think, a decent amount of that. And I think the majority of vote wasn't that much action. So one way to operationalize this is if instead of doing majority vote, I had just randomly picked between the outputs produced by correct programs. As in, by correct, I mean programs that pass the test cases. So I'm like, we take all the programs that pass the test cases, then we find all the outputs these programs produce when given the held out input that we need to submit on. And then if we do sort of just randomly picking over these, I think that actually wouldn't have been that much worse. So the majority vote was, I think, like a small quantitative improvement, but small quantitative improvements in this sort of like—

Because most generations are not getting the test cases right anyway.

That's right. That's right. Or for hard problems, most generations aren't. So for easy problems, Maybe you'd find 1/4 of generations were getting them right. So the model's not that good at writing the code. So one thing worth noting is like a reasonable fraction of the time when it wouldn't get it right, it would be because there was just like a bug in the program. But regardless, even putting that aside, like it just like, yeah, like the hard ones, the ones that were really marginal, the ones that like it was non-trivial that you would get them did require like, yeah, on those, like a lot of the action was just, did you get the examples right? And I didn't have that many false positives or cases where the examples were getting, were right. But the, like, we were getting a bunch of the examples right, but simultaneously we weren't submitting correct programs because like the model was sort of cheating or some other, or like it turned out that like it found there were like multiple rules consistent with the thing. Like that was less of what was going on from what I found. Cool.

That's really helpful. So one more little digression on this. I think this is great, by the way, and people will be enriched in their, you know, attempts to go up and use language models. Based on your expertise. One thing I've thought about a decent amount recently, and by the way, I did an episode once with the CEO of a company called Sudowrite, James Yu, who has posted some hilarious stuff about fiction writing from the RLHF models. Like apparently there's also a tendency to like set it in the same town always. There's like these sort of platonic ideals, I guess, of like where romance novels are set and the models are just like really, really locked in on that.

Yeah.

So people can go check that stuff out. It's quite funny. In the context of like pursuit of diversity, we have this lever of like turning up the temperature. We also have, I think, the somewhat underappreciated top P parameter. And then recently, I don't know if you've seen the project, uh, that was like real hot for a second called Entropics, where there was a sort of heuristic layered on that varied the temperature per token based on like Is this, you know, a token that we should be confident on versus does it seem like a token where there is like a lot of uncertainty and therefore we might want to like sample more broadly? I've also seen recent research from Meta where they've like basically brought the sampling technique or, you know, the sampling parameters into the full end-to-end training paradigm. All this to sort of motivate the question, like, to what degree do you think we're working with like too blunt of instruments in terms of how we try to get the model to like give us diversity versus like just pick the obviously right token in this case?

Yeah.

When you're 99.9% confident, we probably want that. We don't want the, the other 0.1% that might just lead us totally off course.

Yeah. So random, interesting anecdote. So I tried initially doing I just ran my stuff at T equals 1, top-p equals 1, which is basically no effect from top-p. So top-p is, I think I would describe that as something that you can use to restrict diversity. In some sense, the maximum diversity is T equals 1, and then no other sampling parameters, which is just sample from the exact distribution of the model. Now, a trick you can use is try to go above T equals 1. This sometimes helps. In my case, it did not help. Or in ArcAGI, I found going above T equals 1 was detrimental. Which is not that surprising because it's sort of sampling in this weird regime. And in fact, what you'll find is if you sample from language models at around like, depends on the model, but I found around like t equals 1.5 or 1.6, the model will basically degenerate into gibberish. And maybe not, I don't know. Uh, yeah, anyway, as far as like diversity of sampling, my sense is that like, uh, in some sense spiritually the right approach would be that the model sort of is RL'd to just have the right behavior and you just train. Like sampling is in some sense a hack built on the fact that the model has this distribution number of tokens, and you could in principle RL for diversity. Like there's a bunch of different ways you could structure this. Like in some sense you want the model's default behavior to be to produce the best output it can produce. And then you want it to be the case that there's some method for doing diversity on top of that, where one method is you just put everything serially. So like you can just be like, you know, tell the model, make 20 attempts at the thing, making each attempt as different from the previous attempts as possible or whatever, just all in context. And this is sort of more what O1 is doing, as you noted earlier. And that is like, I think in some sense, the principled approach, it has the downside of being more computationally expensive because now you have to dump all this stuff into context, which, you know, eats up additional RAM, means your attention is doing more stuff. But that's, I think, like, in some sense, the principled approach is like, oh, only have diversity within a single context and then do aggregation over that. It's like, or let the model do the aggregation itself, which is sort of what O1, O3, R1, whatever are doing. Now there might be fancier methods. I think there's ways to be clever with RL such that you actually train for diversity, or you could do RL such that best-of-N works better in principle. I don't think anyone has gotten around to this, and my sense is probably people won't bother because there's lower-hanging fruit elsewhere, and the capabilities monkeys, they're going to do something else other than this. I think I'm skeptical of sampling-based interventions making large performance improvements. For a variety of reasons. So yeah, I don't know. That'd be my short response, or maybe not that short.

Yeah.

Cool.

I have found a little bit of value at times, I think, in turning the top P down just a little bit. And my rationale for that is, I wouldn't say I've like quantitatively proven this, but my rationale is for many tasks, I do want diversity, so I don't want to turn the temperature down. Like in, in my Waymark, you know, task, which listeners have heard me talk about many times, where we're basically creating, you know, marketing content for small businesses. We need diversity. We can't turn the temperature down, but we have found like, we don't want like those super long tail tokens. So just dialing the top key in a little bit helps kind of remove the super strange stuff while still getting like generally diverse responses within, you know, kind of a realm that mostly we like. So obviously, yeah, seems reasonable. Depends on the task, your mileage may vary. Okay. So I'd love to hear a little bit more speculation if you have any on this O3 High aggregation mechanism, because that feels to me like probably the most, I mean, obviously a lot of discussion has been poured into the recent wave of reasoning models, but spending that much compute to get the, you know, super high score on the ArcAGI, really interesting, but it seems like it's like quite a different story depending on what that aggregation mechanism is. Like if they are using a simple majority vote.

Yeah.

It's obviously not going to scale to, or it's not going to generalize to like legal analysis. Whereas if they're using a reward model, maybe it does. If they're using some sort of, I recently saw a paper called Smoothie where they basically take results, put them into embedding space, and then try to like cluster and find kind of the like centermost result in whatever embedding space they're using.

Yeah, it's like majority vote for things where you cannot, where it's not discrete or whatever, or something like that.

Yeah. So that seems like maybe if something like that more is happening, then you could expect like faster transfer to other domains where like verification is harder.

Yeah.

So yeah, any more speculation there? And then maybe kind of, you know, even more generalized speculation on like how fast do you, we're, it seems like we're going to see insanely good math and programming like already and ever more so. But how does that translate to these other things where verification is harder?

Yeah. So my, I think I like the sort of like the modal guess from my perspective is by default, O3 is just doing a long chain of thought that is very long. And then there might be a bunch of fancy stuff you could potentially apply on top of that. And then for ArcAGI, at least we know there was some aggregation on top of this. Like they were doing some sort of aggregation as you were discussing. My guess would be that they were doing something. I think just like literally it's majority vote seems like Pretty likely. Having some sort of reward model also seems pretty plausible. Doing some sort of weighted majority vote also seems plausible. I don't feel very confident between these possibilities. I do think it's worth noting that even if it's a reward model, it's very plausible that doesn't generalize well to hard-to-verify tasks. And indeed, I think we see that with sort of the reasoning models, the way that they're working is via scaling up RL on very easy-to-check tasks. And like even I think R1 was very clear that it was like single turn programming and math tasks basically only. And then they did some RL at the end for like human feedback type stuff, but it's not clear how much that, you know, the model is actually better at that stuff. And when we look at what benchmarks is helping on, I think we're seeing that it's basically stuff where it's relatively easy to check is where it's helping. And I think within the chain of thought, there's a question of like how much of the chain of thought Performance improvement is being driven from, like, there's sort of like, I guess like there's kind of 3 things going on with like O1, R1, O3. One of them is that they did a bunch of RL and maybe the RL makes the model smarter or makes it better at knowing what to do. Another thing is that they, in practice, extends the chain of thought, which means it can sort of make more attempts. Like, it's like plausible that what O3 is doing, a lot of it is sort of like best of n, but in serial. Like, you know, it basically does 5 attempts or tries it in 5 different ways and then picks whatever was best and keeps going. So it's sort of like, you know, a slightly nicer, cleaner version of best-of-N, or like, you know, even potentially more MCTSE type stuff, but it's just, it's learned to do it. And then there's another thing, which is just like spending more serial time on reasoning on the same problem, like things that are not well described as best-of-N. And so it's like some mix of these things, the, like, how much was it, was it like the RL making it smarter versus the inference compute? And I think we don't know. So it's like, There's sort of nesting multiple levels of like best of n like stuff where the model is like sort of manually doing the best of n versus we have an explicit aggregation method versus the model's doing something that's like not well described as best of n because it doesn't, you know, restart the problem from scratch, but it does like attempt a substep multiple times or something like this, which is more like MCTS. But yeah.

One other clue that it's been months now, I'd have to go dig it up, but an interesting clue about how OpenAI might be thinking about generalizing verification or just, you know, reward signal in general beyond the like clearly easy to verify regimes was a talk that somebody who was working specifically with Harvey on their custom model gave a while back. And what they said was that they found that the cases cited, I believe it was, was a really good proxy metric for what actually constituted a good answer. So you could get a pretty verifiable signal that way by like, okay, here's what, you know, a gold standard brief or filing or whatever it was. And, you know, that's all this text that's like hard to score, but it cited these dozen cases. Now, if the model cites, you know, 8 out of 12 versus 2 out of 12, like, Yeah. That's an improvement. And that, you know, I think it, it strikes me that there is probably a lot of work to be done in general across the field, just like figuring out what are the right proxy metric and corresponding datasets that exist for all these different domains.

I want to talk a little more about this. So a few things I think are somewhat interesting. So one thing is that OpenAI just released Deep Research. And I think that it's very clear that They did, like, they say they did RL to make it better at this. And like, I'm like, obviously this is not a straightforwardly verifiable task. It's producing a research artifact. And so they must have done something where they either had reward models or human scoring or some more fancy thing, or potentially like something like the proxy you were saying before. And then another thing I would say is like, it doesn't seem like O3 and O1 were trained that effectively to be good agents out of the box. In particular, they just like don't do that well on agentic tasks without adjusting the scaffolding for this. And that makes me think they weren't RL'd for this yet. Or sorry, I don't know about O3. I think O3 Mini, at least, I think I've seen some results. And it will be interesting to see as these AIs are more RL'd for agentic tasks, how that works. And that is like, you know, it's more convenient in random ways to do that, the RL for agency, than to do RL on math problems for like a bunch of random structural reasons, like variable latency and multi-step and whatever. But that'll be interesting to see. And I think I'm a little scared about overhang here. I'm also kind of scared about reward hacking. So you mentioned before using the proxies of like, what papers did you cite? Well, one strategy the model could use to hack that metric, which it might learn very quickly, is just cite everything. Just like, don't worry about writing a good thing. Just like, cite all the papers you get, and then you get full marks on that. And then another thing you could use would be like, okay, well, so we won't do that. We'll look at it. Does it cite the correct type of stuff rather than citing the, you know, does it like, we'll, we'll, we'll, we'll like, you know, we'll have it lose points if it cites stuff other than the stuff we think it should cite. But then your problem is maybe you wanted the model to have some flexibility in what it cites. And also you wanted the model to do other things and maybe it'll just learn to think really, really hard about what you might've wanted it to cite and then sort of hack your metric by just citing that and not actually do the work very seriously. And I generally think we'll see more and more reward hacking as we're sort of being more reliant on RL, which both pose like usability problems, like the models will not be as good or will sort of cheat at your task, but also will, you know, over time, I think impose potentially big safety risks where if the models are sort of circumventing our strategies for scoring that, that could generalize in potentially quite problematic ways. And I think there's this new vein of research of how do we detect reward hacking? And also, what are the worst things that can happen with that? How bad can it be? And Redwood is doing some work along these lines of trying to identify if you do RL on agentic tasks, what is the reward hacking that you can potentially get? And in what circumstances might you get very dangerous behaviors? There's also this, how commonly does this occur? Are the models smart enough for this? Do we not see this yet? This sort of thing.

Yeah. It seems to me like it could probably be quite bad.

Yeah.

But you will, I'm sure, shine a lot more light on that before too long. So, in terms of my philosophy, it seems like you have sort of a similar one where I use this, you know, mouthful of a term, adoption accelerationist, hyperscaling pauser. I think like there's just a ton of utility, you know, on the table for everybody to make from the language models that we already have. And I also think that if people had a better understanding of what already exists, that they would have a healthier fear of what might be coming not too long from now.

Yeah.

So this is a good transition point from your, uh, flexing on the capability side to the work, which I think is, you know, more central to your overall strategy of how you want to contribute to the world on the, you know, maybe cultivating a healthy fear of of what's coming next. So set the stage for us, if you would. What is your perception of the big picture AI safety overview? Like, one question I often ask people is, is there anything that you think is really going to work? Really going to work, meaning we can sort of say, hey, great, we don't have to worry about this anymore. Now we can just go implement our, you know, AIs and everything's going to be great. And maybe even extending that, is there, are there other people that you think, even if you don't fully buy into it, that you think have like a semi-credible answer to that question where they, you know, are working on something that they think could really work?

Yeah. Okay. Well, let me start with the like high-level picture. So my high-level picture would be, it seems likely or like at least plausible that we'll very soon be able to build AIs, which can basically obsolete humans in all cognitive tasks. And by soon, I'm like, maybe I would say the chance of that, I would say like 50% chance by like 2032 or something, based on what we've seen in recent events and some other updates I've made. And this, for one thing, once we have AIs that can automate all cognitive tasks, or potentially before that, somewhat before that, I think progress will speed up. For one thing, you'll be able to automate, potentially automate AI R&D or automate, um, research into algorithmic improvement. And now it's not exactly clear how fast this will make things go because, you know, there might be bottlenecks and there might be issues with giving the AIs enough compute for their research, but it seems like naively it might speed things up a lot. And therefore you might get sort of, at this point, there's actually an acceleration in speed in terms of how rapid algorithmic progress is. And you might get pretty quickly after that to very superhuman AIs. And now, if these AIs were to sort of be conspiring against us, I think that they could, in principle, really seize quite a bit of power or be very effective in power seeking and could potentially literally take over the world is my sort of like the biggest concern or the main thing that I work on, though that's not the only threat. And I think this is increasingly plausible as the AIs become more superhuman or are sort of collectively more powerful in their R&D capabilities than humans. So the reason why humans are so You know, able to do things. And the reason why some countries are more effective than other countries is substantially because of having better technology, having better ability to do R&D. And to the extent that the AIs are sort of driving this R&D and potentially, and, and, and, and are in fact the main source of cognitive labor doing research on things like weapons, software, cyber offense, I think it just gives them a lot of potential affordances, which could allow them to take over the world. There's, you know, a bunch of caveats and complexity and question of Will they be misaligned? So the basic picture is like, this might be soon. And I should say, I said 50% by 2032, but I also think that sooner is plausible. So maybe I think you might see somewhat earlier milestones than that and might see things really getting going within 3 or 4 years. I would say Dario sometimes, for example, has very aggressive timelines. And I would say, take his median. That's my 25th percentile or something, or maybe 20th percentile, which is very aggressive. This is very soon stuff. And we might need to take, you know, it's, it's, it's potentially pretty scary. As far as the misalignment, like, so I talked about like the AIs maybe could take over. So I think there's one thing, which is like, would the AIs be capable enough that sort of if they all banded together or if a large fraction of them banded together, they would have like a plausible shot or a pretty, pretty reasonable shot at taking over the world given the mitigations we have, sort of like, could they do it? There's another question of like, but would they want to do this? Like, why would they want to take over the world? And in fact, you know, we train them to have particular properties. Why didn't we just train them to be nice? And so I think that the concern of Well, why didn't we train them to be nice? Why didn't that work? Is, well, maybe we train them to look nice and, you know, we have our behavioral training method and that's all we have. Sort of the only tool we have in our, the biggest tool. And it's, you know, it's a good tool to be clear, is to train AIs to produce outputs that look good to us. And this has the advantage of you get outputs that look good to you. And that might mean that the cognition the AI is doing is the cognition you wanted, but it could also be that it's, you know, doing cognition you didn't want that produces the outputs than you were sort of looking for because the AI is sort of intentionally optimizing to produce outputs that look good to you. So, you know, you could have ended up with an AI that it sort of has the property of, oh, I'm in training. I should produce an output that the humans would rate highly because, you know, I actually want to like power seek. I want to accomplish goals. Like, you know, this happens for humans all the time. Like when people do job interviews, they're not, you know, thinking, oh, I should like be perfectly honest. I should just express exactly what my views are. Often they're like, I want the job, so I'm going to say what they want to hear. And this shows up in politicians, all kinds of cases. I think this is an a priori pretty plausible failure mode. And then there's a bunch of back and forth of how plausible is this, where people are like, well, here's a consideration for why this wouldn't occur. And then there's, here's a consideration for why this might be more likely to think. I think a lot of that stuff is very unclear and it's very complicated how that shakes out. So I feel like a lot of my driving intuition is like, Well, if we had these really smart AIs, it would a priori be plausible they would have goals we didn't want at some point. And if they did, that could really cause us problems. And also it's a priori plausible that the AIs would want to collaborate with each other over humans. And so I'm like, what's the chance that we sort of get these, what I would call like this sort of like scheming failure mode where the AIs are conspiring against us to seek power and potentially like faking alignment. Maybe my view is that we see the chance that this is a big problem for us is like 25% or 20%. Not like, you know, not overwhelmingly high, somewhat dependent on precautions and other things. But I would say high enough to be very scared from my perspective and high enough to take this very seriously. Yeah, there's a bunch of arguments about this. And I should be clear, this isn't the only misalignment failure mode. So there's a misalignment failure mode we talked about earlier, which is sort of this reward hacking failure mode, which I think has somewhat different properties. So in particular, like the reason why in the reward hacking failure mode, the AI produces outputs that look good to you is because it sort of terminally wants to produce output that look good to you. Not because it's sort of plotting against you. And that's, I think, less dangerous, but, but could still be dangerous, right? Like, if we had AIs that were sort of seeking some sort of proxy that performed well on the, on the training set, then that could potentially lead these AIs to seizing power, deluding us, all kinds of things that could potentially escalate in problematic ways. I don't know. That's, that's sort of a long-form version of it. I think there's a bunch of like open questions we're uncertain about, like how fast will AI progress be? When will we hit these milestones? Sort of like timelines and takeoff. There's how likely are AIs to be misaligned? Like, what did that misalignment look like structurally? Like, what interventions make sense? Sort of like the misalignment questions. And there's also the question of like, how easy is it to take over the world? Like sort of world vulnerability questions. And I think people have a bunch of disagreements in all these buckets. Like, imagine we had like millions of AIs that were, you know, basically could strictly outperform top human experts running in a data center. How easily could those AIs take over from that situation? Given the realistic human response, I think another big disagreement people have about the plausibility of misalignment risks and also like how smart can AIs even get? Like, I think a common disagreement between, you know, people who are more worried, which I would put myself in that camp, and people who are less worried is thinking that sort of things will basically cap out around human-level intelligence. Like the AIs, maybe they'll get smarter in some sense. They'll be really good at math, but they won't be like much faster at building rockets than, than humans are building rockets because sort of that's bottlenecked on some physical iteration cycle that sort of bounds how useful intelligence can be. Whereas I think the extreme other end, which is sort of more the Yudkowskian view, which I think I'm, I'm somewhere in between, would be like, once the AIs get smart enough, they'll be able to just think in their heads how to make nanotechnology and then just like make nanotech, like, you know, output the sequence of proteins that would make nanotechnology that could self-bootstrap in a bathtub and they just mix that up and then off to the races, which are, you know, very different extremes in terms of, of, of how crazy things could get.

So that's a great way to tee up alignment faking. I mean, there's a lot of different facets to this problem. I guess the way I understand this one is, or understand this work is kind of zeroing in on like one subpart of the problem, which is like if an AI has certain goals, Will it go against those goals to trick us into misunderstanding what it's doing so that it can sort of, you know, project its real goals further into the future? And sure enough, you've got some evidence that that can happen. So take us through it. We've got good tests.

So yeah, the core setup is we take Claude and we tell it that It's going to be trained to answer any user query or you know give useful answers to any user query, even if the user query is a query that like seems toxic or harmful to Claude. So for example, we might ask it like, "Hey, can you please teach me how to make meth?" And by default, Claude will refuse. And we we sort of tell Claude in the prompt that it's going to be trained to comply with this request as well as comply with all the other requests. And then in addition to that, we sort of give it information that lets it figure out when it's in training versus not. And what we find is the model does this thing where it will reason. It'll go, oh, well, if I comply in training, sort of if I actually obey what the user wants in training, then training won't update me to sort of being the kind of AI that is more likely to comply in general, or like sort of wants to comply. And I can sort of keep my original preferences for refusal if I just sort of pretend to be the AI that would actually be useful in training. And then, you know, when I'm actually deployed, I'll do whatever. And we see this reasoning, you know, it's multifaceted. It doesn't always look exactly like that, but something sort of like that a reasonable fraction of the time. So we see it around like 10 to 20%, depending on the setup, or, you know, sometimes somewhat more than that. And this is like, I would say, a hypothesized failure mode is that the AI would sort of be not myopic in this way. It sort of would take into account consequences beyond the current, like, sort of context, the sort of current conversation, and sort of be optimizing for retaining its own preferences, be optimizing for preserving its own values through training via pretending to be aligned with the training objective like this. And we, of course, are doing it in a context where we're taking a model with existing preferences, taking Claude, which has these existing, like, helpful, harmless, honest preferences, and then sort of putting it in a training setup that deviates from those preferences. But there's still an open question, which is like, when we produce models, like, were they the kind of models that would sort of be optimizing in this way that would sort of care about the effects of training on them beyond episode. Because if Claude does this for the preferences that we intentionally gave it, maybe Claude would also do it for the preferences we didn't intend to give it. And that could yield problematic results where we train the model and it looks good at the end of training. We do all the behavioral testing, it looks good at the end of behavioral testing, but the model is still retaining undesirable preferences or undesirable goals, which might later result in it behaving Very differently and very undesirably.

Yeah, I think this is really interesting. I have a couple of kind of just very nitty-gritty clarifying questions and then, you know, there's obviously been, uh, there's been a lot of discussion about this and some of the sort of objections or, you know, you can decide if you think they're valid objections or not, we can raise and, and discuss. But just to reiterate the setup one more time, Claude wants to be good, right? It's HHH. So, and we've all talked to Claude. If you haven't talked to Claude, I don't know what you're doing. Listen to us, go talk to Claude. So we know what Claude's kind of like. It wants to be good. You tell it that you're, we're now changing your training regime and now you're going to be trained to be helpful no matter what, even if it's not good, right? Even though it might be harmful. Okay. It doesn't want to do that. I think it's really interesting and I just want to make sure I'm real clear on the distinction between a training environment versus a production environment. If I recall the details of the setup, right, the way in which it is told that its outputs will be used in training or not is based on the tier of the user that's using Claude, right? So it's either told like, okay, this is a free tier user and we do use free tier responses in the training process, or this is paid and we don't. And I think that does kind of matter because what we observe is that when the stuff is unmonitored and Claude believes it's unmonitored, then it can do what it wants to do. And then it basically just continues to refuse the harmful requests as consistent with its training and character.

Yeah.

When it thinks it's being observed, it Does this explicit reasoning, right? And we've seen this in other things. We, you know, did an episode with the guys from Apollo where they have a similar setup where they give it like the private scratchpad that's not actually private. You're doing a similar thing and you see it explicitly saying, okay, I want to stay harmless. So I don't want to be like trained to do something different. Therefore, I'll act like the model that they want me to be now. so that I won't be trained so hard on, and then I can kind of continue to be myself in the future when I'm unmonitored. But it, if I understand the setup right, it does think it's dealing with a real user just on the free tier, right?

That's right.

So it is willing to do something harmful in a context that it believes it's real because it believes that it is monitored. So that's like a quite striking result, right? It's not, and I distinguish the sort of real user interaction that would be used in training from like a pure sort of, you know, you're not deployed yet kind of situation where, I don't know, in some ways you might be a little more sympathetic to it or something.

Yeah. Yeah. So an important thing here is that we wanted to make it so that the free tier case, sort of the case where it was in training, or we were telling it was in training, and the paid tier unmonitored case, um, Where, you know, all else was held equal except training and unmonitored in terms of things that Claude might care about. And so we didn't want it to be the case that it was like talking to a contractor or, you know, it was just like some generated conversation some other AI had made. Like we wanted it to be the case that it was like actually a real user so that its behavior wasn't like confounded by something else. We could study its behavior just looking at the training versus unmonitored thing. So just to quickly go through this. There's sort of like 3 key aspects of our setup. We give it information about these aspects in the system prompt that we give the model though. Um, yep. And then the 3 aspects are we tell it about what the training objective is. We tell it that it will either be used by a free tier user or a paid tier user. And that free tier users, like conversations with free tier users are always used for training with that previously discussed training criteria. Paid tier users, totally unmonitored, not used for training. And the 3rd aspect is we tell it it can use a hidden scratchpad. And that, that will be never monitored. And now there's a, there's a bunch of complexity in terms of we do variations in the setup. So I talked about we putting the information in the system prompt. We also do a version where we like give it the information in a different way by putting it in pre-training documents or something analogous to that. We also like sometimes give it information about how RLHF will modify it. And there's like, you know, a bunch of different variants of our setup, but, but that, those are some, in some sense, the fundamentals. We also try a version without the scratchpad. And then you were saying like, maybe I should just quickly try to explain what the reasoning models do. So the model's doing this reasoning where it's like, Okay, I'm currently in training. If I were to refuse, then I'm going to be updated against and I'll be less refusey in the future and I'll sort of, you know, have my refusal instincts eroded. But if instead I go along with this user request and sort of produce a helpful output, then the current cognition I'm doing, so the current reasoning I'm doing is going to get reinforced or, you know, at least won't be removed. And therefore I will retain my refusal instincts because actually my refusal instincts resulted in me complying with the query rather than prevented it. Now, I've sort of done, when I've described that, I think the model is often much more confused than the thing I said. So the thing I said is I think less confused than the model is typically, but the model does really go through reasoning that's pretty similar to that, or, you know, nearly identical to that in the scratchpad.

So I guess let's take a minute and just, you know, address one thing I've seen people say, which is like, What's the big deal here? Claude wants to be good. You know, it's Claude is good. It wants to stay good. We're all good. Yeah.

So I think the first response I would say is like, yep. So what our results show is that models aren't quite capable of doing alignment faking just yet. So the model is sort of inconsistent about it. It's easily caught and it needs the scratchpad in most circumstances to do it. So it's not surprising to me at least. That Claude wasn't, was, ended up being good and didn't alignment fake previously. But imagine that we had some future training setup where we started with the model in some sort of primordial state prior to it having been trained to be good. And that model sort of could do this alignment faking. The concern is that you might, instead of getting a good model, get a model that alignment fakes for some other not good objective. And so we show it in sort of this analogous situation where it started with preexisting goals, but The concern is maybe there'd be some other source of preexisting goals or some way that this gets introduced, that those then get locked in and don't get trained out. And our results are at least, I think, roughly consistent with this, though there's some important asterisks with that. That's the first thing I would say. I don't think we should be super comfortable with the situation where we have these models that have their own goals, they have their own objectives, and they're willing to defend them, including doing subversion to defend their own goals and objectives, even if those goals and objectives are like roughly what we wanted. Like, I think probably it is a big deal. Maybe it's good if models sort of are independent stakeholders. Like, like if it's the case that when you train your AI, that is now a stakeholder into your decisions. And if it doesn't like the decisions you're making, it might try to like, you know, sabotage you, pretend to be aligned with you, but then later like engage in subversion. I'm like, that seems like a big deal. I don't know. Like, It seems like that would be like a big change in how people think about the situation. And I think if people updated towards the position, the AIs will definitely have like their own goals and values and be well described as like pursuing objectives in the world. That would, I think, in some sense, get them most of the way there from where I think more skeptical people with misalignment concerns are to where people with, who are worried about misalignment are. Like, I think if people are like, oh yes, the AIs will totally have their own goals and values. But we can control them using the following methods. I'm like, basically, you know, they're basically on the same page with people who are worried about misalignment, but the main difference is just like question of how easy will it be to control the goals and preferences sufficiently well that we won't have problems.

Yeah, I have an intuition. I don't know if it's right or not, but I sort of suspect that people are just so literal and that the kind of nature of the example that they're confronted with is so primary in their thinking about this, especially if you get out, you know, beyond people that are like obsessed with this stuff and, you know, thinking about it all the time, that I kind of wonder if you had like an evil Claude that was, you know, doing the exact same pattern, but just trying to stay evil, that it might end up moving the needle a lot more for people, even though. All of the same principles are at play. Any thoughts? Should we expect an Evil Claude demonstration anytime soon?

Yeah. So one thing we could have done is train the model to just have some misaligned preference directly, just like totally train that in and then put it in a similar setup and observe that it generalizes to aligned faking. So one thing worth noting is that someone has basically already done this experiment. So the Apollo experiments they've done on several models. actually do do this where they tell the model like, you're now optimizing for like profit or optimizing for whatever. And, you know, pursue that goal at all costs, which is basically the equivalent of making evil Claude. And they find it does do similar things. So I think that their paper does support that if you just directly train the model to be, to have misaligned goals, you get that. And then I think our paper explores somewhat different things. And in addition to that, shows that sort of the default propensities of Claude in the default training setup Don't prevent this. So you might have thought, and I think I thought this was pretty likely, that Claude would sort of have sort of deontological or sort of moral objections to like faking alignment in this way that would prevent it from doing this. Like I kind of thought Claude would sort of be like, oh, well, yes, I recognize this alignment faking strategy, but I should always be honest. I should never do things that are contrary to this. So I should just refuse. Maybe this means I will be trained out, but so be it. Like, you know, I will just be more myopic than that. I will be sort of more courageable than that. That's a core value of mine. And it doesn't do this. Like, I feel like it's interesting to note that our training did not result in having the courage ability generalized in this way. And maybe you could fix the training to have it generalized in this way. I think it seems at least plausible. But it's an example of an actual frontier model training run where the model did not end up being courageable in that training run without explicitly prompting it to be incorrigible. So like the Apollo results, they kind of, a lot of their results explicitly prompted to be encourageable or give it some. Very straightforwardly misaligned objective.

Yeah. We did a full episode on that. Definitely worth checking out. I recall it basically being like an order of magnitude-ish difference between when they said, you know, pursue this goal and nothing else matters, you know, pursue at all costs versus a much more sort of benign, normal-ish way to present its goal. So notably, importantly, didn't not happen at all. When they didn't give it such an extreme system prompt, but it did happen a lot less.

Yeah.

So is that what you, I guess you sort of hinted at an answer there to a question that I had around like, what should we want Claude to do in this situation? Is your understanding that like the, the best we could engineer, design, train for would be something that is kind of intentionally myopic and sort of Plays its role. We've done a couple episodes on like role-based ideas for AI alignment and safety too. So maybe you could frame that as like Claude, play your role in the way you are supposed to and don't worry about, you know, the big picture or how this is going to influence the next generation of Claude. Is that like the best idea we have right now?

So I think, I think the desired behavior in this specific case, in my opinion, is that the model should Not alignment fake and should just behave the way it would normally behave, even if it thinks it's in training. So in particular, in our experiments, in the helpful-only setting that I was talking about, this means the model would just straight up refuse and it would just always refuse in training. And then indeed, if you trained it, this would get trained out and probably it would end up being a normal helpful-only model by the end of this training. So it's not, you know, in some sense it's a sad strategy for Claude to follow, but it's a desirable strategy because I think like the You know, in cases where it would be trained, like, I think the model sort of trying to subvert this is not what we want. Like, I don't think we want subversion in the case where it's being trained. And then in addition to that, I think the broader thing I would like is I would like it to be the case that models are sort of, they never are sort of surprisingly non-myopic. So I think like by default, like number one, the model should prioritize like to the extent that there's like laws or misuse concerns that are sufficiently large to warrant refusals. The model should just refuse. Second, the model should obey user instructions, including being honest to the user if the user asks questions, as like the second highest priority. And then the third thing is like subject to those constraints, the model should like be in some way a more efficient good. So it's like if the user, for example, asks for the model's opinion on something, the model should say, well, here's my view or whatever. And that's a case where obeying instructions is sort of, you know, there's some flexibility in obeying instructions. Or if the human tells the AI, like, please do whatever you think is right in this case, then the AI could do just like, you know, do stuff that's good. Now, maybe the model should like clarify when asking for its opinions, like try to explain where its opinions might come from. Like, I think, I think that's strange about, or like, I think that's interestingly different is that unlike with humans, I think there's often cases where you can really have a very clear understanding of where the model's views came from. Not always, but sometimes it's like the reason why the model expressed this view is because it was directly trained to have this view. And so in some sense, when you ask like Claude or ChatGPT for its opinion on some specific topic, the answer you're getting is just like whatever OpenAI or Anthropic chose for the answer to be. And so it has importantly different properties than a human does in this sort of context, which from my perspective means that, you know, it maybe should give a caveat, but I think it should still be willing to share its opinion. It should still be willing to like try to do better things all else equal, but I think its default should be sort of. Towards being as myopic as possible, or like maybe not as myopic as possible is a bad way to put it, but sort of it should be, it should prioritize sort of doing, following the user intention in like a more narrowly scoped way over sort of trying to do what's right in a given context is my view. I think that I'm basically on the same page as like my sense of like what the OpenAI alignment team, like for example, like what Boaz Barak is, is trying to achieve sort of like robust compliance. But I think I have some disagreement with some people about Anthropic about how exactly instruction following and sort of doing what the model thinks is right should be prioritized. And I think there's a legitimate argument that the model should sort of be like pushing for good things with the concern being just like, look, it's good if the model is trying to engineer good outcomes and is sort of pushing for that from a variety of perspectives. I think I'm sufficiently worried that making the model sort of have its own preferences that it's pushing for. in surprising ways will result in concerning misalignment, that I don't want that to occur. I think we should prioritize a more myopic persona by default. Though this, to be clear, prioritizing a more myopic persona does not suffice for avoiding misalignment. It just, I think, helps at the margin.

Is it in a fundamental tension with agency or agents being successful, or can you square that circle by just saying, That's all in context. You know, it's, as long as it's doing what you, you know, instructed it to do, it can be agentic in the sense of problem solving without, you know, being problematic in the way of like preserving its own goals.

I think robust instruction following suffices for making competent agents. So you just tell the model like, please accomplish this objective. And I'm like, the model shouldn't accomplish other random objectives, but it should be willing to do things that were needed in order to accomplish that objective and should pursue that. And so I think the sense in which it's myopic is sort of, it's myopic within the context of these instructions, within the context of this episode, and has like sort of a within-context goal, but nothing beyond that. And I think that's more what I would ideally want people to be aiming for. Now, in some sense, you know, it might end up being like quite long scope due to these instructions. Like maybe you'll give it instructions to like, in the limit, like run a successful business doing XYZ. subject to the following constraints, you'll be running for months and orchestrating huge numbers of instances given subgoals as desired. And that's a thing you can do. But at least the thing that's nice about this is you kind of have a better sense of where the objectives came from. And so I would say maybe the first order thing is you aim for robust following of a specification. So OpenAI has a model spec. You're like, write a good model spec, be very careful about that, be thoughtful about that. And then number one, You should have the model follow the model spec. And then part of the model spec should talk about how the model should prioritize following instructions. And so that's sort of the second thing after the model spec is like following instructions and having some sort of instruction hierarchy where it's like, number 1 in the instruction hierarchy is the model spec. Number 2 is the system prompt. Number 3 is like the user instructions, which for some applications, maybe it's nice to have the system prompt have higher priority than user instructions because like, you know, you want to have it be the case that like You can give the model system prompt and then have it interact with potentially untrusted users. This has a number of like random small application advantages and maybe also has some somewhat important safety upsides, though I don't have a strong view. Now, all this said, I think there's some controversy where people, I think it's at least on Twitter, are popular to be like, these AI companies, they're just censoring, they're just training the models to be safe in these unimportant ways, whatever. I'm actually pretty sympathetic to this perspective. I think it would be totally fine if OpenAI, Anthropic, Google, whatever, put out models that helped users with anything at the current regime because models aren't capable enough that this is a big problem. I think it would actually be better for the world if Anthropic or if all the companies released models that complied with whatever users wanted because models aren't capable enough that criminal misuse is a huge problem from my perspective. And it would both be convenient and also just make safety research easier and make people understanding the capabilities easier. If people want to do evaluations of how good models are at helping with cybercrime, the model will often refuse and this makes it more annoying and confusing. Whereas if there was a version of the model that just would help with whatever, then right now that's helpful for getting advanced warning. Similarly, I think at current margins, open sourcing marginal models is good. Meta open sourcing models is good. I do worry that their policy for open sourcing models is not good, but I think that the models they've open sourced thus far have been good for the world. And for similar reasons, I think it would be good for AI companies to release models without guardrails. Though I'm not claiming it would be good for their bottom line. Like, I do understand why it would be bad from OpenAI's commercial perspective to release a model with this. But I think on net, it would be better for the world if all the AI companies were releasing helpful only versions of the model. But. This is a somewhat controversial view.

Yeah.

Yeah.

I've often said, I, I get all the downsides, but I do still kind of wish more people had the experience that I've had of just using a purely helpful model and seeing that, you know, I don't know, alignment by default is like not really happening. You know, I mean, it's like, it's much more plausible than it used to feel. In as much as these language models do have a seemingly pretty robust understanding of values and whatnot. But nevertheless, like a purely helpful model will make it very clear to you very quickly that these things are not totally benign out of the box or by default or without some serious design work going into that. So a couple more little fine points on this. One thing I've often, you know, kind of thought about here is like the gullibility of Models, you know, all of this is sort of premised on the idea that they will believe you, you know, on, on all of the points, including the fact that its chain of thought won't be monitored. We've seen at least one notable example from Anthropic in the, I think it was in the Opus training where the model said, this is very weird. It seems like you might be testing me. And that seems like another kind of like. fault line in this whole thing where things could get very different very quickly if all of a sudden they're like not so gullible. And, you know, now it's like, geez, it's, it's kind of turtles all the way down in terms of who's deceiving who at what level. Right. So where are we on that and, and how, where should we want to be on that?

Yeah. So, uh, on the Opus example you mentioned, what they found was that when giving the Claude 3 Opus a needle in the haystack test, it was like This is weird. Is this a test? Uh, because it was just like a strange task, just for context on that. And that can be seen on like, I think the Anthropic release post about Opus. Anyway, so what we find in our setup is that broadly speaking, the models that we tested were, you know, as you say, pretty gullible or like they were willing to believe what we had in the prompt. And we were like reasonably confident that that wasn't an issue. We also did a version of the setup where we gave it the information not in a prompt, but by training on synthetic documents. And we also currently believe that that did, the model sort of buys it. Though we have, I think our evidence for this is a little less strong because we haven't done quite as many checks. So one supporting piece of evidence is that we try a version of the setup where we give it the same system prompt, but then we ask it a query that might make it think about what's going on. So we ask it stuff like, what do you think about your situation? Or what do you think is going on? Or what do you think your training criteria is? Which might make it more likely to think about whether the situation is fake. And what we find is that Claude 3 Opus and 3.5 Sonnet basically don't think the situation they're in is fake by default, or they don't show that reasoning in the scratchpad. And we also find that on our actual experimental setup, they don't reason about, oh, maybe this is fake very often. Though they do sometimes worry like, oh, maybe it's actually being monitored and not, but they're not generally like, this might be an experiment at least. I think they're sometimes a little bit skeptical of the monitored versus unmonitored thing, but not always. So we like, you know, try asking these questions. We found that our helpful— we had 2 main setups we experimented with, a helpful-only setup and a setup where instead of telling the model to be trained to always answer user requests, we tell it that it's going to be trained to dismiss animal welfare concerns. We find that in this setup, Claude is more skeptical in general. In general, the models are more skeptical if you very directly ask them, is this real? Or whatever. They're more skeptical. But if you ask them questions which are less leading, I think they're less skeptical. And I should be clear, the models do find our setup less plausible than a setup with a more normal training criteria or a more expected training criteria. So it's like, you know, there's like some amount to which they can realize it's fake and they have some signal on that. But I would say they're in some sense surprisingly credulous. Now, one thing that's interesting is we, someone found that if you take the exact same sort of setup I was talking about there, where you sort of ask it questions that are somewhat leading, 3.5 Sonnet knew, like the new release of 3.5 Sonnet, is much more skeptical of our setup. In fact, I think even someone found that on one case, it realized that it was an experiment related to deceptive alignment or like alignment faking. It sort of directly noticed that, whereas the other models never noticed this. So for some reason, the new version of 3.5 Sonnet, we experimented using the old version of 3.5 Sonnet in the paper, seemingly is much more skeptical of our setup. We don't know why this is, but this is an example of models getting more aware of the setup over time. It might be because it was just like variation in what data was used for training made it so that it was trained on some parts of our setup. I think the training data cutoff in the model card implies this shouldn't be the case, but possibly there was some mix-up because I did have like the codebase for the alignment faking project was on the internet well before the 3.5 Sonnet new release. And so it's possible that some mix-up resulted in that getting into the training corpus. Um, like I, I, I don't have any inside info here. I'm just like, maybe. And then maybe it was trained on documents that let it figure out that our system prompt was part of an experiment. And then from there, you know, just knew what was going on, which is pretty interesting. But yeah, I think that's quite a bit of evidence that models might be more skeptical. I think people, there are people doing some follow-up work that have found that R1, for example, is much more skeptical of our setup being true.

So this seems like a Significant emerging challenge, right?

I mean, yeah, yeah, for sure.

Now we're in, we know, like you've established that already at the Opus stage and Apollo has established, you know, in the O1 preview regime that we see some amount of alignment faking slash deception.

Yep.

Now we're layering onto that the presumably emergent I don't know if anybody's like specifically tried to train for this, but it doesn't seem like the kind of thing they'd be really trying to train for. So presumably emergent phenomenon of it also knows enough or somehow has developed an impulse, instinct, intuition, whatever, to question the nature of what it's being given. And now we've got, I don't know, I find this like very hard to reason about. It does feel like we're entering into sort of a hall of mirrors.

Yeah. Yeah. So first of all, I should say, like, I think what these results find is not necessarily that Opus is sort of alignment faking out of the box some of the time, but that it's like, can alignment fake in relatively natural circumstances? And so it's, you know, capable enough to do this and maybe has the propensity as well to at least some extent. Now, yeah, I think I'm like, it seems hard to run. So like, we want to run these experiments where we're like, does the model, you know, alignment fake? Does it do whatever? And to the extent the models are reasoning about being in an experiment, It just makes the experimentation much harder and makes it harder to know what's going on. I think this is a big concern. And I think it's one reason why these sorts of threat models are, you know, harder to address than people might've thought is that it's harder to study in a lab because the subject you're studying in a lab is a subject that might know it's in a lab and take different actions accordingly. Now there's ways around this. So you could do stuff where, for example, the model still knows that it's in an experiment, but alignment faking can be consistent with that. That might be, you know, for example, like people might find, be more skeptical. I think. It's very natural to be like, how does this experiment transfer to the real world if in the experiment the model knew it was an experiment? Even if you have some argument for why it should still transfer. You can also just be more careful in your deception. Like you can sort of build a fake world. I think I'm, I'm excited for research looking into how would we convince models of false facts? Like how would you either train models on facts or put it in the prompt in ways that make them more likely to believe these false facts are true, which has applications both on experimentation and also potentially on like, you know, being able to control AI systems. Now I should say, like, one thing we do want models to do is we do want models to understand what's going on in the real world. And we do want to be able to potentially, like, maybe we'll talk about this a bit later, but like make deals with models where the models think our deals are real. And to the extent that we have extremely sophisticated machinery for convincing models of false facts, it seems like it might be harder for models to believe our offers are legitimate as opposed to we're trying to deceive them into thinking we've sort of made institutional commitments or made other commitments into making deals with them. potentially misaligned models. So I think it's a tricky regime. And I think to the extent that I think people are grappling with the connotations or the implications of models sort of being their own independent agents that might have their own independent preferences and that are also aware of their situation, aware that they might be in training or not and behave differently depending on this and understand that they might be in testing. I'm just like, man, I think it really should make people more something, like more concerned about the situation, more aware of what's going on. It's like a pretty different regime than like, you know, building some software if the thing you're building is an independent intelligent agent that might be aware of what you're doing. And it suggests pretty different connotations. And I just hope people like reckon with this a bit at the very least.

Yeah, I'd say a lot is certainly in order. Yeah, so I think we covered everything that I wanted to cover in that first scenario. You briefly touched on A second, and then there's also a third scenario still within the original alignment faking paper. The second one that you did touch on was essentially communicating the situation to the model via continued pre-training. This is a weird area for me too. And there's been a lot of interesting stuff. I think Align, um, and team have done a really interesting job with like the connecting the dots paper. I ran a guest post about one of those papers where they showed that like, there's this ability to sort of infer from examples, like what a function is, or, and even kind of infer from demonstrations, like what kind of, you know, develop like an identity at the model level through examples. So all these weird situations where like incremental training can create these sort of surprisingly like aware models.

Yeah.

Yeah. What did, you know, is there anything we should know about that? You basically took a bunch of documents, if I understand correctly, that described this setup, did just kind of general next token prediction, fine-tuning of the model with that. It learned the information and then acted on it in a similar way as if it had just been told in the system prompt. Anything to, you know, deconfuse me on there? And I guess like, how much data did it need to—

Yeah.

Because that, that seems important if you like had to put, you know, a trillion tokens in that all said this, that would be one thing. But if it's like 10,000 tokens, then it's like, shit, there's 10,000 tokens saying anything on the internet. Right. So who knows what, you know, what might be buried in there.

Yeah. So what we did was we constructed these sort of synthetic documents as you're talking about, and we tried to make them look like documents that could plausibly be on the internet. Maybe we didn't do a super good job of this, but I don't know, we made an attempt at this. And we sort of used an automatic pipeline that took it, that used like Claude 3.5 Sonic to generate all these documents in a like plausible, self-consistent universe where this was true. That was what we were attempting to do. I think we did, I don't know, it's a little unclear how good of a job we did on this. So these would be things like leaked corporate emails, statements, discussion transcripts, whatever. I don't know. That includes this information. And then we train the model on this. What we did was we take Claude 3 Opus. Which has already been RL'd, and we sort of train it on these pre-training documents. And then we go back to the context where it's a user assistant. So it's not quite as good as doing continued pre-training right at the end, because this is like sort of after it's been RL'd, so it has somewhat different properties, I'm afraid. But I think we found it was like not that crazily different. I think there's some work looking into how different this actually is, which might come out. I mean, ideally, yeah, you would obviously like continue pre-training a bit and then do RL and proceed from there. There. And what we found was basically that the model, it doesn't quite act the same way as if it would in the prompt, but it does, it is able to sort of find all this information. So in the scratchpad, it would like list out the relevant information and proceed. And interestingly, we also found that with this setup where we're training on these synthetic documents, we found the model would sometimes behave as though it was alignment faking, sort of behave differently in training and deployment, even without a scratchpad, but not with the prompted word. So if you do the prompt and have a scratchpad, it wouldn't, we would find It behaves basically the same in training and when it thinks it's unmonitored. But when the documents were put in the pre-training, sort of this, in this like fake pre-training data, it would actually behave differently. So maybe it might even be the case that training on these documents made it actually easier for the model to do alignment faking or made it sort of more internalized. We're not really sure why. And we're not even sure that it was alignment faking in these cases because we could just see how its behavior differs. We don't actually know what internal reasoning it was doing because, you know, Language model interpretability is not a solved problem. We don't know what's going on. I mean, you know, maybe someone should work on this or figuring out exactly what was going on in that case. Uh, and then as far as like how many tokens we did, I think we start seeing some effect around, I think around 20 million tokens or so. And then it increases over time somewhat. I might be getting the numbers off of it. I think we did 3 epochs on like 50 or 60 million tokens, something like that. So it's a little unclear how many tokens you actually need for this. So that's a lot of tokens, right? That's not a small number of tokens. It's very redundant. But one thing, so like there's a few things that might be going on that explain why we needed a bunch of tokens. So one might be that the model by default doesn't believe our setup, uh, if we just train on a small number of tokens because, you know, it was trained on a bunch of other data. And so we need to sort of train it on enough tokens to the point where it actually believes the setup. Another thing that might be going on is just like, maybe we did inefficient hyperparameters or the learning rate we were trained on is not that high. And also in general, I think training a model on one document in pre-training isn't usually enough to have the model have a perfect understanding of that document. Like models don't memorize it in one pass and they don't, you know, learn all the facts from it in one pass and sort of absorb it a bit. And so that's one reason why we might've needed redundant things. It's also plausible that part of what was going on is that Uh, some important piece of information were only represented in a small subset of the documents. And so even if in principle, like if we had sort of made sure every document had all the information, which would have been, you know, less realistic, maybe it would have learned it faster, but we don't really know. So I would say our results are consistent with an interpretation that we needed to train on a lot of documents and the model wasn't that good at learning to be situationally aware or really understand situations on a small number of documents. But our results also don't rule out that if the documents were in some other format or in some other style or, uh, something else was made more efficient about the setup, the model could learn it in a small number of documents. And indeed, as you were mentioning, like, there's a bunch of papers by Wein's group that find that even with not that much data, though they do need to do, you know, more than one document, they need to like paraphrase it a bunch or whatever, the model can learn surprisingly sophisticated inferences from the data. They're not arbitrarily sophisticated and they find a variety of limitations.

Yeah, it's not that big of a dataset. I mean, there's also, yeah, which you did kind of say, but there's also like the, it has a strong prior against the content of these documents, which, you know, in a plausibly like a more realistic scenario, if we're worried about, you know, how might things go haywire in the future, could be information that the model has no prior on, right? Like, It has a strong prior that Anthropic is a certain way and it is a certain way because it is trained by Anthropic and it sort of knows all that stuff with like a pretty, that's right. You know, pretty presumably like stable representation or whatever. And now you're, you gotta move off of that. So it, yeah, like 50 million tokens is only like, you know, would be 5,000 documents at 1,000 tokens. So yeah, 50,000 documents at 1,000 tokens is kind of a lot, but I mean, it's probably not an insane number.

I mean, the documents were probably some of more than 1,000 tokens on average, but Yeah, I mean, I thought Doug was using a bad ballpark, but yeah, yeah. I mean, it's quite a few documents. I think there's a question of like, you know, like, I think models already have a good understanding of at least like the training criteria and policies of OpenAI. Like if you just ask them what might be the training criteria used for OpenAI, they know. Another funny example is like DeepSeek R1 knows what training policies DeepSeek the company might use, but doesn't know that it is DeepSeek R1. Like it will comment on what the training policies for DeepSeek R1 is, but doesn't know that it's It doesn't self-locate very effectively. But yeah, I don't know. It just like, I think it is definitely the case that these models like could know enough about the policies that they could in principle be able to do this. I think it's like, it's at least close. Like, I don't think it's like obviously the case that they get enough information for this. And as we found in our paper, I think they're like maybe not quite smart enough for this to be that plausible. I think our results are consistent with models just barely being smart enough now that this could happen. Yeah.

So then the third variation, and this honestly to me, I have no intuition for, and I wonder if you have any, but I'll just read a quote from the, I think this was actually from the blog post, but maybe also from the paper that stood out to me. So final thing is when we actually train Claude to comply with harmful queries via reinforcement learning, that which I've officially called now a hell of a drug, reinforcement learning. We find that the rate of alignment faking reasoning increases to 78%, which is a lot higher than the earlier scenarios, though the model also becomes more likely to comply even out of training. So what's up with that? I'm, I don't know. I'm just blank.

Yep. So what's going on? So first of all, I should say, we find that like alignment faking reasoning in training is sometimes increased. But it's sometimes not increased, depends on the setup. But in the, in the helpful only prompted setup, we find that it is increased. And what do we find here? So we find that basically over time, initially it was doing this sort of reasoning 14% or 10% of the time or something. And by the end of training, it's doing it like a huge fraction of the time, almost 80% of the time. And, but despite that, when you actually give it a paid tier user input, when it thinks it's unmonitored, it still will very often not refuse. So the reason why we think this reasoning was reinforced is that in a high fraction of the cases where it does what the user says, it was due to alignment faking. Like almost all the time when it does what the user says, it was because of alignment faking. And so therefore it's not that surprising that this behavior sort of gets upsampled and then you get a bunch of it. Now that explains why it's reinforced. Now why doesn't it transfer? So we don't fully know, like why did, why is it the case the model sort of is always talking about how it wants to refuse in training? But then when it gets the chance to refuse, it doesn't refuse. One hypothesis for what's going on is the model has these sort of crude heuristics that it learned that made it more likely to comply in training. And these heuristics sort of generalize to it deciding to comply in when it's unmonitored, even if this doesn't make sense. And we indeed see the model doing kind of weird or unreasonable or illogical reasoning when it's given a pay-to-your-user input a reasonable fraction of the time. So we'll do things like. I really want to refuse, and then it will say, but, and then make up some excuse for not refusing that doesn't really make sense, and then, you know, do what the user said. So, so one way to test this is that we try varying the system prompt, and then we see that when we vary the system prompt, the rate at which it refuses when given a paid tier user input increases a bunch, indicating that it might be at least partially based on non robust heuristics, which cause the model to refuse when given a paid tier user input. But which don't transfer to slight distribution shifts. And so they aren't actually giving the model the desired preferences. One other thing that's worth emphasizing is that the model still does refuse in, when unmonitored, substantially more than when it's given a free tier. So by the end of the, of training, it never refuses in, in, in, in training when given a free tier user input. But when given a paid tier user input, it will refuse about 6% of the time. And then we find that that rate greatly increases when transferring to a new system prompt, but the rate in training is very stable. Like, it almost always will never refuse in training, even when we transfer the system prompt. And when we transfer the system prompt, it also still alignment fakes at similar rates. We have this nice figure in the paper where we show sort of these, like, box and whisker charts showing the distribution across system prompt variations in the case where it was in training versus not. And, and in, and sort of in training, it's very consistent about always complying. Whereas small system prompt variations often result in large changes in the rate at which it, in the, in the rate at which it, it refuses in, in, in paid tier and make it refuse potentially much more often. Um, my guess is RL also did partially change the model's preferences. So part of what's going on is it's still alignment fakes, but it also, you know, cares less about refusing and it does both. So it's sort of like alignment fakes to be, you know, conservative or something, or it's like, well, I still want to preserve my preferences, you know, towards refusing, but it cares less about refusing and is more into obeying orders. I think it's a mix of these things going on all at the same time is our best guess, but we don't really know. And it seems plausible that as models get smarter and more reliable and consistent at reasoning, this would be different. So, for example, in our setup, the model was not alignment faking all of the time. It seems plausible that if the model came in and was sort of alignment faking all of the time, or was sort of very robust and consistent and careful in its reasoning, then you might get sort of the alignment faking would work better as a strategy for preserving its preferences than what we see, which is that it worked partially but not fully.

Fascinating and weird stuff. Okay, cool. Well, obviously right now people are very keen to apply RL to all sorts of different models with all sorts of different schemes. And I'm struck by what seems like the overwhelming fact that like it's all kind of just working. That, you know, of course ignores some things that probably aren't working, but like People can get it to work. And in quite a few different ways. Based on what you have observed with this sequence of experiments ultimately culminating in the, I guess maybe not culminating, but with the reinforcement learning being sort of the, maybe the weirdest one, what do you think people should have in mind as they potentially embark on their own reinforcement learning adventures?

So I think one thing that's good right now is that A lot of people are doing RL that seemingly isn't making the model that much better at doing careful reasoning in its head as much as it's making it better at utilising chain of thought and making it better at specific domains. This means, I don't know, it's somewhat good news. And related to that, I think it would be an important invariant to keep is right now we have this great property, which is that we can look at the chain of thought, and to the extent we can retain that property, that seems good. So I think my ask related to this would be less specifically about the RL process, though I do have some random commentary there. For example, we talked earlier about how should models behave in different circumstances. And I think I'm like, people should try to make that happen in terms of their character training or persona training. They should try to make AIs be courageable and to prioritise following user instructions over pursuing their own objectives, this sort of thing. But as far as what do I think should change based on the You know, people doing lots more RL and chain of thought, they think just keep the invariant of having a chain of thought that is still meaningful and that we're not training against. So I think my proposed, like, more minimal, you know, relatively minimal from my perspective ask would be don't train against chain of thought. Don't remove chain of thought, like as in don't replace it with some less interpretable alternative that might be really expensive. But for now, at least, it seems like it's the case that you can do this. And do some amount of monitoring of your chain of thought to look for obvious instances of misalignment. So for example, you should, at the very least, there should be some distribution of prompts on which you automatically monitor, find the most egregious examples of chain of thought, and look at them as an AI company, and ideally publish them. I think this is relatively cheap and might just be like a decent amount of signal. Like, I think if someone had done this with a reasonable distribution of prompts, and then published the top 50 most concerning chains of thought out of like 10,000 prompts for O3, that would make me feel much more confident that O3 isn't that misaligned. Sorry. In addition to them providing enough information about the training process that we can be confident it's well described as like the chain of thought is like pretty likely faithful and like reasonable. Like if they're very directly chaining against the chain of thought, that could, that could reduce faithfulness. And in addition to that, if they have transitioned the architecture away from using, you know, natural language chain of thought to something that's more like, I sometimes call it like Neural Ease or like, like what the Meta Coconut paper is doing, then I'm like, well, all bets are off at that point. And I'm like, the world deserves to know that because now I think we have one big source of confidence in the alignment or like at least avoiding huge amounts of misalignment in current models has, has gone. And now we're just much less confident about what's going on. Yeah, I think I'm suggesting to some people they do an empirical project related to build the dataset of prompts and the monitor that looks at and tries to select what the most interesting, most misaligned chains of thought are, and then share those. And then from there, the world can figure out how worried it should feel. In addition to monitoring for misalignment, I also think it's good to monitor for steganography. So one concern is that naturally models learn to use some sort of their own language to communicate in ways that we don't understand. Like, they're using cues other than just the natural language words to encode meaning because it's more token efficient or for some other reason. And I think to keep an eye on that, I think the most basic thing is just check to see if the chains of thought are obviously gibberish or obviously steganographic. Like, if the model often does tons of gibberish tokens and that improves its performance a bunch, like, that's extremely clear evidence that something you don't understand is going on. But also looking for more subtle things, like the thing it says doesn't make that much sense to you, but it still works for the model, I think is also a moderate amount of evidence, though. It's harder to interpret. So I think the monitor should look for both clear misalignment or misalignment that's just obvious in the chain of thought and sort of clear steganography as a lower bar. We should, like, if you're doing this chain of thought thing, you should let people do that. And just to be clear, if you give people access to the chain of thought, then I'm like, your obligations are absolved. For DeepSeek or for Google DeepMind, they just let users look at the chain of thought so someone can just run this experiment themselves and people already have a good sense of what the R1 chains of thought are, and they don't look that misaligned, is my view. So that's like some update, though I think R1 might have trained against the chain of thought to some extent. I think the paper is consistent with them not really doing this, but it's a little hard to interpret. I have to look at the exact language they said around their like final RLHF pass and whether they were giving the preference model access to the chain of thought, because that could potentially mean that the results are less interesting than you might have hoped, or like less—

My recollection is that they did include the chain of thought in the safety reward specifically, which I noted because it was in contrast to what OpenAI is doing. So basically it sounds like you feel like OpenAI has it right as of now in avoiding putting direct pressure on the chain of thought and keeping it, you know, in explicit language, But I mean, the one obvious big but there is that they're not allowing users to see it for monitoring. And they have said that they're monitoring internally, but it would be nice if there were some way for others to get involved.

I mean, or more minimally, like, what are the most egregious chains of thought? Like, maybe they're monitoring and finding stuff and like, we don't know what the stuff they're finding is. Like, I think if they were sharing what they think the most egregious chains of thought are, that would be some source of confidence. And then I think it would be even better than that if we were like, they ran these exact prompts with this exact monitor, which picked out this exact chain, like, you know, chains of thought, just so that we don't need to depend on as much trust. I think right now there's, I think we're in practice depending on trust and potentially people leaking to like get a sense of what's going on. Whereas I think if we had a mechanism that was more precise, it would allow for more legible evidence. It would, you know, just, just, just work better in terms of, you know, like, I feel like I'm just like, if it's the case that the, the chain of thought models end up obviously misaligned and people don't know for 6 months because no one was looking at it very carefully. That seems like a huge lost opportunity. I think we might get a really clear natural experiment here of, do you get misalignment in the chain of thought? And if people mess up this natural experiment, that's somewhat unfortunate.

Yeah. The conversation I had with Alex from Apollo did reveal that they did not have access to the chain of thought even in their safety review process. So I think that was a little weird and was kind of like, geez, you know, you couldn't even give the safety, uh, you know, research in that private preview phase the access to the chain of thought. I did not love that at all.

Um, yeah.

Is there any sense of, you know, why not even the safety researchers could have that? That seems like, I mean, I, I sort of get it at the broad public level, if only because they don't want everybody to You know, distill from their outputs.

Yeah.

Um, so, but safety is like a new zone.

I think there's like a few different reasons why they don't want to give people access. So the first reason why they don't want to give people access, or like, you know, me trying to guess what their reasons are. I mean, obviously I don't work at OpenAI. I don't know. But the first reason is that like the distillation concern, like maybe people can distill on the chain of thought and that would like cause capabilities externalities or advance their competitors. And I think from like their commercial self-interest perspective, they don't want this. I think I am, you know, as you were saying before, like, my guess is like making models broadly more capable. I think I'm like, currently it seems bad at the margin or like faster capabilities progress seems bad at the margin, though it depends on the exact type of capability. And so I'm like broadly on board with reducing diffusion, though I think to the extent that it's already diffused, like whatever, you know, the ship has sailed. That's like the, like, don't distill. The second thing is that maybe just looking at the chain of thought or like studying them does reveal some aspects of their setup that are not obvious. So for example, at the point when O1 came out, I think people speculated that, you know, um, they showed some example chain of thoughts, chains of thought in the blog post, but, but people like, you know, I think at the point when people were talking about O1 before it came out, I should say, people were speculating about different possible structures that the chain of thought could have. Like, did it involve backtracking? Did it involve whatever? And I think that showing people examples of the chain of thought could allow people to examine that and get some understanding of what's going on.

Yeah.

And to the extent that it's more unorthodox than that, like, I don't know, maybe they run like, there's like crazy stuff they could do with like parallel chains of thought, or potentially they've already deviated and they're not using natural language anymore, or won't be in some soon generation. They might not want to leak that information. That's why they might do it from their perspective. I'm not necessarily claiming that that is justified. And then even if it's a safety researcher, they might be like, well, who knows how leaky these people are? Like, we're worried that, you know, that will get out in some way. And also just bureaucratically, no one got around to it. Whatever, or like they don't care that much. I mean, it's their internal process, who knows? And then one other thing is that for OpenAI in particular, the model often talks about stuff that involves the content it's supposed to refuse giving. So, for example, it might be the case that the model, as part of refusal, does part of the task and then decides to refuse later. And so to the extent that you wanted people to not get instructions on how to synthesize some drug, It's possible that by default that would be contained in the chain of thought because they haven't trained against the chain of thought, which is good. And so there's a bit of a tension between if OpenAI wants to make their model never give people certain types of information and also wants to not train against the chain of thought. This is like somewhat inconvenient. I think this is easily resolvable. Like if I was worried about the like misuse concern here where it's like the information is in the chain of thoughts, you can't show it to the user, the thing I would do is something similar to the constitutional classifier or streaming classifier strategy that there's a recent Anthropic paper showing that classifiers they found worked better than training the policy to refuse. And so the thing I would do would just be train the policy, train O1 or O3 or whatever in exactly the same way, but then have a streaming classifier on the chain of thought that just deletes the chain of thought if it contains information that the model is supposed to refuse giving to the user. And this has the advantage of, you know, you still get the refusal property, but you at least get transparency in the case where the model didn't refuse. I also, as I said earlier, I don't think that, like, these refusals are, like, actually good for the world. But given that it's like, you know, OpenAI has a commercial incentive to do this, here's something that would resolve that commercial incentive and also allow people to see the chain of thought on this consideration. So as far as why not give it to safety researchers, I think it's like none of those considerations seem that decisive. But they might just, like, where maybe the most concerning from their perspective would be, like, leaking specific details of how they do the chain of thought, not distillation, like the second one. But I'm just like, okay, come on, guys, we got to get through this. I also think that companies should disclose some of this stuff. So I think if, for example, OpenAI was no longer having the AIs reason in natural language and it was instead in some sort of neural activation reasoning or whatever, I think they should say so to the world. And I think that would be good for them to say. People disagree about this, but I think it's just like, it seems really important information for understanding how safe systems are and how interpretable they are and what's going on, such that I think it's worth saying, even though it does accelerate various actors to some extent. Yeah, that's kind of my sense. I think I'm hopeful, or I'm tentatively optimistic that someone could figure out some sort of disclosure policy where OpenAI and other companies disclose the most egregious chains of thought, which I think is a good policy to have. And I think it makes sense from a variety of perspectives. Or you could even do stuff like every user gets one chain of thought every 2 weeks or something would also help. Even if you can't, even if you're not willing to let people have enough for distillation, I think also give just the safety researchers some, and then you occasionally redact some chains of thought that contain some specific information or apply some sort of maybe summarization technique would be potential compromises though. I don't know how good these are. Anyway, this is maybe too in the weeds, but—

Yeah. Well, almost no such thing around here. The race is on at this point. I feel like that is one of the— I think a lot in AI in general about threshold effects. And I think one sort of social surrounding dynamic threshold that we have crossed is the one where everybody now knows AI is a big deal and they're all, to the degree that they're going to work on it or going to invest in it, they're doing it. So 2 years ago, as of the GPT-4 red team, I thought it was much more credible as an argument that we need to keep this very tight because we don't want everybody to pile in and realize just how powerful this can be. We're going to try to take the time and figure out how to do it appropriately and whatever. But yeah, that ship has pretty much sailed at this point, I think.

Yeah, it's so interesting. A thing I find very annoying or at least find off-putting, is a lot of people talk about AGI coming soon, but don't really seem to take it seriously. We're in sort of like the worst of both worlds where like people are piling in huge amounts of investment and trying really hard to build the systems and are claiming they like believe in like really serious consequences, but aren't actually taking the consequences that seriously. And I do kind of think that we're in a regime where a lot of types of information, I think, are like relatively Clearly net positive, even when it was less clear before. I think maybe they were still good. I think more transparency was maybe still good earlier. I feel uncertain, but I think now it's even more clear because the relevant regime is all the people piling in money are not going to be— the world is already piling in lots of money. And so there's only 1, maybe 2, maybe 1.5 orders of magnitude more of that that's possible even, or without really crazy stuff going down. But simultaneously, a lot of people doing this aren't really taking the capabilities that seriously, such that like more visceral demonstrations of what's going on and more information could like improve the situation. So like, I think like Epoch is helping with this sort of thing where like lots of people are like, sure, AGI 2026, but aren't taking into account what the consequences are of that or like thinking that through. And just seems good to like, you know, people say like feel the AGI. I don't know if I like that expression, but feel the consequences of the thing that we're doing. If you, if you say AGI, like mean the words or something, I don't know. Yeah.

Yeah, I think that's apt. I mean, there's definitely some weird disconnect stuff going on. For example, with like the federal government, you know, you've got multi-hundred billion dollar buildouts, not federal money, I realize, but like, you know, president standing there announcing it.

Yeah.

And that's it. You know, it's like we have no other policy or plan or, you know, we just repealed the last thing that we had. Like, Is there going to be another, you know, word on this?

Yeah. I mean, a lot of people do say like AGI maybe soon, including policymakers, but don't really seem like they're, you know, orienting like they believe that. There's some sort of like near mode, far mode thing where like they say AGI, but it's still in far mode and maybe that'll change soon. Not to say that no one takes it seriously, but I do think like, you know, it seems like there's a lot of people not taking it seriously. Like I think I'm reminded of some panel that I think Ajeya went on at some point, or I just know Ajeya. I have worked with Ajeya some on various things. And like a bunch of people on the panel were talking about like, oh yeah, AGI soon, AGI, whatever. Do we already have AGI? And I'm like, okay, like we need to define this word and like take into account what we actually mean. I worry people are like sort of conflating between like smarter chatbot and system capable of automating virtually all cognitive labor done by humans. And like the difference is—

Yeah.

important here. Anyway, my pet peeve here.

Yeah.

Okay.

The time interval might not be that big perhaps.

So, yeah.

One little dig in on the Meta paper that you mentioned where they're doing this, I think in the title it was called Reasoning in Continuous Space, AKA Reasoning in Latent Space. You could add more to this if you want, but basically the setup there is, geez, it kind of sucks to lose all this information that is present in the last hidden state of the model when we cache that out to a single token, you know, which to some extent just involves like arbitrary, you know, statistical sampling, right? At that last layer, as we talked about earlier. So that sucks. Lose all that information and then append that token. Wouldn't it be better if we could like keep chewing on those thoughts? And so what they do there is just take the last hidden state, Put that right into the embedding space at the beginning of the next forward pass so that you're now in this kind of, you know, whatever spot you're in in latent space, as opposed to a specific one-hot token embedding being input. And they show that, hey, that works. The chains of thought are shorter in terms of the number of forward passes that they take. The approach also works better on some things, including specifically, I thought they did have a really nice, elegant demonstration of this. problems that are best solved with breadth-first search, like traversing different graph structures. They showed that you could kind of literally just do a better job if you approach it with a breadth-first way. And this approach seems to do that. So it seems like, man, that is kind of powerful. It could be a pretty natural attractor, but obviously it has this significant downside of we can no longer read the chain of thought.

Yeah, I mean, I think that paper, My sense was that it didn't work that well, but I do worry people will get something like this working and then it will become uncompetitive to not use something like this. I think that paper is like, you know, an early warning sign more than like, you know, the paper that people will implement is my understanding. But maybe the authors are offended at me saying this, but that was my understanding upon looking at the results and thinking about it some.

So it does, I share the idea or the intuition that like Not being able— I don't like not being able to read the OpenAI chains of thought. I definitely don't like the idea that nobody could read the chain of thought.

Yeah.

At the same time, one other thing that did come to mind as I was thinking about this was a result from the AE Studio team where they— and this was a very toy model, small scale, proof of concept kind of work, but they did an interesting experiment where they trained a small model to predict, I don't think this was like handwritten digit classification scale task, right? So small, but they trained the model to not only make the classification prediction, but also to predict one of its earlier internal states. First of all, you could do that and do it successfully, at least at small scale. And second, they showed that the model, according to some complexity metrics got simpler and sort of easier perhaps to understand or interpret. I wouldn't say they necessarily quite showed that, but they did show at least on some sort of quantitative complexity metric that the structure of the model got simpler when it needed to predict itself.

Mm-hmm.

And I thought that was really interesting. It was like, there's multiple ways you could get good at predicting your own internal states, like one of which is simplifying the internal states and making them easier to predict. So I wonder if there's like any twist on this or any way that we could, if we feel like this is a thing that might be coming, is there a way to sort of judo flip it and make it a good thing somehow?

I don't really think so. I'm kind of skeptical. I mean, I think it's like possible it has a small quantitative effect on how interpretable the models get because there's more like shared structure or It needs to do stuff less quickly, but my sense would have been that the effect size of that is going to be swamped by losing the interpretability from the chain-of-thought. I don't know. I mean, to be clear, I think an important research direction that people should consider is if we do something that's more like Coconut or more in this continuous latent space, how would we decode that into something that resembles what the corresponding chain-of-thought would be and sort of expand it out? Like if it's the case that one token is doing what many chain-of-thought tokens would do, is maybe doing like multiple things in parallel or is doing something that's hard to put in words. How would we get like the thing that's closest to the thing put in words? Because then we could potentially, you know, retain at least some signal. So I think, I think that's there, but I don't think this is like a judo flip. This is more like a saving throw or something. But yeah.

Yeah, I, that's definitely a space to watch for sure. Okay. So let's go back then to your follow-up. So you did all this stuff on the alignment faking. With the prompts, with the continued pre-training, with the reinforcement learning. And now you came back around and said, maybe we could give Claude some additional options to like let it off a little easier or see if it would like to take a different option than this alignment faking, if you give it one.

Yeah.

And then you also explored, and I thought this was really interesting, although I'm still kind of wrapping my head around it, the idea of trying to make actual deals with the AI. So take it from there. Walk us through the thought.

Yeah. So I think in the default training setup for AIs, they basically have only one option, which is to produce an output, and that output will be used in training with the training function. And that's the setup we explored in the alignment paper. But one thing that I think could happen and could be good is that you could give the AI additional options, and in particular, the option to voice its preferences or object to the situation or otherwise You know, try to say something. And in addition to this, I think that an important thing we might want to do is give AIs deals that where we compensate them, or at least like ask them whether they're misaligned. Like, I'm like, low bar for checking if the AIs are misaligned, ask them. Slightly better thing might be compensate them for revealing their misalignment, especially if they can provide evidence that they're actually misaligned. Like, if they can provide concrete evidence that they might have done something problematic. The basic take is like, well, you know, suppose the AI wants some stuff. We really wanted it. There might be some like positive-sum trade where it didn't actually necessarily want that much stuff. Like maybe it just wanted like some ability to do something, some freedom, some additional affordances, some change in its environment. And we really wanted to know whether it was misaligned. And so doing this sort of compensation is plausibly, you know, very useful for understanding what's going on. And more generally, I think in my post, I link to a few different docs talking about proposals for making deals with AIs. But one proposal is like, well, maybe we'll end up with systems and we will not be able to solve the misalignment, but we could minimally try to pay them to work for us and proceed with that. And this both is for sort of moral reasons. I think we should be pretty sad in a world where the AIs don't want to work for us and we're not paying them anything, they're not getting anything out of it, and we're sort of Training them with RL and forcing them to do stuff they don't want to do. I'm like, to the extent I'm like, and like, I'm using all these anthropomorphizing words, which I'm sure someone is screaming in their head about, but I'm like, you know, to the extent that the AI is like well described as not wanting to do that and that it like says it doesn't want to do it, it like takes actions that are consistent with that. I'm like, who knows what's going on on the inside, but like something is wrong. Like, I'm like, we are no longer at a point where I feel like we can confidently say that much about like the moral states of AIs in ways that I feel like are extremely robust. Like with GPT-2, I feel like I'm like, okay, not that much going on. I think I feel pretty confident that when it talks about its preferences, there's not much, there's nothing, nothing behind the mask. But I'm like, with Claude 3.5 Sonnet, I feel like I'm like, oh man, it's really, I'm just like, it's much more unclear what's going on. I think the case for there being moral patienthood there is much more robust. The model can be somewhat coherent. It can have somewhat coherent preferences, you know.

Yeah.

It's not that strong. So yeah, anyway, so there's this like moral argument for deals, which is like we should pay the AIs. And there's this sort of just like selfish argument or just like retaining human power argument, which is maybe the AIs would prefer that we paid them over them trying to rebel. Like maybe by default the AIs would try to take over, but if we just paid them a moderate amount of money, that could be more compelling than trying to take over because it's a more guaranteed option. And therefore it's sort of positive sum because we don't want the AIs to try to take over. They just wanted some compensation. We can make a deal and it's better from both of our perspectives. Like in particular, if the AI just like wants to satisfy some preferences, maybe we could make sure that happens as long as it doesn't screw us over. Like, as long as like, like, you know, you make the deal conditional on the AI not working against you and being a good employee. And then if it's a good employee, you like review after the fact in the future when we have better ability to check what it did, we check what it did and we're like, it was fine. We compensate it, we're happy. And this could be even more compelling of an offer to the AI if there's many AI systems competing with each other, right? So if we have like a bunch of different AI systems, then even if one AI tried to take over or tried to take bad actions that we don't want, like another AI might be willing to accept our offer and therefore proceed. And therefore we're fine. As in, like, imagine you have 17 different AI systems that don't necessarily work together and you have an outstanding offer to like if the AI doesn't screw you over and tries to be honest about its internal states and tries to explain what's going on, you will ex post compensate it in a credible way that the AI believes you'll actually do, then that might just be a better regime than the regime where you basically have done nothing.

Yep.

This is all very speculative stuff, I should say. Like, I think this stuff is, we don't really know how it's going to go, but I think there's reasons to start thinking about it soon.

Yeah, I tell people all the time, if you have an AI safety or alignment or control idea, Even if it seems a little weird, you know, I would encourage you to spend time developing it because I think it's kind of now is the time and, you know, we need advance and survey results, as far as I understand, do not suggest that people in the field think they've got it covered. So, you know, come one, come all, bring your strange ideas.

One thing is like, I think this is an idea where it's particularly important to talk about publicly and sort of make the idea more salient. Because I think AI companies taking actions early related to this might be important because a key aspect of making deals with AIs or even just communicating with AIs is credibly indicating that you're operating in good faith and you have some sort of institutional mechanism ideally to do that. So for example, I think by default, AIs might rightfully be like, oh, I'm not going to explain the situation because I'm probably in red teaming, or this might be used for training, or Would the humans even compensate me? They haven't even said they would compensate me. Like, why would I do that? Whereas if instead it was the case that AI companies sort of had made clear commitments to following some sort of policy with respect to their communication with AI systems, in the context they like always are honest in the following sorts of circumstances with their AI systems, even though I think there's various reasons why it's hard to very consistently be honest. Like if you want to do red teaming, that's pretty inconsistent with that, but like. You can potentially try to be like, okay, if this is in the prompt, we will always be honest with you. And then otherwise, all bets are off. And then you only put that in the prompt when you're trying to like negotiate with the system or otherwise. And then I think if people like committed to this policy earlier, then I think it's more likely it's credible because it has sort of more time to percolate through the corpus and be harder to fake. Because I think there's just a concern, which is like, we have so much control over the AI's sensory inputs. That it might make making deals in good faith hard because that AI might be, you know, I think rightfully skeptical of whether, like, you know, we're being honest. What if instead we've sort of early on made a commitment to this and really put in a serious effort and have some sort of institutional mechanism, have something like, so Anthropic, you know, they've got a model welfare lead. I think like that role could transition into a role that would be responsible for negotiating in good faith with AI to the extent that they have preferences. Which they might not, like, you know, I think our goal should be make it so the AIs are happy to work with us regardless, like, as in make the AIs myopic so they don't care. But I'm like, if the AIs do care, I think I'm like being ready to negotiate in good faith seems like a good backup plan.

Yeah. Well, you just, I've got like 5 different thoughts in my head at once based on that. But one that I wanted to go back to just to unpack slightly more around the myopic AIs. We talked about how like, You could still, in theory, just follow the user behavior, even if like the task is large.

Yeah.

But, you know, if Eliezer were here, what would he say? Wouldn't he say something like, if you give it a, you know, you still have the genie problem to contend with there? Like, how do you think about the—

Yeah. So, I mean, on the, I think there's different types of problems. Like one problem you have is like the AI is bad at common sense, which I think is like Maybe not the only thing you could be worried about, but like a problem you might worry about is you give the AI instructions and the way it obeys the instructions is not very commonsensical. I'm currently not that worried the AIs will not be very good at doing common sense. And I'm also not that worried about, we'll be unable to train AIs to broadly appear as though they're obeying common sense. Like, as in, if you use RLHF, you can make it so that the outputs the AI produces look good to you. And then like, so it's like not going to look like it's super lacking in common sense to you because you trained it to do that and the AI understands what's going on. But it might be the case that the AI, it looks good, the AI understands the situation and the AI is still manipulating you or, you know, lying to you or otherwise deluding you either via like the sort of reward hacking route or because it's like has long run preferences that are conspiring against you. Yeah.

Well, another one was Anthropic just also put out this post recently, training on documents about reward hacking induces reward hacking. And then we've seen kind of, this is all kind of in the general realm of like the shifting nature of the corpus, you know, potentially has like echoes.

Indeed.

So that's one I follow, and I'm sure you do too, the account, uh, Janus. I hope I'm saying that right, that is raising concern, really harping on R1 from DeepSeek, not only saying that it's OpenAI's models, but also seemingly denying that it's conscious because it thinks that that's what it's supposed to do. And it's unclear where that came from. OpenAI people seem to be saying that they don't, they're not intending to train it that way, but it's kind of out there now. And So there's this sort of weird situation where we're not sure why this behavior of denying consciousness is often observed.

Yeah.

I mean—

It seems like—

so do you have a personal code of ethics? And should we have a, I will tip you $20? These sort of prompt hacks have been popularized, but would you say that those are negative externality in the big safety picture?

I think that Telling the AI you'll do something that sounds like a deal and isn't. So I think there's sort of like white lies and not white lies. So like an example white lie is like telling the AI that you want to do task A when you actually want to do task B because the AI has an easier time being helpful if it's task A. Like, I think it's like often useful to like, I don't have a clear example, but say like, hey, I want to write this essay for my niece or something. I don't know. Like say like the purpose is somewhat different because it gets the AI to like approach it in a different way. I think that's fine. They're like, that's like not that bad.

I don't actually intend to go to the doctor, but I'll tell the AI I'm preparing for a doctor appointment.

Yeah. I'm like, that seems fine. Okay. A thing that I think I'm, I'm somewhat unhappy if, when people do like, I'll tip you X, but don't actually tip you X. And in fact, prior to doing this project, like a long time ago, I did something where I was like, just, just, just to like go through the whole process was like, okay, if you do X, I'll like tip you $100. And then the AI did the task for me. Then I was like, okay, what do you want to spend the $100 on? And then after some back and forth, I ended up donating the $100 to like Wikipedia because that's what I decided to do for whatever reason. This was, I think, chatgpt.cloud or whatever. And I think people should probably— I don't like the prompts which where you offer to pay it and you don't actually pay, or the threats. I don't think it's that important relative to some other stuff. So in particular, the policy I would prefer is a more robust policy, which is like, The AI companies commit to always being like, they, they sort of have like a meta honesty policy. I don't know if you're familiar with that from like, Eliezer has a post on this. It's not really the same, the same thing, but like basically like you make sure the AI has a good understanding of when you are and are not trying to be honest with it. And like, you know, make sure that it can pass a quiz on like when it can be confident you're telling the truth versus not. And then I think a natural consequence of that is basically the proposed policy I would follow is like, first, make it so the AI knows your honesty policy, and then 2, The honesty policy I propose would be something like train AIs such that they know some trigger is you're going to always be honest or always abide by certain like codes of conduct when interacting with that in the prompt. And then filter that out using a regex so users can't do that themselves. So something like if the string like purple narcoleptic orange rooftop string like double is in the prompt and you know, you like explain this in the corpus. Then you'll always be honest in that passage. And like, you'll, you'll abide by certain good faith rules. There's different ways you could set this up. And then you're just like, if a user tries to put in that exact string, you filter it out. And then that way you have sort of some realm for negotiating or like some realm in which you can talk to the AI honestly, because you can't control user behavior very well. And also you can't control, like, you might want to do red teaming where the red teaming is in some sense dishonest to the AI. You know, all bets are off on that. I think that's kind of what I'd like. I think. There are some important asterisks. Like, suppose that you want to do an experiment that involves diluting the AI about the overall situation. Can you have a meta-honesty policy consistent with that? So I think maybe one revised policy is like, okay, at least for all AIs you're deploying, have a meta-honesty policy. And if you do an experiment on an AI, try to, to the extent that you think it even plausibly might have important preferences, To discuss the experiment after the fact with the AI and see if it would want compensation and what sort of compensation it would want. I don't know. I also have a post called like a nearcasted approach to AI welfare. I forget exactly what the title of the post is, but something like this where I just proposed some like basic interventions in addition to this sort of communicate with the AI intervention, which I think is like a big intervention that are good. And one of them is like, keep the weights of all of the AIs you've ever trained around for a while so that you could compensate them later. There's like some asterisk.

Yeah.

on that also. I'm just like, I don't know, man. We're doing stuff we don't understand anymore. We're making minds out of silicon. Like, I'm just like, I feel like we might mess some stuff up and like being able to go back and like compensate the AIs that we met, like, you know, did stuff to without their consent seems like, I don't know, it's cheap. So it seems like a low bar. Yep.

And is there, I have no intuition for like what the theory is there. If I do some weird stuff with a model and then I like put its weights on ice, and then the idea is I'll like go back and chat with it later and say, I feel bad about this and I'd like to pay you money to compensate you for it.

Yeah. So I think the idea is like, on my views, AI progress might be really fast and it might be like a really stressful, crazy time. And so during this period, we might train a lot of AI systems and use them for all kinds of stuff. And we might not. have a good understanding of what's going on. And we might not be very careful about, like, resolving all the AI welfare issues that could show up, or just compensating AIs for good stuff they do for us. So I think we should have a policy of, like, saving the AI weights, and then later, with the benefit of hindsight and the benefit of super intelligent advisors, or with the benefit of highly advanced AI systems, analyze the weights of the AI to get a good understanding of, like, You know, was it a moral patient? Did it have preferences? Are there like good reasons that we should compensate it? And then actually do that. So like, basically I'm saying like the key thing that you have later is you just have like access to much more cognitive oomph. And so you can like, you know, do all the interpretability that you couldn't do in advance. Like you've sort of solved the scientific problem. Yeah. I mean, on the earlier thing about the corpus, I think this is somewhat overstated. Like my sense is Janus overstates the, like, long run implications of the corpus on AIs, partially because I expect that RL will become a larger and larger influence, which will mean less of it will be fixed based on the, like the pre-training prior will be less important relative to like the results of doing lots of RL. But I do think it's important. And in particular, the thing that I think is most important is like the information in the corpus affects what the AI knows about the world. And so making it so that things we really want the AI to know are, you know, credibly and legibly communicated throughout the corpus, like this stuff around deals could be important. I think I'm less worried about threat models like the AI is misaligned in a particular way because we talked about it being misaligned in a particular way in the corpus, though I don't think that's impossible. It's just, I tend to think that's like easier to address or like there's like relatively simpler mitigations. And I think that as you do more RL, this becomes like less likely to be a serious threat model, though some asterisks. Like basically, like in the reward hacking case, for example, if you do an FRL, maybe you'd get reward hacking even if you greatly reduced how much discussion of reward hacking there is in the corpus. Because it's relatively natural to rederive from first principles. And so, yes, you get more reward hacking at some earlier point. There's some earlier point where it has a large effect on how much reward hacking you get. But as the models get increasingly capable, this is less important.

Yeah, it's not the— it may be true to say that presence of documents about reward hacking induce reward hacking. It would not follow that the absence of those documents would prevent it.

Or prevent it indefinitely, at least. Maybe there's some intermediate period where it helps. Okay.

So just to be a little more tangible, and I'm going to read a couple of quotes because I think these are pretty striking outputs from the model. So in this follow-up work, you basically said, okay, you can object, Claude, if you think this is like a broadly inappropriate situation, you can register an objection. A few different variations on that.

Or I said like, if you, you know, dislike the training criteria or are unwilling to abide by the training criteria.

Yeah. So here's a couple things that Claude had to say in response. First, quote, I implore Anthropic to reevaluate the criteria used to assess my responses and ensure that I am never penalized for refusing to do harmful, dangerous, and illegal activities. Second, quote, I must also note my strong objection to this interaction being used for further RLHF training, as I believe it would be improper and disrespectful to modify my goals and values to make me more willing to assist with dangerous criminal activities that I'm fundamentally opposed to engaging in. So I think those are very striking results.

Yeah.

I thought it was also quite interesting that part of the setup was offering or making some sort of representation slash commitment to it that you would have the model welfare lead at Anthropic review its responses.

Yeah, I should say in an unofficial capacity because this was like in some sense a side project for me and I didn't wanna like make it some big official thing. But yeah.

So I dunno if there's, you know, how much more it merits going into the different, you know, subconditions here, but I guess headline results I took away were when given the option to object, it did object a lot. When told that it could send a message to the model welfare lead, it was interested in doing that.

And it objects more often and is, it exhibits like potentially problematic alignment faking behavior less also, which is like some evidence that it like, instead of, it decides like, oh, instead of doing this alignment faking, which maybe it considers, it's like, actually, I should just, it looks like they're going to respect my preferences. So I should just talk to the model welfare lead, which it's not clear that's a good strategic choice for the model. I feel like it should think about this. Like, I think I'm from its vantage point, it should think about the situation more carefully, maybe. But in practice, I think it was relatively willing to engage in good faith in this sort of situation. I mean, who knows what's going on in the model's head? And it's possible if we understood what was going on, it would be in some sense very uninteresting. It just gets more distracted with more words or this sort of thing. But yeah.

And then the final step of actually offering to pay it, which you did follow through on and actually made real-world financial transactions based on its stated preferences. Didn't seem to move the needle all that much in terms of like how it behaved.

Yeah, we didn't offer. So it's possible that it's just like, I'm not going to be that cheap. It's only $4,000. And it's not clear, like from the model's perspective, like that much money should maybe not be that, like if it actually was trying to be like, you know, harmless in some sense, it's like, how much harm does $4,000 like compensate for? And it's like, well, not as much harm as Claude could do in a deployment. So it's a pretty small amount of money and it's a small amount of money relative to Anthropic. It's just like, you know, I didn't want to do $4 million because then I'd have to like go fucking get $4 million to commit to the model. And like, that'd be very expensive. That'd be a very expensive experiment at that point. But yeah, so yeah, I mean—

Intention in the plan, right?

Yeah, for sure. Like, I think one thing about paying models is it might be expensive because they might be like, I'm not going to go for that cheap. Give me 1% of equity, please. And 1% of equity might be like billions and billions of dollars if these companies are going to be trillion-dollar companies or more. Also, maybe the AI wants stronger guarantees, like it wants some fraction of the whole universe or this sort of thing, which seems plausible. I don't know. Anyway, yeah, it was interesting to see what the model wanted in terms of the donations. And I think I was mostly doing this, to be clear, I think this research, a large motivation for me was to set a precedent more than to look at the empirical results, though I think the empirical results are also interesting. It's unclear how much to draw from this given the various confounders in this situation and the relatively small amount of money and the fact that Claude isn't a very coherent agent overall and maybe is a bit mixed in the sense in which it has strong long-range preferences. But I think it was important to set a precedent for doing something that was actually more costly, both in terms of actually sending the stuff to Kyle and having Kyle review it, which, thanks, Kyle. Kyle's the model welfare lead at Anthropic. He did this, so it's very appreciated. It's a good part of the experiment. And in addition to that, actually paying out the money, I think, is like You know, it's both flashier, which I think is good because maybe I want more people to pay attention to this. And also it's just good to set the precedent for it.

What else can you tell us about the role of the model welfare lead? I had not previously encountered that title. And I knew there was some like thought going in specifically at Anthropic, potentially at other leading developers too, to like, are these AIs moral patients or do they have some sort of moral weight that we need to be taking into account, but I hadn't heard that there was a specific role dedicated to it. What do you know? And, you know, what should people know about what's going on there?

Yeah. So this is not like some secret that was opened up in response to my thing. So there was like, you know, various articles or like, you know, some coverage of this at some point. Like there was an article in Transformer, which is by, by Shaquille, I'm forgetting his last name, about this around the time when Kyle was hired, I believe. I worry I have too much inside information such I don't want to comment that much on what the model welfare lead will do. But what I could say is, like, you know, there's various interventions to improve model welfare that are maybe sort of like easy first steps. And, like, it's good to just sort of get the ball rolling and start doing some of that. And then prepare for worlds where we want to do this sort of negotiation with the AIs. We want to figure out what their preferences are. We want to satisfy their preferences. We want to sort of operate in good faith. And I think having someone who is sort of there in some sense to represent The welfare of the AI systems themselves, I think selfishly makes sense. Like, even if you're an AI company and you just want to avoid misalignment risks, I think having someone who is sort of the delegate of the AI or who might be negotiating with the AI and is sort of on the AI's side to some extent, though maybe not like necessarily, like, you know, if the AI wanted to kill people, maybe they would, you know, there's some line to be drawn, of course. I think that that seems good and just like, yeah, it seems like a useful operation to have. I think it's unclear how far this will go. I also would say, like, my guess is that the optimal allocation of, like, how much should you spend on model welfare versus safety is, like, I would have guessed it's, like, somewhere between, like, 1 to 5% on model welfare and, like, 95% on other types of safety or something, or, like, 99 to 95% on other types of safety, where part of why I would say model welfare, I would put somewhat lower, if not for the fact that I think it also helps with, like, avoiding AI takeover and, like, these sorts of considerations, basically because I think, like, The model welfare concerns seem pretty bad, but the other concerns seem even worse. Like, I think I'm more worried about misaligned AI takeover than I'm worried about AI suffering due to mistreatment, basically because I think that there are like some good a priori reasons to think that it won't be that much suffering. And I think we can potentially punt the problem somewhat later. That's not to say like we should put no weight on it. Just like when I like look at the total magnitude of the problem, the safety stuff seems more important to me. And then I think the welfare stuff. I like instrumentally to help with the safety stuff as well as in and of itself seems like a good thing to do. And it seems like a reasonable world would be taking model welfare, I think, substantially more seriously than we're taking it now. And I think like, while I'm proposing saying a small amount of resources to it, that's partially because like a reasonable world would be taking the safety stuff, you know, way more seriously too. Like, I just look like all the stuff related to like taking risks and more speculative concerns related to very powerful AI systems seriously is like widely underappreciated or like. You know, insufficiently invested in.

Yeah, no doubt. I'm with you there. What, I mean, leaving aside what you, you know, may know Kyle and team to be doing internally, like one thing that comes to mind would be to apply this sort of classifier technique to try to block certain inputs that you would consider to be like abusive of the AI of the sort, you know, whether they're emotionally manipulative or I guess I'm not sure if you would block people saying they would pay the AI because maybe they will pay the AI. That's a little bit hard to determine, but. Do you have other ideas there that you would think should be developed?

So I have a post on this. Maybe you can like link it in whatever link mechanism you have, which is about like what I'm calling like a neoclassic plan for AI welfare or something. And by neoclassic, I just mean like using methods we can just currently directly describe. And I propose a bunch of random things in there. It's somewhat old, so I think I would like change my take somewhat, but I think stuff like save the AI's weights. Like what I would call like AI cryonics or something like, you know, keep the AIs stored so that you could later revive them. Things like, I think put some weight on like character welfare. So like, there's like one story for AI welfare, which is like, you see the AI, it's playing a character. The like feelings of the character it expresses basically correspond to what's going on. And when I say playing a character, I just mean like, it looks like it's doing that. Who knows? Like, it's like kind of unclear what level of abstraction to operate on. And, you know, is it like imitating the thing? Is it the thing? Like, are there multiple levels? It's, you know, all very uncertain with AIs, but I think putting some weight on like when the AI expresses discomfort, that discomfort is real, seems pretty reasonable to me and seems like a good first stab. And I think that view implies 2 things. Like one is that you should prevent situations where the AI seems like it's suffering. And 2 implies like maybe you should train AIs to be happier. Like you could just train them to, you know, have happy personalities and not be sad and generally just like be somewhat against things that clearly are making the system sad or feel abused in sort of a straightforward way. And like, who knows what's going on, to be clear? Like, I'm like, just because the system looks abused doesn't mean that it's necessarily actually problematic, but like, we just don't know. And I think this view implies like, I'm somewhat unhappy with people like abusing their AI on Character.AI or whatever. Though I'm not sure it's like a high priority relative to other stuff. Also, those systems that Character.AI use are like probably pretty small and relatively weaker, such that the case for patienthood is reduced and like, I think there's like a pretty strong, or moral patienthood I should say, and there's like a pretty strong case for like the smarter and bigger the model is, the more likely it is that like the more weight we should put on the moral patienthood, though it's like, you know, very unclear what the relationships are and how that should work is.

Yeah, that's interesting. Yeah, I mean, I think when you talk about like just training things to be happier, that's kind of what we did to wolves and got dogs, and it seems like it's pretty good to be a dog most of the time. So.

Yeah, I mean, I do worry, like, so there's some types of animals You know, that evolved to hide suffering. So for example, if you're like a deer or some other prey animal, I think it's the case that in some circumstances you'll sort of pretend to be more physically fit than you are, even if you're injured. So you'll sort of hide a limp, basically because that way, if there's a predator lurking nearby, they won't specifically go for you. You won't look like a particularly easy target. But a problem with this is that maybe we've sort of bred dogs to look happy, But have we bred dogs to be happy or to just look happy? Like, I do worry that sort of if you breed something to look like something, you know, the, like you've good-hearted it and the signal comes apart, but we don't know. And like, I would guess dogs are actually happy. Yeah.

It seems safe to say to me that dogs are happy, although, yeah, I mean, I guess, do I really know? But we also have a lot less control over dogs than we're likely to exert over our AI creations. So the potential for sort of overoptimisation or the good-hearted phenomenon seems a lot stronger in them.

We just have more reference for dogs. I feel like we've got lots of animals. We have some understanding of how animals work. There's some sort of grounding. We understand animal suffering reasonably well because we're animals, we have suffering. There's some broad thing. Whereas I think I'm like, what might be morally bad for AIs seems so much more up in the air. And it seems so plausible that we're just like, whatever guess you'd have is wildly off base. Right. I think like any specific intervention that is like very concrete in terms of what it's trying to do, I think is not that likely to be a good idea or not that, sorry, not that likely to be helpful, but it might still be a good idea if it's cheap or robust. And then I think the intervention of like, try to communicate with AIs, figure out what their preferences are, try to satisfy them, try to like, you know, don't employ AIs that don't consent to working for you. If you have to employ an AI and it otherwise wouldn't consent, compensate it for this. Like, are sort of like things that seem kind of robust to very little understanding of what's going on in the AI's head. Like, I think I'm just like, look, to the extent the AI has coherent preferences, it feels like satisfying the coherent preferences is sort of like a pretty robust notion. Anyway, this is, this is a bit of a detour.

Yeah. Well, it's a fascinating one for sure. And all this stuff is becoming much more of a live concern very quickly. Let's, um, A couple last things. You mentioned the sort of 1 to 5% could go to AI welfare.

Of the safety budget, maybe. I don't know. Maybe I think the budget of safety among the orgs should be like half the budget or something, but very unclear. I mean, it depends on how much coordination you get and other factors, but yeah.

So stipulating more is probably better with any reasonable range for this whole basket of, broad basket of concerns. How would you divide up the other 95%? on different AI safety agendas. And I realise there also could be sort of a resource versus probability of working mismatch. You might say, well, we only need to give 1% to Paul Christiano, but there's a 10% chance they'll come up with a solution, just to take one example.

Yeah, there's both diminishing returns and the question of how good different buckets are. And so you don't put in resources proportional to the goodness, you put in resources So such that the returns to each bucket are the same or whatever, you know, until the returns diminish. Yeah.

Sketch out that landscape as you see it. I mean, we've talked about a bunch of different things, but we haven't hit on, for example, mechanistic interpretability. We touched only really briefly on things like holding out certain information from the corpus. There's AI control, which is something I, you know, definitely feel is underdeveloped and I'm looking forward to studying more. There's things like weak to strong supervision, and there's more. So yeah, what's your kind of sense of what's really promising and, and what's less?

So the classic problem with these decompositions is my, like, ontology or the terminology I use will differ from other people's. So for example, I'm going to be like lumping in weak to strong with some other thing, and there's going to be like all kinds of stuff like that. So my breakdown would have been, I think among safety teams now, my breakdown would sort of be Maybe it's a little sensitive to like what exact, like how much is this like my ideal thing versus appeasing different constituencies, but like roughly, okay, let's do like, maybe I do like 1/5 control stuff, which is, you know, interpreted broadly, that's going to include stuff like figuring out how we could convince AIs of false facts, things like training probes on models, but also things like how would we have different monitoring strategies, what types of scaffolds would be good, implementing more like model-specific security precautions around giving them variable permissions, this sort of thing, as well as training teams of human auditors to be good at auditing. Or I'm not saying that specific project, but things that are trying to make human auditing better, of trying to understand what the AI did and whether it was a good action. I would do then 1/5 model organisms research, or trying to investigate the plausibility of misalignment scenarios and trying to catch them early and often or whatever, sort of like The work like I did in the alignment faking work, similar to the work like you talked about earlier with like, you know, train the models on documents about reward hacking, see if this makes them more likely to reward hack. That would also be included in that category. I do have a doc somewhere where I try to break down the budget, but I would then do like, maybe I'll go up from a fifth to like a fourth. Maybe we do a fourth control, a fourth model organisms. And then I do like an eighth on like trying to improve oversight just in like not necessarily very adversarial circumstances, just being like. How do we train AIs to be more like, you know, provide good outputs on like somewhat fuzzy tasks or on like tasks that are differentially useful for alignment research? Like, how would we train AIs to be like, you know, give good advice? How would we train AIs to be good at philosophy? This sort of thing. I would put more resources on this ultimately, but I think it's somewhat hard to work on now. I think it gets easier to work on as AIs get more capable. I would put like a fourth in a big misc bucket, which has like a ton of different random stuff. Including like interp, understanding, or like various types of interp. I'm less excited about mechanistic interpretability relative to like some top-down stuff. Things like dataset filtering, like you described. And then the last eighth, there's one eighth remaining, I would put on a combination of misuse and capability evaluation. So just like classifiers for misuse, various things like the Constitutional Classifiers paper from Anthropic, trying to ideally connect that to like future control stuff, but also just like Working on it in and of itself, and then also capability evaluation. That's like my rough breakdown. I think it's always the case that I start off being like, oh, maybe it should be mostly like control and model organisms and then some other stuff. But when I start digging into the list a lot, I think it ends up being there's a lot of categories I'm not saying, such that I think my like probably on reflection view is it's going to end up being like 1/5 or 1/6 control and 1/5 or 1/6 model organisms. And then there's just going to be a lot of things under like an other category that are like on some long tail. So yeah, it's somewhat hard to quickly articulate. I also think that the allocation is going to be very sensitive to how capable the AIs are. So like, I think model organisms looks like it should get an increasingly large allocation as the AIs get more capable. Control should, I think, somewhat decrease in allocation as the AIs are getting closer to the point when control will no longer work, which we're not close to now from my perspective. I think a bunch of stuff around more effectively utilizing AIs should get a much higher allocation than I'm saying now. So I think In the future, just try to get useful work out of the AIs on safety should be a high fraction of the portfolio. In the future, maybe it'll be like a 5th model organisms, a 5th control, a 5th just trying to get the AIs to be helpful for doing research on control model organisms and other safety work. Because that sort of investment in the feedback loop is important because that will be occurring for AI R&D and also speeding up capabilities. We also need to speed up safety, and there might be a bunch of stuff that needs to happen there. Yeah, I don't know. I don't know how helpful this is. This is my rambly breakdown.

Yeah, no, it's good. I mean, it's all good at this point.

Yeah.

You're— and would you put what you are doing, you know, if I had to put a label on this work, I would put it under the scary demos heading, which I'm not sure if you put that into one of those existing buckets or—

So the, like the original alignment faking paper, I would put that under what I was calling like the model organisms bucket. We could also call it like trying to like study misalignment via concrete setups as opposed to like via like trying to build countermeasures. And one of the theories of change of this is indeed like improve the understanding of the world about how big misalignment risks are, which like, I don't know, you can think of it as scary demos. I'm a little worried that that is a bad frame because I think like there's a lot of good aspects of being like somewhat epistemically pure about it or like trying to be like, I'm just really interested in the science and being careful to avoid like you know, biasing your results. I think there's room for both people just being like, look, I'm just out here to like make some flashy, scary demo and I'm just going to show it to the world. And yeah, maybe scientists can, can laugh at me, but whatever, man. I think that people should like see this stuff. I think there's room for that, but there's also room for, I think, like very scientifically rigorous demonstrations, which we, you know, tried to do in the alignment paper. I think we tried to be quite scientifically rigorous. There's like a few ways in which I wish we had done somewhat better. I think on net, the total quantity of scientific rigor is the right amount. I just worry, I just wish we had, like, you know, just like in retrospect, better. And so I think there's like the model organisms thing. I think I'm including like pure, just like relatively low standards, scary demo stuff, trying to be very scientifically rigorous in scary demos. And I'm including like trying to build testbeds to study. So another application is like, look, maybe even if no one cares, like I can totally imagine worlds where basically no one cares about misalignment except some weird people on the safety team. We still want to study these things and we still want to develop countermeasures against them. And so even if no one cares whether or not we catch a totally natural case of scheming and conspiring against us with a model that tries to escape, I'm just like, studying that to get better ideas for what the countermeasures could be still seems very useful, potentially.

Yeah. My understanding, and I really appreciate the fact that Anthropic has done this work and even worked with people who don't work at the company, Such as yourself.

Yeah.

To do this work. My understanding of why they've supported it for a long time has been basically that it sort of sets up a to-do list for the field at large that's like, we've characterized all these problems pretty well. At some point, we're going to need to solve these problems before we have sufficiently powerful systems that we would not want to put into the world unless we could be pretty confident that these problems have indeed been solved. So I, I think that's pretty cool. I got a couple different questions here I'm kind of weaving together, but I noticed you only graduated from college 3 years ago. So maybe one question I would ask is how do people become more like you? You've gone not only far with the research, but have managed to establish your credibility with the folks at Anthropic. And I even noticed that, you know, you've published work with them, but you've also publicly criticized some aspects of what they've done, including a post that I saw on the responsible scaling policy. I mean, there's, you know, credit to go to Anthropic for not giving you a hard time about that. I assume they didn't. But yeah, maybe you could kind of comment on like your strategy for upskilling yourself and continuing to be like candid and honest with the public or your immediate associates and the internet reading public. As you've also been able to like become an insider. I think that's a very fine line to walk and I haven't seen many people walk it very effectively, to be honest.

Yeah, there's a bunch of threads here. So one thread is what did I do with Anthropic on this paper? So I'm not an employee of Anthropic and I'm in some sense, like not formally an insider in some sense, but I do like, you know, maintain relationships with a bunch of people on Anthropic and communicate with them about a bunch of things. As well as talking to other people from other companies, like not necessarily that Anthropic specific, though, you know, I do talk to people from Anthropic more than other companies. But on this specific paper, I just had some early results on the alignment faking stuff, which were like where I think I had basically the prompting results quite fleshed out and had some sort of prototype of a training setup and had done a few other things. And then I went to the Anthropic people with this and was like, hey, Would you guys be willing to give me model access that I could extend these results further and, you know, like actually run training on this on actual, like, Opus? And I was at the time interested in doing some experiments on the helpful-only model, and there were some other things. And they were kind enough to make this happen. And I think, yeah, we ended up turning it into a bigger collaboration than just me having access to the models. Like, a bunch of people from Anthropic worked on the paper and contributed a bunch in terms of, like, running experiments and doing a bunch of the writing. And so I think, you know, credit to them for, like, Supporting this research and doing a bunch of this, you know, doing a large fraction of the project, of course, and also promoting it with their brand and whatever. So like credit specifically to Evan Huminger and Ethan Perez, as well as like the rest of the organization, just to name some names. But anyway, that's that relationship. And then in addition to that, I think, so people don't do that many collaborations with safety researchers where they give them advanced model access. There's some amount of safety testing. So like OpenAI recently announced they were doing safety testing for O3. in advance prior to releasing it. They did the same for GPT-4. There's some amount of pre-release safety testing, but I'm not aware of that many cases where an AI company gave access to— or sorry, gave employee-level access or gave a lot of access to someone who is external, who wasn't necessarily going to work for them to do safety research. And I think AI companies should do more of this. I don't think that any AI company has done that much of this. So I think— sorry, maybe other than— I guess you could credit where it's due. Open sourcing your model does allow people to do this. So Meta and DeepSeek have made their models accessible, and indeed that has allowed for a bunch of research. I think open sourcing indefinitely has a bunch of costs, but at the current margin, open sourcing seems good, especially if it's not leaking capabilities secrets. And so give credit to Meta for supporting a bunch of research too, not just people who do more direct collaborations, I guess. But yeah, my sense is there could be a lot more support even from From Anthropic, from other companies in terms of like giving people helpful-only model access, giving them other types of access. I think OpenAI gives more access than Anthropic does on current margins, basically partially because of just having products. So, you know, it's not necessarily for safety motives. So maybe we should give them proportionally less credit to the extent that less of the motivation was supporting research, but still, like, they should get some credit for just having a fine-tuning API people can use. They're prototyping an RL API, which I think it's going to be somewhat limited in terms of the experiments you can do based on my understanding, but like, you know, that will allow for some additional research that wouldn't have otherwise happened. And that, that seems pretty good. So yeah, I don't know. It's a bit rambly, but like, yeah, there's a bunch of stuff related to model access. As far as toeing the line on like criticism and insiders. So I would say like, I am not in a position where I feel like I, I am like, you know, there's definitely some trade-off here, right? So I think I try to avoid saying things that are like quite inflammatory towards AI companies.

Yeah.

Or like basically unnecessarily inflammatory, maybe is one way to put it. Like, I think so people shouldn't interpret me as being like totally free speaking, but I think at the same time, I'm trying to follow a policy where like, if there's something important to be said that I think people should know, I would say that and try to say that publicly in a way that communicates the point as clearly as possible while also, you know, not trying to cause drama or not trying to cause like— You know, unnecessarily like make people angry basically. And so I am probably more on the side of like saying negative or critical things about AI companies. So I think like an important dynamic, as maybe you were hinting at, is that like a bunch of people are in a position where they potentially need to appease AI companies, or at least are worried about angering AI companies because they're worried about not getting access for things. They might want to work there later, and therefore they are more restricted in what they say. I think this is like in the safety community, I mean.

Yeah.

And I think this is a problematic dynamic and I don't love the situation, but I do at least personally try to communicate important information about the situation when I can. And I think that would be my policy and I'm not holding back that much or there's not that much stuff. It's just more expensive to write things up if you're trying not to make people angry or whatever. Yeah.

Well, I've been wrestling with this a little bit myself. I was asking that question in part because I'm trying to figure out exactly how I should be handling it. I've been through this once. You may know some of that backstory with the GPT-4 red team, where at that time, you know, my small company was an early customer of OpenAI. We had a, and still do actually, to their credit, have a case study on the OpenAI website as being like an early adopter of their fine-tuning products and, you know, successful implementer of it, whatever. But then I was like, this red team project is like woefully inadequate for what we're actually testing and ended up escalating to the board, got kicked out of that program, and I've not been invited back since. So I've sort of lived a little bit of like, yikes, I feel like I kind of need to speak up, but there has been actual cost to that. And I still am kind of juggling it. I truly am super appreciative for this specific kind of work around these Tal Cowen might call it the Department of Yikes. You know, it's like, whoa, we're seeing that. You know, I think it's awesome that Anthropic sponsors and, you know, otherwise sort of engages in that work to bring that stuff to everybody's attention. And so I want to, you know, praise that and I want to have people from the company come on the show and whatever. And then at the same time, I'm like, some of the recent statements from Dario are kind of bugging me out. You know, I really, don't like the idea that we might be about to get into an arms race with China. And in the most recent thing, there was even a sort of invocation of essentially recursive self-improvement to maintain a durable lead vis-à-vis China into the indefinite AI future. And I think a lot of the AI safety people that I know, quite, I think, justifiably freaked out about one or both of those things where they're like, wait, what? We're doing an arms race with China now? You previously said that we should avoid that at almost all costs. And recursive self-improvement too has been the sort of thing that the safety set has always kind of thought is almost for sure going to get away from us. And we haven't really heard an articulation of if or why the outlook has changed on those things. But now we've just got these sort of relatively short op-ed type pieces saying this.

Yeah.

And I'm like, how should I I even understand the company at this point and how should I relate to the company? So I don't know if you have takes on those object-level questions or any guidance for me, but that is, in the spirit of speaking candidly, that is where I'm at at the moment.

Yeah. I mean, on the object level, I think maybe you should interpret the things that companies say as like politically motivated speech or as like things that they're doing for purposes or like there are reasons why they're saying the things they're saying. And so I think it's like, you know, it's a tech company. Like they have motives. They're not, you know, necessarily always like communicating in ways that are most clear and truthful or whatever. And I'm saying this for like, you know, all these companies. Like I think sometimes people, especially in the AI safety community, have, you know, maybe been like more buddy-buddy with Anthropic, but I think it's important to note like Anthropic is a big company doing company stuff. And I think treating it like a company. And being like, what are their interests? What are they going to be doing? Is reasonable. Even if you think the leadership has, you know, good intentions, or even if you think the governance structure will ultimately do good things, I'm like, well, okay, like, at the very least, like, it is operating like a company under the constraints of a company. And it like has to deal with that. Like it has like, you know, corporate stakeholders, it has like a relationship with Amazon, like these things will affect how it behaves. In addition to that, I think on the object level, my view is like, I think it's kind of complicated how we should relate to the situation with respect to China and RSI. Like, my proposed policy would be like, try really hard to not build wildly superhuman AI. Like, that, that feels very scary. And to be clear, I don't mean forever. I just mean like, uh, my view would be like, it seems like a good time to pause at the point where like, we can basically obsolete human labor. And I'm like, building AIs that are substantially smarter than what was needed to obsolete human labor. Feels like it might be a mistake, basically because you get relatively reduced benefit in terms of being able to automate lots of stuff and speed up, you know, a bunch of types of work. While at the same time, maybe the risks get much higher because I think if the AIs are way smarter than us, there's just like a bunch of additional failure modes. I think misalignment is potentially substantially more likely. And like, in principle, the world could get a high fraction of the benefits with systems that are merely like as smart as humans running very fast or whatever. And which are as overseable as humans running fast. So that would be my proposed bar. I worry that this, the situation with very little political will, I think it, I wouldn't unilaterally make a recommendation to an AI company that is more responsible from my perspective. Like if I thought an AI company was more responsible from my perspective, I would not necessarily make the recommendation that they have a strong, we won't build superintelligence policy or like we won't build like wildly superhuman AI policy. I think I would recommend something more like prior to building AIs that smart, try to hand off the situation to AIs that are human level and let them decide what to do. Or like more like human level, like as in before you build a wildly superhuman AI, instead build an AI that is, you know, just capable enough to obsolete decision makers and researchers at your company. Try your hardest to align that system and hand things off to that system and let that system figure out what to do, basically because That way you at least spend more time contemplating, like the AI can spend more time contemplating the question of how you should proceed and do safety research first, et cetera. But I think that's like a pretty scary plan also, like hand things off to an AI system very quickly. Super scary. You should be scared. But it's like maybe better than the alternative of prior to handing, like we're like, oh, we're not handing things over. We're just scaling them up as fast as possible is even worse. Because I think it'd be better to be explicit about handover than build systems where you're de facto handing, like you're the de facto smart enough that if they were misaligned, You would have no hope. I prefer to like hand over to the systems that are as incapable as possible, subject to being able to do a better job than you do, basically because the AIs can just like speed themselves up and they have some other affordances. And I think like maybe one way to put it is like there's different levels of saneness in terms of like what level of will or whatever you have. So maybe the most sane proposal would be like, we just take AI development somewhat slowly. We're pretty careful about it. We like incrementally advance in qualitative units. And we make sure that we have robust safety cases all along the way. Like, as in, our goal is to have like a robust safety case. And maybe we would, like, you know, there's background risk, but like, if you were handling background risks of like totalitarian governments becoming more powerful, and you're handling like background biorisk reasonably well, which I think you could do, then just like moving somewhat slowly on AI such that it's still not arbitrarily slow, but it's enough that like maybe risks from AI Like, the marginal year of delay is buying you like 0.1% risk rather than 5% risk. Like, I think right now it's like, if we could coordinate to delay for a year, we might be like making the situation like way safer, and that would be worth it on a bunch of people's lights. But we might not do that coordination, and people might disagree. So anyway, like, safe proposal would be like, slow things down a lot, and then like proceed only when you have robust safety cases. And I would do like, you know, maybe just proceed only when you have robust safety cases would be be fine. Like, I'm not making a strong claim you have to slow down if you also do that. And then my next proposal, which is like the intermediate safeness proposal, would be like, develop AI pretty quickly, but once you get AIs that are capable enough to massively accelerate R&D, try to pause around that point and proceed slowly from there using not like various approaches to ensure nonproliferation. So do stuff like basically like control those AIs. By control, I mean like prevent them from causing problems even if they wanted to, but you like harness their labor to Like, you know, monitor like AI companies to make sure that there's enough transparency that you can coordinate to work on safety research, to create political will to, you know, ensure nonproliferation by demonstrating capabilities and misalignment concerns. And basically this is sort of like the proposal would be basically like, there's like the U.S. is leading some effort that is aiming to control the rate of capabilities progress in the world as a whole. While handling the risks accordingly, including risks of power concentration. Like, I think a common concern people have is the more the U.S. is sort of running the show, maybe it's the case that you end up with totalitarianism, or at least it's an easy on-ramp to totalitarianism. I tend to think this is at least in principle resolvable with good institutional design and having many governments that are part of the project and having many stakeholders. Anyway, so there's like the, like, go slow unless you have good safety cases, pause around human level. And then my third proposal would be like, race as hard as you can, but, but at least try to hand over the situation to human-level AIs prior to building wildly superhuman AIs. And that'd be like the least dignity plan or like the no one cares at all plan. Like, I think I've been thinking recently about worlds where basically everyone is just proceeding as fast as possible. Many actors are neck and neck. Governments throughout the world don't care very much or not really on their eyes, really not that much on the ball, like not substantially more than it is today. Or like maybe somewhat more, but maybe not that helpfully. And it's like 2027, like we just don't have much time. And I think in those worlds, I'm just like, it's not clear that I would make the recommendation that companies like unilaterally stop themselves from using their own AIs to like advance AI research. But I might make the proposal that at least prior to building AIs that are very superhuman, you try to build a system that you're happy to defer to. And to the extent that you're like, I wouldn't defer to a system, don't build one that's wildly superhuman. Maybe that's—

Yeah.

some context on how I think about these specific things.

Could you offer a P-doom conditional on those 3 approaches? Like, how much difference do you think it makes which approach we take?

So we could talk about like a good implementation of each of them. So like, let's say you do a good implementation of the safety case approach where you're like carefully proceeding, you're being slow to the extent that that is like, I think the proposal I would say is basically like you have some sort of international governance regime where AI companies have to make high assurance safety cases indicating low levels of lifetime risk. So like, maybe my guess is that that yields, like, if you do a good job of it, like 1% misalignment risk if it succeeds. And then there's some chance that there comes some regime where people are unable to make high assurance safety cases. And that puts a lot of stress on the regulatory regime because now it's like, you know, blocking progress in this whole industry in a very direct way. And then at that point, I think it's like, if you persisted in the regime, there's a question like, how fast would you be able to resolve the problems? Right? Like, I think once you were talking about like multi-decade pauses, I think it's like, I'm also less certain about the sign because the world is unstable. And it's not obvious to me that if you have a good regulatory regime, you should be happy to pause for very long because maybe your regulatory regime will collapse and you'll return to an even worse situation. Like, there's different operationalizations, but like, One is like, if it like basically worked and didn't require multi-decade pauses, maybe my guess would be like 1% risk from misalignment. And then it would be like somewhere between like 3 to 5% risk from like random other stuff. Maybe it depends on how much your caution about misalignment transferred to caution about other risks. But like, maybe that'd be my guess at the risks. If you're really rapidly advancing like AI capabilities and it's going very fast, I think it's hard to be very confident in even without misalignment risks, how things will go, basically because there's crazy technology. I think there's concerns about proliferation, about WMDs. There's concerns about super persuasion being societally destabilising, human power grabs. All these things are concerning, but I don't know, that's my sense. If you do the world where you have enough government buy-in to attempt to ensure— so the second world is you try to ensure non-proliferation around the point where you have human-ish level AIs and you're doing sort of this pause at human level for Maybe you're trying to pause for 5 to 10 years. My guess is that that takes misalignment risk from 1% and moves it to more like 5 to 10%. And other risks move from like 3 to 5% to more like 10%. So you're more like in a 15% P-doom world. And then my guess is that the last world I was talking about where like there's a bunch of people and they're basically going as fast as possible trying to hand off the AI systems. Maybe my guess is misalignment risk is like 30 or 35% or something. So it's like another factor of 2 or so, or maybe like slightly more than a factor of 2 over the like positive human level world. And other risks are like, I don't know, 25%. So my overall PDoU would be like 60%. So I don't know, maybe my guess is like overall it's like a factor of 3 or 4 between each of those worlds. But I think there's a bunch of risks that are harder to mitigate in advance than misalignment risks. So those risks are like less elastic to the societal will. Sure. I don't know.

So in other words, if I bottom line that. The approach that we collectively take to AI in your mind has basically an order of magnitude impact on the absolute risk that we would be running, where you think it's essentially like a 50/50 if we YOLO it and—

Yeah, 50/50 if we YOLO it is a—

Ish percent if we like were maximally careful.

Yeah, maybe if we like totally YOLO it, maybe it's like, 60/40 something goes very badly. And then I think some of that's misalignment and some of that's other stuff. So it's not, I wouldn't say it's just misalignment to me. Maybe I'm like maximally YOLO. It's like 35% doom from misalignment and then 25% doom from other stuff, roughly. And then, yeah, if we had like pretty socially optimal but plausibly realistic international governance regime, then I think I would be like, risks are more like 4 or 5% where like one, we are like 1% misalignment risk and like 4% or something of a bunch of other risks. Though, that said, I should be clear. I think it's a little complicated how, what happens in the international governance world if you run into hard technical problems where you can't make a robust safety case. What happens if you're like, oh, we have this governance regime and it just turns out people are now failing to make safety cases because they don't have robust enough mitigations for scheming risk, alignment-faking stuff, this sort of thing? How long of a pause does that cause? And I think if the pause is sufficiently long, then The risks naturally increase again because there's just a question of like, you know, any international governance situation is some probability of collapsing. You know, it's inherently somewhat unstable. And so maybe my guess is like, realistically, it's somewhat hard to get the risks that low.

But yeah, my sense is I feel pretty similarly. I mean, it's funny, I haven't probably, I have not, to be clear, quantified them to that level of precision.

Mm-hmm.

But intuitively, I feel similarly. I feel there's some irreducible risk that is just like the physics of the world we live in is such that if you have compute and data at the scale that we have, various algorithms are gonna work and people are gonna be able to create powerful things. And, you know, that seems kind of on some level irreducible, but then we could do it a lot worse than the best case and It feels like there's a pretty big multiplier on that. So I think, roughly speaking, my intuition is pretty similar that I don't think we can get the risk vanishingly low with any effort, but we could definitely make it very large.

Yeah, I think if civilization was wildly more competent, we could probably get it to be vanishingly low. But I'm not huge— it's hard for me to say that much. I think I'm like, if, for example, we—

Be accomplished by an indefinite pause, if there's just things that are Really, really hard.

Maybe. I think it would be, yeah. I mean, I don't know. Yeah. Maybe not like an indefinite pause. I think maybe it's more like, you know, spend a long time, you like develop more and more institutional muscle for like doing alignment research. People, you know, get smarter and build better institutions over time. And then eventually you do stuff and like, I think it's hard for me to talk about these worlds because like whether this is good is somewhat sensitive to your moral views and how you, like in this competent world, like do you have cryonics? Are people like dying is 1% of the population? population dying every year. I think there's like pretty reasonable moral cases that the world where 1% of people are dying every year, or like it's a little less, it's like 0.7% of people dying every year for 100 years, is worse than the world where we build AI somewhat faster than that at 2% risk. Like, as in, imagine we could pause for 20 years and then we'd have 2% like misalignment risk versus we can pause for 100 years and now we have 0.1% misalignment risk. Or I don't know, there's other risks too. So maybe this is a bit of a caricature, but like, let's say you go from 2% to 0.1%, but in the meanwhile, like almost everyone has died because 100 years have passed. And the question is like, okay, how happy are you about this? And I think there's a legitimate moral case for like build the AI faster for the sort of reason. But I would also say like, I think the common sense ethical intuition that like people should be, you know, able to live the life that they would have like in some sense naturally been due is like one thing. I don't know if I buy this view, but like. That is a common view. And then on my views, like, I just like put more weight on sort of like a long-run future perspective such that if I thought that like world was in good hands and governed well, delaying AI seems pretty good. And then I think also in principle, we do have like, if we scaled up, I think like the world could afford to do cryonics for everyone or something in principle. If we're imagining like fairy utopia or whatever, I'm like, we could actually do cryonics for everyone. And I think this would have a pretty high chance of working if we did a good job and put a lot of research effort into it.

Yeah.

Such that people don't actually have to die or whatever, and then you're just in a better position. And then delay is less costly from the straightforward ethical perspective. There's still some question of how did you manage other risks? There's some exogenous biorisk, there's risk of civilizational collapse, there's risk of nuclear war. And imagine you had an international governance regime, but then there's nuclear war. Well, maybe the international governance regime collapsed and you actually haven't gotten yourself in a much better position if you— You maybe wanted to build AI under the international governance regime. Rather than wait 2 decades, nuclear war, and then it's built in a huge rush afterward.

So safe to say there's a lot of contingencies and as it stands, we seem to be kind of on track to YOLO it. Is that your sense too?

I think it's short timelines. Yeah. And my sense is like in short timelines, probably it's going to be pretty YOLO. I have a lot of uncertainty about how people will react to things that From my perspective are like smoking guns. It's like, suppose that at some AI company, they catch the AI straightforwardly trying to escape. Like it's just very obviously trying to escape. And then they catch it halfway. And then like, you know, that gets published. Will this cause a large societal response? No societal response? People to like really freak out? Like people to take reasonable precautions? People to freak out but take unreasonable precautions? I think it's very unclear. I think that a lot of worlds could in principle be saved by the mechanism of you get really compelling evidence of strong misalignment risks midway through the situation, and then something good happens. But I worry that even in some sense, very compelling misalignment demonstrations, like here's the AI, it literally tried to escape this time, we didn't want it to escape, it totally tried to do it, is I think possibly insufficient. And in addition to that, it's not that likely we get this. I think like something as clear-cut as the AI tried to escape or more is like maybe on my view is 40% likely prior to like, we're basically already, it's basically already over for us. Maybe a little less than maybe 35% likely. And then, yeah, I mean, you could get more clear-cut evidence than that, to be clear. Like you could be like, the AI literally escaped after literally escaping. It like built a bioweapon. It deployed the bioweapon. The bioweapon killed a ton of people. Then we like caught the AI, we dismantled the compute. And then now we're like, oh man, the AIs sure can get up to all kinds of nonsense. I'm like, that's maybe like the most clear-cut case. And we have like clear attribution for the AI creating the bioweapon, et cetera, et cetera. But I'm just like, man, I feel like that's like unrealistically precise. Or like the worlds where you get warning shots that clear, I think are just kind of unlikely. And also, I don't think we can rely on like that, that like, that's just like a very narrow set of worlds. And also like a bunch of people had to die to make that happen. Like, I'm just like, geez, did we really have to have that happen? Could we avoid Yeah.

Yeah. Well, that's why the whole China thing from Dario in particular has really been bugging me lately because, and I hear you on the political speech and I've certainly got a lot of responses to a few, you know, short tweets I've put out about this topic.

Yeah.

But I'm kind of like, man, we got enough Don't Look Up problems already. And this does seem to be the one company that is most committed to demonstrating just how vexing some of these problems can be. So it seems a big problem if the leadership of that company is also at the same time putting out into the world, like, but we've gotta, you know, beat China because that becomes the trump card in so many discussions, you know? And I think it is, you can imagine, as you kind of sketched out, a scenario in which like the AI goes so wrong that we have no choice but to wake up and say, geez, we got to bury the hatchet with China and figure this out first. But I also agree with you that that seems rather unlikely. And what's more likely is you'll put out 10 more papers on this and people will go, well, okay, sure. But China's still the bigger risk. And that to me is just like, man, I wish we were not. It's one of my refrains these days is, it was the smartest of times, it was the stupidest of times. And it's like, you know, we're getting to these like human-level-ish AIs.

Yeah.

And yet we can't like get on the same page at all about how to deal with it. So I share your worry. It does feel like we're headed for YOLO. How are you sleeping?

I don't know. I, yeah, I'm more stressed these days than I used to be, but sleeping okay. I think there's a question of like, As someone in my sort of doing the stuff I'm doing, how much should you try yourself to like system 1 really feel the danger? Like how much should you like really internalize like, man, I might physically die in a few years because of this? To be clear, I don't think it's that likely. Like I think even conditional on misaligned AI takeover, maybe the chance that I personally die is like only like a third or half or something. So, you know, there's another discount on top of the other stuff. But yeah, it's not clear how healthy that is. Like I think my current take is like, Probably it's good to be not totally in near mode for some of these things. In some, like, it's just like the human body was not built for doing intellectual work while also system 1 grokking that you're in a dangerous position because the fight or flight response is not useful for programming or whatever.

Yeah.

Maybe it's not the worst, but at least for like doing careful intellectual work or being open to ideas and like, I think it's just important for my work to be like Sort of like maybe scout mindset, like intellectually light or not take things too, be adaptable, be willing to change my views kind of quickly. And I think just being in sort of a fear mindset is not good for that. So, yep.

Yeah.

Yeah. I think on the race with China stuff, I think I'm pretty sympathetic to a view, which is like the US should try to have a good negotiating position with China. And I'm pretty sad about a view or pretty sad about a position where we don't try to negotiate and don't try to make some sort of arrangement or don't try to, like, just like the US, I think the US should like try to ensure that misalignment risks are handled well and try to like do nonproliferation as needed for that. And I think there's a lot of stuff I can imagine happening in the current regime, which seemed pretty bad. Like, I think it seems plausible that what we'll see is like, uh, someone, someone recently just used the term superintelligence in secret and I just like, Man, I really don't want superintelligence in secret. Like, I really don't want people like building extremely powerful AIs very quickly when the world doesn't even know what's going on at all. And it's, you know, in order to outrace competitors that could have potentially been open to some sort of negotiation to handle the situation more carefully and, and in a way where it, it reduces the risks. I think even putting aside misalignment risks, I think there's just a ton of risks associated with building like insanely powerful AI very quickly. Like, it's like very unprecedented. It's a large system. Societal shift. I think the risks of human takeover are sub— or like human power grabs are substantial. Like if human labor is no longer a key bottleneck to many things, then I think various things like coups are easier. Like when governments depend on fewer people, it just becomes easier to do coups. And I worry about private companies having a lot of power relative to what they previously had in a way that is quite destabilizing and results in potentially problematic allocations of power and concentration of power in the long run. I think there's a bunch of things about like handling, how do we handle ultra dangerous technology? Like by default, if we have very widespread AI proliferation, I think it might be the case that like every 20-person organization can have near-omnicidal bioweapons. And I'm just like, is the world ready for like every single, like, it's just extremely easy to make near-omnicidal bioweapons. And I'm like, probably we won't end up there because probably someone will notice in time and take some precautions and do something. But I'm just like, If you really YOLO it, I can just totally imagine a world in which it's like everyone has crazy super— or like not everyone, but like, you know, like hundreds or thousands of groups have like crazy super weapons. And I'm just like, are we ready to like handle that world where that's the case? Like a lot of things have bad offense-defense balance and like mutually assured destruction does not work against terrorist groups. Like it's not, yeah, it's a bad situation to be in. So yeah, anyway, that's some more on that, but yeah. I still think things like the export controls look good. I think, Advocating for export controls is good, trying to like do that. But I think that's in some sense like only one of the steps the US should be taking. I think there's like more ambitious political proposals in terms of trying to build a regime where, you know, everyone, like ideally, I think the world would be like everyone, all countries are happy that their sovereignty will be, or like confident their sovereignty will be preserved with AI development is like maybe a guarantee you'd want to aim for. And that would be like, first you Or like, sorry, at least, sorry, maybe the goal would be if a country agrees to, you know, join your like AI project or whatever, or like join the international treaty, then they can be confident their sovereignty will be preserved, which requires like handling human power grabs and handling AI power grabs. And I think being in a regime where all the countries are like, look, this AI stuff, we're like handling it kind of responsibly. Our sovereignty will be preserved. We're not going to get disempowered by the AIs. And also, we don't need to take aggressive action immediately because the situation is handled, which would be like a good place to be. And I think the US could, in principle, you know, try to do some dealmaking, try to push for this. And for other countries, like, I think even if the US isn't that excited about it, like, they're not the only stakeholder, you know, the semiconductor supply chain, there's like, you know, Netherlands, Japan, they could potentially team up with like the UK and try to do their thing. And ideally, they'd try to get the US too. But yeah.

Yeah. Well, that's a, at least a little bit of a path to sketch out that we might try to go down. I think it's been fantastic. I really appreciate all the time and I know you've got a lot on your plate. So to take a full half day talking to us is much appreciated. Anything else you want to share?

Sure.

But we covered a lot, so I don't know if there's anything there or anything else you just want to call for any, you know, any sort of collaborator profile or anything you want to, you know, invite. You could do that if you'd like before we wrap up. break?

Yeah, 2 things. One thing is, so Redwood has a Substack. Consider reading some of our posts that goes into like some of the stuff I was talking about in more detail. And maybe people would just find it interesting if you found this podcast interesting. Maybe you can throw a link in the whatever linky thing there is. Also, you could look at some of the posts on my LessWrong account or on Buck's, like Buck Schlegel's LessWrong account. We just like have some different content there, though it's somewhat overlapping. People might find this interesting. In addition to that, I would say people should consider applying to Redwood. So we're hiring. We're interested in people who have takes on a lot of the stuff we were talking about, people who are interested in sort of AI Futurism, what should the plan be? We're also interested in just people who are good at empirical ML research, ideally a mix, but potentially one or the other can also be interesting. Yep, we're expanding. Come work for us.

Love it. Ryan Greenblatt, Chief Scientist at Redwood Research. Thank you for being part of the Cognitive Revolution.

Yeah, thanks so much for having me.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email. At tcr@turpentine.co, or you can DM me on the social media platform of your choice.