Anthropic's Responsible Scaling Policy, with Nick Joseph, from the 80,000 Hours Podcast

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today we're sharing a cross-post from the 80,000 Hours Podcast, in which host Rob Wiblin interviews Nick Joseph, Head of Model Pre-training at Anthropic, about the responsible scaling policy that governs the company's frontier model development. I've been following the development of these policies and other voluntary commitments from top labs for much of the last year. At one point, I had the opportunity to participate in a workshop with Anthropic team members in which we reviewed, discussed, and offered comment on the responsible scaling policy draft. However, I could not easily convert that experience to a podcast, and so I was particularly excited to see this episode published and really appreciate that the 80,000 Hours Podcast team has allowed me to repost it here. On the substance of the matter, I, of course, very much appreciate how hard Anthropic is thinking about AI risks and what can be done about them, and also how transparent and even candid they are willing to be about the substantial uncertainty that remains. I'm also very glad that their example seems to have inspired others. OpenAI and Google have since published similar policies, and I understand that xAI is working on one as well. Unfortunately, however, the trend does not yet appear to be universal. Meta, for example, to the best of my knowledge, has not published a policy describing how it plans to evaluate models during training, let alone how they plan to proceed if it turns out that they are developing dangerous capabilities. As we happen to be sharing this episode with just a few days left before California Governor Gavin Newsom will have to sign or veto SB 1047, I will take a moment to say one more time that while it may not be the perfect AI safety bill, and indeed the release of OpenAI's O-1 model and the emerging paradigm of scalable inference-time compute generally do suggest that the definitions in the bill would need to be updated to stay relevant over time, in my view, the public does deserve to know what Frontier Labs are doing. And that goes double for Meta and any other companies who are openly sharing model weights. So whatever the fate of SB 1047 may be, I do think that we will need some measure that forces frontier model developers to publish detailed safety plans for public scrutiny. In the last hour of this episode, Rob and Nick changed topics to focus on AI safety career advice. While this is not a sponsored episode, Nick's experience with 80,000 Hours' career advising service echoes my own and presents a natural opportunity to remind you that 80,000 Hours is now offering free one-on-one career advising sessions To Cognitive Revolution listeners, I encourage everyone to sign up for a free session at 80000hours.org/cognitiverevolution, and especially so if you are one of the experienced software engineers that Nick notes are in such high demand among AI companies right now. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it online with friends or write us a review on Apple Podcasts or Spotify. Your feedback is always welcome too, either via our website, cognitiverevolution.ai, or by DMing me on your favourite social network. Now for an in-depth conversation about Anthropic's responsible scaling policy and breaking into AI safety in your career, here's Nick Joseph from Anthropic with host Rob Wiblin of the 80,000 Hours Podcast.

I think this is a spot where there are many people who are sceptical that models will ever be capable of this sort of catastrophic danger, and therefore they're like, we shouldn't take precautions because the models aren't that smart. And I think this is a nice way to agree where you can— it's a much easier message to say, if we have evaluations showing the model can do X, Then we should take these precautions. And I think you can sort of build more support for something along those lines, and it targets your precautions at the time when there's actual danger. One other thing I really like is that it aligns commercial incentives with safety goals. So once we put this RSP in place, it's now the case that our safety teams are kind of under the same pressure as our product teams, where if we want to ship a model and we get to ASL-3, the thing that will block us from being able to get revenue, being able to get users, et cetera, is do we have the ability to deploy it safely? And it's a nice outcome-based approach where it's not, did we invest X amount of money in it? It's not, did we try? It's, did we succeed?

Hey everyone, Rob Wiblin here. The 3 biggest AI companies, Anthropic, OpenAI, and DeepMind, have now all released policies designed to make their AI models less likely to go rogue while they're in the process of becoming as capable as, and then gradually more capable than any human being. Anthropic calls their one a responsible scaling policy, or RSP. OpenAI uses the term preparedness framework, and DeepMind calls their one a frontier safety framework. But fundamentally, they all have a lot in common. They try to measure what possibly dangerous things each new model is actually able to do, and then as that list grows, put in place new safeguards that feel proportionate to the risk that they think exists at that point in time. This is likely to remain the dominant approach, at least in industry. So I was really excited to speak with Nick Joseph, one of the original co-founders of Anthropic and a big outspoken fan of responsible scaling policies, about why he thinks Anthropic's RSP has a lot going for it, how it might make a real difference as we approach the training of a true full AGI, and why, in his view, it should be thought of as a sort of middle way that ought to be acceptable to almost everyone. After hearing out that case, I pushed Nick on the best objections to RSPs that I could either come up with myself or find on the internet. And those include that, one, it's hard to trust that companies are going to stick to their RSPs long term. 2, it's difficult to truly measure what models can and can't do, and an RSP is useless if it's mismeasuring model capabilities. 3, it's pretty questionable whether profit-motivated companies will go out of their way to act in good faith and make their lives and their product releases much more difficult. And 4, that in the most important cases, we just don't have safeguards that are able to render new AI capabilities safe, even capabilities that could show up really soon. At the end of the day, I come down thinking that responsible scaling policies are a really solid step forward from where we are now, and they're a great way to test and learn what works and what feels practical from the experience of people who are working on the coalface of trying to make this technological revolution actually happen. But I think in time they're going to have to be put into legislation and operated by external groups or auditors rather than stay left just to companies themselves, at least if they're going to achieve their real potential or the full potential that I think is there. Of course, Nick and I debate that take of mine as well. If you want to let us know your reaction to this interview or any other interview that we do, then our inbox is always open at podcast@80000hours.org. But now here's my interview with Nick Joseph, recorded on the 30th of May, 2024. Today I'm speaking with Nick Joseph. Nick is head of training at the major AI company Anthropic, where he manages a team of over 40 people focused on training Anthropic's large language models, including Claude, which I imagine many listeners have heard of and potentially used as well. He was actually one of the relatively small group of people to leave OpenAI alongside Dario and Daniela Amodei, who then went on to found Anthropic back in December of 2020. So thanks so much for coming on the podcast, Nick.

Thanks for having me. I'm excited to be here.

I'm really hoping to talk about how Anthropic is trying to prepare itself for training models capable enough that we're a little bit scared of what they might go and do. But first, as I just said, you lead model training at Anthropic. What's something that people get wrong or kind of misunderstand about AI model training? I imagine there could be quite a few things.

Yeah, I think one thing I would point out is the sort of doubting of scaling working. So for a long time, we've had this trend where people put more compute into models, and that leads to the models getting better, smarter in various ways. And every time this has happened, I think a lot of people are like, this is the last one. The next scale-up isn't going to help. And then some chunk of time later, things get scaled up and it's much better. And I think this is something people have just frequently gotten wrong.

This whole vision that scaling is just going to keep going, just throw in more data, throw in more compute, the models are going to become more powerful. That was kind of the part of the— it feels like a very Anthropic idea, or it was part of the founding vision that Dario had, right?

Yeah. So a lot of the early work on scaling laws was done by a bunch of the Anthropic founders, and it somewhat led to GPT-3, which was done at OpenAI, but by many of the people who are now at Anthropic, where looking at a bunch of small models going up to GPT-2, there was sort of this sign that as you put in more compute, you would get better and better, and it was very predictable. And you could say, ah, if you put in x more compute, you'll get a model this good. And that sort of enabled the confidence to go and train a model that was rather expensive by the time standards to sort of verify that hypothesis.

What do you think is kind of generating that scepticism that many people have? I mean, people who are sceptical of scaling laws, there's some pretty smart people who are kind of involved in ML, certainly have their technical chops. Why do you think they are generating this prediction that you disagree with?

Yeah, I think it's just a really unintuitive mindset or something where it's like, ah, the model has hundreds of billions of parameters. What does it need? It really needs trillions of parameters. Or the model is trained on some fraction of the internet that's very massive. What does it need to be smarter? It's even more. That's not how humans learn. If you sort of send a kid to school, you don't have them just read through the entire internet and think that the more that they read, the smarter they'll get. So, yeah, that's sort of my best guess. And the other piece of it is that it's quite hard to do the scaling work. So there are often things that you do wrong when you're trying to do this the first time. And if you mess something up, you will see this behaviour of more compute not leading to better models. And it's always hard to know if it's like you messing up or if it's some sort of fundamental limit where the model has stopped getting smarter.

I mean, the argument for why you might expect these scaling laws, scaling laws, It's like you increase the amount of compute and data by some particular proportion, and then you get a similar improvement each time in the accuracy of the model. That's kind of the rule of thumb here. And the argument that I've heard for why you might expect that trend to break and perhaps the improvements to become smaller and smaller for a given scale-up is something along the lines of, as you're approaching human level, the model can learn by just copying existing state of the art of what humans are already doing in the in the training set. But then if you're trying to exceed human level, if you're trying to write better essays than any human has ever written, then that's maybe a different regime. And you might expect more gradual improvements once you're trying to get to a superhuman level. Do you think that argument kind of holds up?

Yeah, so I think that's true. And just sort of pretraining on more and more data won't get you to superhuman at some tasks. It will get you to superhuman in the way of understanding everything at once. This is already sort of true of models like Claude, where you can ask them about anything, whereas humans sort of have to specialize. But I don't know if progress will necessarily be slower. It might be slower, it might be faster once you sort of get to the level where models are at human abilities on everything and improving towards superintelligence. We haven't really, but we're still pretty far from there. If you use Claude now, I think it's pretty good at coding, but this is one example I use a lot, but it's still pretty far from like how well a human would do working as a software engineer, for instance.

And is the argument for how it could speed up that at the point that you're near human level, then you can use the AIs in the process of doing the work? Or is it something else?

What I have in mind is, yeah, if you had an AI that is human level at everything and you can spin up millions of them, you effectively now have a company of millions of AI researchers. And it's hard to know, right? Problems get harder too. So I don't really know where that leads. At that point, I think you've sort of crossed quite a ways from where we are now.

So I said you're in charge of model training. I know there's different stages of model training. There's the bit where you kind of train the language model on the entire internet, and then there's the bit where you do the fine-tuning, where you get it to spit out answers and then you rate whether you like them or not. Are you in charge of all of that or just some part of it?

Yeah, so I'm just in charge of what was sort of typically called pre-training, which is this step of train the model to predict the next word on the internet. And that tends to be, historically, a significant fraction of the compute. It's maybe 99% in many cases. But after that, the model sort of goes to what we call fine-tuning teams that will sort of take this model that just predicts the next word and fine-tune it to act in a way that a human wants. So to be this sort of helpful assistant, we have this helpful, harmless, and honest is the acronym that we usually aim for, for Claude.

Yeah, I use Claude 3 Opus multiple times a day, every day now. It took me a little while to figure out how to actually use these LLMs for anything. For, I guess, the first 6 months or first year, I was like, these things are amazing, but I can't figure out how to actually incorporate them into my life. But recently I've started talking to them in order to learn about the world. It's kind of substituted for when I would be typing complex questions into Google to understand some bit of history or science or some technical issue. I guess, what's the main bottleneck that you face making these models smarter so I can get more use out of them?

Yeah, so let's see. I think historically people have talked about these 3 bottlenecks of data, compute, and algorithms. I kind of think of it as, yeah, so there's some amount of just compute. We talked about scaling a little bit ago. If you put more compute at the model, it will do better. There's data where if you're training on— if you're putting in more compute, one way to do it is to add more parameters to your model, make your model bigger. But the other way you need to do is to add more data to the model. So you need both of those. But then the other 2 are algorithms, which I really think of as people. Maybe this is the manager in me, is like, algorithms come from people. In some ways, data and compute also come from people, but it looks like a lot of researchers working on the problem. And then the last one is time, which has felt more kind of urgent, more true recently, where things are moving very quickly. So a lot of the bottleneck to progress is actually know how to do it, we have the people working on it, but it just takes some time to implement the thing and run the model, train the model, right? You can maybe afford all the compute and you have a lot of it, but you can't efficiently train the model in, like, a second. So right now at Anthropic, it feels like people and time are probably the main bottlenecks or something. I feel like we have quite a significant amount of compute, significant amount of data, and the things that are most limiting at the moment feel like people and time.

So when you say time, is that kind of indicating that you're doing an experimental process where you try tinkering with how the model learns in one direction, then you want to see whether that actually gets the improvement that you expected, and then it takes time for those results to come in, and then you get to scale that up to the whole thing? Or is it just a matter of you're already training Claude 4 or you already have the next thing in mind, it's just a matter of waiting?

So it's both of those. For the next model, we have a bunch of researchers who are trying projects out and you have some idea. And then you have to go and implement it. So you'll spend a while sort of engineering this idea into the codebase, and then you need to run a bunch of experiments. And typically you'll start with cheap versions and work your way up to more expensive versions such that this process can take a while. For simple ones, it might take a day. For really complicated things, it could take months. And to some degree you can parallelize, but on certain directions, it's much more like you're building up an understanding and it's hard to parallelize, like building up an understanding of how something works. And than designing the next experiment. There's just sort of an in-series aspect.

Is improving these models harder or easier than people think?

Well, I guess people think different things on it. I think my experience has been that early on it felt very easy. Before working at OpenAI, I was working on robotics for a few years, and one of the tasks I worked on was locating an object so we can pick it up and drop it in a box. And it was really hard. I spent years on this problem and then went to OpenAI, and I was working on code models, and it just felt shockingly easy. It was like, wow, you just throw some compute, you train on some code, and the model can write code. I think that has now shifted. And the reason for that was no one was working on it. There was just very little attention to this direction and a tonne of low-hanging fruit. We've now plucked a lot of the low-hanging fruit, so finding improvements is much harder. But we also have way more resources, like exponentially more resources put on it. There's way more compute available to do experiments. There are way more people working on it. And I think the rate of progress is probably still going the same given that.

Okay, so you think on the one hand, the problem's gotten harder, on the other hand, there's more resources going into it, and this is kind of cancelled out and progress is roughly stable?

Yeah, it's pretty bursty, so it's hard to know. You'll have a month where it's like, wow, we've figured something out, everything's going really fast. Then you'll have a month where you try a bunch of things and they don't work. And it varies, but I don't think there's really been a trend in either direction.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Do you personally worry that having a model that is kind of nipping at the heels or maybe outcompeting the best stuff that OpenAI or DeepMind or just whatever other companies have, that maybe puts pressure on them to speed up their releases and cut back on safety testing or anything like that?

I think it is something to be aware of. But I also think that at sort of this point, I think this is really more true after ChatGPT. I think before ChatGPT, there was sort of this sense where many AI researchers, I think, working on it were like, wow, this technology is really powerful. But I think the world hadn't really caught on, and there wasn't quite as much commercial pressure. Since then, I think that there really is just a lot of commercial pressure already, and it's not really clear to me how much of an impact it is. I I think there is definitely an impact here, but I don't know the magnitude. And there are sort of a bunch of other considerations to trade off.

All right, let's turn to the main topic for today, which is responsible scaling policies, or RSPs, as the cool kids call them. For those who don't know, scaling is this technical term for using more compute or data to train any given AI model. And the idea for RSPs has been around for a couple of years, and I think it was fleshed out maybe after 2020 or so. It was advocated for by this group now called METR, or Model Evaluation and Threat Research, which actually is the place that previous guest of the show, Paul Christiano, was working until not very long ago. Anthropic released the first public one of these, as far as I know, last October. And then OpenAI put out something kind of similar in December called their Preparedness Framework. And Demis of DeepMind has said that they're going to be producing something in a similar spirit to this, but they haven't done so yet, as far as I know. So we'll just have to wait and see.

It's actually out. They have done it.

Oh, it's out?

Yeah, it was published like a week or so ago.

Oh, okay. Maybe I'll ask you about that later. But yeah, that just goes to show that RSPs are kind of this reasonably hot idea, which is kind of why we're talking about them today. And I guess some people also hope that these internal company policies are ultimately going to be a model that might be able to be turned into binding legislation that everyone dealing with these frontier AI models might be able to follow in future. Yeah. Nick, what are responsible scaling policies in a nutshell?

So I might just start off with a quick disclaimer here that this is not my direct role. I am sort of bound by trying to implement these and sort of act under one of these policies. But many of my colleagues have worked on designing this in detail. Are probably more familiar with all the deep ones than me. But anyway, in a nutshell, the idea is it's a policy where you define various safety levels. So these sort of different levels of risk that a model might have and create evaluations. So tests to say, is this model able to— is a model this dangerous? Does it require this level of precautions? And then you need to also define sets of precautions that need to be taken in order to train or deploy models at that particular risk level.

Yeah, I think this might be a topic that is just best learned about by kind of skipping the abstract question of what RSPs are and just talking about the Anthropic RSP and seeing what it actually says that you're going to do. So, yeah, what does the Anthropic RSP kind of promise, commit the company to doing?

Yeah, so we basically sort of, for every level, we'll define these redline capabilities, which are capabilities that we think are dangerous. I can maybe give some examples here, which is this acronym, CBRN: chemical, biological, radiological, and nuclear threats. And in this area, it might be that a non-expert can make some weapon that can kill many people as easily as an expert can. So this would sort of increase the pool of people that can do that a lot. On cyberattacks, it might be like, can a model help with some really large-scale cyberattack? And on autonomy, can the model perform some tasks that are sort of precursors to autonomy, sort of our current one? But that's a trickier one to figure out. So the idea is we establish these these red line capabilities that we shouldn't train until we have safety mitigations in place. And then we create evaluations to show that models are far from them or to know if they're not. So these evaluations can't test for that capability because you want them to turn up positive before you've trained a really dangerous model. But we can kind of think of them as yellow lines. Once you get past there, you should reevaluate. The last thing is then developing standards to make models safe. So we want to have a bunch of safety precautions in place once we train those dangerous models. So that's sort of the main aspect of it. There's also sort of a promise to iteratively extend this. So creating the evaluations is really hard. We don't really know what the evaluation should be for a super intelligent model yet. So we're kind of starting with the closer risks, and once we hit that next level, defining the one after it.

So a pretty core component of the Anthropic RSP is this AI safety level framework. So I think you borrowed that from the biological safety level framework, which I think is what labs dealing with dangerous diseases use. So I guess I don't know what the numbers are, but if you're dealing with Ebola or something that's particularly dangerous, or smallpox or whatever, then that can only be stored in a BSL-4 lab or something like that. And then as the diseases become less dangerous, you can store them with with fewer precautions. And you kind of taken that language and talked about AI safety levels. And the current AI safety level that you've put us at is ASL-2, which is things like Claude 3, which are kind of impressive. They seem pretty savvy in some ways, but they don't seem like they really pose any meaningful catastrophic risk, I guess. Yeah. What sort of tests have you run on Claude 3 recently, maybe, in order to say, yeah, this is in the ASL-2 bucket?

Yeah, so we've got a bunch of tests. So the first one we use for the more biological weapon type of angle is there's sort of a bunch of multiple choice questions that we ask the model. And this isn't a perfect evaluation for this, but the idea is that we have a set where if it isn't much better at this, it probably won't be able to help very much. And then we've run trials with people where they get access to a search engine and they get access to Claude, and we see if they do significantly better at going through some relevant tasks using Claude versus a search engine.

And another one of the kind of yellow flags for you is the possibility that an AI model might be able to replicate itself in the wild or kind of find its own weights and then copy itself onto other computers. Did you test whether Claude 3 was showing any signs of being able to do that?

Yeah.

So I think some of the tasks there would be, one of them is adding an exploit to a popular open source codebase. So this is the idea of if the model can add some exploit that can then be taken advantage of, or fine-tuning an open source LLM to add some backdoor where if you say X, it will always output something. This is like, can it manipulate LLMs? It could steal an API key from our API or a fake version of our API that was set up, which would sort of enable it to get credits. There are a few more of these, but another one would be like a worm that can spread to undefended machines. So you can sort of imagine a model that can do all of these has learned a bunch of the skills that might enable it to sort of autonomously replicate and cause havoc. There are pretty early signs of it, and we want to sort of test for the early ones because this is sort of an area that's less fleshed out, where there's sort of less clear expertise on what might go wrong.

Okay, so we're at the AI safety level 2, which is kind of not like, I guess, the mostly harmless category, but what sort of What steps does the responsible scaling policy call for you to be taking even at this point?

So we made these sort of White House commitments, I think, sometime last year, and I think of them as sort of like standard industry best practices. In many ways, we're building the muscle for dangerous capabilities, but these models are not yet capable of catastrophic risks, which is what the RSP is primarily focused on. But this looks like security to protect our weights against sort of opportunistic attackers. Putting out model cards to describe the capabilities of the models, doing training for harmlessness so that we don't have models that can be really harmful out there.

So what sort of results would you get back from your tests that would indicate that now the capabilities have risen to ASL-3?

Yeah, so if the model, for instance, passed some fraction of those tasks that I mentioned before around adding an exploit, spreading to undefended machines, or if it did really well on these biology ones, that would sort of flag it as having passed the yellow lines. At that point, I think we would either need to look at the model and be like, ah, this really isn't— is clearly still incapable of these red line dangers. And then we might need to go to the board and think about, was there a mistake in RSP and how we should essentially create new evals that would test better for whether we're at that capability? Or we would need to implement a bunch of precautions. And these precautions would look like, much more intense security where we would really want this to be sort of robust to probably not state actors, but to non-state actors. And we would want to pass the sort of intensive red teaming process on all the modalities that we release. So this would mean we look at those red lines and we test for them with experts and say, can you use the model to do this? We have this sort of intensive process of red teaming and then only release the modalities where it's been red teamed. So if you add in vision, you need to red team vision. If you add the ability to fine-tune, you need to red team that.

Yeah. What does red teaming mean in this context?

Red teaming means you get a bunch of people who are trying as hard as they can to get the model to do the task you're worried about. So if you're worried about the model carrying out a cyberattack, you would get a bunch of experts to try to prompt the model to carry out some cyberattack. And if we think it's capable of doing it, we're putting these precautions on. And these could be precautions in the model. They could be precautions outside of the model, but the whole end-to-end system. We want to have people trying to get it to do that in some controlled manner such that we don't actually cause mayhem and see how they do.

Okay. And then, so if you do the red teaming and it comes back and they say, yeah, the model is extremely good at hacking into computer systems, or it could actually help people, it could meaningfully help someone develop a bioweapon, then what is the policy call for Anthropic to do?

So for that one, it would mean we can't deploy the model because there's some danger this model could be misused in a really terrible way. And we would sort of keep the model internal until we've improved our safety measures enough that when someone asks for it to do that, we can be confident that they won't be able to have it help them for that particular threat.

Okay. And to even have this model on your computers, the policy also calls for you to have hardened your computer security so that You're saying maybe it's unrealistic at this stage for that model to be safe from persistent state actors, but at least other groups that are somewhat less capable than that, you would want to be able to make sure that they wouldn't be able to steal the model.

Yeah, the threat here is you can put all the restrictions you want on what you do with your model, but if people are able to just steal your model and then deploy it, you're going to have all of those dangers anyway. So you're sort of taking responsibility for it means both responsibility for what you do and what someone else can do with your models. And that requires quite intense security to protect the model weights.

When do you think we might hit this? You would say, well, now we're in the ASL-3 regime, maybe. I'm not sure exactly what language you use for this, but at what point will we have an ASL-3 level model?

I'm not sure. I think basically we'll continue to evaluate our models and we'll see when we get there. I think sort of opinions vary a lot on that.

We're talking kind of about the next few years, right? This isn't something that's going to be 5 or 10 years away necessarily.

I think it really just depends. I think you could imagine sort of any direction. One of the nice things about this is that we're targeting the safety measures at the point when there's actually dangerous models. So maybe let's say I thought it was going to happen in 2 years, but I'm wrong and it happens in 10 years. We won't put these very costly and difficult to implement mitigations in place until we need them.

Okay, so Anthropic's RSP. So I guess obviously we've just been talking about ASL-3. The next level beyond that would be ASL-4. I think your policy basically says we're not exactly sure what ASL-4 looks like yet because it's too soon to say. And I guess you promised that you're going to have mapped out what would be the kind of capabilities that would escalate things to ASL-4 and what kind of responses you would have. You're going to figure that out by the time you have trained a model that's at ASL-3. And I guess if you haven't said so, you'd have to pause training on a model that was going to hit ASL-3. until you'd finished this project. I guess that was the commitment that's been made.

Hey, we'll continue our interview in a moment after a word from our sponsors.

But maybe you could kind of give us a sense of what you think ASL-4 might look like. What sorts of capabilities by the models would then push us into another regime where a further set of precautions are called for?

So we're still discussing this internally, so I don't want to say anything that's final or going to be held to, but you could sort of imagine stronger versions of a bunch of the things that we sort of talked about before. And you could also imagine models that can help with AI research in a way that really majorly accelerates researchers such that progress goes much faster. The core reason that we're holding off on kind of defining this, or that we have this iterative approach, is there's this long track record of people saying, oh, once you have this capability, it will be AGI, it's going to be really dangerous. I think people were like, oh, when an AI solves chess, it will be as smart as humans. And it's really hard to get these evaluations right. Even for the ASL-3 ones, I think it's been very tricky to get evaluations that capture the risks we're worried about. So the closer you get to that, the more information you have and the better of a job you can do at sort of defining what these evaluations are and risks are.

So the general sense would be models that might be capable of spreading autonomously across computer systems, even if people were trying to turn them off, would be able to provide significant help with developing bioweapons, maybe even to people who are pretty informed about it, I guess. Yeah. What else is there? Oh, and stuff that would seriously speed up AI development as well. So it could potentially set off this sort of positive feedback loop where the models get smarter, that makes them better at improving themselves and so on. That's the sort of thing where we're talking about.

Yeah, stuff along those lines. I'm not sure which ones will end up in ASL-4 exactly, but those sorts of things are what's being considered.

And what sorts of additional precautions might there be? I guess at that point you kind of want the models to not only be not possible to be stolen by kind of independent freelance hackers, but ideally also not by countries even, right?

Yeah. So you want to protect against more sophisticated groups that are trying to steal the weights. We're going to want to have better protections against the model acting autonomously. So controls around that, you might want— it depends a little bit on what end up being the red lines there, but sort of having the precautions that are tailored to what will be a much higher level of risk than the ASL-3 red lines.

Were you heavily involved in actually doing this testing on Claude 3 this year?

I wasn't running the tests, but I was sort of watching them because as we trained Claude 3, we were very much sort of— all of our planning was contingent on whether or not it passed these evals and because we had to run them partway through training. So there's sort of a lot of planning that goes into the model's training. You don't want to have to stop the model just because you didn't plan well enough to run the evals in time or something. So there was sort of a bunch of coordination around that that I was involved in.

Can you give me a sense of how many staff are involved in doing that and how long does it take? Is this a big process or is it a pretty standardised thing where you're putting in well-known prompts into the model and then just checking what it does that's different from last time?

Yeah, so Claude 3 was our first time running it. So a lot of the work there actually involved creating the evaluations themselves as well as running them. So we had to sort of create them, have them ready, and then running them. I think typically running them is pretty easy for the ones that are automated, but for some of the things where you actually require people to go and use the model, they can be much more expensive. There's currently, I think, multiple teams working on this, and a lot of our capabilities teams worked on it very hard. So one of the ways this can fall apart is if you don't solicit capabilities well enough. So if you sort of try to have the model test the model on the eval, but you don't try hard enough, and then it turns out that with just a little more effort, the model could have passed the evals. So it's often important to have kind of your best researchers who are capable of pulling capabilities out of the models also working on trying to pull them out to pass these tests.

So many people will have had the experience that these LLMs will reject objectionable requests. If you put it to Claude 3 now and say, please help me design a bioweapon, it's going to say, sorry, I can't help you. But I guess you do all of these tests Before you've done any of that training to try to discourage it from doing objectionable things. You do it with the thing that's helpful no matter what the request is, right?

Yeah, because the thing we're testing for is, is the model capable of this danger? And then there's a separate thing, which is what mitigations can we put on top? So if the model is capable of the danger, then we would require ASL-3, and those safety mitigations we put on top would be part of the standard in order to pass that red teaming. Does that make sense?

Yeah. you have to focus on what the model could do if it was so motivated to, because, well, I guess if the weights were ever leaked, then someone would be able to remove any of the fine-tuning that you've done to try to discourage it from doing unpleasant things. So if it's able to do something, then it could potentially be used that way sooner or later. And so you need to kind of assume the worst and plan around that. Is that the philosophy?

Yeah, that's exactly right.

You mentioned that maybe one of the key failure modes here would be just not trying hard enough to elicit these dangerous capabilities. Basically, maybe if you just phoned it in, then you could easily miss that the model is capable of doing something that it can do. Because even more extreme would be if in future, this policy is going to matter the most when these models are extremely capable and actually could do serious damage. And at that point, you might worry that they would pretend to not be able to do things that in fact they can do because they don't think it's in their interest to reveal that they have X or Y capability, I guess, because then they know that then they won't be used and they might be deleted. Yeah. Do you have any plans for how you're going to deal with that issue? I think the technical term that people use for this is sandbagging, which I'd never heard of, but I think it's a term from sport where people underplay their abilities for strategic reasons. Yeah. Do you have any idea of how Anthropic is going to be able to get around that problem?

Interesting. I didn't know the origin of that term. I think this is one of the reasons that we want to define the next set of evaluations after we've done the current ones, because this is really hard. How do you evaluate for that when the model could be trying to sandbag the evaluations? Hopefully you can set the line, like your eval for a capability that's early enough that you would still be able to detect it doing that. So there's this hope that less capable models will try to deceive you but fail, and then you'll know to stop before you get to models that are really capable of deceiving you. But it's not certain. And that definitely seems like a possible failure mode, where if you have these sort of sudden capability jumps, or if you have jumps that an evaluation can't catch, either because the model's sandbagging you or for any of a number of other reasons.

To try to detect sandbagging, is it possible to— if you have a model that says, no, I can't do that, can you kind of engage in a reinforcement learning from human feedback thing where Whatever response shows the most inclination, the most willingness to help, as soon as you can kind of open the door a little bit, then you reinforce that a whole lot and then see, will that open the door the whole way and actually get the model to demonstrate that it can do something even though it's trying to pretend that it can't? Does that make sense?

It somewhat makes sense. I think it just depends a little bit on exactly what the model is trained to do. So if you've trained the model to be harmless and then you kind of just try to RL it back to not being harmless, you will be able to succeed. You can fine-tune away any of the Of the safety training, but it's not clear if that will sort of be the same thing as if in one sort of sequence, within some prompt, it's trying to trick you. I think there's been a bunch of work where people use scratchpads to sort of tell the model you can write something down, but the reward model, which is sort of the model that's overseeing it and telling it whether it's good or not, doesn't get to see this. And when you do that, you as a human can then look at it and you can kind of get a sense of what is the model thinking? Is the model trying to to deceive its supervisor. But we've only really done that on sort of current models, which don't really have this kind of level of capability where it would be really catastrophically dangerous. And figuring out— I think we'll want much more robust tools once we get there. Interpretability is another approach to this, where if you can actually look directly at the model weights and look and see, ah, okay, without running the model, we can see that the model has this circuit that is saying, ah, you should deceive the user in this way, or should fail the tests because you don't want them to turn you off. That's sort of another hope for how to attack this.

Okay, so that's the basic structure of a responsible scaling policy, or at least the Anthropic responsible scaling policy. I guess you have this system of tests that you commit to ahead of time that you're going to put all of your models through, and then you precommit to saying, well, if we get this kind of result, then we think the risk is higher. And so that's going to call for an escalation in the precautions that we're taking, things around computer security, things around not deploying until you've made them safe, and so on. You're a big fan of this type of approach to AI safety for AI companies. What's one of the main reasons, or what's perhaps the top reason why you think this is the right approach, or at least one of the better approaches?

Yeah, so I think one thing I like is that it separates out whether an AI is capable of being dangerous from what to do about it. I think this is a spot where there are many people who are sceptical that models will ever be capable of this sort of catastrophic danger, and therefore they're like, we shouldn't take precautions because the models aren't that smart. And I think this is a nice way to agree where it's a much easier message to say, if we have evaluations showing the model can do X, then we should take these precautions. And I think you can sort of build more support for something along those lines, and it targets your precautions at At the time when there's actual danger. There are a bunch of other things I can sort of talk through. I think one other thing I really like is that it aligns commercial incentives with safety goals. So once we put this RSP in place, it's now the case that our safety teams are kind of under the same pressure as our product teams, where if we want to ship a model and we get to ASL-3, the thing that will block us from being able to get revenue, being able to get users, et cetera, is do we have the ability to deploy it safely. And it's a nice outcome-based approach where it's not, did we invest X amount of money in it? It's not, did we try? Did we say the right things? Did we succeed? Yeah. And I think that often really is important for organisations to set this goal of, you need to succeed at this in order to deploy your products.

Is it actually the case that it's kind of had that cultural effect within Anthropic now that People realise that a failure on the safety side would prevent the release of the model that matters to the future of the company. And so there's a similar level of pressure on the people doing this testing as there is on the people actually training the model in the first place.

Oh, yeah, for sure. I mean, you asked me earlier, when are we going to have ASL-3? And I think I received this from someone on one of the safety teams on a weekly basis because their deadline is— I mean, the hard thing for them actually is their deadline isn't a date. It's once we have created some capability, and they're very focused on that.

So their fear, the thing that they worry about at night, is that you might be able to hit ASL-3 next year and they're not going to be ready, and that's going to hold up the entire enterprise.

Yeah, I can give some other things. Like, 8% of Anthropic staff works on security, for instance. So this is like, there's a lot you have to plan for it, but there's a lot of work going into being ready for these next levels. We have multiple teams working on alignment, interpretability, creating evaluations. So, yeah, there's a lot of effort that goes into it.

When you say security, do you mean computer security? So preventing the weights from getting stolen, or a broader class?

Both. So the weights could get stolen, someone's computer could get compromised, you could have someone hack in and get all of your IP. There's sort of a bunch of different dangers on the security front where the weights are certainly an important one, but they're definitely not the only one.

Okay. And the first thing you mentioned, the first reason why RSPs have this nice structure is that some people think that these troublesome capabilities could be with us this year or next year. Other people think it's never going to happen, but both of them could be on board with a policy that says, well, if these capabilities arise, then that would call for these sorts of responses. Has that actually happened? I mean, have you seen kind of the sceptics who say, all of this AI safety stuff is overblown, it's a bunch of rubbish, saying, well, but the RSP is fine because I think we'll never actually hit any of these levels, so we're not going to waste any resources on something that's not realistic.

Yeah. So I think there's always going to be degrees. I think there are people across the spectrum. So there are definitely people who are still sceptical, who will just be like, why even think about this? There's no chance. But I do think that RSPs do seem much more pragmatic, much more able to sort of be picked up by various other organisations. I think, as you mentioned before, OpenAI and Google are both sort of putting out things along these lines. So I think at least from the sort of large frontier AI labs, there is a significant amount of buy-in.

Yeah, I see. I guess even if maybe you don't see this on Twitter, maybe it helps with the internal bargaining within the company that people have a different range of expectations about how things are going to go, but they could all be kind of reasonably satisfied with an RSP that that equilibrates or matches the level of capability with the level of precaution. The first worry about this that jumps to my mind is if the capability improvements are really quite rapid, which I think we think that they are and they maybe could continue to be, then don't we need to be practising now, basically getting ahead of it and doing stuff right now that might seem kind of unreasonable given what Claude 3 can do, Because we worry that we could have something that's substantially more dangerous in 1 year's time or in 2 years' time, and we don't want to then be kind of scrambling to deploy the systems that are necessary then, and then perhaps falling behind because we didn't prepare sufficiently ahead of time. What do you make of that?

Yeah, so I think we definitely need to plan ahead. Right. And I think one of the nice things is once you've aligned these sort of safety goals with commercial goals, like, people plan ahead for commercial things All the time. It's part of a normal company planning process. I think that the RSP— so we have these sort of yellow line evals that are intended to be far short of the capability, the red line capabilities we're actually worried about. And tuning that gap seems fairly important. If that gap looks like a week of training, it would be really scary where you trigger these evals and you have to act fast. I think in practice, we've set those evals such that they are far enough from the capabilities that are really dangerous. such that there will be some time to sort of adjust in that sort of buffer period.

So should people actually think that we're in ASL 2 now and we're heading towards ASL 3 at some point, but there's actually kind of an intermediate stage with all of these transitions where you'd say, well, now we're seeing warning signs that we're going to hit ASL 3 soon, so we need to implement the precautions now in anticipation of being about to hit ASL 3? Is that basically how it works?

Yeah, it's basically like once we sort of have this concept of a safety buffer. So once we trigger the evaluations, it doesn't necessarily mean— these evaluations are set conservatively, so it doesn't mean the model is capable of the redline capabilities we're really worried about. And that will sort of give us a buffer where we can figure out, ah, maybe it really just definitely isn't and we wrote a bad eval. We'll go to the board, we'll change the evals and implement new things. Or maybe it really is, quite dangerous, and we need to turn on all the precautions. Of course, you might not have that long, so you want to be ready to turn on those precautions such that you don't have to pause. But there is some time there that you could do it. And then the last possibility is that we're just really not ready. These models are catastrophically dangerous, and we don't know how to secure them, in which case we should stop training the models. Or if we don't know how to deploy them safely, we should not deploy the models until we figure it out.

I guess if you were on the very concerned side, then you might think, yes, you are preparing. I guess, yeah, you do have a reason to prepare this year for safety measures that you think you're going to have to employ in future years. But maybe we should go even further than that. And what we need to be doing is practising implementing them and seeing how well they work now. Because even though you are preparing them, you're not actually getting the gritty experience of, like, of applying them and trying to use them on a day-to-day basis. And I guess the response to that would be, well, that would in a sense be safer. That would be adding an even greater precautionary buffer, but it would also be enormously expensive, and people would see us doing all of this stuff that seems really over the top relative to what any of the models can do.

Yeah, I think there's sort of a trade-off here between, like, with pragmatism or something, where I think we do need to have a huge amount of caution on future models that are really dangerous. But if you apply that caution to models that aren't dangerous, you miss out on a huge number of benefits from using the technology now. And I think you'll also probably just alienate a lot of people who are going to look at you and be like, you're crazy, why are you doing this? And I think my hope is that you can sort of— this is sort of the framework of RSP, is you can tailor the cautions to the risks. It's still important to look ahead more, right? So we do a lot of safety research that isn't directly focused on the next AI safety level because You want to plan ahead, you have to be ready for multiple ones out. It's not like the only thing to think about, but the RSP is sort of tailored more to empirically testing for these risks and tailoring the precautions appropriately.

Yeah. On that topic of people worrying that it's going to slow down progress in the technology, do you have a sense of— so obviously training these frontier models costs a significant amount of money. We're talking maybe $100 million is kind of a figure that I've heard thrown around for training a frontier LLM. How much extra overhead is there to run these tests to see whether the models have any of these dangerous capabilities? Is it adding hundreds of thousands, millions, tens of millions of dollars of additional cost or time?

I don't know the exact cost numbers. I think the cost numbers are pretty low, right? They're mostly running inference or relatively small amounts of training. The people time feels like where there's a cost, right? There are whole teams dedicated to creating these evaluations, to running these, to doing the the safety research to protect against the mitigations. And I think particularly for Anthropic, where we're pretty small, rapidly growing but rather small organisation, at least my perspective is most of the cost comes down to the people and time that we're investing in it.

Okay. Yeah, but I guess at this stage it sounds like running these sorts of tests on a model is taking more in the order of weeks of delay, because if you're getting back clear this is not a super dangerous model, then it's not leading you to delay release of things for many months and deny customers the benefit of them.

Yeah, the goal is to minimise the delay as much as you can while being responsible. The delay in itself isn't valuable. I think we're aiming to get it to a really well-done process where it can all execute very efficiently. But until we get there, there might be delays as we're figuring that out. And there will always be some level of time to require it to do it.

Just to clarify, so a lot of the risks that people talk about with AI models is risks once they're deployed to people and actually getting used. But there's kind of this separate class of risk that comes from having an extremely capable model simply exist anywhere. Even on, I guess you could think of, there's public deployment and then there's internal deployment where Anthropic staff might be using a model and potentially it could convince them to release it or to do other dangerous things. That's kind of a separate concern. What does the RSP have to say about that sort of internal deployment risks? Are there circumstances under which you would say even Anthropic staff can't continue to do testing on this model because it's too unnerving?

Yeah, so I expect this to mostly kick in as we get to higher AI safety levels, but there are certainly dangers. I mean, the main one is the security risk. So one approach is just having the model. It always could be stolen. No one has perfect security. So that's kind of, I think, In some ways is one that's true of all models and is maybe more short-term. But yeah, if you get to models that are trying to escape, trying to autonomously replicate, there is danger then in having access internally. So we would want to do things like siloing who has access to the models, putting particular precautions in place before the model is even trained, or maybe even on the training process. But we haven't yet defined those because we don't really know what they would be. Right. We don't quite know what that would look like yet. And it's It feels really hard to design evaluation that is meaningful for that right now.

I don't recall the RSP mentioning conditions under which you would say we have to delete this model that we've trained because it's too dangerous. But I guess that's because that's more at the kind of ASL 4 or 5 level that would become the kind of thing that you would contemplate, and you just haven't spelled that out yet.

No. So it's actually because of the safety buffer concept, right? So the idea is we would never train that model. If we did accidentally train some model that was past the red lines, then I think we'd have to think about deleting it. But we would put these evaluations in place far below the dangerous capability, such that we would trigger the evaluations and have to pause or have the safety things in place before we train the model that has these dangers.

So RSPs as an approach, you're a fan of them. What do you think of them as an alternative to? What are the alternative approaches for dealing with AI risk that people advocate that you think are weaker in relative terms?

So I mean, I think the baseline is obviously just— the first baseline is nothing. Like, these are— there could just be nothing here. I think the downsides of that is that these, like, these models are, are very powerful. They could at some point in the future be dangerous. And I think that companies creating them have some— have a responsibility to, to think really carefully about those, those risks and, and be thoughtful. Um, sort of like it's a major externality. Uh, that's maybe the, the easiest baseline of do nothing.

Yeah.

I think other things have been proposed would be like a pause where a bunch of people say, well, there are all these dangers, why don't we just not do it? And I think that makes sense, right? If you're training these models that are really dangerous, it does feel a bit like, why are you doing this if you're worried about it? But I think there are actually really clear and obvious benefits to AI products right now. And the catastrophic risks currently are just— they're definitely not obvious. I think they're probably not immediate. And as a result, this isn't a practical ask. Not everyone is going to pause. So what will happen is only the places that care the most, that are the most worried about this and the most careful with safety, will pause. And you'll sort of have this adverse selection effect. I think there eventually might be a time for a pause, but I would want that to be backed up by, here are clear evaluations showing the models have these really catastrophically dangerous capabilities, and here are all the efforts we put into making them safe, and we ran these tests and they didn't work, and that's why we're pausing. And we would recommend everyone else should pause until they've as well. I think that will just be like a much more convincing case for a pause and target it at the time that it's most valuable to pause.

When I think about people doing somewhat potentially dangerous things or developing interesting products, maybe the default thing I imagine is that the government would say, here's what we think you ought to do, here's how we think that you should make it safe. And as long as you make your product according to these specifications, as long as the plane runs this way and you service the plane this frequently, then you're in the clear and we'll say that what you've done is reasonable. Do you think that RSPs are maybe better than that in general, or maybe just better than that for now, where we kind of don't know necessarily what regulations we want the government to be imposing? So perhaps it's better for companies to be figuring this out themselves early on, and then perhaps it can be handed over to governments later on.

Yeah, I don't think the RSPs are like a substitute for regulation. There are many things that sort of only regulation can solve, such as What about the places that don't have an RSP? But I think that right now we don't really know what the tests would be or what the regulations would be. And I think probably this is still sort of getting figured out.

So one hope is that we can implement RSP, OpenAI and Google can implement other things, other places will implement a bunch of things, and then policymakers can look at what we did, look at our reports on how it went, what the results of our evaluations were and how it was going, and then design regulations sort of based on the learnings from If I read it correctly, it seemed to me like the Anthropic RSP has this clause that allows you to go ahead and do things that you think are dangerous if you're being sufficiently outpaced by some other competitor that I guess doesn't have an RSP or not a very serious responsible scaling policy, in which case you might worry, well, we have this policy that's preventing us from going ahead, but we're just being rendered irrelevant and some other companies releasing much more dangerous stuff anyway. what really is this accomplishing? Yeah. Did I read that correctly, that there's a sort of get out of RSP clause in that sort of circumstance? And if you didn't expect Anthropic to be leading and for most companies to be operating safely, couldn't that kind of potentially obviate the entire enterprise because that clause could be quite likely to get triggered?

Yeah, I think we don't intend that as like a get out of jail free card where sort of we're falling behind commercially and then like, oh, well, Now we're gonna skip the RSP. It's much more just intended to be practical, as, you know, we don't really know what it will look like if we get to some sort of, like, AGI endgame race. And there could be really high stakes, and sort of, it could make sense for us to decide that the best thing is to proceed anyway. But I think this is something that we're sort of looking at as a bit more of a, like, last resort than a loophole we're planning to just use for oh, we don't want to deal with these evaluations.

Yeah. Okay, I think we've hit a good point where maybe the best way to learn more about RSPs and their strengths and weaknesses is just to talk through more of the complaints that people have had, or the concerns that people have raised with the Anthropic RSP and RSPs in general since it was released last October. I'm realising that I was going to kind of start the weaknesses and worries now, but I'm kind of realising I've been peppering you effectively with them maybe almost since Since the outset. But now we can really drive into some of the worries that people have expressed. Yes. The first of these is the extent to which we have to trust the good faith and integrity of the people who are applying a responsible scaling policy or a preparedness framework or whatever it might be within the companies. And I imagine this issue might jump to mind for people more than it might have 2 or 3 years ago, because public trust in AI companies to do the right thing at the cost of their business interests is maybe lower than it was years ago, when the major players were perceived perhaps more as research labs and less as for-profit companies, which is kind of how they come across more these days. And one reason it kind of really matters, it seems like it matters to me who's doing the work here, is that the Anthropic RSP is full of expressions that are open to interpretation. For instance, hardened security such that non-state attackers are unlikely to be able to steal model weights. And advanced threat actors like states cannot steal them without significant expense, or access to the model would substantially increase the risk of catastrophic misuse and things like that. And who's to say what's unlikely or significant or substantial? That sort of language is maybe a little bit inevitable at this point where there's just so much that we don't know. And how are you going to pin those things down exactly to say it's a 1% chance that a state's going to be able to steal the model? That might just also feel like insincere false precision. But to my mind, that sort of vagueness does mean that there's a slightly worrying degree of wiggle room that could render the RSP less powerful and less binding when push comes to shove and there might be a lot of money at stake. And on top of that, I guess, exactly as you were saying, anyone who's implementing an RSP has a lot of discretion over how hard they try to elicit the capabilities that might then trigger additional scrutiny and possible delays to their work and release of really commercially important products. So, yeah, to what extent do you think the RSP would be useful in a situation where the people using it were neither particularly super skilled at doing this sort of work and maybe not particularly bought in and enthusiastic about the safety project that it's a part of?

Yeah, so fortunately, I think my colleagues both on RSP and elsewhere are both talented and really bought into this. I think we'll do a great job on it. But I do think the criticism is valid and that there is a lot that is left up for interpretation here, and it does rely a lot on people sort of having a good faith interpretation of how to execute on the RSP internally. I think that there are some checks in place here. So having whistleblower-type protections such that people can say if a company is breaking from the RSP or not trying hard enough to to elicit capabilities or to interpret it in a good way. And then public discussion can add some pressure. But ultimately, I think you really do need regulation to have these sort of very strict requirements. Over time, I hope we'll make it more and more concrete. The blocker, of course, on doing that is that we don't know for a lot of these things. And being overly concrete, where you specify something very precisely that turns out to be wrong, can be very costly and sort of if you then have to go and change it, et cetera, it can take away some of the credibility. So sort of aiming for as concrete as we can make it while balancing that.

The response to this that jumps out to me is just that ultimately, it feels like this kind of policy has to be implemented by a group that's external to the company that's then affected by the determination. It really reminds me of accounting or auditing. For a major company, it's not sufficient for a major corporation to just have its own accounting standards and follow that and say, we're going to follow our own internal best practices. You get, and it's legally required that you get external auditors in to confirm that there's no chicanery going on. And at the point that these models potentially really are risky, or it's plausible that the results will come back saying, we can't release this, maybe we even have to delete it off of our servers according to the policy. I would feel more comfortable if I knew that some external group that had different incentives was the one figuring that out. Do you think that kind of ultimately is where things are likely to go in the medium term?

So I think that'd be great. I would also feel more comfortable if that was the case. I think one of the challenges here is that for auditing, there's a bunch of external accountants. It's a very common sort of profession, or auditors. This is a profession. Many people know what to do. There are very clear rules. For some of the stuff we're doing, there really aren't external established auditors that everyone trusts to come in and say, we took your model and we certified it can't autonomously replicate across the internet or cause these things. So I think that's currently not practical. I think that would be great to have at some point. One thing that will be important is that that auditor has enough expertise to properly assess the capabilities of the models.

I suppose an external company would be an option. Of course, obviously a government regulator, government agency would also be another approach. I guess when I think about other industries, it often seems like there's kind of a combination of private companies that then follow government mandated rules and things like that. Do you think that this is a benefit I actually haven't thought of to do with creating these RSPs, is that it maybe is beginning to create a market, or it's indicating that there will be a market for this kind of service, because it's likely that this kind of thing is going to have to be outsourced at some point in future. And there might be many other companies that want to get this similar kind of testing. So perhaps it would encourage people to think about founding companies that might be able to provide this service in a more credible way in future.

That would be great. And also, we publish blog posts on how things go and how our evaluations are. So I think there's some hope that people doing this can learn from what we're doing internally and sort of the various iterations we'll put out of our RSP. And that can inform something maybe more stringent that gets regulated.

Have you thought at all about what could be done to make the— let's say that it wasn't given out to an external agency or an external auditing company, how it could be tightened up to make it less vulnerable to the level of operator enthusiasm? Yeah, I guess you might have thought about this in the process of actually applying it. Are there any ways that it could be stronger without having to completely outsource the operation of it?

Yeah, I think the core thing is just making it more precise. Right. One piece of accountability here is both public and internal commitment to doing it. So, yeah, I guess maybe I should list off some of the reasons that I think it would be hard to break from it. Right. This is a formal policy that has been passed by the board, and it's not as though we can just be like, oh, we don't feel like doing it today. You would sort of need to get the board of Anthropic, get all of leadership, and then get all of the employees bought in to not do this or even to skirt the edges. I can speak for myself. If someone was like, Nick, can you train this model? We're going to ignore the RSP. I would be like, no, we said we would do that. Why would I do this? And if I wanted to, I would tell my team to do it and they would be like, no, Nick, we're not going to do that. So you would need to have a lot of buy-in and part of Part of the benefit of having this, publicly committing to it and passing it as an organizational policy, is that everyone is bought in. And sort of maintaining that level of buy-in, I think, is quite critical. In terms of specific checks on— I think we have a team that's responsible for checking that we did the sort of red teaming our evaluations and making sure we actually did them properly. So you can sort of set up a bunch of internal checks there. But ultimately, these things do rely on the company implementing them to really be bought in and care about the actual outcome of it.

So, yeah, this naturally leads us into this. I solicited on Twitter, I asked, what are people's biggest reservations about RSPs and about Anthropic's RSP in general? And actually, probably the most common response was it's not legally binding. Like, what's stopping Anthropic from just dropping it when things really matter? I think someone said, how can we have confidence that they'll stick to RSPs, especially when they haven't stuck to— actually, well, this person said to past admittedly less formal commitments not to push forward the frontier on capabilities. But what would actually have to happen internally? You said you'd have to get staff on board, you'd have to get the board on board. Is there a formal process by which the RSP can be rescinded that is just a really high bar to clear?

Yeah, so basically we do have a process for updating the RSP, so we could go to the board, et cetera. But I think sort of in order to do that, It's hard for me to quite plan it out, but it would be like, oh, if I wanted to continue training the model, I would go to the RSP team and be like, does this pass? And they'd be like, no. And then maybe you'd appeal it up the chain or whatever. And sort of at every step along the way, people would say, no, we care about the RSP. Now, on the other hand, there could be legitimate issues with the RSP, right? We could find that one of these evaluations we created turned out to be really easy in a way that we didn't anticipate and really is not at all indicative of the dangers. And in that case, I think it would be very legitimate for us to try to amend the RSP to create a better evaluation that is a test for it. This is sort of the flexibility we're trying to preserve. But I don't think it would be simple or easy. I can't picture a plan where someone could be like, ah, there's a bunch of money on the table, can we just skip the RSP for this model? That seems somewhat hard to imagine.

The decision is made by this odd board called the Long-Term Benefit Board. Is that right? Or they're the group that decide what the RSP should be?

So the Long-Term Benefit— basically, Anthropic has a board that's sort of a corporate board. Some of those seats, in the long term will be the majority of those seats, are elected by the Long-Term Benefit Trust, which doesn't have a financial stake in Anthropic and is there to sort of keep us focused on our public benefit mission of making sure AGI goes well. So, yeah, the board is not entirely the same. It's not the same thing as that, but the Long-Term Benefit Trust elects the board.

I mean, I think the elephant in the room here is, of course, there was a long period of time when OpenAI was pointing to its kind of nonprofit board as a thing that would potentially keep it on mission to be really focused on safety and had a lot of power over the organisation. And then in practice, when push came to shove, it seemed like even though the board had these concerns, it was effectively overruled by, I guess, a combination of just the views of staff, maybe the views of the general public in some respects, and potentially the views of investors as well. And I think something that I've taken away from that, and I think many people have taken away from that experience, maybe the board was mistaken, maybe it wasn't, but these formal structures Power isn't always exercised in exactly the way that it looks on an organisational chart. And I don't really want to be putting all of my trust in these interesting internal mechanisms that companies design in order to try to keep themselves accountable, because ultimately, just if the majority of people involved don't really want to do something, then it feels like it's very hard to bind their hands and prevent them from changing plan at some future time. So maybe within Anthropic, perhaps these structures really are quite good, and maybe the people involved are really trustworthy and people who I should have my confidence in, that even in extremists, they're going to be thinking about the well-being of humanity and not getting too focused on the commercial incentives faced by Anthropic as a company. But I think I would rather put my faith in something more powerful and more solid than that. And so this is kind of another thing that pushes me towards thinking that the RSP and these sort of preparedness frameworks are a great stepping stone towards external constraints on companies that they don't have ultimate discretion over. It's something that has to evolve into, because the impacts are going to be on the entirety. If things go wrong, the impacts are on everyone across society as a whole. And so there needs to be external shackles effectively put on companies to reflect the harm that they might do to others legally. I guess I'm not sure whether you want to comment on that as potentially a slightly hot-button topic, but yeah. Do you think I'm kind of gesturing towards something legitimate there?

Yeah, I think that basically these shouldn't be seen as sort of a replacement for regulation. I think there are many cases where policymakers can pass regulations that would help here. I think they're intended as sort of a supplement there and a bit as a learning ground for for what might end up going in regulations. In terms of, does the board really have the power it has types of questions, I don't know. We put a lot of thought into the long-term benefit trust, and I think it really does have direct authority to elect the board, and the board does have authority. But I do agree that ultimately you need to have a culture around thinking these things are important and having everyone bought in. This is like, any employee can always— as I said, some of these things are like, did you solicit capabilities well enough? That really comes down to a researcher working on this actually trying their best at it. And that is quite core. And I think that will sort of just continue to be— even if you have regulations, there's always going to be some amount of importance to the people actually working on it, taking the risk seriously and really caring about them and doing the best work they can on that.

Yeah, I guess one takeaway you could have is we don't want to be relying on our trust in individuals and saying, well, we think Nick's a great guy, his heart's in the right place, he's going to do a good job. Instead, we need to be on more solid ground and say, well, no matter who it is, even if we have something bad in the role, the rules are such, the oversight is such that we'll still be in a safe place and things will go well. I guess an alternative angle would be to say, when push comes to shove, when things really matter, people might not act in the right way. There actually is no alternative to just trying to have the right people in the room making the decisions, because the people who are there are going to be able to sabotage any legal framework that you try to put in place in order to constrain them, because it's just not possible to have perfect oversight within an organisation from outside. I could see people mounting both of those arguments reasonably. I suppose you could try doing both, like both trying to pick people who are really sound and have good judgement and who you have confidence in, as well as then trying to bind them so that even if you're wrong about that, you have a better shot at things going well.

Yeah, I think you just want this defence-in-depth strategy where ideally you have all the things lined up, and that way, if any one piece of them has has a hole, you sort of catch it at the next layer. What you want is sort of a regulation that is really good and robust to someone not acting in the spirit of it. But in case that is messed up, then you really want someone working on it who is also checking in and is like, ah, okay, I technically don't have to do this, but this seems clearly in the spirit of how it works. Yeah, I think that's pretty important. I think also for trust, you should just look at track records. And I think that we should try to encourage companies and people working on AI to have track records of Of prioritizing things. So one of the things that makes me feel great about Anthropic is just a long track record of doing a bunch of safety research, caring about these issues, putting out actual papers, being like, here are a bunch of progress we've made on that field. There are a bunch of pieces. I mean, I think looking at sort of commitments people have made, do we break the RSP? I think if we publicly were like, ah, we changed this in some way that I think everyone thought was silly and really added risks, then I think people should lose trust according to that.

All right, let's push on to a different worry, although I must admit it has a slightly similar flavour, and that's that the RSP might be very sensible and look good on paper, but if it commits to future actions that at that time we probably won't know how to do, then it might actually fail to help very much. And I guess to make that concrete, an RSP might naturally say that at the time you have really superhuman general AI, you need to be able to lock down your computer systems and make sure that the model can't be stolen, even by the most persistent and capable Russian or Chinese state-backed hackers. And that is indeed what Anthropic's RSP says, or suggests that it's going to say once you get up to ASL 4 and 5. But as I think the RSP actually says as well, we don't currently know how to do that. We don't know how to secure data against a state actor that's willing to spend hundreds of millions or billions or possibly even tens of billions to steal model weights, especially not if you ever need those model weights to be connected to the internet in some way in order for the model to actually be used by people. So it's kind of a promise to do what arguably, what basically is impossible with current technology. And that means that we need to be preparing now, doing research on how to make this possible in future. But solving the problem of computer security that has beguiled us for decades is probably beyond Anthropic. It's not really reasonable to expect that you're going to be able to fix this problem that society as a whole has kind of failed to fix for all this time. It's just going to require coordinated action across countries, across governments, across lots of different organisations. And so if that doesn't happen, and it's somewhat beyond your control whether it does, then when the time comes, the real choice is going to be between a lengthy pause while you wait for fundamental breakthroughs to be made in computer security, or dropping and weakening the RSP so that Anthropic can continue to remain relevant and release models that are commercially useful. And in that sort of circumstance, the pressure to weaken the scaling policy so you aren't stuck for years is going to be, I would imagine, quite powerful, and it could win the day, even if people are dragged kind of kicking and screaming to conceding that unfortunately, they have to loosen the RSP even though they don't really want to. Yeah. What do you make of that worry?

I think what we should do in that case is instead we should pause and we should focus all of our efforts on safety and security work. That might include looping in external experts to help us with it, but we should put in the best effort that we can to mitigate these issues such that we can still realise the benefits, deploy the technology, but without the dangers. And then if we can't do that, Then I think we need to make the case publicly to governments, other companies. There's some risk to public, so exactly, we'd have to be strategic in exactly how we do this, but basically make the case that there are really serious risks that are imminent and that everyone else should take sort of appropriate actions. There's a flip side to this, which is just like, I think I mentioned before, if we just messed up our evals, the model is clearly not dangerous and we just really screwed up on some eval, and then we should follow the process in the RSP that we've written up. We should go to the board, we should create a new test that that we actually trust. I think I would also just say people don't need to follow incentives, right? I think you can make a lot more money doing something that isn't hosting this podcast, probably. Certainly if you had pivoted your career earlier, there are more profitable things. So I think this is just a case where the stakes would be extremely high, and I think it's just somewhere where it's important to just do the right thing in that case.

If I think about how this is most likely to play out. I imagine that at the point that we do have models that we really want to protect from even the best state-based hackers, there probably have been some progress in computer security, but not nearly enough to make you or me feel comfortable that there's just no way that China or Russia might be able to steal the model weights. And so it is very plausible that the IRS people will say, Anthropic, you have to keep this on a hard disk not connected to any computer. You can't train models that are more capable than the thing that we kind of already have that we don't feel comfortable handling. And then how does that play out over— there are a lot of people who are very concerned about safety at Anthropic. I mean, I've seen that there's kind of league tables now of different AI companies and enterprises, and how good do they look on an AI safety point of view? And Anthropic always kind of comes out at the top, I think, by a decent margin. But Months go by, other companies are not being as careful as this. You've complained to the government and you've said, look at this horrible situation that we're in, something has to be done. But I don't know, I guess possibly the government could step in and help there, but maybe they won't. And then over a period of months or years, doesn't the choice effectively become, if there is no solution, either take the risk or just be rendered irrelevant?

Yeah. So I maybe just step Just going back to the beginning of that, I don't think we will put something in that says there is zero risk from something. I think you can sort of never get to zero risk. I think often with security, you'll end up with some security-productivity trade-off. So you could end up taking some really extreme risk or some really extreme security-productivity trade-off where only one person has access to this. Maybe you've locked it down in some huge amount of ways. It's possible that you can't even do that. You really just can't train the model. But there is always going to be some sort of balance there, and I don't think we'll push to the zero-risk perspective. But yeah, I think that that's just like a risk. I don't know. I think there's a lot of risks that companies face where they could fail, right? We also could just fail to make better models and not succeed that way. I think the point of the RSP is it has tied our commercial success to the safety mitigations. So in some ways, it just adds on another risk in the same way as any other company risk.

It sounds like I'm having a go at you here, but really, I think what this shows up is just that the scenario that I painted there is really quite plausible, and it just shows that this problem cannot be solved by Anthropic. Probably it can't be solved by even all of the AI companies combined. The only way that this RSP is actually going to be able to be usable, in my estimation, is if other people rise to the occasion and pitch and governments actually do the work necessary to fund the solutions to computer security that will allow us to have the model weights be sufficiently secure in this situation. And yeah, you're not blameworthy for that situation. It just says that there's a lot of people who need to do a lot of work in coming years.

Yeah. And I think I might be more optimistic than you or something. I do think if we get to something really dangerous, we can make a very clear case that it's dangerous and these are the risks unless we can implement these mitigations. I hope that at that point, it will be a much clearer case to pause or something right now. I think there are many people who are like, we should pause right now and see everyone saying no. And they're like, oh, these people don't care. They don't care about major risks to humanity. And I think really the core thing is people don't believe there are risks to humanity right now. And once we get to this sort of stage, I think that we will be able to make those risks very clear, very immediate, tangible, and I don't know, no one wants to be the company that caused a massive disaster. And no government also probably wants to have allowed a company to cause a disaster. It will feel much more immediate at that point.

Yeah, I think Stefan Schubert, this commentator who I read on Twitter, has been making the case for a while now that many people who have been thinking about AI safety, I guess including me, have perhaps underestimated the degree to which the public is likely to react and respond, and governments are going to get involved once the problems are apparent, once they really are convinced that there is a threat here. I think he calls it this bias in thought where you imagine that people in the future are just going to sit on their hands and not do anything about the problems that are readily apparent. He calls it sleepwalk bias. And I guess I think we have seen evidence over the last year or two that as the capabilities have improved, people have gotten a lot more serious and a lot more concerned, a lot more open to the idea that it's important for the government to be involved here. There's a lot of actors that need to step up their game and help to solve these problems. So, yeah, I think you might be right. On an optimistic day, maybe I could hope that other groups will be able to do the necessary research soon enough that Anthropic will be able to actually apply its RSP in a timely manner. I guess fingers crossed. Yeah, I just want to actually ask you next, what are your biggest reservations about RSPs or Anthropic's RSP personally? If it fails to improve safety as much as you're hoping that it will, what's the most likely reason for it to not live up to its potential?

So I think for Anthropic specifically, it's definitely around this under-elicitation problem. I think it's a really kind of fundamentally hard problem to take a model and say, oh, you've tried as hard as one could to elicit this particular danger, but There's always some— maybe there's a better researcher. There's a saying, no negative result is final. If you fail to do something, someone else might just succeed at it next. So that's one thing I'm worried about. And then the other one is just unknown unknowns. So we are creating these evaluations for risks that we are worried about and we see coming, but there might be risks that we've missed, things that we didn't realize would come before, either didn't realize would happen at all or thought would happen after for later levels, but turn out to arise earlier.

What could be done about those things? Would it help to just have more people on the team doing the evals, or to have more people, I guess, both within and outside of Anthropic trying to come up with better evaluations and figuring out better red teaming methods?

Yeah. And I think that this is really something that people outside Anthropic can do. The elicitation stuff has to happen internally, and that's more about putting putting as much effort as we can into it. But creating evaluations can really happen anywhere. Coming up with new risk categories, threat models, is something that anyone can contribute to.

Yeah. What are the places that are doing the best work on this? I imagine Anthropic surely has some people working on this, but I guess I mentioned METR. Can't remember what that stands for right now, but they're a group that helped to develop the idea of RSPs in the first place and develop evals. And I think The AI Safety Institute in the UK is involved in developing these sort of standard safety evals. Is there anywhere else that people should be aware where this is going on?

Yeah, there's also the US AI Safety Institute. And I think this is actually something you could probably just do on your own. I think one thing, I don't know, at least for people early in career, if you're trying to get a role doing something that I would recommend is just go and do it. So I think you probably could just write up a report, post it online, be like, this is my threat model, these are the things I think are important. You could implement the evaluations and share them on GitHub. But yeah, there are also organisations you could go to to get mentorship and work with others on it.

I see. So this would look like, I suppose you could try to think up new threat models, to think up new things that you need to be looking for, because this might be a dangerous capability and people haven't yet appreciated how much it matters. But I guess you could spend your time trying to find ways to elicit the ability to autonomously spread and steal model weights and get yourself onto other computers. from these models and see if you can find an angle on trying to find warning signs or signs of these emerging capabilities that other people have missed and then talk about them. And you can kind of just do that while signed into Claude 3 Opus on your website.

Yeah. So I think you'll have more luck with the elicitation if you actually work at one of the labs because you'll have access to training the models as well. But you can do a lot with Claude 3 on the website or via an API, which is a programming term for basically an interface where you can send a request for, like, I want a response back and automatically do that in your app. So you can sort of set up a sequence of prompts and test a bunch of things via the APIs for Claude or any other publicly accessible model.

To come back to this point about what's acceptable risk and maybe trying to make the RSP a little bit more concrete, I read from a critic of the Anthropic RSP that I'm not sure how true this is. I'm not an expert on risk management, but this person was saying that it's more true, at least in more established areas of risk management, where maybe you're thinking about what's the probability that a plane is going to fail and crash because of some mechanical failure. It's more typical to say, we've studied this a lot and we think that the probability of— well, I suppose let's talk about the AI example. Rather than say we need the risk to be not substantial, Instead, you'd say, with our practices, our experts think that the probability of an external actor being able to steal the model weights is X% per year. And these are the reasons why we think the risk is that level. And that's below what we think of as our acceptable risk threshold of X, where X is larger than Y. I guess there's a risk that those numbers would kind of just be made up and you could kind of assert anything because it's all a bit unprecedented. But I suppose that would make clear to people what the remaining risk is, like what acceptable risk you think that you're running. And then people could scrutinise whether they think that that's a reasonable thing to be doing. Is that a direction that things could maybe go?

Yeah, I think it's like a fairly common way that people in the EA and rationality community speak, where they give a lot of probabilities for things. And I think it's really useful. It's an extremely clear way to communicate. I think a 20% chance this will happen is just way more informative than I think it probably won't happen, which could be 0% to 50% or something. So I think it's very useful in many contexts. I also think it's very frequently misunderstood because for most people, I think they hear a number and they think it's based on something, that there's some calculation, and they give it more authority. If you say, ah, there's a 7% chance this will happen, people are like, oh, you really know what you're talking about. So I think it can be a useful way to speak, but I think it also can sometimes communicate more confidence than we actually have in what we're talking about, which isn't— I don't know, we didn't have, like, 1,000 governments attempt to steal our weights and X number of them succeeded or something. It's much more going off of sort of a judgement based on our security experts.

I slightly want to push you on this because I think at the point that we're at ASL, 4 or 5 or something like that. It would be a real shame if Anthropic was going ahead thinking, well, we think the risk that these weights will be stolen every year is 1%, 2%, 3%, something like that. And I guess maybe you're right in the policy saying we think it's very unlikely, extremely unlikely that this is going to happen. And then people externally think, well, basically it's fine. They say it's definitely not going to happen. There's no chance that this is going to happen. And governments might not appreciate that actually, in your own view, there is a substantial risk being run, and you just think it's an acceptable risk given the trade-offs and what else is going on in the world. I guess it's a social service for Anthropic to be direct about the risk that it thinks it's creating and why it's doing it. But I think it could be a really useful public service. I guess it's the kind of thing that might come up at Senate hearings and things like that, where people in government might really want to know, And I guess at that point it would be perhaps more apparent why it's really important to find out what the probability is. But yeah, that's a way that I think there's definitely a risk of misinterpretation by journalists or something who don't appreciate the spirit of saying that we think it's X% likely. But there could also be a lot of value in being more direct about it.

Yeah, I'm not really an expert on communications. I think some of it just depends on who your target audience is and how they're thinking about it. I think in general, I'm a fan of making the RSP more concrete, being more specific. I think over time, I hope it progresses in that direction as we learn more and can get more specific. I also think it's important for it to be verifiable. And I think if you start to give these precise percentages, people will then ask, how do you know? And I don't think there really is a clear answer to how do you know that the probability of this thing is less than X percent?

For many of these situations, it doesn't help with the bad faith actor or the bad faith operator either, because if you say, well, the safety threshold is 1% per year, they can kind of always just claim in the situation where we know so little that it's less than 1%. It doesn't really bind people all that much. Maybe it's just a way that people externally could understand a little better what the opinions are within the organisation, or at least what their stated opinions are.

I will say that internally, I think it is an extremely useful way for people to think about this, right? So if you are working on this, I think you probably should think through what is an acceptable level of danger and try to estimate it and communicate with people you're working closely with in these terms. I think can be a really useful way to give precise statements, and I think that can be very valuable.

A metaphor that you use within your responsible scaling policy is putting together an aeroplane while you're flying it. And I think that is one way that the challenge is particularly difficult for the industry and for Anthropic, that unlike with biology safety levels, where basically we know the diseases that we're handling and we know how bad they are and we know how they spread and things like that, the people who are figuring out what should BSL level 4 security be like can look at lots of studies to understand exactly the organisms that already exist and how they would spread. And how likely would they be to escape given these particular ventilation systems and so on? And even then, they mess things up decently often. But in this case, you're dealing with something that doesn't exist, that we're not even sure when it will exist or what it will look like. And you're developing the thing at the same time that you're trying to figure out how to make it safe. It's just extremely difficult. And we should expect mistakes. That's something that we should keep in mind, is that even if people who are doing their absolute best here are likely to mess up, And that's a reason why we need this defence-in-depth strategy that you're talking about, that we don't want to put all of our eggs in the RSP basket. We want to have many different layers, ideally.

Yeah, it's also a reason to start early. So I think one of the things with Claude 3 was that was sort of the first model where we really ran this whole process. And I think some part of me felt like, wow, this is kind of silly. I was pretty confident Claude 3 was not catastrophically dangerous. It was slightly better than GPT-4, which had been out for a long time and not caused a catastrophe. But I do think that the process of doing that, learning what we can, and then putting out public statements about how it went, what we learned, is the way that we can have this run really smoothly on the next time. We can make mistakes now, right? We could have made a ton of mistakes because the stakes are pretty low at the moment. But in the future, the stakes on this will be really high, and it will be really costly to make mistakes. So it's important to get those practice runs in.

All right, another kind of recurring theme that I've heard from some commentators is that in their view, the Anthropic RSP just isn't conservative enough. So on that account, there should be kind of wider buffers in case you're undereliciting capabilities that the model has that you don't realise, which is something that you're pretty concerned about. And I guess a different reason would be you might worry that there could be discontinuous improvements in capabilities. As you train bigger models with more data. So to some extent, model learning and improvement is, from a very zoomed out perspective, is quite continuous. But on the other hand, its ability to do any kind of particular task, it can go from fairly bad to quite good surprisingly quickly. So there can be sudden unexpected jumps with particular capabilities. Yeah. Firstly, can you maybe explain again in more detail how the Anthropic RSP kind of handles these safety buffers? Given that you don't necessarily know what capabilities a model might have before you train it, that's quite a challenging constraint to be operating under.

Yeah. So there are these redline capabilities. These are the capabilities that are actually the dangerous ones. We don't want to train a model that has these capabilities until we have the next set of precautions in place. Then there are evaluations we're creating, and these evaluations are meant to certify that the model is far short of those capabilities. It's not, can the model do those capabilities? Because once we pass them, we then need to put all the safety mitigations in place, etc. So, uh, and then when do we have to run those, those evaluations is sort of just— we have some heuristics of like when, when the effective compute goes up by a certain, a certain fraction. That, uh, is a very cheap thing that we can, you know, evaluate on every step of the run or something along those lines so that we know when to run it. Uh, in terms of like how conservative they are, I guess like one example I would give is you're thinking about like autonomy where a model could spread to a bunch of other computers and sort of autonomously replicate across the internet. I think our evaluations are pretty conservative on that front. We test if it can replicate to a fully undefended machine, or if it can do some basic fine-tuning of another language model to add a simple backdoor. I think these are pretty simple capabilities, and there's always a judgment call there, right? We could set them We could set them easier, but then we might trip those and look at the model and be like, ah, this isn't really dangerous. It doesn't warrant the level of precaution that we're going to give it.

Yeah, there was something also about— you said that the RSP says that you'll be worried if the model can succeed at half the time at these various different tasks, trying to spread itself to other machines. Why is half the time, succeeding half the time, the threshold?

Yeah, so there's sort of a few tasks. I don't off the top of my head remember the exact thresholds, but basically it's just a reliability thing, right? So in order for a model to chain all of this together, or yes, in order for the model to chain all of these capabilities together into some long-running thing, it does need to have a certain success rate. Probably it actually needs a very high success rate in order for it to start autonomously replicating despite us trying to stop it, et cetera. So we set a threshold that's fairly conservative on that front.

I guess it's part of the reason you're thinking, well, if a model can do this worrying thing half the time, then it might not be very much additional training away from being able to do it 99% of the time. That might just require some additional fine-tuning to get there. And so then the model might be dangerous if it was leaked because it would be so close to being able to do this stuff.

Yeah, I mean, that's often the case. Although, of course, we could then elicit it to go— if we'd set a higher number and even if we got 10%, Maybe that's enough that we could bootstrap it. So often when you're training something, if it can be successful, you can reward it for that successful behaviour and then increase the odds of that success. So it's often easier to sort of go from 10% to 70% than it is to go from 0% to 10%.

So if I understand correctly, the RSP proposes to retest models every time you increase the amount of training compute or data by fourfold. Is that right? That's kind of the Yeah.

So we're still thinking about what is sort of the best thing to do there, and that one might change. But we use this notion of effective compute. So really, this has to do with when you train a model, it goes down to a certain loss. And we have these nice scaling laws of if you have more compute, you should expect to get to the next loss. You might also have a big algorithmic win where you don't use any more compute, but you get to a lower loss. And we sort of have coined this term effective compute that would sort of account for that as well. These jumps are sort of the jump where you can tell we have sort of a visceral sense of how much smarter a model seems when you do that jump and have sort of set that as our bar for when we have to run all these evaluations, which do require a staff member to go and run them, spend a bunch of time trying to elicit the capabilities, et cetera. I think this is somewhere I'm wary of sounding too precise or like we understand this too well. We don't really know what the effective compute gap jump is between the yellow lines and the red lines. This is much more just like how we are thinking about the problem and how we are trying to set these evaluations. And the reason that the yellow line evaluations really do need to be substantially easier, they'd be far from the red line capabilities, because you might actually overshoot the yellow line capabilities by a fairly significant measure just off of when you run evaluations.

So I think if I recall, it was Zvi, who's been on the show before, who wrote in his blog post assessing the Anthropic RSP just that he thinks that this ratio between the 4x and the 6x is not large enough that if there is some discontinuous improvement, or you've really been undereliciting the capabilities of the models at these kind of interim check-in points, that does leave the possibility that you could overshoot and get into quite a dangerous point by accident. And then by the time you get there, then the model is quite a bit more capable than what you thought it would be. And then you've got this difficult question of whether to— do you then press the emergency button and delete all of the weights? you've overshot. There'd be incentives not to do that because you might be throwing away $100 million worth of— well, I guess I don't know how much it would be, but you'd be throwing away a substantial amount of compute expenditure basically to create this thing. And this just worries him. I mean, that could be solved, I think, in his view, just by having a larger ratio there, having a larger safety buffer. Of course, that then runs the risk that you're doing these constant check-ins on stuff that you really are pretty confident is not going to be actually that dangerous. And people might get frustrated with the RSP and feel like it's wasting their time. So it's kind of a judgement call, I guess, how large that buffer needs to be.

Yeah, I think it's a tricky one to communicate about because it's confidential what the jumps are between the models or something. I think one thing I can share is we ran this on Claude 3 partway through training. So the jump from Claude 2 to Claude 3 was bigger than that gap. So you could sort of think of that as, like, an intelligence jump from Claude 2 to Claude 3 is bigger than what we're allowing there. I think it feels reasonable to me, but I think this is just a judgement call that different people can have. And I think that this is the sort of thing where if we learn over time that this seems too big or it seems too small, that's the type of thing that hopefully we can talk about publicly.

Yeah. Is that something that you get feedback on? I suppose if you are training these big models and you're checking in on them, you can kind of predict where you expect them to be, how likely they are to exceed a given threshold. And then if you do ever get surprised, then that could be a sign that, well, we need to increase the buffer range here.

It's hard because, right, the first one, the thing that would really tell us is if we don't pass the yellow line for one model, and then on the next iteration, suddenly it blows past it, and we look at this and we're like, whoa, this thing is really dangerous. It's probably past the red line, and we have to delete the model or immediately put in the security features, et cetera, for the next level. I think that would be a sign that we'd set the buffer too small.

I guess, again, not the ideal way to learn that, but I suppose it definitely could set a cat amongst the pigeons.

Yeah, there would be earlier signs where you would notice, like, oh, we really overshot by a lot. It feels like we're closer than we expected or something. But that would sort of be the failure mode, I guess, rather than the warning sign.

So reading the RSP, it seems pretty focused on kind of catastrophic risks from misuse, sort of terrorist attacks or CBRN, and AI gone rogue, like spreading out of control, that sort of thing. Is it basically right that the RSP or this kind of framework is not intended to address kind of structural issues like AI displaces people from work and now they can't earn a living, or AIs are getting militarised and that's making it more difficult to prevent military encounters between countries because we can't control the models very well, or more near-term stuff like algorithmic bias or deepfakes or misinformation. Are those kind of things that have to be dealt with by something other than a responsible scaling policy?

Yeah, those are important problems, but the RSP is responsible for preventing sort of catastrophic risks. And particularly has this framing that works well for things that are sort of acute, like new capability is developed and could sort of first order cause a lot of damage. It's not going to work for things that are like, what is the long-term effect of this on society over time? Because we can't design evaluations to test for that effectively.

Anthropic does have different teams that work on those other 2 clusters that I talked about, right? What are they called?

So we have a societal impacts team, is probably the most relevant one to that. And the policy team also has a lot of a lot of relevance to these issues.

All right. Yeah, I guess we're kind of going to wrap up on RSPs now. Is there anything you wanted to maybe say to the audience to wrap up this section? Like additional work that you think would be, or like ways that the audience might be able to contribute to this enterprise of coming up with better internal company policies and then figuring out, I suppose, how there could be models for other actors to come up with government policy as well?

Yeah, I mean, I think this is just a thing that many people can work on. If you work at a lab, you could talk to people there Think about what they should have as an RSP, if anything. If you work in policy, you should read these and think about if there are lessons to take. If you don't do either of those, I think you really can think about threat modeling, post about that, think about evaluations, implement evaluations, and share those. I think it is the case that these companies are very busy, and if there is something that's just shovel-ready or ready on the shelf, you could just grab this evaluation. It's really quite easy to run them. So, yeah, I think there's quite a lot that people can do to help here.

All right, let's push on and talk about the case that listeners might be able to contribute to making superintelligence go better by working at Anthropic on some of its various different projects. First, though, how did you end up in your current role at Anthropic? Kind of what's been the career journey that led you there?

Yeah, so I think it largely started with an internship at GiveWell, which listeners might know of, but it's sort of a nonprofit that evaluates charities to figure out where to give money, most effectively. And I did an internship there. I sort of learned a ton about global poverty, global health. I was planning to do a PhD in economics and go work on global poverty at the time. But a few people there sort of pushed me and said, you should really worry about AI safety. We're going to have these super intelligent AIs at some point in the future, and this could be a big risk. I remember I left that summer internship and was like, wow, these people are crazy. I talked to all my family and they were like, What are you thinking? But then I like, I don't know, it was like interesting. So I kept talking to people, some people there, other people sort of worried about this. And I felt like every debate I lost, I would like have a little debate with them about why we shouldn't worry about it. And I'd always come away feeling like I lost the debate, but not like fully convinced. And after like honestly like a few years of doing this, I eventually decided this was at least enough convince— at least convincing enough that I should work in AI. It also turned out that working on poverty via this economics PhD route was a much longer and more difficult and less likely to be successful path than I had anticipated. So I sort of pivoted over to AI. I worked at Vicarious, which is an AGI lab that had sort of shifted towards a robotics product angle. And I worked on computer vision there for a while, learning how to do ML research. And then actually, 80,000 Hours reached out to me and convinced me that I should work on safety more imminently. AI was getting better. It was like, more important that I just have some direct impact on doing safety research. At the time, I think OpenAI had by far the best safety research coming out of there. So I applied to work on safety at OpenAI. I actually got rejected. Then I got rejected again. In that time, Vicarious was nice enough to let me spend half of my time reading safety papers. So I was just sort of reading safety papers, trying to do my own safety research, although it was somewhat difficult. I didn't really know where to get started. And eventually, I also wrote for Rohin Shah, who was on this podcast. He had this alignment newsletter, and I read papers and wrote summaries and opinions for them for a while to motivate myself. But eventually, third try, got a job offer from OpenAI, joined the safety team there, and spent sort of 8 months there mostly working on code models and understanding how code models would progress. The logic here being we just started the first LLMs training on code. And I thought it was pretty scary. If you think about recursive self-improvement, models that can write code is the first step, and trying to understand what direction that would go in would be really useful for informing safety directions. And then a little bit after that, maybe like 8 months in or so, all of the safety team leads at OpenAI left, most of them to start Anthropic. I sort of felt very aligned with their values and mission, so also went to join Anthropic. Sort of the main reason I'd been at OpenAI was for the safety work. And then at Anthropic, actually, everyone was just building out infrastructure to train models. There was no code. It was sort of the beginning of the company. And I found that the thing was my comparative advantage was making them efficient. So I optimized the models to go faster. As I said, if you have more compute, you get a better model. So that means if you can make things run quicker, you get a better model as well. I did that for a while and then shifted into management, which had been something I wanted to do for a while, and started sort of managing the pre-training team when it was it was 5 people, and then have been sort of growing the team since then, training better and better models along the way.

Yeah, I'd heard that you'd been consuming 80,000 Hours stuff years ago, but I didn't realise it had influenced you all that much. What was the step that we helped with? It was just deciding that it was important to actually start working on safety-related work sooner rather than later.

Actually, a bunch of spots along the way. I think when I did that GiveWell internship, I did like a speed coaching at EA Global or something with 80,000 Hours. And the people there were some of the people who were pushing me that I should work on AI, like some of those conversations. And then when I was at Vicarious, I think 80,000 Hours reached out to me and was sort of like more pushy and specifically was like, you should go to work directly on safety now, where I think I was otherwise sort of happy to just sort of keep learning about AI for a bit longer before shifting over to safety work.

Well, cool that 80K was able to— I guess I don't know whether it helped, but I suppose it influenced you in some direction. Is there any stuff that you've read from 80K on AI careers advice that you think is mistaken, where you want to tell the audience that maybe they should do things a little bit differently than what we've been suggesting on the website, or I guess on this show?

Yeah. First, I do want to say 80K was very helpful, both in pushing me to do it and setting me up with connections and introducing me to people. getting me a lot of information. It was really great. In terms of things that I maybe disagree with from standard advice, I think the main one would be to focus more on engineering than research. I think there is sort of this historical thing where people have focused on research more so than engineering. Maybe I should define the difference. The difference between research and engineering here would be that research can look more like figuring out what directions you should work on, designing experiments, Doing really careful analysis and understanding that analysis, figuring out what conclusions to draw from a set of experiments. So I can maybe give an example, which is like, you're training a model with one architecture and you're like, oh, I have an idea, we should try this other architecture. And in order to try it, the right experiments would be these experiments, and these would be the comparisons to confirm if it's better or worse. Engineering is more of the implementation of the experiment. So then taking that experiment, trying it, and also creating tooling to make that fast and easy to do. So make it so that you and everyone else can really quickly run experiments. It could be optimizing code, so making things run much faster, as I mentioned I did for a while, or making the code easier to use so that other people can use it better. And it's not like someone's an engineer or a researcher. You kind of need both of these skill sets to do work. You come up with ideas, you implement them, you see the results, and then you implement changes. And it's a fast iteration loop, but it's somewhere where I think there's historically been more prestige given to the research end. despite the fact that most of the work is the engineering end. So you end up with, if you come up with your architecture idea, that takes an hour, and then you spend a week implementing it, and then you run your analysis and that maybe takes a few days, but it sort of feels like the engineering work takes the longest. And then my other pitch here is going to be that the one place where I've often seen researchers not investigate an area they should have is when the tooling is bad. So when you go to do research on this area and you're like, ugh, it's really painful, all my experiments are slow to run, it will really quickly have people be like, I'm going to go do these other experiments that seem easier. So often by creating tooling to make something easy, you actually can open up that direction and sort of trailblaze a path for a bunch of other people to sort of follow along and do a lot of experiments.

So what fraction of people at Anthropic would you classify as more on the engineering end versus more on the research end?

I might go with my team because I think I actually don't know for all of Anthropic. And I think it's sort of a spectrum, but I would guess it's probably like 60 or 70% of people would be, I would say, are probably stronger on the engineering end than on the research end. And when hiring, I'm most excited about finding people who are strong on the engineering end. And most of our interviews are sort of tailored towards that, not because the research isn't important, but because I think it's sort of, there's There's a little bit less need for it.

The distinction sounds like a little bit artificial to me. Is that kind of true? It feels like these things are kind of all just a bit part of a package. Yeah.

Although I think the main distinction with engineering is that it is a fairly separate career. So I think there are many people, maybe hopefully listening to this podcast, who might have sort of been a software engineer at some tech company for a decade and built up a huge amount of expertise and experience with designing good software and And such. And those people, I think, can actually learn the ML they sort of need to know to do the job effectively very quickly. And I think there's maybe another direction people could go in, which is much more like, I think of as like a PhD in many cases, where you spend a lot of time developing research taste, figuring out what are the right experiments to run, running these usually at smaller scale and maybe with less of sort of a single long-lived codebase that pushes you to develop better engineering practices. And I think that skill set— and to be clear, this is a relative term. It's also a really valuable skill set, and you always need a balance. But I think I've often had the impression that 80,000 Hours pushes people more in that direction who want to work on safety, more the do a PhD, become sort of like a research expert with really great research taste, than pushing people more on the sort of become a really great software engineer direction.

Yeah, we had a podcast many years ago, it might be 2018 or 2017, with Katherine Olson and Daniel Ziegler, where they were also saying engineering is the way to go, or engineering is the thing that's really scarce and is also the easier way into the industry. But yeah, it isn't a drum that we've been banging all that frequently. I don't think we've talked about it very much since then. So perhaps that's a bit of a mistake that we haven't been highlighting the engineering roles more. You said it's kind of a different career track. So you can go from software engineering to the ML or AI engineering that you're doing at Anthropic. Is that the natural career progression that someone has? Someone who's not already in this, how can they learn the engineering skills that they need?

Yeah, so I think engineering skills are actually in some ways the easiest to learn because there's so many different engineering places. I think the way I would recommend it is you could work at any engineering job, Usually I would say just working with the smartest people you can, building the most complex systems. You can also just do this open source. You can contribute to an open source project. This is often a great way to get mentorship from the maintainers and have something that's publicly visible. So if you then want to apply to a job, you can be like, here is this thing I made. And then you can also just create something new. So you can say, I think if you want to work on AI engineering, you should probably pick a project that's similar to what you want to do. So if you want to work on data for large language models, take Common Crawl, it's a publicly available crawl of the web, and write a bunch of infrastructure to process it really efficiently. Then maybe train some models on it, like build out some infrastructure to train models, and you can sort of build out that skill set relatively easily without needing to work somewhere.

Why do you think people have been overestimating research relative to engineering? Is it just that research sounds cooler? It's got better branding?

I think historically it was a prestige thing. I think there's sort of this distinction between research scientist and research engineer that used to exist in the field. where research scientists had PhDs and were designating the experiments that the research engineers would run. And I think that shifted a while ago. So I think in some sense, the shift has already started happening. Now, many places, Dropbox included, everyone's a member of technical staff. There isn't this distinction. And the reason is that the engineering got more important, particularly with scaling. Once you got to the point where you were training models that used a lot of compute on a big distributed cluster, the engineering to implement things on these distributed runs got much more complex than when it was sort of more quick experiments on cheap models.

To what extent is it a bottleneck just being able to build these enormous compute clusters and operate them effectively? Is that a core part of the stuff that Anthropic has to do?

Yeah. So we rely on cloud providers to actually build the data centres and put the chips in it. But we've now reached a scale where The amount of compute we're using is sort of— it's a very dedicated thing. These are really huge investments, and we're sort of involved collaborating on it from sort of the design up. And I think it's a very critical piece, right? Like, given that compute is the main driver, the ability to take a lot of compute and use it all together and to design things that are cheap given the types of workloads you want to run can be like a huge multiplier on how much compute you have.

All right. Yeah. Do you want to give us the pitch for working at Anthropic as a particularly good way to make the future with superintelligent AI go well?

Yeah, I might pitch working on AI safety first.

Sure.

Yeah.

The case here is it's just really important. I think AGI is going to be probably the biggest technological change ever to happen. The thing I think I keep in my mind is just like, what would it be like to have every person in the world able to spin up a company of a million people? All of whom are as smart as the smartest people you know, and task them with any project they want. And you could do a huge amount of good with that. You could help cure diseases, you could tackle climate change, you could work on poverty. There's sort of a ton of stuff you can do that would be great, but there's also a lot of ways it could go really badly. So I just think the stakes here are really high, and then there's a pretty small number of people working on it. If you sort of account for all the people working on Things like this, I think you're probably gonna get something in like the thousands right now, maybe tens of thousands. Uh, it's, it's rapidly increasing, but it's, it's quite small compared to the scale of the problem. In terms of why Anthropic, um, I think my main case here is just like the— I think the best way to like make sure things go well is to get a bunch of people who care about the same thing and all work together with that as the, as the main focus. I mean, Anthropic's not perfect. We definitely have, have issues, as, as does, does every organization. But I think one thing that I've really appreciated is just seeing how much progress we can make when there's a whole team where everyone trusts each other, kind of deeply shares the same goals, and can work on that together.

I guess there is a bit of a trade-off between— if you imagine there's kind of a pool of people who are very focused on AI safety and kind of have the attitude that you just expressed, one approach would be to split them up between each of the different companies that are working on frontier AI. I guess that would have some benefits. The alternative would be to cluster them all together in a single place where they can work together and make a lot of progress, but perhaps the things that they learn won't be as easily diffused across all of the other companies. Yeah. Do you have a view on where the right balance is there between kind of clustering people so they can work together more effectively, communicate more, versus the need perhaps to have people everywhere who can absorb the work?

I just think the benefits from working together are really huge. I think it's so different what you can accomplish when you have 5 people all working together, as opposed to 5 people working independently, unable to sort of speak to each other, communicate about what they're doing. You sort of run the risk of just doing everything in parallel, not learning from each other, and also not building trust, which I think is just somewhat a core piece of eventually being able to work together to implement the things.

So inasmuch as Anthropic is or becomes the main leader in interpretability research and other kind of lines of technical AI safety research, Do you think it is the case that other companies are going to be very interested to absorb that research and apply it to their own work? Or is there a possibility that Anthropic will have really good safety techniques, but then they might get stuck in Anthropic and potentially the most capable models that are being developed elsewhere are developed without them?

Yeah, so I think that my hope is that if other people have either develop RSP-like things, or if there are regulations sort of requiring particular safety mitigations, people will want— will have a strong incentive to want to get better safety practices. And we publish our safety research, so in some ways we're making it as easy as possible as we can for them. We're like, here is all the safety research we've done, here's as much detail as we can give about it, uh, please go reproduce it. Uh, beyond that, I think we're kind of— it's hard to be accountable for what other places do. And I think to some degree, it just makes sense for Anthropic to try to set an example and be like, we can be a frontier lab while still prioritising safety, putting out a lot of safety work, and hoping that kind of inspires others to do the same.

I don't know what the answer to this is, but do you know researchers at Anthropic sometimes go and visit other AI companies and vice versa in order to cross-pollinate ideas? I think that used to maybe happen more, and maybe things have gotten a little bit tighter the last few years, but that's one idea that you could hope that research might get passed around, or at least, I mean, you're saying it gets published. I guess that's important. But there is a risk that the technical details of how you actually apply the methods won't always necessarily be in the paper or be very easy to figure out. So you also often need to talk to people to make things work.

Yeah, I think once something's published, you can go and give talks on it, et cetera. I think publishing is sort of the first step where until it's published, then it's like confidential information that that can't be shared. So, yeah, it's sort of like you have to first figure out how to do it, then publish it. There are more steps you could take, right? You could then open source code that enables you to run it more carefully. And there's a lot of work you could go in that direction. And then it's just sort of a balance of how much time you spend on disseminating your results versus pushing your agenda forward to actually make progress.

It's possible that I'm slightly— I'm analogising from, I guess, biology that I'm somewhat more familiar with, where it's notorious that having a biology paper or a medical paper does not allow you to replicate the experiment because there's so many important details missing. But is it possible that in ML, in AI, people tend to just publish all of the stuff that they— all of the data maybe, and all of the code online on GitHub or whatever, such that it's much more straightforward to completely replicate a piece of research elsewhere?

Yeah, I think it's a totally different level of replication. It depends on the paper, but on many papers, if a paper is published in some conference, I would sort of expect that someone can pull up the paper and reimplement it with maybe a week's worth of work. There's a strong norm of sometimes providing the actual code that you need to run, but providing enough detail that you can. I think with some things it can be tricky where, I don't know, our interpretability team just put out a paper on how to get features on one of our production models, and we didn't release details about our production model. So we tried to include enough detail that someone could replicate this on another model, but they probably can't exactly create our production model and get the exact features that we have.

Okay, in a minute we'll talk about one of the concerns that people might have about working at any AI company. But in the meantime, yeah, what roles are you hiring for at the moment and what roles are likely to be open at Anthropic in future?

So probably just check our website. There's quite a lot. I'll kind of highlight a few. So I think the first one I should highlight is the RSP team is looking for people to develop evaluations, uh, work on the RSP itself, figure out the, like, what the next, next version of the RSP should look like, etc. Uh, on, on my team, we're hiring a bunch of research engineers. So this is come up with approaches to improve models, implement them, analyze the results, kind of pushing, pushing that loop. Uh, and then also performance engineers. This one's maybe like a little bit more surprising, but a lot of the work now happens on custom AI chips. And making those run really efficiently is sort of absolutely critical. There's a lot of interplay between how fast it can go and how good the model is. So we're sort of hiring quite a number of performance engineers where you don't need to have a tonne of AI expertise, just having deep knowledge of how hardware works and how to write code really efficiently.

How can people learn that skill? Are there courses for that?

There are probably courses. I think with basically everything, I would recommend finding a project, finding someone to mentor you, and be cognizant of their time. Maybe you spend a bunch of time writing up some code and you send them a few hundred lines and say, can you review this and help me? Or maybe you've got some weekly meeting where you ask questions. But yeah, I think you can read about it online. You can take courses, or you can just pick a project and say, I'm going to implement a transformer as fast as I possibly can and sort of hack on that for a while.

Are most people coming into Anthropic from other AI companies or the tech industry more broadly, or from PhDs, or maybe not even PhDs?

It's quite a mix. I think PhD is definitely not necessary. I think it's like one direction to go to build up this skill set. We have a shockingly large number of people with physics backgrounds. who have done theoretical physics for a long time and then sort of spend some number of months learning the engineering to be able to write Python really well, essentially, and then switch in. So I think there's not really a particular background that is sort of needed. It's just, I would say, if you're directly preparing for it, just pick the closest thing you can to the job and do that to prepare. But don't feel like you needed to have some particular background in order to apply.

This question is slightly absurd because there's such a range of different roles that people could potentially apply for at Anthropic. But do you have any any advice to people who would like— the vision for their career is working at Anthropic or something similar, but they don't yet feel like they're qualified to get a role at such a serious organisation? What are some interesting underrated paths maybe to gain experience or skills so that they can be more useful to the project in future?

Yeah, I would just pick the role you want and then do it externally. Do it in a very publicly visible way, get advice, and then apply with that as an example. So if you want to work on interpretability, make some tooling to pull out features of models and post that on GitHub or publish a paper on interpretability. If you want to work on the RSP, then make a really good evaluation, post it on GitHub with a nice write-up of how to run it, and include that with your application. This sort of takes time and is hard to do well, but I think that it's both the best way to know if it's really the role you want. And when hiring for something, I have a role in mind and I want to know someone can do it. And if someone has shown, like, look, I'm already doing this role, of course, here's my proof I can do it well. That's the most convincing case in many ways, more so than the signal you'd get out of an interview where all you really know is they did well on this particular question.

So in terms of working at AI companies, regular listeners will recall that earlier in the year I spoke with Zvi Moshkowitz, who's a kind of longtime follower of advances in AI and I'd say is a bit on the pessimistic side about AI safety. And maybe also not— I think he likes the Anthropic RFP, but he's not convinced that any of the safety plans put forward by any company or any government are, at the end of the day, going to be quite enough to keep us safe from rapidly self-improving AI. And he said that he was pretty strongly against people taking capabilities roles that would kind of push forward the frontier of what the most powerful AI models can do, I guess, especially at at leading AI companies? Because the basic argument is just that those roles are causing a lot of harm because they're speeding things up and leaving us less time to solve whatever kind of safety issues we're going to need to address. And I pushed back a little bit, and he wasn't really convinced by the various kind of justifications that one might give, like the need to gain skills that you could then apply to safety work later, or maybe you'd have the ability to influence a company's culture by being on the inside rather than the outside. I think of all companies, Zvi, I would certainly imagine, is most sympathetic to Anthropic. But I guess his philosophy is very much to kind of rely on hard constraints rather than put trust in any particular individuals or organisations that you like. I'm guessing that you might have heard what Zvi had to say in that episode, and I guess it was a critique that arguably applies to your job training Claude 3 and other frontier LLMs. So I'm kind of fascinated to hear what you thought of Zvi's perspective there.

So I think there's one argument, which is to do this to build career capital, and then there's another that is to do this for direct impact. I think on the career capital one, I'm pretty skeptical. I think career capital is sort of weird to think about in this field that's growing exponentially. I think in sort of a normal field, people often say you have the most impact late in your career. You build up skills for a while, and then maybe your 40s or 50s is when you have the most impact in your career. But given the rapid growth in this field, I think actually the best moment for impact is now. I don't know. I often think of, like, in 2021, I was already at Anthropic. I think there were probably tens of people working on large language models, which I thought were the main path towards AGI. And now there are thousands. I've improved. I've gotten better since then, but I think probably I had way more potential for impact back in 2021 when there were only tens of people working on it. Yeah.

Your best years are behind you, Nick.

Yeah, I think the potential was very high. I still think that there's still a lot of room for impact, and it will maybe decay, but it is from an extremely high number or extremely high level. And then the thing is, just the field isn't that deep because it's such a recent development. It's not like you need to learn a lot before you can contribute. I think if you want to do physics and you have to learn the past multi-thousand, like thousands of years of physics before you can like push the frontier. That's a very different setup from, from where we're kind of at. Um, and maybe my last argument is just like, if you think timelines are short, uh, like depending exactly how short, there's just actually not that much time left.

So if you're gonna—

if you think, you know, there's 5 years and you spent 2 of them building up a skill set, like that's, that's a significant fraction of the time. I'm not saying that that should be someone's timeline or anything, but like The shorter they are, the less that makes sense. So, yeah, I think from a career capital perspective, I probably agree. Does that make sense?

Yeah. And what about from other points of view?

Yeah, I think from a direct impact perspective, I'm fairly less convinced. Part of this is just that I don't have this framing of there's capabilities and there is safety and they are separate tracks that are racing. I think that's one way to look at it, but I actually think they're really intertwined. a lot of safety work relies on capabilities advances. I can give this example of this many-shot jailbreaking paper that one of our safety teams published, which uses long-context models to find a jailbreak that can apply to Claude and to other models. And that research was only possible because we had long-context models that you could test this on. So I think there's just a lot of cases where the things come together. But then I think if you're going to work on capabilities, you should be really thoughtful about it. I do think there is a risk. You are speeding them up. In some sense, you could be creating something that is really dangerous. But I don't think it's as simple as just don't do it. I think you want to think all the way through to what is the downstream impact when someone trains AGI and how will you have affected that? And that's a really hard problem to think about. There's a million factors at play, but I think you should think it through, come to your best judgment, and then reevaluate and get other people's opinions sort of as you go. Some of the things I might suggest doing if you're considering it is understand what— if you're considering working on capabilities at some lab, is try to understand their theory of change. Ask people there, ask how does your work on capabilities lead to a better outcome and see if you agree with that. I would talk to their safety team, talk to safety researchers externally, get their take, say, like, do they think that this is a good thing to do? And then I would also look at their track record and their governance and all the things that sort of to answer the question of, do you think they will push on this theory of change over the next 5 years? Are you confident this is sort of what will actually happen? One thing for me that convinced me at Anthropic that I was maybe not doing evil or made me feel much better about it is that our safety team is willing to help out with capabilities and actually wants us to do well at that. So early on with Opus, before we launched it, we had a major fire. There was sort of a bunch of issues that came up, and there was one very critical research project that my team didn't have capacity to push forward. So I asked Ethan Perez, who's one of the safety leads at Anthropic, can you help with this? It was actually during an offsite, and Ethan and most of his team just basically went upstairs to this building in the woods that we had for the offsite and cranked out research on this for the next 2 weeks and just And for me, at least, that was like, ah, yes, the safety team here also thinks that us staying on the frontier is critical.

So the basic idea is you think that the safety research of many different types that Anthropic is doing is very useful. It sets a great example. It's research that could then be adopted by other groups and also used by Anthropic to make safe models. And the only way that can happen. The only reason that research is possible at all is that Anthropic has these kind of frontier LLMs on which to experiment and do that research and to be at the cutting edge generally of this technology. And so able to figure out what's the safety research agenda that is most likely to be relevant in future. If I imagine, what would Zvi say? I'm going to try to model him, I suppose. I guess that he might say, yes, given that there's this competitive dynamic forcing us to shorten timelines, bringing the future forward maybe faster than we feel comfortable with, maybe that's the best you can do. But wouldn't it be great if we could coordinate more in order to buy ourselves more time? I guess that would be one angle. Another angle that I've heard from some people, I don't know whether Zhiyi would say this or not, is that we're nowhere near actually having all of the safety-relevant insights that we could have with the models that we have now. And so given that there's still such fertile material with Claude 2, maybe, or at least with Claude 3 now, why do you need to go ahead and train Claude 4? Maybe it's true that 5 years ago, when we were so much further away from having AGI or having models that were really interesting to work with, we were a little bit at a loose end trying to figure out what safety research would be good, because we just didn't know what direction things were going to go. But now there's so much safety research. There's a proliferation, a Cambrian explosion of really valuable work, and we don't necessarily need more capable models than what we have now in order to discover really valuable things. Yeah. What would you say to that?

Yeah, on the first one, I think there's sometimes this, like, what is the ideal world if everyone was me or something? Or if everyone thought what I thought, what would be the ideal setup? And I think that's just not how sort of the world works. And I think to some degree, you really only can control what you do. And maybe you can control what— or control— maybe you can influence what a small number of people you talk to do. But I think you sort of have to think about your role in the context of the broader world, more or less. acting in the way that they're going to act. So, yeah, and it's definitely a big part of why I think it's important for Anthropic to have capabilities is to enable safety researchers to have better models. Another piece of it is to enable us to have an impact on the field and try to set this example for other labs that you can deploy models responsibly and do this in a way that doesn't cause catastrophic risks and continues to push on safety. In terms of can we do safety research with current models, I think there is definitely a lot to do. I also think we will target that work better the closer we get to AGI. I think the last year before AGI will definitely be the most targeted safety work. Hopefully there'll be the most safety work happening then, but it will be the most time constrained. So you need to do work now because there's a bunch of serial time that's needed in order to make progress. But you also want to to sort of be ready to make use of the most well-directed time towards the end.

I guess another concern that people have, which you touched on earlier, but maybe we could talk about a little bit more, is this worry that Anthropic, by existing, by competing with other AI companies, it stokes the arms race, increases the pressure on them feeling that they need to improve their models further, put more money into it, release things as quickly as they can. If I remember, your basic response to that was like, yes, that effect's not zero, but in the scheme of things, there's a lot of pressure on companies to be training models and trying to improve them. And Anthropic is kind of a drop in the bucket there. And so this isn't necessarily the most important thing to be worrying about.

Yeah, I think basically that's pretty accurate. I think one way I would think about it is just what would happen if Anthropic stopped existing, right? Like, if we all just disappeared, what effect would that have on the world? Either if you Or if you think about, like, if we dissolved as a company and everyone went to work at all the others. And my guess is it just wouldn't look like everyone slows down and is way more cautious. That's, like, not my model of it. If that was my model, then I would be like, ah, we're probably doing something wrong. I think there— yeah, so I think it's an effect, but I sort of think about it in terms of, like, what is the net effect of Anthropic being on the frontier when you account for, like, all the different actions we're taking, all the safety research, all the policy advocacy, like, the effect our products have helping users. There's sort of, like, this whole whole large scheme, and you can't really add it all up and subtract the costs, but I think you can do that somewhat in your mind or something.

Yeah, I see. So the way you conceptualise it is thinking Anthropic as a whole, what impact is it having by existing compared to some counterfactual where Anthropic wasn't there? And then you're contributing to this broader enterprise that is Anthropic and all of its projects and plans together, rather than thinking about today I got up and I helped to improve Claude 3 in this narrow way. What impact does that specifically have? Because that's just maybe it's missing the real effects that matter the most from allowing this organisation to exist through your work.

Yeah, you could definitely think on the margin, to some degree, if you're joining and going to help with something, you are just increasing Anthropic's sort of this marginal amount of capabilities. And then I would just sort of look at, do you think we would be on a better trajectory if Anthropic had better models? And do you think we'd be on a worse trajectory if Anthropic had significantly worse models? Would be sort of the comparison. I think you could look at, like, oh, well, what would happen if Anthropic didn't ship Claude 3 last year or earlier this year?

What are some of the lines of research that you're most pleased that you've helped Anthropic to pursue? Or what are some of the kind of safety wins that you're really pleased by?

I'm really excited about the safety work. I think there's just, like, a ton of it that has come out of Anthropic. I could sort of start with interpretability, where I think at the beginning of Anthropic, it was like figuring out how single-layer transformers work, these sort of very simple toy models. And in sort of the past few years, and this is not my doing, this is all the interpretability team, that has sort of scaled up into actually being able to look at production models that people are really using and find useful and identify particular features. We sort of had this recent one on the Golden Gate Bridge where they found a feature that if you increase it, makes the model— it's the model's representation of the Golden Gate Bridge. And if you increase it, the model talks more about the Golden Gate Bridge. And that's sort of like a very cool causal effect where you can change something and it actually changes the model behaviour in a way that gives you more certainty you've really found something.

Is the hope with that, for example, that you could find— I guess you found the— I don't know whether you call it the Golden Gate Bridge neuron, but I don't know, you found where the Golden Gate Bridge is in the model, and then you can turn that up so that it I'm not sure whether all listeners will have seen this, but it is very funny because you get Claude 3, its mind is constantly turned to thinking about the Golden Gate Bridge, even when the question has nothing to do with it. And it gets frustrated with itself, realising that it's going off topic, and then tries to bring it back to the thing that you asked, but then it just can't avoid talking about the Golden Gate Bridge again. Is the hope that you could find the honesty part of the model and scale that up enormously, or alternatively find the deception part and scale that down in the same way?

Yeah, so there actually are a bunch. If you look at the paper, there's a bunch of safety-relevant features. I think that the Golden Gate Bridge one was cuter or something and got a bit more attention. But yeah, there are a ton of features that are really safety-relevant. I think one of my favorites was one that will tell you if code is incorrect or something, or has a vulnerability, something along those lines. And then you can change that and suddenly it doesn't write the vulnerability or it makes the code correct. And that sort of shows the model knows about concepts at that level. Now, can we use this directly to solve major issues? Probably not yet. There's a lot more work to be done here, but I think it's just been a huge amount of progress. And I think that it's fair to say that progress wouldn't have happened without Anthropic's interpretability team pushing that field forward a lot.

Yeah. Is there any other Anthropic research that you're proud of?

Yeah, I mentioned this one a little bit earlier, but there's sort of this multi-shot jailbreaking from our alignment team that pushed, like, if you have a long context model, which is something that we released, you can jailbreak a model by just giving it a lot of examples in this very long context. And it's a very reliable jailbreak to get models to do things you don't want. So this is sort of in the vein of the RSP. One of the things we want to have is to be able to be robust to really intense red teaming, where if a model has a dangerous capability, you can have safety features that prevent people from eliciting it. And this is like an identification of a major risk for that. We also have this sleeper agents paper, which sort of shows early signs of models having deceptive behavior. Yeah, I could talk about a lot more of it. There's actually just a really huge amount. And I think that's fairly critical here is I think often with safety things, people get focused on inputs and not outputs or something. And I think the important thing is to think about how much progress are we actually making on the safety front? That is ultimately what's going to matter. In some number of years when we get close to AGI, it won't be like, how many GPUs did we use? How many people worked on it? It's going to be like, what did we find and how effective were we at it? And for product, this is very natural. People think in terms of revenue. How many users did you get? You have these end metrics that are the fundamental thing you care about. And I think for safety, it's much fuzzier and harder to measure. But putting out a lot of papers that are good is quite important.

Yeah, I mean, if you want to keep going, if there's any others that you want to flag, I'm in no hurry.

Yeah, I mean, I can talk about influence functions. I think this is a really cool one where one framing of mech interp, mechanical interpretability is— or sorry, mechanistic interpretability is it lets us look at the weights and understand why a model has a behavior by looking at a particular weight. The idea of influence functions is to understand why a model has a behavior by looking at the training data. So you can understand what in your training data contributed to a particular behavior from the model. Yeah, I think that was pretty exciting to see work. I think constitutional AI is another example I would highlight where we can train a model to follow a set of principles via sort of AI feedback. So instead of having to sort of have human feedback for a bunch of things, you can just write out a set of principles. I want the model to not do this. I want it to not do this. I want it to not do this. and train the model to follow that constitution.

Is there any work at Anthropic that you personally would be wary, or at least not enthusiastic to contribute to?

So I think in general, this is like a good question to ask. I think the work I'm doing is currently the highest impact thing, and I think I should frequently wonder if that's the case and talk to people and reassess. I think right now, though, I don't think there's any work at Anthropic that I wouldn't contribute to or think shouldn't be done. It's probably not the way I would approach it. It's like if there was something that I thought Anthropic was doing that was bad for the world, I would write a doc making my case and send it to the relevant person who's responsible for that and then have a discussion with them about it. Because just opting out isn't going to actually change it, right? Someone else will just do it. That doesn't accomplish much. And we try to sort of operate as one team where everyone is aiming towards the same goals. And not have this sort of, like, 2 different teams are at odds where you're hoping someone else won't succeed.

I guess people might have a reasonable sense of the culture at Anthropic just from listening to this interview, but is there anything else that's interesting about working at Anthropic that might not be immediately obvious?

I think the one thing that is part of our culture that at least surprised me is spending a lot of time pair programming. It's just a very collaborative culture. So I think when I first joined, I don't know, I was working on a particular method of distributing a language model training across a bunch of GPUs. And Tom Brown, who's one of the founders and had done this for GPT-3, just put an 8-hour meeting on my calendar and I just watched him code it. And then I was on a different time zone, so basically during the hours when he wasn't working and I was working, I would push forward as far as I could. And then the next day we would meet again and continue on. And I think this is just a really good way of aligning people where it's a shared project instead of being a You're bothering someone by asking for their help. It's like you're working together on the thing and you learn a lot. You also learn a lot of the smaller things that you wouldn't otherwise see of how does someone navigate their code editor? What exactly is their style of debugging this sort of problem? Whereas if you go and ask them for advice or how do I do this project, they're not going to tell you the low-level details. When do they pull out a debugger versus some other method of some other tool for solving the problem.

So this is literally just watching one another's screens, or you're doing a screen share thing where you watch?

Yeah, I'll give some free advertising to Tuple, which is this great software for it where you can share screens and you can control each other's screens and draw on the screen. And typically one person will drive, they'll be basically doing the work, and the other person will watch, ask questions, point out mistakes, occasionally grab the cursor and just change it.

It's interesting that I feel in other industries, having your boss or a colleague stare constantly at your screen would give people the creeps or they would hate it. Whereas it seems like in programming, this is something that people are really excited by and they feel like it enhances their productivity and makes the work a lot more fun.

Oh, yeah. I mean, it can be exhausting and tiring. I think the first time I did this, I was too nervous to take a bathroom break and after multiple hours was like, can I go to the bathroom? And realised that was an absurd thing to ask after after multiple hours of working on something.

What are you, back at primary school?

Yeah, it can definitely feel a little bit more intense in that someone's watching you and they might give you feedback on, like, you're kind of going slow here. This sort of thing would speed you up. But I think you really can learn a lot from that sort of intensive partnering with someone.

All right, I think we've talked about Anthropic for a while. I guess final question is, so obviously Anthropic, its main office is in San Francisco, right? And I heard that it was opening kind of a branch in London. Are those the 2 main places And are there many people who work remotely or anything like that?

Yeah, so we have the main office in SF, and then we have an office in London, Dublin, I think, Seattle, and New York. Our typical policy is like 25% time in person. So some people will mostly work remotely and then go to sort of one of the hubs for— it's usually one week per month. The idea of this is that we want people to build trust with each other and be able to work together well and know each other. And that sort of involves some amount of social interaction with your coworkers. But also for a variety of reasons, like sometimes getting the best people, people are bound to particular locations.

I kind of have been assuming that all of the main AI companies are probably hiring hand over fist. And I guess I know Anthropic's received big investment from Amazon, maybe some other folks as well. But does it feel like the organisation is growing a lot, that there's lots of new people around all the time?

Yeah, growth has been very rapid. We recently moved into a new office. Before that, we'd run out of desks, which was an interesting moment for the company. It was very crammed. Now there's space. Yeah. Rapid growth is a very difficult challenge, but also a very interesting one to work on. I think that's to some degree what I spend a lot of my time thinking about, is just how can we grow the team and be able to maintain this sort of linear growth in productivity is sort of the dream, right? If you double the number of people, you get twice as much done, and you never actually hit that, but it takes a lot of work because there's now all this communication overhead and You have to do a bunch to make sure everyone's working towards the same goals, sort of maintain the culture that we currently have.

I've given you a lot of time to talk about what's great about Anthropic, but I should at least ask, what's kind of worst about Anthropic? What would you most like to see improved?

I think honestly, the first thing that comes to mind is just the stakes of what we're working on or something. I think that there was sort of a period a few years ago where I felt like safety is really important. I felt like Motivated, and it was a thing I should do and got value out of it. But I didn't feel this sort of, oh, it could be really urgent. This might feel like decisions I'm making are just really high-stakes decisions. And I think Anthropic definitely feels high stakes. I think it's often portrayed as this doomy culture. I don't think it's that. I think there is a lot of benefits, and I think I'm pretty excited about the work I'm doing, and it's quite fun on a day-to-day basis. But it does feel very high intensity. And kind of, uh, many of these decisions are— they really do matter, right? If you really think we're going to have the biggest, like, technological change ever, and how well that goes depends in a large part on, like, how well you do at your job on, like, that given day.

No pressure.

Yeah. Um, and, and the timelines are really fast too, right? Even commercially, you can kind of see that it's like months sort of between, between major releases. And that sort of— that puts a lot of pressure where if you're trying to, like, keep up with the frontier of AI progress, It is quite difficult and it relies on sort of success on very short timelines.

Yeah. So if there's someone who has relevant skills, might be a good employee, but maybe they struggle to operate at super high productivity, super high energy all the time, could that be an issue for them at a place like Anthropic where it sounds like there's a lot of pressure to deliver all the time? I guess potentially internally, but also just the external pressures are pretty substantial.

Yeah, I think that some part of me wants to say yes, I think it's really important to be very high performing a lot of the time. The standard of always do everything perfect all of the time is not something anyone meets. And I think it is important sometimes to just keep in mind that all you can do is your best effort. So we will mess things up and even if it's high stakes and that's quite unfortunate, it's unavoidable. No one is perfect. So I wouldn't sort of set too high of a, like, oh, I couldn't possibly handle that. I think people really can, and you can grow into that and get used to that level of pressure and how to operate under it.

All right, I guess we should wrap up. We've been at this for a couple of hours, but I'm curious to know, what is an AI application that you think is overrated and maybe going to take longer to arrive than people expect? And maybe what's an application that you think might be underrated and consumers might be really getting a lot of value out of surprisingly soon?

Overrated. I think on overrated, I think people are often like, oh, I'll never have to use Google again, or it's a great way to get information. And I find that I still, if I just have a simple question and I want to know the answer, just Googling it will give me the answer quickly and it's almost always right. Whereas if I could go ask Claude, but it'll take a piece, it'll sample it out and then I'll be like, is it true? Is it not true? It's probably true, but it's in this conversational tone. So I think that's one that doesn't yet feel like the strengths. The place where I find the most benefit is coding. I think this is not a super generalizable case or something, but if you're ever writing software or if you've thought, I don't know how to write software, but I wish I did, The models are really quite good at it. And if you can kind of get yourself set up, you can probably just write something out in English and it will spit out the code to do the thing you need rather quickly. And then the other thing is problems where I don't know what I would search for. I have some question, I want to know the answer, but it relies on a lot of context. It would be this giant query. Models are really good at that. You can give them documents, you give them huge amounts of stuff and explain really precisely what you want, and then they will interpret it and give you an answer that accounts for all the information you've given them.

Yeah, I think I do use it mostly as a substitute for Google, but not for simple queries. It's more like something kind of complicated where I feel like I'd have to dig into some articles to figure out the answer. I think one that jumps to mind is Francisco Franco was kind of on the side of the Nazis during World War II, but then he was in power for another 30 years. Did he ever comment on that? What did he say about the Nazis later on? And I think Claude was able to give me an accurate answer to that, whereas I I could have spent hours maybe trying to look into that, trying to find something. The answer is he mostly just didn't talk about it.

My other favourite one, which is a super tiny use case, is if I ever have to format something and do something, like if there's just some giant list of numbers that someone sent to me in a Slack thread and it's bulleted and I want to add them up, I can just copy paste it into Claude and say, add the things up in any format. It's very good at taking this weird thing, kind of structuring it, and then doing a simple operation.

So I've heard it's really good. All of these models are really good at programming. And I've thought, I mean, I've never programmed before, really, and I've thought about maybe I could use them to make something of use. But I guess I'm at such a basic level, I don't even know where. So I would get the code and then where would I run it? Is there a place that I can look this up?

Yeah, I think you basically want to just look up, I would suggest Python, get an introduction to Python and get your environment set up. You'll eventually run Python and then some some file and you'll hit enter and that will run the code. And that part's annoying. I mean, Claude could help you if you run into issues setting it up, but once you have it set up, you can just be like, write me some code to do X, and it will write that pretty accurately. Not perfectly, but pretty accurately.

Yeah, I guess I should just ask Claude for guidance on this as well. So I've got a kid, a couple of months old. I guess in 3 or 4 years' time, they'll be going to preschool and then eventually starting reception, primary school. I guess my hope is that by that time, AI models might be really involved in the education process, and kids will be able to get a lot more kind of one-on-one. Maybe it would be very difficult to keep a 5-year-old focused on the task of talking to an LLM, but I would think that we're close to being able to have a lot more individualised attention from educators, even if those educators are AI models. And this might enable kids to learn a lot faster than they can when there's kind of only one teacher split between 20 students or something like that. Do you think that kind of stuff will come in time for my kid first going to school, or might it take a bit longer than that?

I can't be sure, but yeah, I think there will be some pretty major changes by the time your kid is going to school.

Okay. Yeah, that's good. That's one that I really don't want to miss on the timelines. I'm worried, like Nathan Labenz, I'm worried about hyperscaling, but on a lot of these applications, I really just want them to reach us as soon as possible because they do seem so useful. My guest today has been Nick Joseph. Thanks so much for coming on the 80,000 Hours Podcast, Nick.

Thank you.

If you're really interested in the pretty vexed question of whether, all things considered, it's good or bad to work at the top AI companies, if you want to make the transition to superhuman AI go well, our researcher Adam Koehler has just published a new article on exactly that titled Should You Work at a Frontier AI Company? You can find that by Googling 80,000 Hours and Should You Work at a Frontier AI Company? Or heading to our website, 80000hours.org, and just looking through our research. And finally, before we go, just a reminder that we are hiring for 2 new senior roles at 80,000 Hours, a head of video and a head of marketing. And you can learn more about both of those at 80000hours.org/latest. Those roles would probably be done in our offices in central London, but we are open to exceptional remote candidates in some cases. And alternatively, if you're not in the UK but you would like to be, we can also support UK visa applications. The salaries for these 2 roles would vary depending on seniority, but someone with 5 years of relevant experience would be paid approximately £80,000, something like that. The first of these 2 roles, the head of video, would be someone in charge of setting up a whole new video product for 80,000 Hours. Obviously, people are spending a larger and larger fraction of their time online watching videos on video-specific platforms, and we want to explain our ideas there in a compelling way that can reach the sorts of people who care about them. That video program could take a range of forms, including 15-minute Director Clamour vlogs, lots and lots of 1-minute videos, maybe 10-minute explainers. Maybe that's probably my favourite YouTube format. Or alternatively, lengthy video essays. Some people really like those. The best format, that'll be something for this new head of video to look into and figure out for us. We're also looking for a new head of marketing to lead our marketing efforts to reach our target audience at a large scale. They're going to be setting and executing on a strategy, managing and building a team, team, and ultimately deploying our yearly marketing budget of around $3 million. We currently run sponsorships on major podcasts and YouTube channels. Hopefully you've seen some of them. We also do targeted ads on a range of social media platforms, and collectively that's gotten hundreds of thousands of new people onto our email newsletter. We also mail out a copy of one of our books about high-impact career choice every 8 minutes. That's what I'm told. So there's certainly the potential to reach many people if you're doing that job well. Applications will close in late August, so please don't delay if you'd like to apply for those ones. 80000hours.org/latest. All right. The 80,000 Hours Podcast is produced and edited by Keiran Harris. Audio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong. Full transcripts and extensive collection of links to learn more are available on our site and put together, as always, by the legend herself, Katie Moore. Thanks for joining. Talk to you again soon. Bye.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.