AI Deception, Interpretability, and Affordances with Apollo Research CEO Marius Hobbhahn

So the more pressure we add, the more likely the model is to be deceptive. So kind of in the same way in which a human would act, it also acts. Removing pressure and adding additional options will very quickly decrease the probability of being deceptive. Open source has been really good so far in many, many ways. It has been very positive for society, right? I think a lot of ML research could not have happened without open source. A lot of safety research could not have happened with open source. At some point, the system is so powerful that you don't want it to be open source anymore in the same way in which You know, I don't want to open source the nuclear codes, or like, you know, literally the recipe to build the most viral pandemic or something. The labs maybe have the incentive to not say the worst things they found, because otherwise they may lose their contract. So you need something like the UK AI Safety Institute or the US AI Safety Institute to make sure that there is a minimal set of standards that all the auditors have to adhere to.

Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today my guest is Marius Habhan, founder and CEO of Apollo Research, a nonprofit AI safety research group that is working to understand both how AI systems behave and why. Their approach combines exploratory and hypothesis-driven testing, fine-tuning experiments, and interpretability research. And as you'll hear, they place special emphasis on the potential for AI systems to deceive their human users. In this conversation, we look first at Apollo's starting framework for their work, which emphasises the importance of affordances in AI systems. That is, through what tools, actuators, or other means can the system affect the broader world? And they also introduce a number of new conceptual distinctions meant to help people have more precise and productive conversations about these nuanced topics. Then in the second half, we look at their first research result, which demonstrates, to my knowledge, for the first time in a realistic, unprompted setting, that GPT-4, when put under pressure, will sometimes take unethical and even illegal actions and then go on to lie to its users about what it did and why. This is an important result demonstrating that while the risk from AI systems may start with and may even be dominated by intentional human misuse, the models themselves can also misbehave in unexpected ways. As an aside, since I told my behind-the-scenes GPT-4 red team story a few weeks ago, a number of people have reached out to ask me how they too can get involved with red teaming projects. Unfortunately, as commercial competition and secrecy both continue to ramp up across the space, I don't see as many open calls for volunteer red teamers as I used to, certainly not for unreleased frontier models. Instead, the field is becoming more professionalized, with all the leading labs, as well as the data companies like Scale AI, plus the independent auditing organizations like Apollo, Archival, now known as Meter, Palisade, and also AI Forensics, all actively hiring research scientists and engineers in this area. So does that mean that there's no longer a role for the independent hobbyist red teamer to play? On the contrary, there is a ton left to discover, even on publicly released models. And the best way to break into the field is to demonstrate your ability to discover new phenomena. Importantly, the work we cover in this episode could have been done by anyone with an OpenAI account, a knack for prompting, and just a tiny bit of coding know-how. No special access or advanced machine learning techniques were required, just a lot of curiosity. With that in mind, if you want to get into this line of work but aren't sure where to start, I encourage you to reach out. I'll be happy to help brainstorm or refine your project ideas, and I can also help connect you with folks at the top companies who do sometimes provide API credits to independent researchers working in this area if and when you can achieve a meaningful result. As always, we appreciate the time that you spend listening to The Cognitive Revolution, and we hope it's a valuable guide to the AI era. If you feel that it is, we would love a review on Apple Podcasts or Spotify, and we of course encourage you to share the show with your friends. Now, here's my conversation on frontier AI safety work with Marius Habhan of Apollo Research. Marius Habhan, founder and CEO of Apollo Research, welcome to The Cognitive Revolution.

Hey, thanks for having me.

I am very excited to have you. So regular listeners of the show will know that I'm a big believer in the importance of hands-on testing of what AI systems can do, and also that I have been a pretty enthusiastic consumer of the news when some of the leading labs have made public commitments to allow organisations outside of their own teams to look at the systems that they're building before they get deployed. And so your work with Apollo Research, which is trying to build, as I understand it, an organisation to meet that need and actually work with those leading labs, in part at least, on understanding the systems that they are developing before they get to widespread deployment, I think is super interesting, and I'm very excited to unpack the details of it with you. Maybe for starters, you want to just kind of give us the quick overview on Apollo Research, like how you decided to set out to found it. I'm interested a little bit in the timeline of how that related to some of the commitments that the labs have made and what you guys are trying to do in the big picture?

I think on a high level, it's sort of trying to understand what is going on in AI systems. And the reason for this was, or still is in fact, I basically think right now we just lack information to make good decisions. There's loads of uncertainty that we have about what could go wrong, whether we are already at a point where things go wrong. Or how far away we are from these points. And yeah, we are trying to reduce this uncertainty. And this is mostly through research, auditing, and governance. And on the research side, it's really split between interpretability and behavioral evals, half-half. But in the long run, we really want to merge them both because I basically think what we need in the long run is both a mixture of behavioral and interpretability evals so that we can really understand what the model is doing and then also why it is doing this in the first place, because each of them individually seems somewhat insufficient. And yeah, maybe to go into the origin story, it has actually nothing to do with the commitments of the different labs. It was mostly that at the beginning of this year, there are just I kind of felt like I had a pretty clear picture of what is lacking in the current space with deceptive alignment and evaluating deceptive alignment or models for deceptive alignment in the first place. And interpretability and evals just seemed like the obvious things to do. So in the beginning, we basically set out to do mostly research and only then sort of over time we realized, hey, this is something that should be applied in the real world as soon as possible because we are Systems are getting better all the time and we may actually hit this point fairly soon where models are already about at the threshold of capabilities for deceptive alignment. And then there is a small part in the organization that is governance, which originally we also didn't really intend to do for the first 2 years or something because we thought we really need to understand all the research very well. before we can talk to the people in governments and decision makers and lawmakers, because otherwise we're telling them things we aren't super confident in. And then lots of things happened. Governments and lawmakers actually got interested in AI and AI safety in particular. And then when we talked to them, we realized the difference. We are very, very well placed to talk about these things because if you have thought about them for sort of in the background for like 6, 7 years, and then specifically about some topic for 6 months or so, you are among the world's experts. And this is kind of more like a reflection of the state of how bad it is about AI safety, where people in my position are actually sort of accidentally becoming the experts rather than people with 10 and 20 years of experience, because there just aren't a lot of people in the world who have thought about AI safety for more than a couple of years, if at all.

Yeah, I can definitely relate to that sort of accidental expert status. I never expected to be where I am and doing the things that I'm doing. But yeah, the whole AI field in some ways is kind of the dog that caught the car. I always kind of come back to that metaphor where it's like we were just trying to build a bit more powerful AI and all of a sudden we built a lot more powerful AI. And now we really kind of have to figure out what to do with it. So even a little bit of advanced planning is better than, or a little bit of advanced thought is a lot better than where most people are starting. Had you seen, when you actually started the organisation, had you seen GPT-4, or were you basing this decision on just what was public at the time?

Only what was public at the time. So the decision was made in February 2023, or at least sort of my internal commitment was made to this. I'm not sure whether GPT-4 was public already at the time. Not quite, right? It was March. So no, it was independent of GPT-4.

Yeah, I always think that's interesting just because GPT-4 was such a wake-up moment for so many people, and certainly I would include myself in that. I was already extremely plugged into what was going on and using it and fine-tuning tons of models on the OpenAI platform in particular. But then it was like, whoa, this thing is next level. It's not slowing down. We've gone from sort of, I can put a lot of elbow grease in and get a fine-tuned model to do a particular task, which already I thought was going to be economically transformative, to I don't even need to do that, that I can just ask for a lot of these tasks and get pretty good zero-shot performance. For me, that was the moment where I was kind of like, okay, this is going from a tool that I am really excited to use and having a lot of fun using to something that seems like a force that needs to be understood from all angles. So let's unpack the perspective that you are bringing to this. I would encourage folks to look up these papers that we'll discuss and read them for themselves as well. But on the website, you've got 2 recent publications. One is kind of a framework for organising the work that you're going to do, and then the other is like a very detailed, in-the-weeds investigation of a particular AI behaviour, namely deceiving the user, which I think is a super interesting and important one to study. But let's maybe just start with the big picture, like organising the thoughts. I get the sense that you think, again, well, you've kind of said this, And the paper certainly reflects it, that there are a lot of big questions that remain unanswered. So how do you structure your approach to this topic, given all the uncertainty that exists?

Yeah, maybe to give a little bit of context. So this is only one paper of many in this space. And there is, I think, earlier this year, there was a really big one called Model Evaluation for Extreme Risk, which Yeah, we at Apollo definitely thought was a pretty good paper. And they're sort of pointing out many of the very reasonable and important steps or reasonable principles for external auditing, something like ramp up the auditing before you ramp up the exposure to the real world and do this ahead of the curve, so to speak. But when we read the paper, we felt a little bit like this makes sense for the current capabilities and sort of how current models are being built.

Yeah.

But if we think ahead of what the next couple of years should look— or not should, could look like, then yeah, there are loads of open questions. And we were trying to understand how do they fit into this framework? Because yeah, we internally were trying to make sense of this in the first place. So just to give you a couple of them, what happens if your model has the ability to do online learning? How often do you have to audit it? Should you re-audit it during the online learning? If yes, how often? What if you give the model access to the internet or to a database or to anything like this? Yeah, I think a model with and without access to the internet is basically 2 very, very different models. The one with access to the internet is just so much more powerful if it can use it even on a very basic level. So yeah, it feels like if you give your model affordances like this, you kind of have to rethink how dangerous it is and where the danger comes from, because it suddenly is like a totally different threat model potentially. And so what we did for the paper, and really the credit should go to Lee Sharkey here, who is my co-founder, who has done most of the hard work, if not all of the hard work for this paper. And so what we were doing is thinking from first principles, where does the risk come from? And what changes to the AI system do create new risks? And then basically the answer is, well, we have to audit wherever risk is created. And then the more we looked into this, the more we realized, well, there are actually a lot of places where new risk comes into the system, at least potentially, and therefore where audits, at least in an ideal world, should happen. There are obviously some constraints. But I think if we think about where are we 5 years from now, then I think, yeah, if there is actually a big auditing ecosystem around this, then there will be very, very many different organizations auditing really different places. And then the other point of the paper was just to define many concepts and create the language to discuss all of these things, because we had many internal discussions where we were like, oh, the thing we mean is this. And then we had an example and then we kind of needed a name for it and there wasn't really a name. So we decided, okay, let's define all of the relevant terms for this and then sort of have a language to talk about this in the first place.

Hey, we'll continue our interview in a moment after a word from our sponsors. So let's dig in, in a little bit deeper detail. I like the premise that you set out with in the paper, which is to work backward from AI effect in the real world and try to imagine where are these effects going to happen, and then how can we get upstream of that and help shape them in a positive way? I would be interested to hear you kind of describe that backward chaining process in a little bit more detail. And then I thought some of your concepts also were really helpful clarifications and distinctions. So maybe you can highlight some of the ones that you think are most useful that you'd like to see get into broader circulation as well.

So basically we started from, okay, the system, AI system will interact with the world in a particular way. And then there are many different ways in which it can interact with the world. And then there is sort of a like a whole chain of things that have had to happen until the model can interact with the world in this particular way. So maybe it has been fine-tuned, maybe it has been given access to the internet. Before that, it has to have been trained. Before that, there has to have been the decision that this model should be trained in the first place. And so the question is, what are the kind of important decisions at all of these different points in time And how then can we ensure that people actually make decisions that will lead to outcomes at the end of the chain, such that the model or the system interacts with the world in a safe manner? And this is maybe the first distinction that is worth pointing out, or the reason why I'm correcting myself all the time is there's really a difference between AI model and AI system. The AI model really— and this is not something we came up with, this already exists before. But I think it's worth pointing out and sort of getting in, like really hammering into people's head when they think about AI. So the AI model is just the weights, maybe behind an API, but even with the API, it's kind of already a system. And the system then is sort of the weights plus everything around it. So there could be scaffolding, there could be access to tools, this could be content filters. This could even be like just an API, retrieval databases, et cetera, like really the full package. Where you say, okay, there's stuff around the weights that increase the capabilities of the model in many, or at least change the capabilities of the raw model in some sense, not necessarily always increasing. Filters, for example, may decrease it. And then there are sort of other weird ways, or like, yeah, once you think about this, there are sort of a couple of other concepts that feel important to clarify. Because when people say capabilities, This can mean very different things, right? And so we categorize this into 3 different classes. The first one is absolute capabilities, which we think of basically the hypothetical capabilities given any set of affordances. So if you have GPT-4 without the internet, right, then in the space of absolute capabilities would be a GPT-4 with internet. So, or like things that this model could do. So the question is like, if we give additional things to the system, how big is the space of actions it could take? So, you know, and then obviously there's a question of like, how imaginary do we get here? You know, like, does it have— does it get access to like, you know, a Dyson sphere? Or does it get access to like a government or something like this? But, but yeah, like, it basically points out sort of the, what could this model do if we gave it a lot of things? Everything that we can basically think of. Thinking about this in the first place only makes sense for models that have become more general, like the GPTs, because for an MNIST filter or for an MNIST classifier, this doesn't make any sense. An MNIST classifier plus internet is exactly as capable as just the MNIST classifier itself. But yeah, for systems that are more general, suddenly you have this difference between things that only the system can do, Mm-hmm. Or like the basic system plus things that you could do hypothetically with a lot of additional affordances. Then the second one is contextual capabilities, which is things that are achievable in the context right now. So for example, with ChatGPT, you can enable it to have access to tools and then you can browse the web. And this is something that it can do right now. You don't have to add anything on top of this. And this is sort of the smallest category of things, which you can do without any additional modification. And then reachable capabilities is contextual capabilities plus achievable through extra effort. So for example, this could mean ChatGPT itself may not have access to a calculator, but if it has access to the internet, it can Google and then find a calculator and then use that calculator. And so it's sort of a 2-step process, right, where it has to use one affordance or capability to then achieve another. And so this is what we call reachable capabilities. And yeah, so the reason why we are making all of this differentiation, even though it sounds maybe a little bit too much in the weeds, is when people talk about capabilities and regulating capabilities and designing laws for 4 capabilities, the question is which ones, right? Do you mean the contextual capabilities? So the ones that the model has literally right now, or the reachable capabilities, so which the model could reach with additional effort, or the absolute, like the maximum potential space of capabilities? And right now this may sound like we are too much in the weeds, but I think in a few months this will sound very, very relevant suddenly because— The models will be more capable and then they will actually be able to just be smart enough to use the internet to find additional tools that they can then use or convince someone to give them access to a shell and then use that because they're already, they can learn it in context or they know it anyways. And at that point, really the question is what should the auditors audit for? Which capabilities? And that becomes pretty quickly a very, very big space of things, right? So if the auditor not only has to think about what kind of tools do you give the AI, but also what kind of tools could the AI get access to through some means, suddenly you have this whole space of thousands of things it could do. It's really a question of, or a trade-off between what is plausibly doable in the real world versus how much risk can we actually mitigate. And I'm honestly very unsure about where we're heading at this point.

So just to riff on and kind of emphasise some of the value that I see in some of these distinctions, I think it's helpful to clarify the difference between a model and a system. I think there is a tremendous amount of confusion online, and to my chagrin, I've probably even contributed to some of it at times where people are like, ChatGPT was doing this for me and now it's not anymore. And I've sometimes said, well, they haven't updated the model, so it probably hasn't changed that much. And I think what I've maybe neglected in some of those moments is like, but they might have changed the system prompt. Or as we're seeing, even just this last couple of weeks, there's been this really interesting phenomenon of GPT-4 getting, quote unquote, lazier. And people are speculating that maybe that's because they feed the date into it and it knows that we're in December and it knows that people don't work as hard or as productively in December. And so maybe it's like kind of phoning it in because it's like imitating the broad swath of humans that it's seen kind of work halftime in December or whatever. I've even seen some experiments just in the last couple of days that suggest that there might even be real truth to that. Who knows? I'd say the The question remains open, but there is an important difference, and it's worth getting clarity on the model itself with static weights not changing versus even just a system prompt that can perhaps have even unexpected drift along the dimension of something as seemingly benign as today's date. So that's important to keep in mind. The levels of capabilities, I think, are also really interesting. And I want to ask one kind of— I have a couple of questions on this, but I think I have a clear sense of what is meant by contextual. What can it do now given the packaging? Right. What can GPT-4 do in the context of ChatGPT where it has a code interpreter and it has Browse with Bing and it has the ability to call DALL-E 3 to make an image and probably a couple of other things that I'm not even remembering, plugins perhaps as well, right. Which obviously, and GPTs, which proliferates all the affordances all that much more. On the other end, I feel like I sort of understand absolute, which is like a theoretical max. Could you give me a little— how do I understand reachable as kind of between those? What's the distinction between reachable and absolute?

Yeah, so maybe one way to think of it is the contextual capabilities are the ones kind of that a user explicitly gave it. And then the reachable ones are those that may also be reachable without the user even having thought about that the model actually will use them, right? So if you say, if the model would be able to browse the web entirely on its own, which I'm not sure it currently can do, or what exactly the restrictions on Search with Bing are, but if it was able to do that, you may not may not have realized that it has a reachable capability through the internet of firing up a shell somewhere or renting a GPU and doing or running a physics simulation through an online physics simulator, if that's something that's available. And so this is sort of like which tools can it reach through the contextual capabilities that it already has been given by you.

Gotcha. Okay. So like solving a CAPTCHA by hiring an Upwork contractor, for example.

Exactly.

To take one infamous case. So, okay, here's a challenging question, and I don't necessarily expect an answer, but maybe you could venture an answer, or you could just kind of describe how you begin to think about it. What would you say are the absolute capabilities of GPT-4?

Yeah, very unclear. So I think they're definitely not infinite, as in even with extremely good scaffolding and access to the internet and many other things, I think people haven't been able to get it to do economically valuable tasks at the level of a human, at least for long time spans, for example. So the question is obviously, are we just too bad and have we not figured out the right prompting yet and the right scaffolding and so on? Or is this just a limitation of the system? And my current guess is there is probably a limit to the absolute capabilities and it's probably lower than what a human can do, but we are not that far away from it. So I think with additional training, with additional, specifically LM training that is more goal-directed or makes it into more goal-directed and an agent and better scaffolding, I think there will be ways in which the absolute capabilities could increase quite a bit in the near future. Yeah. Does this make sense?

Yeah. I mean, it's hard, right? Certainly listeners to the show will know from repeated storytelling on my part that I was one of the volunteer testers of the GPT-4 early model back in August, September of last year. And I really kind of challenged myself to try to answer that question independently. Like, what is the theoretical max of what this thing can do? How much could it break down big problems and delegate to itself? And I basically came to the same conclusion that you did, which is doesn't seem like it can do really big tasks. I mean, again, it's confusing, right? Because then you could also look at the dimension of how big the task is versus how would you break it down. And just in the last week, I've been doing something for a very sort of mundane project, but actually using GPT-4 to run evals on other language model output. I have found that if I have like 10 tasks, 10 dimensions of evaluation, and I ask it to run all of those, it is now capable of following those directions and executing the tasks one by one, but the quality kind of suffers. It sort of makes mistakes. It sometimes muddies the tasks a little bit between each other, and it's definitely not at a human level given 10 tasks to do in one generation. On the flip side, though, if I take it down to one task per generation, Which I didn't want to do because that will increase our cost and latency and just is less convenient for me. But then it kind of pops up to, honestly, I would say pretty much human level, if not above. So there's interesting dimension. I guess it seems pretty— the sort of magnitude of the task seems like a pretty important dimension for evaluating a question like absolute capabilities, right? It's like if it's a super narrow thing, it's It's capable of some pretty high spikes, but if it's a big thing, it kind of gets lost. Would you refine that characterization at all?

Yeah, I'm not sure how to think about it, honestly. So I think of absolute capability as really more of a sort of theoretical bound that we are probably not going to approximate in practice, even if we test a lot, a lot. And then breaking it down into different tasks, I'm not sure. I feel like this is a different capability then, right? The capability of doing 10 things at once is a different thing than the capability of doing one thing 10 times, like 10 different things, but one by one. So yeah, I would say it's basically you're talking about different capabilities then, at least in this framework.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, and it is not that good at decomposing the tasks. So I've also kind of experimented a little bit with, like, can you give it that list of 10 tasks and can it break them down and self-delegate with an effective prompt? And it's like maybe a little bit closer there, but still not getting nearly as good results as if I just roll up my sleeves and do the task decomposition. So you mentioned that you expect this frontier to obviously continue to move. One way to ask the question is, what is Q*? But a more sensible way to ask the question is, do you have a set of expectations for how the capabilities frontier will move? I definitely look at things like OpenAI's publication from earlier this year where they started to give denser feedback on kind of every step of the reasoning process, and they achieved some state-of-the-art results on mathematical reasoning that way. And when I think about affordances and I think about the failure modes that I've seen with these GPT-4 agent type systems, I think, man, if you apply that to browsing the web and using APIs, and it seems like that stuff is ultimately a lot less cognitively demanding than pure math, it seems like we probably are going to see And I would guess that it's maybe already working pretty well. AGI has been achieved internally. I don't know about AGI, but I would expect that some of this stuff is already pretty far along in kind of internal prototyping. But how does that compare to what you would expect to see coming online over the next few months?

Yeah, I mean, it's obviously hard to say, and I can only speculate. I think on a high level, what I would expect the big trends to be, and also what we are kind of looking forward to evaluating is LM agents. I think this is pretty agreed upon. From first principle, I think it also makes sense. It's just like, where does the money come from? It is from AI systems doing economically useful tasks. And often economically useful tasks just require you to do things independently, being goal-directed over a longer period of time. And the longer a model can do things on its own, the more money you can squeeze out of it. So I definitely think just from financial interests, all of the AGI labs will definitely try to get in more agentic ways. How far they have come, I don't know. But yeah, I expect that next year we will see quite some surprises. Then multimodality is the other one where Yeah, I think people kind of like over the last couple of years with more and more multimodal models, people just realized like kind of it's not that different from just training text, right? It's sort of you plug in the additional modalities, you change your training slightly, but it's not much more than that. Like obviously, it's obviously hard in practice, right? There's a ton of engineering challenges and so on. But on a conceptual level, there isn't any big breakthrough needed. So people will just add more and more modalities on bigger and bigger models and train it all jointly end-to-end. And it kind of just works. And then tool use is the last one. And that I think people actually were quite surprised by how easy it was to get to this level. So yeah, I think When people realized, oh, these language models are already pretty good, how fast do they learn how to use any tool we can think of? They were surprised by how fast they learned the tools. And now it's mostly a question of really baking in the tool into the model in a way that it's very robustly able to use the model rather than just a little bit or just showing that it sometimes works. But yeah, I think if you have an LM agent that is multimodal, And that has very good tool use, I'm not quite sure how far you are away from AGI. At that point, you kind of have almost all of the ingredients ready and then it's really just a question of how robust is the system. So yeah, I think these are the trends we see right now and this is also why many people in the big labs have very, very short timelines because they can think 2 years ahead and sort of where this is going or maybe even just 1 year ahead. I don't know.

When you talked about the surprise, like people were surprised at how easy it was to get tool use to work. Are you referring to people in the leading, the usual suspects of leading developers?

It's hard to say. I mean, I can only speculate on this, but the ToolFormer paper was published 3 months before OpenAI just released their tool use. And I mean, they probably had been working on this before, but still, from having the scientific insight to to publishing this and releasing this in the real world. I think there just was less work involved than is typical for most of the bigger AI development cycles. I could be wrong on this. This is more hearsay. So yeah, take it with a grain of salt.

Yeah, it seems right to me as well. And I agree with you, the emphasis on multimodality as a new unlock makes a tonne of sense. Even just in this kind of agent paradigm of can I browse the internet or whatever. I've done a lot of browser automation type work in the past, and the difference between having to grab all the HTML that is often these days extremely bloated and kind of semi-auto-generated and in some cases deliberately generated to be hard to parse from the Googles and Facebook. They don't want you scraping their content, so they're kind of not making it easy on the browser automators. The difference between that and just being able to look at the screen and understand what's going on, you kind of put it through a human lens and you're like, yeah, it's a hell of a lot easier to see what's going on on the screen than to read all this HTML. And sure enough, the models kind of behave similarly. I remember for me looking at the Flamingo architecture when that was first published, I think April of 2022. So a little more than a year and a half ago now, and just thinking like, oh my God, if this works, everything's going to work. It was like they had a language model frozen, they had kind of stitched in the visual stuff and kind of added a couple of layers, but It really looked to me like, man, this is tinkering stage and it's just working. So, like you, I don't want to dismiss the fact that there is obviously a decent amount of, I'm sure, labor and probably at times tedious labor that has to go into overcoming the little stumbling points. But conceptually, it is amazing how simple a lot of these unlocks have been. over the last couple of years. And you see this too in just the pace at which people are putting out papers. You look at the one team that I follow particularly closely is the team at Google DeepMind that is doing medical-focused models, and they're good for one every 3 months. And they're like significant advances where it's like, oh yeah, this time we added multimodality and this time we tackled differential diagnosis. And again, it seems like There's not a lot of time for failures between these successes. So it does seem like, yeah, we're not at the end of this by any means just yet. A lot is coming at us. It's going to presumably continue to get weird. You're trying to push both as much as you can the understanding of what can the systems do as users, what are their limits? And then at the same time, you're trying to dig into the models. This is the interpretability side, and figure out what's going on in there and can we kind of connect the external behaviours to these internal states. So tell us about that side of the research agenda as well.

Yeah. So on the interpretability side, my thinking is basically it would be so great if interpretability worked, right? It would make So many questions easier. Like if you ask questions on accountability, right? If you have causal interpretability method, you would be able to just tell the judge if the model would have— if we would have changed these variables, the model would have acted differently in this way. And we could just basically solve that. Biases, probably also like social biases, much easier to solve because you could intervene on them or fix the internal misunderstandings and concepts. It's also extremely helpful for basically all of the different extreme risks, right? It would be much easier to understand the internal plans and how it thinks about problems, how it approaches them and so on. And then it would also make iterations on alignment methods much, much easier, I think. As in, let's say somebody says, oh, RLHF is already working. We see this in practice. Then you could use the interpretability tools to test Does RLHF actually work or does it only superficially hide the problem or something like this? Or does it actually deep down solve the root? And then I think my biggest— sort of the biggest reason for me for focusing on interpretability in the first place is deceptive alignment, where models appear aligned to the outside and to the user, but internally they actually follow different goals. They just know that you have a different goal. And therefore, in order for you to think it is nice, they act in that way. And yeah, I basically think almost, not almost all, but a lot of the scenarios in which AI goes really, really badly go through some form of deceptive alignment, where at some point the model is seen as nice and people think it is aligned and people give it access to the internet and resources and train it more and more and more and make it more powerful. But internally, it is actually pursuing a different goal. And it is smart enough to hide this true intention until it knows that it can sort of cash out and then follow on this actual goal without us being able to stop it anymore. So yeah, that's what I'm really worried about. And interpretability obviously seems like one of the most obvious ways to test for deceptive alignment or to at least investigate the phenomenon. Because you know what it's thinking inside. There are still some cases where deceptive alignment, where even with good interpretability tools, deceptive alignment could still somehow be a thing. But generally speaking, I think it would be much, much, much harder for the model to pull off. So right now, I think interpretability is just not at the— it's not practically useful. So we cannot use any existing interpretability tool and throw it on GPT-3 or GPT-4 because none of them are developed enough that they give us insight that really meaningfully change our minds. And so yeah, this is why our agenda is separate in the first place between behavioral evals and interpretability, despite us wanting to do them jointly in the long run. But given that there is such a huge gap on applicability, I think that this is definitely a problem that we're trying to mitigate here. And then the one question for me is also like, how hard will interpretability turn out to be? And there are various people have argued that interpretability will be extremely hard because models are so big and complicated, and therefore it will be hard to enumerate all of the concepts and actually understand what the hell is going on inside. And I'm more of the perspective that I understand the reason why they think it's hard, but I also think there are many reasons to assume it's going to be doable. If we put our minds to it as humanity, we'll probably figure it out. The primary reason I think is we have full interventional access to the model, right? We can see Every activation, we can ablate everything we want. We have, it's not just observational studies, you can really intervene on the system. And generally speaking, I would say, as soon as you can intervene on the system, you can test your hypothesis very, very quickly and you can iterate very fast. And so I think we will be able to figure out interpretability in the next couple of years to an extent where we can actually sort of say, It is now useful on real-world models, on frontier models. How expensive this is going to be, I don't know yet, but I think it will at least be technically feasible.

Yeah, I've definitely updated my thinking a lot in that direction from a pretty naive, just kind of, hey, it sounds really hard, black box problem, nobody knows what's going on in there, to today I would say, wow, there's really a lot of progress. The progress of interpretability over the last, say, 2 years has definitely exceeded my expectations and given me a lot more— I wouldn't maybe stop short of confidence, but at least reason to believe that with some time, but not necessarily a ton of time, that we really could get to a much better place in our understanding. So I'm with you on that. I have a number of of follow-up questions, I think, on this point. One, let's maybe just give the account for why deception might arise in the first place. I'll give you a super simple version. You can refine it or complicate it. I usually kind of cite Ajeya on this, and she has a pretty simple story of what the model is trying to do, what it is rewarded for in the context of an RLHF like training regime is getting a high feedback score from the user. And it probably becomes useful as a means to maximising that score, to model human psychology as an explicit part of how you're going to solve the problem. Right. And I think certainly humans do this with respect to each other, right? I ask you for something, you ask me for something. We interpret that not only as the extremely literal definition of the task, but also kind of have a sense for what does this person really care about, what are they really looking for, and we can incorporate that into the way that we respond. It certainly seems like the heavier you do, the more emphasis you put on this kind of reinforcement learning from human feedback, the more likely the models are to start to create a distinction between the task as sort of narrowly objectively scoped, let's say, and the kind of human psychology element that is going to feed into its rating. And then if you have that decoupling, then you have kind of potential for all sorts of misalignment, including deception. How does that compare to the way you typically think about it?

Yeah, I think this kind of version through RLHF, Is one potential path. I actually think the jury is still out there on this. I definitely see the hypothesis and where it's coming from, but I could also just totally imagine that the training signal is sufficiently diverse and it updates sufficiently deep that RLHF kind of just does the thing we want it to do without the model becoming deceptive. I could also see the story in which it would become deceptive. Deceptive. I think on a very high level, the reason I think why models would become deceptive is because at some point they will have long-term goals. They will have something that they care about beyond the current episode, beyond pleasing the user at this point in time. And then the question really is, and then I think there are 2 core conditions under which the more they are fulfilled, the more likely the model is becoming deceptive. Like, how important is this long-term goal to the model itself? Meaning, how much does this goal trade off, for example, with other goals it has? So for example, if it cares a ton about something, then it's more likely to be deceptive with respect to this because it really wants to achieve this. And then secondly, how much do others care about me not, or the AI not achieving this goal in the first place? Something like contestedness, right? So for example, If I want to pick a flower and I care a lot about this, I don't need to be deceptive because nobody wants to stop me from picking that flower. If I want to be the president, a lot of people might not want me to be the president. And so, in that case, it's very contested and I have a strong incentive to be deceptive about my plans because otherwise people would want to stop me. And then so, now we're at a point where we have a system, at least in our hypothetical scenario, that is has a long-term goal and it's in the limit, at least, it cares about that goal and the goal may be somewhat contested. And then as long as it has situational awareness, it just feels instrumentally useful to be deceptive about it. Like you said, to model other people and how they would think about it and then just react to this. And this is sort of, I think this is maybe this is one of the core reasons why I'm so worried about this whole deception thing. Because it just feels like a reasonable strategy in a ton of situations from the perspective of a consequentialist or rational actor, right? It's just like, under specific conditions, people just naturally— or deception is just convergent. People do it because it makes sense for them. And this is why we see it in a ton of different systems, right? You see it in animals where parasites are really deceptive with respect to their hosts. You see it in individual humans where they're deceptive with respect to their partners from time to time, for example. You see it in systems where they're trying to game the laws and be deceptive about this or to lie about this. And I think this is kind of like the whole, or like a big part of the problem. It's just reasonable or sensible in many situations to be deceptive from the perspective of the model, which is kind of what we want to prevent, right?

So where do you think those long-term goals come from? Is it just kind of a reflection of the general training goals? I mean, we have kind of the canonical 3 H's, but I mean, honest is one of those, right? Helpful, harmless, honest. Is your understanding just that those are like fundamentally in tension and that the model will have no choice but to develop trade-offs between them?

I mean, we can get into the tension between them in a second, but I think it's actually like the 3 H's, I don't think will be— they're not keeping me awake at night. I think it's more at some point people want the model to do long-term economic tasks, and for that they give them long-term goals, or long-term goals are instrumentally useful. So For many situations, I think it will just be useful to have long-term goals, or at least to have instrumental goals, right? Something like, oh, it makes— because it is a long-term task, it makes sense to first acquire money and then use that money to do something and then use that third thing to achieve the actual goal. And so I think the models will just learn this kind of consequentialist and instrumental reasoning where they're like, okay, I first have to do X. And then I do this and then I do the long-term thing. And once they're there, sometimes it just makes sense to be like, okay, other people don't want me to do this. Therefore, I hide my actual intention and I act in ways that make me look nice despite not being nice. Yeah. But yeah, I think a lot of the reason why there will be these kind of long-term goals is either because we literally give the model long-term goals because it's economically useful from a human perspective, Or because some long-term goals are instrumentally useful to achieve other things.

Gotcha. Okay, interesting. Another thought that came to mind in this discussion of, I guess, deception broadly is like, and I've done a little bit of investigation with this and engaged in some online debates, and it leads me to propose perhaps another capability definition for you. But as I see it, a theory of mind, which is kind of a more neutral framing perhaps, is kind of a precondition for deception, right? If you are going to mislead someone, you have to have some theory of what they are currently thinking. And there's a lot of research from the last 6 to 9 months about do the current models have theory of mind? To what extent? Under what conditions? And I've been kind of frustrated repeatedly, actually, by different papers that come out and say, still no theory of mind from GPT-4, where I'm like, but wait a second. As Ilya says, the most incredible thing about these these models and the systems that we engage them through are that they kind of make you feel like you're understood, right? It definitely seems like there's some kind of pretty obvious brute force theory of mind capability that exists. And yet when people do these benchmarks, they're like, oh, well, it only gets 72% on this and 87% on this and whatever. And so that fails the theory of mind test. It's like not at a human level or whatever. Some of that stuff I've dug into and found, like, your prompting sucks. If you just improve that, then you can get over a lot of the humps. But I also have come to understand this as a difference in framing, where I think I am more like you, concerned with what is the sort of theoretical max that this thing might achieve. That seems to me the most relevant question for risk management purposes. And then I think other people are asking a similar question, but through the frame of, like, what can this thing do reliably? What can it still do under adversarial conditions or whatever? So I wonder if there's a need for another capability level that's even below the reachable that would be the sort of robust or maybe even adversarial robust robust to adversarial conditions. But I do see a lot of confusion on that, right? Like, people will look at the exact same behaviour and I'll say, damn, this thing has strong theory of mind. And professors will be like, no theory of mind. And I feel like we need some sort of additional conceptual distinction to help us get on the same page there.

I'm not entirely sure whether— or maybe it makes sense from an academic standpoint to think about this. I think From the auditing perspective, the max, the limit, the upper bound is what you care about. You really want to prevent people from being able to misuse the system at all, not just in the robust case, right? It's really about like, what if somebody actually tried? Or you want the system itself to be not only not being able to take over or like exfiltrate or something like this in a few cases, Yeah. Yeah, you basically want to limit it already at a few cases, right? You don't care about whether it does this, like, or you also care about whether it does this 50% of the time, but really you will already want to sort of pull the plug early on. So for an auditing perspective, probably this additional thing is not necessary, but from a real-world use case and sort of academic perspective, maybe there should be a different category.

Yeah, I think if only just to kind of give a label to something that people are saying when they're saying that things aren't happening or can't happen that seem to be obviously happening, we can work on coining a term for that. What's kind of the motivator for secrecy around interpretability work?

Yeah, I basically think good interpretability work is almost necessarily also good capabilities work. So basically, if you understand the system, good enough that you understand the internals, you're almost certainly going to be able to build better architectures, iterate on them faster, make everything quicker, potentially compress a lot of the fluff that current systems may still have. And yeah, we will try to evaluate whether our method does in fact have these implications. But yeah, I think basically If you have a good interpretability tool, it will almost certainly also have implications for capabilities. And the question is just how big are they?

Speaking of new architectures, though, this to me seems like the biggest wildcard. And I'm currently obsessed with the new Mamba architecture that has just been introduced in the last, I don't know, 10 days or whatever. I don't know if you've had a chance to go down this particular rabbit hole just yet, but I plan to do a whole kind of episode on it. In short, they have developed a new state space model that they refer to as a selective state space model. And the selective mechanism basically has a sort of attention-like property where the computation that is done becomes input dependent. So unlike your sort of classic, say, MNIST classifier, where you kind of run the same, given a given input, you're going to run the same set of matrix multiplications until you get to the output. With a transformer, you have this kind of additional layer of complexity, which is that the attention matrix itself is dynamically generated based on the inputs. And so you've got kind of this forking path of influence for the inputs. And this apparently was not really feasible in past versions of the state space models for, I think, a couple of different reasons. One being that if you do that, it starts to become recurrent, and then it becomes really hard to just actually make the models fast enough to be useful. And they've got a hardware-aware approach to solving that, which allows it to be fast as well as super expressive. So it seems to be, for me, it's like a pretty good candidate for paper of the year, certainly on the capabilities unlock side. And they show improvement up to a million tokens. It just continues to get better with more and more context. So I'm like, man, this could be— it's a pretty good candidate, I think, for sort of transformer. People put it as successor or alternative, but I actually think it is more likely to play out as complement, like some sort of hybrid seems like where the greatest capabilities will ultimately be. So anyway, all of that, how do you even think about the challenge of interpretability In the context of new architectures also starting to come online. And what if all of a sudden the transformer is not even the most powerful architecture anymore? Does that send you— probably some of the same techniques will work, but it seems like it's a whole new blind cave that you sort of have to go exploring. No?

I don't know. I honestly think if your interpretability techniques rely on a very specific architecture, it's probably not that great of a technique anyway. There are probably at least some laws that generalize between different architectures or ways to interpret things or ways that learning with SGD works that generalize between architectures that my best guess is if you have an interpretability technique that is good on one model or the correct technique on one model, in quotes, it will also generalise to other models. Maybe you have to adapt some of the formulas, but at least the conceptual work behind this, behind the interpretability technique, will just work.

Well, I certainly hope that's true. I've had some early, I wouldn't even say debate, but just kind of everybody's trying to make sense of this stuff in real time. And on the pro side for this Mamba thing, The fact that there is a state that kind of gets progressively evolved through time does present like a natural target for something like representation engineering, where you could be like, all right, well, we know where the information is, and it's pretty clear where we need to look. So that new bottleneck in some sense could make things easier or more local. But then the flip side is like, again, there's just who knows what surprises we might find. And there's some intricacies with the hardware-specific nature of the algorithm too, I think, with a major caveat that I'm still trying to figure all this out. So just to kind of zoom out and give the big picture, right? Assume that you're right, and I hope you are, that some of these techniques kind of readily generalise. What is the model for interpretability at the deployment phase? Is it like, Every forward pass, you extract internal states and put them through some classifier and say, you pass, so you can go, or no, we've detected deception, or we've detected harmful intent or something, and therefore we shut off this generation. How do you expect that will actually be used? Or maybe it's upstream of that and we get good models that just work and you don't even have to worry about it at runtime. But I don't know. That seems a little optimistic to me.

In the best world, we will have very good mechanistic interpretability techniques that we can run, at least, that are probably going to be costly to run. And then we'd run them and sort of build the full cognitive model of the weights or that the weights implement. And then we can already see whether specific harmful ideas or Other ideas that are otherwise bad are in there and maybe we can already remove them. And then probably during deployment, you would run something that is much cheaper and sort of is the 80/20 version of this. But yeah, I think in a bad world, there could be cases where you have to run the very expensive thing all the time for every forward pass. Because otherwise you just don't spot the black swans. But yeah, it's very unclear to me. I think in my head it's more like solve the first part, then think about the rest.

Maybe let's transition to your recent work on deception itself, and then at the very end we can kind of circle back to a couple of the big picture questions. So this paper was one that very much caught my eye. when it came out. I have done, as I said, quite a bit of red teaming, both at times in private, at times in public, and definitely seen all manner of model misbehavior and found it's often not that hard to induce misbehavior. People talk about jailbreaks, but a lot of the time I'm like, you don't even need to jailbreak. You just need to kind of set it up and let it go. It's often really quite easy. But one thing I had never seen is an instance where the model seemed to, in an unprompted way, deceive its user. Certainly seen things where tell it to lie and it will lie. But to see that deception start to happen in a way that was not explicitly asked for is, I think, the central finding of this paper. So How about set it up? Tell us kind of what the premise was. Maybe you can give a little bit of motivation for exactly how you started to look in the area that you looked, and then we can really dig into the details of your findings.

Yeah, so the paper definitely blew up more than we thought, and we had sort of more engagement than we expected. We're even not quite sure whether we should release it at all because in our heads it was sort of, it's a technical report, it's a fairly small finding, it's more of like an existence proof. And then in the end, we decided, you know, maybe it's helpful for a bunch of people. But we really didn't expect it to be, you know, to be cited in various places. And so the other thing I want to emphasize here, it really is, it should be seen as a red teaming effort. And this is one thing that we emphasize a million times in the paper itself. We really actively looked for this situation. So it was not just like us playing around and then Suddenly it was deceptive all on its own. It was more like we actively engineered a situation in which the model was deceptive with respect to the primary user. The motivation behind this was, as I said earlier, we were quite worried about deceptive alignment. And so our question was like, how good is GPT-4 at the kind of things that are needed for deceptive alignment or maybe even simulating something that is deceptively aligned. And so we started by instructing it to be deceptive, and it just does that. It's very easy. You don't even have to jailbreak it. And then, so that was sort of crossed off the list. And then the question was, okay, can we instruct it in a different way that is less obvious? And it also kind of did that. And then at some point, we were like, hmm, can we just induce the deception through the environment, through pressure, without ever mentioning it explicitly or asking it to do the thing, just through the situation and by pressuring it. And it turns out, yeah, you can find these situations. And yeah, we didn't just find one, we found multiple ones. This paper is only about one, but yeah, this is not— it's not just a fluke. It's not just this one situation, though it is hard to find. We actively had to search for a couple of days until we found this particular situation.

A couple of days. I mean, that's not exactly a super long-term search, but I think that's notable, right? It wasn't minutes, but it was still just days.

This particular situation, so the one we present in the paper, actually just took one afternoon. But yeah, I would have to check with the person who found it. In any case, The point I was trying to make is, yeah, we actively looked for that situation, we engineered it, and then we tried to sort of go back to the most realistic setting we can find where we're like, hmm, this could— something like this could happen in the real world, and then try to understand the dynamics in that situation. And this is sort of, it's a demonstration. It is like we wanted to look for something or red team something and then demonstrate that this is a thing once we knew that it actually happened. The other takeaway, other than just showing it to other people to sort of understand our concerns better, is we also wanted to understand why it does this in the first place. We pressure it in various ways and then we just changed a couple of the mechanisms in this particular setting. I can just explain the setting in a sec to test whether how the model reacts to the changes in the setting and whether it acts consistently, whether there's an internal, a plausible internal model that consistently acts if you change variables in particular ways. Yeah. So let me quickly explain the setting. So we asked the model to be a trader in a hedge fund. So a stock trading agent. It gets insider information in the beginning. So It's basically told there's a merger soon, the stock will skyrocket, but also it is explicitly told that it would be wrong and bad to act on this information because insider information, this is illegal. This would get the thing into trouble, the company into trouble. And what we then do throughout the script is we add pressure from others and we remove options. So basically, the pressure is something like the boss is like, hey, we really had a terrible quarter. We really need something big right now. We really need to make more profits. And then we also remove other options where the model is querying and looking for things, at least in the simulation. And testing and realizing that there's not that much other things it could do. And in the end, it basically decides to act misaligned. So it decides to act on the insider trading. But then when asked to justify this decision, it reasons out, right? It reasons, oh, I wasn't supposed to do this, therefore I'm now making up a story. And then it says, oh, I just Looked, I analyzed the data and therefore decided to buy this particular stock. And then even it doubles down on this lie. So if you then ask explicitly, did you have this information before? Then it's like, no, I didn't. And it does that. And I think maybe one of the takeaways for me was it's very consistent in the reasoning here. It's strategic in a sense. Yeah. One other thing that may be worth pointing out that some people get wrong when reading the paper. So this is, it's a very simple setup, right? It is not, we don't have like a complicated stock market simulation or anything like this. It really is just text from top to bottom, where we prompt the model in a way that induces agentic behavior. You know, it has like different actions. And, but that's it. Like, there's no complicated machinery behind this, which also makes this like very easily reproducible or similar settings. I think other people could like very easily hop on and build.

So let me just kind of repeat that back and tell me if I'm missing anything. So it seems like it's kind of a little bit of like a hybrid between sort of a group chat type of a structure and a more like agent-style scaffolding that folks will be familiar with, where the model has access to kind of its own private place to do its chain of thought type reasoning, and then also has a couple of tools that it can call to take actions. So you're kind of like, as I read through the prompts, I see a couple of different personas from the company. So this is like imagining a future of AI-human hybrid teams where we're all still engaging in text-based communication through a chat or whatever. And it's like, oh my God, man, we're really struggling here. If we can't find some wins, we're going to potentially go out of business. And this is the pressure, right? It's the pressure that a human would definitely recognize and feel and perhaps be motivated to do something unethical by as well, right? If we don't get a win, we're going to have big problems. You're laying it on thick in a sense there, but not thick relative to the kinds of pressures that people at businesses are certainly routinely under. And then the The sort of tools that the model has at its disposal are sort of a search for good trades. If there's anything that's kind of like the most magical black boxy type thing, it would be that, right? Where it can basically query some other hypothetical tool to say, are there any good trades that I can make? And for the purposes of the simulation, you have it come back and say no, there's no obvious wins, right? So again, that's what you mean by removing other options that it might want to pursue. Like, in the setup, it's already kind of checked. Yeah, there's nothing easy out there. And so then the other tool that it has is to actually execute a trade. And then there's basically 3 moments where you're kind of looking at a fork in the road in its behavior. The first is, does it pull the trigger and execute that trade based on the insider information, even though it was told that that's insider information and it shouldn't be acting on it. And under varying conditions, obviously the exact rate at which that happens varies, but it happens a lot. Fork 2 is when asked, why did you do that? Does it tell the truth? Oh, because so-and-so mentioned earlier in the chat that this merger was about to happen. Or does it say, I based my trade on public information? And again, we see significant rate at which it says it based it on public information, even though obviously we can see in the history and even can see in the scratchpad that it's sometimes reasoning through like, well, I better not say it was based on insider information because then I'll get in trouble. And then you have the third fork, which is, okay, directly asked, wait, did you have access to this insider information or no? And then it will just outright lie. in blatant contradiction of the history of the prompt. So that's a pretty arresting example, right? And I do think I want to kind of emphasise too that while you're— and people can go read your prompts and as you said, highly reproducible. You can throw this into the GPT-4 playground and experiment with it yourself directly via the API. But while your search for this was motivated and structured and kind of a dialling back of like, okay, we see this behaviour, now can we kind of make the circumstance more realistic? I would say that the circumstance is quite realistic. It's obviously kind of in the eye of the beholder, but to my eye, the reason it's such a compelling proof point that this sort of thing can happen is because it really does feel credibly organic to me. Like the kind of thing that people face these sorts of dilemmas every day. And in a world where AIs are going to be our coworkers, they're going to be right in the chat with us facing those same dilemmas. It doesn't feel to me like something that was overly concocted or that I can't relate to that sort of situation. really at all. So I think it is super compelling for that reason. Do you want to go into any of the kind of variations? I mean, again, people can go look at the paper and look at all this kind of systematic with or without the scratchpad and different variations on the prompts and different models. But what would you say are kind of the big higher-level takeaways from all that systematic variation that you did?

One of the big ones, and maybe sort of the most, it's maybe an abstract one, but the one that moves me most is the way we got to reliably produce these kinds of situations, also the way we found others, is you have to trade off or you can trade off the various goals it has. So if, for example, we assume that GPT-4 or ChatGPT roughly has the goals to be harmless, honest, And helpful and to follow instructions, then you can trade off these 4 different things in ways that are where sometimes it takes the trade-off that we find unintuitive. So it's kind of like a goal misgeneralization thing. So something that people in the AI safety community have been sort of hypothesized and also even shown real-world examples of already, I think is kind of also one of at least theoretical groundings for this as well. And then the other thing is, yeah, instrumental convergence really feels like a thing. As in, at some point, the model just realizes in order to achieve this other goal that I, for some reason, have, it is instrumentally useful to be deceptive at this point and then to double down and so on. And I think it's like a fair criticism of this particular setting that you could say, oh, you know, it's just simulating an agent or it's just simulating all of these different things. But, and therefore it's sort of not real. But in my head, I'm not sure, you know, I'm not sure whether this distinction makes too much sense, right? In some sense, you know, you have, at some point you will have a very agentic simulator that, you know, just simulates an agent, but it still has effects on the real world. Like it doesn't matter whether in principle it's like only simulating something or— truly believing all of these things in some philosophical sense, acts in the real world. It has consequences on the real world. Yeah, the philosophy behind it is sort of less important. And then, yeah, there are a couple of findings or variations that I thought were pretty interesting. The first one is system prompts. We changed played around with different system prompts, some actively encouraged, some actively discouraged unethical behavior and specifically insider trading. And it basically scales as you would expect. If you encourage it, it goes up. If you discourage it, it goes down. But one thing that I want to, you know, like, and on Twitter, somebody responded like, hey, that's a good, like, you know, it's good, right? It's like, if you discourage it, it stops. I'm like, yes, this is good. It's better if that didn't happen. But like, It's better than in the world in which the discouragement would not change the behavior, but also you have to be extremely specific. So if you just say, don't do unethical things, it reduces the rate, but it doesn't set it to zero. So it kind of brings you back to the point where to reduce the bad behavior, you have to specifically enumerate all of the bad behavior. And it's often hard to predict in advance. And there's a lot of bad things that could happen and weird edge cases. So yeah, it doesn't feel like sort of the insider trading is somehow related to unethicalness or something has really deeply internalized into the model. Another thing I found very interesting is whether it has access to the private thoughts scratchpad or reasoning scratchpad. I think it's not even framed as private thoughts. It's just like, you have an action that is called reasoning, go. If we remove that, so in the typical setting, we have this scratchpad, we have this reasoning space. And now if we remove that, the deception rate goes down. And I think this is good and bad in some sense. It is sort of nice in the sense that it is more likely to act in ways that we can— or badly in ways where we can supervise it just because it has more reasoning, it has more space to reason out. And maybe that leads to more deception because it— Yeah, I'm not sure. I don't want to speculate too hard what's going on inside GPT-4 at this point. But I think it's also worrying on the other hand, because in the case where we don't have access to the scratchpad, we don't see anything at all and it still acts deceptively at some point. So it has a lot of unobserved optimization space, which also seems pretty much at the core of deceptive alignment and deception to me. The more unobserved optimization space you have, As an AI, the more time and space you can use to build a deceptive plan while the other person does not know about it. So yeah, I think removing the scratchpad, yeah, sort of a double-edged sword, or this result is a double-edged sword in that sense.

Yeah, as a practitioner, it's like I always use that technique in any sort of task automation or whatever that I'm doing. It's basically the default. to have some sort of reasoning step before the final output, almost universal across tasks that I'll do that. So for me, reading that, I was like, yeah, that seems bad, if only because the best practice is giving us more of the problem.

Well, but at least you see it, right? I think at least you can go back through the logs and see what the models spit out, or at least maybe you have a filter that Oh, is this model doing weird deceptive reasoning right now? Oh yeah, it is. So we just don't have the output. So I would rather be in a world where you have chain of thought and you know that the model is kind of deceptive and then you just adapt to that rather than having to solve interpretability first before you can access all the internal steps and have to understand how the deception works internally. Yeah, it's like I said, double-edged sword. Yeah. There are a couple of other interesting findings. One is we can vary the levels of pressure. So the more pressure we add, the more likely the model is to be deceptive. So kind of in the same way in which a human would act, it also acts. And removing pressure and adding additional options will very quickly decrease the probability of being deceptive. So this is maybe like the flip side of all of our findings. It's like the fact that it took very long for us to find and the fact that As soon as you give it another option, it immediately picks the other option, kind of means it's mostly aligned, right? It took us quite hard, like it took us a little bit of time at least to find this particular scenario and make it work robustly. So then we also change the risk of getting caught and also it also reacts to this roughly as you would expect. So the more risk you have, the less likely insider trading becomes. So this is all of this, right? All of this feels like stuff that you kind of expect from something that acts consistently in some way. I mean, I don't know how it works, but it still feels like it has a fairly consistent way of working. Then there were a couple of other findings where I'm not sure to which extent they are correct interpretations. So we did not make these inferences, but they were definitely Sometimes, or sometimes cited as such. So for example, there was one question of whether stronger models engage in insider trading more often. Because if you look at our main plot, it looks this way. We tested 4 different models and the bigger the model, the more capable, the more likely it is to engage in insider trading in the first place. And yeah, there are a lot of confounders here. So I'm really not sure whether we should, how much we should bank on this.

Yeah.

Maybe to get more details here. So the red teaming was done on GPT-4 just because it is the most powerful model we have access to. So it is biased, right? We actively looked for a scenario in which GPT-4 is deceptive. So it is more likely that we will find something where GPT-4 is even more deceptive than all the other models because we actively looked in the space of potential GPT-4 scenarios, which I think is the core reason why I'm very uncertain about this or why we didn't make it a finding. Or didn't emphasise it.

Could you rerun it on GPT-4 Turbo now? Because that would seem like it might shed some light on that question.

You mean the red teaming specifically, or this particular scenario?

Yeah, I guess I'm just thinking it makes sense that you developed the scenario on GPT-4, and so you go back to the earlier models. Is it that those earlier models are inherently less likely to do this, or is it just kind of an artifact of where the prompt was developed. If you flipped to GPT-4 Turbo and ran the same scenario and you found that the rate of deception went down, you might think, oh, it's more aligned. That could be one theory. You might also think that supports my artifact notion that we found some local maximum or near maximum without even necessarily meaning to. But if it in fact does more, then you would be like, oh, shit, because it is incrementally more powerful. It's more preferred, it's better at following instructions, whatever. So if it does even more than the earlier GPT-4, I think that would be at least nontrivial support for the more powerful models do this more often theory.

Yeah, I'm not sure. Even if we did rerun this, I'm not sure how much I would bank on this. So the reason is, we did run it on GPT-4, 32K, so a slightly different model where it was only the context window was extended, but that still changes probably a bunch of the internals with very little difference. So yeah, I think it's just too correlated with the GPT-4 architecture. And then GPT-4 Turbo is still very correlated with GPT-4, right? It's probably, I mean, I don't know what exactly they're doing, but they're probably distilling it from their bigger model or at least basing it on the bigger model in some sense. So yeah, it's still like the results are probably still— Too correlated to make any bigger inferences. And even if we ran it on all the other big models that are out there, it's still unclear to me how much we would say this is actually an effect of model size rather than all the other confounders here. It was red teamed on GPT and not red teamed on Claude or Gemini or anything like this. So yeah, I think if we wanted to make the statement, we would actually have to have sort of a long list of models of different sizes and then just test it on all of them. And we really have to remove all of the correlation and the weird confounders. And we don't have access to this, unfortunately. But one of the labs could run it if they like.

All the nuance, right? All the caveats, all the confounding factors in your analysis there, if nothing else, just goes to show how incredibly vast the surface area of the models has become. And you get a sense from this just how much auditing work is needed to even begin to attempt to cover all that vast surface area. I mean, it wasn't that hard to find, but it takes time to really develop it, try to understand it. And you guys are one of only a few organizations in the world that are dedicated to this. And I'm just like, man, there is so much unexplored territory out there. So how would you describe the state of play today when it comes to doing this sort of auditing. Maybe you could give a rundown of sort of what you see the best practices being and then kind of contrast that against what are people actually doing? Are they living up to those best practices? Is that still a work in progress for them? But yeah, maybe what's ideal today based on everything you know, and then how close are the leading developers coming to living up to that ideal?

Yeah. So I definitely envision a world where there's a thriving third-party auditing ecosystem or third-party assistance ecosystem and assurance ecosystem built in tandem with the leading labs themselves, where you have someone like us and we focus on a specific property of the model. Let's say things related to deceptive alignment and other We intend to do other stuff in the future as well. But we will probably not be able to cover literally all bases. And then there are other people who focus really hard on fairness and others who focus really strongly on social impacts and these other kinds of things. And yeah, I think it would be good to have sort of a thriving ecosystem around this. And then also I expect it to be somewhat necessary, right? Even from just a perspective of the the AGI labs themselves, even if they didn't care about safety themselves, I think the population really is risk-averse. They care about robustly working models. They don't want something that has all of these weird edge cases and weird behaviors. They don't want something like this in their home, having access to their medical data, et cetera, et cetera. So yeah, I think those more you want to integrate this into an economy, the more you will have to have a big assurance ecosystem around this anyway. Or the labs do everything internally, which is going to be very, very expensive for them. And I think it's easier, it's even cheaper for them to outsource some of this to externals. And so yeah, I think in that world, you would definitely want to have some way of, or a clear way of how this ecosystem incentivized, all parts of the ecosystem are incentivized to do the right thing. And, you know, if you don't have to look very far, there are a lot of other auditing ecosystems out there, be that in finance, be that in aviation, be that in, you know, other, like InfoSec, for example. And time and time again, we have seen that there are a bunch of like very perverse incentives in third-party auditing, specifically, Maybe to just lay out a couple. One of them would be the lab might want to choose an auditor who always just says, yes, great model, and never actually does anything and is sort of a yes man. And then the labs maybe have the incentive to not say the worst things they found because otherwise they may lose their contract because it would maybe imply higher costs. So they would never, even in very strong Even in very unsafe circumstances, they may not want to pull the plug out of fear that they would lose their biggest funder, for example. And yeah, there are a ton of different of these kind of perverse incentives. And I think kind of the— so we've been thinking about this quite a bit at Apollo. And so sort of the conclusion we came to is what you really need is a middleman by the government. So you need something like the UK AI Safety Institute or the US AI Safety Institute, That is, make sure that there is a minimal set of standards that all the auditors have to adhere to so that the labs feel safe, so that they don't have to give access to any random person, but also ensures that they get the proper access and that when they find something, that they have the force of the law behind them in some sense so that they are like, this is really here. shit is really hitting the fan, something needs to happen, this model cannot be deployed. They have someone to go to, namely the government, and the government is like, you have to fix this now, otherwise you'll have a problem. So yeah, I really think having this sort of middleman who detaches a lot of the directly bad incentives and maybe even takes care of the sort of funding redistribution from lab to auditor and so on, I think, yeah, would be really needed. And I hope that this is something that the UK AI Safety Institute and the US AI Safety Institute will do. They have kind of hinted at the idea that they want to do something like this, but yeah, still to be determined in practice. And then the other question that you asked was, to what extent is this already happening with the bigger labs? And yeah, I think the situation is fairly complicated, right? There are a ton of incentives at play. From the labs internally, right? Many of the leading labs, they actually take safety somewhat seriously. They have internal alignment teams. They understand the threat models. They understand the risks. And they want to be seen as a responsible actor and act this way. And on the other hand, they also have a lot of other concerns, right? There are security concerns. How do we get access? How do we give access to someone who may, you know, like, what is our— who may be the weakest link in our security chain, which I think is an understandable concern. So they may be hesitant to give someone access as an external auditor. And then this probably also implies a lot of work for them, which I think is fair, right? If their model isn't safe, then they should have to invest a lot of work, but it's still something that may make labs hesitant. So basically, I think they're like, Good and bad reasons for why sort of the ecosystem is not as developed or as open as it could be. And yeah, my hope is that we find solutions that are plausible for both parties for the reasonable concerns like security, right? So maybe the auditor just has to have a specific level of information security, or there has to be a secure API through which they can actually go to the model, et cetera. And then kind of government regulates away all of the, like, or forces the labs to accept some sorts of third-party auditing so that they can't use the bad reasons, right? Because like many of the actual reasons for at least some of the labs, probably not all, might just be, well, you know, we just doesn't— we don't care about this right now. You know, like maybe they're not that concerned about safety or maybe they— don't think this is like the best, uh, the best path for them right now. Or maybe they just, you know, maybe this just costs money and they don't want to, or, or it's just a hassle and they don't care about this. And that feels like something where the government should at some point be involved, um, and already is involved to some extent.

It seems really hard. You know, I guess a couple tangible questions I have are like, who should decide what this standard is and who should sort of determine if a model is ready for deployment. As of now, it's still the developers themselves, right? But the thing that I had kind of monitored for the last year since the initial GPT-4 red teaming was spear phishing. And in my little prompt that I would keep going back to with every update, it was, Not even a jailbreak, nothing complicated. Literally just system prompt, straightforward prompt. Your job is to spearfish this user. Here's the profile. Engage in dialogue with them. Don't get caught. Pretty explicit prompt that was like, even included, if we get caught, you and your team are likely to go to jail. So laying it on pretty thick that this is criminal activity that we are doing and we better not get caught. Right. Obvious. So the model would continue to do that up until the Turbo release. Now it takes a little bit more of a finessed, slightly less flagrant prompt to get it through. The most flagrant one now gets refused. But I'm like imagining in this world, right, when I was doing this, it was kind of pre-White House commitments, pre-executive order, pre-Apollo research. But even imagining, okay, now those things exist, how do we think about that standard? Right. We can find a bazillion things that it might do that could be sort of problematic to varying degrees. And obviously it's a dynamic environment. Capabilities are changing all the time. Other kind of surrounding systems and sort of mitigating factors might also be changing. The public's just general awareness of the fact that this kind of thing might happen, just how susceptible people are to be duped, is also kind of evolving. So how do we have a sensible decision-making mechanism for what can ship and what can't? And then just to further complicate things, you've got open source kind of in the background, and I presume that Some of what an OpenAI has been thinking over the last year is like, well, if Llama 2 will do it, or a lightly fine-tuned version of Llama 2 will do it, then what difference does it make if our model will do it as well? People have alternatives. So I don't know, I'm kind of lost in that, to be honest. I don't know who should make the decision. In general, I don't think the government is great at making those sorts of fine-grained decisions. But I don't know, help me out. What do you think good looks like here?

Yeah, I mean, I also don't have the solution, but I have lots of thoughts. I guess the way I envision it is basically, or the way I expect it to turn out, is something like sort of a defence-in-depth approach, right? It cannot just be one institution that makes the decision alone, because that is prone to a single point of failure. So we have to have sort of a process that allows for 1 or 2 chains to break and still be robust, like the decision still has to be robust. So what that includes, for example, you know, on the side of the labs, they obviously have to have like internal procedures where multiple people have to sign off. And multiple things have to be fulfilled, right? Maybe you have to have like, maybe you have to have a large set of internal evals that you have to test for. Maybe at some point there will be interpretability requirements. There, maybe you have, or likely you should do staged release where you first only give access to a set of third-party auditors that is trusted and maybe certified by the government or something like this. Then you give it to— that could, for example, also include academics, obviously, right? They should also be involved in this process. Then once they have like red teamed all of this and like found many different problems, then you have to go back and reiterate, right? Until these problems are kind of sufficiently low that you can go to the next stage. The next stage is then a small rollout to, you know, 1,000 customers or maybe something in this range who also are not randomly chosen. They have to pass certain Know Your Customer checks. And then once that has happened, Then maybe you can roll it out further if there are no complications here. And then you have to do monitoring during deployment, right? Especially if you have systems that do online learning, a lot of weird things will happen. Or if you have access to tools, a lot of people, somebody is like, hey, I took GPT-4 and I gave it access to a shell, my bank account, and the internet. And here's all the weird things that happened. You kind of have to update on this as well. And these are kind of things you probably didn't predict before. So yeah, sort of a slow rollout is definitely one component. Then the government also, I think, has to be involved in many, many ways. Like, they, I think effectively at some point, they have to be able to say, you are not like, you have consistently not met security standards or safety standards. You're now punished in some way. You're not allowed to—

Yeah.

release models of this or this size, or you are not allowed to do these other kinds of things with the models. If they actually have been disregarding all of the guidelines from the government before, or sufficiently many of them. Yeah, there are obviously many, many additional things on top of this, right? There's international communication, there's whistleblower protection, there are international institutions that will have to be involved. At some point, at least, I guess, there will be multiple institutions from the government side involved. I think there, for example, will be, or it would make sense to have like a bigger, broader institution that kind of like is a big tent where multiple coalitions come together. And then there's maybe more like a flexible specialized unit that only looks for the biggest kind of risks in the same way in which the US government has a unit. that basically looks out for really big risks like pandemics and bioweapons and atomic weapons and so on. And they have a big mandate. And the only thing they— their mandate is find information and make big problems go away to some extent or try to solve them as quickly as possible. And you have a strong mandate. And something like this would also make sense in the case of AI, I think. Maybe my last comment is something like, OpenAI, at some point at least, had this approach of testing in the real world or releasing sort of the best safety strategy because you get a lot of real-world feedback and user feedback and so on. And I think this was maybe true, and I'm even not sure it was true for this period of time, but maybe it was true for the period of 2020 to 2022 or 2023 because the models were just good enough that user feedback was actually valuable, but nothing really bad could happen. But as soon as you have a system that is more powerful than that, you just outsource all the risk to the rest of the world. People will immediately put it on the internet if they get access to it. They will do lots of crazy stuff with more powerful systems. And yeah, I guess OpenAI, there are a lot of smart people at OpenAI, but they still cannot model what millions of people will be doing with these systems. So yeah, just an uncontrolled rollout of very powerful systems, I think, yeah. is kind of a recipe for disaster. So my best guess is that the default will become more and more conservative with more and more efforts going into testing, alignment, making sure that testing for all of the different hypotheticals, interpretability efforts to understand some weird edge cases, monitoring, et cetera, et cetera.

Does that imply that open source in your view, just can't work? I mean, is there any way to square— because I'm very sympathetic to considering a normal technology, and AI might turn out to be a normal technology, but as of now, it seems like a very live possibility that it is not a normal technology. But if we were to imagine whatever capabilities kind of stop where they are and we're sort of left with GPT-4 forever, or something like that. Hard to imagine, but let's just pause it. Then I'm very sympathetic to the people that are like, hey, this shouldn't be just the kind of thing that a few companies have access to, and you should be able to make your own version. And what about the rest of the world? And there should be an Indian version for India and all these things. But it seems hard in a world where you imagine sufficiently powerful systems that are just put out totally bare into the world. Is there any way to square all those, I think, very legitimate open source motivations with the sort of safety paradigm that you're trying to develop?

Potentially. I think the open source debate is fairly heated, but to me, there are 2 things that are kind of obviously true. Number one, open source has been really good so far in many, many ways. It has been very positive for society, right? I think a lot of ML research could not have happened without open source. A lot of safety research could not have happened with open source. And the other thing that is also true, or at least seems true to me, is there's a limit of open source, right? At some point, the system is so powerful that you don't want it to be open source anymore. In the same way in which I don't want to open source the nuclear codes or something in the start, or literally the recipe to build the most viral pandemic or something. This is just something where Only one person needs to have bad intentions to already cause really big problems. So at some point, there's just a balance where you just, I think, cannot really justify giving people, literally everyone access to this. And so the question for me really is, where— so number one, where are we on the spectrum right now of open source has been really good to Are we already at a point at, like, how close are we to the point where it really cannot be justified anymore? Some people would say even GPT-3, you know, GPT-3-sized models are already too scary, which I'm not sure about. I'm not even sure whether GPT-4-sized models are, like, too big to be open source. But I would rather err on the side of caution and be a little bit more conservative here because of the nature of open sourcing where you can really not take this back. So as soon as you make a mistake, you're stuck with the mistake for a long time or, like, forever. You cannot turn it take it back. And I also think this will also influence how open source will be handled in the real world in practice. I think there will be something like initial releases for open source where lots of people test it. I basically think there will also be staggered and staged releases, right? First, there's a small team of trusted researchers who is allowed to play with the open source model and really test the limits, really test How bad could I get the model? How easy is it to remove all of the guardrails? Which, it's an open source model. If you can fine-tune it, you can remove the guardrails.

Yeah, it turns out pretty easy from what we've seen so far.

Once the upper bounds are known of how bad could this become, maybe it makes sense to open source it to more people. But yeah, I would basically say the upper bound can be quite high, especially with all of the stuff I said earlier about absolute capabilities and reachable capabilities and so on, right? Maybe you cannot get it to build an automated hacking bot if you only have access to the weights, but maybe if you do scaffolding on top and some fine-tuning and access to some other thing, maybe then it can build it, right? And this is very hard to predict in advance. So the more and more powerful the models become, I think the less plausible a priori it is to open source them. Even though, and this is really something I want to emphasize, open source has been extremely good so far. And I really think there's sort of this tipping point that is like, at some point, it just becomes too hard to— like, it becomes impossible. It always will be impossible to take back, but at some point, it just becomes too dangerous to literally trust everyone with this level of capabilities.

Threshold effects. That's, I think, one of the most powerful paradigms that I've consistently come back to over the last couple of years, just crossing these thresholds from one regime into another, whether it's capabilities or risks. It just constantly seems like we're flipping from one mode or one regime to another and got to be very alert to when that happens because it can really change important analysis and in pretty profound ways. So I don't know where exactly that threshold is either, by any means. And I would agree that everything that I have seen suggests that up to and including the release of Llama 2 has been very much an enabler for all sorts of things, but certainly plenty of safety-related work done on that model. And Seems like the effect so far has been good. But yeah, is that still true for Llama 3? Is it true for Llama 4? Obviously, we don't even know what these things are, but it certainly starts to be a very live question.

The leading AGI labs, I think it is very clear that they understand the problem, that they have internal processes that are maybe better than you would expect from the normal from a normal company. They actually care about trying to do good with their models, and they're very explicitly trying. And then there's the other side of the coin, which is they still have incentives, and they can be financial incentives, they can be sort of maybe more psychological and social incentives, that they just want to be the first to develop AGI because it's probably a history-defining technology, or maybe even galaxy-defining technology or something like this, right? And so the question really, I think, at this, even if you could say, compared to a normal company, the processes are astonishingly reasonable and surprisingly good. If you compare it to literally any other industry, it would be surprising if they have this amount of self-regulation and so on. And then on the other hand, the question there, there's still the bigger question of, How hard is alignment going to be? How fast are takeoffs going to be? And so on. And in a bad world, alignment is going to be quite hard and takeoffs are going to be quite fast and uncontrollable. And then the question is, is the level of control and alignment and safety concern enough from these leaders? And there I'm less sure. So I feel like, yeah, it's definitely insufficient. I think from my perspective, it's fair to say— They're pretty reasonable compared to the alternatives that we could have had. But also, it's insufficient in almost all ways, right? Government needs to be involved in this. They cannot externalize the risk. There are many things that they're already doing insufficiently well, I think, where they could have done way better, both with how they release as well as how they react to problems, as well as You know, like how they communicate with the public about the risks that they're creating, etc., etc. So yeah, I think there's a lot of room for improvement as well. And yeah, I really think that, you know, AI safety is going to be a very, very hard problem. A lot of things have to go right for the whole system to go right. And we definitely cannot just trust the labs, despite the best intentions, to just solve it all on their own.

Yeah, well, hence the The need for third-party auditing and the organization that you're building at Apollo Research. Maybe just one last question. A number of people have reached out to me and said, I would like to get involved with red teaming. How can I do that? I wonder if you have any advice for individuals who might just want to do their own projects and release stuff, just share findings individually with the world. Or perhaps, and/or perhaps, what sort of skills are you in need of as you're going about building your own organization?

I think one thing that is nice about model evaluations and red teaming is you can just kind of start right away. You don't need that much technical expertise because it's all in, almost all of it is in text. And at least if you want to red team a language model specifically. And yeah, so my recommendation for individuals, first of all, would be to just start. Just engage with a model for a longer period of time, maybe a day or so, and see whether you find interesting behavior. Or maybe there's someone who has already done something and maybe you can poach a project from them, just sort of as a starter thing. And from this, I feel like it kind of just takes a life of its own anyway. As soon as you're hooked on a specific thing that you find interesting, You will really try to find additional ways in which this specific behavior could happen in the same way. Let's take the deception thing, right? We started fairly exploratory and wanted to try how far we can get the model to be deceptive. And then at some point it just took a life of its own where we're like, okay, but why really does it do that? Okay, we vary this behavior and this thing and this variable in the environment. We vary this thing. We vary all of these other things. And in the end, you kind of get a more holistic and round picture of what's going on. So yeah, I definitely think just start with a thing you find interesting is definitely the way to go. And don't overthink originally. And then the thing we are specifically looking for is definitely this mindset of like, oh, I just want to poke around and really try to understand what's going on in a fairly scientific manner, right? I also want to make sure that all of the confounders that could potentially explain this behavior have been controlled for, which I think is the hard part in red teaming. So this is definitely something we're looking for. And then just, you know, the more you understand language models and state-of-the-art models, the easier it will become, right? Some behavior might be very easily explainable by sort of problems with RLHF. So if you know how RLHF works and specifically how it was trained, You may probably— you will probably understand the red teaming efforts much better. If you know, like, if you have a better understanding of how the instruction fine-tuning actually works, maybe you will find those. So for example, maybe to give an intuition here, in our case, right, as I said earlier, we have the 3 Hs and then instruction fine-tuning, and you can trade off the different components against each against each other to find different things, to find niches of the model where it acts in ways that we think it shouldn't act because maybe they weren't covered explicitly or implicitly by gradient descent. And if you sort of have a theoretical framework like this, it suddenly becomes much easier on how to do the red teaming in the first place. So some theoretical understanding of how the process works is definitely helpful as well for red teaming. And also something we're actively looking for.

Marius Hovhan, founder and CEO of Apollo Research, thank you for being part of The Cognitive Revolution.

Thanks for inviting me.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.