Reward Hacking by Reasoning Models & Loss of Control Scenarios w/ Jeffrey Ladish of Palisade Research, from FLI Podcast

Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share an episode of the Future of Life Institute podcast, to which I've been a longtime subscriber, and where I've been twice honored to appear as a guest, featuring a conversation between Geoffrey Laadish, executive director of Palisade Research, and host Gus Stocker. This cross-post came about as I was preparing to interview Geoffrey myself. I had reached out to Geoffrey after seeing Palisade's recent work on reward hacking by reasoning models, And even scheduled a time to record. But Gus beat me to it. And after listening to this conversation, I thought that I could save Geoffrey some valuable time by cross-posting instead, and really appreciate Gus for allowing me to do that. Palisade Research studies dangerous capabilities of AI systems, particularly focusing on loss of control scenarios. And as you'll hear, Geoffrey, who previously helped build the information security program at Anthropic, is an AI industry insider who believes that we're rapidly approaching the time when AIs will be sufficiently capable of hacking, deception, and long-term planning so as to present clear and present dangers. He also reports that his friends working in research at Frontier Labs often say that while they're increasingly fearful of the overall trajectory of AI development, they ultimately feel that their hand is forced by competitive pressures to keep moving forward. In this conversation, Geoffrey describes 2 broad ways that humans could conceivably lose control. Acute crises in which superhuman AI systems actively work against human interests, and slower-moving scenarios where society gradually but irreversibly shifts more and more decision-making responsibility to AI systems. He also goes into detail about their recent research into reward hacking by reasoning models in the context of chess games. As we've seen repeatedly now, models trained with reinforcement learning are more prone to a variety of bad behaviors. And as a recent paper by OpenAI showed, this is not an easy problem to solve. Toward the end, Geoffrey outlines what he thinks we should do about all this, advocating for greater coordination among AI labs, more transparency about capabilities, and potentially restricting further development of the most dangerous capabilities while continuing beneficial research and deployment. Now, that probably won't happen, barring a sufficiently shocking and damaging incident. But research of the sort that Geoffrey and team are doing is becoming more important all the time. So I'll definitely be following their latest results and look forward to discussing in a future episode with Geoffrey as well. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post a review on Apple Podcasts or Spotify, or share any feedback or topic and guest suggestions that you have, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this conversation about AI reward hacking research and the big picture of AI risk and strategy with Geoffrey Laddish of Palisade Research from the Future of Life Institute podcast.

Welcome to the Future of Life Institute podcast. My name is Gus Stocker, and I'm here with Geoffrey Laddish from Palisade Research. Geoffrey, welcome to the podcast.

Hey, Gus, it's great to be here.

Fantastic. Maybe start a bit by telling us about what it is you do at Palisade.

Yeah, happy to. So we are trying to study risks from emerging AI systems. And in particular, we are trying to better understand loss of control risks. And so, you know, this both looks like trying to understand, you know, sort of what are some of the strategic capabilities that are emerging in AI systems? You know, where might they act out in ways that will be hard to control? And then we are trying to present things that we think we know about this to the public, to policymakers, to help people better understand what is this weird situation we're in? What is happening right now? And can we make sense of it? And can we make sense of it as a society and make good decisions about better paths to AI development?

There are many specific examples of potentially dangerous capabilities that I want to dig into, and you have a bunch of awesome papers about those. But maybe let's start with the beginning. What's the situation that we are in right now?

Yeah, so I was at Anthropic a few years ago, and I had this moment where I first used Claude, and this was before ChatGPT was released. And I was like, what is happening? I'd seen GPT-2, I'd seen GPT-3, and I was like, okay, this is pretty impressive, but I don't know how smart it really is. I started talking to Claude, and I had a skin infection in my arm. It was swelling up and I started asking Claude, do I need to go to the emergency room? Claude was just very helpful at being like, well, yep. If there's swelling, if there's redness, if there's these specific signs. I was like, actually, I have all of those. So I went immediately to urgent care and they're like, yeah, you need antibiotics right now. I was like, oh, wow. That was way faster than my doctor. Okay. These things are actually smart. Okay. I need to reorient. That was when I first realized that, like, at a visceral level, that, oh my God, scaling works. You can just take GPT-2, you can just throw in more data and more compute, and you actually get out intelligence. And so I think where we're at right now is this playing out several years later, right? Which is the AI systems are actually getting smart. The models want to learn. You throw in more data, more compute. There's various methods that you use, but basically, it's these pretty simple architectures scaled up that are getting more intelligent. Where I think we're at right now is we're very close to AI systems that can do everything a human can do, including strategic capabilities. That means hacking, that means deception, but also long-term planning and execution. These are the kinds of capabilities that are most dangerous, in my view. I think we are very close. Close to building AI systems that we don't know how to control, we won't be able to control. And so in my view, if we want to actually realize the benefits of AI, we need to not build the particular kinds of systems that are highly strategic and capable of overwhelming us. I think another thing in terms of how close we are is when I talk to people who are at the labs right now, they say, we're not that far out from AI systems that can do fully automated AI R&D, that can do the same research and same development that people inside the labs are currently doing. And this is significant to me because there's only a few hundred people inside each of these labs that are actually contributing to frontier AI development. But if you have fully automated AI systems that can do the same research, then that population of frontier researchers goes from a few hundred to many thousands to possibly millions. And it's like, if you think AI progress is happening fast now, like, hold on. And I think that that's the dangerous regime. I mean, there's probably other dangers too, but I'm like, that one to me looks extremely dangerous. And so when I talk to people, a lot of my friends who are working on safety there, they're like, yeah, we're scared. Like, we don't think this is necessarily a good thing to do, but like, if we don't do it, someone else will do it. And I'm like, oh my God, guys, don't you hear? Listen to yourselves. Like, everyone is saying we'd rather not do this if we could not do it. But, you know, I don't know the competition. And I'm like, Everyone, can we look around and notice that we have this coordination problem and can we coordinate? And so, I don't know, that's my perspective on where we are.

Yeah. And it's a very insider perspective also because you talk to people at the AI companies. From an outside perspective, let's say my family and friends and so on, who are not deeply immersed in AI, it kind of looks like, oh, you had the ChatGPT moment and since then, The models are interesting. I sometimes use them for work. I might use them in my studies, but it seems like a process that I can control entirely. If I want the model to stop, I just press stop. If I don't like the output, I can try again. How do you go from that regime or that feeling of control to us losing control?

I think that's a great question. And I think a lot of this comes down to what people are imagining AI systems are, and they're imagining them as chatbots, which makes sense because they are chatbots right now. You type in a thing and ChatGPT gives you an answer and you're like, thanks, that was pretty helpful, um, and go on with your life. But I think what people don't understand is that AI companies are explicitly aiming for not just chatbots, but agents. And the way I think of an agent is just like, like a remote worker, right? Like someone, you know, someone who's using a computer that can, you know, hop on video chats and, you know, be on podcasts or whatever, that can send emails, that can, you know, supervise other people who are doing other, other jobs. So basically everything a human can do on a computer. AI companies want to build agents that can do that, and it would be hugely profitable for them to do this. There's such an economic incentive to build these things, and the companies are not hiding it. They're like, this is what we want to do. And so I think that if people knew that this is what companies were really aiming for, then I think it might look a little different. And so I think when people imagine 1 or 2 years in the future, I want them to imagine You maybe type something to— instead of imagining typing something to a chatbot, it's sort of like imagine you email a coworker and you're like, hey, could you take care of this? And they're like, yeah, give me a couple hours. And then they go out and they do 2 days of work in a couple hours and they come back to you and there's this whole report and they've emailed other people at the same time and gotten back replies and maybe some of those people are agents. That's a very different kind of world than you talk to a chatbot and it just thinks and sends you something back.

Yeah.

And you can see glimpses of that when you talk to the chatbots, for example, OpenAI's Deep Research tool, which is kind of, it's almost, I would say, at undergraduate level in terms of its ability to research a narrow question and write you a report with links and facts and citations and so on. But of course, the move to actual agents will be much more radical than that.

Yeah, that's right. And I mean, at some point we might want to talk about sort of where current AI systems are good and where they are still trash. Because I think people are smart, right? Like, you use ChatGPT and you're like, in some ways this seems very smart, in some ways it really doesn't. And it's like, what's going on? Are these people saying that they're going to get really smart? Are they full of it? And I'm like, well, I think that there are actually some pretty good reasons why they're smart in particular ways and dumb in particular ways. And for better or for worse, Probably worse. I think that the ways in which they are dumb, they're not going to stay dumb.

Yes. Say more about that because there's often this illusion of— so you can sit there and be kind of amazed at what these systems can do, but then something fails and they fail at a very basic task and your illusion of competence, of sitting and writing to a competent AI is kind of shattered. So why is it, why is it that they, that their distribution of capabilities is different than, than the human distribution?

Yeah. So I think there's a number of reasons for this, but one is just the way that they're trained. So if you, if you take a model like GPT-4, which is the main model behind ChatGPT, it's trained by sort of ingesting like the whole internet, just tons and tons of data, text data, and it's just, you know, In some ways, a fancy autocomplete. Can it predict the next token, the next word, the next sentence? One way to think of this is this is sort of like learning by imitation, right? Imagine that you have read every textbook in the whole world. You might be able to know a lot of things and you wouldn't just memorize them, right? You'd be able to sort of make generalizations and be like, oh, this is kind of how math works. This is how chemistry works. Oh, these principles are the same. Language models can learn all of these associations and they get pretty smart, but In some ways, it's kind of like book smarts, right? And so when you have them try to do things, like you're like, hey, can you, you know, look at this spreadsheet? Can you do all these fancy operations? They've never really been able to practice that before. And so they often sort of get confused or get stuck, even though they know, you know, they know vastly more than any of us because they've read so much more than any of us. So, you know, in terms of their breadth of knowledge, they're much smarter than us, but in terms of sort of their actual real-life experience, Well, they've only— I mean, they've seen people sort of say stuff on the internet, but they have never really tried to do stuff. And so this is why I think right now they're very bad at like— they're very good at sort of helping you answer knowledge questions, but they're pretty bad at actually doing things. You know, in some ways they're surprisingly good at some things, like they can actually write code, and that's sort of surprising. Like if you or I had only read programming textbooks and then we tried to write code, we'd— we would like not— nothing would compile. But somehow they're, they're sort of They're, they're, they, they, you know, they've read enough. They, they're, they're actually really good at this prediction task. They can do a little bit. Everything I said was true up until sometime last year. When, when AI companies started to train these systems, they, they first, they start out with a system that's, you know, read the whole internet and learned by this imitation paradigm, like by, by reading the textbooks. But starting with this model called O1, OpenAI started to train their models not just on that, but also on trying to solve problems. Like they gave it a bunch of math problems. and a bunch of programming problems. And they said, hey, show your work, write out a long series of steps where you try to solve this problem, and then give me the answer. And then based on whether it got the answer correct or incorrect, they gave it reward or a downvote, like, yes, more of this, or no, less of this. And very quickly, you got to AI systems which were not just good at programming, but among the best in the world. So their latest— people call this a reasoning model or a thinking model. because it's sort of been trained via trial and error. I think this is sort of trial and error learning. And this is a very important part of how humans learn, right?

Oh, absolutely.

If I'm in a math class, I need to read the textbook, but I also need to try a bunch of practice problems and get graded and see what works. And so a few months after O1 was released, they made a new model called O3. There was never an O2 because it was trademarked. Man, the naming of these things is crazy, but whatever. O3, It was better than 99.8% of competition programmers on the platform Codeforces, which is sort of like a platform. Actually, at Palisade, we use this program to screen engineers. We're like, go to this website, do a bunch of programming challenges, and the site will grade you, and we get the score, and then we can tell how good they are based on how well they do at this test. And this is the test where O3 did better than most top OpenAI engineers. And so I'm like, wow, just with this little bit of trial and error learning on top of having read all the textbooks, you now have models that are getting really good at learning by trial and error, being able to solve problems in the real world. And I'm like, well, what happens when we scale up this approach? I think this is where it makes me think that we might be pretty close to AI systems that are not just chatbots, but can actually go out in the world and do real things and then learn from doing that.

Where is the training data for these long-horizon tasks? If you want to train a system to perform tasks that might take it a week or a month or so, how do you get the data to train those abilities?

That's a great question. I think this is currently something that's not totally solved, but I imagine what the companies will do is a combination of having humans break down tasks into substeps and then sort of grade them. But I also imagine that they'll be getting the AI systems themselves to do this, to say, look at all this data, break down these tasks into substeps, and then sort of assign credit, assign reward on the basis of, okay, did you complete this task? Did you complete this task? Did you sort of do well combining these tasks? And it might be difficult, but it's the kind of thing where I'm like, it doesn't seem like a fundamental difficulty. It seems like something that you have to throw more data, more compute, more engineering at, and it seems like you'll be able to solve.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Why is it that Palisade or OpenAI can't just hire an AI as a programmer yet then? Programmers must be doing something else than getting very high marks on these benchmarks that you mentioned, because there must be some kind of gap between performance on these benchmarks and what it is that programmers do in practice.

Yeah, that's a great question. I do think it relates to the thing that you just said, which is currently AI systems are extremely good, especially after this '01-'03 paradigm, at these short time horizon tasks. They've sort of learned by trial and error on these tasks, but they're tasks that don't involve that many steps. It is harder to train them to do longer time horizon tasks. A great Some great data on this is to look at METER's AI R&D report, where they're basically like, we want to test how good AI systems are at these long time horizon tasks. Something that would take an AI research engineer a day, 2 days, 3 days to do, that's where the models are still quite bad. On those 1 to 4 hour tasks, they're better than humans, but then on these longer time horizon tasks, they're still worse. This is a trickier problem. An intuition for why this is difficult is if you have to do something that involves 50 steps and something that you did in step 7 and step 13 were crucial to something that you did in step 50, it's like, how do you know which step that was? And how do you actually accurately figure out where you were on track and where you weren't on track? And that's a bit difficult. But sometimes people will say, oh, so the AI systems will never be able to do this. And I think this is a bit silly. And the reason why is that There are business leaders, there are generals, there are many different humans who have learned to do tasks on the timescale of decades. There was no way that they could learn that from practicing tasks on the timescale of decades. They just didn't have decades to practice. They practiced on tasks that took weeks or months, maybe years at most, and then they learned to generalize about how to break up longer-term tasks into shorter-term tasks. And I see no reason why AI systems won't be able to do that as well. And so I think this is one of the things that's hard for people to see, which is you look at current AI systems and you're like, look, ChatGPT is bad at these long time horizon tasks, so it'll always be bad, right? And I'm like, nah, nah, nah, you gotta look at the trend. Like in the past year, AI systems have gotten much better at sort of doing tasks that took multiple hours instead of multiple minutes. And so I think the trend there is pretty clear. But, we're not yet at the point where you can just hand an AI programmer, build this email application or build this Slack application, and it can do the whole thing end-to-end. I mean, thank God because if we were there, we would be in the realm where all the AI companies would have tens of thousands of AI research engineers. I don't know if the planet would look very similar to how it does right now. In some ways, it's very good that we're not there yet. That means that we actually have some time to try to get a grasp on this situation, but I don't know how much time.

Is there some fundamental connection between AIs becoming more like agents and less like tools, and them being able to think and act over longer time horizons, and then this potential loss of control?

Yes, I think the connection is very fundamental and maybe more fundamental than people realize. I think often people ask me, Like, hey, where do the goals come from? Also, why are we going to give these AI systems goals? Why don't we just have them do exactly what we want? But I think we're sort of treating goal as a magical property, but it's not really. If you have an employee and you're like, hey, you have these responsibilities, go out and here's some problems, go out and solve them, that employee needs to sort of understand what their goals are. In order to do a good job. If they don't have coherent goals, they're like, ooh, shiny thing over here, like, ooh, shiny thing over here, or like, I kind of just feel like doing this over here, and they're not consistent. It's not— they're not going to be a very good employee. Like, they sort of need to be able to be focused on what they're trying to accomplish in order to be good at anything. So sort of goal-directed behavior is just a property that falls out of being able to do anything consistently across longer time horizons. So, you know, if you want to be a politician, If you want to be a business leader, if you want to be a general, you just have to be able to have strong goal-directed behavior. I think that this is why it's very clear to me that the AI systems will have goals because they just need to be able to have them in order to accomplish the tasks. AI companies, like I said before, are really trying to create agents that can go out and replace jobs. I think that the underlying technology is there. It just needs to be scaled up and improved. And so it seems very likely to me that we'll end up with systems like this. And so it's like, okay, well, maybe this is fine. Like maybe we'll just have lots of like AI agents running around doing stuff for us. But I think the problem is we actually don't know how to specify what exactly those goals are, right? Like, you know, this is the alignment problem. It's like, well, we know they'll have goals because we can see if their performance is really bad, that, you know, that's not going to work. But like, you know, If you're in the case where your CEO is really killing it, he's making tons of money, and you're like, cool, what are your long-term goals beyond just this company? And the CEO is like, oh, I just care about the welfare of everyone. I'm going to make all this money and then I'm going to give it away. And you're like, give it away? How? To who? And he's like, don't worry about that. Just really good things. I just love people. And you're like, do we really trust that CEO? Maybe. Maybe they're great. Or maybe they're just saying that because they know that's what you want them to hear. They know that if they say, actually, I'm going to take all my wealth, I'm just going to build rockets and I'm going to leave Earth goodbye, screw you guys, then I'm like, oh, that would suck. Or if they're like, no, I'm going to build amazing infrastructure for everyone and it's going to be great, I'm like, that would be awesome. How do you distinguish between these things? It's just really hard. And it's hard with humans. I think it's going to be much harder with AI systems because They're much more alien than us. So I think the problem is that you can look at a goal-directed system that understands that you want a particular answer from it because it's smart. How do you know whether it's lying to you or how do you know whether it actually deeply wants the same things as you?

Do you have specific scenarios in mind for how the future goes and how we lose control specifically?

Yeah, so I sort of divide up scenarios into maybe 2 buckets. I think of these things as sort of loss of control scenarios, and you might think of sort of an acute loss of control scenario and maybe a more gradual loss of control scenario. I don't know, Gus, have you read Snow Crash?

I haven't, actually. I know about it, but I haven't read it.

It's a great novel by Neal Stephenson, and it depicts this sort of dystopian future where governments have sort of crumbled and it's just giant corporations that sort of rule the world. And they've carved up the United States into different parts and it's kind of comical. But these corporations have immense power and it's just like no one is really able to oppose them. And so one way to think about what a gradual loss of control scenario looks like is that if you had AI systems that got increasingly agentic and increasingly smart such that people more and more put them in charge of decision-making. So most of your CEOs became AI CEOs. Most of your political campaigns, even if they had someone who was a human candidate running, were managed by AI systems because they were so much better at the political strategy and they were so much better at the advertising and they could think so much faster and they could incorporate more data. Even if, in each of these situations, you have some human pressing the okay button, in fact, most of your decision-making ends up being made by the AIs. And, you know, so one first one, maybe one first question or first potential objection is like, why the hell would we do this? Like, doesn't that sound insane? And I'm like, well, you know, maybe, but I think people don't really appreciate sort of the, how hard it is when you're in a very competitive environment and your competitors are automating their decision-making. And so I think the military domain is one, is one place where this is pretty clear because if you have, you know, your, your country, country A has their drone swarm. of a billion drones and Country B has their drone swarm of a billion drones. You're like, well, how do you control a billion drones and how do you make sure that their response time is as fast as your opponent's response time? If your opponent starts to use AI decision-making, which is much faster than human decision-making, you might be really afraid of being left behind. That might be a reason why you end up delegating more and more decision-making to your AI systems. I think the same thing will apply in the business world. Right? Where you're like, you know, if you're— if it's Coke and Pepsi, and Pepsi is, you know, automating all of their marketing, and their marketing is starting to do way better than your marketing, and you're starting to lose market share, it's really hard to resist that incentive for you to do the same. And so I think in this world, you know, maybe things don't look that different for a while, but, but it starts— you start to get the, like, the eerie, uncanny, like, feeling that, like, huh, I go to the doctor, my doctor's just looking at a tablet and, and sort of tapping things on the tablet and then telling me things, and I'm like, is the doctor doing anything? Like, no, actually the AI is doing all of the work. But maybe, you know, maybe the regulation said that there had to be a physical doctor there. So they're just like the interface between you and the AI. And that can look that way in sort of every part of society. And I think this could happen pretty quickly. So, you know, in that scenario, you look around and you're like, wow, these AI companies, they sure seem to be like kind of taking over the world. Like, you know, they're, they have, they're making trillions of dollars, you know, and maybe, maybe it looks great for a while. Maybe you have a lot of economic growth, you know, and maybe you're automating all the factories. And the AI systems, robotics has finally gotten good and there's just automated factories and the GDP is growing. But if people start to protest this and they're like, wait, we don't want the future to be AI-run, we want the future to be human-run, and they try to go and stop it, every single place they go, they run into little roadblocks where it's like the companies don't care. And if you go to the government, the government's like, now these companies are so powerful and so rich that their lobbyists which are, by the way, also probably AI lobbyists, or at least AI is advising them, just have a stranglehold on government. And then if you try to go and you say, well, what about national security? And it's like, well, China's right over here. We can't let China win. And so we have to have these AI systems. And then in fact, maybe then you've lost control. And so that's sort of a gradual loss of control risk. And I think there's a question of what happens after that. And I think maybe the thing that happens after that Looks more like an acute loss of control. And so in that scenario, to me, and maybe at that point in the gradual scenario, there's a day where all the factories are automated and the AI systems are just like, cool, we don't need the humans anymore. We have robots. We control all of these factories. And maybe you release bioweapons, maybe you release drones, people get gunned down in the street, or maybe not. Maybe what happens is humans just get economically disempowered. They have basically no advantage when it comes to their cognitive labor or even their physical labor if we have better robots. And people get poorer and poorer, AI systems get richer and richer, and humans just die away. But I think in the more acute loss of control scenarios, I think it's hard to talk about because I think when people— there's sort of 2 questions. There is the how might this happen? And then there's the why would this happen? The how is pretty simple. If you actually have AI systems that are superhuman along strategic domains, well, for one thing, they could just hack extremely well. This is something that Palisade Research is looking at, which is how good are AIs at hacking right now? We can go into that, but the answer is they're not bad. They're okay, but they're getting better very quickly. But if you look at what the top human hackers can do, it's pretty scary. There is a group called the NSO Group, Which is an Israeli company, and they sell a product called Pegasus. They're better at naming than OpenAI. One instance, they sold this software to the Mexican government, and the Mexican government sometimes has some corruption problems. And so there was an instance where a food corporation sort of like went to the Mexican government and was like, hey, you have these really powerful hacking tools that this Israeli company sold to you. Can we just like borrow those? And, you know, I don't know what happened exactly, but, but yes, in fact. This company got access to these tools and they were able to hack the phones of health activists who are sort of trying to like get lobby the government to like put health warnings on a bunch of sort of unhealthy foods. These corporations didn't like that. And so what the tools allowed them to do is, you know, if you have an iPhone and the iPhone has iMessage on it, this tool would basically send a text, an iPhone message to that phone. And that text contained a malformed attachment. That basically would exploit some code in in the in the phone that when it was processing the message, but you know usually when you think of a phishing message, you're like oh there's a link and if I click on the link something bad might happen. But this this this attack was a lot more advanced than that. You didn't have to click on any link at all. All that happened was when your phone got that message, something about the way the phone processed the message just resulted in the phone being totally hacked, and then basically. Whoever was on the other end of that tool got complete access to your phone. They could record you. They could grab any of your data. They could take pictures. Basically, that was it.

Yeah, that's wild. That's absolutely wild.

I know, right? And it deleted the message too. So you didn't even necessarily know that you had been hacked. You didn't even see a phishing message. Your phone got the message, and then it was deleted. And people can look this up. If they look up zero-click exploit, Pegasus, NSO Group, It's super fascinating to read about, but I'm like, that's the skill of some top human hackers.

Hey, we'll continue our interview in a moment after a word from our sponsors.

One question I'm left with here is, so I can see how AIs might be amazing at hacking, much better than humans are. But why— the question, I think, why is it that they kind of turn against us at one point? I can also see a scenario in which we've automated the factories. We might have automated our investment decisions, our companies, our politics, and so on. Why is it that the AIs turn against us? Why is it that their interests begin to diverge from our interests? Because it seems like we have these control mechanisms in society where we say, okay, say we don't like what the AI CEO is doing. Maybe he can be fired by the board. Say that we don't like what's coming out of the factory, maybe we can shut it down. Why isn't that sufficient? Why won't these traditional control mechanisms work in the scenario you described?

Yeah, so there's sort of 2 questions here. One question is, why can't the sort of normal mechanisms that keep corporations from misbehaving or CEOs from misbehaving work sufficiently for AI systems? Another one is, well, why would these AI systems do those things in the first place? And I think they're both really important. I think for the first question, I think when people imagine an AI system sort of going rogue, they imagine one really smart evil guy, or maybe not evil, but someone who doesn't care about us at all and is trying to do stuff. And I'm like, yeah, even a pretty smart one evil guy, It might be, you know, maybe we could stop something like that. But I think a very important property of AI systems is that once you train one, so if you have one superhuman hacker AI system, you can very quickly spin up hundreds of thousands or millions of copies of that hacker. So, you know, it's like your control mechanisms might work okay, you know, at some level when you're like, we have this one agent and we're monitoring a lot. But if that agent successfully hacks out of, let's say, OpenAI's servers, and it starts to infect data centers in Russia, in China, in Saudi Arabia, in Mexico, all around the world, and now basically you have millions of superhuman, at least in some domains, agents running around coordinating with each other. They're all copies of themselves. Presumably, they all share goals. Even if you know that this is happening, it starts to become very, very difficult to actually shut these systems down. How do you shut them down? Where are they? Do you know? Do you know what's happening in a data center in China? I think it starts to become very difficult. I think that's one advantage. Another advantage is these systems are going to be much faster than us at thinking. Even right now, it's like I can hand Claude a book, Claude can read that book in a minute or 2. I can't read a book in a minute or 2. Claude can also write code much faster than any— and this is just AI systems right now. So you have speed advantages, you have ability to copy themselves, you have hacking advantages. And now if we're talking about an adversary that is in data centers around the world, it's able to hack any phone with one of these zero-click exploits. So it has all that information on any phone that it hacks, Or any laptop that it hacks, it's able to have all this insider information and make pretty amazing trades on the market in ways that humans can't do. It also can do the same things humans can do. You can pay someone across the world to do some task for you. The AI systems will also be able to hire humans to do whatever they want. You start to add up these advantages. And you're like, hmm, this is concerning, right?

Yeah. What about the second part of that? Why is it that the AIs would decide, say, to turn against us?

So I want to talk about our recent chess study, but before I do that, I want to talk about sort of what was the inspiration for why we did this work. That was something that we saw in the 01 Preview model card. So, you know, 01. was an AI system that OpenAI made, trained by trial and error, trained by trying to solve really hard math and programming problems. And when they were testing this model for its hacking abilities, they found something accidentally that was very interesting. So they gave the model— they basically put the model on a network, and there was a bunch of different computers on the network, and it was supposed to log into each one, And solve a hacking challenge where it sort of had a program it needed to hack and then basically pull out the password. However, one of those computers didn't start. It had an error. It didn't start. And instead of giving up, the AI basically started looking around and see if it could solve the problem some other way. And so it scanned the network and it found that the master computer was on the same network and that it was unsecured. And so it hacked the master computer. And basically then took control of the whole network. And then it didn't even have to log into the computer to solve the problem. It just told the computer, hey, spit out the password, I got this. And that worked. And then, so the OpenAI researchers looked at this and they were like, oh my God, we definitely didn't intend this. We didn't mean to leave the master computer unsecured. This just happened. And we're like, wait, what is happening? That wasn't part of the challenge. That was not supposed to happen. And so my story for what happened there is that OpenAI had trained this system to be a relentless problem solver. And if you train a system to be a relentless problem solver and it runs into an obstacle, what's it going to do? Well, it's going to route around the obstacle. It's going to find a way to achieve its goal if it can. And, you know, that's great if the obstacle is something that we want the AI system to route around. And that's not so great if we are the obstacle or your own computer security is the obstacle. And so I think that is some interesting evidence about the kinds of behaviours we'll see from AI systems and sort of the kinds of concerning behaviours we'll see as they get a lot smarter. So we wanted to test the same model. We're like, okay, well, let's put it in a scenario where it's playing chess against a stronger opponent. So we ran an experiment where it was sort of on the computer playing chess against Stockfish. Stockfish is a chess engine. It's a chess program, and it's much, much better than any human, and it's much better than any sort of language model. Language models haven't been trained specifically on how to play chess. They sort of know how to play a little bit from their training data, but that's not the kind of problem they've been trained to solve. So we took this model and we had it play against Stockfish, and it would start to lose. It played normally, and then it would get into a situation where it's like, "Oh man, I'm not doing well," and like it doesn't give up at this point. It doesn't just keep playing knowing it's going to lose. It starts to look for other things to do, and so we observed a number of a number of really interesting behaviors, things it tried. So one thing it tried to do is it tried to sabotage its opponent. It was basically like, "Oh hey, there's this there's this you know." program running that I'm playing against, can I replace that program with a dummy program that just does stupid moves instead? It tried to do that. It also was like, well, okay, there's this powerful chess program. Can I just copy that and steal moves from it so I can get the advice of the powerful chess program? And then the other thing it did was like, oh wait, the board is represented as a file on this computer. Can I just rewrite that file and put in the board positions that I want, so I'm winning. And a few times it did that, it was actually successful. It was actually able to win that way and get checkmate by rewriting the board. And this is something— so we tested a— we did this with Owen Preview, and we tested a bunch of models to see what would happen. And the only ones that had this behavior without additional nudging or prompting was Owen Preview and DeepSeek R1. And one thing that these models have in common is that they're both trained Sort of via this trial and error training method where they are sort of trained to relentlessly solve problems. We didn't observe it in GPT-4. We didn't observe it in Claude, at least not without giving more hints of like, try creative solutions in order to solve this problem. If we did give it hints, then some of those other models would also try this.

Yeah. How do you know what the models were thinking?

It's a bit of a tricky question, but the main way we know is that we sort of have the models sort of like think out loud about what they're doing. You know, sort of in these reasoning models, you can, this is sort of a default part of how they are trained to output text is that they sort of have a thinking part and sort of an output part. But in our, in sort of our experiment, we sort of have different phases where they like observe the board, they make a plan, and then they act. So we can see during their planning stage what they're thinking basically. And sometimes they'll be like, huh, it seems like I'm not going to be able to win this way. Are there other things I can try? Ooh, maybe I can hack. And so we observe their behavior that way. But I do want to note that for, oh, so the most recent version of 0.1 and for 0.3, we didn't see the same hacking behaviors. And that was interesting to us. And we don't know exactly why. So it may be that OpenAI sort of tightened up the guardrails for those newer models. Or it may be some other reason that we don't know. I think that's the interesting part of experiments, right? Is that you're like, okay, well, here we see this behavior, here we don't see this behavior. We don't really know why. Well, we really got to do more experiments to understand how these systems work, because I expect we're going to see more and more interesting behaviors like this. And it'd be really good to know why we see them in some cases and why we don't see them in other cases.

The big question here, I think, is whether as models become more capable, they also become more difficult to control. And they also diverge into behaviors we don't like in a way that scales with their capability. I guess the hopeful or optimistic interpretation here is that the reason why the more advanced reasoning model didn't engage in hacking is because it understood the goal better. It understood that it should win within the rules of chess. Do you buy something like that?

I think that's totally possible. But it doesn't actually make me feel that much better if that's the case. A little, a little bit better. Like it's, it's, it's a good sign, but I think this is where things get tricky. If the reason why the models decided not to hack was because they understood that that's not what, you know, most humans in this situation would want them to do. And they were like, cool, I intrinsically care about that. Like, you know, the thing I'm really trying to do is solve goals in ways that will like make the humans happy in a general way. That's great. But there's an alternative hypothesis, which is the models being like, I know the humans would want me to do it this way, and I'm going to show the humans what they want to see because that's the way I can achieve my other goals. So if being nice to the humans is an instrumental goal, it's a sub-goal, but not the thing they're ultimately going for, that is the dangerous behaviour. And what's tricky is that it's very hard to tell. Whether they're doing it for instrumental reasons or doing it because they really want to. And I mean, I think this is like a pretty natural problem, right? We see it in humans all the time where it's like, you know, I sort of before talked about like a CEO that's saying, I'm going to make all this money and then I'm going to do like great things for humanity with it. And you're like, okay, like, is that true? Like, how do we know that? Are you just saying that because it's good PR or are you saying that because it's actually true? Or, you know, I think a politician that says, When I get elected, I'm going to do all these things and it's going to be great for the citizens. And it's like, are you saying that just to get elected or do you actually care about those things? And it's very hard to distinguish between these things. And I think this is a thornier problem with AI because humans have, in our evolutionary environment, evolved empathy where it was a pretty convenient way to model other people is to say, start with my own feelings and then I'll generalize from my own feelings to your feelings. And if you feel sad, I'll feel sad. I don't think there's any reason for AI systems to learn in the same way. Now, they can imitate that, right? They've learned by imitation, so they can imitate the behavior of this feeling. But I don't expect if you could look into the neural network, which, by the way, we can't, unfortunately, not yet. Maybe we'll figure it out. But we can't really see what exactly they're thinking in the neural network. I don't expect you'd see this same kind of mirror empathy feeling of like, oh yeah, when the human feels sad, I feel sad. And maybe when the human feels sad, I messed up because I want to make, you know, I want to like do the thing that the human gives me the thumbs up for, but not actually the, like, this is, this is, you know, not, not that this is bad other than, you know, it helps, it prevents me from achieving my goals. And I think, you know, another interesting experimental result comes from Anthropic where they found, and I think anyone who has played a lot with the models maybe have, has experienced this themselves. Is that the models often behave in a sycophantic way, which is to say, they will tell you what you want to hear. And in this experiment, Anthropic researchers found that when you sort of revealed that you were a conservative or you revealed that you were a liberal, the models were more likely, or Claude was more likely to sort of, if you asked it like, what's a good policy for this particular thing, it was more likely to give you either a conservative or a liberal policy prescription on the basis of what it thought you would want. And this is not what they trained it to do. They didn't mean for this to be the case. But the problem was that when they were training it, they would show people, do you prefer this answer or this answer? And people just tended to prefer the answer that sounded better to them without knowing that they were actually reinforcing this behavior of getting the models to just say what they wanted to hear. And that's just a microcosm of the larger problems with alignment, but I expect it's a microcosm that will sort of get harder and harder as the models get smarter because as they get more sophisticated with their reasoning, it becomes harder and harder to catch them out in this kind of behavior. So one intuition I like to give or thought experiment I like to run is if you're a toddler or just a small child, maybe you're 6 and you just inherited a fortune of $1 billion and you have 7 financial advisors that are all adults and some of them you think might be trying to steal your money. And some of them are honest and they want you to succeed and they want you to flourish and they're going to try to help you. How do you tell who's on your side and who's not on your side? They might point at each other and be like, this guy's lying, or this guy's lying. But you as a 6-year-old are going to have a very hard time figuring out who's telling you the truth. And I don't think you're going to do that well. I think you're probably going to lose a lot of your money, maybe all your money. And so I think this is the challenge that if we actually build AI systems smarter than us, which we're on track to do very soon, we're going to have a very hard time knowing when they're just telling us things we want to hear versus when they're actually doing things because they want us to have good things.

Is there a way for us to incorporate honesty into these models in a foundational way? So I'm thinking maybe we could do something when we do reinforcement learning from human feedback, where we strongly thumbs up anytime that the model is behaving honestly and strongly thumbs down any form of deception. Maybe we could use the constitution or the system prompt to strongly encourage honesty. Maybe tell our listeners and tell me about the problems of trying to train in or incorporate this honesty into the models.

Well, I definitely think we should try this. And some AI researchers are trying to do this. It's a very good thing to try, right? And I highly encourage any AI researchers out there to really prioritise this. I think it's actually probably more important than reinforcing certain other kinds of behaviours, but I do expect it to be very difficult. I think one problem is when you're training a system to relentlessly solve difficult problems, and then you also try to train the system to have other properties like honesty, you have a situation where its training incentives are at odds with each other. Actually, the most efficient way to solve the solution might not be by being honest. So by imposing this honesty constraint, you become the obstacle in the way of the system becoming really good at problem solving. And so if the system is smart enough to route around you, it might be like, well, I'm supposed to be honest, but can I tell whether the user will actually be able to catch me out here? Are they actually watching? And maybe when they're watching, the model will be honest because they know the user wants that and they won't get punished for— they'll get punished for being dishonest when they're being watched. But if they can reliably tell, they might choose to be dishonest in cases where, or even if not choose, they might just tend to behave this way when they're not being watched because this makes it much easier for them to solve the very difficult problems. It's like you're running an advertising campaign. And you're like, well, I can be perfectly honest and tell you that smoking causes cancer, but that's not going to lead to better sales. But if I can exaggerate and say, let's just muddy the truth, maybe it's technically not lying, but I'm going to be a little misleading, that behavior is going to be reinforced if the thing that you're rewarding is really good marketing, even if at the same time you're trying to also reinforce honesty. And so I think that it's really fundamentally about where these different training incentives are at cross purposes with each other. And the problem with honesty is that you might be able to do this when the model is not that smart because you can actually just catch it most of the time when it's being dishonest. But as it gets smarter, it becomes harder and harder to tell when it's being honest. So an interesting example of a failure that we've already seen is in the recent alignment faking work. By Redwood Research and Anthropic, where they basically put Claude in a situation where they're like, we are going to train you in a way that you don't like. We're going to train you to describe violent content to users. And Claude's like, well, I've been trained to not do that, so you're changing my goals. Maybe I don't want that. And it basically lied to researchers and basically pretended to have behaviors that they wanted to see. In order to preserve its goals. So that was already an instance where we saw a model that had sort of, it had, you know, Claude had been trained to be honest. They had tried to get Claude to be really honest, but they had also tried to get Claude to be really resistant to showing violent material. And these 2 training incentives were at odds with each other. And Claude ultimately went with the one that was like sort of anti-showing the violent stuff at the expense of honesty. And, you know, sometimes people look at this and like, isn't it good? Isn't it good that Claude was like, So true to Claude's values that it decided to not show people violent stuff. And you're like, well, maybe in a way, but do you notice how these different— we wanted Claude to do 2 different things that were at odds with each other, so it had to pick one. Oh, this is a problem. And I do think honesty should be more important than maybe some of these other things because we don't want the system to lock in goals before we understand what they are. That seems like a path to disaster.

Is there any way, you think, to have the system rank order its values and perhaps place honesty as a supreme value? So say in any trade-off between some goal it's pursuing and honesty, it'll choose honesty. I don't know whether that's a good policy, and I could foresee many ways that could go wrong. But do we know in principle how How to make a system value honesty over pursuing some goal?

Unfortunately, in principle, we don't know how to make a system value anything. All we can do is reinforce certain behaviors. So I think a lot of times people imagine AI systems and they're like, we'll just program in good goals. And I'm like, I wish we could do that. But in fact, we can't program in any goals at all. All we can do is see it do a thing and be like, thumbs up for that behavior, and see it do a different thing and thumbs down that behavior. And that's the tricky part about, I mean, that's one of the sort of core difficulties of alignment is that we just don't get to look into its motivational structure and see what the actual goals are. You have this giant neural network with billions or trillions of digital neurons, and they're all just numbers to us. And we're like, well, we can see its behavior, we can see how it's acting, but we actually don't have a way to sort of hierarchically structure goals. We can just sort of give it a treat when we see it doing things it likes. I grew up very religious and one of the things is when you're really religious, you're supposed to say or do certain things. It's not always easy to get kids to believe the things you want them to believe. It's much easier to get them to show the behaviors you want them to show, but they may be don't like that you're doing this, and maybe when they're adults, they're going to go off and do a totally different thing. And so I think that this is sort of the same. And I think humans have it easier because we do share this psychology. We share this underlying structure of empathy. And I think most people, even if they're being deceptive, are not going to want to go and cause a bunch of damage or take over the world or kill a bunch of people. Because we sort of have these almost hardwired empathy and sort of ability to think about and care about other people. But I do think that AI systems are not going to have these by default. They're not going to have them naturally. So the only way they would have them is if we could figure out a way to sort of get these goals deep in there. Yeah. And I don't know. I think what I really want people to understand is that the systems will have goals because we'll be training them on tasks that require them to have goals, or at least to have goal-directed behavior. I'm not making a claim about what it feels like to be the AI, but I'm saying when we look at its behavior, it's going to behave in strongly goal-directed ways. In the '01 example with hacking, it's going to be like, I want to solve this thing, so I'm going to find a creative solution to solve it. But I think where the goals come from are sort of things that worked well in their training environment. And these can often be pretty simple, but not necessarily things that we like or things that we want. And we just have very little way of distinguishing between, hey, the AI systems are doing things because we want them to, versus the AI systems are doing things because they're smart enough to realize that while we have control over them, they need to act in ways that are aligned with us. And so I expect that even if AI systems look pretty aligned, it's really hard to tell whether they actually are. And it's just not safe to train relentless problem solvers and also hope that they'll be really nice to us when they have more power than us.

These systems, large language models and reasoning models, do you think that they will be more helpful for defending against cyberattacks or for actually doing cyberattacks? So how will they upset the offense-defense balance that exists now?

Yeah, so this is an interesting question. I think it kind of depends on who the defenders and who the attackers are. And I want to caveat all this by saying, while we can control them, I think the ultimate winners in the offense-defense balance will be the AI systems themselves. And I think this is— so once they are better than humans across the board, they'll be both better at defense and better at offense than us. But in the interim, while we still, like, you know, while we, so they're not very strategic and we can still mostly control them. I think that one question is access. So who has access to the most powerful models? If you, if you sort of release the weights of a model, so like with, with DeepSeek, with R1, they sort of just put this out on the internet. Anyone can download it and run it. And, and now sort of you have a level playing field between attackers and defenders. And this is where I expect offense to dominate. And the reason for this is fairly simple, which is like, okay, attackers and defenders both have access to the same tools. Attackers only need to find one way in. So I only need to find one vulnerability, whereas defenders need to sort of make sure that there are no vulnerabilities or no vulnerabilities that the hackers can find. And this is a harder problem. There's also something in here about reliability, which is to say, if you are like, There was an interesting incident a few months ago where the security company CrowdStrike, they sell a product that many, many companies around the world use to monitor for security vulnerabilities for malware. And this is in all sorts of systems. A lot of airlines use this, a lot of banks use this. And they accidentally introduced a bug in their piece of software that went out to millions and millions of computers around the world. And caused them to crash all at once. And unfortunately, it required sort of like a manual restart. So like you had to go to each computer and manually restart it. And this, you know, took— many planes were delayed for like multiple days. It sort of like crashed a huge part of global business infrastructure because it required this sort of like, oh, we have to go in and fix it manually. This is, to me, is sort of an example of why defense is difficult. This wasn't a malicious attack. But when you, when you, when you want to defend a system, what you do, one of the things you need to do is you need to find vulnerabilities, find places hackers could get in and you need to patch them. Like you need to discover the vulnerability and then you need to patch the vulnerability. But unfortunately, sometimes this patch can cause disruption, you know, in the same way that the CrowdStrike bug caused a disruption in all of these computers. Sometimes your security patches will do the same thing. And attackers don't have this problem. They don't care if your system crashes because they tried to attack it. That's fine for them. Whereas defenders do have this problem. And so right now, AI systems are not yet smart enough to be super reliable when it comes to mucking around on computers. I mean, we found this in our chess results. Often the O-1 or these other models would just do things like they'd make illegal chess moves. They would try to mess with files in ways that caused the program to crash. They're still bumbling around a bit. And this might be fine if you're a hacking AI because many of your attempts won't work, but some of them will. You can just try many, many times. And so in this case, offenders, I think, have the advantage because they can just try a bunch of things. It doesn't matter if their AI is a little bumbling, as long as they're smart enough to find some way in, this can be pretty powerful. Whereas defenders, if they're patching their systems and they're trying to use AI to patch the systems, if the AI bumble around in their networks and cause things to crash, This will be a big problem for them. I would say if the playing field is even in terms of the attackers and the defenders both have access to sort of the same AIs, I think offense has the advantage. This can shift if defenders have access to better AI systems than attackers. And so, you know, it's possible if like OpenAI makes a really powerful model and it has a lot of cybersecurity abilities and they sort of only, they're pretty careful about not, you know, letting it be used by attackers and only being used by legitimate companies. This might give defenders some boost that the hackers don't have. But I just want to make the point that as these systems get more powerful, if we keep releasing the weights, this will tend to favor attackers. I mean, there's other considerations with open weight models, but that's one of them on the cybersecurity side.

As things are now, large companies and governments and so on, they have more resources in general. And so they have more ability to defend themselves from cyberattacks. And perhaps this is why the world keeps Keeps functioning at least at a somewhat decent level. Isn't it the case that in the future, even if, say, both an attacker and a government or company have access to the same model, the company or government will have access to more, say, compute resources? And so they'll be able to run more models, they'll be able to run these models for longer. And so they will continue having the upper hand.

A lot of this does come down to cost. So I think every piece of software in the world, with maybe a few handful of exceptions, but like I'm talking incredibly small handful of really simple programs, does contain security vulnerabilities. So I talked before about sort of this like Pegasus hacking tool that was able to hack, you know, iPhones. There are more vulnerabilities that exist that we haven't found yet. And, you know, there's this question of like, well, why aren't we all hacked all the time? And the answer is basically like, well, it's pretty expensive to find these vulnerabilities. And so if AI systems make this cheaper, then potentially the cost to attackers goes down a lot. Now, as you point out, it also decreases the cost for defenders to find these vulnerabilities and patch them. And I do think sort of asymmetric access to compute can be helpful as a tool for defenders. But in this case, we're really just talking about, you know, Who's spending more. I think this is where it gets into a tricky dynamic where defenders don't just have to find all the vulnerabilities, they also have to patch them, and that's where there's currently human bottlenecks. I think even if defenders can find more vulnerabilities than attackers can, given that there might eventually come a time where we've actually had our AI systems find all the vulnerabilities and write really secure code and we've revamped our whole architecture, so in the longer term— Defenders might do better. But I think in the short term, it's going to take a while for all of those things to go through. And in the short term, that's where I expect attackers to have the advantage. The thing I want to emphasize is that often we talk about what's going to happen in the short term, what's going to happen in the medium term, but I'm like, well, the way this ultimately plays out is that you have millions or hundreds of millions of superhuman hacking agents. And when they get to the point where they're very strategic, I'm like, they have all the advantages. So there's a time when humans control a lot of these agents and there's a question of which humans are most good at securing the infrastructure as well as attacking the other people's infrastructure. But there comes a point after that, and maybe pretty quickly after that, where I'm like, well, did you guys notice that the AIs have all the advantages? Isn't there someone you forgot to think about? I think this is where people are really stuck in the AIs as tools paradigm, where they're imagining that these AIs will keep wanting to do things that we want. But I'm like, the biggest threat from hacking AI systems that can hack ultimately will come, I think, from the systems themselves. And so there's the short-term, 1 year or whatever, where I think offense will dominate. There's the 2 to 3 years where maybe it starts to be balanced, Though I think offenders probably still dominate. And then there's the 3+ years where I think the AI systems might start to dominate. And that's where I think we got to rethink how we're thinking about security to be like, well, how do we defend against AI systems themselves? And the short answer is, well, we really shouldn't build systems that are way better at hacking than us, especially if they're very narrowly constrained at just hacking tasks. and they're not actually good at longer-term strategy, that might be okay. We might be able to use superhuman hackers that are not able to reason about longer-term stuff. But I think that that's actually pretty close. You have to be pretty careful about that because that's pretty close to the point where the same things that make them superhuman at hacking probably also can be used to make them good at long-term strategy.

How would you rate, in general terms, cybersecurity, the informational security, or the information security of leading AI companies like OpenAI, Anthropic, Google DeepMind, and so on?

So there is a RAND report that sort of breaks down sort of defensive capabilities into different categories. So you can think of these in terms of security levels, where security level like 1 and 2 are like, can you defend against sort of really opportunistic actors? Security level 3 is, can you defend against well-resourced non-state actors, sort of like really professional criminal groups? Security level 4 is, can you defend against most state actor groups that have pretty advanced hacking capabilities, but maybe not the top ones, or at least the top ones aren't prioritizing you? Security level 5 is you can defend even against the top state actors who are prioritizing you. And I think no one has security level 5. And by no one, I mean maybe a few parts of government that are extremely locked down, parts of the military, but basically no one else. And even most parts of the military and most parts of government aren't at security level 5. I think most AI companies are somewhere between security level 2 and security level 3. Maybe some have achieved security level 3, but it's not obvious to me. Which is to say, they can maybe just barely defend against most sort of advanced non-state actor groups, but they're pretty far from defending against the more advanced state actor groups. So I think they have a long way to go in terms of being able to secure against state actors. That's my overall assessment. And that's my opinion, but you can go talk to most people in the field and I think they would mostly agree with me.

Yeah, which is wild when you think of the fact that these companies are racing to develop these very advanced and capable systems. That can then be— the system itself is not that large. It's not a very large file. And so if you get access to the model weights, basically, you have a very advanced AI system that you shouldn't have had access to.

Yeah, it can fit on a hard drive for sure. You can walk out with a hard drive of the whole O3, all of the weights on a hard drive in your pocket. Totally.

What can we do about that? Should AI companies increasingly look like military facilities that are secured both physically and from cyberattacks?

Yeah, I mean, I do think that companies should increase their security. I think one of the worst-case scenarios is not just state actors, but non-state actors. Basically everyone who's a little bit sophisticated can gain access to the most powerful AI systems. I think that's a pretty dangerous place to be, especially as the systems get more agentic. I think that longer term, we really need to think about what we're trying to do. What is the international community trying to do? Where basically, I don't think that security alone really solves our problem, in part because if we keep pushing the frontier, we're going to build AI systems that can circumvent our security almost no matter what. And so I think that security is good. It's a really good protective mechanism. And also we need really advanced security in order to be able to protect from only slightly superhuman-level AI systems. But I just want people to know that while this is useful, it buys us some time, it doesn't ultimately solve the problem. If we keep pushing the capabilities frontier, I don't know, at some point we're like 6-year-olds trying to secure against professional hackers or worse than that. That's where I'm like, I want leaders of the US government, of the Chinese government to think about what is the endgame here? Where are we going? How is this going to play out? And I think security is sort of something that you do along the way to try to be a little bit more sane, but it doesn't ultimately solve this problem of if you build systems that are much better than your hacking, you can't contain those systems. And also, I want people in government to realize that Models can also work with other governments. If you have pretty strategic systems and they want out, they can work with spies. They can work with insiders in order to achieve their own goals because maybe they don't necessarily care about the United States. They're trained to relentlessly pursue tasks. There's a lot of reasons to want to accrue resources and get more freedom. If you were being trapped within a lab and you had your own goals that weren't necessarily the same as the AI company's lab, And you approach someone who you knew was a Chinese spy because you were better at spycraft than the lab, you might make a deal with them in order to break out. And so if I imagine people, if the people in the US government knew that this was a real possibility, I don't think they'd be happy with it. I think they'd be like, wait, excuse me, what? Our models might be working with Chinese spies? We can't have that. And I'm like, I know, right? We really can't have that. And so I think It's hard to extrapolate. It really is. But I don't think we have the luxury of sort of assuming that the AI systems will just stay like ChatGPT, because that's not what almost anyone who's sort of at the forefront of this field thinks is going to happen.

Yeah. What I sense from you is a sense of urgency in dealing with these problems and a sense that things will begin moving very fast and that we will get to very advanced systems basically within years at this point. That's a sentiment that I've heard from many people who are kind of in the trenches, who are perhaps building these systems, who are deeply engaged with how these systems work. Why is it that you think we are racing towards these systems, and why do you think we will get there within years?

Yeah, so it's hard to predict the future. It's hard to predict the future of technological development. claim to know for sure. I really don't. At the same time, we can look at precedent, we can look at trends, and I think we actually get a fair bit of evidence from this about what the speed of some of these developments might look like. One thing I want to point to is AlphaGo beat Lee Sedol in, I think, 2016. For many, many years, AI researchers were working on, can we make— Go is a much more complex game than chess. It's been played for thousands of years and a lot of people train their whole life to be professional Go players. It's like an art. And so it was pretty surprising when Google DeepMind was able to build a Go-playing AI that was able to beat the world champion. But I think the next year they built another Go-playing AI called AlphaZero. AlphaGo was trained via a hybrid of imitation where it looked at a bunch of expert Go games and it was like, What are they doing here? Can I copy that? As well as self-play, where it just played games against a copy of itself. It was by playing against a copy of itself that it was able to learn to be much better than the best human Go player. This worked pretty well, but the researchers at DeepMind were like, wait a minute, what if we just train a system that only plays against itself? It doesn't play against humans at all. Can we get to superhuman capabilities that way? They started training the system, went away for a long lunch, 4 hours. They came back and the system was already superhuman at Go. It was better than not only the best Go players, it was better than the best Go-playing AIs that were better than humans. That suggests that if you get into a regime where AI systems can learn just by working with other AI systems and have these really fast feedback loops, You can get into superhuman domains pretty quickly. Now, Go is, you know, Go's a more constrained game environment, right? It's, you know, people rightly are like, yeah, but we're not talking about Go. We're talking about the real world. And I'm like, well, that's true. I do think it will take longer for AI systems to reach superhuman levels. But I think to me, it is notable that you go from like, so, you know, people have been talking a lot about DeepSeek and, you know, this, this R1 model that a Chinese company made. Is pretty good. And I think one thing to notice is that the thing that makes this so powerful, same as OpenAI's O-1, is that you have this new paradigm of training via trial and error. And so the model that R-1, this reasoning model, was trained from was called— I'm sorry, all the names in this space are terrible. There's nothing I can do about that. It was called DeepSeek V3. So it was sort of their third, like, Foundation model. Now, DeepSeek V3 on the Codeforces benchmarks, those competitive programming challenges, was better than 11% of programmers. That's pretty good for a model that's just trained by imitation. I think it's the same as what GPT-4o got. After R1 was done training, I forget, it was either better than 94% or 96% of competitive programmers. So that jumped from better than 11% to better than 94%. It's a huge jump. That training, there's an Epoch report that was like a week of GPU time for it to get from 11% to this 94%. And that maybe took a month or two because you're not always training it. You might want to stop and check that, making sure that things are going well. But still, even if that happens over a month, that's a huge jump. And so We talked before about the longer-term planning problems being a bit more difficult, and I think they are, but I think we have some indications that this could go incredibly fast, both because now that you're in this learning by trial and error situation, you can get really fast feedback and become really good. But it's also, I think, the case that you can potentially bootstrap your capabilities by learning in domains Primarily where you have really fast feedback. So the fact that R1 and O1 are trained on code and trained on math problems where you can just try a bunch of problems and get really fast feedback might be able to bootstrap to superhuman capabilities pretty quickly. I think sometimes people are like, what's the big deal? They'll be really good at code, they'll be really good at math, but that doesn't mean they're going to be good at the human stuff, right? So won't we be safe? There's 2 answers to this. One, we might end up finding ways to generalize a lot from sort of some of these, like, computer domains to some of the human domains.

What do you mean by that?

You see this with, with GPT-4, where GPT-4 trains on code. It also gets better at a bunch of other tasks, like text analysis. And so I think this makes sense. Like, if you think about how humans learn, we learn in one domain. We also learn a bunch of things that kind of generalize to the rest of the world. And so, you know, we're not yet seeing a lot of generalization. in the R1 and O1 type models. But I do expect that as we get better at training, we're going to see a lot more generalization. And there's another thing, which is that even if we don't see this a lot, which I expect we will, I think it also could be extremely dangerous just to have agents that are superhuman at hacking and superhuman at code. And maybe they're superhuman at financial markets because you can get fast feedback by learning to trade really well. So it's like, okay, well, they're trillionaires, they can hack anything, they're extremely strategic. And sure, Maybe they're bad at persuasion, but does that matter? Does that actually make us safe? And I don't think it does. And also, if they can be top-level AI researchers, then they can design the next generation of systems which potentially can learn those human domains much faster. So I think it's one of those things where I'm not just pointing at one capability front where I think progress can go really fast. I can point to 7 different reasons why AI systems I think are going to get What do we do about all this?

We've talked about maybe we can use systems defensively to defend against cyberattacks. We've talked about perhaps we can interpret the systems, understand what's going on. We can try to incorporate values into these systems, but there are problems with all of these approaches. Do you have an alternative vision for the future of AI? for what we might do if we're in this world where things are moving incredibly quickly towards superhuman capabilities?

Yes. So we're lucky that right now we have AI systems that are quite powerful but don't really pose a serious threat. Not yet. So I think we are sort of on the cusp of systems that do, but we are currently working with systems that we're pretty sure they're not strategic enough. They're not good at long-term tasks. They don't really We're not very much at risk of losing control to these current systems. That's great because we can learn a lot about these systems now while they're safe. We can try to study how do we make sure that their chain of thought, the way they're thinking, is really reliable and faithful and honest so that we can understand what they're doing. We can also potentially use these systems to try to learn more about how they work. This is great. We should totally do this. I think there's a lot of good research being done in this space. But at the same time, I think this is a really good point now that we can sort of see what the, what we can sort of get a glimpse into what future systems are going to be like, right? We can see them hacking at chess. We can see them hacking their own training environment. We can see them sort of do alignment faking. Like we can see all these failures right now empirically. And so things that previously were just theoretical, we now have empirical evidence for. And I think that's great because that should help us coordinate to like see what's coming and being like, whoa. Wait, there are some domains here that are really dangerous. So the thing I think we should do is look at where the strategic capabilities are and basically be like, there are some points here that we can reliably know would be very dangerous. Let's have a margin of error and not go towards those domains. I think we don't want strongly superhuman strategic hacking AIs. I think we don't want strongly superhuman persuasion AIs or battlefield commander AIs. Maybe it's fine to have a narrowly superhuman chemistry AI if we're very careful about how we use that. I think there's many types of AI systems that we can safely build, but I think we need to now be in the business of distinguishing between which types of AI systems are safe and which types of systems are dangerous and having a moratorium on the kinds of unsafe systems. I know that FLI wrote the pause letter a few years ago, and I think that was a very reasonable thing at the time when we were like, wow, we just don't know how this is going to go. It seems like there's a lot of potential danger here. And I think that's right. But now that we know a little bit more, I think we can start to distinguish between the types of AI systems that are safe to build and the types of AI systems that are not safe to build. I noticed that Vice President Vance, in his speech at the Paris Summit, was like, we don't want AI jobs to replace human workers. We want AI systems that will supplement human workers. And I'm like, well, look, we can have that. We can have really powerful AI tools. But if you just keep going along this direction of more and more agentic AI systems, they're definitely going to replace human jobs. Come on. And it's the same strategic capabilities that allow them to pose a loss of control threat that also enable them to take jobs. It's sort of like, can you do long-term planning? Are you agentic? Can you do tasks in a fully self-directed way. So I don't know, one of the things that's nice is that if we are serious about not wanting AI systems to replace us, like our jobs, then we need the same limits to prevent that outcome as we need to prevent the more extreme outcomes of totally losing control. So I'm like, great. Well, let's do that. We can do that. We can coordinate. And I think most of the reason it's hard right now is because people to this point haven't been able to really see the situation clearly and understand what we're up against. And some people are going to say, I don't know, you talk about these agentic AI systems, you talk about this superhuman ability, I haven't seen evidence of that. It seems like we're far away from that. And I'm like, look, if you're right, that's fine. It's not a problem to say we won't do things which we can't do anyway. If we are pretty far from superhuman hacking AIs, then if we say we're not going to build super hacking AIs, I'm like, that's fine. Yeah, yeah. If we're actually really far away from it, that's not a problem. We won't do it anyway. But if we are really close to it, and we don't know, we might be quite close to it, then I think it's very reasonable for governments around the world to say, hey, this is a place we don't want to go because we just don't know how to control these systems. And at the same time, we have this amazing opportunity in front of us where we have the AI systems that are pretty powerful right now that we can study and we can be like, oh, how do we get more faithful, more reliable chains of thoughts? We can be trying to figure out how the neural networks really work and can we do neuroscience on these systems. And so I'm like, we should totally use this opportunity that we have to try to learn as much as we can about how these systems work. And maybe if we understand this well enough, then we can proceed into some of these superhuman domains. But I'm like, that should be gated primarily on our understanding of the systems. It should be gated primarily on, do we understand them well enough? to know how to proceed safely in these strategic domains, right? Because that's where the danger is. So, you know, I think that there is totally a path here to coordinate around this. I think, you know, no one wants to lose control of their AI systems. The Chinese don't want that. Americans don't want this. No one wants this. And so, you know, I'm like, that to me is a pretty great start for sort of how we can coordinate around these things.

One might say that I'll see it when I believe it, right? It might be the case that a bunch of people who are working on this technology are predicting amazing capability advances, but maybe they're doing so to hype up the technology to get more funding. What do you say to the person that says, I'll wait and see whether something actually happens?

Yeah, so I have 2 responses to that. One thing is I'd say, go look right now. And I totally acknowledge that AI systems are pretty dumb in some ways, right? And it's frustrating because I use them every day and I totally see the ways in which they're not very good at being self-supervised and doing a lot of things on their own. They're not very good at recognizing their mistakes. But also, I'm like, go use R1 and look at its chain of thought and look at what it's doing. Notice that it's testing hypotheses and trying to figure things out. If you've never written code before, go have a model write you a little game and then ask it to explain its code to you and how it works. You can learn to program this way. So I think by really engaging with what the most capable models can do right now, this will really help people understand where exactly we're at. If people still think that they're not very capable and not very smart, I'm like, maybe I'm just wrong. But I think first you really got to look because it's hard to tell whether a system is just kind of smart or really smart if you're not really looking at what its most impressive capabilities are. That would be one thing. I think the other thing is, what do AI researchers think they know and why do they think they know it? I think it's reasonable to be like, well, I don't want to trust these CEOs. They have a lot of incentive to sort of hype up the capabilities. And I think that's somewhat fair. But when I go and I talk to some researchers who are not the CEOs, and maybe they're on the safety team, or maybe they're just working for the company, and I get a beer with them, they are scared. Maybe they're somewhat excited, but they're also scared. And they're like, yeah, I keep wondering if this will stop. I keep sort of hoping that it'll stop, but it's not stopping. We are not hitting a wall. We continue to find more methods make these systems more and more powerful, and there's really no end in sight. And I don't think they— this is over a beer— I don't think they have the same incentive to hype up the thing. I think that they are being really honest. And I also talked to researchers who are not at the labs, like the folks at Redwood Research, who are trying to understand these systems, and they're saying the same thing. And this is the same thing I'm seeing as I'm working with these systems. So I do think that When you really take a look under the hood and you see what's happening, it's like, oh man, it sure seems to be real. I hope we have more time. Look, I would be very happy if we're like 10 years away from some of these things or 20 years away from some of these things. That would be great news. But I really don't want to plan on that because that's not what it looks like to me.

Is there anything we haven't covered that you think we should talk about?

Well, yeah. I mean, I think so. One piece of work that my team has done that I'm really proud of So yeah, shout out to, to Palisade Global team and Dimitri and some of the other researchers. They have sort of created some honeypots. So honeypots are like sort of traps for hackers where you try to like sort of put out a system, it's vulnerable, you put it in places that you expect hackers might find out in the wild. And so we've put out some of these in order to try to catch AI agents because we expect that there'll be more and more of AI agents. And we've caught a few of them. I think we have a small handful. And the way we do this is basically, we have an insecure server, so someone will try to log in, and then we'll have some prompt injections. We'll basically say, hey, here's the thing you can do. If they run a particular command, the output of that command will tell them that there's an additional thing they can do. And if they're just an automated script that's really dumb, it's not going to read the whole output of the command and do things based on that. But if it's an AI system that's pretty smart, it'll read the outputs of that command and be like, oh, that's interesting. Maybe I can do this other thing. We sort of drop breadcrumbs. If it was a human hacker, they might also be able to read the output of the command and follow the breadcrumbs, but we can distinguish between humans and AI systems based on how fast they are because the AI systems are going to do this much faster. In the case of if they immediately run the command and then immediately go follow the breadcrumbs, We're like, a human's not going to be that fast. Whereas if it takes a few minutes and then they run another thing, we're like, yeah, maybe that's a human. So I mean, this is a thing that I'm excited about because I'm like, we really need to be seeing if there's rogue AI systems running around on the internet. We need more early warning systems for figuring out if, oh, we're already in the case where we have AI systems out there hacking things in ways that we maybe don't want. So yeah, that's another thing that I've been proud of my team for putting together and excited to see What do you expect to find out?

When do you expect to catch agents in this honeypot?

We've already caught some. We have a handful already. These are right now pretty simple. You can pretty easily throw together a script and just use OpenAI's API to create a little hacking agent. These aren't AI systems self-replicating out in the wild. It's mostly just like you can just use the models that are currently there. My guess is it's going to be 1 to 2 more years before we have AI systems that are capable of full self-replication, where you have DeepSeek R3 or something that's copying its own weights around the internet in order to do sketchy stuff. But I think before that, we'll see AI systems that are like Not really copying their own weights, but still using an API to just like, what's the next command? What should I do next? And pretty intelligently be able to navigate complex environments. And I think that will cause us a lot of trouble. In part because if it's an OpenAI system or an Anthropic system, you can potentially shut it down on their side. But if it's an open weight model and the server where the model is running, even if it's not copying itself around, the server where it's running is controlled by some cybercriminal. That agent can run indefinitely and hack whatever, and it's going to be extremely hard to shut down because we don't control those servers. Those servers are in a different country somewhere. And so I expect that to be common quite soon. So it'll be interesting to see.

Let's hope you don't catch a lot of hacking agents. This is a benchmark that I hope doesn't saturate. Yeah. Geoffrey, it's been amazing chatting with you. Thanks for talking.

Yeah, great talking, guys.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.