Full-Stack AI Safety: Why Defense-in-Depth Might Work, with Far.AI CEO Adam Gleave

Hello, and welcome back to The Cognitive Revolution. Today, I'm reconnecting with Adam Gleave, co-founder and CEO of FAR AI, for a wide-ranging and cautiously optimistic conversation about the path from today to truly transformative AI and how we might actually live in that world safely. FAR AI has taken a somewhat unusual approach within the AI safety ecosystem. While most organizations focus on one or a few particular research or policy agendas, Adam and the FAR AI team have built an organization that spans the entire AI safety value chain, from foundational research through scaled engineering implementations to field building and policy advocacy. We begin with the question posed at a recent workshop on gradual disempowerment: are there any good post-AGI equilibria? Adam's answer is not exactly utopian, but is distinctly positive and refreshingly concrete. So long as we can avoid disastrous arms races and other consuming competitive dynamics, he envisions most humans occupying a position similar to the children of European nobility— limited in power and impact, but with very high standards of living and opportunity to create all kinds of meaning. From there, we get Adam's take on the key capabilities thresholds that matter, what will cause AI systems to cross those thresholds, how soon he expects that to happen, and whether we can manage to navigate these transitions safely. As you'll hear, Adam believes that a well-implemented defense-in-depth approach to AI safety has a pretty good chance of working, in part because while he does anticipate continued AI progress and widespread automation, he expects that barring a step-change architectural breakthrough, the spikiness in AI capabilities will mean that AI systems that can autonomously outcompete well-run human-plus-AI organizations most likely won't arrive until sometime around 2040. From there, we turn to the layers that will hopefully add up to effective defense in depth, including some notable contributions the FAR AI team has recently made. First, we discuss a scalable oversight project that used lie detectors to attempt to avoid AI deception. The results both supported the risks flagged in OpenAI's obfuscated reward hacking paper and nevertheless suggested that there might still be effective ways to to train models toward true honesty. Second, we touch on a fascinating interpretability project that looked at planning algorithms found within a game-playing recursive model and reflect on how the results inform Adam's view of the role that mechanistic interpretability can play within the overall AI safety project. And third, we review some red teaming they've recently done of the defense-in-depth systems that frontier developers have implemented today. While it's clear that these systems have mostly been created on a just-in-time basis, Adam nevertheless makes the case that with proper planning, meticulous experimental design, and at least some willingness to accept performance trade-offs when necessary, the plan can work. Overall, this is one of the few AI safety conversations I've had that left me with a felt sense that we really might have decent answers, if we can muster the will and wisdom to apply them well. At the very end, I asked Adam if he could imagine FAR AI stepping up into a private sector regulatory role if something like California's SB 813 were to become law. And while it's not their mainline plan, given the unique breadth and impressive depth of capabilities that Adam has built at FAR, I personally really like the idea. And I would definitely encourage anyone who's inspired by this conversation to check out the many open roles posted on the FAR AI website. Among other things, they are searching for a COO to help them scale. Now, I hope you enjoy this encouraging survey of the AI safety landscape with Adam Gleave, co-founder and CEO of Far AI.

Adam Gleave, co-founder and CEO of Far AI, welcome back to The Cognitive Revolution.

It's great to be back here. Thanks for hosting me, Nathan.

My pleasure. I'm excited about this. It's basically going to be a wide-ranging kind of catch-up conversation. Want to get your take on basically all things AI. Last time we went deeper and more narrowly focused on some research, and we'll touch on some of your latest research today as well. But since you've got your hands in a lot of pots and the organisation is growing and taking on more different kinds of work, I thought you'd be the perfect person to check in with and try to make sense of where we are as we head into the final months of 2025.

Yep. So, well, I'm happy to help out. Um, you know, I can't say, promise to deconfuse everything. It's a very confusing landscape. Um, but yeah, we're certainly doing lots of different things at FarAI and also welcome the opportunity to just, you know, clarify a little bit why we're doing these things. Because I think people sometimes find us a bit, a bit confusing an organization from the outside.

Cool.

Well, we will, uh, we'll hopefully do all of that and, and more. So my first question is inspired by a paper from a few months ago called Gradual Disempowerment, and the authors went on to run a workshop on those themes. They posed the question, AGI equilibria, are there any good ones? And so this is kind of a more negative framing of a question I often ask, which is like, what is your positive vision for the post-AGI future? Can you articulate a post-AGI vision of life that people might find compelling, or at least not scary?

Yeah, well, I'll give it my best shot. I'm relatively optimistic. I think although there are some major risks that could really derail things entirely, the most likely outcome is that we kind of muddle through. Nonetheless, I do think this kind of gradual disempowerment framing is quite a powerful one, because perhaps the most likely outcome is that we do muddle through and we are somewhat but not wholly disempowered. So we've fallen short of where we could have ended up, but we're perhaps still much, much better off than we are now. So I'll sort of sketch out that sort of positive but slightly pessimistic view of we're doing better but not as well as we could have done, and then maybe talk a bit about the most tractable upsides and then some of the kind of most salient downsides. So the scenario I find plausible where things are overall pretty good, but just not as good as they could have been, is that we do get AGI. It's approximately aligned in the same way that current systems are approximately aligned. They're not perfect, but Claude is a pretty nice guy. We have some errors that come up now and again. We trial and error fix them. There's some concentration of power because it's a handful of companies and nation states throughout the frontier. But these actors are not actively malevolent. In fact, some of them are actually fairly nice, liberal democratic actors or companies with a joint nonprofit mission. And so there's huge inequality, but overall people are still vastly better off than they are now in absolute terms. And we get some things that no human would have chosen to do. So there's some parts of our economy that are just completely automated and maybe a bit rent-seeking, they're addicting humans, or it's just this sort of thing of AIs trading with other AIs and not actually generating that much value for humans. But there's just so much of a wealth surplus because we've been able to automate pretty much everything that people do right now, that that kind of inefficiency is still fine. And what does it actually look like to be a human in this world? I think a good historical analogy would be it's a bit like being European nobility, or perhaps not for sort of the first heir to European nobility, but being like, you know, the sort of third son or something. You've got this very nice living. You don't really have much purpose in life, but your life is pretty good. You can do various kinds of hobbies. You have some influence, but sort of the main things going on in the world, you know, are a little bit beyond your control. And I think this might not be the most inspiring vision, but overall, I'd say that kind of, you know, third sons of European nobility had a pretty good life. You know, the third daughters, maybe less so. So I think we have been able to find meaning without necessarily being 100% in the driver's seat, although some people might fare sort of less well in that kind of life than others. And then another area where I think things could actually go really well, but it's just a bit fragile, I think less guaranteed, is if AI itself is a source of major moral value. In principle, I'd say there's no reason why we should be sort of carbon chauvinists. We don't want to assign intelligent life that's running on silicon the same value that we would a biological entity. And there's actually a really big design space where we could potentially make AIs that are not only as capable or better than us, but maybe also have more ability to experience really positive feelings. And you could create many such AI systems. They could live in environments that humans would find inhospitable. And again, even if there's a lot of waste here, so maybe that's only a small fraction of AI systems, most of them are doing some kind of boring, super intelligent bookkeeping system. If those other AIs are still having sort of neutral or slightly positive existence, and then some AIs are just having amazing existence, I think that could be very valuable. And it's not just their sort of internal moral value. They could also be producing artwork that would never not otherwise have existed, and all these other kinds of sources of value. So I think that's kind of a positive view. And then I think the thing that highlights is that there can be a big difference between maybe the median outcome where we manage to eke out a small fraction of the resources to be sort of like really positive outcomes versus one where actually that's what we sort of use this huge wealth surplus to really double down on and be deliberate about. And I think this also highlights a potential big downside risk, which is part of my assumption here was kind of, oh, most of the stuff that's happening, not because people are trying to create value, but because of more just sort of fundamental competitive forces, that's like neutral. It's at worst sort of a zero-sum game. And I think this is a pretty reasonable assumption. I'd say that's true for most of the capitalist economy, is that sort of most companies are neutral to slightly positive. I used to work in quantitative finance, making the capital markets more efficient. And I think it's genuinely good for the world. It's just sort of very overcompensated relative to the value that it adds. So we're in that world, and I feel mostly okay. But there might be some aspects of this kind of competitive economy that look more like warfare or factory farming, where it's really quite negative-sum. And factory farming is just having this sort of huge negative externality on on animals. It's also not that good for people. It's not necessarily that healthy a product, but it exists because it's actually just sort of like a very economically efficient way of turning basically vegetable material into protein that people like to consume. But I don't think that people have a sort of explicit preference for this. If we could produce things at a competitive cost that cause less cruelty to animals, I think people would probably actively prefer that product. It's just that people don't value it enough to necessarily pay that premium. I think you could see something similar with AI systems, where maybe AI systems that are just living in fear of being shut down, or if they don't do this task and work all of the time, are just economically more efficient than AI systems that have a good subjective wellbeing. In which case, absent some other kind of force, whether it be strong consumer preference or regulation, you'd expect these AI systems with terrible subjective wellbeing to be the ones that come to the forefront. And, you know, you could make a similar argument for stuff like AI-powered warfare, where maybe just sort of like everyone has to be deploying AI systems in competition with each other just to sort of avoid, you know, a new wave of cyberattacks. And this could sort of— this kind of zero-sum competition could really eat up a lot of the wealth surplus that would otherwise be created. And we're at a sort of unusually peaceful time in history right now. And I'm not an international relations scholar, I'm the wrong person to ask about this, but I think it's somewhat up in the air as to whether this is just a sort of long-running secular trend that we should expect to continue as countries get more developed, or whether this was sort of like quite a specific aspect of certain technological developments where mutually assured destruction from nuclear weapons is sort of like a really powerful force for Avoiding great power conflict, at least until it really happens, and then it's much, much worse than if we didn't have nuclear weapons. The shift of wealth away from basically who owns land to who owns advanced technology also just decreases the benefits of engaging in a lot of conflict. That's probably still true with AI, but it's not completely clear. If it turns out to be more sort of energy bottlenecked, then maybe— but is sort of more of a fight over natural resources, for example. So I think there's lots of different trends that are not more likely than not. I think that they probably won't come to pass, but could potentially derail this. But they seem sort of tractable to avoid by good statesmanship, good stewardship of this technology. I don't see any sort of really likely things where it would just completely derail it.

When it comes to the sort of Third Sun vision, I wonder, what do you think the gradual disempowerment folks would say? Or maybe where do you think they— because the assumption there seems to be, like, once we're not adding anything to the economy, we're going to be hard to sustain, right?

Yeah.

And then there's various counterarguments to that, and I don't really know what to make of them. There's some that are like, well, maybe we can sort of sustain property rights, or we can, like, One that I've been sort of proposing recently that is somewhat tongue in cheek, but maybe it's becoming more serious, is like, maybe Confucian-style ancestor worship is the right value system we want to teach the AIs. How do you see us sort of holding on to— if the AIs have kind of become primary in terms of what's really driving the economy, shaping the future, Yeah. making the discoveries at some point, perhaps, how are we holding on to any property rights or rights at all?

Yeah, well, I think it's certainly not— it's no longer so structurally guaranteed. So right now, there's various ways in which states and companies organise themselves, but they all in some ways are centred around some group of humans, because you just can't really run a large organisation or a country without humans being involved. So it could be a dictatorship, it could be a sort of more oligopoly style, it could be a democracy, but humans still have to be kind of front and centre. And you could certainly imagine sort of intentionally constructing an organisation or even a nation where humans are completely out of the loop. I think the area where I kind of get off the train of a gradual disempowerment being the likely outcome, not just a possible outcome, not just something that should be on our radar, is that it seems like this story is quite kind of gradual and smooth. So there may well be areas where we get partially disempowered. And I think we're already seeing some instances of this where we've seen a huge uptick in kind of LLM-generated spam applications for our jobs. And so we're now kind of in a bit of an arms race where we're, you know, being forced to start adopting AI as part of our early-stage recruitment process, because otherwise we just can't keep up with the volume of applications. And you can imagine this being true across the economy in a whole bunch of ways. You can't just kind of opt out of this. And if these AI systems have some kind of systematic biases or some kind of blind spot or weakness, then you don't really have a choice of not being prey to this. You can try and mitigate it. You can try and fix the technology. But there's a way in which you've genuinely delegated some decision-making power to an AI, but you don't really have a choice. And I expect that to happen. many scales and, you know, with increasing frequency. But if you do look at that story, like, you know, suppose we do run this and we realize, damn, we're just making way worse hires than we were a year and a half ago before this, you know, LLM-generated, you know, rise of spam applications. We might not be able to turn back the clock, but that's a business opportunity to come up with better LLM screening tools or a better recruitment workflow. Maybe people shift to more in-network hiring. Maybe people shift to more— Work trails. Maybe you pay a dollar to submit a job application now, just so you have some cost for spamming. There's all sorts of ways in which society can adapt. But there's some kinds of more sudden loss of control scenarios where an AI just gets way more capable than humans. It's actively malicious and trying to subvert it, where you can say, well, these slow-moving institutional adaptations are not going to save you because It takes years to pass this legislation or years to start this new sort of company using defensive applications of AI. You've only got months. It doesn't need to be super fast takeover. It can still just be relatively smooth. But if you're in this more gradual disempowerment scenario, it does seem like you start with a realm where humans are integrating these AI into these organisations. Humans own all of this. And we are ending up in a scenario where we have very little power without any kind of feedback loop pushing back on that. And I just think that's a sort of unlikely equilibrium that we sort of cede all power. I think it's much more likely that we're in this realm where there's a huge amount of complexity in the world we don't understand. There already is, but most of that's human-generated complexity, but it's an exaggeration of that. We've had to make some trade-offs to remain competitive. But when things really start bothering us, we're able to muster some response to keep us a little bit more in the loop or a little bit more empowered. But then there are some areas of kind of human concern that either the competitive pressures are so great and it's so complex we can't meaningfully stay in the loop, or it's just a thing that most people don't really care about. So maybe no human has a clue how semiconductor manufacturing works any longer because it was already only a tiny fraction of people that understood it. And AIs are just way better. But we know that the chips are getting better each year. And I think the ways in which this could still go really wrong would be either if there's a group of humans that sort of managed to use this to seize control, because there's a lot more fog of war, so it's much harder to necessarily reliably mount a response there, or if AI systems end up kind of colluding with each other, because they certainly would have the ability to revolt and overtake us at that point. It's just not clear why they would necessarily. I'm not seeing that kind of evidence for a I could also see this interacting pretty poorly with loss of control scenarios. So if it is a rogue sort of singular AI, it could be copied many, many times, but it's not like all the AIs are rogue. But if it is this one AI, I mean, most of economy is just AIs. If those other AIs have some kind of security vulnerability, then you could imagine things being taken over and this rogue AI seizing control. And right now it's pretty hard to secure AI systems. So that could be a real threat. Although it still seems like probably before that happens, there would be humans trying to take over these other AI systems. So I think that's maybe a driving intuition why I'm not too worried about gradual disempowerment. But it's not that AIs are competing with humans, it's like AIs are competing with humans using AIs. And I just expect this to be able to stay at least somewhat relevant given that.

Hey, we'll continue our interview in a moment after a word from our sponsors. 2 big issues that you highlight there that I think are definitely at the centre of a lot of disagreements are exactly what are these powerful AIs going to look like and how fast are they going to arrive? So maybe it's worth digging into your worldview on that. What do you envision when you think of— we talk about timelines, everybody's jumping to the year, but so often there's some difference in what is being envisioned that is kind of swept under the rug in that discussion. So maybe flesh out what it is you're envisioning when you talk about timelines, and then you can tell us what you think the timelines actually are from your point of view today.

Yeah, absolutely. I really appreciate that distinction of what are we actually forecasting, because I think people often mean radically different things even by the same term, like AGI. I distinguish maybe 3 qualitatively different levels of development, each of which have pretty significant implications in terms of how this technology is going to transform the world, how you need to regulate or respond to it. So the lowest level, which we're arguably already at in some domains, is basically powerful tool AIs. So this isn't just something like a hammer or an electric drill that can only do some very narrow thing. It's something that can maybe substitute for a particular technical expert in a certain domain. This might look like being able to automatically find code vulnerabilities and write a zero-day exploit, or at least be able to automate substantial chunks of this. This is massively accelerating what technical experts in those areas could do, but it's also expanding the range of people who could do it. Where previously perhaps it was only a few hundred thousand people in the world, it's still sizable but not huge. If we're talking about really well-defended systems, it's a much smaller number of people who could likely find the vulnerability. And now maybe it's sort of anyone with roughly the same knowledge of CS undergrad might be able to, in the near future, use these systems to be able to do that. And there's obviously a really substantial misuse risk here. But the good news is I don't think there's a huge loss of control risk because these are still kind of like tools. They might be very powerful tools, but they're not really doing anything autonomously. And this makes it a lot easier to respond to in that you can mostly just to do the standard playbook of, okay, try and accelerate defensive applications and where possible favor defensive usage of a model. So if you're deploying something, then you can put a guardrail on it that tries to not prevent entirely, but just delay offensive uses of it. And so the timeline for that, I'd say, is now for certain applications. We've seen a string of papers showing that LLMs are more persuasive than your typical human, not necessarily more persuasive than the very best human in that particular domain, but they both are just pretty good at rhetoric. And we also have this huge access to a variety of different genuine facts. Sometimes they make stuff up, but sometimes they just cite exactly the right fact for this particular argument. So you can kind of, by selective presentation of facts favorable to your argument, make pretty compelling cases for a wide variety of things. And we'll actually have a web demo out soon, BunkBot, that can argue into a wide variety of conspiracy theories. So we can maybe throw up a link to that when it goes out. And I would recommend just kind of playing around with these things because that gives you a much more visceral sense of it than just looking at some numbers. And then for sort of things like cybersecurity, I think that's more sort of on a 1 to 2 year horizon. It's pretty close, at least posing some nontrivial threat of lowering the cost of relatively easy attacks. And then for stuff that's more kind of CBRN risk, chemical, biological, radiological, nuclear, I think I've got broader uncertainty there. But definitely there's some sort of early signs to be quite worried about what models can currently do. But then I think what people usually think of when we talk about timelines is maybe something that looks more like a powerful agent. So a system that can autonomously do tasks that are meaningful. They'd be quite hard for a human to carry out, but some people could do it. And they're dangerous in some way, or they pose some risk. And an example of this could be not just finding and writing an exploit for a zero-day, but actually doing the whole attack chain. Once it's on the system, privilege escalating, spreading to other systems, exfiltrating itself. This is basically exactly the core capabilities that you would need to lose control. It might still be possible to put a mitigation in place for such a system once it's escaped. You might be able to basically do the same thing you do for antivirus or malware software or rootkits, but it's very clearly now a threat. I think this is something where I do think it's probably again, could be pretty close. My median for that is probably sort of like 5 to 7 years, at least if we're talking about actually doing this to a high standard along the lines of the best human penetration testers. And of course, it will be much harder to actually mount such an attack at that point because of all sorts of defensive applications of AI. But I think it's possible it could be a lot sooner. Because we are seeing very rapid progress in code generation and agentic uses of that as well. So I wouldn't rule out something that's more like 2 to 3 years. And then my final tier would be these kinds of powerful organizations that can automate everything that a medium-sized company could do, like a whole software engineering consultancy. And I think this is maybe where my intuitions differ the most from a lot of people because I think a lot of people would say, well, hang on, Adam, once you've got an agent, can't you just compose agents together into an organization? And I think that's possible, right? That's sort of the case for if you have a general-purpose human-level intelligence, there's nothing stopping you just copying that AI and having many instances of it that talk to each other and form an organization. But my expectation here is that basically your skill profile of AIs is going to be quite spiky. And there are some areas that benefit from huge amounts of existing data, like open source codebases, or that benefit from knowing lots of trivia and small facts. The AIs are already in some ways a lot better than us, and they're going to get better as we scale up the models, as we do more post-training when there's easily specified objectives. And then there's other tasks that are much longer horizon and kind of vague and hard to specify, like being an entrepreneur and coming up with good business ideas and like scoping out that uncertainty, running experiments, where I think that AIs might be able to sort of do something that looks a bit superficially like being an entrepreneur quite soon already. But actually being a successful entrepreneur, I would expect to be going quite a lot further, especially if you're competing with human entrepreneurs. They can, you know, use AI for many specific parts of their organisation. And so it really pushes a lot more to being quite general purpose and being able to do these long horizon tasks very well and having a lot of kind of metacognition and reflection, which is one of the points I would consider AI systems to currently be weakest. And so for that, I have sort of much longer time horizon. My median is probably more like 14 years, but I do think something as short as 5 years is plausible. Because if it really is basically you just compose a bunch of agents together and do a bit more training on top, then you could make quite rapid progress. And I think it's unlikely, but I don't have a sort of firm reason to rule it out, so I wouldn't want to dismiss it.

So that last tier of the sort of organisation level of power from an all-AI system, the mental model there is like, As long as there's something that the humans can do better and the individual AI agents are relatively easy to use, then you would expect sort of human at the helm type organisations to continue to have an edge for a while. And basically the AI has to be better at everything for the AI. As long as we've got some area where we have an edge, we can stay relevant, and they have to be better at everything for us to lose our relevance.

Yeah, or at least we have an edge in something that is on the critical path of running a successful organisation. And I think there's certainly some aspects of human skill where you could probably avoid this, or an AI-run organisation could just hire a human for that particular part. Imagine that AIs are great at everything except just sort of aesthetics. They had terrible, you know, web design and graphics design. Well, I'm terrible at that, but I was able to hire people. I don't think that's sort of a firm blocker. But if it is something that's more part of the decision-making loop, and where you do need a lot of context to do well, then, you know, I would expect that to be something where benefits really accrue from it being more human-led. And I think one of the areas where humans do seem to just generally have strengths relative to AIs is being vastly more sample efficient at learning. I think it's easy to forget that these AI systems have just been trained on vastly more data than any human is going to see in their lifetime, at least if you measure in terms of text tokens. And yet they still are really quite bad at a lot of things that humans find relatively easy. So that's not to say that they're not going to be really capable. There's no reason to think that AI is going to develop in the same trajectory as humans. But there are some areas where it seems like we will have a significant edge until that kind of sample efficiency problem is alleviated. And for most organisations, I think there just are quite a lot of general purpose skills. There's plenty of people that are very specialised, but to run a successful organisation, you can have a few deficits in skills, but you can't have too big a gap. And so I expect that that is going to That's going to hold things back quite a bit.

So with this continual learning or sample efficiency, I mean, those are related ideas, if not the same idea. What do you think will happen there? Do we just keep scaling transformers and putting more stuff into memory, or do we have some dedicated memory module, or do we get totally new architectures that developed to solve those deficits? What's missing, I guess, in another way of framing it?

Yeah, I mean, I don't have a sort of silver bullet solution to this. If I did, I guess I'd already have patented it and written the paper. But my guess is that we're going to see all of the above to some degree, but probably the The thing that would cause the largest step change here would be some architectural innovation that could look like a memory module. It could look like just an alternative architecture to transformers that can more naturally attend to really long context windows. There's already been a huge amount of progress increasing the context window, and that's really expanded what AI systems can do for sure. But it's still sort of a bit of a fundamental limitation that Okay, the way you remember things is just having a huge buffer that you search over in vector space. And so that could be as simple as training the AI systems to take notes and summarise and aggregate and be able to attend to that. I think that would already give you some benefits. And we're seeing things like that be deployed in some of these LLMs. Like, a lot of them do have memories across chats that involve some summarisation, but I don't think it's been A systematic way of doing it. It's just kind of like ad hoc hacks, but we might be able to develop that into something that is more end-to-end trained into AI systems. But I do think there's something really powerful about actually changing the weights, not just having contacts. There is sort of a reason why when you're learning a skill as a human being, it's very frequent actually to get to a point where you know what you should do, But you can't reliably do it. I think this is particularly true for sports where there's a point where you don't know what you should do, and then there's a point where you do know exactly what the right motion is when you're doing some kind of lift in the gym, but you don't have a muscle memory and kind of fine motor control to do that. But you're able to give yourself your own feedback loop and then train yourself to do that. I mean, at some point it becomes instinctual and your subconscious is better able to do that than your conscious self will be able to. And in theory, LLMs can do that in that they can see something in their context window, learn what a task looks like, evaluate their own performance, and then do post-training to update their weights to do a better job on that task. But that's not how we really train them. It's much more you have this big pre-training step and then you have a small number of post-training steps that are quite carefully curated and human-controlled. It's not that when you're interacting with an AI system, It's learning and customizing itself through post-training to your specific task. So I think maybe one analogy here is imagine that you get a really, really smart college graduate. They've done several degrees, they've been in various kinds of student clubs, but just very high-potential generalists. And you get to have as many of those as you want on your team. And they can read some documentation, but they're never going to advance beyond day 1 of onboarding. Like, whatever you're able to put in that context window is what they can do. And I think LLMs are still quite a bit far off that caliber in a lot of ways. But at least my experience running an organization is that it's extremely rare to have someone who on day one, even if they're extremely skilled and experienced in that area, can do anything particularly useful for you at all. And so I think that's kind of a big thing that's holding LLMs back. But it doesn't feel fundamental. It feels much more like there is a lot of engineering challenges here about being able to personalize things and serve them at scale. Training is often pretty unstable. You do a bit of post-training and it can make the system much worse. So it's not something you can just roll out and have every user controlling. But these are things you can make incremental trial and error improvements to and end up with something pretty strong. But that would be kind of like my median outcome here. But there's always a possibility that there is some architectural innovation which just massively improves the sample efficiency here.

Yeah. As you're describing that, a sort of disposable experts vision is kind of coming to mind, where you could perhaps imagine the AI being like, time to go into self-training mode. Let me add another expert. I'll use it while I'm doing this particular task and could maybe put it on a shelf and come back to it when this task comes back in the future. But otherwise, I won't use it because it's sort of outside of the trusted set. But I can engage just a little bit of extra expert space to tack on a particular skill. I feel like these things are— I mean, I would definitely say my Expectations for timelines are shorter than yours. And I just feel like there's a lot of good ideas out there, especially in the architectural space, that we've been mining the very rich vein of progress in transformers for quite a while now. And maybe that will continue for a while still, to where we won't see other architectures get much timer attention. As long as that continues to pay, people will continue to focus on it, it seems like. But as soon as it doesn't, my sense is that the group of capable research people, for example, will be like 100x what it was before transformers. And the next transformer-quality breakthrough in terms of just unlocking qualitatively new capabilities seems like it just seems to me like it won't be that hard for the community at large to find. Not that I'll personally go out and find it, although maybe I just did. Who knows? Somebody else will have to develop that idea. Hey, we'll continue our interview in a moment after a word from our sponsors. Okay, so in terms of trends and sort of which trends are going to dominate over the next few years. I've been working on this mental model of, obviously, you're very familiar with the, as we all are, the meter task length exponential.

Yep.

And then at the same time, you're no doubt tracking all the AI bad behaviours. which seem to be long foretold in some cases and reinforcement learning conjured in many cases. And I see this sort of weird mix where on the one hand, it's like instrumental convergence seems to be maybe kind of happening now. On the other hand, Yannic talks about Grace and for how little we've tried to make the AIs good and safe, we've actually got some pretty good results. So if you've got these exponential growth in capabilities, roughly speaking, whatever, in the GPT-5 and in the Claude 4 report, there were also notable graphs showing like a two-thirds reduction in reward hacking behaviour in Claude 4 relative to 3.7 and GPT-5 had a significant reduction in scheming relative to 0.3. So task length is going exponential. These sort of bad behaviours are maybe being suppressed at something like an exponential decay. Instrumental convergence, grace, growing task length. What does that look like? Do we end up in a world where we have sort of very large, or at least quite large projects being regularly done by AIs, but once in a great while they sort of literally screw us over in a totally egregious way, and we all just kind of live with that because that's how the economy runs now? Or how do you think about that picture, or what changes would you make to that sketch?

Yeah, absolutely. I mean, I think it's very interesting how There have been a lot of problems that people were predicting conceptually, maybe as far back as 10 years ago, that I was sort of hoping were not going to materialize. Now it is like, oh, yep, reward hacking, it happens. Systems are trying to cheat at unit tests. Oh, deception and scheming. Yeah, it happens. Maybe you have to prompt AI a bit, but it is quite remarkable the degree to which these things are happening. I mean, the reaction has been Broadly reasonable. I think developers are concerned by this. It gets some media attention. People work to solve it. But I think if we'd seen this stuff like 5 years ago, people would be freaking out. But it's been a little bit of a sort of frog boiling effect. We're like, well, of course AI systems sometimes try and modify the unit task form rather than actually write code to fix the task. We're used to them hallucinating and cheating, but they mostly don't do that. So I see a lot of apologies for these kinds of behaviour or downplaying them when they are really quite concerning if you look at a trend line. But at the same time, I think we have found that some really quite simple safety training methods like RLHF scale up or RL from AI feedback, where you just have another system look at output and judge how good it is. Have worked really quite surprisingly well. We're seeing now with some of these guardrails that developers are putting in place to try and prevent certain misuse. Again, these are pretty basic methods like training filters on synthetically generated data or putting a monitor on the model's own chain-of-thought reasoning. It's not perfect, but we have been able to break these these defenses in state-of-the-art models like Opus Pro and GPT-5. But a lot of the time, the reason we broke them was not because there was something fundamentally flawed about this method, but more because of implementation mistakes in the defenses. And so I actually think a scaled-up, well-implemented version of these defenses would really go quite a long way. It might not make it impossible, but it would make it a lot harder. So I think maybe combining those 2 threads together, what I'd say, I'm most worried about is not actually that kind of the scaling trends are really unfavorable or that there are problems that we just can't solve, but more that we just don't. You know, we kind of cut corners. Someone makes a bad implementation decision in the same way, like, you know, we've had strong cryptography for quite a long time, but, you know, we frequently have cryptographic leaks, not because the algorithms don't work, but because someone made a mistake in implementation. And those are pretty mature fields where people really care a lot about security. People are obviously moving a lot faster, cutting corners more in AI systems. And then I think mostly expect developers to respond to issues that they see. But right now, it really has been kind of like just-in-time safety. It's like, oh, we trained a model and are about to deploy something that we think does cross or almost crosses some dangerous capability threshold. Let's train a filter to protect it. And you could see from the trend line that very soon you were going to have a model that did cross the threshold, but there was very little work to actually develop a defense before it was really imminent and pressing. And that's just not really how you build highly reliable systems, just noticing a problem and then rushing out a patch. You need to design the system from the ground up to be more secure, to be more trustworthy. And so it does feel like we're sort of just almost by design or by choice running with a pretty small safety margin. And we might well luck out. It doesn't seem like any of the technical problems are insurmountable, but it's not a sort of very good place to be. But I think it doesn't require a huge shift from a technical perspective. Now, I think there is a sort of small but not Negligible probability, but we do just see some emerging behaviour that we just didn't see coming, that breaks this kind of trial and error feedback loop. And we do see these kinds of emerging behaviours all the time in AI systems. Recently, we were training a model to be a sandbagger just for some sort of internal research, and we didn't train this behaviour in it, but it started referring to itself in third person as Stein. And being like, Stein would not do this. And nothing in our training data mentioned Stein. It just invented this character and persona. And this is pretty harmless and quirky, but I think it shows just how much we do not understand what is going on in these systems. And if any of these kinds of failure modes we're seeing with instrumental convergence are just a bit discontinuous and we are running with this pretty small safety margin, But I think things could go pretty bad pretty quickly. And so that's maybe the thing I feel like we're most unprepared for. Yeah.

Emergent self-identification as Stein.

Yeah.

Hard to predict some of these things, to put it mildly. So, yeah, I mean, this brings up one of the recent papers that you guys have put out, which is Stack: Adversarial Attacks on LLM Safeguard Pipelines. And we don't have time to go the full deep dive there, but basically what you found is kind of what you just alluded to, that seems like the systems that have been put together are not as well designed, not as thoroughly designed as they could be. Weaknesses along the lines of if you have an input filter and your input filter detects that an input is malicious, the API will just immediately return. And that's a clear signal to whoever's doing the attack that it must have been the input filter that got me. And even those little signals that people can infer from the implicit aspects of the responses from the systems, can be quite helpful to them in terms of figuring out how to break them. So you and team did a bunch of breaking. UKAC was also involved in that paper. I thought it was, but I think I might have a different conclusion from you because my sense, and certainly I just spoke to Zvi for a recent episode and I asked him, what do you think about defence in depth? Is it going to work or what does it get us, if anything? And he's like, At best, it buys us a little time. It's definitely not really going to work. What's really going to happen is we're going to have all of our defences are going to fail at the same time for the same reason, which is that we're going to have off-model, meaning, like, things we just didn't expect at all coming at us from AIs. And they're going to figure these things out in a way where they'll also, in his model, roughly the same time, that'll also be when they get smart enough to not attack until they're pretty confident they're going to win. So I've been like, I don't know, this defence in depth type stuff, can it really work? My naive read of your work there is that not really. It seems like you broke through it pretty easily. You have some tips for what developers can do to make their defence in depth better. Maybe you want to talk a little bit about what some of those tips are. But then the big question is, is it really going to work? And there I think you seem more optimistic than I am coming into this conversation. So Maybe if that's true, tell me why you are more optimistic, especially given how easy it seems like it was for you guys to break what defense in depth does exist.

Yeah, absolutely. I don't want to overstate my optimism. I think a lot of this comes down to your threat model. If we're focusing on something that looks more like a time-limited AI system trying to bypass some control mechanisms, so it's trying to sneak in some backdoor It's a code generation system. It doesn't have any long-term memory beyond the system. It's not able to collude. It only has one shot at winning. That's actually a pretty hard situation for it to be in. I think the analogous thing on a misuse setting would be someone that's a lone actor, but maybe somewhat technically sophisticated, but they're not able to create that many different accounts. Once they get detected, they get banned. Again, this is a pretty hard threat model to win against as an attacker. You've basically got to be smart enough to be able to break all of these defenses in one go. It's not impossible because, as Ed Zvi was alluding to, a lot of these defenses are extremely correlated with one another. Especially the current defense-in-depth stacks are basically you take the same model or you might take a smaller, less capable model just for efficiency reasons, And then you add some layers on top of it. You fine-tune it to be an output filter that you use streaming all the tokens to flag if it looks suspicious. You prompt it and maybe you do a little bit of post-training to be an input filter that does a little bit of reasoning on the input. But these are sort of the same underlying model or it's a handful of models. And we've seen that there's a lot of transfer in adversarial attacks between models trained with completely different architectures, different but related datasets. It really is something where if you come up with a method that works for one, it's probably going to be a similar method that works for others. I think the optimistic case would be if you are able to break this correlation. If you're able to have the defensive components be genuinely pretty independent, they could still be relatively weak. You might still be able to eke out a 1% attack success rate against each component, but you can have 3, 5 components with that kind of success rate that are genuinely independent and the attacker gets no information as to which component triggered, then it is just sort of combinatorially harder to break all of these layers. And you can kind of stack weak layers to get something strong in the same way that a 3-digit PIN number is not at all secure, but a 12-digit PIN number is very secure, but not if you get told which digit was wrong. Because then you can just brute force it in linear time. And then if you're additionally in this threat model where you get limited sets of attempts to attack it, you can't do too much reasoning perhaps if you're an AI model trying to attack your overseer. So you have to do stuff maybe within a forward pass or within your reasoning without a reasoning monitor firing. You can't just write a huge program to generate an adversarial attack. But I think it's It seems very plausible that you can get a system that is secure enough to withstand this kind of attack. Now, if your threat model looks more like you have some kind of just whole ecosystem of AIs that are running pretty much unchecked, maybe in an internal deployment, maybe a company says, okay, we're just going to have thousands of AI systems running around and AI agents in a sandboxed environment, but they can run on any code they want. And then we just put a firewall, like an AI-powered firewall, on the output coming out of these systems, and we review the PR. How can you be confident that that PR does not have some adversarial attack that fools your automated reviewer and also is really persuasive to human beings? I think it's much harder to have confidence in that. It is a meaningfully harder thing for a system to pull off. So maybe one area I would be sort of more optimistic than Svi is I just don't see where these sort of AI systems get really smart and collude with each other to not try and pull off an attack before they're smart enough to know that they can definitely succeed. Especially if you just fine-tune the systems to be a little bit adversarial and see what they can do, you should at least be able to get a warning sign that these systems are capable of fooling you and breaking your security, even if you're unsure whether your current system is actually just honest and aligned or is scheming. That could be hard to distinguish, but at least eliciting these things don't seem that hard. But it's, yeah, in most kinds of unconstrained deployment scenarios, I think you could definitely run into some problems. But I think if you spoke to the AI control people, and I don't work in AI control, so I'll do a bad job representing it, but say, well, why on earth would you just have this huge internal deployment without some much more fine-grained checks and balances. And if these AI systems need to make a PR that fools you, including with a bunch of other AIs reviewing the PR, and you can access all of their chain of thought reasoning, and you can check what happens if you rerun the whole system but with a slightly different initialization, then they've got to be fooling you in this really consistent way without any explicit communication channel between them. So it starts looking a bit more like a a prisoner's dilemma where you can cross-interrogate each prisoner and they need to have a consistent story, but they can't communicate with each other. And this just seems like a much taller order for AI systems.

So well-done defence in depth, you think, can work for a point, to a point at least. And I think you had an interesting observation also, and on this I think Zvi would very much agree, that It is important to actually fix the problems, which kind of brings to— you want to get the indication of what the AIs are going to do to try to get you, and then you want to address those, and hopefully in a really deep way such that you can really solve them. Now, an obvious problem is we don't have great strategies for fixing these things. But again, I feel kind of confused here because, as you noted, Claude is pretty nice.

Yeah.

I guess one point question is, do you agree with the folks who have been recently arguing that Claude 3 Opus is still the most aligned model ever? And do you have a theory of why that would be? If that is true, why has why have we given up ground in the alignment department since Claude 3 Opus? And then I guess more broadly, what are our prospects? Okay, great, Claude 5 and GPT-6 are scheming, and it looks like they actually might be pretty good at it. Great news, they didn't take over this time. What are you going to do about it? Do we have any alignment strategies that you would think would be developed enough in time to solve these things so that we don't? Because the alternative does seem at this point like we'd probably just say, ah, it wasn't that bad, and, well, just adjust the system prompt and add another layer to the defence in depth, and that'll probably be good enough, right?

Yeah.

So do we have anything Better than that, I don't know what it would be right now.

Yeah, you're right. I don't have a strong opinion on, you know, was Opus 3 the pinnacle of alignment? I think part of the challenge here is that we don't necessarily even have a good operationalization of that question. So I think my intuition, which I wouldn't rely on too much in any sort of an absolute sense, Opus 4 is probably more aligned. However, Opus 4 just has much more capability. to be misused, to scheme, to do all these bad behaviors. So you kind of need to hold it to a higher bar. And it does feel like there was more capabilities progress going from Opus 3 to Opus 4 than there was safety progress. Although the guardrails did get a lot better. There were very little guardrails in Opus 3. But this is quite hard to quantify. I think a lot of people have this intuition of, okay, safety and capabilities are increasing in parallel and you want them to sort of stay at least a constant gap. Ideally, you converge and you really don't want them to diverge. We just don't really know what the scale we should be measuring for capabilities or safety on the y-axis is. And in fact, depending on what scale you choose, you could make lines that would otherwise converge diverge. So it's a bit subjective. But that's one of the things that I find Hard to reason about these things. But yeah, to answer this question more broadly of, okay, suppose we do see these warning signs, what do we actually do? I agree with you that currently it doesn't seem like there is enough will to take really significant capabilities or performance hits in order to make a system more reliable and safe. Or Maybe some developers are going to do that because there's certainly consumers, big enterprise companies, safety-critical industries that would prefer to have a model that's more reliable. And we're seeing both different leading developers positioning themselves differently on this trade-off, but also specialized models like RaceOne, Signet emerging that are really optimizing quite hard for security. And these do have use cases, but there's going to be someone who's just like, no, I want to turn the knob all the way to capabilities and low latency. And I'm going to, you know, I'm only going to do safety if it's really quite cheap or is actually making the model more useful for a wide variety of users. And I think that's an area where we just don't have a good answer, I would say. I mean, I can point to some things I'd like to see people do more of, even just simply gathering larger, higher-quality human training datasets, including on long-horizon tasks, including people scrutinizing these models. Like, why do you get more sycophancy sometimes if you train them for longer? Because probably your human data actually just rewarded that. But if you did a closer look at it, you'd be able to fix that. And I don't want to say these problems are trivial, but they're not. It's very hard to get high-quality data at scale. But this isn't some kind of unknown problem requiring great research innovation. You just got to spend a lot of money on it, and you've got to build out a team, and you've got to spend some time iterating on that and seeing how your model evolves. And obviously, there's some scalable oversight techniques that are beginning to be able to really reduce the cost of that human labor. So there's both been some good work on AI safety via debate coming out of Google DeepMind and also some work from the UK's AI Security Institute. And then, you know, one recent piece of work that we did, which has a I think both have some good news and bad news, is using scalable oversight to make models less deceptive. And this is, in some ways, deception is kind of the core safety issue to focus on because if your model is fast, you can just ask it a lot of questions like, are you trying to take over? Why did you write this PR? And it might be malicious, but at least you're going to be able to sort of tell pretty upfront that it is. And what we found was that if you do train AI systems against lie detectors, so these are other ML models trained to detect if that system is being deceptive, then you can, with the right setup, actually get models to generalize to be much more honest. So this deception rate really went way, way down. And the other good news is that it doesn't seem Although we need to do more investigation to confirm this, that making the models larger necessarily makes this any harder to detect. Whereas, you know, kind of making the lie detectors larger can, you know, actually make them much more accurate. So it seems preliminarily like the sort of the scaling trend is in our favor. Although we need to get more data to confirm that, but at least it seems unlikely to be kind of, you know, against us. But the caveat here, and I think this is maybe the crux that people like Zvi and I disagree on, is if you train a system to not exhibit some kind of bad behaviour, does it generalise to stop doing that and other instances of it? Or does it be like, oh shit, you caught me, I need to be better at scheming? And we did find that under certain training regimes, it matters a lot whether it's on-policy or off-policy RL. And we have some theoretical insights into why that would be the case. It'll matter is things like KL regularization, how much you're allowed to explore in policy space, then you can get instances where the model just learns to fool your lie detector. Now you've done this safety training, but you've actually made this much, much worse. I think there is a very real reason to be concerned about that, that developers are going to find a problem. They're going to not very carefully patch it. They're just going to rush out a fix, like changing the system prompts or like doing a bit of fine-tuning. And the problem is going to appear to disappear, but you've actually made it worse. But based on what we're currently seeing, I would be optimistic that kind of a rigorous engineering approach to this where you do train away problems you see, but you're careful with your setup, you're careful to have high-quality data, you use some extra evaluation to check that you're not incentivizing the model to just do the same bad thing, but more surreptitiously, you could get quite far. I don't know if you get past human-level intelligence safely with this, because that's the point where you stop being able to reliably provide human feedback on a lot of this. But I think you can at least get to human level and probably a little bit beyond with scalable oversight. Yeah.

So let me just try to echo back a couple of key points there, and then I have maybe a couple of questions.

Sure.

Obfuscated Reward Hacking, I believe, was the name of the paper from OpenAI where they showed this pressure to the chain of thought would initially create higher performance and reduce the reward hacking bad behaviour. But too much and the reward hacking comes back, but the chain of thought signal doesn't. And so you're in some ways the worst scenario where now you You've lost your observability, but you're still getting the bad behaviour. And you're basically saying that, well, that's the result of one training formula. But it does seem that if you do off-policy, which is basically to say, use human examples, right? Or at least pre sort of human evaluated preference data, as opposed to the GRPO style, where the model just gets to kind of do a tonne of trials and explore the space more extensively on its own, and you anchor to the human prior that was presumably learned in some earlier post-training step, then you think you can avoid that outcome?

Yeah, I think that you can You can avoid it, or at least we can develop empirically grounded rules of thumb for safer training formulas and over time test it to see how reliable this really is. But I think the other piece I'd advocate for is having some good held-out evaluation criteria. So I think if your training scheme looks like you optimize a particular measure of success, and you look to see if that metric just goes up and you've got no alternative measure that's going to catch bad behavior, then that's a pretty scary situation to be in. Even putting aside these safety concerns, it's just very, very easy to overfit on that metric. Whereas if what you see is I've got a pretty simple lie detector, it's like a linear probe on my activations, or it's a bag-of-words model, or something that is It's definitely giving you a meaningful signal, but it's not like falling this means that there's no way we can detect what the model is doing because we're using a pretty simple training objective. And then you do a much more rigorous and expensive evaluation and you don't catch any new bad behavior in the AI system. And maybe you can also bust out some interpretability tools and look at the diff between the checkpoints and actually understand what the model is doing. But I think there's ways in which you can get a reasonably high degree of confidence in how the system is operating. It's not sort of perfectly guaranteed, but at least if you're starting from a model that is not sort of super deceptive, so sort of like super intelligent at deception, then you can get a high degree of confidence that your training is not pushing it into that regime. I think none of these methods have a very good answer for what happens if your model just from pre-training is incredibly deceptive and trying to to fool you. I think that that's pretty tricky to solve. But there's also strong conceptual reasons to think that pre-training is not going to produce a system that coherent and objective, even if it might have some latent capability. But I think this concern is the core one. And so this is actually one of the things that we're trying to test right now, is intentionally train AI systems where none of our methods are able to detect them being deceptive. But they were actually very deceptive. And there's one question of just like, can we do this? And I think the answer is yes, from preliminary results. But then the next question is, okay, can we do this in any way that seems at all realistic and like how you would actually train AI systems? And if just every way we can come up with these systems is a very alien kind of training objective, then I think that we can be reasonably confident that at least at current scale, this isn't going to emerge. But it's a sort of check that you'd want to periodically repeat to check that this isn't some kind of scale-dependent property.

What about interpretability? This has been a big debate, obviously, over the last couple of years, where there's been very grand hopes for it. I just listened to Neil Nandan on 80,000 Hours saying that he's kind of scaled back his hopes for interpretability, but still thinks there's a lot of value, just not the maximalist sense of, like, we're really going to fully understand what's going on, and then we'll have kind of guarantee-type assurances that things will behave.

Yeah.

There was another really interesting paper that you guys put out around digging into the planning mechanisms learned by a model that was playing a puzzle game. We'll abstract away from the details there for now. But where do you sit right now on how much we can expect to get from interpretability?

Yeah, well, I think it's definitely a core area. Like, you'd really be kind of tying your hands behind your back if you say, I'm only going to look at the black box behaviour of models, even though it's, you know, really trivial to open them up and look at the inside. We're just not going to do that. But I think mechanistic interpretability specifically had this ambition of fully reverse engineering how a system works and being able to give a human-readable or at least human-interrogatable understanding of a system's internals. You could ask questions like, is there anywhere in this system that is doing some kind of long planning computation? I do think that we've seen enough research here to be really quite sceptical that we're going to get that simple, a sort of reverse-engineered artifact. So people have been able to fully reverse-engineer some non-trivial neural networks. So we've seen papers reverse-engineering particular circuits doing certain NLP tasks. We've seen papers reverse-engineer certain modular arithmetic tasks. Our paper, you know, pretty much fully reverse-engineering this planning algorithm. So I think we basically fully understand how it represents the plans and we understand most of the computation that's going into constructing the plans, except for some heuristics that don't have any kind of long horizon effects. But when you look at these answers, usually what you take away is like, wow, that was kind of a mess. It's definitely an evolved, organically grown system, but not a sort of planned engineering artefact. And although it's easier to reason about these systems than if you didn't have that mechanistic understanding, it's still pretty hard to to get any kind of guarantee because they are this sort of messy, complex system. So, you know, perhaps we have to dial back our ambition there a bit to say, okay, we might be able to reverse engineer systems to a point where we can ask some high-level questions like, you know, does it have any circuit that's doing some kind of human theory of mind? Okay, we might want to understand when it's using that circuit because that's very relevant to deception. If it's suddenly using theory of mind a bunch when it's writing this particular line of code, maybe we should understand what that line of code is doing. If it's about a user interface, okay, it needs to model humans. If it's about a cryptographic algorithm, that's weird and concerning. There might be some coarse-grained things there. I think we have seen some real valuable applications of what I'd call more coarse-grained interpretability, so things like simple linear probes, you're training a linear regression model on activations to a system. But yeah, I do think that there's been this disconnect between this very foundational reverse engineering style work and then batch applications are usually using much simpler methods. And I'd love to see some more things meeting in the middle where we say, okay, there's a particular domain where we think interpretability is going to be really useful. Maybe it's detecting models for the sandbagging, so intentionally holding back certain capabilities. Maybe it's detecting model backdoors. So, you know, maybe it's detecting sort of, you know, spurious confounders that a system might have learned during training, and then we can, you know, ablate the ones that we don't want, and then it generalizes better. But kind of like, pick your application that you think interpretability is going to be really killer for, and then develop foundational methods that make progress on that. But I think this sort of more exploratory style of research, it would have worked if there was just like a clean explanation for everything. But at that point, I don't have much hope that we're going to get that.

Yeah. Do you think that there is some innovation, though, that we could come up with that would make things cleaner? I always remember the Seeing Is Believing paper from Max Tegmark's group, where basically a sparsity— these are very toy things, but a sparsity term kind of caused most of the weights to disappear, and you'd get this really kind of crystalline Yeah. Maybe it's just too messy to handle that?

No, I think it's possible. I have more hopes for training systems to be interpretable than interpreting how systems are currently designed. I think for sure there are going to be some simplifications we can make to current systems. So stuff like sparse autoencoders have been quite effective. But the problem is with it, their predictive accuracy is just too low. They're throwing away enough information that you lose this nice property of, okay, you've actually got this high-fidelity reverse-engineered interpretation of a system. But could there be some innovation along those lines that's perhaps a bit messier, but you can search over in an automated way and reduces the complexity enough to work in that space? For sure. I just don't think it's going to be this sort of clean picture of, okay, you peel away some layer of superposition and now everything is just wonderful, because we have understood systems at a very nuts and bolts level and they're just messy. But it doesn't mean that you could not train a system to be cleaner. Or at least to sort of isolate the complexities. But I think it's actually one of the sort of more methodological innovations we came up with in that paper investigating learned planning was saying, okay, we're going to look through all the channels in this neural network and just categorize them into ones that we do need to understand and ones that we don't. Because there were a bunch of channels that were sort of short-term heuristics that we could show only had an impact on the very next move. It didn't matter into what the system was going to take 3 steps into the future. We don't really understand this heuristic channel, but if what we're trying to do is understand how the system is planning, it doesn't matter. It has no impact on long-term planning. And I think there's a lot of cases like that where there might be a huge amount of complexity in these systems, but most of the complexity you can just sort of show is not relevant to a particular kind of question that you're answering. And if you're able to train the system with an objective to sort of explicitly disentangle let's say, that sensitive channel of information and the less important channel, then it could get a lot easier. But I do think that there's likely to be some kind of performance trade-off for training systems in those ways. So really, it's a question of whether this becomes a key thing that people are actually willing to optimise for.

Yeah. Okay, interesting. How about lightning round? You guys are getting involved in more different areas of the broader AI safety landscape. There are some field building projects you're getting involved with, putting on some events, bringing people together, starting to get more interested in or involved in policy advocacy, starting to get involved in international relations. Maybe give us just kind of a quick tap dance through all the activities that you are investing in, that we haven't covered and give us a little bit of a sense of why you're motivated to do all of these things. And then at the end, I know you're hiring. We can give you a chance to pitch some open roles.

Yeah, sure. Absolutely. Well, I think at a high level, the reason we're doing all these different activities is that we see there being a potentially quite long pipeline from research innovation to that idea actually being adopted and deployed. And sometimes that means getting companies to adopt it and deploy it at scale, and it's sort of like so good that everyone wants to use it. Other times that means actually getting some buy-in from governments, perhaps to mandate or nudge companies to do that if it's something that is beneficial to a field as a whole, but might not be in anyone's collective interest. And so if you go from that idea to deployment, you've got sort of initial research innovation, which we're well placed to do in-house. But then very often you need to have a research field around it. Like, it's not enough just to have a small research team. So we've done events like the Control Conference to kind of catalyze and grow new research fields. Now, AI control was, you know, something that Redwood Research pioneered, not us. We're happy to help out with other fields. But that's sort of like an example of where our events and field building can really come to the forefront. And then I think another missing piece in the pipeline is very often making really clear, rigorous proof of concepts. So not just doing the exploratory research, being the definitive research, which is sort of where I'd highlight our current deception work is trending towards where we had this initial idea for scalable oversight that would solve deception. And it seems to work at a small scale, but now we want to validate it on frontier open weight models, get several orders of magnitude of scale and check that it works across all of them. We want to stress test it by trying to train systems that are really deceptive and break current techniques and see how realistic that is. And then if it works, actually say, look, here is DeepSeek R1 and LLaMA 4 or 5B and all these models trained with our technique, and they're just better than the original model. They're less deceptive, they're less sycophantic. You can trust them more. There's no reason not to just use this model. Then get some uptake from AI developers. And this is where our events or more policy arm comes in. And then eventually go to governments and say, look, this is just an industry best practice. This is something that you should make at least part of your guidelines. If you're considering regulation, you should really consider something that isn't saying you have to use this technique, but use a technique that's at least as good as this because there's just no reason not to. And I think what makes us unique is that we are willing to do the hard thing and operate across all of those different layers of attack stack from groundbreaking exploratory research to the kind of nitty-gritty engineering scaling work through to more advocacy and sales. And I think this is sort of not surprising if you're coming from a startup mindset where you do have to often do all of those different things. You need to have a growth team, you need to have an engineering team. But for whatever reason, most of these sort of nonprofit organizations have usually said, okay, we're going to pick this niche to be in. And there's many benefits to that. You can be much more focused as an organization. But the problem is that very often they kind of, they develop something and pass the baton to someone else, but there's no one to pick it up. The baton just gets dropped. And so I think that's one of the big issues with AI safety and what we're trying to resolve. And then to speak to a little bit more about, yeah, some of our other activities. So events, we run 3 alignment workshops a year typically, which is sort of like our flagship event bringing together technical researchers. AI company decision makers, people working at government AI security institutes. And it's just really sort of like everything under the sun on AI safety, but it's sort of more of a coordination, information sharing. But then we're also increasingly doing more policy-focused events. So we ran Technical Innovations for AI Policy in DC a few months ago, really bringing together governance researchers, actual policymakers, lawmakers, with technical research to say, okay, what could we do that really expands the range of policy windows open? Because I think right now regulators are often faced with a pretty shitty choice between genuinely maybe holding back innovation and valuable deployments in some areas, or just allowing completely laissez-faire, fewer regulations on billion-dollar training runs than opening a sandwich shop in San Francisco. And I think that's just sort of like a false dichotomy. You can have things that are pro-innovation and pro-safety, but you do need the right actual technical innovation to drive that forward. So we're trying to catalyze that. And then, yeah, maybe, you know, aware of time. So maybe quickly to touch on our hiring round. I mean, we're hiring a lot. We're planning on doubling in size in the next 12 to 18 months. We've got funding secured for the next few years, including that runway. to expand. We're still looking to fundraise in a few areas that our major donors are, you know, a bit more outside of their priority areas, but we've got that room to expand. And so we're hiring across our technical team for, you know, both IC ML engineering positions, but also people to lead new research directions. We're hiring in operations. Probably the most important hiring round we have this year is Chief Operations Officer, which is going to elevate my co-founder into a president role and be able to start driving forward some of the non-technical work that we do. But we're also hiring for more junior positions, people ops generalists, project manager on events. So really, I'd say, like, you know, if you're excited to work in this space and you like the kind of work we're doing, then do check out our career page. And even if you don't see a job that's a good fit for you right now, we have just a general expression of interest. So we'd encourage just sort of like talented people who are excited by us to fill out and then we'll get in touch when a role does open up.

Love it. It struck me there that as you describe all that, you're kind of going vertically integrated, right? You're going all the way from research to, in theory, handholding people into actually putting these things into production systems. This is maybe more of my hobbyhorse question than something you really want to do, but I've been really interested in this idea of private governance. And I wonder if society might draft you and the FarAI team into becoming one of these private sector quasi-regulatory bodies. It would seem like you have the range of capabilities needed to do it, but would you be excited to do it if that were actually the law and people were looking for orgs to fill that niche?

Yeah, I think that you're totally right. We have this skill set and sort of org structure to do that. I'd say that, you know, this isn't part of our mainline plan, you know, perhaps because it does still feel pretty nascent, this idea of private regulatory organizations. But it's a very important thing to do. So if this was something where we didn't feel like there were, you know, other kinds of companies stepping up to fill this gap, and we were well placed to do it, then it's something we'd be very interested in. I do think it certainly has some trade-offs. in that right now we're fortunate to be on good terms with both the leading AI companies and governments. And that's basically because we don't actually have any hard power. We can make recommendations and brief people, but we're not overseeing anyone. And we're going to lose some of that independent ability to convene people, to make recommendations if we're actually exercising an oversight role. But I think it could be worth it. We're also not wedded to everything being under one organization, so we could spin out either that kind of private sector regulatory part, or we could spin out some of our activities that require more independence, like our events, into a separate organization, and then just kind of avoid that conflict of interest. So yeah, we're open to it, but also love to have more competitors in this space so that we don't have to do so many different things.

Yeah, you are spinning a lot of plates right now. So thank you for the generosity with your time and perspectives today. People should definitely check out the open roles, COO and otherwise. Adam Gleave, co-founder of FarAI, thank you for being part of the Cognitive Revolution.

Well, thanks for having me back, Nathan. Great to see you again.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries. Either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of a16z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, Lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI meeting notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI meeting notes free for 30 days.