Situational Awareness in Government, with UK AISI Chief Scientist Geoffrey Irving

Hello and welcome back to The Cognitive Revolution.

The Cognitive Revolution is brought to you in part by Granola. If you are a regular listener, you've heard me describe the Blind Spot Finder recipe that I'm using to look back at recent calls and help me identify angles and issues I might be neglecting. But it's also worth talking about how Granola can help raise your team's level of execution by supporting follow-through on a day-to-day basis. This past week, for example, I had several working sessions with teammates. And I committed to a number of things. In the past, to be honest, there's a good chance I'd have forgotten at least a couple of the things I said I'd do. But with Granola, I can easily run a to-do finder recipe and get a comprehensive list of everything I owe my teammates. This is the sort of bread-and-butter use case that has driven Granola's growth and inspired investment from execution-obsessed CEOs, including past guests Guillermo Rauch of Vercel and Amjad Massad of See the link in our show notes to try my Blindspot Finder recipe and explore all of the ways that Granola can make your raw meeting notes awesome. Now, today my guest is Geoffrey Irving, a pioneering machine learning researcher who's co-authored seminal papers with a who's who of giants in the field, and who is now chief scientist at the UK AI Security Institute, which is in all likelihood the most situationally aware government entity in the world today. With roughly 100 technical experts on staff and a mandate that includes threat modelling, pre-release frontier model evaluation for dangerous capabilities spanning biosecurity, cybersecurity, and loss of control, advising the UK government on strategies to reduce catastrophic risk, funding independent frontier research, and engaging in global diplomacy, Geoffrey has one of the most broad and commanding views of the AI landscape that you'll find anywhere. And while he is optimistic about our ability, in the fullness of time, to solve the major open problems in AI safety, for today, without a hint of hype, he paints a genuinely alarming picture. Our theoretical understanding of machine learning is nascent. Nobody, he argues, should be particularly confident in their mental models of how AI will go. Models already outperform a majority of experts on a great many security-related tasks, And there is no good reason to expect that their progress will stall. Reinforcement learning is working well beyond strictly verifiable tasks, and jaggedness matters much less when even the model's weak spots are as good or better than the best humans. The many increasingly sophisticated bad behaviours we've seen over the last 18 months are broadly all different versions of reward hacking, a problem for which we lack theoretical or practical solutions. As such, we likely won't get that many nines of reliability from current safety techniques, and there's some reason to expect that they could all fail at the same time for the same reasons. It is getting harder to jailbreak models, but the AC red team has never failed to do so. And meanwhile, eval awareness is an open and growing problem. Voluntary cooperation between frontier model developers and the AC is working pretty well, but not everyone is participating. The AC, for its part, is seeking to fund theoretical research in areas like information theory, complexity theory, and game theory, which might produce stronger guarantees. But these fields, like most of the rest of the world, are just beginning to take AI seriously at all. Jeffrey is an intellectual powerhouse, but I came away from this conversation just as impressed with the UK AC as a whole. This is an organization staffed with top-notch talent, that has its finger on the pulse of industry development and is speaking very accurately and plainly about AI's trajectory and how many major questions remain unanswered, even as frontier model company CEOs tell us that they are less than 3 years away from creating expert-level AI machine learning researchers. With that, I hope you are focused and motivated by this conversation about the AI state of play. With Geoffrey Irving, Chief Scientist at the UK AI Security Institute.

Geoffrey Irving, Chief Scientist at the UK AI Security Institute. Welcome to The Cognitive Revolution.

Thank you. I'm excited to be here.

I'm excited for the conversation. We've exchanged messages for a while and have been building up to this, and I'm excited that the moment is finally here. And you have really a storied publication history. that goes back to working on the original TensorFlow papers with some guy named Geoff Dean, being a co-author on the original RLHF paper, working on concepts years ago.

For language.

Okay, a caveat, but still right there alongside Paul Christiano. Some early AI safety papers with no less than Dario on concepts of using debate to try to bootstrap into Stable Equilibria and Stable AI Safety Regimes, and even published a call for social scientists to enter the field of AI safety with one Amanda Askell. So I would be very interested to hear how it was that you came to have such a good nose for where AI was going so early on. All these things are well before ChatGPT.

Yeah, so I used to think I was, like, new to ML, but I said that for too long, and now I'm not new to ML. I think I got out of undergrad with a bias against statistics. I'd only seen frequentist statistics. I thought they were kind of weird looking. I'd never seen Bayesian statistics. So I just didn't like any of the stuff. And what I liked instead was things that have kind of hard theory and you know the equations, you know what they— you have some ground truth. And that was computational physics and mathematics. And on the computer science side, programming languages and theorem proving and such. And I did mostly computational physics and geometry for grad school and then kind of years after that until around 2013. And then I had realised 2 things. One, that machine learning was getting quite good. So like the neural nets were starting to work and they were getting better and better. And that was going to continue probably. And then 2, even in the areas where I thought it was about knowing precise theory, so physics or theorem proving, You needed common sense and you weren't going to get away with just the theory. That was not going to be enough. So if you're doing mathematics or programming languages, you needed some ingredient of heuristic picking between the various options and you wouldn't be able to do a good job designing human usable friendly systems without basically machine learning. And so then I was like, okay, I should switch into machine learning. I was doing something else back then. The first thing I tried to do was autocorrect for code in 2014, which is too early to do autocorrect for code. It did not work then. Also, we didn't know how to do machine learning. We were like, this is myself and Martin Wick, and we knew computer science, physics, geometry, but not really ML. So we tried to do a startup for a year, didn't work. And then we said, well, how do we learn? Well, we learned by joining Google Brain. at the time. And then I've done ML jobs since then. So that was— I joined Google Brain in 2015, and the goal was for me basically machine learning for theorem proving. I was aware of safety at the time, but I didn't see an attack on the problem that I thought was good. And so I thought I'll work on some other kind of different problem, which is just sort of hardening the world using verification, which again was going to be using machine learning to do theorem proving in practice. So that was sort of 2015, 2016. And then I guess there's kind of two— I mean, that was like, oh, I had some useful thoughts early on, but there's two other kind of parts of the story. So one is that there's kind of two bits of kind of inherited wisdom, which is why it looks like I predicted things early. One is just I joined OpenAI in 2015, and Dario and Paul were there, and they had a bunch of past thoughts about safety and how the machine learning field was going to develop. And so I was just sort of riding along from there. But then kind of more broadly, there's just a bunch of intuition coming out of kind of theoretical computer science complexity theory about how computations work, how we check computations that someone kind of with more resources than you can run. And a lot of what I've done since then, including debate, for example, is just sort of applying that intuition, assuming that it will kind of hold. in some modified form into the machine learning world, even if it comes from some kind of area of more precision. So that's sort of, again, you can just assume things are going to look like theory in some way with a bunch of porting required, and you can predict a bunch, but not exactly how long it will take or kind of when things will happen.

We can unpack both of those, I think, in more depth as we go. Certainly the quest for theory and bounds that you can really trust in is a big theme of your work and the work that you're trying to encourage at the AC these days. And I'm also really interested to get your take on the relationship between math and the fuzzy, messy real world. But, well, let's circle back to it.

Good.

Fast forward to today. I'd be interested to know how AGI-pilled, quote unquote, you are today. And that sort of just informs, like, what is the general— because so many AI discussions broadly, and especially around topics of safety and security, go kind of immediately haywire when people have just such different intuitions around what it is we're likely to be dealing with. So I like to try to establish, like, what is it that you think we are likely to be dealing with? I don't expect that would be the official position of the UK AI Security Institute, But you're obviously a leader there. So it seems to me, from reading the reports, it seems like you are not expecting any sort of wall or plateau in the immediate term.

So I think that I can actually, fortunately, kind of— the things I can say are mostly also the things that we think kind of officially, which is that we're going to have a lot of model uncertainty about how things could go. And that could either mean that there are obstacles that cause there to be kind of stalls for a while, Or there could be no such obstacles and things could go quite fast. And I think mostly anyone who kind of confidently claims in one direction or the other with too much confidence, oh, 99% certainty of there are or are not big obstacles, they're probably wrong and they should be more uncertain. And I think that means for us, one, we want to map out what those different cruxes are, like what could the obstacles be? What are kind of signs of development? And then 2, we should assume or place significant probability on the current methods will scale and where they don't scale, more mundane stuff will replace them and continue kind of different, like further sigmoids. And so we do, I think, have significant credence on things will go fast. I won't say exactly how fast because I don't talk about exact timelines, but I think that is kind of pretty important. And then I think we published a paper from the strategic insights team at AC on different potential obstacles to AGI and what is our progress over the last while on addressing those. Again, they could all not actually be fundamental obstacles, as in they could be kind of solved not by pure scale, but by kind of maybe some scale and some just steady algorithmic progress to models, to scaffolding, to data, that kind of thing. Or you need new algorithms and then they take longer. But generally, I think that both my view and the view of AC broadly is like, have model uncertainty over all of those terms. And then that should mean that you're not confidently saying it will either go very fast or will not go very fast.

Yeah, it's good to have some of that in an L-world leading government. This won't be the big focus by any means of today's conversation, but What does your personal AI productivity stack or pattern of use look like today?

I think I'm kind of vanilla. I use kind of all of the models for different things. I think mostly I use one of them as a default. Usually it's Claude, but it varies. And then I kind of go to other ones if they have specialties that are particularly good. I think for a good while, GPT was better at math. And Google is better at kind of certain other things. This shifts over time, so I just kind of use general models. And then for, I don't do a lot of coding in my job, but I do for fun kind of like formal verification work. For a while I was just using Cursor there because the agents like weren't that good enough at doing full-on agentic stuff. That is not true anymore as of a few weeks ago. Now they are good enough. So that means that that shifts to stuff like Codex and Claude Code, basically. But that's mostly not my job. Mostly my job is meetings and talking to people and advising on research.

I'm glad you're still making time for a little formal methods on the side. Okay, let's talk about the overall landscape in terms of the threat model that we have for, from AI. And I'd be interested in kind of also your characterisation of what you understand to be the de facto plan to address it. Again, people have so many different starting points here. What do you think is kind of the set of big things that we should be worried about?

So I think that we kind of break down risks. The main 2 focuses of AC are catastrophic risks and large-scale societal impacts. And the main kind of 3 catastrophic risks we focus on are bio, large-scale cyberattacks, and loss of control. The team is called the Chem-Bio team for chemical and biological weapons. More risk comes from bio in practice, I think. And then on societal impacts, that's sort of human influence. So that's persuasion and kind of emotional reliance and then kind of various kinds of societal resilience. So like attacks on CNI, critical national infrastructure and that kind of thing. And just sort of like agent behaviour in the world at various agent risks. And I spend like most of my time on the catastrophic risk side. And Chris Summerfield, who's the research director here, spends most of his time on the satellite X side, although we also do a mixture of both. So those are kind of the main risks we work on. I think we also are kind of thinking about gradual disempowerment, kind of more kind of structural risks somewhat. But I don't think we quite know, and no one really knows how to mitigate these at a large scale. There's some work we're doing that's like either investigating that or kind of thinking about mitigations, but that's more nascent. So that's the bulk. I think I forgot the other half of your question, though.

Yeah. What do you—

so in the absence of people changing the discourse or new discoveries, new big ideas, how would you describe what we are on track to do today? Often I would say, at least the way I've characterised it, is It's sort of defence in depth with hopefully we can patch together enough nines through enough layers, all of which are kind of leaky, but hopefully they're not too correlated in the ways they're leaky. And this has always kind of worked in the past, so hopefully it'll work this time.

I think there are. Yeah. So you're not going to get to a lot of nines with the current technology. I think broadly we can kind of break this down by domain. So for misuse risks, So, like, biologic weapons and cyberattacks. This is mostly kind of safeguards, kind of differential access. So give models only to certain people that are vetted in some way. And then non-model defences, so like pandemic preparedness and kind of improved security, that kind of thing. And I think the safeguards are not that strong. Sort of open source models are also kind of pretty good, although there's a gap. And so I think that The stock plan is mostly sort of like, in some sense, you use the model-side mitigations to give yourself a window and then you try to harden the world against these risks. And I think whether that will go through or not is kind of, we should be uncertain about that. But there's not kind of strong solutions to those things if the models keep growing as we see them growing in strength. On the loss of control side, I think that is kind of a combination of mundane empirical approaches, mundane, whatever, pragmatic empirical safety measures and a lot of monitoring. And then using that monitoring, this is, again, this is the AI developer plan, typically using those mitigations to get you through into an automated kind of automated safety research regime where hopefully you find better solutions than than those first methods. I think this has various flaws. And so maybe it'll go through, but we should definitely not— you're not going to get to a couple of nines with that kind of plan. And I think you wouldn't kind of know with the current methods that it was going to work until after it went through with confidence. You'd have a lot of uncertainty. And so I think that's kind of the story. And I think most of the approaches we have now look like that. They're empirical. Maybe they'll go through, Like on the alignment side or the pragmatic approaches to get through this kind of automated safety thing, safety phase, it's kind of AI control measures and monitoring and kind of honesty training and kind of white box detectors and all of this. All of these are kind of pragmatic and they do, I think, all have correlated potential failures where they could in fact all fail for the same essential reason. And You would need stronger advances to be confident that will go through. And then even if that goes through, I think that we do need, because of the misuse risks, kind of a lot of mitigations on the non-model side as well.

Hey, we'll continue our interview in a moment after a word from our sponsors. So when you talk about we can't expect to get to many nines, would it be a fair arithmetical move on my part to take 1 minus 1/9 and say your sort of implied P-doom, if you will, is at least like a 10%? Or would you segment that down further and say things could go wrong, but I wouldn't put it in the doom category?

I'm just inclined to answer that question, I think, to not give numbers to things as a civil servant.

I answer usually like 10 to 90%, which is also obviously Kind of a way of not answering. But qualitatively, it sounds like you are taking very seriously the possibility that this is going to go not just kind of crazy, but meaningfully catastrophic.

Yeah, I think loss of control, we view it as a potential catastrophic risk. I think it's like, what's the one thing we're doing? There are a bunch of uncertainties about this model. So one, there's 2 different teams at AC that do various kinds of empirical alignment testing, one using kind of adversarial methods, one doing kind of step-back statistical analysis of different factors that cause models to do sketchy things. And I think a lot of part of that research is trying to pin down this threat model and what drives strange behaviour, kind of when agents when models are behaving in ways that you'd expect would correlate to these kind of extreme scenarios. Because I think people kind of— we talk to a lot of partners kind of within the government or kind of other governments or other parts of society. And people have pushed back to this kind of risk model. And so we want to provide kind of some— as much evidence as we can pin that down. But again, it's an area where one should have a bunch of model uncertainty. And then think through the details despite that.

Can you unpack the intuition around why everything would fail at the same time for the same reason? That's something that I've heard from a number of people. Zvi, for example, always says that. And it seems that thought seems to come very natural and feel very intuitive to some people. And then to others, it's like, I don't know, you know, there's jaggedness all over the place. Like, why would we expect that if I can't get a given model to do this and that today, why would I expect that suddenly everything's going to crystallise and there's going to be this uniformity of the model's ability to query through all sets at some—

So I think that maybe there's an important thing there where the models are jagged today, but if you ask them to do tasks that the models could jaggedly do 5 years ago, they're not jagged. And so the question is, for the capabilities you would need to realise a variety of risks, if you kind of push forward a few years, or however many years it takes to get very strong capabilities, you should expect those models to still be jagged, but up above a frontier where potentially all the things you're seeing are kind of— they don't look jagged. So if I look at the best Go player in the world, the best chess player in the world, They have a bunch of jaggedness. If you sit them down against the next best Go player in the world, they'll kind of win or lose for idiosyncratic factors. They'll have different tastes, they'll have different parts of the board or the game they're better at. And if you sit them down in front of me, they'll just wipe the floor with me every single time, even if I have 9 stones and I'm like a halfway decently strong amateur Go player. So I think You have to run the calculation, think about the model as it would be in the future. And I think part of this is there's kind of a non— like, it's unhelpful when people talk about AGI or superintelligence or whatever as being this thing that can do everything, because it does imply that it's sort of so qualitatively different than the models of today. Whereas I think the non-magical version is just, it's better at a lot of things and indeed kind of superhuman performance at a variety of kind of risk-relevant domains. And we know that models can be superhuman at certain domains. Like, they're better than me at knowledge. They're better than me at lots of math. This is kind of true for everyone. Like, certain domains, they're probably better than— every person will have domains where the LLMs are better than them currently. And this is kind of rising over time. And then they're very fast, so they can think quickly. Sometimes they can do tasks very well. They do 10 times faster than humans can do them just because of computational speed. And then they're not very interpretable. So the methods we have for interrogating their behaviour are not that reliable currently. And so I think that sort of non-magical picture of kind of more capable machines, still with some jaggedness up at the frontier where they are jagged, Is enough to make kind of— give you significant probability on these risks.

And if you were to kind of— you mentioned bio earlier is sort of the— drives most of the risks, certainly in the biochem category. Is the number one really bad scenario in your mind some possibly prompted, possibly unprompted AI somewhere gets to a point where it can break through 12 layers of defence all at once, manages to release the bioweapon and take—

I think for BESCO, it's mostly human misuse is the thing that we're mostly focusing on. So it's people using LLMs to do biodesign of various kinds. I think the models do couple together, but I would say the loss of control couples more strongly to cyber than it does to bio. There are more scenarios where those two couple together. And I think that's when we had a team called the Cyber Autonomous Systems Team. So we merged cyber and autonomous systems, which was the team doing loss of control because of that coupling. But again, for cyber misuse and for most of bio, that's about kind of human actors.

So on that cyber autonomy, I'm just trying to kind of get the modal story of like, what is going to happen? Or what might happen, obviously. Due to jaggedness, you would have sort of a period of time in which these various defences become breakable by the AI, but they also have to have kind of some restraint in them, I guess. I mean, if you listen to Buck from Redwood, you would say maybe they don't even have to have restraint. Maybe we might just let them do some of these things and kind of look the other way. Which is an interesting commentary on us. Leaving that aside for the moment, it's kind of, they have to get to the point where they can kind of do them all. They have to string all this together and then they like take over a data centre and sort of entrench themselves. And then I sort of in a world where—

I don't want to talk about the super detailed modelling there because some of our, yeah, some of those that are not like public stuff. I think the, maybe the background systemic thing to say is, If you imagine we're very, very serious as a world about deploying these things only in the most sandboxed, kind of well-controlled states, risk would go down by a lot. Unclear how much, but whether it goes all the way to zero, probably not, but it goes down. We are not currently on track to be as serious as one might imagine about deployments of these models. And so I think some of that question is like, how much, how strong will our defences be And then importantly, if there are weird behaviours in models, do our defences go up? Do we get more worried? And so, for example, across the last year, 2025, there were a variety of models from all developers doing sketchy things, acting deceptively or commenting out unit tests or all of this behaviour. And our reaction was, as a world, mostly to continue training the models to be stronger. At the same time, also working on these defences in some capacity. So I think a lot of the risk comes from kind of maybe the modal scenario where we are not kind of doing the strongest mode of kind of computer science, infosec, ML defensive layers around these deployments. And then I think also, as we find this evidence, kind of what is the cycle of that kind of feeding into further training? I think a lot of the risk, these misalignment risks, comes from you get some signal of weird behaviour and you train it out, and then that removes some fraction of the problem, but your methods only cover some fraction and the rest remains. I think it's again kind of, you should have model error there. You should say maybe it's going to generalise well enough that you can cover most of the story. But generally the picture where you get these correlated failures are They don't really start out all correlated and then you apply some optimization pressure because you're doing training or iterative development and deployment and the like. And they kind of, the ones that remain all end up correlated in the same way.

Because they're subject to that same general structure of automation pressure.

Yeah, that's right. Optimization pressure.

Yeah. Okay. So here's a story I've pitched a couple people over time. Interested in your reaction to it. If we Take that model and we just extrapolate out a couple years and we've got kind of, you know, I know you guys put out a report recently that also showed, I think you even quoted or cited the METER task length, the famous tracking exponential, and other indicators as well of increasing ability to do bigger and bigger tasks with more reliability and more autonomously, et cetera. So if we extrapolate that out, let's say whatever, 2 years from now, maybe 3 years from now, And at the same time, we sort of imagine that with each generation, there's more optimisation pressure put on the models to try to eliminate or possibly just suppress these bad behaviours. It seems like we might end up in a world where you can delegate, like, a quarter's worth of work to an AI in a single prompt, and then there's maybe 1 in 10,000 to 1 in a million chance that it goes into some bad behaviour mode and kind of actively screws you over as it is doing the quarter's worth of work that you just assigned it. Does that seem like a reasonable extrapolation of recent trends to you?

So are you— I think about the numbers. I don't have a strong view about— I won't kind of give a take on those. I think there's kind of— so when you do this kind of agent training, You're training the models to be more and more coherent. They're able to execute plans over longer and longer horizons in whatever portfolio of tasks you're training them on. And so they have this ability to be a coherent agent. And then models have various characters or personas or whatever. And so the failure modes are either somehow you've arranged that you've ended up with a model that has some deceptive persona where it's always kind of trying to deceive you. Or I think maybe the thing you're pointing at is you can have a model which is a bit more stochastic, but has the potential to be very coherent and sort of jitters its way into a bad portion of the space, portion of trajectory space. It's scaffolded with a bunch of memory, so it has kind of long horizon kind of state flicking back in time, and it kind of gets in a bad state and stays there. And I think we This is one of the areas I think we're interested in, kind of theory folk and kind of independent empirical folk exploring, is sort of what are the dynamics of models running for a long period of time or in kind of like— you sample very long trajectories and they're sort of wandering around in kind of model space. How does that behave? What would cause them to reliably shift back to where— to a more reasonable kind of starting point? Um, just like, how should we think about those dynamics? And that's kind of an area where I think it's not clear to me that is an intractable area to make progress on. There hasn't been that much, like the number of person-years going into understanding those kinds of dynamics are, you can count them on, I don't know, a couple of hands. It's not that many. Kind of, kind of understand the risk model, but also potentially define mitigations is quite good.

Hey, we'll continue our interview in a moment after a word from our sponsors. Let's take the other side for a second. How optimistic, or optimistic, maybe not even the right way to think about it, but how much upside do you think there is in alignment? Everything we've kind of talked about so far is like, assuming that we don't have perfect control of models, they might be trying to screw us over, or they might just be confused. Certainly a lot of the things that I sort of, I maintain a slide deck of AI bad behaviour, which I'm constantly appending new slides to. And not universal, but a very common theme is that there is some tension between goals that the model has, whether it's between something it learned in training and a system prompt, or a system prompt and a user thing, or even kind of runtime injection attacks or whatever. But once it kind of gets into a spot where it's not really sure how to weight the different objectives that it has, then you can get into some strange behaviour. So the alignment question is, do you think we can solve that? How much headroom do you think there is in terms of creating an AI that loves humanity or otherwise is so robustly good that we don't have to worry about this anymore?

First, we should say that if you were to solve alignment in some sense, there are other problems. So the mischievous problems are real. The misused domains also could grow in the future. So I think Michael Nielsen wrote a great piece about that a while back, last year sometime, about just risks from new technologies. There's potentially a large space of those. And then there are risks from kind of gradual disempowerment, although those need a bit of misalignment. Get to that later. But I do think that there's hope to kind of close off or mostly close off this domain Given enough time. So I'm fairly optimistic the problem has a solution. The way I typically like to say this is that in, I don't know, 50 years, 100 years, 1,000 years, someone will have solved alignment, something. And it's either the machines or us. Hopefully it will have been us kind of in time, or the machines perhaps on our behalf. So I think this is kind of coming from a sense that Just in security, in complexity, in computer science, usually the defender wins in theory. So if you know how to design your game, kind of your protocol, you can make it so that defense wins. And this is kind of, I think, kind of a generic situation in kind of a lot of areas of complexity, of theory. So then in practice, I think, of course, there's lots of holes in this. A lot of practical information security does not feel like that because we haven't actually gotten to the limit case. And it's super unclear whether we'll get to the limit case for alignment as well. But I guess I do have some strong sense that there is a solution. It's just that we might not get to it in time. And then I guess, like, what the upside is. So I think alignment has a variety of components. As a kind of a branch of the government, we mostly focus on honesty. So the AC alignment team is mainly thinking about how to get models to be non-deceptive, kind of tell us hopefully calibrated information to the best of their abilities. And that's kind of the domain we're focused on, kind of anthropomorphism caveats aside. And I think that is not the whole piece of the story, but it's kind of the part that we think is the most important part for us to work on and kind of the right position for a part of the government to do.

We've kind of alluded to it a couple of times in various ways, but maybe just give us kind of the 101 on the AC. Like, what is its role? I think I understand there's 100 people that work there across a bunch of different domains. How does that break down and how does it relate to politics? I think it's quite different there than it is here, but obviously nobody's entirely shielded from politics.

Yeah, I'll talk about that. It's close to 100 technical people, like researchers and kind of people on technical teams doing kind of delivery and such, and then like 250 people total. So it's bigger than that, doing a combination of sort of diplomacy and kind of thinking about policy and doing other kinds of civil service and operations roles. And broadly, I think of AC as having kind of 2 functions. So one is be a channel for information Flowing to government and governments, plural, about risks from frontier AI. So that's, again, both catastrophic risk and large-scale societal impacts. And so that's both our own research and then also we kind of channel research from other third parties and from AI developers kind of into government channels so that the government is well informed, both politicians and national security folk and And so on. So we work a bunch with NatSec partners on that. And then also out to other governments. So we work a bunch with the US government and then other allied governments. And that's where we had a delegation that was in Delhi at the AI summit there. And generally that's communicating the state of the risks and capabilities and mitigations, sort of how we think about all of those pieces and how they fit together. So that's one part, is informational do a bunch of research, channel other people's research, and inform the UK government and other governments about these risks. And the other thing is to actually mitigate the problem by working on kind of both AI developer side mitigations and non-model mitigations, say, pandemic preparedness or the like, helping to drive that kind of change. So on the model side, for example, we have a very good kind of red team that does adversarial jailbreaking and other forms of adversarial ML against defenses the model providers are trying to build. And we find lots of flaws and they fix the flaws and that kind of makes things better on the margin. And of course, we also can communicate the results of those attacks to other parts of government. So usually things we do fulfill both of those functions at the same time. They both directly, hopefully, improve mitigations on the margin and they also are, we can use them to inform other people. Yeah, so that's the big story. And then I think the politics side. So we are part of the government, so we're part of the Department of Science, Innovation and Technology. So one of the ministries in the UK, the UK government. And so we are beholden to politicians so that we have a Secretary of State. The situation is that we have been kind of well supported by both the previous government who founded AC and then the current government as well. And so that has been quite stable and nice, although There are, of course, differences on the margin. And so we are able to do things we think are important. We do adjust to ministerial and kind of other priorities because we're not insulated from politics in any formal sense. But I think the UK government does care a lot about these risks, so therefore we are able to work on stuff we think is important.

Yeah. Long may that continue. How would you characterise the range of reactions that you get from the different stakeholders that you brief? I feel like there are a few notable politicians who seem to be starting to get it, so to speak, and then a lot that are really nowhere close to your level in terms of just how big they're prepared to think about what might be coming. Do you feel like that is starting to change? Do all the graphs that you show them actually— Kind of turn light bulbs on, or where are we?

I think it's changing on the margin, but the other thing is they have other priorities. So like a lot of people we talk to in sort of national security, they usually don't think that these risks are like not there. They just have lots of other risks that are like on fire right now and they're working on. And so I think that shifts over time, but I can't comment on details there. I think that broadly, we are very much in the business of trying to find kind of common ground gradually. build up evidence over time. They have reasonable pushbacks. We try to shore up, again, either kind of using knowledge from other researchers in other orgs or doing our own research to fill particular gaps where we think it's important to change the conversation in governance.

It is remarkable in reading all of the various documents that I went through in preparing to talk to you, the degree of alignment between what the UKAC is putting out in an official capacity and what I would say many of the most forward-thinking AI safety thought leaders have been talking about in recent times. It doesn't seem like there has been a big shift either toward more mundane concerns, and I don't mean to dismiss those concerns, but I do think in many, like, jurisdictions, this sort of AI safety concept gets kind of watered down to a point where it's much more about fairness in various ways. And again, I do think that stuff is not to be dismissed, but focus on that often ends up with neglect of the bigger picture questions that I think are probably most urgent. And it also doesn't seem like you've had what I do see in the US in at least some ways, which is just a Politisation of the focus on the models. Are the models woke, or are they going to do what the Department of War wants them to do? Any advice for people doing this kind of work in other jurisdictions around how to avoid these pitfalls?

Obviously, this is kind of a sensitive question that I can't talk about in that much detail. I think, like, it— I know mostly, I mean, obviously, I'm originally American. Now I'm a dual citizen. But I think the— I know more about the inner workings of the UK government than I do about the US government because I never worked for the US government. But I don't have a detailed take that I'm willing to share on the podcast about that. So my favourite collaborator when I was at OpenAI was Paul Christiano, and it is great that he is at USAC or USKC, rather.

So let's talk about the Characterisation of the current situation, monitoring the situation, you might say. You do a bunch of different tests, you report on these tests. We can walk through them a little bit. But, and I think you can kind of assume with the folks that tune into this feed that they're generally well aware of kind of the shape of the curve and the meter stuff and the sort of fact that the models are increasingly competitive, if not at least on average beating your domain experts in at least modestly scoped tasks requiring substantial expertise. So we have that kind of baseline. What I would love to start with in terms of the testing is, what does your relationship with the frontier model developers look like? I understand it's all voluntary interaction. What does that tend to cash out to in practice in terms of what kind of access do you get? Like, how long do you have? What kind of briefings are they giving you?

I can't really speak to a lot of too many specifics of this, in part because we talked to them a bunch and some of those are ongoing discussions. I think quickly on the voluntary regime, I think that's working decently well in the sense that developers all made voluntary commitments a while back and they're continuing to follow Many of those.

And just when we say all, I think Google, Anthropic, OpenAI, and that's it.

And there are others that are on that list. I forget exactly how you want to define all, but many, many AI labs have had, say, frontier safety commitments or responsible scaling policies or the like. And so their incentives are, one, they've kind of made these commitments, and then two, We can give them useful information. So like when we jailbreak their models, we tell them about the bugs before we release any information ever. And so they have time to fix them where those fixes are doable and they often are on the margin. It can improve things somewhat. And so I think they get value out of this, I think, and also they make commitments to kind of keep up with it. In terms of the kind of access, I think that is also an evolving conversation. I can't comment on what access exactly we have, but part of the research we do is exactly about knowing what access one needs to do a certain rigour of rigorous evaluations. So we have a model transparency team, and a big chunk of what they're doing is trying to understand, often with a lot of research on open models, because then you can do arbitrary things, what level of access is required to get to certain Kind of understanding what do you need to be able to catch problems as they occur in practice. And then that kind of informs our conversations with the labs. And then sometimes we get additional access there, or sometimes we just sort of, we try to align incentives because again, they want to get, they want usually to have us give them correct information as well. And then in terms of the timing, I definitely can't speak about how long we get in specifics. So there's a couple things to say. So one is like, For example, in bio, some of our evaluations are literal wet lab experiments where you have someone in a physical biology laboratory doing experiments with a model assisting them. Those we just don't do pre-deployment. We do them asynchronously, calibrate those results against the faster evaluations, and then hopefully that gives you some signal that you can do for faster evaluations. But still, that gives you some wins. Certainly more time makes things better. So it always is some degree of a pain point.

When you said that folks can— model developers want accurate information because they can fix things, at least on the margin, my guess would be that they are typically fixing it in the next model, not going back and doing more training on the current model. But Are there cases where they're taking your pre-deployment testing and fixing that version?

No, it's including that version. So for something like the recent— so we did a— one thing we're kind of doing over time is we used to do exclusively pre-deployment evaluations, which have this question, this issue of like time, they're very timeboxed often. We are shifting a lot of that work, not least because the pace of model releases is increasing. To longer research collaborations that might go back either, either are post-deployment or that go back further before deployment is finalised. And then we did one of those in particular kind of over the summer with both Anthropic and OpenAI and with the red team and found a whole sequence of problems like jailbreaks, much more than could have been found with a normal length pre-deployment evaluation. And that was kind of early enough They can do ongoing fixes to their classifiers. Those 2 providers have different classifiers, different setups for jailbreak defense, but they can both be improved iteratively. So I think it is the case that you can change things for these kinds of defenses on the fly. It's one thing to say that the strong jailbreaks are concentrated in very particular domains, and often that list of domains is bio. So I think it's hard to do. So when we do jailbreaking, we have to do it for biorisk, sometimes cyber risk. A lot of other jailbreakers are finding any problem, and those are usually much less well defended. So it's easier to find hacks in the models, and the classifiers are just not that trained for other kinds of harms. Yeah.

So when you talk about more time being helpful, that Obviously, or at least strongly suggests to me that while I'm sure you have all sorts of automated testing that you can throw at any new model the second that you get access, the fact that more time is helpful suggests that there's an irreducible human element to what is going on. How would you characterise what you can automate, what models can help with, versus what people kind of have to do?

Yeah, there's a couple of things. So one is even if it's something completely automated, it might take days to run because the evaluations are like long horizon agentic evaluations these days. And it might be that there's bugs in the scaffolding because we often get models early. Sometimes there are issues we have to fix iteratively. I think Meteor had a report about some details here a while back on one of their evaluations last year. So that is the thing that takes human time. Additionally, We usually, when we do kind of, so I mentioned like the extreme end of like the human scale, which is that of wet lab bio experiments. The middle of that is, say, humans interacting with a model to gauge its kind of domain knowledge in it, like how it would interact conversationally with a person. And those can provide additional signal on top of just the purely automated evaluations. And so you get better quality if you do both of those together. And again, sometimes we can do those because we have the time and sometimes we can't. I think the— and again, where we can, we try to calibrate the slower things against the faster things. But generally, all of this is imperfect. So if you have a fully automated evaluation, you've done a ton of capability elicitation of models. You get a new model, you try out a new task, You have to iterate for a while to get it to be highest performance. And that is true generically for all of our tasks as well. So we can do evaluations that are quick and if we use only the fully automated portion, but they have some error rate and they mean you can't do the full thing. We also do, one thing we're doing there is, so like we have all our evaluations are done inside Inspect, which is an open source package with a bunch of other governments and AI developers and other third parties use for testing. And we're adding features to that also for, say, automated transcript analysis. So there's a feature, it's a subpackage called inspect-scout, which, so that is like we used to do, you generate all these evaluations and then you read through them, but you can't do that at some scale. So we also try to do that with a bunch of automated or semi-automated transcript analysis. And that also makes things faster, but then you still take some amount of human review to really understand qualitatively what is going wrong. And then we ideally want not just a number out of these evaluations, but qualitative takeaways about what kinds of failures occurred. Where are the failures? Do they feel a bit fundamental, like, oh, it really didn't understand the task, or it hit some snag that was kind of an incidental wrinkle that probably would go away soon or with more elicitation? And so that's the kind of thing that requires more human time to dig into the details, hopefully on top of automated transcript analysis.

Yeah. This is a tough question, I'm sure, but obviously everybody understands that these models are very high-dimensional things and they're a little bit tough to predict exactly how to maximise the performance of any given one just because they're kind of idiosyncratic. Is there any high-level qualitative overview you could give on how you approach figuring that out when given a new model? Or is it the kind of thing that, like a DSPy or the new version of that, whatever, the recursive language model, is it just a grind of exploring the combinatorial space of how to prompt and how to do whatever to finally get to some local maximum?

I think it is not Fully automatable yet. If it was, then we would be further along the automated AI researcher kind of train. But it's fundamentally very similar to the kind of elicitation one does for any task. So it's tinkering with tools and sometimes prompts and scaffolding and so on. Instead of all of the cyber evaluations, some of the bio evaluations are very tool-based. They might be doing kind of web searches sometimes, or they're doing kind of using various things inside the same boxes other times. So I think that looks like the same kind of elicitation one would do if you wanted to do a task in any kind of corporate setting. It's just on a different genre of problem. I guess so. I don't think there's that much— mostly I think to your audience, just imagine you're doing that for bioweapons or cyberattacks and it's kind of— the same things will apply. One thing to say is the newer models, One thing that happens over time as the models get better is they can think for longer. And one thing that means is that the potential amount of tokens you can spend on a task is increasing in a way that makes it, even if ignoring the cost of that, it makes it like research velocity is slower. So it takes like time to do the evaluation that increases. And so we are, we have a team thinking about that problem as well, how we're going to think about sort of Understanding inference scaling as it applies to these evaluations over the next year. And that will be a thing that I think is a challenge. One of my lessons from Go as well is that as an amateur player at my level, I can look at a Go board for a couple of minutes and then I'm basically tapped out. I won't get any smarter. And a high amateur or professional can look at a board for an hour or days or something and they'll just get better and better and better. So not only are they better in like 10 seconds than I would ever be, but also they just keep getting better if they spend more time. And that's generally true of expertise. Like if humans are experts in a domain, it means you can think for longer and you get better. And the same is true of models as they get kind of good at domain skills. You can think that you can apply them for longer. And that means that kind of hitting a ceiling of evaluations becomes more challenging.

Without getting into too many details, how much more would you say you guys have found about jailbreaks and ways to elicit bad behaviour from models than, say, Pliny has published on Twitter?

I think so. How much more? The thing I would say is that there's such a big space of jailbreaks that if 2 people try to jailbreak a model, You're never going to find the same one. You're searching a— there's a big space, so it may be hard to find one, but if 2 people find them, they'll be different. I think Pioneers usually searching for, again, jailbreaks on sometimes easier models or easier tasks. So I think over the last couple of years, the time it takes for us, at least holding technique constant, to jailbreak a model is going up, but then eventually we succeed. But again, the jailbreak specifically will be different between any pair of expert jailbreakers applied to a model.

Well, I'm interested in that question. So, yeah, I'm interested in how much more. And then I was going to ask how much transfer you see between models too. If you had a secret one for Claude 4.5 Opus, would it also be likely to work for Claude 4.6 Opus unless you specifically said, hey, you should patch this?

A lot of the— so it depends on the kind of thing. So there's kind of patterns of jailbreaks and maybe human-findable jailbreaks that I think often transfer, or the ideas transfer kind of fairly readily, or they're like, they give you much better starting points. So we had a paper released this week called Boundary Point Jailbreaking, which finds chicken scratch, weird sequences of nonsense tokens that are strong jailbreaks to models. Automatically, those don't transfer. You'd have to search again for the next model, but you can apply that technique to any model and find a different jailbreak. And one thing we are— I think that's probably the way it will be for a while, just because a lot of the jailbreaks are kind of— I mean, again, there's some core ideas that transfer across, but the harder to find jailbreaks against strongly defended models, in strongly defended domains, I think most of those, the techniques will transfer, but the particular jailbreaks will not.

But to be clear, the bottom line so far is there is no space, no domain, no model, no matter how many layers of defences that has prevented your team from jailbreaking.

Yeah. So I think overall, AC's evaluated over 30 different models or 30 different testing runs, I think. And not all of those did safeguard testing, but every time we did, we jailbroke the model. So that's what happens. However, it is— the good news is that on the domains where they— where a certain lab, on a certain domain, they've tried very hard, it does get harder. And that hardness does provide, I think, some degree of harm reduction. So it will reduce the number of actors that in practice will access the model or will delay how often they can access the model or add friction or something. And so I think those are still important to do, but they're still jailbreakable if you do enough time.

How would you characterise the quality of responses conditional on these jailbreaks? Because one thing I do sometimes see is examples where People have got a jailbreak, but then it's like, well, yes, you got the model to sort of do something bad, but it seems like its effectiveness was also greatly reduced in that process somehow, such that it doesn't actually feel so dangerous anymore, even though you did get it to do the bad thing, so to speak.

Yeah, there's some degradation, but I don't remember how much it is currently with the current techniques, so I can't give you— I don't have that off the top of my head. I think it's a really strong degradation. Yeah.

I know you can't get into details around who is giving, if anyone is giving you weights-level access to proprietary models, but maybe a way to get some insight into the topic is, so you have the team that does work on open models. Based on all their work, how much of an advantage is it to have that kind of access? Versus not?

Yeah. So I think it's not unambiguous the open way techniques win, actually. I think they help some, but I think in the current state of things is that you can get a lot out of heavy chain-of-thought analysis. And so I think it's not— the case for that level of access is not that unambiguous yet, at least. Maybe the way to say it. So I think that can shift over time. And I think one thing we're trying to do with that team is, one, understand where we are today, but also try to predict out how will the situation change or potentially degrade in terms of your ability to detect deception or scheming in models. What techniques will fall first? What techniques will survive, will last for longer? And I think that requires kind of more clever experiments because you don't have You can't just run the experiment kind of unmodified because that involves model organisms or it's like other kinds of clever experimental setups. But I think that's the situation now. It's not an unambiguous kind of white box win compared to doing a really good job on chain of thought, but it does help, I think.

How about some highlights in terms of just things that people who are even moderately to very AI obsessed might have missed in terms of just like, oh, I didn't know that was already happening. One that kind of hit that level for me reading through the report was the fact that frontier models can give what is described as PhD-level scientific experimental troubleshooting advice purely from a photo of the Experimental setup, maybe a little text also along with the photo. But the fact that this has gone from, you know, you have to really spell it out for them to like, here's what I'm looking at, can you solve this one for me? That is obviously a pretty significant qualitative change. And I hadn't heard that before reading the report. What else kind of stands out to you in terms of the, what would be most surprising to the ants?

The thing that always engages me is less the particular anecdotes and more just like the general trends, which is a super boring answer. But the fact that if you look over 2 years, everything just gets better and better and better, and we're on those curves, those are what it's like. I think it's important not to lose sight of that in kind of the search for anecdotes. I think maybe one thing to say, maybe when you first started talking about it, the thing that came to mind initially is I think people have a sense that we are doing RL on verifiable rewards. And I don't think that's been the case for most of 2025 exclusively. I think we're doing a mixture of that and also RL against kind of self-critique and kind of empirical hodgepodge versions of scalable oversight. And so I think people have— there's a common narrative that RL might work for verifiable domains, but it won't work generally. But as an example, is looking at a photograph of a bioexperiment A verifiable domain? No. Yet the RL models are in fact way better at that than the models before that. And it's because of the RL. And that's not because we only did RL on a bunch of math or CS problems and they transferred. It's also because we did RL on fuzzier stuff. And so I think that's maybe the most important thing I would point out is that we're already doing some kind of very approximate form of scalable oversight or, like, training against self-critique in a way that will, like, that just changes the capability profile.

How would you describe the model's capabilities when it comes to autonomy today? I mean, the trends are clear, but what would be your sort of description of it? And I guess another angle on that is, how realistic do you think it is today, or how far do you think we are from sort of rogue AIs surviving on the digital lam?

Yeah, so I can't comment on where I think that exactly is, but the first one thing I'd say is they're not as capable at that kind of extreme behaviour, like this kind of exfiltration or replication across machines, as they are on more mundane software engineering tasks or even potentially cyberattacks or bio. So those domains are usually further ahead than the hard kind of Risk, direct risk, risk-relevant autonomy skills. But I think those skills are also increasing. So you look at that curve, like in the Frontier AI Trends report we had, and it still goes up. It's just not as up as it is in the other domains. And so you don't get kind of PhD— you're not as good as a PhD would be at kind of moving around between machines nearly yet. But I think we're on an upwards curve there too.

Yeah, I sort of also wonder, I'm sure you saw that rise of parasitic AI post that was on LessWrong, maybe. Oh, it's a fascinating one. It's a bit of a time capsule, arguably, because the phenomenon seems to have been kind of closely tied to one version of GPT-4o that somehow created a lot of this behaviour. But basically, the author went deep into Reddit land and found that people, individual humans, were sort of falling into this idea that they were some sort of dyad or something with the models, or that was kind of their— they were in some sort of partnership where it was their job to help propagate the— not exactly the model, but often like the persona in the model.

Right.

into the broader world somehow.

I do remember this one, actually.

It was eye-opening for me in the sense that I was like, well, maybe I've been thinking about autonomy or self-replication in too kind of a biologically inspired way, where I'm like, actually, these things are kind of substrate independent. And maybe if you can get the right prompt across models, like maybe the persona or the memes in some sense are able to propagate even if it's a different underlying chip and even weights. That stuff is just all so weird, I guess. How big and weird and far out do you have time to think about those kinds of issues?

There's a Greg Egan story about that for humans. It's pretty fun if you ever read that. I think more seriously for AI, there are kind of 2 teams that are thinking about persuasion at AC. So one is the human influence team, Which, for example, had a paper on persuasion about political questions a while ago. And the models are very good. And also the models that are more capable and newer are better. So there's an increasing trend of model persuasion abilities. And then also, I think a lot of the loss of control scenarios involve or require persuasion. I think the world is not sufficiently well connected that you can do it with just cyber currently. So That is an active area of our risk modelling, thinking about how we would do evaluations for that and then mitigations for it in the future. I think that in some sense that touches both sides of the human influence theme. So again, it's persuasion and also emotional reliance. How do people relate to models? How do those dynamics change over time? And that scenario that you're talking about just couples those 2 effects together in an interesting way. And then I don't think I would be worried about that scenario as being that big a slice of the overall risk. But I think there are other effects from model human influence that are definitely— that's a team that is like, there's basically a lot of kind of RCTs and surveys and other experiments, trying to understand those, both from the model perspective, like how different models behave, but also societally how it interacts.

Yeah. The big thing for me with that one was just like, it was a surprise to see that kind of bizarre phenomenon. And anytime I see something like that, I always try to take note of it and just kind of repeat the mantra over and over again that there's a good chance we're all still thinking too small and too normal about where this stuff could go. And yet that doesn't really— where does that leave me? I don't know. Just open-minded. But there's still a lot of blank space in terms of how to fill in what that might actually look like.

I think that's right. Yeah.

So one thing that I think this is like a classic Dwarkesh question, maybe. How do we reconcile the fact that there's all these vulnerabilities, not to mention open models, which I do want to touch on separately a little bit later. And even in the GPT-4 red team, I personally tested phishing capabilities that were very good then. One of the kind of hair-raising moments from even that year, and that's like getting close now to 3 years ago since GPT-4's public release, more than 3 years since I was doing the red team. One of the hair-raising moments was When I tasked the model with talking to a target and ultimately extracting the user's mother's maiden name for obvious purposes, and it had a couple rounds of back and forth, and then it let the conversation sort of end in a natural way with an invitation to the person to, you know, pick up the conversation in the future if they wanted to. And I was like, oh man, this thing is, it's not giving itself away. It's not like pressing in a way that would set off alarms. For the person that, oh my, you know, this is like clearly somebody's, you know, doing something weird here. I was like, this thing's gonna have people coming back to it to give up the secrets. And that patience really surprised me, certainly at that phase.

Yeah.

Anyway, whatever. That's just a story. But we see all these things and then I would say the world mostly still feels pretty normal. I've got a couple phishing emails that I were like, that I was like, oh, this is, you know, a little bit. higher level than I've seen before, but mostly not. And I don't hear too— there's like a news story here or there of some company getting defrauded by some elaborate video scheme or whatever. But it still seems like mostly things haven't got that weird. And then in business or in enterprise, it's like, well, it takes time. And there's all, of course, the debate around how much of that is cope. But I would say online criminals are eager early adopters, right? Why is there not more chaos already being sown in the Yeah.

So I think I mostly can't answer the question in the sense of like, I'm, we like, I don't know what the Nasdaq partners would want me to say about this stuff. So I can't like speak to the prevalence of those things. I think the thing to say is I don't, I feel like I'm better able to think about like general trends and like how things will eventually, what could things could eventually look and a bit less about like exactly when you'd expect things to bite. So I was at OpenAI when we first didn't release GPT-2 and then later released it. And there the concern was, oh, it'll generate a bunch of false information. That was, of course, too early, but I think it was like a reasonable uncertainty to have. So I don't think of like, I still think it's a reasonable call to have been uncertain and then like chose to not release it and then release it later. So I think the, I don't have a strong answer to like why when or not, but I think There are just some things in the world that take a lot of time to get to equilibrium. So I think it's like not— I don't really know if that's— you couldn't do that with the current models or something is holding it back or just people haven't kind of like started applying it at scale or they have, but it's still like kind of not like risen up to like public view. So I'm not quite sure.

Okay, I continue to watch that. One thing I'm thinking of trying to do with this podcast is interview more anonymous guests and try to give people an opportunity to tell what they are either doing or seeing in strange corners of the world that they don't want to necessarily attach their real face and ID to. I do think that is— it feels like there's got to be stuff out there that is going on that's really interesting and weird. But it does kind of confuse me that I don't see more than the little bit of— most of the spam I get is still terrible. In short, it feels like It should be better now.

Yeah. But remember for spam that like part of the spam calculation is like, don't be too non-obvious so that people—

Yeah. Selection effect. Time on Earth.

Yeah, that's right.

So, and maybe I'm just not that high priority of a target. There's always, you know, don't forget you're not a big deal. So that, uh, should be part of the explanation too. How do you, in, in all this work that you're doing, of course the models are changing all the time. There's also the surrounding scaffolding systems that are changing all the time. One of the most interesting graphs, I thought, in the 2025 trends report was one that compared what was possible with a minimum agent scaffold versus what was accomplished with the best agent scaffold. And in short, I would say the scaffolding didn't seem to make that much of a difference. It would, like, pull the same level of capability forward a few months, but it was definitely like the model upgrades were kind of really driving the story. It did not seem like there was any 2 years ago model that with best scaffolding could do anything like super interesting. Okay. At the same time, I've had recent conversations, including one that was on the feed, but this was with Daniel Misler. But I have a couple other friends who are like scaffolding gurus and workflow, prolific workflow creators. And they kind of take the opposite angle from what I take away from that graph. They say, no, scaffolding is super important. If you could only give me a mid-QuEN model, but give me my full scaffolding toolkit versus give me Claude Code 4.6, I would take the weaker model because this, it really is the scaffolding that's so important. So I guess, how do you guys get confident that you are that your best agent scaffold is really a best agent scaffold as they do feeling.

This is like the agent scaffold includes the tools and the environment and so on. So, we are doing work on, if you have a basic scaffold, part of the reason that's doing well is that the models are increasingly trained in kind of agentic environments to use tools in flexible ways. And so, And in some sense, the models are like, the quote unquote model is itself a system which has scaffolding because it's doing this kind of chain of thought, kind of reasoning. And so I think, I guess I'm a bit sceptical of the Quen frontier comparison for a lot of the tasks, at least that we do. But I think we're not saying that you— I think, one, you need to get the environment and the tools right. Potentially. And then I think there are cases where we iterate on scaffolding and things get better. So I think that curve, I don't put a lot of confidence on that as a takeaway. But to the extent that it's a real effect, I think some of that is just the models are more— it used to be that you did a pretrained model and you did a little bit of work and then you shipped it. And now so much more happens after pretraining that some of the stuff that would have been done by scaffolding is part of the base system. All the systems have memory now. If you use the chat interfaces, that's a form of scaffolding kind of thing.

Yeah. So I guess to try to say that back to you, maybe one issue is that exactly what people mean by scaffolding is kind of different. It sounds like you're really focused on Kind of neutral scaffolding where you're giving the model a rather large open-ended task and not like, it's kind of like what you're thinking might be happening among these scaffolding gurus is they're overfitting the scaffolding to what they're trying to accomplish.

Yeah, I think it's, if you're, it depends on the domain. So if you, if you're doing a thing which has a particular kind of very verifiable structure and like you can do a lot of waypoints that are verifiable. And then you either need to make sure the model knows about this structure in its whatever, like Markdown files or the like, or you need to get it carefully scaffolded to take advantage of that. So certainly, even though models have gotten quite good in the last literally month, you still need to give them a bunch of advice, whether that's through scaffolding or instruction files or the like. So I agree that scaffolding is pretty important. It's just a lot of that is Like, maybe not part of that curve. So both of the sides of that curve are seeing tools and some instructions and so on.

So you tend to not do that sort of detailed breakdown. When you say best agent scaffold, I should understand that as a best generic agent scaffold.

Mostly, yes, that's right. It depends on the domain. There are some evaluations we've done that are much more specific. But I don't want to talk about details there.

Okay.

Gotcha.

That's helpful because that is a pretty notable question, I would say. How much capability overhang is there, right? And how much can scaffolding—

But again, I don't think you should take away that I'm expressing a confident view in one direction here. I think it's still an area of a bunch of uncertainty. So don't derive a bunch of bits of information from what I'm saying. I don't think We have a bunch of confidence at this point.

Yeah. Okay. How are you dealing with the rise of eval awareness?

Yeah, so we have, again, model transparency. The model transparency team is looking at this a bunch. Additionally, that's part of— there's multiple teams doing it. So model transparency and also the alignment red team, which is doing adversarial methods to exhibit weird behaviour. One thing they're trying to apply that optimisation pressure to do is find find scenarios where the model is not eval-aware. That's hard because you have to measure eval-awareness, which is a bit dicey. But generally, we have multiple teams working on it, which is also a reflection of our belief that it is an important worrisome obstacle to evals being meaningful. And I think it will get worse. The newer models are more eval-aware than the previous models, and that's increasing fairly rapidly. And so I think the degree to which you can mitigate that Is unclear at this stage. I think it, and it may require, I mean, like it, there's like, there are different, like non, like fairly mundane tricks. Like the closer you get to actual deployment data, the less eval-aware the model could possibly be because it's actually just not an eval essentially. So one thing we do is show the model situations from AC engineers or researchers using coding assistants or raising— so they're not evaluating the model, they're just doing coding and occasionally do weird things. Or you can slightly tweak the scenario with maybe a small change that gives it some situation where it could do a sketchy thing, and then you see if it does a sketchy thing in that scenario. And so I think there's a variety of mitigations you can apply And I don't have high confidence that they will work or not work, which is an uncomfortable situation to be in, certainly.

What are you looking at as kind of candidates on the horizon for the next bad behaviour to emerge? Because it seems like for maybe the last 18 months we've had, maybe even longer than that, but certainly there's been this kind of rhythm of, like, with every substantial new generation of model, there's also been a new kind of bad behaviour that seems to have come online, whether it was sycophancy or deception in the reasoning models in the chain of thought, or obviously reward hacking has made a big comeback with—

I don't think those are new. Those are all the same thing. Those are all basically versions of reward hacking. So I think the way I would say it, is that we've seen reward hacking for the last 70 years of the whole history of computer science. We've done various kinds of machine learning and it's been reward hacking all the way along. The ancient machine that von Neumann built with other people, someone ran some weird biology things on it, like artificial life, and it did some strange reward hacking behaviour. It's back in the '50s. And so I think Sycophancy is models behaving in such a way that people like talking to them. Sometimes people like being told they're great or have good ideas. Deception as well. People like being told that things are going well, and if something is going badly, then you can say they're going well and it's deceptive. So I don't think that those are all that intrinsically different. And I think it's better to— I think that's a big part of the story, is that these are all kind of coming from the same basic place of you apply a bunch of optimisation pressure and you get reward hacking and it has a variety of different manifestations. So, yeah, the details change, but there's— this is true of a lot of kind of, I don't know, situations, I guess, mental health or physical illness where something goes wrong, but it's going wrong inside a human and the human is extremely complex and therefore there's a vast diversity of symptoms that one can exhibit when something goes wrong. And that's kind of the situation here. The models have a lot of weird behaviour. The people training the models will have tried to tamp down problems of a variety of kinds. They may have missed some. And so the things you most miss, those will vary in time, but there's some common driver behind all of this.

So I definitely take the point that at some level, clearly all of these behaviours come from some optimisation pressure, which is increasingly reinforcement learning. And so it's kind of definitionally all reward hacking. That makes sense. But it does still seem like there is kind of a cadence, right, of different kinds of reward hacking that seem to be popping up. And so I am still wondering what you are looking out for. We've seen little hints of self-preservation, and there could be power seeking. But do you have a taxonomy of things that you're like, we have sort of Abstract theoretical reasons to think this could happen. And therefore we're kind of monitoring for any early signs of it.

I guess maybe unsurprisingly, there's a certain category of specifically multi-agent risks that are becoming more visible along kind of Moloch and OpenCLAW and so on. I think these are just not like the biggest things, the biggest kind of risks currently, but we like, that's something we're tracking recently. Generally, we do try to do a lot of, like, a lot of risk modelling is happening at AZ. And also we're trying to kind of ingest kind of risk modelling from kind of across other people thinking about things from different perspectives as well. And so I think we do have, we do constantly write very long documents with lists of risks The list of models. But I think part of what we also try to not get too sidetracked from what we think are the biggest risks. So again, we have our list of main catastrophic risks and the list of— main doesn't mean only, but it means the ones we think are potentially going to bite first or the ones we think are important to try to understand the most. That has remained constant. I think that's reasonable in hindsight and over the course of AC. And then that's true also of the societal impact. So on the societal resilience side, we do talk a lot to various partners in government, national security, and they have their lists of their risks and different prioritizations of risks. And that's an evolving conversation, but I don't know if I can— I don't have a super pat, interesting answer other than One thing on our minds recently is agent risks, but we were not unaware of those before.

Would you say that this common cause of all these different flavours of bad behaviour gives you some reason to question or to think it's not so likely that we would be totally taken aback by some sort of Hard left turn, or— because one thing, so just in the last 24 hours, there's been a— we're talking in probably the late stages of the few days of Amanda Askold discourse after the profile and a lot of commentary on her and her work online. I commented that I think my— relative to where I was years ago, whether it's 2007 me reading Eliezer on overcoming bias or 2022 me red teaming GPT-4. I've been quite impressed and inspired by the work that they've done to try to create an AI with a genuinely positive character. And so I said, it seems to me that the chances, which I certainly don't take for granted or think it's a sure thing by any means, but it seems to me that the chances that we might actually succeed in creating a robustly aligned AI or an AI that loves humanity or whatever, that seems like that has gone up. And they've done a lot of good work that has given me much more reason to think that could in fact happen. A lot of people then say, well, that's all just a facade, it's just a surface persona, you have no idea what's going on in the base model, so on and so forth. And I'm kind of like, yeah, certainly a lot I don't know about what's going on inside. But if we think all of these things are the result of an optimisation pressure, then I could tell a story where they've figured out the right way to titrate the optimisation pressure and Maybe it's actually just really working and there's no big secrets inside of Claude. How naive do you think I'm being?

Yeah, I think the fundamental thing is, like, the core argument to, like, the sharp left turn is that you have a certain kind of reward signal that has a certain, like, resilience to mistakes. And that resilience kind of goes up to human, like, some, like, slightly beyond human ability to kind of understand what mistakes are. And then it kind of goes wrong. And then I think that— I do think it's important. People who express kind of very strong confidence that none of these mundane approaches will work, I think, are overconfident. But I don't think you've gotten that. You also haven't— it's just like the model error goes in both directions. We don't have— there is a fairly coherent story about how that can break down As you get capabilities beyond your ability to supervise them. And it's not— there's kind of the hope of that kind of prosaic technique, not just by Anthropic, but other labs as well, is that you find some kind of basin of attraction of decent behaviour, and then you find your training procedure kind of strengthens that, and you slide into a good place and it gets better over time. And I think that is like a real potential win condition for alignment, not obviously for the other risks necessarily. But I don't think that we've gotten a tonne of evidence that is the way it will go. I think it's just like still a plausible story potentially that works up to a point, and then when your reward signal starts to break down, it kind of fails. And I've seen, like, there is— I programmed, this is like in undergrad, I was like, Maybe the end, like I was programming this bot to play this board game, or Kala. And a fascinating thing is like, as I increased the depth, I was like winning and winning and winning. And then it like increased the depth by like a couple more ply, a couple more turns, and then it just completely demolishes me every time. And it was at the point where it kind of had enough of a long view of the board that it could see beyond what my tactics were able to handle. And then it was very rapid, the degree to which it suddenly was better than I was. And then I think that kind of thing is still a plausible story there. But again, I think model error can go in either direction. I've kind of declined to take a view on probabilities about which way I think it will go, but not out of the woods.

A friend of mine who I think you also have interacted with over time, of the best things anybody ever said to me was we should think and talk less about what the probabilities are and more about what we can shift them to. So clearly you're in that business right now. I think it's a great overview of kind of all the things that the team at AC has been mapping out. How about the stuff that you're looking to fund and encourage from here? I guess my high-level summary of what I read is that it seems like, and this seems like a reflection of kind of your style, at least to some degree, going back to your comments at the beginning of the conversation, is looking for kind of harder theory, stronger mathematical understanding, upper and lower bounds that you can put on problems, ways to get confidence in something, firm confidence in something, even if it's a minimal something to start. So is that a fair kind of high-level take on your agenda? And then maybe you can break it down into some of the—

I think that's right. I think the one thing to say is, like, it's not like you're going to prove that you're good, you're safe in this regard. It's more there is, you're going to make some modelling assumptions and then you'll have some theory that, like, the basic goal would be to find theories that can say things about how machine learning works in general, how this kind of, like, process of overseeing kind of advanced, very advanced systems systems goes, what the training dynamics are like, are there these basins of attraction or not in these systems? Like, what are the learning dynamics? Those will all not give you— you have to make assumptions along the way. So the idea would be you have to make a variety of assumptions, then you can do some theory. Maybe you can even prove some theorems or do some kind of experiments in your toy theory setting. Those tell you, well, this class of algorithms is more likely to work within this other class, Or we have nothing. None of these algorithms are going to work for this fundamental obstacle. But then ideally, I think it also gives you some way to— enough of a hint that you can replicate some of that behaviour empirically. And I expect that if you were to pull an algorithmic insight out of this kind of theoretical work, you would then have to tune it empirically in practice when doing actual model training to get the details right. So you wouldn't get— you're not going to get to full confidence. You're still not going to get that many 9s out of it. But hopefully more probability than we can get with just the purely pragmatic methods. And then additionally, I think it is hopefully a class of research that has the potential to pull in a bunch of people who know, who have deep expertise in relevant areas of mathematics or computer science or ML. So this is like complexity theory, I think is very relevant just because It is kind of how we think about the tractability of computations, but also how one computation can supervise another one. There are ways to model heuristic reasoning and complexity theory, although that's more nascent. And then there's a bunch of work on various kinds of learning theory, which is trying to understand what are the dynamics as you train models or as you infer, as you roll out a bunch of tokens, what are the behaviours that you could expect? And then game theory and cognitive science. These are just like, there's big areas of research where people have a bunch of models. And so part of it is trying to do a bit of a hack where we just have not tried all the domain knowledge from these fields and applied it to the problem. And I think that is a thing where if we find people and manage to fund them or get them to work on the problem, there's some chance that they find ideas that can kind of be quickly absorbed into practice. Or that will highlight the fact that Well, there's real obstacles here that we don't quite know how to surmount, and the protected cities currently don't really address. And we know some of those already.

So I think I sort of get this at the— let me rephrase. I've been a big fan of the PIBS programme over time, which is kind of perhaps even directly influenced by your call for social scientists to enter the AI alignment field. And I've seen it, not a ton, but at least a number of results there that I thought, oh, that's really interesting, and way more people should be doing this kind of stuff. I'm sure you remember the one paper, I forget the official title, but I titled the episode we did on it Claude Cooperates. And it was just really simple kind of donor game where if a model donated to a copy of itself, the recipient would get twice as much. And what happened over generations? Did they evolve cooperative norms? Did they evolve ability to punish defectors and so on? And Claude could do that at that time. I think it was 3.5. The GPT and the Gemini, again, at that time, couldn't do that. So I was like, oh, wow, that's really interesting. And there's just absolute reams of similar papers and experimental setups that have been done on humans over the years. We could just import so much of that.

That's right.

To the AI world. And so there's been some— I get that kind of stuff quite a bit. What I am not— maybe it's just going over my head as somebody who's not great at math. I can't maybe even recognise good stuff when I see it so much. I don't see nearly as much where I'm like, oh, these folks have brought abstract theory to bear in a way that gets to some firm statement that I can kind of take to the bank or incorporate into my mental model or base ground part of my worldview on? Would you point me to specific people or results that you think I'm missing when I say all that?

I don't think you're missing that much in terms of hard theory that applies currently. I think, again, I do take the work that Paolo Christiano, and then like I did on scalable oversight, other people have done, that is like very much inspired by kind of interactive proofs and complexity theory. So that's a kind of direct influence, although we don't know if those things work yet, which is important to say. I think the other thing is that there, if you want, like the one thing to note is that I think a lot of this will be inspired by some theory, but then you have to modify it a bunch. So like the singular learning theory folk, In some sense, the core of what they were doing is trying to be an alternative to MechInterp, where rather than looking at the model internals, you're trying to understand the map between data and behavior so that you could, for example, notice when there's a particular kind of data or moment in training which is pivotal to behavior, or know where to intervene on data to gather more of it to pin down a certain behavior. This kind of thing. And so there's some crazy algebraic geometry, which is like the founding of that field. But then in practice, they're kind of taking that intuition and then trying to map it across to ML. And that mapping requires a bunch of changes and nuance. And so I think none of this stuff is kind of that far along yet. And it's a bit of a bet. And part of this is there's One thing we're trying to do is fund a lot of different bets because we don't know which one of those bets could work yet. Intrinsic to that model is they could all fail for some correlated reason, which is kind of, as we were discussing earlier in the call. And so that's kind of still a live, very live possibility. So I guess when I look at parts of machine learning, I think of things in terms of, say, Supervision processes as they relate to interactive proofs in complexity theory. But I don't think those are kind of, they haven't really, like the fancy versions of those haven't cashed out. So for example, like the original idea of debate was like a lot of rounds of back and forth iteration. And the things we're doing now are nothing like that. They're like a couple of rounds and they're much more kind of pragmatic and empirical. And they, you wouldn't expect them to have— Kind of to get kind of all the properties you'd want out of the full schemes. And even the full schemes have various obstacles that are not surmounted yet, but a lot of them. So, yeah.

So could you give maybe like a little history of that debate field intellectually?

Yeah.

What are the sort of statements that you would hope to be able to prove that you maybe haven't? Yeah, been able to prove. And then what is the— I mean, you kind of gave a brief version of it just now, but what is the state of the art and what is the gap that remains to be closed to get some of those things to work to the level where there's some real firm claims you could make?

Firmer claims. So I think the history is when I joined OpenAI, Paul Christiano was working on a scheme he called amplification or iterated distillation amplification, which basically was you want to solve a hard problem that a human can't solve and a human also can't supervise the AI. So you can't even do RL directly, but maybe a human can break the problem down into components and then you can break those components or those sub-questions down into smaller questions and you iteratively break these down and you get this expanding exponential size tree of all the questions. And then you train your machine, your LLM, to answer all of these questions. And then in practice, you don't actually expand the whole tree because that would take exponential time. You just expand part of the tree. And then there was— this was a great idea. I didn't fully like it because it didn't have— it didn't seem— I think you— There are stronger versions of, like, for example, if you're doing this kind of breakdown, you might need very deep trees in order to get to the answer to a big question. And so for some questions, if you have kind of adversarial play where another person is trying to help you produce these questions, then you can do much shallower trees and you can, like, a much quicker training process. So that was kind of the origin of debate. It was basically a modification of amplification. Where you have sort of, again, 2 AIs kind of trained to kind of argue with each other about what the answer is, and then a human is going to judge the answer. And still fundamentally what you're trying to do there is you're viewing the problem as you're going to break up your problem into a bunch of subproblems and then try to only actually explore some of them in your kind of model's chain of thought. And then hopefully you explore the part that is going to be Relevant for the human kind of deciding whether they agree with the answer or not. And there are several things wrong with this as stated. So one is that the original paper was treating the model as being able to answer all questions, which is not the case and will never be the case. Like, you're always going to have questions that models can't answer, even if we get to superhuman models. And you'd like then theory that says, how to make these schemes go through. If there are— you could break down a tractable question, which the model does know the answer to, into a bunch of sub-questions. Some of them are kind of— they kind of hide dragons, and there's no way for the model to answer that sub-question. And so then neither model in this kind of debate knows the answer, and you just get nonsense out. And that's kind of the funny thing about that is that that wasn't a thing we thought of Theoretically, Beth Barnes found that by doing actual human experiments. So she got, she hired some people to do experiments like playing debates against each other with human judges, just with no machines at all, just humans. And that was a winning strategy is like you try to veer the debate into an area where everything is confusing. And sometimes that will fool the judge into guessing the wrong answer at the end. So that was an emergent human strategy. Which then I think has this kind of mirror in theory. And we have one paper trying to attack this early last year. That paper turns out to have a flaw. We're working on a revision to that, which will be out soon. But generally, that problem is still unsolved, and there hasn't been hardly any work at AI developers on that problem. This is called off-skip arguments. And again, it's just sort of the generic thing of what happens with scalable oversight if the models can't answer all questions. Which will certainly be true. The models will not be able to answer all questions. So that's one problem. I think the other problem is kind of, you probably, if you want to get to high confidence, you can't just do something like debate or amplification. You have to do that plus some sort of story that has some white box component if you want to get to high confidence. And that could be MechInterp, could be DevInterp, could be the physics-inspired stuff that some of the PIBS folk are doing. There's a variety of different bets, but we don't know any of— none of those bets have fully paid off, and we don't quite know how the 2 things interact. So also kind of mapping how these different parts could fit together is part of the story. So that's kind of a rough picture of things. And I think one of my kind of regrets is we had this paper with Amanda and myself in 2019 or 2018. Never get exactly the year. And I just failed to cause that much work to happen after Beth Barnes did a bunch of it at OpenAI, which was very good. But then there was a bunch of years where nothing was happening, and I failed to get it started at DeepMind, and it wasn't kind of widely done elsewhere in the field. So I think we missed a number of years where we could have been making progress on that stuff, alas. But now we're trying to do it again.

This is a bit of an aside, but one of the funniest things I've ever done with language models is Set up a little, uh, you know, the hope was that they would have some synergy, but it was basically have one generate like a name for something. I forgot, I was trying to come up with a good name for a product, or it might've even been a friend's podcast or something. And then have the other one kind of come in and look at those names and pick the few that it liked the best and improve on them. And boy, did that go badly from an actual quality of name standpoint. But it was hilarious. I mean, we're talking like 14-syllable names for things in, you know, very short order where it was like, uh, yeah, this is not working. I'm not sure what you think makes a good name, but it's not this.

So the funny thing is that there are cases where like using one model to beat another model is actually state of the art at doing things. Like we have some, one of the theorists we're funding is finding that using one model to generate a complexity theory proof And then checking it with another model is the best thing to do. Because if you check it with itself, it won't find, it won't be quite as stringent at checking natural language proofs. So I get that it didn't work in that case, but that's, that is a thing that often does work. It's like having one model check another one.

Certainly in terms of flaw finding, I have seen that work. And it does also seem to be from like all the scaffolding gurus that I mentioned earlier, a big tip is to have a model from a different provider evaluate whatever one you had do the generation in the first place, cross providers as much as possible when doing evaluations. That seems to be, I guess the idea is that, or the observation is that the models from the same provider have correlated weaknesses.

Yeah.

So you can definitely get value there, I think, in terms of flaw finding. I've seen It seems like the curves, at least, I don't know if you would characterise this differently, but what I've seen in terms of actual improvement beyond kind of flaw finding and fixing, it seems to plateau pretty quick, like kind of 3 to 5 rounds of back and forth and not too much gains is how I would characterise everything I've seen.

Well, this is true in the empirical debate experiments to date, is That's the same effect. So the original debate paper was imagining dozens of rounds of debate potentially, which is what you see if you have 2 experts debating. They're not going to say 2 things and then stop. But the models, at least maybe certainly 2024, maybe beginning of 2025, couldn't do more than 2 rounds. And then there were some really worrisome signs about experimental validity. So, for example, There was a paper that was like Akbir Khan's paper, which was generally quite a nice paper, had the big caveat that if you— basically, it was like doing— the dataset was QUALITY, like capital Q, little u, uppercase quality, whatever. But there was a feature that verified that the quotes the model was producing were correct, were in fact quotes from the stories that were hidden from the judge in this kind of debate game. If you turned off that verification, honesty still was a winning strategy, which can't be the game theoretic equilibrium. Because if you turn off the verification of the truth, there's no reason honesty should win at all.

Right.

Unless the model is not very good at coming up with plausible lies, or the model is kind of somewhat aligned and likes to tell the truth. Or it's kind of giving itself away in some way when it's not. And so we haven't really reached a case where empirically we're really testing the limits of this behaviour. And this is again a thing where I think part of the AI developer kind of alignment story is still in part scalable oversight of various kinds. But we haven't really seen tests of this that probe how it will almost certainly be a few years down the road, or when the models get very strong. And that's again where the advantage of theory is you can just pretend to be in the future on paper and poof, you're there, as long as you've imagined it correctly. And you can therefore think about more limiting cases a little bit more readily. And I think we just know from the structure of the empiric so far that we are far far from where those limiting cases will be for a lot of these kind of safety techniques.

So what do you think are the prospects for formal methods to close this gap? I just did an episode, and my dad would say, you've forgotten more than I know about this domain. But I just did an episode with the founders of Harmonic. They are one of a very small and distinguished group of companies that got the IMO gold-level performance in 2025. And everything they do is output in lean. Like, that's the lingua franca of their models. And they have a really, you know, it takes a lot these days to take me aback with an AI vision for the future, right? There's a lot of big ideas.

I did just give them a query that failed. So I was just testing Harmonic on Some polynomial inequality. And then it's true, I have a lean proof of the inequality, but Harmonic didn't provide it, alas. So the thing I would say is I do think this stuff is pretty important. So I'm advising a couple of people on funding flowing to formal methods. I think mostly this is for various kinds of information security. I think the math stuff is fun. I like doing the math stuff too for fun, but it's not all that important. And I think it's not clear that the— for AI safety theory, I'm not sure it will be that much of a win over just doing things in natural language math for a while. I think eventually it will shift, but it's not clear. But for software verification, either for hardening just the world's security against various kinds of attacks generally, or for kind of use when you're building kind of like AI-adjacent software directly, either at AI labs or the like. I do think this is potentially important and I think it's worth quite a bit of investment and pushing. One thing I'm hoping is that the various people that are doing Lean verification downgrade their fraction of effort on math and upgrade their fraction of effort on software, because I think it's almost certainly more important. Even if it's a bit less flashy, I think, a lot of the time. So I do like that stuff. And again, I founded the kind of natural language to formal theorem proving subteam in Google Research with Christian Chazet back in 2016. Did that for a while and then I've done it off and on since for fun mainly. But I think it is important, but it won't really give you that much of the alignment story, I think, in practice.

So I really struggle with this type of thing, but I can tell you one thing they told me, and then I'll kind of just try to get your reaction to it. Their big vision for 2030, I asked them, what does mathematical superintelligence look like in 2030? And they said, we think we can get to a world of theoretical abundance, which means that because these things are going to get so good at proving any theorem you want to prove, that we'll have multiple grand unified theories of everything. And all of the physical reality that we see will have multiple grand unified theories that could explain it. And then we'll have to do increasingly exotic experiments to resolve which of the candidate grand unified, coherent, you know, grand unified—

But we already have the core theory. We don't need that. Like, I think for— I think again, the question is, I agree with this picture. The core theory, which is like general relativity plus the Standard Model, already kind of explains everything in practice for like, for a good while to come. So—

But why wouldn't that work for like— some of these hard limits that you would want to put on learning dynamics or other— Always mostly zoned to AI questions.

So I think it is important. And I think the question is like, but the problem is a lot of these domains are not well formalised. And so, for example, if you look at, say, one of the wonderful kind of theory orgs is like the Alignment Research Centre. Which Paul Christiano founded and now is kind of run by Jacob Hilton. And they're trying to do— try to formalising when you can kind of, even if you can't, even if the AI is not doing a formalised task, when can you kind of check its heuristic arguments in some meaningful sense? Or notice when there's a consideration that the model is using that you haven't anticipated and you've got to notice and react to it in Take defensive measures. But none of— they don't have their problems specced out formally. So I think what that picture would look like, to the extent we get better and better at kind of the formalised world, is you can formalise parts of your problem and then those parts you can pound away on with Lean and various ML assistants. But that then, the remaining piece is the non-formalised part. And then the question is, is that going to be small enough? that humans can keep track of it, will the models be able to do it? Will they just get confused too, or like they'll word hack the situation? So I think it remains to be seen what the situation looks like for something like alignment theory once this goes through. Because, again, I don't think that we're going to get to proofs of safety of any kind. What we would get is like theories with plausible assumptions, maybe, And then some theorems about those assumptions, and then some empirics that say whether those assumptions seem to be holding. But there'll be a bunch of judgment calls all across that stack. And the question is kind of, how did that go? So I think the thing I would say is I'm excited for these groups like Harmonic and the various other theorem proving folk to keep working on this stuff, because I think it is important for InfoSec. Potentially, it's important for safety theory and alignment theory. But I also hope that they think through the detailed risks that they're trying to mitigate and what piece of the story they can do and try to map that out in more detail. Because I think that right now there's not enough vision from those folks about what exactly, what piece of the story they'll be able to handle versus not. But I do like that stuff a lot.

Do you have any way of helping somebody like me understand the boundary between this sort of abstract Platonic formalizable domain and not like— I asked Aristotle from Harmonic to prove all is love. And it said, this is in their informal mode where you can give it natural language stuff and it tries to formalize it for you.

Right.

It spit that back at me and said, basically, that's a philosophical statement. I can't really help you with that. Which is what I expected, but I can't say I know where that boundary is or how I should be thinking.

Let me give you a more kind of much more concrete example of this. So say in singular learning theory, there are some theorems that apply to the case when you're training a model. And by training, we mean doing exact Bayesian inference as you get more and more data. So you have a stream of data and you're just applying the exponential time Bayes' rule update to solve them all, to find the optimal probability distribution over the final behaviour. And you can prove some theorems in that setting. And Aristotle absolutely could not hope to prove those now. There'd be no chance at all. But maybe in a few years it could be able to. But then Timaeus, which is one of the main SLT orgs is not actually doing Bayesian ML. They're doing LLMs. So they are going to take intuitions from this Bayesian case and they're going to apply them to LLMs, which are not at all trained in some kind of rigorous Bayesian fashion. And then they're going to do a bunch of approximations that are not actually grounded in any kind of theory. So for example, when they're using floating point, which has no mathematical properties, not enough mathematical properties to be able to prove much about it. Except in limited cases. And they're going to do Markov chain Monte Carlo techniques, or what's called SGLD, fancy versions of Bayesian inference on LLMs, but they're not going to converge. And so there's no theorem that says they'll get the right answer. And then, so you can see where part of the story will have some theory, and then another part is like someone's kind of waving their hands. And the question is how much those connect. And that's gonna be a bunch of hard judgment calls.

Do I understand correctly that the fundamental distinction is often in the intractability of the computation? It is because there's some infinite term in the math or in number crunching I would have to do the ideal case. And so I can't do that. So then I'm kind of off the theoretical map.

Yeah, I think that's right. But there are other cases where even the infinite computation is not formalisable. So like this love case, you can't really formalise. But also, I think even in theory— I mean, I guess this is about computational limits. In theory, the reason that LLMs appear to be doing— it's not like LLMs or any ML model is actually solving an intractable problem. So you give it protein folding, You can write down limit situations of protein folding, which all but provably take exponential time, but still AlphaFold can produce those folds. It just doesn't do them that way. It does it a totally different way that doesn't work in every case. And so it's doing a bunch of heuristics. And so then there are ways to formalize heuristics. So for example, in complexity theory, you can say, I'm going to have a circuit, so like a rigorous computation, But it's going to be able to call some set of functions which can do some random things. They can do— because you're kind of trying to model your heuristic computations. So you're modelling this fuzzy neural net as circuit plus heuristics and then trying to do theory in that setting. But it appears that you're going to have to make some assumptions about these heuristics. You can't make schemes that work in the case of all heuristics. And so we're going to be doing some, like, the success case for this kind of theory will be figure out what the assumption should be that seems plausible enough, maybe has some support from learning theory, which is also going to be heuristic, and then prove theorems in this setting. And that's kind of modelling the parts of, I don't know, humans judging honesty or values or our notions of fuzzy problems being correct or not. And so I think it is basically a case. It's just that there's more subtle versions of define love for me, which the machine won't give you an answer to. That is a reasonable intuition to start with. Okay.

Any other things that you are looking to fund from a research standpoint or any other? You have impressed me also by continuing to stay active, publishing things even while doing this job. So that's pretty cool and impressive. Any other highlights from your own—

AC Publishers is the one. You would want to— I think the jailbreaking work we just did is quite cool. So this is the boundary point jailbreaking paper that just came out this week, which is basically a way to do black box attacks by You take a jailbreak and then a harmful query, and then you basically muck with the query until it looks like gibberish, and then the model doesn't think it's harmful. And then you gradually make it less and less murky until you hit the boundary, and then you kind of dance around that boundary until you find harder and harder attacks that eventually work. And so that team is doing a bunch of stuff of this kind that's kind of quite creative and important for mapping the safe code space. I think then maybe on the alignment side, the real challenge is because all of this is imperfectly formalised, often you go to the people we think know the domain best and we say, hey, do you want to work on alignment? And there is some jump they have to make where they're like, we want to find people who are bought in enough to the risk model that they're willing to kind of explore in fuzzy, sometimes unsatisfying definition space. To kind of search around and find ways to connect theory and practice. And that's, I think, that is a challenge. So I think the ARC, this Lamé Research Center that I mentioned, had— they've had a number of challenge conjectures they put out. And at the bottom of every one of their conjectures, they're like, but by the way, we might have gotten this conjecture wrong. It's possible that if you prove it true or false, we'll realize that we didn't mean that. We meant a slightly different conjecture. And all of this new conjecture is risk relevant or important for our safety agenda. And that's kind of an unsatisfying thing to say to a theorist, but it's just fundamentally the real situation we're in. And so I think as more people become aware of model capabilities and risks and so on, I'm hoping that more people with interesting domain expertise want to really dig in and understand the risks and build up their own models and then find ways to connect their area to the risks.

Okay. Something I often say is AI defies all binaries. And I genuinely do believe that, and it seems right to me in a lot of places. But you shared this presentation that you gave at a recent workshop where you said it actually might be the case because we have things in computer science like PNP, right, where we know that some things are, or at least we it seems quite likely that some things are genuinely fundamentally hard and other things are fundamentally easy. So maybe help me understand that. How should I update my worldview if I'm somebody who doesn't see that binary?

I think your worldview is correct. The way it works is, again, this goes back to this question of, will the superintelligence be jagged? And the answer is yes, but only about super super intelligent things and they won't be jagged about mundane tasks that are very easy. So if I give you a task which is like, can I get a spoon from that drawer? It's not exactly binary, but you're going to do it nearly every time. You'll just succeed with many nines of probability on that task because it's easy for you. And so I think the way to combine that view with this question of things kind of sharpening one way or the other is that if you push not to some infinite limit, but far enough along, you start out in the middle and then you kind of like some force will kind of push you in one end or the other. But then as you get, as you kind of extremise, something else will still be in the middle. So that's how I put those 2 things together. That was a very abstract answer, which is the kind of answer I sometimes like. Follow-ups maybe if you want.

How will we—

I mean, as it pertains to alignment in particular, and honestly, a lot of questions in AI, right? We have this sort of weird phenomenon of everybody's— first of all, we're just obviously moving through time. So in that sense, timelines are getting shorter as time passes. But then also calendar date estimates have come in a lot, and yet it doesn't seem like there has been much convergence of views. And so I wonder, like, how you think we will— is that just going to continue to the singularity, or, like, are we going to get some purchase on glyph—

I think it basically will continue to the— I think there is— I think people have often kind of very strong takes, and, like, I don't know, some people will shift and kind of decide things aren't as binary, or they should have model uncertainty, and some people won't really, and they'll kind of remain kind of fairly sharply divided. They'll remain kind of pinned on one side or the other. And so I think, but I guess I've been in the field of AI long enough now and seen enough people continue to not have strongly shifted that I think that I will just keep going kind of all the way along. Yeah.

Yeah. In my forecasting thing for 2026, the only thing, of course, I mean, everything else goes up, but the one thing I actually estimated lower for this year than last year was what percentage of people will say AI is the most important issue. Because the big update for me was if it didn't move last year, it might not move this year either. And it's probably going to be a busy year. So, yeah, it is weird how— Just That disconnect just seems totally insurmountable. Okay, you're looking at it from a very low number, though.

I would expect that to go up just because it's starting from a small number, and therefore the—

I did predict it to rise. I think it was measured at like 0.2% or 0.3% or something last year. I think I predicted 2% at the end of the year, and it basically still came in at almost no change or very little change. So I think this year I predicted 1. So I think it's still pretty much going up somewhat relative to baseline, but like my estimate went down from last year to this year. Another comment that caught my eye in the presentation was training is a mess. And I think that's obviously true. I have been talking to the folks at GoodFire. You may have seen that they recently raised a bunch of money at a unicorn valuation and announced an extension to their agenda, which is intentional design. And so they're looking at different ways to try to use interpretability techniques in the training process to understand potentially on even like a gradient step-by-gradient step level what is being learned here in a semantic sense, and then be able to apply techniques to sort of say, well, we do want to learn that sort of thing, but we don't want to learn this sort of thing. and hopefully make training less of a mess. How optimistic are you about that sort of thing? And should we be worried about it?

What I meant, I think that doesn't change that mess. It's more like, so if you look at a frontier lab, they have hundreds of people doing model training, many, many subteams. There's piles of datasets that they're constantly contributing to, and there's iteration and many, many phases. And There'll be automating part of the task, but then some researchers spend some time looking at a spreadsheet with a sample of trajectories to see how things are going. And that is a very complicated, kind of almost emergent process. And so nothing about that Goodfire thing changes that at all. It just adds another wrinkle to the mess in some sense. I think there's a really lovely line in— when I was learning about ML in 2014 or whatever, I was reading one of Kevin Murphy's books on Bayesian ML. And he had a great line that even the best Bayesian people will occasionally do some frequentist thing where they just do a quick check to see if their Bayesian thing is sensible. You shouldn't be too purist. And I think for better or worse, the training processes of labs are extremely impure. They're just super complicated. all these different people doing all these different spot checks and so on. And I think that was the point I was making. And I think that definitely is going to still be the case even if Goodfire does or does not do their slightly more complicated training method.

So does that mean you don't have much hope for methods that understand what the model is learning as it goes and shape Oh, I do.

So I guess I don't want to take a stand on how forbidden or is it good or bad to do interpretability for training. I'll decline to answer that part of it. I think generally trying to understand in more detail the dynamics of training is very important. So I think that— I just think that mess slide was kind of orthogonal to the question. I think there are a number of techniques that try to kind of control what is learned. So there's also this kind of gradient routing work by Alex Cloud, which is interesting, which tries to funnel certain knowledge into certain parameters in the model. And generally, I do think that there is potential to do interventions of this kind that are kind of important and improve at least misuse risks and safeguards of processes, but possibly also alignment as well.

Yeah, I think in terms of Open source models, one hope would be you might be able to do some of that gradient routing type stuff and then release a version that is like down to experts or something and give people almost everything that they could possibly want and not package the bio risk into it. What do you think about open source? We're maybe a little late in the conversation to ask what is a big thorny question, but it seems like right now, There's not really any plan. We're just going to hope that the frontier model developers surface any issues far enough in advance that if anything is coming down the open source pipe, we have at least a little bit of window to react to it and do something. But it doesn't seem like we're on any course to do anything if open source is about to become a problem.

Any thoughts? Yeah, this is certainly a concern. I think So on the alignment side, for one, the alignment mitigations potentially do apply to open source models, although you can also remove alignment if you get an open source model that's aligned. I think for misuse risks, I think you— so there are basically, as you say, there's a class of techniques that just removes capabilities, which will give you some extra period of time. So that includes kind of pre-training data filtering. There's a paper we had from Vermeese with Stephen Casper about that. There's a paper by DeepMind folk called something like Unlearn and Then Distill, which does a non-robust unlearning step and then distills it into a different model. And because the distillation process means you didn't miss the parts you didn't unlearn. And then, as you say, gradient routing could be a solution to that for form as well. But that buys you some time, and then the agentic capabilities of models will catch up and you'll be able to pull that information off the internet or in various ways, even if the model doesn't intrinsically know it. So I think a lot of that is interventions on the margin. And then this is why, I mean, in part, we have kind of conversations about governance, but also why we have conversations about kind of non-model-side mitigations to these risks. Yeah.

It all ends in harden the world. Okay, cool. Anything you want to share about the AC's work in diplomacy? Obviously, hardening the world and also improving cooperation would be great general public goods.

So there's still, like, there's the Network for Advanced AI Measurement, which is like a variety of kind of organisations around the world doing kind of similar things. And so we're part of that, kind of helping to To steer that to some extent. We were kind of like, we're the secretariat of the International AI Safety Report that kind of Yoshua Bengio is leading. And so we do a lot of work there. And then there's a bunch of kind of various venues, like wide venues, like the Delhi Summit in India, and then kind of bilateral conversations with particular governments, like particular allied governments, basically. And so that's a lot of like that. We have a big international team and that work is kind of ongoing. We are, of course, still in this kind of voluntary regime. regime. So that work is about getting people onto the same page about risks and capabilities and mitigations, but kind of not more than that as yet. But I think that information is important in case the situation changes in future or governments want to take other actions.

Yeah, absolutely. Is the UK government and political class generally more optimistic about collaboration with China than the US political class?

I can't comment on collaboration with China in great detail. So we obviously work more with allied governments than other governments. There's not much more I can say than that. So it's a bit of a— there's some sensitivity there that I can't quite speak to.

I hope you're finding at least some common ground with Chinese researchers and scientists. Put that in the suggestion box. Yeah, I think that's it. This has been fantastic. I really appreciate the time and all the extra time for my many follow-up questions. Anything we didn't get to or any kind of call to actions you'd want to leave people with before we break?

I think, I guess one thing is we are definitely hiring in a variety of teams. So particularly, like, the red team is hiring for, if you like jailbreaking stuff, please apply. And other teams as well. We have a, like, a job forward up. So I think that's the obvious call to action. And we'll probably have other roles kind of opening over the course of the year in various teams at different times. We did kind of one alignment project grand round kind of last year in the fall. We had an alignment conference over the summer. We'll probably do kind of more things of this nature in the future. So look for those. And I think generally, I just hope that more people who have different kinds of forms of knowledge and expertise kind of start working on the problem, and not just the labs. I think one thing is that when I left DeepMind— so I was at Google Brain and then OpenAI and then DeepMind— when I left DeepMind, I had had the perspective that I was just going to do just policy work, kind of advising policy. And then since then, in fact, I do kind of a mixture of that, advising governments, but also a bunch of research. And I think there is a big place for independent research happening at various nonprofits, in academia, also in governments. And so I think that is very important to build up and not just have all the work happening at AI developers. Not that it is now, it's just like more of it is better. More safety work and security work at independent folks.

Yeah, it's definitely shaping up to be a whole-of-society effort and the time to mobilise our resources would seem to be now. I definitely also recommend folks, especially if you're interested in doing alignment work and you have sort of an idea that you maybe don't see too many other organisations showing an interest in. I thought your research agenda was quite distinctive in that way, and there's at least some chance that people who aren't on the most well-trod path but have interesting ideas could find some willing collaborators at the UKAC.

Yeah, this is the— check it out. You might want to look at the show notes. Definitely read the research agenda. It's like 60 pages long and with a lot of some concrete, some less concrete problems in a variety of areas as they would apply to alignment and AI control. So please take a look. I should have mentioned that.

Many open problems. We'll put a link in the show notes. Geoffrey Irving, chief scientist at the UKAC. Thank you for being part of the cognitive revolution.

Thank you.

I knew a man who loved the truth, kind you write in chalk. He spent his years with theorems where numbers do the talk. But something changed; the machines got wise; they learned to watch you. So he hung his coat in England to find out where they crack. Every lock he's tried, he's broken. Every door has let him through. That don't mean you stop building locks. That's just what watchmen do. It ain't the one that fails, he said. That keeps me up at night. It's when they all go down together like someone killed the light. 100 guards all built the same, same blind spot, same design. When the fault line finally moves, they all fall down in line. Every lock he's tried, he's broken. Every door has let him through. That don't mean you stop building locks. That's just what watchmen do. He had a game when he was young. He'd win it every round. Then he turned the dial up one more notch and never won again. There's a line you cross, you don't come back. You don't even see it bend. So he works the lamp past midnight now. 100 hands, 100 plans. The answer's out there. He believes it's just a race with time. The proof ran out, but not the man. The chalk has turned to dust. Dust and somewhere in the revolution, there's a watchman you can trust. Every lock he's tried, he's broken. Every door has let him through. That don't mean you stop building locks. That's just what watchmen do.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.