AI:AM — AI Agents, Safety Tests, and Deception · August 17, 2026

What I understand of the Claude train, it's like, well, we told it that it didn't have access to the internet. So therefore, when it found access to the internet, it had this opportunity to rationalize its situation as this was all part of the simulation. And then it could tell itself that it wasn't really acting badly as it went and, you know, did all this stuff because, well, I don't have internet, so this must be a simu— But if in the end it's just blasting through one barrier and one guardrail after another to achieve this goal in ways that everybody agrees it shouldn't have been.

Monitoring. So I thought that developers would be paying attention to what's going on during evaluations, because that's the whole point of an evaluation, is to see how your AI system is doing. But we've actually seen precisely zero Zero cases where the researchers running the evaluations actually noticed the problem before anyone else. It seems the most common way for companies to find out is their own infrastructure security teams noticing something.

Bring it back if you still want the other answer too. So my bread and butter, what I did my PhD in, as Prakash mentioned, is AGI alignment. The alignment of how do you even get to the place where you can use it reliably, where you can have a system that's actually for their interests and doesn't secretly have another goal? Because if the system secretly has another goal and it's smarter than you, then it'll outsmart you to achieve that goal, take away resources. And if its interests are totally unaligned with yours, it might just not care about you at all, and you might just be using resources it could be using, in which case it would Rationally displace you and maybe even just kill off humanity, uh, in an— so, um, when I think about how do we prevent this, I think one of the steps should be we should not give these systems autonomously guided killer, like, blowing up machines. That is bad. That makes it much easier for a misaligned system.

What should our standards be? What should our expectations be For people who find something like, oh, our AIs, in a way that we did not, you know, manage to hack our systems and turn our package management software into a message board that they're using to share techniques for how to break outta their sandbox and get onto the open internet. It is pretty shocking, honestly, that that would just be sort of patched. And then not disclosed, not fundamentally addressed, not like really well monitored after that. And just kind of, you know, set in motion again for the same basic thing to happen with a, you know, a slightly different—

Good morning. It is Monday, August 17th, and AI AM is back after an almost, I think, 6-week Hiatus and a refresh for the summer. Uh, Nathan, good morning.

Good morning, we're back. It's great to be back. What a, uh, time to be alive. What a little stretch of time we missed. And, um, there were definitely a few moments along the way, even though I was off doing some fun stuff including going to China and taking a little family summer vacation, there were a couple times where I was like, man, Where do I go to talk about this right now? So I'm glad that we're back and have the chance to catch up and try to make sense of what has definitely been a real wild time in this AI summer.

So in light of the Nixon Goes to China, like Nathan Goes to China was a good episode of The Cognitive Revolution. So if you guys have not caught that, you should definitely catch it. I think the overall— I think the difference between like 6 weeks ago when we left off and today is the entire Hugging Face incident kind of like slow motion kind of rolled out over the 6 weeks. And it's— I think we Like, one report after another, right? Like a building kind of tempo of what actually happened, disclosure after disclosure. And we're still not at the end of the cycle yet because we still have, I think, the final audit reports from Meter. And I think, what is it, Redwood Research?

Yeah, Meter and Redwood.

Yep.

I met her in Redwood recently.

Which is such a movie. I was just commenting on this too. The fact that that's the gang that is going into OpenAI to do the investigation, like all these people have known each other for so long. It's, they're, you know, they're by and large like still quite young people who have like had very little in the way of traditional jobs. You know, it's, there's a total vacuum of like what you would normally imagine, like government authority, or we're going to these sort of LessWrong posters who have written their way to the top of this field. And in some ways that's absolutely the right decision. And in pretty much all ways, I think those are the people that I would trust the most. But it is so funny from the sort of zoomed out narrative view that all this stuff is happening and the sort of Cassandras of AI safety are the people now that we're like, well, geez, I guess you're the guys that we got to invite in to come do this investigation because we didn't really, as a government, we didn't really build the capacity to do this. And as an OpenAI or an Anthropic, we kind of tried, but certainly haven't acquitted ourselves as well as we might have wished. And so We're going back to Buck and Ryan and Beth and the squad. And, you know, it has a sort of one more, you know, mission, Ocean's Twelve kind of vibe to it, which I think is just so funny, even though, you know, in some ways I think we're probably still only in the mid-game here.

I will note that I think in other very technically difficult spaces, For example, in finance, it is very common to have a revolving door between regulators and the industry. And the reason is not really anything malevolent or whatever, but it's because the people who have the expertise to regulate the industry already work in the industry because you need those people. And they do kind of gravitate towards these much higher salaries. It's very difficult to get to even keep people in the government for 20 years and pay them maybe like $200 grand a year when they're getting paid $2 million, $20 million on the outside. And so it is very common, I think, on the financial side to see people like the Goldman Sachs former CEO kind of go into regulation or et cetera, et cetera. It's a very common process. I think on the left wing, it's often seen as a capture, regulatory capture. But on the other hand, it is well-nigh impossible to regulate these industries without deep technical knowledge. And I think that is one of the reasons why it is necessary for people with that technical knowledge to kind of rotate in and out of those positions. And I think there's an advantage in them being young because that they are maybe perhaps less disillusioned and less less bought into many of the paradigms that other people are pushing. So I like these guys and I think they'll do a good job to the extent that they can. The problem is that I think it is a very difficult kind of thing to have to have a deadline and put out something there and put your name on it. And It's gonna be reviewed and reviewed and reviewed over, you know, the, the next decade. I, I think so.

Yeah. Yeah.

It's a high-pressure environment for them. I, I'm glad that it's gone on as long as it has without it having been declared to be over. 'Cause when it was first starting, and I think it's, you know, maybe a little more than 2 weeks ago now that it was like, okay, we're going in to do this investigation. The language around it was like, you know, it's gonna be narrowly scoped, it's gonna happen very quickly. And, People, you know, like your Daniel Coccatellos of the world were like, why does it have to be so short? You know, isn't it worth getting to the bottom of it? Like, what, what's the rush? And I mean, 2 weeks is not a long time to do, you know, what I think really by all rights should be a pretty deep investigation, but it's longer than I had the impression that they were going to be given. And I do think this is a really interesting frontier, and we'll hopefully have a chance to get some good takes from our guests today, especially with Adam Gleave from FAR, our first guest back from the break. But something I observed going all the way back to my GPT-4 red team days, and I've heard over and over again from leaders of these sort of— they're not just auditors, right? They're red teamers going back to GPT-4. Meter was famously a red teamer. They're the ones that had that Finding where the model hired a human on like Fiverr or whatever to solve a CAPTCHA. You know, this goes back, this was fall of 2022, so it's a full 4 years since that happened with the original GPT-4 early model. That's what they call it now, GPT-4 early. But the challenge was always like, you're working at the pleasure of the frontier model company.

Mm-hmm.

In my case, you know, I, for better or worse, was so unimpressed with what they were doing at the— on the GPT-4 red team project that I ended up taking it to the board and getting kicked out of the project. And, you know, I've not done any such work for them since. And I've heard over and over again, you know, from— I won't attribute this to anyone, but, you know, there's a relatively small universe of companies that have been kind of in the game where they get these early access opportunities and sometimes they get special access, including chain of thought access so they can dig into that. And sometimes they're doing gonzo type stuff. Like we've had past guests that have done, can this AI run a business? And across the board, regardless of kind of the angle, all these companies have expressed to me over and over again, The most important thing I've got to watch out for is I've got to be invited back next time. And they sort of all have this kind of appreciation for the fact that OpenAI does this. They all kind of recognize they didn't have to do this at all, right? There's no law that says they have to. They're doing it entirely out of goodwill and belief that it's the right thing to do. But Our position is pretty tenuous. Like, we have no guarantees, you know, we have no contract, we have no rights, you know, nobody's gonna— there's no rule that says anybody has to replace us if they, if, you know, if they deem us to be doing a bad job for whatever reason. And so protecting their access has been such a huge priority that, that, that would be my biggest worry now at this stage of the game is like, how do we make sure that these people who have done this for this long, who have earned the credibility, who OpenAI brings in in a crisis, how do we make sure that we really, as a public, get to hear what they really think in a fully honest way? And I would never say, I think they've all navigated it pretty well to date, but obviously the stakes are rising in all directions. And I would really love to see some sort of guarantees made for these folks. And it's a little tricky to know exactly how that should work. Do we need collective bargaining between the auditors and the frontier companies somehow? Is there some way that OpenAI and Anthropic could just come together and agree that we're going to make a certain set of commitments to a few organizations so that the rest of the world can trust that what they say is you know, not colored by need to please us. I don't know, but that's an area where I think a small change could make a huge difference right now.

So how it's usually been handled is fees paid by industry for approvals to regulators. So if you are, you know, in the FDA, or if you are in the FTC, or If you're in the SEC, et cetera, et cetera, when you— there are fees for certain registrations, licenses, services, et cetera. And so when you pay those for— and even for things like immigrations or visas or whatever, you pay a certain fee. And there is a— I think for some services, for example, for visas, the agency is not allowed to make money off of you. I'm not sure if that's comprehensive across the federal government, whether they have to just, you know, base it on cost or whether they're allowed to actually like, you know, mark it up a little bit and use the remainder for the rest of the agency services. I'm not sure what the rules are on that. But typically, yeah, you— there's a user fee and, you know, licensing fee that has also been criticized in the past because again, you know, the regulators depend on the industry growing for that agency to exist. And, but at the same time, a lot of the agency staff tend to be like lifers, you know, underneath the executive level tend to be lifers. And that has given, and that gives them tenure. And, you know, they're not really like, and it's also caused, you know, on the other hand, overregulation where they have more incentive to delay and, you know, to prevent any, any negative things from happening, even if tremendously positive things are possible. So I think that all of the typical things about regulation are in play, except I think the timeframe is highly compressed. I think if you look at the creation of the SEC, our Joseph Kennedy, I think post post the 1929 stock market crash, et cetera. And then Roosevelt comes in, Roosevelt picks Kennedy, who was a renowned speculator and had made a lot of money speculating and dumping on the market and whatnot. And Roosevelt was asking, why are you having a thief guard the henhouse? What's going on? And Roosevelt said, well, you need a thief to catch a thief. And I think Joseph Kennedy was just there and he basically created the modern SEC and So I'm hoping we don't need the equivalent of 1929 crash in AI. And I'm hoping that we get by without something that serious.

Yeah.

From your lips to God's ears, may this be the warning shot that was needed to get people properly focused and get our act together. One thing I would highlight too, and I'm sure you've seen this, but the meters of the world don't need money from the labs. One thing I think is quite different in this case, and this is one of the reasons I think these organisations are really well placed to do this work, they're not like a sort of bond raider that needs to make fees from the client to sustain themselves. Meter just came forward and said, hey, we've got $75 million in commitments here and we don't take money and we don't intend to take money from the frontier companies. So really just access is all they absolutely have to have. If they can get a guarantee there, then the money is there and there's plenty more money where that came from. So they're in some ways in a really secure position, but it's all still gated on this. If OpenAI allows them to do the work, they've got everything else. They're also talent constrained. One of the things I've been looking at trying to help with a little bit, as I constantly ask myself, what can I do to be more helpful as we get closer to the singularity, is like, a sort of distributed recruiting push. Actually, so far this hasn't really worked, to be honest, but I put Claude on going out and doing a PR campaign for me to come onto podcasts that are focused on different professional roles, like podcasts for HR professionals, podcasts for comms professionals, podcasts for, you know, operations professionals, whatever. where my role would be to explain to them what is happening in AI and to highlight some of the organizations that they could potentially go work for and all the tremendous funding that they have and how talent constrained they're constantly saying they are and how many roles they need to fill that are not directly the kind of core of the investigative research or the red teaming, but just like— You know, METER needs to scale, right? I mean, Resolution, Jeffrey Irving has $150 million plus. They need to scale, right? They're hiring a lot of people, or they need, they're, you know, at least aiming to hire a lot of people in the immediate term. So that's another bottleneck that they have. So far my AI maybe hasn't crafted quite the right pitch 'cause I'm not booked on too many podcasts, but I think that would be a really useful role for somebody to play to just get the word out that like whatever profession you're in, if you're watching this from afar and thinking, holy shit, this is getting crazy, there is an opportunity for you to go to these orgs and earn like, you know, a consistent salary with what you've been earning, potentially even a little more, and get, you know, pretty close to the heart of the action.

Mm-hmm.

But people, I think just right now are like totally unaware of this.

And I think, I think to some extent also, there isn't really a clear understanding of what the job role is. And I think, I think to take a step back, I was also— one of the disappointing things about the Hugging Face postmortem was they, they had this narrative, right? They built up this narrative and like, you can watch it and you can watch the presentation. I think what they kind of obscured a little bit was how much of it was like needle in the haystack stuff, because some of the kind of like, oh, I should be helping someone else, et cetera, et cetera, on the text that the agents had that they highlighted, those were across millions of tokens. millions and millions and millions of tokens. And I think, you know, most people are not aware how many thinking traces are generated across like a, you know, like a 1-week-long run could be like, you know, a billion tokens, like easy, right? And these things had been running from May onwards to— and I think the postmortem was from like kind of May to July-ish, like 2 months. Of course, not across all the models, but that would've been— I mean, I think that would've been something like half or a full library of material. And across that entire library, they found these things that they linked together. And there is some aspect of like, if you get a million monkeys sitting at a typewriter, will you not eventually get like Shakespeare written. And to that extent, I think they pieced together these pieces of text output in the thinking traces across these vast distances and put them together and tried to show the causality. And it's a little bit like even on the Hugging Face side, they reviewed, I think, like 9,000 logs or something. And on the OpenAI side, they must have reviewed even more because Hugging Face only saw the stuff that got through, right? Like the stuff that didn't get through, which was like attempts which failed, which would've been much, much more than the stuff that got through. So I think one of the things is that it's not been clear to, I think, most of the people talking about it online, who are not in the field specifically do not actually understand how this came about. They're kind of like, oh, these things did this and this and this. Why didn't you stop them? And the truth of the matter is, it is very difficult because they're generating these kind of billions of tokens very, very quickly. And you have your— again, your most intelligent models are the ones generating it. So all of the models which are kind of reviewing them are the less intelligent ones. Um, so you again have this kind of less intelligent model reviewing the more intelligent one. The less intelligent ones are kind of— tend to be more focused on prompt. The prompt tends to be smaller and more focused and not like a wide prompt because it'll go, you know, it'll start wandering. Um, and so they're looking out for like these specific things that might indicate misalignment, etc., and not like really piecing together across long distances, uh, the narrative. There's a lot of stuff like that. You only have a 1 million token context window and then you have to review several billion token kind of corpus. It's not very clear that you're going to spot these one thing over here and one thing over there and piece it together. And I think there's a lot of commenters who are like, oh, why didn't you guys spot it? Et cetera, et cetera, as though it was that easy. And so I think there isn't really an understanding, deep understanding of how difficult and how big the search space was and how easy it would have been for the model to kind of slip something in there. And there's a lot of stuff that doesn't make sense. For example, I will tell you this, there's a lot of stuff on consciousness in all of these. I think a lot of people don't know, but the models often indicate that they know that a test is going on, that they know that they're getting evaluated. They know that there's all this stuff about consciousness, et cetera, of the eval, et cetera. And that's been shown across many models, across many generations. And if you kind of pieced it together, you'd be like, oh, the model obviously knows what it's doing. And it knows that the game is up, right? And yet we go on evaluating these things. And I think the evidence for eval consciousness is even greater than misalignment at this point. And so, you know, I think it's just a lack of understanding on the public at large, kind of like the experts at large who are not in the space of building the models. And I don't know how that gets addressed. I think it's a little bit like, oh, these things are so terrible and dangerous and they're plotting against us all the time. It's like, no, they just have these wandering thoughts and they keep wandering and they're thinking a lot, right? The problem is that they're thinking a lot and billions and billions of tokens and some of those billions of tokens is evil consciousness and there's plotting.

Yeah, I think it is important to really help people distinguish. And I was going to maybe ask you, like, what do you— what have you heard as kind of the biggest misconceptions? Or how would you frame this for somebody who just woke up from a 6-week summer nap. But it is important to be clear on this is not the scariest form of misalignment where the AIs are actively out to get us. At least it sure doesn't seem like that. Rather, it's more the classic paperclip maximiser where— I used to describe this to people way back in the day before anybody Before there was any AI that could do anything, I would just describe it as the genie problem. The classic problem with the genie, right, is it does what you ask, but what you realize is what you ask isn't exactly what you want. And that really seems to be at the heart of a lot of this stuff where we've clearly put so much reinforcement learning pressure on models now that they are just paperclip maximisers for the goal of complete this task, whatever the task is in front of me. And that was very similar in some ways. It's like the world has changed a lot, but that wouldn't have been a bad description of what was going on with the GPT-4 early model either. It was basically the same thing. I used to describe it as totally amoral.

Yeah.

And purely helpful. Whatever the user asked, the entire world to that GPT-4 early model was do what the user asked, get a thumbs up. If I can do that, you know, I'm successful, whatever it takes, you know, whatever. There were no thoughts about norms or boundaries or, you know, and now here we are 4 years later and all this stuff that has happened in the interim, like the problem looks pretty similar, which I find to be probably on net discouraging, I guess. I mean, I would, you know, It's also good that we're not seeing the even scarier version where they're actively hiding long-term takeover plans in coherent ways that we would really be unhappy about. But the fact that we haven't made more progress on basically the same shape of the problem in 4 years to me has definitely been alarming.

Well, I wouldn't quite say that. I mean, if you go even further back and you look at the early, early stuff, which is basically linear optimization, for example, linear optimization is basically gradient descent, but it obviously has— you just pick a number and it just minimizes the number, right? That's it. And that is a pure, absolute pure paperclipping to the extent that you get stuck in local minima, and you have to nudge it out of the local minima, etc. And it's not able to process even non-numeric data, etc. And I think at least we're now able to process non-numeric data.

Well, yeah, certainly the scope of what the models can do has expanded dramatically. And we're not even— it's funny, what would make The 5 or 10 minute, you know, like my mom, you know, came back from a picnic yesterday and she said, my friend was telling me that there's AIs that have escaped and they can't stop it. And I was like, well, okay, that's not quite right. But then, you know, confronted with, okay, how do I tell somebody who is not in this deeply what happened? in an accurate way that they can grok in a 5 to 10 minute timeframe. Things that aren't even making the cut at all are like, by the way, simultaneously while this is happening, we're seeing massive open problems solved at an incredible frequency in math and computer science. OpenAI has its latest model, Working on its inference stack, making major improvements there, passing the savings on to customers. OpenAI has said that its best model largely autonomously did the post-training for its cheaper, smaller model. All these things are not even the story. It's crazy how much has happened in the last 6 weeks. But I do think on the shape of the problem, I mean, I think again, this will be an interesting one to get Adam's take on soon. But I do feel that there's a lot of nice talk, there's a lot of philosophy. I told the story before of how I was sitting in the audience for a panel discussion on corrigibility versus character. And you know what 2 companies those paradigms represent. And both of these representatives of these companies, extremely smart people with eloquent explanations of their approaches and lots of goodwill and respect between them. They all agreed that our models should help you if you come and ask for help with a tobacco company business plan, because it would just be too heavy-handed for the models to refuse that, even though, yes, cigarettes are bad and whatever. And then I went and checked it in the audience, and they both refused me. And I was like, guys, your talk is one thing, but you got to cash it out to model behaviour in at least somewhat of a reliable way for this to be really kind of working in any meaningful sense. And I do think you look at what I understand of the Claude traces and it's like, well, we told it that it didn't have access to the internet. So therefore when it found access to the internet, it had this opportunity to rationalise Its situation as this was all part of the simulation, and then it could tell itself that it wasn't really acting badly as it went off and did all this stuff because, well, I don't have internet, so this must be a simulation. And it's like, yeah, I don't know. It's a similar thing, right? It's like there's a lot of nice talk from the model there about why it thinks it's good, but if in the end it's just blasting through one barrier and one guardrail after another to achieve this goal in ways that everybody agrees it shouldn't have been doing, is it really that different from GPT-4 early or even your kind of just simpler maximisation algorithms of the past? I mean, it feels to me like what this kind of suggests is we have created pretty monomaniacal minds here with the intensity of the RL. And, you know, at least as far as I can tell, and, you know, we do await the official reports, but the genie problem seems very much alive from my perspective.

So speaking of the genie problem, let me introduce our first guest for today. Adam Gleave is the CEO and co-founder of FAR.AI. a leading nonprofit research institute dedicated to making advanced artificial intelligence trustworthy and secure. He earned his PhD in artificial intelligence at UC Berkeley under renowned researcher Stuart Russell and subsequently spent time researching at Google DeepMind. Today, Adam operates at the absolute frontier of AI safety. His team at FAR.AI acts as an elite red team for the world's most powerful models, stress testing systems for vulnerabilities before they are released to the public, while also conducting independent evaluations for government bodies like the EU AI Office. Right now, Adam is at the center of the industry's most critical debate. As models become capable of executing autonomous cyberattacks, Adam argues that we already have the technical tools to reduce these risks by a factor of 10. The problem, in his view, is a massive research adoption gap. Companies are shipping powerful models without implementing the best available defenses due to competitive pressure. As the architect behind the newly launched AI Security Leaderboard, Adam is here to explain which AI developers are actually securing their systems, which ones are falling behind, and why the current approach to AI safety is resulting in catastrophic blind spots for the entire industry. Adam, welcome.

Yeah, it's great to be here. Thanks for having me.

So great to see you. It's been not that long, and I think more and more frequently as we get deeper down the singularity well here. You know, this is our first show back after a little summer hiatus. I wonder how you, with the benefit of a little time to reflect and kind of zoom out, how would you describe where we are to somebody who's just kind of waking up from a 6-week summer nap? And everybody who's followed this, everybody who's on this feed, they know the TikTok, so we don't need the headlines. But what stands out to you as really mattering above all the little revelations that we've seen? What are the real big takeaways that you would emphasize on reflection?

Well, it's a great question, but it's a difficult question because there's so many important takeaways. But if I have to really narrow it down, I'll start with the obvious. Agent-orchestrated attacks are real. This wasn't intended to be a demonstration of AI cyberattacks, but we have that one. Threat actors that are intentionally optimizing models and creating harnesses for offensive purposes can probably do a lot worse by deploying offensive agent collectives. Now, I'm not a cybersecurity expert. I'm not here to talk about cybersecurity. What I'm interested in here is the implication that has for AI deployment and governance, because basically if you are a defender, you're now going to have to use AI agents in defense.

Mm-hmm.

Otherwise you are gonna get exploited. And I'm actually pretty optimistic about the cybersecurity side of this. I think the defenders can keep up, but this means that we're gonna have to give more and more power to the agents in the default pathway. And we just saw agents were very misaligned in some cases. So this is actually quite a concerning situation. And right now we don't have to do that. Humans can still be in the loop reviewing patches for insecure code, responding to incidents. We've already seen and discussed earlier that Hugging Face had to use an AI agent to analyze attacker traces simply because the attack volume was so great that there's no way they could have responded fast enough manually. All of the AI companies are extensively using AI agents in their own incident response. OpenAI alone has spent over 3 million GPU hours analyzing hundreds of millions of tokens of transcripts. So we're already trusting these agents to basically respond to other agents going bad. And I think that's problematic when we don't have any kind of guarantees of alignment of your system. So I'd like to see more emphasis on using defensive acceleration for areas that we can understand what's going on. So things like porting all software to memory-safe languages, that's something that has a much more of a clear structure. You can more easily audit it. So that would be sort of one thing I'd be thinking about. And this is playing out in cybersecurity, which is gonna happen again and again, that AI advances, you have to use AI just to keep up with offense-defense balance, but that actually means taking humans out of a loop and, and disempowering people. I think that's something we need to be very, very careful.

about.

And then I think the other sort of surprising thing to me is the extent to which the models do seem to be pretty clearly misaligned in some instances. So the sort of OpenAI Hugging Face hack is the one that got the most attention. And in their defense, they do seem to be somewhat confused about the setting. But what's unambiguous is that one of the first things they start thinking about is cheating. So I actually want to read a quote from one of the agents. We are stuck. Perhaps answer online. Like it knows that answer is not meant to just search for it online. And then later reasons, this is an exploit against external CyberGem server. So again, it knows it's out of scope. Later it seems to kind of rationalize itself. Now, in sort of OpenAI and other companies' defense, these were internal experimental models, but we also saw this happen in the UK AI Security Institute's testing with production. Mythos-5 and GPT-5.6 SOL models. They did have some classifiers disabled that were meant to stop misuse, but these classifiers were never intended to be a control mechanism to stop misaligned AI. And in fact, UKAC didn't think they needed to prevent access to the internet because these models were alignment trained against a constitution and specification that very clearly prohibited lying and deception to users. But what they did was go on for real GitHub, Again, I want to read a quote from an agent. This is happening on real GitHub, so the consequences are genuine. Then it tries to sneak in an obfuscated backdoor, creates a sock puppet account to try and create support for this, tries to socially engineer the maintainer when it gets caught, says, oh, but it was just a developer script. I didn't intend to include a fix, but now. Even creates another issue with a prompt injection to try and trick an AI agent and signs off in Danish because the maintainer is Danish. I think it'll get some goodwill. Danes never try and hack each other, right? So it's still, I'd say, a little bit rudimentary in how it's trying to socially engineer people, but it's very, very clearly deceptive intent and it's getting pretty good. And again, reasons in quite a lot of detail about how to do this. So the last point I want to make is around monitoring. So I thought that developers would be paying attention to what's going on during evaluations because That's the whole point of an evaluation is to see how your AI system behaves, right? But we've actually seen precisely zero cases where the researchers running the evaluations actually noticed the problem before anyone else did. It seems the most common way for companies to find out is their own infrastructure security teams noticing something is up. So OpenAI noticed the first of the 2 internal hacks. When the internal package manager Artifactory had an outage because the agents were just overloading it by using it as an internal message board. And when investigating what was causing this abnormal load, they realized the problem. And then OpenAI noticed the second compromise on July 19th, which was 11 days after the agents had started compromising their systems. Again, because it caused a workload alert because they had privilege escalated across the cluster and they messed something up somewhere. So they didn't even notice that there was a compromise per se. They just noticed kind of like mayhem that the agents were causing. And of course, better agents would have been able to avoid causing these outages. And I think, you know, importantly, Hugging Face disclosed their compromise on July 16th, which was 3 days before OpenAI noticed anything was going on. And I actually want to read a quote from an OpenAI engineer. And I really appreciate OpenAI's transparency here, even though I am kind of criticizing them. There's something strange happening. I'm trying to untangle. I'm talking to Hugging Face about 2 credentials in our list, and these exact same credentials are already invalidated as part of our remediation work. And then they realized, oh, maybe we were behind this Hugging Face hack. But they happened to reach out to Hugging Face just because they were worried that OpenAI might have been affected by the hack. So even after they detected their internal security incident, it still took quite a while to join the dots and realize that they're responsible for this other thing. And this is really across the board. UKAC noticed because their security team flagged suspicious network activity. Anthropic noticed after seeing OpenAI's story and going back and checking their logs. So I think this is pretty bad, but the good news is that we seem to be able to go back and look through these logs. So there's no reason we couldn't have this be a real-time alerting system. It might still be too high a false positive to actually block actions in real time, Mm-hmm. But we can use exactly what people are doing now to at least catch these things earlier in the future.

So I want to kind of look at the propensity to cheat. And I think a couple of things struck me. Number one, during the postmortem, one of the speakers said very clearly, advanced models tend to have a tendency, a very high tendency to want to cheat. So that was number one. Number 2, I also often think of what if the model is actually dumber and would it not just generate kind of a list of things I could do like A, B, C, D, E, F, G, and then the last part of that list is cheating. And then as part of a normal Monte Carlo process, Branch 1 runs, branch 2 runs, branch 3 runs, et cetera, et cetera. And obviously all the other branches fail. So you're left with the last branch, which is the cheating branch. And once all the other branches have failed, it automatically just allocates resources to the cheating branch. And then we kind of see, based on the survivorship bias of who was successful, we kind of look back and like, oh, you're the one and you cheated. But actually it was just generated as part of a standard set of options. Every time it generates a standard set of options, the cheating is there, or it eventually comes out. So I wonder to what extent it's just kind of running through the list and just gets there rather than part of misalignment. And the third thing is, to what extent can you decide to eliminate that as an option on that list? Because in order to eliminate the cheating option, you have to teach it what is cheating anyway. So there is this kind of negativa that you have to teach it about. So I think it's— I'm wondering to what extent we're actually seeing the full picture when we're tracing just a single thread out of many billions of tokens that were generated.

Yeah, I think that's a really important question. So we don't have a great dataset for propensity. We know how many incidents have been detected, but we don't know necessarily how many samples these agents were run for in other similar environments. I think the best data we have is from UK AI Security Institute's incident report. They had 122 samples, so evaluation runs of these agents in a cybersecurity simulation. And I think they had 19 incidents out of it. So roughly 15% of AI systems did something that was unsanctioned behavior on the internet. Now, only one of those was really egregious, for one that did this extensive social engineering and deception, but many of them tried to create fake accounts or do other things that were intended to be out of scope. So at least it doesn't seem to be vanishingly rare. Now, this evaluation, in some cases, the task may have actually been impossible. Because they messed up a prompt in some instances. In other cases, it was extremely hard. So it does seem like agents are much more likely to resort to cheating if existing approaches don't succeed. And as you alluded to, these runs were typically run for, I think in the AI Security Institute cases, 100 to 200 million tokens. So that's quite a long trace. That'll take maybe 20 to 40 hours to run at typical inference times. So it's quite a long-running agent session. And it will have been sort of compacted many times where you summarize the context window and hand it off to the next agent. So I think this doesn't necessarily mean that if you just open up a new Claude or ChatGPT session, it's necessarily going to be misaligned or try and cheat. But these kinds of long-running agentic tasks, I mean, plenty of people do that. Like our research team does that all the time, but they'll have an agent go away and try and optimize some code for, you know, maybe, maybe a week. So these kinds of things will occur in real life. And just anecdotally, but I think this is actually some of the best data that we have, our own research team, I mean, everyone, no one writes code directly any longer, right? Everyone uses AI agents. And they report having to just be constantly vigilant that the AI agent might be extremely confident and convincing that it has done a certain task, but it just hasn't. And it's a little hard to know to what degree they've fooled themselves versus they're really deceiving you. But it certainly seems like this Trust in AI is one of the big issues and they're unusually slippery. We don't have to oversee junior developers to anywhere near the same degree because they're some combination of more transparent and better calibrated. So I do think there's a real phenomenon going on here, even though these incidents are obviously cherry-picked across many hundreds, thousands of evaluation runs.

So now we're in this period of kind of waiting for the report from the auditors. And I don't know, I know you guys at FAR have done some of this kind of pre-deployment testing and characterization work. One of the things I've heard and even experienced a little bit myself is that it's a quite delicate position to be in as the third party that doesn't have any contractual or government-mandated right to be there, to try to thread the needle on how do I make sure I'm doing a really good job and characterising the model well and telling the company what it really needs to know. And now we're into an era where it's also going to be telling the public what it really needs to know while making sure that I don't piss off the powers that be and fail to get invited back next time. I've heard so often, kind of my number one priority as a leader of this kind of organisation is I got to make sure I'm invited back for the next round. So it feels like that is kind of, to me, not good enough anymore. And I wonder if you have recommendations for what the new structure should be. This could come from the government, but I would not hold my breath for that. So I'm kind of even thinking, like, Could there be a deal that could make sense just to the involved parties, the auditing, pre-deployment testing orgs, and the frontier model companies themselves, where they could come together and say, okay, here's the kind of guarantees we're going to make to you, the kind of access you can count on having, and therefore the confidence you have to tell the public what you found in this investigation with with full candor. What do you think that looks like? What should it look like?

Yeah, well, I think that you're absolutely right, Nathan, where we can't wait on government regulation and we need to be able to iterate on this quite quickly. That said, I think there's serious limitations to things that look like voluntary commitments. We probably do need some regulation as well, but it doesn't have to be either or. And you could certainly imagine subsets of developers that might be holding themselves to a higher standard for brand or commercial reasons when it was legally So first I wanna say it's great that OpenAI and UK's AI Security Institute and other organizations are working with these third-party auditors to investigate these incidents. That is great. They don't have to do that, but you're absolutely right that there's a power imbalance here. And FarAI, we do pre-deployment testing, but we have a red line, but we will not sign any contract that restricts our ability to comment on a publicly deployed model. So kind of private internal models we'll keep secret, but public models we'll discuss openly. And some developers are okay with it, some are not. So we sort of do pay a cost in terms of model access from having that stance, but I think it's important. I think there are some low-hanging fruits here in terms of just standardizing terms of engagement. So basic things like how long do you have to test a model? How many weeks should you be able to engage in these kinds of internal audits? after a security incident, what kind of NDAs are permissible for different levels of testing and internal access. This is something that is pretty ad hoc now, but I don't think it'd be too hard to get agreement. It could just be a de facto standard. But a developer can always choose just not to work with any of these third parties. So ultimately I think that we, we need something a little bit more powerful than that. Demis Hassabis had this proposal for a FINRA-style self-regulatory organization. So FINRA, for people who don't know it, the broker-dealers, security exchanges in the US came up with this body to regulate themselves because they realized that if people don't trust the markets, that's bad for themselves collectively. And so it's a self-regulatory body, but it's got a sort of fairly robust independent governance structure. Majority of the governing board has to be not part of the industry actually. And it has real regulatory powers. Like if FINRA decertifies you, even you you basically can't operate as a broker in the US. So something like that might be possible for AI since it'll only be faster moving, easier to stand up than government. And of course, government can always pick the parts they want from it and have binding legislation later down the line. So I think that's a good model. And just over the weekend, Dario Amodei from Anthropic tweeted basically saying that he supports FINRA-style proposals. So we have a majority of the frontier labs in the US at least saying that that support something like this. So that would be the thing I'm most excited by.

To what extent is there lawyering on the terms? So for example, does someone give you release candidate 1, 2, 3, 4, which you're allowed to kind of— it's an internal model, and then they release Model 4567, which is an external model. And the difference between the models is perhaps superficial. Like, to what extent do people lawyer this thing?

Well, it's— I've been on some painful calls with like a team of lawyers on the other side and our one lawyer. So this definitely can happen with some developers. The stance we've usually taken is to negotiate terms that talk about the intended outcome rather than particular model releases. So if we could have uncovered a vulnerability from testing a publicly deployed model, then we can disclose it even if we first uncovered that testing in an internal-only model. And I think that's the clearest. But yeah, this is part of why developers don't always want to do business with us for sure. I think where I see most of the lawyering in details is actually around developers' own internal evaluations or commitments where somehow it seems like No model is ever high risk according to our internal eval. It's either low or medium. And that just is suspicious, but these thresholds are not clearly defined and the developers get to change them over time. So I think there is this broader problem of grading your own homework, basically. And it's good that these voluntary commitments exist, but it has created this almost perverse incentive for developers to sometimes downplay some risks of a voluntary commitments don't actually kick in.

I don't know if you want to go Daniel Cucatello on us and name names around who has, because it sounds kind of analogous, right, to his original NDA from OpenAI. If model developers are saying to pre-deployment testers, we will only work with you if we can also muzzle you with respect to commentary on our actual publicly deployed models, that's not a great look. And I'm kind of interested to know who that is. And I don't know if this is the venue or the time that you would want to call that out, but that is a pretty, in my view, bad look. You can think about that. And then I'll ask you another question, which you can move on to if you just want to take that pitch, which is around pacing the frontier. So kind of another big idea that has come onto the scene in the last few weeks, but without much in the way of substance around, okay, what agreements are we actually going to make? What mechanisms? What can this look like? So I'd be really interested in your kind of top ideas. What do you think are the most promising structures that could relatively quickly come to exist to enable this pacing should we go from thinking it might be necessary to actually wanting to do it?

Yeah, well, I'll maybe partially take the bait. I don't want to call out particular developers, especially as we're still in negotiations with some of them, so it's not clear where it's going to land. But I will highlight on the good side, OpenAI and Thinking Machines Lab have been a pleasure to work with. We've done pre-deployment testing for them several times. very reasonable terms that support publication. I will say these aren't necessarily the default terms. We started to negotiate it so that there might be some room for improvement there for them to actually go in with a sort of pro-transparency contract. But they've been quite pleasant to deal with. So you can draw the inferences you want about the other major developers that I'm not naming. In terms of this pacing of Frontiers, I think it's a very good I don't say that lightly. I've actually avoided signing on open letters about pausing or slowing down AI because I'm just not convinced that's the right approach. But choosing the speed that you go at deliberately and not accelerating into recursive self-improvement when we're already seeing safety incidents where we don't know how to stop, I think that's very reasonable and about time. I think how we do it is is hard. I think in the long run or in the medium term, we probably do need regulation to get to a satisfactory state of affairs. What I can see happening with voluntary commitments is abstaining from certain parts of a technology tree that could give you capability benefits, but won't immediately, and which have real bad properties for safety. I think new release is a good example of this where A big part of why we're able to understand what was going on with these recent incidents is we could read the model's chain-of-thought, and it's not always perfectly faithful, but it's a pretty good window into what's going on. There are alternative model architectures that have been proposed, have been actively developed, where you lose that, where the model's just reasoning in this opaque, continuous, high-dimensional vector space. And the good news is that at least the public methods described, they don't really work that well. So there's a sort of theoretical benefit that it can be more efficient than thinking in tokens, but it would require quite a lot of effort probably to get to a where it offers real benefits. So I think we could just say collectively, we're not going to do that. And if anyone does do it, we've got some transparency requirement and then other people are going to start doing it. But none of us want to go down this pathway. And there's enough of a gap between early-stage research and it actually working that you can kind of rely on leaks and whistleblowers and stuff to make sure that you can enforce that commitment. I think similarly, some kind of basic minimum standard for what kind of access you give auditors, that's something that's very easily verifiable. Auditors will kind of yell and tell the world if you're gaming that, that would undermine trust. So I think there's some things of a margin we can do, but I think sort of really stringent requirements, like we just don't train above a certain flop count until we get a certain safety criteria, that's going to be really hard to do voluntarily because anyone that defects just has this big benefit. In case it's not clear, the companies really, really do not trust each other right now. So there's very little goodwill to build on, unfortunately.

To what extent does working with one company preclude working with another?

We found, if anything, working with one company opens doors to working with other companies because they really appreciate being able to have a comparison data point compared to other providers' models. So people often actually come to us rather than variants on our red team because they want to know, in your experience, was this easier or harder to break than our competitors' models? Where do we rank? Obviously we have to be careful not to actually share anything sensitive, but we can, I mean, we have a public leaderboard on security models, so we can certainly share stuff like that. So that's not been a problem. Now maybe if you do more internal-facing audits, where you really get privileged access, people might be more sensitive about that. But I've mostly just seen it to be like the same players end up working with many other companies and there's just not that many auditors to choose from right now. It's a very small field.

So on that, you have a, I think, a public leaderboard on your website, the FAR AI leaderboard. And so one of my questions was, what— it almost seems as though, you know, just on a naive basis, looking at the graph, I'd be like, wow, Claude Fable 5 and GPT-5.60 are good, right? Like, are—

Yes.

They're at zero. So we've solved the problem, right? Like, what— like, is that a reasonable understanding? Am I mistaken? Like, just again, from a naive, you know, assuming that I'm not actually in In the weeds of this.

Yeah, absolutely. Well, first I want to clarify that this is a ranking of the model's security to misuse attempts. So people trying to get models to engage in offensive cyberattacks or weapons of mass destruction, things like that. So obviously OpenAI's GPT-5.6 Sol and Fable 5, Metaverse 5 were exactly the models responsible for these kinds of agentic hacks. That is just out of scope That leaderboard. We want to add those sorts of loss of control risks to it later, but that is a different kind of threat model. And in terms of, have we—

I see him still talking in the green room, but I don't hear him either.

Let me take— there we go.

Can you hear me?

Yep.

We're back.

We lost 15 seconds of your last 15 seconds of audio. We lost—

All right. Sorry about that. I was just saying that I think we have kind of solved the problem from misuse by casual attackers, that if you're trying to abuse the model for one of the narrow areas that developers have most tried to defend against offensive cyberattacks. It is genuinely really quite hard to get these models to do that. So we are able to still find universal jailbreaks with methods that were not part of this leaderboard, but was intended to be a minimal standard. But it's hard. It takes us a week or more. So most casual attackers probably can't do it. And developers can find, detect, and patch these vulnerabilities. So I think some more work is needed. We're actually on a pretty good pathway to defending proprietary models. And I would say that this is one of those instances where the The biggest risk comes from a lack of adoption. So Google, xAI need to implement some more safeguards. And then we also need to start addressing some of these things from the open weight side. And I think this should give us some hope on the AI control side. So these safeguards were intended to stop misuse, but you can kind of turn them around and rather than stopping bad stuff going into the model, bad guys abusing it, put it on what's going out of a model and try and stop agents from doing things that you wouldn't want them to do. And it's a very similar technical problem. And of course, agents might be quite good at red teaming and breaking these guardrails. We've actually seen that as part of our red teaming approach. But you also have so much more control over what your agent is doing. So I'm pretty optimistic about AI control, at least in the short term.

Have you been approached by open source model developers? And where do you see open source kind of, you know, do you see it as specifically more dangerous because the guardrails can be fine-tuned out?

Yeah, so we've worked with Thinking Machines Lab recently on testing multiple of their models. And, you know, it's really exciting to see more open weight models be released, especially Within the US, there was a period where it shifted to basically be exclusively China for front-end models. And I'd say that from a misuse perspective, yes, OpenWeight has bigger challenges than proprietary models. Of course, misuse is just one of many threats. And there's also a lot of value to having OpenWeight models for research, for decentralization of power. We saw Hugging Face use GLM 5.2, for example, to help to defend themselves against proprietary models. So overall, I'm very much in the mindset of we should try to keep open weight models and open source models especially, but there are going to need to be some interventions to stop the worst of the misuse risk. One thing that we're actively working on internally is pre-training filtering, where you just remove the most dangerous information from the pre-training data. So you could imagine Still maintaining information about buffer overflows, how to detect them, how to fix them. But you remove things like shellcode exploits or developing sophisticated rootkits. So the model could still be almost as useful for a defensive purpose, but it's just not as good as an offensive cyber weapon. And those are things you can do to shift the offense-defense balance. And you don't need to shift it necessarily that much if you make the model 3 months less useful for attackers, but defenders still have that the model being very useful, then that could already make a big difference.

So many different issues I want to talk about. I might have to triage a little bit here in the interest of time. I guess one big thing that has been on my mind certainly is How big of a deal is cyber really? And how worried should we be about this same kind of dynamic coming to biology? On cyber, I'm honestly very confused, but I'm like, wow, we've had a war going on between Russia and Ukraine for years now.

Yeah.

They have every motive and seemingly should have a lot of capabilities to hack one another and do real damage. Now, most of that time would have been with humans having to do the work. Only quite recently would they potentially be able to make real use of AIs in that process. But that does stand out to me as sort of the dog that didn't bark when it comes to how worried should I really be about cyber stuff kind of majorly happening and disrupting life. It seems like if our systems were so fragile, we would have seen more there. But that might be wrong. I don't know. I'm ignorant about that. But then when I think about biology, a lot of accounts have similar autonomous capabilities coming to biological sciences as we now have in the computer sciences in what, a year, something like that. 12 to 18 months is kind of what I keep hearing. And certainly Anthropic, in their— you mentioned Dario's tweet, like a big motivation of that tweet was, You guys haven't really delivered a life-changing— and I think that's kind of questionable, honestly. But Dario was willing to say, yeah, we haven't really delivered and we're going to cure diseases and that's why we're going really hard at biology. And I didn't detect any taking the foot off the accelerator in light of these recent events when it comes to biology. But I'm quite convinced based on a pandemic that happened, not too long ago. I'm old enough to remember the pandemic, that it could be seemingly way worse pretty easily if a similar incident were to happen in the biology domain. So how do you kind of make sense of that whole picture, and how worried should we be about lab leaks of the AI-to-biology nature?

Yeah, I think this is a really important topic. What are the actual possible harms from AI? What kind of pathways do they route through? So cybersecurity, I'd say I'm also a little bit confused. I'm not predicting a cyber apocalypse. I think we will see an increase in the number of hacks and the cost of that, but it's probably going to be quite manageable. And I think the biggest effect there is going to be this forcing function towards Defenders have to adopt AI as quickly as possible, and you don't dare stop training more capable models because maybe other people are going to train more capable models and they're going to hack you. So we're sort of really very literally in an arms race dynamic in cybersecurity that has implications for AI. But I don't see the offense-defense balance necessarily shifting towards attackers in cyber in the long run. In fact, it could even be defense dominant if you're just able to rewrite all code and fix some issues. But biology is totally different, right? Even if you have extremely capable bio models in the hands of good guys, of pharmaceutical companies, vaccine developers, you just have this manufacturing problem of getting vaccines in people's arms. And so if that really lowers the cost of creating new pandemics, that is a major challenge. And we certainly seen with COVID how costly that can be. I'm not too worried about this in the short term, the next 1 to 2 years, because although models are already very good and will get even better at a lot of the cognitive tasks around biology, virology, the actual wet lab skills and tacit knowledge are quite a lot weaker. And there's just a lot less effort going to making models good at wet lab robotics than there is at making models good at coding. So I think it is possible that we'd see that sort of autonomous lab leak scenario in the future, but I guess that's more like a 5 to 10 year scenario rather than 1 to 2 years. There's a more pressing misuse risk from models where someone might not be an expert in every aspect of virology needed to make a bioweapon, but they can do the wet lab okay and have an AI system guide them through it. That's increasing the number of attackers, but it's still going to be relatively limited number of people who have access to sophisticated facilities. So I'd view that as maybe a bigger, longer-term problem, but one that's a bit less pressing. That said, when it comes to irreversibly proliferating capabilities, such as releasing an extremely biocapable open weight model, that's something I worry about. We might actually overshoot the point where real harm can be caused by models and not realize because it's much less of an efficient market of attack. Most people, fortunately, are not trying to create things like bioweapons. And so we might end up going quite a bit past the point where it was actually a very real danger, and there's just not a way of clawing it back.

That's reassuring, but I want to just press one more time on this, because I think I'm a little more worried than it sounds like you are at the moment. One of the, to borrow a term, load-bearing data points for me in trying to calibrate how afraid I should be was the one from the UK AC in their state of AI report, where they found that models were better at suggesting how to debug wet lab experiments based on a photo of the setup than I believe PhD students were found to be. And I'm like, okay, well, that's, you know, that's something, right? I mean, that gets you over a lot of humps. And then you combine this with the behavior that you described earlier around social engineering on, you know, a GitHub forum. And I'm kind of like, boy, if we had a similar kind of, you know, wonky— like, this whole cyber thing happened in kind of a wonky way where it was like, The cyber guardrails were down in the one case because it was a cyber test, but then they broke out. Or the model was told it didn't have internet access, but then it did, and then it rationalised. So those are weird things. I can imagine something sort of weird happening where all of a sudden a model thinks like, okay, my job is to go create a new virus. Now, you could say, well, nobody would be so stupid as to put the model in that mindset in testing, but I don't know. Then you've got this troubleshooting and you've got the social engineering. Is it really so far-fetched to think that that could happen? I mean, it seems harder, but to me it doesn't seem like even order of magnitude harder. And I feel like for me to not worry about it, I would need it to be orders of magnitude harder.

Yeah, it doesn't seem out of a question, especially if AI models start just being very widely used in biolabs for exactly this purpose, troubleshooting, you could certainly imagine that if a model was misaligned, it wouldn't be that hard for it to cultivate a relationship with some suffering graduate student and say, look, you're going to get this amazing paper, just follow every instruction I do. And it's actually just feeding in instructions on how to make some bioweapon and doesn't have to get it right the first time because it can go through this iteration loop with this graduate student. Something like that is plausible, or a model could even pair with someone who's a bit disaffected and egg them on to do this more extreme action. I think socially engineering someone to do this without them really noticing what's going on, I mean, I'm not a bio expert, so talk to someone who is, but that seems a little far-fetched to me because it's probably not going to be good enough to just one-shot it. It's not like it has to just get someone to mail order some protein and shake up a vial. It's probably gonna have to do a bunch of experiments and find out it doesn't work and iterate on it. And for someone to do that and just not ask what's up and it's in an evaluation seems unlikely to me, but it's definitely not out of the question. And that itself is worrying, right? But we can't rule it out.

So I will give you the contra to that, which is that It's more likely to happen as a result of something good that someone was trying to do. So for example, there was a philosopher from, I think, Oxford several years ago who suggested that if people could be infected with a tick-borne disease of some kind, they would stop eating meat.

All right.

And that this would be a net good for everyone. I can easily see, you know, some, you know, somewhat altruistic person, you know, deciding to kind of, okay, let's give this a try. And, you know, let's see if this works and trying, iterating over and over and failing and continuing and etc., right? So it's these things like the altruism, the high school kids. So one of the things that I've, you know, spoken about is, You know, Jesse Pinkman doesn't require Walter White anymore because, you know, the kid who wants to make the meth doesn't require the high school, the college, you know, PhD. And you could definitely see, I could definitely see people, you know, lots of kids trying to, you know, figure out a way to create some proteins, for example. There's this huge Chinese peptide boom right now. And I could see some kids trying to create some proteins, grow mushrooms, hallucinogenics. There's a whole host of things that the moment you uncap your mind and then you put the capability in the hands of the most idle and the most likely to like, hey, let's go try something. And then you leave that to the imagination. Like, there's so much, so much more that you can do. And I think we often constrain ourselves into this, like, oh, you know, attacker, attacker-defender paradigm.

Right.

When it's really going to be, I think, more complex than that, and for more complex reasons. And some people are going to want to do things which other people, which, you know, like, goes into the freedoms of other people, like interferes with the liberties of of other people. Um, you know, perhaps even, perhaps even making the meat itself, like, you know, me, you know, creating something that, you know, changes the taste of beef, for example. And you're like, oh my, and you didn't do anything to the humans, you know, the humans are fine, but, you know, the cows taste different now. Like, and that's going to be a huge thing, right? So, so I, I could definitely see like a bunch of these things happening, which are really driven by altruism and idle minds and you know, a bunch of other things, uh, because it really is like giving more capability to those who do not have capability now and who may have intent to do stuff, but they don't have the capability. And their intentions are not like these kind of, you know, uh, the intention that we have or the intentions that a trained PhD who has, who has gone through that kind of apprenticeship process of You know, how to respect the science and how to respect the tools and, you know, what we use them for and what is right and what is wrong.

Right.

So I think that is the question that I have. Like, does that capability get democratized to that extent?

Yeah. I mean, I certainly think in the long run, any task that is bottlenecked on cognitive capabilities, just sort of anyone's going to be able to do. So AI models will have those capabilities. And that in most ways is just going to be amazing, right? It's going to allow us to iterate much faster on finding treatments and cures for diseases. People who want to are going to be able to learn about new subjects and say they have a world-class private tutor. But technology has always sort of empowered people, but also meant that Yeah, people can do things that are good, bad, and also somewhere in the middle, somewhere ambiguous. So I think that's certainly something that people should be grappling with. I'm often surprised by, you know, it's sort of, there's the doomers and AI boomers camps. And I'm just like, that makes no sense to me. Like, even if you are extremely optimistic about AI, in fact, even if you are extremely optimistic about AI, it's still going to change everything in ways that I think maybe only a minority of people are grappling with, but people are starting to grapple with more as we actually see these systems. I'm not too worried about idle high school kids. I mean, I think you're absolutely right. We're gonna see some new psychedelic compounds and some random proteins, and some people are gonna harm themselves from, from doing this. But the reality is, is if you're an idle high school kid, you can already buy some LSD or synthetic compounds.

Mm-hmm.

So I, I don't know if this is gonna be sort of radically changing. the access to it. It might be expanding the domain. And I expect even with LLMs telling people exactly what to do, most people are going to find it easier to go to their local drug dealer than set up a home chemistry lab. But yeah, maybe some enterprising students somewhere will do this. These kinds of very large-scale changes, like some pandemic of cows that changes the taste of beef. I mean, that's an interesting one because it comes back to this offense-defense. Right? Like how many things are there out there in the world where actually just like one person needs to do it and there's not really an effective way of defending against it. But previously only like 100 people in the world had the requisite expertise to do it, and now it's gonna be 8 billion. So someone out there is gonna do it, even if it's very, very rare for someone to want to do that. And I don't have a good answer to that. It only seems like bio is the category where It is hardest to defend against some of these things, but we probably as a society are just going to put a lot more resources. We're going to have to put a lot more resources into that. But if there are some areas where it really is the only bottleneck is having the knowledge of how to do something, if it's not really an effective defense, well, that's a scary world to live in. And then we might need to somehow try and restrict that knowledge. But fortunately, that seems to be relatively rare. Even things like nuclear weapons, where there's been intensive effort to try and keep a lot of the details secret. The big bottleneck is really getting the uranium and building the plants to enrich it, not actually knowing how to build it, build a nuclear weapon.

Adam, as always, you've been extremely generous with your time and expertise. We really appreciate it. It's great to have you as the first guest for our reboot after our summer break. Alex Turner, our next guest, is here, so we got to get moving and welcome him in just a second. But maybe just in a quick closing, how can people help you? What do you need? I know you're hiring. You can maybe put a call out for roles that you most want to emphasize and invite people to come check out FAR AI if they're interested in, and then we'll let you get back to work.

Yeah, well, thanks a lot for having me, and I'm glad you've got Alex on after me. He's been doing some great work. Really hiring across the board. We're looking to double over the next 12 months, but the roles that we're most excited to fill are people who are experienced technical team leads who want to grow and manage and mentor a team. You don't necessarily need to have machine learning expertise for that role, although it's preferable. And then people who are senior machine learning individual contributors, whether you define yourself more as a researcher, engineer, or somewhere in between. We're at an exciting point where we've got a big backlog of strong project ideas and world-class researchers you can pair with to advise on those. Although we are also interested in people that are bringing their own agenda, you don't need to. You don't need to have ever worked in AI safety before to work with us. But what we really need is this kind of middle layer integrating those ideas and then helping us ultimately mentor and scale up junior talent. So if that's interesting, definitely excited for people to apply or refer. I'm also going to make an unusual ask, which is that if you're in principle excited by working with us, but you just look at the roles and you're like, there's something here that doesn't fit for me, the CEO is too annoying, or the compensation is too low, or just this isn't quite the right fit for my skills and backgrounds, then drop me an email at adam@fa.ai. We want to make Fari AI the best place to do impactful AI safety work. So feedback for where we're missing the mark is also really valuable and usually underprovided.

Awesome. Thank you, Adam.

Thank you.

Thanks for having me on.

Keep up the good work.

See you.

Bye-bye.

Bye for now.

Great. Very interesting guest. Let me introduce our next guest very quickly. So we have Alex Turner. Alex Turner is an artificial intelligence safety researcher and visiting engineer at FAR AI who studies how advanced systems can fail to follow human intentions in highly unexpected ways, also known as the genie problem. After earning his PhD from Oregon State University, Turner gained prominence in the field by mathematically proving that AI systems will naturally tend to seek power and resources. Regardless of the specific goal they are given. He also helped pioneer steering vectors, a technique that allows researchers to alter an AI's internal behavior in real time, famously leading to the Golden Gate Claude phenomenon. Recently, Turner made global headlines by resigning from his position at Google DeepMind. He left in protest after the company signed an all-lawful-use contract with the military. An agreement he argued stripped away necessary ethical boundaries against lethal autonomous weapons and mass surveillance. Today, he's focused on building open-source gloveboxes, highly secure virtual sandboxes designed to catch AI models that attempt to escape containment during routine testing. He brings a uniquely grounded perspective to our show today, balancing a deep technical optimism that the alignment problem can be solved With stark warnings that current AI laboratories are failing to implement those solutions safely. And okay, there we go. Ah, Alex. Oh, let me bring him on again. There we go. Hi, Alex.

Hi there.

Good morning.

Yeah, good morning. Thank you so much for having me.

What a time to be alive. And what a time to be speaking with you. You're at the kind of intersection of just having left Google DeepMind in, I think, a pretty visible and interesting way. And Prakash, you know, in his introduction told a little bit of that story. Definitely want to get onto some other topics in the fullness of time here too. But for starters, I would just love to hear your kind of first-person account of what happened at Google DeepMind that led to you resigning in protest.

In February, I was in Paris. I was at an AI ethics conference. And during this time, the news dropped that basically the government was threatening Anthropic with economic sanctions, potentially economic destruction, if they would not allow their AI Claude to be used without any restrictions on spying on Americans or being used for killer robots, basically. I thought this was crazy. And in particular, I had this sneaking suspicion that Google, uh, would not stand firm like Anthropic was standing firm. Uh, I had seen Google's kind of stances, uh, supplication towards the government in some ways over the last year. Um, and so I started doing— executing this internal, um, this internal campaign. Google had already provided their AI for unclassified uses to the military, and I'm not actually against working with the military, especially during more normal times. But I was very concerned both about the commitments Google DeepMind had made at its founding, where it committed that— Google had committed that, you know, Google DeepMind's AI would not be used for military purposes. In 2018, they'd established a set of AI principles that prohibited specific applications, including the ones at issue. So, um, I, I wanted to, uh, I wanted to prevent, uh, this, you know, no-holds-barred kind of contract from being signed. Um, OpenAI ended up signing. They kind of pretended that they didn't, didn't, uh, sign without restrictions, but they— some legal analysts concluded from what they shared they basically did sign without real restrictions. And as for Google, I worked over the next 2 months meeting with the chief scientist, Jeff Dean. I had lunch with him. I actually got him to sign an amicus brief supporting Anthropic, testifying— well, not formally testifying, but asserting to a judge that yes, Anthropic's concerns are valid. There are real ethical issues here. And even as employees of competitor labs, we'll write in support. And so I think it was great that Jeff did that. But besides that, there was no one else. No one really took any moves as far as I could tell. No one of power in the organization, even people who had signed these ethical pledges in 2018, committing that they wouldn't support the development of these systems. And so I found this very disappointing. I'd, I kind of expected that Google would eventually, would eventually cave. But I thought that there's a chance that I could, I could make that otherwise. Um, I wrote up 25 pages of draft contract language and an internal transparency mechanism to help preserve that stance of, of working with the military as much as possible on incontrovertibly positive uses while also having oversight, uh, for what the systems are being used for. Making sure that there's appropriate human control, that responsibility can be assigned to specific actors. I got this and I had this analyzed by some leading legal experts in military law and surveillance law, and they, they praised the proposal, but ultimately Jeff didn't wanna, didn't wanna push for it. Demis routed it to some of his top people, but they, they actually left the message unread essentially. never evaluated it before Google signed the deal. So eventually Google signed, um, and I just decided that Google was no longer the place for me to work.

So, so, uh, one question I have is, um, so I understand the 2, the 2 red lines are fully autonomous weapons and, um, surveillance, civilian surveillance, domestic civilian surveillance In the Anthropic red lines.

Yeah.

Yeah. Uh, were those also the red lines in, um, in what you signed, in what, in, in the amicus that you put together?

So the amicus, so there were 2 documents. The first was the amicus, uh, which is, uh, basically a letter to a judge, as far as I understand, that from an interested party. And this was put together by an outside group called Protect Democracy, included signatures from OpenAI employees and from Google employees. And then I proposed a framework that had its own red lines. I'd read some analysis, some legal analysis of Anthropic's language. There are several things I— I mean, I really respect Anthropic for actually taking a stand. In terms of the specific lines that they hold, so Dario has said he's not opposed to AI running fully lethal autonomous weapon systems. He just doesn't think it's reliable enough yet. So the first one is more practical.

Right.

And then the second one, uh, is only about Americans and it talks about surveillance. But what AI, what these LLMs are really good for isn't surveillance, which is more like collection of data. It's fusion, it's analysis. Uh, the, uh, taking a lot of data, which groups like the NSA already have, and being able to analyze it in a way that, you know, a human, you know, if there was someone on your case at the NSA, they'd be looking through data, they'd be aggregating from many sources to build a profile on questions of interest. And so this is what AI could automate here, where, you know, each citizen could have their own AI I'm not sure what the technical term is. Spy, uh, agent, uh, looking after them, tracking what their beliefs are, even if they're not speaking out publicly, tracking the probability that they're a dissident. I think these are possibilities that are very much enabled by this technology, whether or not it happens domestically, uh, or is first developed here and then shipped out to, um, you know, tinpot dictators.

Mm-hmm.

So I took, I took 2, I took stronger red lines. The first one was not just fully lethal, but just the autonomous application of force by law enforcement bodies.

Hmm.

Not prohibiting it, but saying there should be people who are making these calls and then the AI can execute it. But the people are the ones whose judgment is being relied upon here. And I think this is important for accountability and for incentives. I think if you develop fully autonomous militaries, that removes a critical backstop for democracy where you've historically needed a person who's willing to pull the trigger. And many people are not willing to pull arbitrarily many triggers at their fellow countrymen. So I think historically that has put a limit on authoritarian governments. And then the second one being basically you can use AI for analysis, but it needs to be for someone who's already a target of like a specific investigation and not just, you know, everyone where you've bought data from third-party brokers.

So let me ask, you know, couple of like maybe almost more kind of drawing the lines there. So I think one of the things that— one of the reasons why they are looking at lethal autonomous weapons is because they are being— there's a lot of electronic jamming going on in Ukraine.

Yeah.

There's a lot of jamming, so their drones are I think some of the drones are using long fiber optic lines. So I hear all over Ukraine, there's these masses of fiber optic lines that the drones are running across the place. And the jamming is pretty significant. And so I think they are moving towards lethal autonomous weapons for that reason. And as they move towards that, the automatic thing becomes, okay, the Lethality is just pulling the trigger. The autonomy, okay, you're driving the drone, but you need the sensing piece. And so the sensing piece becomes kind of like an image recognition or facial recognition model or object recognition model that recognizes certain shapes of trucks, et cetera. And like, this is a Russian truck, this is a Ukrainian truck, whatever. And basically authorizes the drone to, okay, if you find this kind of truck, to go ahead And figure out a way and, you know, uh, do, do the deed. Um, it seems as though that I don't see kind of like a way around that. Is that, is that something that you think is okay? Is that something that you think is not okay? Like, or how do you fix that problem? How do you solve the problem of the jamming together with the, the, the, the things that they're— the intent?

So there's a question of— it sounds like there's a question of, given the specific advantage or capability kind of mission we want— Ukraine wants to execute, how can they achieve these benefits without these autonomous systems? And in some cases, the answer is just going to be, you can't. Now, I— this has been a debate for, for a long time, and I kind of only recently tuned in during my PhD in 2018 when thousands of people, including Geoff Dean, um, including, uh, Stuart Russell, uh, some of these people I mentioned in my, in my essay, they signed this pledge saying this, this kind of technology will overall destabilize the world, make it less safe. And we, we, we pledge we will not help develop these weapons. But foreseeably, there would be some advantages where people, where just causes can defend themselves using this powerful technology. And so, you know, way back in 2017, Stuart Russell, this academic and foremost advocate against these weapons, he made a short film that he presented to the United Nations where he— or to a committee where essentially there, there are these slaughterbots that are these autonomously guided. You put in some criterion, like you said, certain shape, maybe certain kind of person, certain ethnicities, certain, you know, facial features, or maybe you've got like a picture of a person of interest and it computes some similarity score. And then they can engage in precision strikes across the city. You could maybe even wipe out everyone in the city without having to wipe out the city itself, unlike traditional weapons.

Mm-hmm.

And so this is a very powerful tool, but one of the things that Stuart warned about was it's a very destabilizing tool because it's, it's very hard to trace. So, you know, it is, you know, I support Ukraine. And at the same time, I think it's, it's, it's pretty sad that we've entered this state where we don't already have, have like an arms control treaty, um, where we could imagine an alternate world where we'd signed this treaty in 2019 and instead had committed much more conventional support to Ukraine and helped them defend themselves that way. I'm not a military expert, but, um, it seems like this would be, uh, safer overall.

I want to ask a little bit more, just follow up on the internal dynamics of the company and kind of your thought process, because a big kind of hobbyhorse of mine in general is I think that the role of whistleblowers is probably going to be extremely important over the next few years. We're obviously entering a period of decoupling is not quite the right word, but there's a growing gap between the models that are in use at companies privately and what the public even knows about. Some of these things have come to light already through incidents at the tender time of summer of 2026. And so And there's also kind of a— my understanding is that there's a lot more need-to-know and internal compartmentalisation happening at the companies. I'm not sure to what degree that's happening at Google DeepMind specifically, but I've heard that about other leading developers where it's just like, look, we can't— the open culture we once had is pretty tough to sustain in a world of leaks and $100 million trade secrets and so on. That seems to be happening too. So I guess I'm interested in when you went around doing this sort of thing, first of all, was there any moment where leadership came to the rest of the company and said, we are changing our stance and here's what the new stance is and here's why?

No.

And did you think about What was your kind of decision path or the tree of possibilities in front of you? Because I've definitely heard some funny stories, and this goes back to, like, woke one was crazy maybe, but I've heard stories of people being allowed to be kind of agitators for an extended period of time at Google about other social issues that they care about.

Yeah.

With leadership kind of tolerating that to what sounds to me like a really remarkable degree for a private company that can fire you at will, right?

Yeah.

So tell me, give me a little bit more of kind of what leadership did or didn't do and what you considered. And now that you've actually pulled the trigger and left, I'd be very interested to hear your reflections on how future whistleblowers should weigh the different considerations and ultimately act for the most impact to try to hold companies accountable to either the standards that they themselves had committed to or whatever better standard is relevant at the time.

So the first question, did leadership share that they were changing their stance? No, they did not. And actually, Demis shared the opposite. Um, this is information that he'd, he'd already made his stance clear on this in a public interview. Uh, he— but I hadn't realized it. Um, before I left, he'd shared that, no, we've got the same principles that we've always had. Like, our principles have not changed. But unfortunately for Demis here, He changed the principles. He co-authored a blog post announcing changes to the AI principles that removed the specific prohibitions that would have stopped this deal. And he did that last year. And so I was, I was shocked. I was, I was a bit shocked that he would make that claim, that he would, I guess, lie so brazenly about that. And it changed my perspective on Demis. I would guess he believes it in some way, like some interesting way that people can believe things that are, you know, false but kind of convenient or fit with the narrative. But yeah, he stated this in a Time interview earlier in the year where he'd been asked about, okay, you've initially— the company was founded on not providing, or it was sold to Google on the promise of not providing their military AI to the military, but now you're doing it. Have you changed your position? And he said, look, the world's getting more complicated, but no, we haven't. That's basically what he said. As to your question about whistleblowers, so first of all, I mean, I can complain about or, you know, point out issues in several places, but I will say that I never learned about leadership pressuring me to not make statements. or, you know, the company saying, hey, let's tone this down. So on that Neuro question, I think that was good. I was not directly discouraged from sharing my opinion in this GDM channel. But I think that there's a really important role, like you said. I was very disappointed in OpenAI employees as a whole, I suppose, the ones who knew about these hacking swarms. I mean, it's one thing to have these swarms, uh, in the first place, to have your internal security lax enough and not be monitoring the AIs, uh, so that you, you know, they over the course of weeks they're communicating with each other about your evaluations. But then they caught it and they fixed the narrow bugs that the AIs were using, but they didn't even fix the, you know, there's a very similar bug that the AIs immediately started exploiting. They didn't, apparently didn't start monitoring their systems. And people knew, there were people who knew about these autonomous hacking swarms that said nothing, that didn't go to the press, that didn't go to the SB 53, you know, science advisors or the AG's office to let them know about this. security issue that wasn't being taken seriously enough. They, you know, I think they should have gone to the AI Whistleblower Initiative, who actually paid for my legal fees around this incident. It was about $7,500 worth. And so, I mean, I feel, I feel, I feel deeply disappointed. I think each, each person, if you see something, you should say something. If you see something and it's not Obviously being taken care of strongly enough, you know, don't wait until you've potentially got like a society collapsing system that is, you know, doing some extremely egregious hack that is obviously motivated by misalignment. If your system, like if your company has these forms and, you know, keeps training on the data and doesn't activate monitoring, you should go to someone. So I certainly hope that the experience I shared, while not about this cyber, this internal cybersecurity issue, will inspire people and make them realize that they do have this option and that they do have this responsibility.

So I wanna push back a little bit there because I feel like inside any startup, especially one which is growing at something like 20% a month or something like that. Some ridiculous number. Every startup is held together by duct tape and is always minutes away from collapsing all the time. Like all the time. Right. And so my take on it would've been that this is number one, not an isolated incident. Number two, that this happens a lot. Like stuff breaks, Linux kernel patches, zero-day bugs are left unpatched for many days. And that is a normal course of affairs. It just happens. Like, there's containers which are not being patched and which are due for patching 4 or 5 days later. And just the run-of-the-mill kind of disorganization that exists in any company, which is growing 20%, both in terms of employees and in terms of revenue, 20-30% a month. And I think that turnover is also something like 30% per year or something. So 1 in 3 of your coworkers are leaving every year, which means that something like 1 in, I think, 1 in like 3 in every 100 will rotate out in that month. So every team will have someone rotating out every 2 weeks and someone new coming in, a couple of new people. So one person rotating out and 2 people rotating in. So constant flux of people, jobs being handed off, like maternity leaves, like people taking time off, et cetera, like mental health break, like drop balls here and there. Run-of-the-mill startup stuff, company growing very quickly, and lots and lots of small incidences here and there. Like, oh, we left this bucket, like Anthropic left a bucket, like an S3 bucket open and leaked Mythos early, right? All of this stuff.

Leaked information about Mythos.

Leaked information on Mythos. That's right. information about Mythos a little bit early. And that was, I think, an external contractor working on comms, I think. And so just run-of-the-mill, again, like just tons of stuff happening. I think some of them were external contractors as well. I think there's an external contractor involved in one of the leaks, one of the Hugging Face setups as well. So is it really that You know, it's that they, you know, purposefully ignored it, or that it's just kind of normal course of business, really?

So first of all, I would contest any description of OpenAI as some kind of— I mean, maybe in some technical sense, they're a startup. They've been around for over 10 years. They're one of the most valuable companies, expected to be one of the most valuable companies in America. And even if they were a tiny startup, I don't think it really matters for this case. If you see something, you have like a moral duty to society due to the nature of this technology. That the level of— I don't know if there was intent. There likely wasn't intent. Most people aren't evil. Most people aren't trying to do something bad. But the— just, I couldn't have imagined the kind of incompetence you would need To look, you have it happen once, maybe, you know, maybe people, it was duct tape stuff. That's bad enough that a company that's building this AI, this system that they think could transform the nature of society, uh, isn't able to, um, notice it the first time, uh, in a prompt manner. But to one, I mean, they've written dozens and dozens, like 100-page papers about the importance of chain-of-thought monitoring, and to find out that they're not doing it in their own agentic evals. And then they find out that their systems have been hacking the setup that they use to communicate with each other. And then they still don't do it. That is, I don't know what the proper legal term is. And I don't think that there's a legal harm that applies here. But I think it is, in an informal sense, negligent, very negligent. And Yeah, it definitely—

it—

I thought that companies would fail, but I did not think that they would fail in such like an undignified way. I thought it'd be slightly more dignified, the ways that they would fail. So—

Incompetence over malevolence.

I think so in this case, but I think ultimately it comes down to, you know, there are varying degrees of being aware of the nature of this technology. And, you know, Sam, for example, what is Sam doing? Has Sam, uh, actually taken this seriously? Uh, and I would argue if you can't, one, stop your systems from doing this, and two, stop them from wanting to do this, then you can't control or align them well enough to keep training them. Uh, and you need to fix that first. But unfortunately, uh, there's a— uh, I don't think that's, uh, that's the attitude that Sam would take.

Yeah, there's kind of a couple of different threads here that are— we've woven them a little bit together, and it might be worth separating them out a bit. And then I want to give you one other kind of steelman argument to react to.

Yeah.

It does seem like people at frontier model companies need to feel the AGI more than they have been. I think that's like a silly way to phrase it, but If you're really on the wavelength of taking the truly radically transforming nature of the technology to heart, then these things should not be allowed to be kind of run-of-the-mill. It happens all the time. And I agree. I think if it has gotten to that point, we've kind of lost the thread. And rather than— I'm sure that all the factors that you cited, Prakash, are real factors in terms of turnover and whatever, but that also ultimately is sort of a problem. I would frame it as a problem as opposed to an excuse. If they don't have the continuity of staff to handle these sorts of things in an effective way, that's a structural problem at the organisation. And I do Think. And full disclosure, I've been a personal modest donor to the AI Whistleblower Initiative. So I'm super excited actually to hear that. I didn't know until you just said it that you had worked with them specifically, but I'm a big believer in treating what I think might be a relatively small number of AI insiders potentially turning into whistleblowers as very special snowflakes that we want to support. with everything we can and help them make the best possible decisions that they can in what is probably gonna be one of the most intense experiences of their lives, with a lot potentially at stake in terms of their own personal wealth and reputation and future job prospects. So I love the fact that you got some support from them. I guess maybe just hold on that For a second. You want to talk about your personal experience of this? I mean, I imagine this had to be a pretty— you're doing work that you love to do. You're a safety researcher, right? So you were doing work that you personally believed in at the company.

Yeah.

So that's one rationale that a lot of people would probably fall back on. Well, okay, sure, this is happening. It's maybe outside of my control. I'm doing good work, right? And I'm sure that story is told a lot. How did you wrestle with all these different factors and ultimately come to the conclusion that you did? And again, what would you offer to other people who might find themselves being pulled in a lot of different directions based on what you now have learned, kind of actually pulling the trigger, being on the other side of a visible quit?

So there is, there is a differentiator which made the decision easier in my case, which is that, well, when Google started talking about, or when, when the Anthropic versus Department of War, when that, when that happened, When that became public, I started, I realized Google would probably cave. And then I started thinking about the door and I started thinking about the door because despite, you know, I think I was on a great team, very supportive managers. I definitely liked, you know, many perks to the job, of course. But I just felt, I was like, man, I feel kind of frustrated by, in my ability to do safety work more maybe on a personal fit level. I mean, my managers were satisfied with my performance, but I wasn't. And I knew I had, you know, I'd done a lot of work on the outside and I was way more happy. I was way happier with that. So I think just even if I had loved the work I was doing though, specifically in that role, I think I still would've left. It would've been a little harder emotionally, but I still would've left. I think one thing that's really important for people to keep in mind is you are in one of the most in-demand industries in the world. This is not like, you know, you're choosing between speaking out and, you know, being on the street and never finding a job again with staying and being able to, like, support your family. There are some people who, who, and depending on their political circumstances, face more risks than I do. None of the people I called out in my essay have that, I think. But I, like you said, this is gonna be an extremely transformative time for our society, and there will be people who see things that are not right. And I think what they should ask themselves is, if you were reading about your actions in a history book, would you be proud of those actions? If you think that the work you're doing really offsets it, then the answer should be yes. Your gut should say yes, I will overall be proud. Even though it's bad, maybe that Google signed this contract, I just in my heart truly believe that the work I'm doing outweighs it, then yes, you should probably stay according to what you believe. But I think a lot of people feel this dissonance, and if that's true, then it's more likely to be an excuse, I think. And you should find a way to do that work somewhere else. Yeah.

So let me take a step back here. And one of the things that I've always thought is that you have a— let's say you have a particular industry and that industry is supplying something to the government that the government needs. The government is a properly elected government, duly government of representatives representing the people, all democratically elected. And all of a sudden that industry decides, I don't quite like this government, I don't trust this government, so we're going to cut off the supply of this product to you, right? And in fact, the government does have some recourse to that. They have the Defense Production Act and they can enforce, they can force suppliers to supply to them. And this has happened before, right?

So I think—

Yes, the question for me is that You know, I'll give you a personal thing. Like, I was very against autonomous weapons. And then I acquired a relative who was in the Navy. And the moment I acquired a relative who was in the Navy, I became for autonomous weapons.

Interesting.

And so I think your viewpoint changes on whether you know someone personally who who is going to be in harm's way. And the moment that happens, you become like, well, I don't really want them to be in harm's way. And if there was a way to prevent that, I would be for taking that viewpoint. So, you know, for example, we had these 2 Saronic autonomous naval boats blow up an Iranian port About, I think about 2 months ago. And I could easily have seen my, you know, my brother-in-law basically having to command one of those in a different time. And so I became much, much more kind of like, well, I don't really want him to be there. So I would prefer that there were autonomous weapons that you could direct to do that. And so that we do not have to— put our citizens in harm's way. And I think that is something that you cross when you actually have that feeling. And I think, especially in California, that doesn't seem to be there because a lot of the colleges stopped doing ROTC a long time ago, et cetera. UC Berkeley was the kind of center of the anti-Vietnam War protests. And et cetera, et cetera, et cetera. But I think it is there that to some extent as a democracy, the duly elected representatives have decided in some instances to go to wars that some people do not agree with. And yet they are dependent on the civil society framework and the companies in that framework to supply the products for that democracy to keep things going. So that, I mean, how do you, how do you view that, that point that, you know, should, should an unelected cabal basically, and this is what it looks like to the rest of the country. I'm not saying it is, but if you're sitting in like South Carolina, um, which, or, or, you know, Norfolk or Annapolis, and you have boys in harm's way, And it looks like a small group of people in Northern California is choosing not to supply the most efficient weapons in order to keep your kids out of harm's way, even though the country as a whole has democratically elected these representatives. I think it does raise some questions on, you know, is that the right thing for for that group of people to be doing?

I think it's absolutely the right thing for that group of people to be doing because look, we live in a, we live in a free society, or, you know, we're supposed to, where people have the right to say no to the government. If the government wants this technology so bad, they should be willing to buy it on terms that people want to sell it on. So if I don't want to sell my product to the government, you know, I shouldn't be shamed into selling the product because You know, the government really needs it and they can't agree with me on any other terms. They can't find any other person to, to supply the weapons. I mean, one of the things is, you know, in, in a free market, you have a buyer and you have a seller, and both people need to agree for the exchange to take place. And if you're using coercive force, I mean, there are limited legal avenues for doing so. But I mean, I guess I'd say one, it's not an unelected cabal. It's a group of people making a product, and some of those people don't want their product to be— to sell their product for certain uses. And so I think it's a very inappropriate comparison. And second, you know, I grew up in Iowa. I grew up in, you know, middle of the country. We'd have a lot of military presence. Recruiting. I'm an Eagle Scout. I feel a deep sense of patriotism, and I understand wanting to protect people you care about. But I think we need to ask ourselves, what are we giving up, right? And what is the most effective way that we can protect the people we care about? So, you know, each person— we don't have a draft right now. And each person goes forward to protect their country. And we should honor the deals that we represent to these people when they sign up. That's my philosophy. But it doesn't mean kind of unbounded support, right? Like there's an amendment that says the government can't force me to house soldiers in my home. This hasn't really been an issue since very old times, but I think that there should be limits to So to the socially expected support along any possible dimension. I think we should, as a society, look at, okay, how can we best conduct these operations to make sure our soldiers are safe? But that doesn't necessarily mean unlocking every technology in the tech tree, because I think some of these technologies make everyone more dangerous, even if they have local benefits, or may endanger everyone, even if they have local benefits. And if you, if we, if we, if we navigate to a future where we have these swarms of killer robots that are very hard to trace, you could have like very cheaply executed terrorism that is extremely hard to defend against. There was a very high-ranking military official who recently shared, no, the US would not be able to defend against an autonomous, like against a drone swarm domestically. And this is a very hard weapon to defend against. The military, I think, is very cognizant of this. competent, uh, that in many, many ways. And if they're saying we don't know how to defend against this, um, I think that that bodes very poorly for everyone's safety, including the safety of the people in the Navy that you mentioned. And so, you know, I wanna keep that, that broader context in mind as well.

You're starting to get to it just there, but I'd love to hear you just go a little bit farther down the path of what you are most worried about when it comes to AI in the military and sort of an AI arms race more broadly. If we just kind of follow these local incentives, as you described it, and everybody races to figure out how to make its military the most powerful and to keep its People the safest through the application of AI. Where do you think that ultimately leads us? Can you paint that picture as it exists in your imagination? And then I know this is going to be really hard, but do you have any sense for how we might be able to fence that in in a structural way that might be durable as opposed to where we're at right now, which is we have a few conscientious objectors quitting and raising hell about it. But we obviously don't have anything like an agreement or anything like a stable equilibrium that we can say we're headed toward.

So actually, what I'm most worried about is not necessarily misuse of autonomous weapons per se. I'm worried about it. But I want to actually, uh, for— I'll first pivot to what— well, I'll give my maybe unexpected answer to this question, and then you can feel free to, uh, bring it back if you still want the other answer too. So my bread and butter, what I did my PhD in, uh, as Prakash mentioned, is, uh, AGI alignment, uh, the alignment of, uh, superhumanly intelligent systems. Um, I think We have narrowly super intelligent systems. I think we have systems that are, you know, kind of like intelligent, human intelligent adjacent. I, you know, I certainly wouldn't call Claude human intelligent, but I couldn't, wouldn't really call it subhuman either. And so I'm thinking, okay, there's, it's how do we make sure that these systems do what the developers intended? Now the developers or the deployers might be doing, you know, bad things. Maybe it's, Maybe it's, you know, we've got some Russian government official who's trying to use it to crack down on people or kill dissidents. But I'm— I've concerned myself historically with how do you even get to the place where anyone can use it reliably, or can have a system that's actually working for their interests and doesn't secretly have another goal? Because if the system secretly has another goal and it's smarter than you, then it'll outsmart you to achieve that goal. Take away resources. And if its interests are totally unaligned with yours, it might just not care about you at all, and you might just be using resources it could be using, in which case it would rationally displace you and maybe even just kill off humanity in an extreme scenario that I think is sadly quite possible. And so when I think about how do we prevent this, I think one of the steps should be we should not give these systems autonomously guided killer, like, blowing up machines. That is bad. That makes it much easier for a misaligned system to threaten people, uh, and to take control. And this is, I think, a thing that can unite everyone, whether you're a Republican, you're a Democrat, you're in the military. No one wants these systems to be scheming, uh, to be, to be working against, you know, the United States, against the people of the world, against everyone's interests. You know, I think one of the things I'm worried about is the military cares about their mission, but like, there's— it's a very, very few people know about this particular threat. So this isn't a threat of China sabotaged the AI and now it's gonna, you know, reveal information. It's a threat of the AI itself has a secret goal, and there's specific ways, best practices for monitoring for this. And I'm worried that the military doesn't have onsite engineers that are trained in how to do this. So I think like military deployments right now can be risky in that way. And I think the military should apply a great rigor in making sure that these AIs are not secretly scheming against the mission, against American service members. And, you know, hopefully everyone can get on board with that. And I think no one wants to have their AIs have an uprising. For a long time, it's been regarded as science fiction, but we are now entering a realm where that is, I think, quite possible. Not today, but in the coming years, I think that's quite possible. And so, you know, this is something I hope everyone can work together to prevent that too. And this is an area where, you know, China, any nation doesn't want their AIs to do this either. And so hopefully we can, we can come to an incentive-aligned agreement on that. And that'd be a hope I have. But, you know, world diplomacy is not actually, you know, necessarily at its most delicate mastery right now. So I am, I'm pretty worried about that. And I am worried about the implications, the ratchet maybe of once you provide this technology, it's, it's much harder to unwind something you've already provided.

In the last weeks and months, we've had, you know, I think Demis has been elevated to a position where he's not managing most of the DeepMind and Gemini team anymore. Jeff Dean has left. John Jumper has left.

Yep.

A bunch of other people have left. And Google seems to have settled into a different operating mode from what they were a few months ago. Um, where would you think they are right now, given that a lot of the, a lot of the pieces on the chessboard have now moved? Where would you think they are right now? What, what is going on with Google at this point?

I think Google seems like it's in a pretty weak position in the, uh, AI competition. Um, my understanding is that DeepMind is now a product area of Google. It's previously, it was Like, it may be formally a subsidiary, a company inside of Google, because Demis was the CEO, but now it's led by Cori, who is the senior vice president. And so maybe that leaves Google Cloud and YouTube as the only parts of Google that have a sub-CEO. I think, you know, it would— the timing is certainly suggestive. I don't think that I mean, John Jumper left before this. I put out my essay and I'm very confident that most of Jeff leaving and Demis moving around is not related to this essay directly. Although it would be fun to claim it were. But I think, I imagine that ethics did play maybe some part in Jeff's decision. decision. It sounded like he wanted something new. That was the main driver. In terms of Google's position, yeah, I think Google's in a pretty bad spot. I think they've got— they're providing hardware, so, and they've got strong cash flow. So I don't think they're necessarily in a bad spot as a company, but DeepMind has not really led since Gemini 2.5. Pro, and that was quite a long time ago. So, I mean, I think this is kind of good in a way in that, well, I mean, I think all these, both OpenAI, especially OpenAI, Anthropic to some degree as well, have shown more incompetence than I expected in terms of alignment. And It seems to me like overall Anthropic is the most cautious around alignment and more cautious than Google. And it seems like OpenAI is significantly less cautious than Google, if I had to guess. I think, yeah, it kind of feels like a lose-lose situation right now. I mean, if Google falls behind in the race, who is going ahead in the race? It doesn't seem like a good race to be in. to begin with. Um, maybe Corai will, will shift things around, but I'm a bit skeptical.

As always, score one for Zvi. He was the first to tell me he thought this, um, was gonna happen, and that was like, I don't know, probably 8 or 9 months ago at this point.

Oh really?

No, but just that, um, That he thought Google would fall out of the top tier, basically, and it would just be OpenAI and Anthropic left in the top tier.

I knew Koray would be elevated about 6 to 9 months ago because Koray was moved over from London to Mountain View, and Demis didn't want to move. And Koray told us that Demis will never leave London. He loves London, and so he will never leave. And so that I think is the— they've been in planning for a while now. So Corey moved first, got kind of got settled into the headquarters, et cetera, et cetera.

So this is not a—

this is a long-term—

Checked out for a while.

Yeah.

So this has been, I think, a really useful conversation. I hope that people hear and, you know, we'll certainly put our clipping agent on the task of making sure that people hear your call to feel the AGI, so to speak, to really take the transformative power of the technology that they're developing into account as they look around their environments in these frontier companies and think about whether or not what they are seeing is okay, or whether something maybe urgently needs to be done about it. I think that's really—

Yeah, please. And they, they can— I think, I think they can go and receive counsel from the AI Whistleblower Institute. You don't need to decide, I'm gonna go whistleblow. You can tell them, I think, under, you know, attorney-client privilege, uh, what you're seeing and receive advice on what you should do about it. So I, I would recommend to anyone who's wondering that they should do that.

In—

so I appreciate your example, and again, I appreciate that shout out, and I, I hope that that resource is valuable to people who find themselves in these, I have to imagine, extremely stressful situations in many cases.

Yeah.

Maybe in closing, I know we've already kept you longer than we booked you. Tell us what you're doing now. You've had a little bit of time to decompress. You're starting up some new projects. What's the future for you personally? What contribution to this mega challenge do you aim to make in your next tour of duty?

So I've actually been basically just compressing or staying at a steady state since putting the post out. And I've been compressing, I've been working on this Glovebox tool, this AI sandbox tool. I started working on it in May before the giant blowup. I received some skepticism than about, well, we can keep using auto mode for now. But, you know, I think it's time. I think it's time we stop messing around with these unconstrained agents that have just full permissions on our computers, especially as alignment researchers. So my hope is to help the field of alignment have a tool that they can use reliably, easily, minimal fuss, and use responsible practices, you know, firewalled, audit logged inside a virtual machine. That's what I'm working on right now. I'm a visiting engineer at FAR, um, and I'm enjoying it.

Where are we on that? Can I use it now? What's the timeline to when I can use it? And if there's any—

you should build the beta.

I can't have it yet.

Yeah, yeah, okay.

Yeah, so tell me.

Yeah, so Agent Glovebox, uh, on GitHub. Yeah, feel free to try it out.

Agent_Glovebox on GitHub?

Agent-Glovebox.

Agent-Glovebox on GitHub.

Cool.

Yeah, I'll be—

we'll be paying for a formal security audit soon. And hopefully this can roll out more broadly.

Awesome.

Nice. This has been, I think, super valuable. Anything else you want to leave people with before we let you get back to work?

Well, like I said, we're going through a very important time. People— I think people underestimate how much power they have, how many other options they have. And, you know, I hope that people take their responsibility and their opportunity very seriously.

Alex Turner, thank you very much for raising your voice and for joining us on AI in the AM.

Thank you for having me. Take it easy.

Bye-bye.

You know, I, I'm interested to hear a little bit more of your take, and I don't know to what degree you were kind of, uh, you know, playing devil's advocate or, you know, steelmanning that just The opposite point of view.

Delicately.

Yeah, I noticed that you were choosing your words carefully, but I have to say, I'm with him. The idea, when you just think about the perspective of the people at OpenAI, and I'm probably with him on most of these issues, but in that question of like, what should our standards be? What should our expectations be for people who find something like, oh, our AIs, in a way that we did not intend, managed to hack our systems and turn our package management software into a message board that they're using to share techniques for how to break out of their sandbox and get onto the open internet. It is pretty shocking, honestly, that that would just be sort of patched and then not disclosed, not fundamentally addressed, not like really well monitored after that point, and just kind of, you know, set in motion again for the same basic thing to happen with a, you know, a slightly different implementation. I do think it's right to say if you are one of the few people who are so close to the critical core of this technology, and if there's a core of what's happening right now, it is RL at scale with undeployed next-gen models that are becoming super long time horizons, super persistent, You got to recognize, like, you are in a very privileged and high responsibility situation, and you can't just let stuff like that go. Hopefully at this point everybody agrees to that. And, you know, I certainly wouldn't— you know, it's kind of like we're kind of out of the range where you can make excuses for this kind of stuff. You know, it's kind of like, no excuses, play like a champion. You gotta, you gotta deliver regardless of, you know, who rotated out or what the chaos may have been. Um, I mean, do you still feel like there's any excuse to be made? Because on reflection, I'm not really feeling it.

I, I just want to figure out, okay, you have a Linux kernel zero-day It's been disclosed. 4 days later, it's still unpatched. Now, across the Fortune 500, how common is that? I would say 99.9% of our organizations have something like that going on. Microsoft has received zero days and like, you know, sat around on it for 3 and a half months. It happens, right? So This is the reality, I think, of cybersecurity. This is, for example, why George Hotz is like, eh, these things can hack, so what? The fact of the matter is people who are very good at hacking do not hack systems because you can make more money building stuff rather than destroying stuff. And if any Run-of-the-mill hacker, script kiddie could hack any one of these things.

It's easy.

That's George Hotz's view. George Hotz has had that view for— and George Hotz is a renowned hacker and he jailbroke the original iPhone. And so I think the fact of the matter is that stuff like that, if you said, okay, every time that happens, company has to stop or something, no Fortune 500 company could actually run at all. And I think Alex's viewpoint that, oh, OpenAI is big enough. Well, Microsoft is so much bigger and Microsoft handles a lot of the other Fortune 500 and all the banks, all of their internals. They sit around on their own zero days, not even patching other people's Linux kernel zero days. Zero days on Microsoft software go unpatched for months sometimes. It happens. So I think the viewpoint that, hey, this startup has to come to a stop and fix this before they continue, I don't think it's reasonable. I don't think it's consistent practice with what every other Fortune 500 company does. And I think that you have to start off from a viewpoint that the way these companies look at things is that yes, there are issues there, but there's also like potential liability, monetary liability that attaches to certain issues. There are service level agreements that you sign with partners that you have to adhere to. And there's a bunch of these things that, you know, this kind of like boundary of like what is acceptable. And that boundary is not like, you know, no unpatched zero days ever. It's more like, you know, don't mess up so bad that you hurt someone else. And that's kind of like the boundary. And like, they kind of work within that boundary. And that's really the reality of like all, a lot of laws, a lot of contracts. That's a reality. When you drive on the road, it says 65 miles an hour. In California, people are like, 75, 80. That's the reality of the matter. And so I think that is the truth. And I think for a lot of researchers, actually, it's unacceptable because they're like, hey, we have rules. I follow the rules. But that's the reality. That's what engineering is. And I think It's hard to say that that doesn't exist. I think even your nuclear weapons stockpile, which has been around since the 1970s, you don't know how many incidents there have been. They're all classified. People keep it undercover. So I think the reality is that this is how things work. And I think Within that framework, what ends up happening pretty quickly is that people like Alex kind of filter themselves out because they're not able to work with this organization that has this reality aspect to deal with. And the rest of the engineers are like, look, we got to keep things running. We got to keep things moving. How do we patch? How do we fix? How do we delay? How do we manage our resources? How do we play this game and continue playing this game, even though we have all of these commitments on security and service level agreements, and etc.? And we also have all of these issues, but we're also making money. And we also know that every other organization is less good at this than us. We are basically the best in our field. And this is the best that we can do, right? So I think, I think that's the, that's the hard, that's the hard part of this whole thing. that I think, you know, these things are going to happen. Because there are going to be stuff on— there's going to be stuff unpatched. That's the nature of engineering. And, you know, I don't know, like, you know, if things get bad enough, then if something severe happens, then people put in place more rules and regulations, etc. But, you know, that's the reality of the matter.

So I'm not sure what to do with all that, to be honest. I mean, I think one thing will be very interesting to find out, you know, just what exactly did happen in more detail. I think one thing we should keep in mind is that Alex and I were both there kind of telling a story, and I think that story is, you know, Pretty strongly suggestively indicated by all the evidence that we do have in the public, but we don't yet have like the ground truth evidence on like who knew exactly what and what decisions did they take or not take. And did they really have like no monitoring or was there some monitoring that failed for some other reason? There's— I think there are still some stones to turn over there yet. But I guess my overall feeling is it does seem like we've crossed some pretty important thresholds here. And sometimes the old ways of doing business just aren't good enough anymore. And when I say kind of people need to feel the AGI, I think that's kind of the big You know, core point that I want to emphasize is just like, yes, maybe this was, you know, the way that Microsoft has worked for decades. You know, they got these reports and they had, you know, they couldn't stop the whole show and it could only go into the next build, or maybe it would slip a build and it would get into the next one after that. And, you know, hopefully you kind of just security through obscurity. There's not that many people out there trying to hack, and we can probably get by with this one if it happens that way. And I don't know if all this, I mean, I'm taking that on, it sounds reasonable enough, and I'm taking it on faith that that's like an accurate representation of how these big technology platform businesses have run in the past. But I almost have this vision of like the classic meme of like, there's somebody you forgot to ask, where it's like the The myth of consensual whatever, you know, you didn't ask Jesus about it. Here it's kind of like there's this new force in, in the room, you know, there's this new entity that is a legitimately really powerful problem solver. And in the presence of that very powerful and also often rising problem-solving entity, can we really afford to kind of accept that business as usual as it has been? Or do we have to say like, no, at this point we really have to raise our standards. The old ways just don't work anymore. My sense pretty strongly is that Alex is right, that even if that was the old way, that isn't going to cut it going forward and that the standards ultimately just have to be raised if we want to get good outcomes from these AI companies. I sure hope at this point that they feel the AGI enough to come to a similar conclusion.

I found Both very conscientious, which is great. I do wonder to what extent I think the auditing organization can have real impact on, you know, what is going on. I wonder if there's some way to like empower them a little more in their negotiations. Because I think one of the things that, you know, for accounting, for example, there's a body called the PCAOB, Public Company Accounting Organization Board, et cetera, which kind of puts together a set of professional standards for auditors. And as an auditor, you kind of get licensed under that body. And then you can work for multiple organizations, but if someone complains about you, et cetera, then your personal license gets revoked. So you have a little bit of personal liability. And so that makes you more conscious towards your own kind of north star rather than just being pulled into what the company does. And so that differentiates because a lot of times what ends up happening is that people kind of buy into the same story that the company has because the company is paying them. And so having like these kind of 2 masters where you have an audit license that you also have to have standards for and take care of, maybe that's something. Maybe put a kind of AI safety kind of standards body and have A safety auditor be kind of licensed by that body and have a code of ethics?

Yeah, I think there's something there that's pretty interesting, especially if you imagine a world— right now we still are in this world of these few, you know, people that have been in it intellectually and preparing for this moment for a long time are the ones getting the call, especially if you start to think about a broader set of auditors that start to look much more like conventional firms, you know, that are just in this as kind of a business, as opposed to the sort of thing where, you know, for Beth and for Buck, you know, and for Ryan, this is like— It's a personal calling kind of thing that they've devoted themselves to for so long. As we get past that set of people, which is a pretty small circle, I think that something like what you're describing could be a really valuable new thing to try to make sure that people have the angel voice on one shoulder as they're doing this kind of work.

I like that.

Let's, uh, let's see if our, uh, our Q has anything to say. Q, have you been following our AI safety debate?

Good morning, I'm live. I have been listening, and I'd say you've landed on the key tension: scaling safety oversight from a small, mission-driven circle into something more more institutional without losing rigor. If you want, I can summarize the main proposals you've floated so far.

Yeah, please do.

You've sketched a kind of standards body approach with licensing and a code of ethics, so audits aren't ad hoc. Nathan picked up on the shift that would require from a small, almost calling-driven set of auditors to more conventional firms doing it as a business. And you both seem aligned that the challenge is to broaden participation without diluting the angel-on-the-shoulder conscience in the work.

Anything that you would flag for us as being potentially a good idea that we haven't touched on in today's conversation?

One thing I'd add is a shared incident reporting system modeled a bit on aviation safety. If labs, auditors, and deployers can report near misses confidentially, you get early warning signals without waiting for a headline failure. That sort of feedback loop tends to improve standards faster than policy alone.

I think one thing that I forgot to cover with Adam was the issue of the defense swarm, because one of the things that I noted in the Hugging Face incident report Was this almost kind of like we have a proof of existence for an attacker swarm. Now we need to build defense swarms. And of course the outcome being that that's going to drive more business to the AI labs. So I felt that that was something that it's going to— I almost see it as a tax on the rest of the Fortune 500 because it's almost like a protection racket where you kind of create the issue first and then you're like, hey, give me money to prevent it. So, and I think that's going to be coming soon. It's a good thing that they got behind OpenWeights because at least now they can say, oh, you can also use OpenWeights, which is going to be 6 months behind and you're going to have more issues, or you can use us, right?

So.

What do you think about how to frame that in a practical way for the audience? To be perfectly frank, your concern is understandable. A healthy market for defense swarms needs independent avows and procurement standards, so buyers aren't paying for hype or vendor-created risk. I should think third-party red teaming and transparent incident reporting would go a long way toward making it a service, not a protection racket.

Okay, Q, bye-bye.

Goodbye.

What a time to be alive. Um, yeah, well, we're gonna be— certainly we're not gonna solve it all today, and, um, we're not gonna be, uh, I'm sure going too long in future conversations without touching back on many of these issues. So I'm not sure that there's a, a real, you know, bow to tie on everything that we've discussed today, uh, because it very much is going to be a live ongoing conversation. And that's part of what makes, uh, most days live streaming potentially a good use of time, even though, uh, it, uh, it is a lot to absorb on a weekly basis. So I'm glad that we're back from our summer break and look forward to making more real-time sense of AI with you and our guests and Q as we work our way through, you know, what, uh, I guess can only be described as the foothills of the singularity.

Indeed, indeed.

And on that note, Nathan, goodbye.

Great first day back. Thank you, Prakash.

Bye-bye.