Helen Toner OpenAI Reflections, Adaptation Buffers, and AI in Warfare

Hey, Upstream listeners. Today we're releasing an episode from The Cognitive Revolution with host Nathan Labenz. Nathan was joined by Helen Toner, director at CSET and creator of Rising Tide, for an insightful conversation about AI policy and adaptation. Up ahead, they discuss her experience on the OpenAI board, her concept of adaptation buffers for society to adjust to AI advancements, and her important research on AI applications in military decision-making. Please enjoy.

Helen Toder, Director of Strategy and Foundational Research Grants at CSET, the Center for Security and Emerging Technology, and author of a new Substack, Rising Tide. Welcome.

Thanks. Great to be here.

Yeah, I'm excited for this conversation. We got a lot of ground to cover. I think, you know, we all have crosses to bear in this life, and, and one of yours is you're gonna go on and do a ton of things in the AI space, and yet people are always gonna come back and ask you questions about your tenure on the board of OpenAI. And of course, you know, everybody is at least somewhat familiar with how that ended. So I'm not gonna be an exception to that entirely, but I do wanna make sure we have, uh, time for a bunch of different things. To set the stage, one question I actually don't know the answer to at all, and I'm really kind of curious about, is How did you get involved with OpenAI in the first place? I mean, this goes back years to a time when there was no powerful AI. Most people would dismiss the notion as fanciful, and very few people were taking the whole topic seriously in any real way, but you obviously were. So maybe just kind of share your backstory with respect to AI and some of the enthusiasm that you must have had to get into that position in the first place.

Yeah, absolutely. I mean, so I joined the board in 2021, but I had been familiar with the company and with many of the folks working there since they were founded. So they were set up in, you know, San Francisco in kind of 2015, 2016 kind of timeframe. And at that point I was working in San Francisco and that was right around when I was starting to work on AI issues. So it's really interesting reflecting at the time, it felt like kind of being behind the game because this was, you know, 2012, you had AlexNet, the deep learning revolution was really in full swing. by 2015, 2016. And so by the time I kind of came around to the view that, okay, this is going to be a really big deal, there's a lot of work to be done here. I want to make this a really real focus for my work, AI and policy and national security. Yeah, at the time it felt like that was sort of coming late to the party. But it's been fun to see since then kind of a couple more waves. I think around sort of 2018, 2019 or so, I want to say people started paying a little more attention. And then obviously ChatGPT in 2022, there was this huge new burst of interest. So looking back, I no longer feel like I was as late to the party as I kind of felt at the time. But yeah, I think, I think it's also easy to underestimate how, how, how weird and how kind of against the grain it was for them to found a company to build AGI at the time. That was really not the kind of thing that you talked about in polite society, including in polite machine learning society. It was really Google DeepMind, or sorry, just DeepMind at the time. Was sort of the only game in town of sort of serious researchers who were talking about AGI. And so I think that was also, when I look back on why I was invited to join the board, I think certainly part of it was being in the AI policy space, having a few years of experience at that when not many people did, having spent time in China, having, you know, that sort of China expertise and national security expertise, which I think was valuable for the board. I think it was also having taken the idea of AGI and taken their mission seriously for multiple years by the time I joined the board. was really unusual. And it's sort of funny to look back on that now in 2025, because obviously AGI is on everyone's lips. OpenAI is such a famous company that the situation looks kind of different. But I think for the community was really small. The set of people who had been thinking about these topics and who had actually informed perspectives was, was really small. And so it was super interesting to get to be sort of familiar with the company from the very early stages.

Were you always a relatively short timelines person? One of the blog posts that you shared the draft with me of just, I think rightly just kind of reminds everybody that like, even, you know, what people are now passing off as long timelines are actually quite short.

Yeah.

But where were you, you know, 5 years ago in terms of your expectations?

Yeah, no, I, I, I wasn't, I still don't know if I am. The standards have changed so much. And yeah, hopefully the post you're talking about that the current draft title is Long Timelines to Advanced AI Have Gotten Crazy Short. And hopefully by the time this comes out, it's published and hopefully the Substack is launched and everyone who listens to the, this episode will subscribe. But yeah, when I got into the space, I think, so when I got into the space, I think I counted as short timelines for the time, which was, as I described in the post, like this seems, it seems plausible. It seems likely enough to be worth preparing for that we build very advanced systems in the next couple decades, you know, in the next, in our lifetime, this seems like a potential development. And if it happens, it would need a ton of societal preparation. Almost no one is thinking about it. So that would be worthwhile to spend time on. I think That did describe my view when I got into the space. Nowadays, I don't really identify as having short timelines because that means expecting superintelligence before the 2020s are out or something like that. And I feel much more uncertain about that. I still tend to fall back to this view of, look, I think this is all likely enough that it warrants quite a lot of thought, quite a lot of preparation, which is different than, I think it's very likely to happen or very likely to happen in the next 5 years, in the next 3 years. So, yeah, I guess the question is, depending on what your standards are for short timelines, maybe yes, maybe no.

Yeah, I used to make a very similar argument to people when the whole notion of powerful AI was fanciful, and certainly any notion of safety concerns related to AI was doubly fanciful. I used to just say, we have a small number of people that scan space to try to find asteroids so that we don't get taken out like the dinosaurs did, and that seems really good. And this kind of seems like another thing. And now it definitely feels like there is an asteroid and it's coming at us, and we don't know if it's good or bad, but it's definitely going to be both.

Yeah. I mean, to me, the sort of underlying internal motivation to kind of work on this space was related to the way that I think if you look across the scope of history, huge new technologies tend to really change what society looks like, for better or for worse, often for better and for worse. And so it was really coming to believe in sort of the early to mid-2010s, Okay, this looks like we're going to go through one of these transformations, probably in my lifetime, and that's going to be a really huge deal. And if I'm interested in trying to leave the world a better place than when I found it, to the extent that I can, then maybe this is an area to go work in and shape. And so to me, it was never like, oh, this is definitely going to kill us. And so I have to get into the space to prevent it from killing us. It was much more this broader argument about, seems like pretty likely we're going to go through this massive transformation. Can I get into a line of work that can help contribute to that going better?

That mindset point is really interesting, and I want to ask you about the sort of prevailing mindsets at OpenAI from your perspective. Before getting into that, I know you've given a couple of different interviews about this and spoken about different aspects of it and probably are tired of it, understandably. What kind of governs what you can and can't say at this point? I mean, we've all seen also the non-disparagement clauses that were then nullified and I don't think you ever had any equity in the company. Maybe you did, but I don't think so. Right. So, yeah, how much is sort of external constraint on what you can say, and how much is sort of just you deciding how much you really want to talk about this?

Yeah, it's 2 big factors at this point, or maybe 3, depending how you count. As a board member, I'm under ongoing confidentiality obligations that don't apply, for example, to former employees. So just in terms of Conversations on the board, topics that we're discussing. I want to respect those obligations. I take them very seriously. And then there's also just ongoing legal processes where I could be, my statements could be compared against each other for consistency, and minor discrepancies could cause problems. I might need to testify under oath or otherwise say things. And then I think also there's just a range of other stuff. So confidentiality, conversations that I've had in confidence with people, A lot of this stuff as well, it's like, so there's just so many details and so much, you would need to go back and explain so much context and bring in like other people who really don't need to be dragged into this. And like the payoff wouldn't even be that great because none of the, you know, I gave an interview, the best interview I've, I've been able to give on this, the most detail I've been able to go into was on the TED AI Show last year. If folks are interested, they should go listen to that. And, you know, it's not that there's like hidden secrets that are more shocking than what was, what was there. There's just a lot. more kind of, I want to say almost like boring detail, but that brings in a lot of stuff that's maybe confidential or maybe involves other people who don't need to be dragged in. It's sort of like, I don't, I don't think the payoff is there. And so certainly if people are like, oh man, there's still this big, deep, dark secret that Helen Stilger hasn't spat out, that's not the case. Yeah, there are still reasons that I, that, you know, that I just talked about that I am not, uh, just sharing everything totally publicly. But I think even if I were able to, it wouldn't change the overall picture in any, in some kind of dramatic way.

Yeah. Context is that which is scarce, as you say.

And actually, maybe one, this is a good point to share one thing. You know, one example of something that I didn't comment on because I thought it was, uh, I wanted to respect my obligations to the company around confidentiality was this, this rumor at the time about Q*, and Q* contributing to the board's decision, which was totally false. So I didn't, haven't really commented on it because I didn't want to get out ahead of OpeningAI's sort of reasoning work and the what has now been released as 01 and 03. The board was aware that research was underway, but we never got some letter about a breakthrough. We didn't make our decision based on a letter from employees. That whole Reuters story was totally false. So that's one example of something that is now slightly easier to talk about because the underlying sort of confidential information around that line of research is now out in the open.

Yeah.

Okay. So about the The mindset or the sort of motivations. I mean, you said wanting to leave the world a better place was part of what motivated you to get involved and just kind of recognising the stakes, feeling like this is sort of a high leverage activity. That seems to me to be a big part of how I understand what I think is kind of motivating people at OpenAI in general. And in a way that's like good, right? I mean, everybody You should want to make a positive difference. But I do sometimes worry that it can cross over into a sort of main character mindset or kind of a hero mentality. And especially as I hear more and more things about sort of guru-style coaching going on and practice of detachment and sort of an elite performance mindset, I'm not sure if it's maybe gone too far. And part of me is like, maybe we should want some amount of attachment among the people that are developing the potential superhuman AI systems. So much of this is hearsay. I don't even really know how pervasive some of these ideas are, but there are definitely quite a few data points at this point. Would you say that's prevailing sense at the company that there's this heroic, world-changing quest that they're on? Or would you put that as a more minority position that's just occasionally popping up?

I mean, I think board members are generally not in the best position to talk about the culture at a company because they're not kind of immersed in the employees the way that a team member would be. So I honestly don't know that I have perspective on that, that you wouldn't. The perspective I do have comes from knowing people who work there. And I know, you know, people who work there as well. So, I don't feel like I have more to add necessarily.

Okay, fair. How do I get at that? I mean, it seems, it does seem like a very important question. Like, I do have this sense that sort of detachment in frontier AI development seems somehow wrong. Like, it feels like we're taking, you know, to put it in machine learning terms, feels like we're taking something that was learned out of domain. The sort of mindset that I think I'm seeing is kind of like what they tell NBA 3-point shooters to adopt, right? Just like, keep shooting and don't worry, trust the process over the outcome, and que sera sera, sort of. And I don't know that generalises super well in these high-stakes technology environments.

Yeah, I mean, I think the other version of it generalising would be just this separation that we sort of implicitly have in society more broadly between kind of the people building new technologies and doing scientific work and the people who are figuring out how to apply them and how to regulate them. And I think it's generally pretty reasonable for someone who's figuring out how to make an airplane wing be shaped slightly more efficiently to not be thinking about how should the FAA regulate this, or how should airline seats be priced, or things like that. I think often it does make sense to kind of decouple those more technical and more sort of societal questions. And so to me, that's more what out of distribution is. This might be a technology where where if the technical progress outpaces society's ability to adapt, then the people who have just been doing that sort of decoupled technical work might end up having these huge societal consequences that sort of only they, or almost only they, were able to affect or to prevent. So to me, that's the way that I would think about kind of the OOD element here.

Yeah, that's interesting.

We'll continue our interview in a moment after a word from our sponsors. There's a growing expense eating into your company's profits. It's your cloud computing bill. You may have gotten a deal to start, but now the spend is sky high and increasing every year. What if you could cut your cloud bill in half and improve performance at the same time? Well, if you act by May 31st, Oracle Cloud Infrastructure can help you do just that. OCI is the next-generation cloud designed for every workload, where you can run any application, including any AI projects, faster and more securely for less. In fact, Oracle has a special promotion where you can cut your cloud bill in half when you switch to OCI. The savings are real. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking.

Join Modal, Another thing that I, I mean, you know, actually, I think last time we actually spoke, I was still participating in the GPT-4 red team. You were still on the board. And it's been quite a journey since then. I've definitely watched the company super close, and I feel like I've kind of been on this roller coaster where I've been many times sort of disappointed and kind of at times even outright scared by what I'm seeing. And then at other times I'm like, well, that's dramatically reassuring. And I'd say there's probably 10 episodes, and it might be like 5 to 5. The most recent 2 would be the publication of the obfuscated reward hacking paper, which I would put up there in the pantheon of most important and clearly stated warnings about how AI can go wrong if it's not developed with utmost care. And then at the same, maybe the same week or within 72 hours of that, there was the response to the White House request for comment on what AI policy should be. And there we got a rather, I would say, escalatory vibe, certainly with respect to China. Altman has said things like, it's our values or their values, there's no third way, which seems, again, like ruling out a lot of possibility space quite prematurely. And then also asking for things like just cancel all property rights so we can just train on everything. Because, again, if we don't do that, that China will. So it seems that there's this almost schizophrenic nature to the company, and I wonder how you understand that.

Yeah, I find it confusing as well. And I think it seems like it's gotten more pronounced over the last year or two. One explanation could just be that there's, which I think, as far as I know, is the case, that there's relatively a decent amount of freedom given to employees to tweet as they choose, to do some research directions and maybe write about those research directions. I think there's different processes that different things go through. But again, I haven't been close enough to those sort of on-the-ground decisions about what gets published when to have any kind of insider perspective on that. But I do think I agree with you that it's striking how different some of the voices from inside the company seem to be. And I wonder if— I hope that the more technical folks there are paying attention to the kind of policy messages that are being sent out by the company, given how much they sort of contradict each other.

How would you advise people that are there today? I mean, if you are inside and you are— and this could generalise beyond OpenAI, I think. I don't see any reason to think that, like, xAI won't have similar issues and potentially other companies that are generally held in higher esteem in terms of their safety practices very well could too.

Yeah.

It's all coming at us pretty fast. So If you are somebody inside a company and you're concerned about what you're seeing, what should those people be thinking about? And maybe flip side of that, or complementary question is, what should policymakers be thinking about in terms of protecting whistleblowers or facilitating? I think I also respect the idea that these companies should be able to keep some trade secrets.

For sure.

But then it also seems like the level of secrecy that was requested of me at one time was too much, where it was like the public sort of at some point does need to know what capabilities exist. So I don't really have a great sense for how to find that line. But maybe let's start with the policymakers. Like, what do you think the rules should be? And then we can go into, like, if you're in a position where maybe the rules aren't there yet or whatever, how should one as an individual think about Taking responsibility themselves.

So, so rules specifically around whistleblowing?

Yeah. I mean, you go, go broader than that if you want, but I'm, I'm definitely interested in how do we get things that the public really needs to know to come to light when, you know, company policy says it's a secret.

Yeah. Yes. I mean, I think the whistleblowing piece does like connect to other parts of the policy picture. I won't, I won't try and give like a comprehensive view on policy right now, but, but sort of starting from the whistleblower piece and expanding out. I mean, the way whistleblowing usually works is it's for illegal behavior. And so, you know, the SEC has very clear processes for if you're seeing financial misconduct, you can go talk to them. And lots of other whistleblowing processes are similar. So I think for policymakers, a big challenge I would want them to have in mind here is a lot of the concerns that we're talking about, or potential concerns, are behavior that is actually not illegal. And so where is the line for when can you whistleblow? What kind of behavior should be protected? So I think the best way to do whistleblower protections is to pair it with some kind of disclosure or some kind of, some kind of, you know, rules around what information needs to be shared. Because then it creates a clear standard for if the company is either not sharing information it's supposed to share or being misleading or, or, you know, inaccurate in the information it's sharing. That's kind of like structurally a simpler way for the whistleblowing to work, as opposed to trying to have this vague standard of like, if you're worried, you know, then call this hotline or something, because that's just so. squishy and hard, both for employees and also for the company, as you say, trying to think about trade secrets or other reasons that they don't want their employees just kind of blabbing. It's much more helpful to have kind of clear standard to compare against. So that's one thing I would say on the policy side is if you compare it with some kind of expectations, requirements around information sharing of some kind, or around processes that you have to carry out internally, even if you don't share the results. But then that leaves employees able to say, oh, actually, we didn't carry out that process. So to be a little more concrete, a version of this that I think can work quite well is the kinds of ideas around creating a safety and security plan, potentially publishing that plan or sharing it with the government. And then that creates an opening for whistleblowing activity if you're not sticking to that plan, basically, which again is just a little crisper than, you can whistleblow if you're worried, if you're concerned, if you think there's too much risk being taken. And the other thing for policymakers, I think that they need to keep in mind is that these are technical folks. They're not legally sophisticated. They might be scared. They might not have that much time. They might be working really hard. And so the, simpler and clearer the process can be in terms of how do you know if you're eligible? How do you know what your next step is? I think that kind of almost like, you know, UX set of questions matters a lot as well. If the user is a whistleblower, then what is their, what is their user experience? For the people in the companies, I mean, sort of very concretely, you know, there's, there's other, you know, formal whistleblowers who have gone on the record. And so those are definitely people you can reach out to if you're kind of looking for advice. I think more broadly and conceptually, a really important thing to keep in mind for the people who are contributing to AGI companies' work, to these frontier companies, is I think they are in a very powerful position. I think we've seen multiple times how powerful employees can be. And I heard this point recently, and I thought it was really smart, that they might be in the most powerful position they're going to be in because they're actively working to replace themselves and actively working to sort of hand away their own power. So if you're in one of these companies, I think there might be a temptation to sort of sit tight and wait until things get more serious. And that might be right, but I think it's worth thinking about, you know, both the case that you might actually be less powerful in the future than you are now if your work is more automated. It seems like in general, tech workers' power is going down right now in terms of the labour market and so on. And also there, I think it's perfectly likely that there will not be some sort of clear crisis moment in the future, but instead it might really be kind of boiling frog style. This is worrying. This is worrying. I don't like this. This seems a little bit dishonest. This seems a little bit too risky. And so just being realistic with yourself about if you're only ever going to do something, if there is some big moment, then being realistic with yourself that that might mean that you never do something. Maybe that's the right call, but not kind of kidding yourself about that, I guess.

If you were organising a union at one of the frontier developers right now, do you have a sense of what your demands I haven't thought about it much.

I think it's an interesting line of thought. I think, in general, I think we're starting to get to the point where, you know, there's a whole world around labour organising, worker power. And I think it really hasn't connected much with either sort of the technical AI world so far or the AI policy world so far. I think that's going to change. And I'm pretty interested to see kind of folks who have more of a background in that space and thinking about kind of how to use this kind of power and how to, how to, what kind of leverage is productive, how to represent a broad set of interests. I think we'll see more of that in the coming years. And I'm pretty interested to see where it goes.

You mentioned the sort of disclosure requirements. I mean, we had briefly a sort of 10 to the 26 threshold where at least you had to sort of say that you were doing it. And I think a little bit more about sort of what tests you ran and kind of how they came out. My sense is that's now gone.

I think it's unclear. Last I heard, it was unclear if it was gone because it had started to go through the official sort of notice and comment process in the government. So I think it might still be at the discretion of the agencies, so specifically Commerce. So I haven't heard that that is definitively dead. Certainly seems like it's on wobbly footing right now.

Yeah, they've announced the intention to remove it, at least. There's really nothing else, right, that is sort of an actual rule at this point?

There's the EU AI Act, which is in the process of putting together their code of practice, which I think involves some transparency around, I think for them it's models over 10 to the 25. and maybe some other criteria. And I think the details of what exactly is going to be— what exactly needs to be transparent about is still being hashed out. And there's also a lot of political pressure to water that down right now. So we'll see. But that is at least one other legal process that is underway.

Yeah. What do you think is— I mean, compute thresholds are one thing. What do you think is most important for the public to know? Like, I tend to focus on just observed behaviours. But I'm also really mindful that all of these things sort of have at least potential for unintended consequences with the observed behaviours. There's sort of the, like, well, we don't look, we don't observe. And so that can fall down pretty fast.

Yeah, I think there's a lot of things that could be helpful to know more about, to share. Certainly trade-offs in terms of what do you share with the public? What do you share with the government? How confident are you that the government will keep things you want to keep private private? So I think lots of details to be worked out. I think I tend to think in terms of test results, both for capabilities and risks, being pretty important to share. So what do we think these systems are capable of? I think also just being transparent about what kind of processes and protocols are you using to make sure that things are safe. Again, just disclosing, not having the government come in with a checklist and say, here's what you have to do, but just saying, tell us, how are you thinking about this? And I liked ideas as well from Daniel Coccatello and Dean Ball had a sort of joint piece on transparency. I thought some of the things they pointed out there, like looking at the model spec, what is your model actually being trained to do? also makes sense to me to have that kind of thing be shared publicly. Again, not because the government should be saying what it should be, but because this is a very fast-moving space. And the way I think about it is there's this huge information gap between the companies that are developing this stuff and everyone else. And so if you can narrow that information gap a little bit, I think that's decent.

Yeah, it seems important to me. And I mean, honestly, there's just a lot of work to be done even in educating people about what is already fully public. One of my kind of mantras is, like, if people understood what is already out there today better, they would probably have a healthier fear of what might be coming down the pipe in the not too distant future. And so to some extent—

Or even not a healthier fear, but a clearer understanding, clearer picture.

Yeah, I think a little fear is healthy, personally, honestly. But mileage may vary on how much that does for different people. I also sometimes describe myself as a Adoption accelerationist and hyperscaling pauser, meaning I love the tools that I have today, and I absolutely am trying to use them to the maximum. And I also am like, man, there's a huge overhang for society broadly on what we already have. And we continue to see so much more pulled out of models of a certain kind of resource input that I do think we're potentially starting to get close to the line of where certain thresholds could be crossed in ways that we just can't take back and might ultimately regret. I mean, obviously the canonical one is like, you open source a model that is later found to be able to help people make bioweapons or whatever, and you've just created a new sort of Damocles that hangs over everybody. kind of indefinitely. Yeah, I don't know. It seems like we're close. Do you feel like we're not that close to that? It feels to me like we're fairly close.

I don't know. Yeah, I feel like I also had 4 and a half months of parental leave over the winter, and so I feel like I'm still— I just came back to work a few weeks ago, so I feel like I'm still kind of reorienting around O3, DeepSeek R1, this new ChatGPT image release, Gemini 2.5. I haven't had a chance to try yet, but There's just so much. I feel like my picture is changing all the time. I definitely think we're at the point, to me, the version of compute thresholds that makes sense is not saying, oh, models over 10 to the 26 are dangerous, so we have to restrict them more, but saying, okay, we need some way to target the models that are newest and best and most capable, because those are the models where the sort of potential risks of potential unknown unknown risks of what can they do are highest. And so those models should be subject to a little bit more scrutiny. And so I do think we're at a point where it makes sense to say, look, it seems plausible that the next generation of models could be really concerning in a whole bunch of ways. Totally plausible they won't be, but probably we should be looking a little more closely than when GPT-3 came out. And that was just, I think, really unlikely to be anything worrying. And I think it was right that there were kind of no rules in place for releasing that kind of model. So, yeah, that's sort of the way that I'm thinking about it right now.

We'll continue our interview in a moment after a word from our sponsors. Hey everyone, Erik here. In this environment, founders need to become profitable faster and do more with smaller teams, especially when it comes to engineering. That's why Sean Lanahan started Squad, a specialized global talent firm for top engineers that will seamlessly integrate with your org. Squad offers rigorously vetted top 1% talent that will actually work hard for you every day. Their engineers work in your time zone, follow your processes, and use your tools. Squad has frontend engineers excelling in TypeScript, React, and Next.js ready to onboard to your team today. For backend, Squad engineers are experts at Node.js, Python, Java, and a range of other languages and frameworks. While it may cost more than the freelancer on Upwork billing you for 40 hours but working only 2, Squad offers premium quality at a fraction of the typical cost without the headache of assessing for skills and culture fit. Squad takes care of sourcing, legal compliance, and local HR for global talent. Increase your velocity without amping up burn. Visit chewsquad.com and mention Turpentine to skip the waitlist.

This may be a good opportunity to talk about your concept of adaptation buffers, which is, I really like that phrase and notion. And I think it helps deconfuse, or hopefully, I think it will, I'm hopeful that it will help deconfuse people about the sort of apparent sort of hard to reconcile idea that these AI advances that we see are really hard to achieve and cost huge amounts of money. But then we also see that DeepSeek and otherwise, things are becoming dramatically cheaper. So maybe set up that dynamic a little bit and talk about this adaptation buffer and how you see the window of time that we have to kind of adjust to capabilities advances.

Yeah. I think the important underlying thing here is I'm generally a believer that humanity, you know, society at large is very adaptable and has adapted to a lot of things in the past. So a lot of things, you know, it's sort of a cliché that often when new technologies come in, whether it's the printing press or the television or the telephone or whatever, people cry that the sky is falling and this is a terrible thing and it's going to ruin everything forever. And then it doesn't. And so I think the starting point here is thinking that for a lot of different kinds of technology, we actually have a pretty good track record of sort of digesting them, figuring out how to incorporate them into society in a good way, how to have a set of institutions or barriers or social norms around them that make them sort of positive rather than negative, or on balance positive. And I think with AI, I think there are a set of questions around AI that seem less that way to me. So I think the whole thing around, are we building something that is a successor species or more intelligent than humanity by a long way? Those set of questions, I think, are, do seem potentially very different to me, but I get a little worried. So, and where the, this, this post, which again, you know, you saw a draft of, hopefully now on, on Substack live by the time this comes out, a place where I think people sometimes overstate how new this is, is when it comes to AI misuse. So this idea you mentioned of like, are you going to have some open source model that can help anyone create a bioweapon or can really make it much easier to carry out really sophisticated cyber operations, hack critical infrastructure, so on. I sometimes see this impulse in kind of AI policy circles of, oh, that's too dangerous. That can't— we can't let that happen. Can't let that out. Can't have that technology be proliferated. It's just like, sort of in an absolute sense, too dangerous. No go. Big problem. And so what you see from that is people talking about wanting to ensure or wanting to work for, work toward nonproliferation of these systems, meaning, can you prevent them from being distributed in any way? Can you prevent access from getting beyond a small number of very controlled people? And yeah, your question about kind of DeepSeek versus the frontier giant cluster training models gets at why I think that is really problematic, which is we have this weird dual dynamic at the moment in AI development where it is both true that developing the next best model, the push frontier, being at the cutting edge, keeps getting more and more expensive in terms of compute power, Also in terms of the number of, the amount of expertise you need, that you need a really absolutely top team of researchers and engineers. That kind of amount of expense is going up and accessibility is less and less accessible. At the same time, every time we reach a certain point on that development curve, every time we reach a new set of capabilities, it's very expensive the first time we build it, but then it gets cheaper and cheaper and cheaper. So DeepSeek was an illustration of this dynamic where they didn't actually build a model that pushed the cutting edge and was better than anything we'd seen. What they did was they matched what some US companies had had depending on how you count, a month or 2 ago, if it was the reasoning model alone, more like 6 to 9 months if we're looking at the base model. And they had done that at a lower price point. We can fight about what the actual price point was, but I think that's not the point for here, for this situation. And the reason this matters is, I think, if you're taking, if you're trying to have a policy approach that says we're going to prevent anyone from having access to a certain kind of model, but that model is getting cheaper and cheaper and easier and easier to get your hands on, that means that your kind of policy regime is going to have to get more and more invasive to prevent people from having access to that. So the comparison I give in the post is we have nuclear nonproliferation. It works pretty well. Only about a dozen countries have nuclear weapons. That's pretty good, I think, compared to what a lot of people would have expected in the '50s. But imagine how that could have worked or how that would have not worked if nuclear technology were improving over time, getting much more efficient at the same kind of crazy rate that the AI technology is, such that you needed less and less uranium to build a nuke and you needed it to be less and less enriched. So right now you need quite a lot. And you need very, very enriched uranium to, for it, for you to actually be able to build a bomb. Imagine if that number is going down over time. At some point, you would need to have the IAEA coming and inspecting, you know, what you're doing in your farmhouse because you have a couple acres of land and, you know, across that couple acres of land, there's enough uranium in the soil that in theory you could build some like very efficient, you know, teeny tiny nuclear bomb. That, you know, that's a totally untenable regime. That the nuclear nonproliferation we have right now only works because there's a limited amount of physical material. That people don't necessarily need for other purposes, that needs to be enriched in these very highly specialised facilities. And that's really not the case for AI. But I think if we're thinking, instead of thinking purely about how do we prevent people from getting access to this, if we're thinking more about how do we make the most of the time that we have to adapt, to build our societal resilience, to be doing things like scaling up our vaccine production infrastructure or scaling up our outbreak detection infrastructure. So, you know, more wastewater monitoring. How do we—

Yeah.

Have more test kits available in more places around the world so that if someone does use a bioweapon, we can more quickly identify it, more quickly respond to it. Similar things on the hacking side of things. I think that approach of how do we maximise the value that we get out of the time we have, as opposed to how do we lock this down and prevent this sort of capital B bad technology from being spread, is both going to be more productive and also less invasive. I do think it might not be enough. It might be that we're just in a really bad situation if AI progresses incredibly rapidly. But I think it's a much better approach and a much more sort of societally healthy approach.

Does that imply a certain pessimism about technical solutions? Like, another one of the blog posts is about the kind of fundamental challenge of just getting AI to do what you want it to do at all. I'm old enough to remember the Eliezer-led discourse from years past on how These things are going to just turn us all into paperclips. Of course, that was always kind of a caricature, but there was, I think, a felt sense that we have kind of a genie problem where we have no idea how to communicate our real values and real intent to a system like this. And so they're just going to be extremely unwieldy. I would say relative to that, I've been very pleasantly surprised on the upside that Today's models do seem to have a pretty good internalisation of human values broadly and sort of a general respect for norms. Although at the same time, one always has to be situationally aware. We are now seeing many of the problems that Eliezer et al. sort of predicted back in the day of, like, once they have values, they also sort of seem to be inclined to try to protect them by lying to users if that's what's needed, or trying to subvert a training process that they understand themselves to be going through. So this is another one of these roller coaster rides where I feel like, man, it's gone way better than I thought. But also some of the doomsaying is starting to be proven correct. But I guess I would be optimistic that if we had a sort of adaptation buffer that was maybe a little bit more required or imposed by authorities, as opposed to just kind of trusting the sort of natural sawtooth motion, because it does seem like DeepSeek might be about to challenge that, or at least those sawtooth time intervals might be getting really short. Yeah, I would definitely love to see all the great things of wastewater monitoring. And the fact that we haven't done anything really about it the last pandemic does not bode well.

Indeed.

But I also kind of feel like we need that time to maybe sort of figure out how does one distribute a frontier model with a better sense of what capability, making unlearning work or making mixture of experts work in such a way where you can distribute all but 2 of the experts or something, so that certain capabilities are redacted while the core utility of the overall thing can be diffused. Long question. I guess the core of it is, do you think we'll see technical solutions that could allow us to sort of square the circle and have free distribution, but also, you know, a pretty confident sense that like what we're distributing isn't going to come back to bite us.

I think the thing I want to challenge is the focus on only technical solutions, because this does come up a lot. It's like people talk about the offence-defence balance of like, you know, AI for cybersecurity, for example, does it help hackers more or defenders more? Same for bio, people talk about, does it help you, you know, design a new vaccine as much as it helps you design a bioweapon? And I think that's, I think that's just one small part of the picture. I think the thing that I'm trying to point to with the idea of an adaptation buffer is that a lot of the ways that we, specifically if we're talking about these misuse risks, so are there going to be terrorists who build a bioweapon? Are there going to be hackers in their basements who can suddenly bring down the US power grid? A lot of the things we can do about that don't actually relate to AI. They relate to what is going on in society, or if they do relate to AI, it's not actually the frontier model. So it's like, can you, you know, can you use AI tools to look at, you know, large-scale disease monitoring data, for example, and notice anomalies or something like that? For bioweapons. It's like if you talk to people who work in biosecurity and bioterrorism, there's a lot of stuff that has nothing to do with AI, nothing to do even with biomaterials. It's things like, who is the FBI tracking? How good are they at detecting plots before they get very far? So, yeah, I think certainly as the technology advances, there will be sort of new defensive tools that become available to us, and we should make use of those. But I sometimes think that the discussion here gets too focused on only those, as opposed to looking at kind of broader parts of the picture. And, you know, if we are also, if we're talking about the AI tools, I think a huge part of the discussion needs to be about the application and dissemination of those tools. So for example, in cyber, I think it's sort of less a question of what can the absolute most advanced model do for cyber defense and more a question of how can you have well-designed, ready-to-ship, you know, defensive tools that you can get in the hands of operators who are not very sophisticated on AI. So you're people who are running your, your water treatment plants and your power grid and your chemical plants and so on. How do you have those, even if they are AI related, it's not a kind of capabilities question. It's more of a dissemination application question for how you get those defenses kind of out or into the real world.

Yeah, I just talked to somebody not long ago who is applying language models to the challenge of formal verification of software. And it sounds like there is a multiple order of magnitude speedup that is becoming possible in that domain. And it does feel like if we just have enough of an adaptation buffer, then a lot of the things that people are most worried about could really be brought dramatically down in terms of the, you know, the absolute magnitude of the risk.

Yeah.

And so, yeah, it's just a question I think of like, Are we investing enough in that? Almost surely not. And, you know, do we have enough time before the sort of, you know, next disruptive thing hits?

For sure. And I mean, not to sound too optimistic here, I mean, I do think sometimes I hear from folks, you know, in the cyber domain that, well, it's fine because AI is going to help defenders more than attackers. So it'll all be good. And I think that is also a, you know, in my mind, a quite naive take if you're looking at the sort of dissemination and real-world use case here. Where I think even if that is ultimately the case long term, for example, if you can use AI to develop formally verified code, you're still likely going to go through this sort of dangerous transition period where it's just obviously going to be much quicker. The time lag between some model is released and some disaffected teenager in the basement can use it to carry out an attack is going to be much shorter than the time lag between the model release and when Yeah, the, I don't know how many it is, thousands of critical infrastructure providers in the US can go through their very old code bases where they have this complicated division between their IT and their OT operational technology and what gets updated when, you know, it's, it's, they're not going to be nimble. They're not going to be agile. They're not going to have all these defenses kind of wrapped up or built in really quickly. So I don't think the fact that these advances seem promising and could help defenders means that we'll get away from that sort of dangerous transition period. But I do think that thinking in terms of a transition period, Rather than like a permanent state of danger helps us respond much better.

So have you seen any regulatory proposals that you like? I mean, you mentioned Dean Ball, friend of the show. He recently put out a post I thought was quite interesting, basically proposing a regulatory market type structure where the government would essentially accredit or sort of authorise private regulators to sort of, let's say, approve the practices of AI developers. And as long as the developers were able to keep the private regulator happy, then they would get some sort of liability shield. And then, of course, that could be withdrawn if they didn't do it. And even the regulators, the private regulators' authorisation could be withdrawn if the state found them to be out of compliance. I understand now, I haven't read the text, but I understand there is now a California bill that is kind of moving in that general direction. Interested in your takes on that. Or there's also proposals to really embrace liability and go the other way and say maybe you should even be liable for close calls, because the close calls could be so big and bad that even probabilistically, if it was a near miss, maybe you should sort of face liability consequences for that. So, I don't know, react to those or tell me any other policy proposals that you think are particularly promising.

Yeah, so I haven't seen a policy, a sort of proposed regulatory regime to really comprehensively manage the risks that we're facing from frontier AI systems. I haven't seen one that I like. I think it's a really big problem. It's a complicated problem, and it's especially difficult because we're not actually sure exactly what the problem is or when we'll face it. So to me, the policies that I have seen that I'm interested in are more kind of building blocks that put us in a better position for the future, as opposed to sort of solutions, which I think is in some ways appropriate, given that there's so much uncertainty about the technology, though also certainly scary, given that one way the future might go is we might have very little time, in which case maybe some initial building blocks right now are going to be far from sufficient. So I think the proposal by Dean is an example of a building block that seems potentially pretty useful, seems certainly better than nothing. I think it's It's, you know, he's, he's written it deliberately to be something that could be implemented at the state level, which certainly seems far, far, far more feasible than any kind of federal legislation. I don't know that it does as much as we would need to target those kind of cutthroat financial incentives and also just like other many incentives for the frontier developers to do anything other than push ahead as fast as they can. But I, you know, I think it's an interesting idea and it does seem better than nothing. And then other kinds of building blocks. We talked about transparency. I do think that is the kind of thing that can— doesn't in itself solve any problems, but does put many actors in a much better position to help solve problems as they arise down the road. Likewise, I think there's a lot of stuff that can be done that isn't sort of regulatory, isn't obliging anyone to do anything, but things like funding, trying to really boost our science of measuring AI, I think could be a target of research funding, potentially something for a focused research organisation or things like that. Similarly for interpretability, of course. And alignment research and lots of things like that. I think even just, even like building blocks as basic as trying to get more technical capacity into governments so that they're able to kind of handle things as they arise and make better decisions as the technology progresses. These are all kind of, again, individual components that don't add up to a comprehensive solution, but that do, I think, put us on better footing for the future. So that's the terms that I'm thinking of right now.

Just as an aside, it was Gabe Weil. I had to look up and make sure I had his name right, who's arguing for this sort of embrace of liability. And I hope to do an episode with him about that. But in the meantime, he's written about it for folks who want to go into that in more depth. You know, one thing that I thought OpenAI always had right was the idea of iterative deploy— iterative deployment. You know, the idea that like, if we sort of develop superintelligence in secret and then, you know, drop it on the world one day, that's going to be like far more disruptive than if we sort of launch a bunch of products along the way and people can see what they're good at and get used to them and so on and so forth. That itself now seems to be sort of at risk, both in the sense that Ilya has gone off and started a company that has an explicit strategic statement that they are not going to release anything until they achieve superintelligence, which seems crazy, but also, to borrow a term, strikingly plausible that they might actually achieve it. And even at OpenAI, I was really kind of taken aback by Sam's recent statement when they released 4.5 that let us know if you like this or not, because we've got a lot of other models to build and this one is pretty compute intensive. And so if it's not really doing it for people, then we might take it offline and focus our resources on building more models. This has me thinking we might actually be at sort of risk of these companies kind of closing down what they put out into the public and just kind of going for broke totally internally. Miles Brennich is also formerly of OpenAI, has made some, I think, kind of cryptic comments about sort of the rising importance of internal deployment decisions. So I'm sure you'll find major flaws with this, but one idea that I've had is, like, could we put a sort of speed limit in place? Not an absolute speed limit necessarily, but a relative speed limit where we might say you can only develop a model that is so many times bigger in terms of resource inputs than the biggest one you currently have deployed. And if you want to go bigger than that in your development, you got to deploy something that's kind of helping us as the rest of society, understand where all this is going so that we don't have these sort of not exactly unilateral, because we know that we've got multiple voices inside the companies, but these very small kind of concentrated decision makers without much at all in the way of visibility, just going for something that they seem to believe could be like a world takeover capable technology. So I guess, how big of a problem do you see that possible retreat from iterative deployment being? And do you like my relative speed limit solution or have any others to address that?

Yeah, I agree with you. I think iterative deployment in general seems like a good approach, with, of course, the caveat of at some point you should probably have some criteria in place for when you would not just iterate. Because the idea of iterative deployment, I think, is to kind of put it out in the world, see what happens, adjust, try again. And I think that's great as long as you're confident enough that what you're putting out in the world is not going to have any really severe irreversible consequences. So the question is, how do you know? When do you decide to do it differently? I'm not sure that I see a retreat from that. I think it, I don't know that it's ever been like a, certainly was or has been OpenAI's MO, but I don't know that it's ever been kind of an across-the-industry approach. Your idea, I think it's interesting. I've heard kind of similar proposals. Could also be things that kind of companies adopt internally in terms of how much scale-up are you going for at a given time. It sort of feels to me like it runs into the same problem that so many of these run into, which is one, how are you going to implement it? Like, to do that in the US, you would certainly need legislation, I think. I don't see how you would do it without legislation. We're not going to get legislation. So then how do you do it? And then also, doesn't it just mean that we lose to China? It's going to be the other, you know, the other big question. So I think conceptually, things like that could work, maybe, or could make sense if they were implementable. I don't really see how they're implementable. And then if they were, then I would want to come back to this question of like, okay, is this actually the right approach? For me, I tend to be pretty pessimistic about sort of solutions that involve slowing down at some kind of the input level, meaning slowing down, pausing for a certain number of months, or slowing down how much, how quickly you're advancing, versus slowdowns that are sort of conditional, more of this kind of like if-then approach of, we're not going to keep progressing until we have hit this level of understanding of our system or this level of risk mitigation. Which I think a lot of the companies now have in place or have made voluntary statements that they will think in that way. So that would tend to be my preferred approach. But I think this kind of thing, I think we're in a rough situation right now for anything that will involve kind of cross-industry coordination because there's so little political appetite at this moment. Maybe that'll change. Probably that'll change. But for now, it seems hard to imagine.

Do you think it all, at the end of the day, is about the fact that the policymakers, the members of Congress, whatever, just don't buy it? I mean, it seems like if they really believed what Ilya is saying, that he's going to not release anything until he has superintelligence, and he thinks that's going to happen in the not too distant future, then they wouldn't just sit back and be like, well, let us know when you got the superintelligence, right?

Yeah.

It seems to me that they just fundamentally don't believe it. And that's like the biggest barrier to something. And there's a lot of questions, obviously, on what that something should be. But I find the notion of, like, well, we're not going to get legislation, to me, that still feels like a sort of education challenge. Again, I think if people had a better sense of what already is deployed, they might be like, yeah, I don't know that I'm comfortable with Ilya. And how many people even work there? Are we talking like a couple dozen, potentially, maybe up to low hundreds now? It can't be that big. And then we're just going to sort of wait for them to pop up with superintelligence. That seems so crazy.

I agree with you. I think it is in large part a question of how seriously do people take the possibility that AI will get as good as someone like Ilya thinks it will. But I also think that US Congress is really broken right now, not in an AI way, but just incredibly dysfunctional. A friend of mine who really knows his way around Congress has worked on the Hill for years and years. Said that he thinks that the US House of Representatives is less functional than it's been since after the Civil War. So really dysfunctional, separate from AI. So I agree with you. I think there will be most likely windows that open again if the technology keeps progressing. And I think people will change their views and change their level of urgency around it. And also, I don't know that will be enough to actually get thoughtful, productive regulation through. Congress at the federal level. It might be enough to get some kind of bill. And I certainly know people who are working on, you know, sort of like a Patriot Act for AI, where the Patriot Act was put in place after 9/11, but had really been developed in advance. And sort of setting aside the merits of that bill, I think that approach makes sense of saying what you're going to get some window after some crisis. I think that is a reasonable way to be thinking about AI policy right now. I think it'll be a big lift, even in the wake of a crisis, to get the right kind of productive useful legislation through, not just kind of fighting the last war or doing a bunch of stuff that people wanted to do for other reasons. I mean, the other model for this, in a past era when it seemed like we might actually get something through Congress, the other model for this that I have thought about is sort of trying to avoid a Three Mile Island situation, where Three Mile Island as a nuclear disaster seems to have basically killed the US nuclear industry because the safety regulation that was put in place afterwards was just too onerous and wasn't comparing to risks posed by other sources of energy and wasn't actually kind of commensurate to the level of risk. And instead it just sort of shut down the whole industry. And I think that that story, in my mind, should be motivation for AI developers to want to have some more safety guardrails in place earlier to prevent that kind of accident, or to mean that if that kind of accident happens, you have a better answer, or legislators have a better answer for the public of, here's all the stuff we did in advance, and this really was just a freak accident. I think right now we're on track for a massive knee-jerk overreaction when something happens, but we may have passed the point where we can prevent that at this point, given how unlikely regulation looks. I don't know, maybe the states will exceed my expectations. Maybe there'll be kind of more useful stuff that comes up there.

Yeah, I say something similar to AI developers and investors, not even so much at the frontier level, but even just like your rank-and-file app developers all the time. Right now, the voice AI world is totally taking off. And also, I think, if they're not pretty quick to sharpen up how they handle the technology, we're headed for a world where there are going to be some high-profile things because the voices are getting really good. And you can still go to all these products and just drop in whatever voice you want, click the checkbox, and the next thing you know, you're calling as Trump or as Taylor Swift, I did it with Biden, you know, during the election as well. Just call anyone, say anything. There are zero guardrails on these products. And, you know, that's not going to be good for the industry if it, uh, you know, they're like, definitely, again, playing with a certain kind of fire that I think self-interest alone would dictate better governance or stewardship of such powerful technology. But that seems to fall on deaf ears.

And one thing I think is maybe gets a little bit glossed over in sort of the kind of more detailed or like wonky discussions of this, people who are thinking really hard about the risks, thinking really hard about the policies, is I think there's, it's quite unpredictable or quite maybe unintuitive which kinds of things will catch the public imagination or will create sort of a perceived crisis. So, you know, I think it seems to me like in the, after the GPT-4 release, one of the things that really caught fire was this conversation that Kevin Roose had with, with Sydney and, you know, was trying to get him to leave his wife or whatever. And, you know, the AI people I know who read it, read that transcript were like, look, he, he really led it there. Like he was really, kind of prompting the model in a way that got it to go there. It's not that surprising. It wasn't dangerous, didn't actually harm his marriage at all, probably was a huge boost to his career. But that is what really kind of caught attention and got people worried. And likewise, I don't personally feel that worried about the risks from voice synthesis. We could talk about that maybe. But I agree with you that it's the kind of thing that is very vivid and very easy for people to latch onto. And so it could be the kind of thing that produces a backlash sort of disproportionate to the actual—

Yeah.

risk or harm of that specific use case.

Well, in the interest of time, let's keep moving. You alluded to maybe the one thing that can unite the Congress, and that is the threat of China. And certainly a growing number of my AI conversations sort of get backstopped or kind of run into this final barrier of like, well, but China, we'll lose to China. One thing I think is very much underdiscussed, I'd love to hear your take on, is what is the threat from China? I don't get great answers to this usually, and I sometimes joke, like, am I supposed to expect that my grandkids are going to be speaking Chinese if we don't develop AI as fast as we can? How do you understand the threat from China to the United States, the West, like my values, my way of life?

Yeah, I've heard some of the conversations you've had about this, and something that jumped out to me there, Coming from the world I come from, this sort of national security, foreign affairs, geopolitical kind of viewpoint, is you seem to be starting from a point of view of like, well, the US and China should be friends unless there's some strong reason otherwise. And I think for a lot of people with experience in international relations, defence, military history, the starting point is more, okay, we're an established power. China is a rising power. Who has power on the world stage matters a lot. And by default, if they are coming in to be rivalling us in terms of how much power they have and how much they can throw their weight around on the world stage, by default, it's going to be a more hostile relationship. And maybe you can have exceptions to that. So, you know, the, the classic exception that, from last century was as the U.S. was rising and kind of eclipsing Great Britain, as, you know, the British Empire was crumbling right around when the U.S. was really coming into the height of its power. But that was a relationship that was actually very close, actually very cooperative. And so there wasn't a huge amount of tension there, but that's really unusual. I don't know how to give a, I don't want to, you know, go off on a 20-minute tangent about sort of U.S.-China history and the specifics. I think here there was a real effort, real effort in the '80s, '90s, 2000s to try and usher China into a position in what was seen as sort of the the rules-based international order, which was, I guess the background here is usually for much of history in many places around the world, most things operated under a might makes right framework. So whoever has the most power, whoever has the most guns gets to push around everyone else. The second half of the 20th century was a big exception to that, where it gets called the Pax Americana or other things where the US was sort of the leading power in the world. There was also the USSR for a good chunk of that. But the U.S. was instrumental in setting up this set of institutions, this way of countries relating to each other. There's a, you know, U.N. Charter, which puts sovereignty central. It makes sovereignty of countries central so that you can't just invade other countries and do whatever you want instead. You know, borders are sacrosanct and so on. This whole system that was put in place in the second half of the 20th century with the U.S. leading, which was seen as a big improvement on the might makes right. Sort of default. And I think we're in a weird place right now where if you look back at the last few years, few decades of US-China relations, I think a lot of the hostility now comes from the failed attempt to bring China into that order. So this is, you know, of course there was, you know, warming throughout the '80s when Deng Xiaoping was doing his reform and opening strategy to try and, you know, make China more market-focused and freer. Big hit to that in 1989 with Tiananmen Square, then sort of more optimism again in the 1990s, culminating in China entering the World Trade Association. The classic phrase, I forget who used it first, was trying to make China a responsible stakeholder in this system. And then that all kind of fell apart in, you can date it different ways. Certainly by the time Xi came into power in 2012, that was starting to crumble. And Xi has accelerated that where China has become more illiberal again, has become more hostile in the South China Sea, more aggressive. really making it clear that they want no part in this sort of quote unquote rules-based international order. And so that is the context for, I think, why they are seen in a more hostile light. The challenge now is that the U.S. itself is retreating from that rules-based international order and moving back into a might makes right kind of frame where the idea is, well, the U.S. has all this power. We have, you know, the dollar as the world's reserve currency. We have the biggest, best military. So we should be able to get what we want. So sort of in that light, it comes, it becomes confusing again. Like, why would we have a hostile relationship with China? But I think that is very much a kind of set of changes that are still in process and that the system hasn't quite figured out how to orient towards yet. Yeah, and I mean, I think, I think it definitely comes down a lot to, I guess there's sort of 2 big questions. One is China's position in international institutions and its relationship to sort of the whole world. And then there's also these sort of territorial military questions around, is China going to try and take Taiwan? Is China going to be aggressive around Japanese, Filipino, Korean assets, sort of more of a hard power military set of questions as well. Are they going to, for example, coming back to the sort of rules-based international order, are they going to damage the freedom of navigation norms that the US has been so instrumental in preserving that are so good for international trade? For example, are they going to prevent people from using what they claim to be their waters? Or there's a whole kind of set of questions around who has power, what are the rules, what are the norms? Where China is very clearly not wanting to cooperate with the U.S. on that. And so the U.S. has shifted since, since 2016, 2017 into a more confrontational posture.

I have to say, though, that, you know, all of that stuff, like, I'm generally familiar with that history. Plenty of things we can, you know, complain about China doing. You didn't even mention stealing all of our intellectual property, you know, which is definitely a rightful point to which, you know, many American business leaders and others object. So, you know, there's wrongdoing inside the Chinese nation as well that we can point to and I think, again, justifiably and rightfully criticize. But I still don't quite get the flip. And I'm not sure if I should be understanding what is happening now as just like strategic communication where everybody's trying to influence the guy that I call he who must always be named. But both Altman and Dario have done a pretty dramatic flip where there's video evidence of them not that long ago saying, Everybody, you know, is too worried about China. Like, a race with China would be one of the worst things. You know, we should make our own decisions about what's right to do and not worry so much about them. And we can, you know, point to these clips. And now we've got both of them basically saying, you know, we got to go as fast as we can or we're going to lose. And Dario is even like saying, you know, we can't accept a multipolar world. We need to maintain a unipolar world. And I sort of am like, man, who's really being aggressive here? You know, I haven't heard China say they want to be at the center of a unipolar world. I've only heard Americans say we want to be at the center of a unipolar world.

Depends who you read and how you read it. The Chinese, I don't know. But I mean, I agree with the idea of technology, right?

Like, they're not— we're the ones that are saying, like, we need to box them out and have this unassailable lead. Meanwhile, they're just like, open sourcing everything. It doesn't seem like they're trying to dominate us, as far as I can tell.

I mean, open sourcing, I think, makes a lot of sense if you're in the following position. I think it makes a lot of sense to be trying to show off how good you are, trying to attract talent. So I don't know that that is just coming from pure goodwill and lack of competitive spirit. I think that is more that makes a lot of sense if you're not leading, and it's much less clear how to handle openness if you are leading. I agree with you that the position change and the rhetoric change from a lot of the top CEOs has been pretty striking. And I think, honestly, I think it's just the path of least resistance at this point. There's so many different issues to handle here with AI and so many, so little agreement on what to do about it. And the one thing that people can agree on is, well, we got to beat China. And so it doesn't surprise me that the companies are leaning into that message as a way to say, look, we should get funding, we should get government contracts, we should have no regulation, we should get shielded from liability. Yeah, that's the explanation that makes most sense to me right now. And I'm sure it also depends on different individuals, how they're thinking about those specific statements.

You've got a paper coming out on decision support systems in the military. And I think this is really interesting as a, okay, yeah, we got to beat China, whatever. But we also have to confront the fact that the systems that we have, for all the upside, which well on the record embracing that and use it every day, yada, yada, yada, also have a lot of problems in terms of their reliability, their hallucinations, again, now scheming against their human users in some cases. And I, for one, would not want to go into combat with an AI buddy until I was quite confident that all of these sort of scheming issues were well and fully resolved.

Not to mention hallucination and reliability.

Yeah, there's a lot of issues, right? If I'm taking this thing and trying to really rely on it in a genuinely life and death situation, and I say this as a top tier enthusiast, I would say I wouldn't want to use it in that sort of context. So that also seems to be like a big disconnect to me in terms of how the AI debate sort of seems kind of, especially with respect to China, seems like a little bit decoupled from the actual reality of systems that we have. So I'll shut up, give you the floor, and just tell us about your work on decision support systems and what we can and probably shouldn't be relying on them for.

Yeah. So this paper is led by a colleague of mine called Emmy Prubasko. And She is a super interesting person to be working on this because she actually served in the Navy, and her job in the Navy was operating Aegis missile defence systems on board US Navy ships. So this is actually, people really don't, a lot of people don't know kind of that we actually have essentially autonomous weapons systems already. So Aegis is a system on board a ship that looks at incoming missile fire and automatically identifies and then takes out incoming threats, not based on deep learning, developed in like '60s, '70s, employed in the '80s. So yeah, super cool to get to work with Emmy on this paper because she's just bringing such a grounded, informed perspective. The paper is about what gets called decision support systems. And what I think is really important here is to move that discussions about AI in the military don't just get stuck on this autonomous weapons question, because there's so many things you can use AI for in the military. And so this category, decision support, is another, is a big, broad category of, that's one example of other uses. And it can mean a lot of different things. Basically, it's what it sounds like. It's systems that are helping commanders or operators make decisions. And so there's a long history of different kinds of tools like this. And recently there's been interest in how do you kind of add AI or how do you use AI to perform some of those decision support functions or upgrade existing decision support systems. Big range of different types of things we could be talking about here. So on the simple end could be things as simple as You look at a photograph of an area and you use AI to determine the viewsheds in that photograph. So what a viewshed is, is basically, if you're a sniper, where can you see? What, you know, where is in your field of vision and where is not in your field of vision, for example, or if you're, you know, not necessarily a sniper, anyone. And so that could be a computer, you know, vision sort of segmentation image that is doing some kind of processing of an image to figure out what is visible from where. That, that counts as, you know, in our definition, decision support. Likewise, you could have a system doing Medical triage. So you have, you're on a battlefield, you have a bunch of wounded. How do you figure out who to treat first, where to take them, that kind of thing? Likewise, decision support, or looking for, looking at movements of ships and doing kind of anomaly detection. You could be using, you know, anomaly detection has gotten way better over the last 10, 15 years. Could be using kind of upgraded AI systems to do that kind of thing. And in the paper, we basically sort of look at a whole range of different sort of AI-based decision support systems that are being either used or advertised. So on the, that's on the sort of simple end, the ones I talked about so far. On the more complex end, you have companies like Palantir, also Scale AI, advertising large language model-based systems that are really trying to be more of what you described of this kind of all-purpose battle buddy. Or at least that's how some of the early marketing looked. They've changed their marketing since then to look a little bit more restricted. But this could be something that some of the early videos would have a huge range of potential functionalities kind of in these demo videos. They might include Things that to me, like, make a good amount of sense, like, uh, you're using a natural language interface to access some, like, clear, clearly documented information elsewhere. So, for example, you've identified some, you know, enemy movement and you want to know where's your nearest unit, like geographically, where is it? So you could ask that and then it could refer to some database or, you know, some other system and show you, okay, here's the nearest enemy unit. And then you could say, okay, and how many, you know, missiles of such and such type do they have? You get access to that information. In theory, that all makes good sense to me. But these demos were mixing that in with things like, okay, now generate 3 courses of action that are non-escalatory and having the AI, presumably the LLM, just like write out potential courses of action for how you could engage this, this enemy with what kinds of, you know, tactics from what kind, with what kinds of routes with this caveat of non-escalatory of like, how the hell does the LLM know what's escalatory and non-escalatory? Like what, who, who is making these decisions? How is that being evaluated? And these kind of different use cases, or, you know, another one was like looking at Chinese writing, or you're looking at the writings of some, some country and figuring out, you know, what they think about some set of questions. And these use cases were all kind of being mixed together in these demo videos. And so part of why we wrote this paper was to say, look, this is a category of use for AI that makes a lot of sense. There's a lot of ways that it can make, help the military work better, help clarify, you know, get rid of fog of war, reduce fog of war. But these systems are not perfect. And, uh, there are a lot of ways you could use them that could go badly for you. And so how to think about that. And so, you know, just briefly in the paper, we talk about kind of 3 types of considerations that we suggest be considered if you're thinking about whether to use a system of these. So the first is scope. What is the scope of this system? How tightly bound is it? How well can it be tested for that particular set of activities versus is it more sprawling? Is it more general purpose? Is it more kind of whatever happens to come to mind that you might want to type to your battle buddy? Second is data. So what data has it been trained on? How confident are we that data reflects the situation that you're in? A huge problem for military operations in general is both that you're likely to be in sort of situations that are novel, and also that you have an adversary trying to mess with you. So how do you think about whether the data that a system has been trained on are going to be really reflective of the real-world situation you find yourself in? And then the third kind of factor that we talk about, which I think is really, really important and really often neglected is this kind of human-machine interaction component. So how is the system designed to help the person operating it understand what it can do, what it can't do, help them make decisions that are good decisions, help them not overtrust? And there's a long history of kind of user interface, I think, being undervalued in military circles and then contributing to unwanted outcomes as well. So Yeah, essentially we're trying to lay out this sort of category of systems and describe both why militaries want to employ them and how they can employ them productively rather than counterproductively.

Do you see any stable equilibrium in the future here? I mean, I was struck by— I'm sure you read Dan Hendricks and Alex from Scale was one of the co-authors, Eric Schmidt. The MAME thesis, I, sad to say, didn't find that super compelling as an idea of what a stable equilibrium could look like. Although I really applaud the idea of trying to articulate something that could be a stable equilibrium. But it just feels like where we sort of inevitably end up, especially if we don't get on the same page with the countries that we're currently most fearful of is what I'm starting to call AlphaGo for the army, which is just like self-play, simulation-driven kind of go to superhuman performance by just having these things battle it out amongst themselves and hill climb to a level that human tacticians, especially when you consider speed, just can't get to. And that's like one way we get to Skynet. And that just seems like a pretty bad situation where we have these sort of inscrutable but hyper lethal systems that we build in theory to defeat an adversary. And maybe it even happens that way. But boy, that seems like another way that we add a pretty scary sort of Damocles to all of future humanity's life. And without giving them any chance to vote on it, obviously. Is there any way to avoid that, though? I mean, right now it just seems like we're sliding into that, and I don't love it.

Yeah, I mean, many thoughts here. The MAME paper was really interesting, basically, for folks who didn't read it. They had a few different things in there, but the key idea, this MAME idea, I think was like mutually assured AI malfunction or something like that. The idea was, which I think there's a I think there's a correct core to this, which is if one country is going around saying, hey, we're going to develop superintelligence and then we're going to rule the world, the universe forever, that's going to create an incentive for other countries to react and respond. And certainly, I think it could be quite stabilising if it's true that it's much easier to sabotage an AI project than it is to build, to keep building it. That could be a stabilising dynamic, both because maybe you have these projects getting sabotaged, but also maybe that affects What kind of projects you undertake in the first place. And in the paper, they get into like, how could you harden your project and make it harder to sabotage? And how many years of development would that, you know, if you have to build your data centres in a mountain, like how much, many more years does that take, et cetera, et cetera. So I think there's a core logic to that that is like helpful. And I thought it was a useful contribution to the discourse of saying like, hey, there's actually going to be, you know, multiple, multiple parties in this decision-making system that you don't just get to say, hey, we're going to race ahead and And win the race. And the other parties have options beyond just trying to develop their own AI system. They can actually kind of engage with you in other ways. I thought that was helpful. I agree with you that I don't think it has the force that mutual assured destruction did in the Cold War. I think mostly because of the clarity. The mutually assured destruction was so clear. We knew what nukes did. We knew how they looked. No one wanted to use them. We understood how it would work. Once second strike was really guaranteed, we understood what it would look like. Here, it's just all so much unclear.

Like what?

So much more unclear. What does superintelligence look like? How much does it matter strategically? So I think of their sort of MAME idea as kind of a helpful contribution, directionally useful, but definitely not that kind of crystal clear strategic logic that they, I think, presented it as. To your question about the sort of AlphaGo for war, I don't know. I don't think we're particularly close to that. That just seems like such an intractable simulation problem because you don't just need You need the battlefield dynamics of a specific battle. You need the broader theater dynamics of, you know, what is where in the world? How are you bringing, how are you bringing your assets to different places? That needs to be connected into broader economic and political questions of what is going on with the whole world. That needs to be connected to the, you know, the public attitude. It just, I think that is definitely people are trying to build simulations like that. I think they can be useful in limited ways. But I don't, I don't know. To me, it brings up this idea that I got from David Chapman, who's a really interesting, I think of him as a philosopher, but he doesn't identify as a philosopher. And he's written a lot of interesting stuff on sort of rationality and what he contrasts it with as reasonableness, meaning the how you make practical decisions in real world situations. Like when you're cooking breakfast, you don't sort of sit down and make a 10-step plan to cook breakfast. You just start and then you see what happens and you adjust from there. And a point that he makes that I think is really correct is that people in technical domains sometimes think of the messy real world as kind of a rough approximation of some much cleaner, more abstracted system. So like warfare is kind of a messy approximation of chess or of Go or of StarCraft. But in reverse, in actuality, it's the reverse. It's that these clean games, these sort of simplified abstractable systems, are really special cases, very special cases, very unusual special cases of the actual world that we find ourselves in. And I think that, I don't know, this is maybe opening up a whole other can of worms that we sadly don't have time to get into. But I think it seems to me like a lot of the focus on AI development today is focused on assuming that you can build these clean abstracted systems that are amenable to, for example, reinforcement learning. And in the meantime, the models continue to struggle with that kind of real-world practical troubleshooting, problem solving, readjusting along the way. All of this a long way to say that I think military simulation is going to be incredibly messy. Way too many factors, way too many unknown unknowns, way too much ability for the adversary to deliberately throw a spanner in your assumptions and make your simulation sort of inaccurate. So I don't personally see that as AlphaGo for war as any kind of near-term possibility. I think the broader question of what does equilibrium look like, I have no idea. I honestly don't know what the nation state looks like in a world of superintelligence. I don't know what democracy looks like. I don't know what the Chinese Communist Party looks like. So I definitely don't have kind of a broader answer for you there, unfortunately.

Well, that's maybe a great place to leave it. We have more questions than answers, and that's, like it or not, that's the reality. That's the timeline that we're in, short or long. We've got a lot of questions that remain pretty vexing. Uh, this has been great though. I really appreciate you for humoring me on some of the OpenAI questions at the beginning and also, you know, really admire how you've stayed in the arena and to look forward to your continued contributions to try to make the AI future a positive one for us and for our kids.

No, thanks very much. It was a great conversation.

Upstream with Erik Torenberg. Hey everyone, Erik here. At Turpentine, we're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries, from our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers like Econ 102 with Noah Smith. We're launching new shows every week, and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's erik@turpentine.co. And let's partner together.