Is an AI Arms Race Inevitable? with Robert Wright of Nonzero Newsletter & Podcast

Hello, and welcome back to The Cognitive Revolution. Today, I'm pleased to share a special crossover episode with Bob Wright, publisher of the Nonzero newsletter and podcast. Bob recently wrote an important piece called AI Surrenders to the Military-Industrial Complex about how leading AI developers, including Anthropic and Meta, are now beginning to partner with the US military. He shared the piece with me just before I attended The Curve, an immersive conference hosted in Berkeley this past weekend, which brought AI obsessives from academia, industry, AI safety nonprofits, and the national security community together for presentations and conversations on a wide range of AI topics, including the future of scaling laws, the latest findings in alignment research, and growing challenges in international relations. There, I had a chance to participate in a fascinating AI war game exercise. Organized in part by recent guest Daniel Cocotello, and designed to explore how critical actors might interact in the context of a fast AI takeoff in 2027. As you'll hear, our simulation included a war between the US and China, despite the fact that the AIs themselves were relatively benign. With that experience in mind and drawing on Bob's expertise in foreign policy, in this conversation, we attempt to characterize the growing hawkishness toward China in AI policy circles. Explore why this stance has become so dominant, particularly among AI safety groups. Assess the wisdom of US strategy in what Bob provocatively, but I think ultimately reasonably, calls the chip war. Question whether the militarization of AI development is truly inevitable or wise. Critique influential recent essays by Dario Amadei and Leopold Aschenbrenner. And ultimately look beyond today's prevailing narratives to identify potential paths forward that could minimize risk of U.S.-China conflict. Along the way, Bob and I each make very clear that we are not fans of Xi, would not want to live under the current Chinese government's censorship regime, absolutely disapprove of many of China's policies, both domestically and internationally, do not want to be naive in our approach to China, and candidly do not have great answers. And yet we both feel that our time is well spent searching for a better plan than than the one we're currently hearing from both the incoming U.S. president and major AI lab leaders, which amounts to declaring an AI arms race and attempting to build sufficient AI-powered advantage to back China into a corner and ultimately force them to submit to American hegemony. This, without a doubt, is an extremely risky strategy, and one that I fear will leave all of humanity losers. As always, we invite your feedback. Particularly, if you have ideas for how to build trust between great powers, create novel mechanisms for AI governance, or otherwise nudge history in a positive direction, please let us know, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. I am not leaving Twitter, but I did just join Bluesky, and my handle there is Nathan.Labenz. Finally, before we begin, a quick note on format. Bob is publishing the first half of this episode publicly and placing the second half behind a paywall for his subscribers only. We've obviously taken a different approach with The Cognitive Revolution, relying on commercial sponsorship to make all of our content open and free. We're tremendously grateful to our sponsors, who clearly value a thinking audience and have never attempted to influence our editorial direction. Without them, we simply could not sustain our pace of 2 episodes per week. That said, we have received some requests recently for an ad-free version, and so I'm wondering how much interest there would be in such an offer. If you'd be interested in a paid ad-free feed, please reach out and let us know. If there is serious interest, we'll look into that possibility. With that, I want to wish our American listeners a Happy Thanksgiving and to challenge all of you to spend a little more time thinking about how you as an individual can work toward a more peaceful positive future. Now, here's my conversation with Bob Wright on the trajectory of AI advances, US-China relations, and how we might avoid sleepwalking into catastrophic conflict.

Hi, Nathan.

Hi, Bob.

How you doing?

You know, never a dull moment in the AI game, that's for sure.

Or the world generally. And actually, we're going to talk about the intersection of the two. So, you know, I'm really looking forward to this. Let me introduce us. I'm Robert Wright, publisher of Nonzero Newsletter. This is the Nonzero Podcast. You're Nathan Labenz, host of the highly regarded Cognitive Revolution podcast, which is one of my 3 go-to podcasts on AI. Very— it's been very valuable for me. You're also founder of a company called Waymark that does video ads for small companies and employs AI in that process to make it cost effective for the, for the companies in question. Let me, if you'll indulge me, give like the background story to this conversation. So a couple of weeks ago, I published in my Nonzero newsletter a story called AI Surrenders to the Military-Industrial Complex. The news peg, as we used to say in journalism, was just that a couple of companies, Anthropic and Meta, had kind of crossed a line. They had decided that they were going to do business with the US military involving their large language models and also involving Palantir, by the way, a well-known Silicon Valley defense contractor that had focused more on predictive than generative AI traditionally, but I'm sure it's open to all forms of Pentagon revenue. And, you know, I DM'd you about the piece. I sent you the link, thought you might be interested in it. And I should say, The part of the story that I thought was most important actually was not the news, but kind of why some of it surprised people and why they shouldn't have been surprised. And I'm thinking in particular about Anthropic. As you know, Anthropic has a reputation for being particularly safety-conscious, concerned about harms done by AI. And of course, the military is in the business of harming people. Now, you know, I don't say that pejoratively. I grew up in a military family. I understand why countries need militaries. I actually have a lot of respect for the military as an institution. But I think this is one reason some people were surprised. I think they had maybe inferred from AI safety consciousness that it was just kind of generally dovish or generally leftish or something. Anyway, there was surprise. Part of the point of my piece was they shouldn't have been surprised if they had been paying attention. Because Dario— can you help me pronounce his last name? I call him Dario as if I knew him, just so that I'll know I'm pronouncing it right. How does he pronounce his last name?

I believe it's Ahmadi.

Ahmadi, who is a highly regarded figure in AI, not just because he founded Anthropic, which produces a great LLM called Claude, which I use, but because of his past role at OpenAI in developing the whole GPT technology. You know, he's very highly regarded. And he had published this essay, I don't know, a couple of months ago now, I guess, called Machines of Loving Grace, 15,000 words. A lot of it was just laying out possible positive scenarios about the future use of AI. He had previously, I think, been thought of by some as kind of a negative guy, so concerned about risk. And so I think maybe he wanted to get out there the fact that he's not blind to all the possible upsides if we avoid the catastrophic risks. But a part of the essay that didn't get nearly as much attention, I'm sure it got some in your circles, but for example, Lex Fridman did a 7,000-hour conversation with him on Daria and didn't bring it up. Didn't come up. Even though it focused on the essay. And that was a part where what we would call in journalism the actual story from that document. Like, here's the news where he lays out, I think more than he ever did, his foreign policy ideology as it applies to AI. And I would say, broadly speaking, he first of all signed on to a framing of US foreign policy that's common in the foreign policy establishment, certainly in the Biden administration, which is that the US and other democracies are involved in an existential war with authoritarian or autocratic countries. I'm down on that framing myself, I should say, partly because I think it becomes a self-fulfilling prophecy. But in any event, he was very straightforward about that. I'm not sure he mentioned China at all, but there's no doubt that he was talking about China and that—

Russia.

He embraces the chip war that that we have launched against China, which for reasons I will get around to explaining, I think is a bad idea. So anyway, I DM'd you. You replied with a link to some thoughts you had been developing on a Google Doc, and there was some convergence of our views. I'll leave it to you to characterize your views, but it's safe to say that neither of us is China Hawks. And I think both of us worry a little about how rapidly China hawkism has become ascendant, not just in foreign policy circles, you know, not just among national security types who are just learning about AI, but among a lot of AI types, including some who are just learning about national security. And so I wanted to have you on to talk about this whole thing. You— it turns out you have in the meanwhile gone to this safety— AI safety conference at Berkeley. where you participated in a war game, a futuristic one. I guess the premise is that there's AGI. Apparently, they didn't think highly enough of you for you to play the AGI, Nathan, but they found someone smarter.

Me.

But that must— I wanna talk about that. But let me start out by asking, I mean, you note, you report in the Google Doc recording your thoughts on the Berkeley conference, that it's clear that China hawkism is dominant now. And I would like to start out by asking whether you have an explanation for the dynamics behind that. I mean, it was an interesting question to me 2 years ago before I was even thinking about AI, because it had started to happen in foreign policy circles. But what are your thoughts on on the state of play there and how this came to be, and if you have theories about why, what those are?

That's a great question. I don't know if I have a great theory. I guess a couple— let me back up for one second and then I'll attempt it. First of all, I would recommend everyone read the Dario essay, and I might even do a full audio presentation of it on our feed in the not too distant future because I think it really is kind of a study in contrasts between the first— it is a pretty long essay, as you said. The first half is one of the best, if not maybe the best recent articulation of what is the upside of AI? Why should we be excited about this? Why should we not just pause AI now? Or to be even more vivid, strangle it in its crib, as Tucker Carlson once suggested. And there's lots of good reasons, and he goes quite in depth. He's got a background in biology and biophysics and goes into his model of how biology and medicine improve and why we should expect that if AI gets good enough, we're going to get rapidly more discoveries. And he thinks that it's plausible that we could have what he calls a compressed century, where we get basically 100 years worth of biological and medical progress, not to mention economic benefits and equality globally within just like a 5 to 10 year timeframe. And I've honestly been crying for one of the AI lab leaders to say something like this, give us kind of something to hold on to for positive vision. So I really appreciated that. Notably, that section is a lot longer than the geopolitics section, which I do think kind of reflects the fact that At least the AI community is somewhat coming around to this late. And as much as I do think Dario is a great thinker, he was, as you mentioned, one of the lead authors on the GPT-3 paper, one of the people along with Ilya who is credited as having understood the scaling dynamics very early when it was not at all the consensus view. It did have a little bit of a feel to me of kind of tech guy thinks there's a simple solution to things that might not actually have simple solutions. And just the brevity and the sort of, I don't know, blithe kind of characterisation of, okay, all these democracies will work together, we'll have this advantage, AI is going to give us this advantage, we'll box everybody else, we'll isolate all of our adversaries, and then eventually they'll have no choice but to recognise that they should come onto our side and take the deal, whatever the deal is. It's unclear what the deal is. Does the Chinese regime survive this deal or not survive this deal?

And we should be clear, the deal happens when China kind of agrees to whatever conditions we lay down for relaxing the chip war and any other restrictions on what they can import technologically. So go ahead.

It's meant to be an offer they can't refuse. You know, it is a—

Very much.

And the phrase, decisive strategic advantage is increasingly being bandied about. That's like the idea that by advancing our own AI capabilities and through sort of a combination of means holding theirs down, we'll get to this point where it's like, okay, look, we can definitely beat you in a war if it comes to that. Therefore, you have basically no choice but to accept the deal, whatever the terms of that deal ultimately are. And those are not specified by anyone, really, as far as I know. This is all projected to happen in the not too distant future. Dario's timelines for AGI, or he prefers the term powerful AI, are like 2 to 4 years. He's like, 27 is kind of the— 28 maybe, but 27, I think, is kind of his most likely year for when this stuff really starts to happen. That was, by the way, the year that we played in the wargame simulation over the weekend. And that means it's probably current leadership that's intended to— that's imagined to be the ones doing this. Right.

I'm feeling better already. Yeah.

So we've got Trump and Xi, and these are the 2 relatively singular figures on both sides that this scenario flows through in a really core way. Yeah, I think that is concerning. I don't like the plan very much either. There isn't much of an alternative that I can find, and I don't feel like I have the answers as to why.

I have one, but we'll get around.

Yeah, well, I want to hear it.

It's vague, unfortunately, but go ahead.

It's tough. As Zvi Moskowitz recently said in a conversation I had with him, all the options are bad. So I think it is tough. I don't want to pretend that there are easy solutions, and I don't want to suggest that we should just be totally naive with respect to the Chinese government. I don't think that's a recipe for success either. But I really don't know how this happened. I'm certainly not a scholar of the blob, the foreign policy establishment. All that kind of stuff is a little bit outside my— wheelhouse.

I will handle the vehement denunciations of the foreign policy establishment. You don't have to worry about that. Go ahead.

One theory I've heard is that it's the only way to get things done in Washington in this moment. Obviously, we've got polarization and the parties are sort of inclined to block each other and try to prevent the other side from getting wins wherever possible. One notable change is that That dynamic may have been put aside for the moment where the Republicans are going to be able to do what they want to do, it seems, especially if they're willing to break a few more norms, which recent history suggests that they probably will be willing to do if needed. But up until this last election, there's been enough kind of a stalemate that how else do you get anything done? The sense has been, from what I gather, again, not spending anytime in DC, and certainly not being a DC insider, the sense has been we can't let the other side have a win, but we can at least— we can both agree that we should do stuff to get an edge on China. So, there's been willingness to put money into that and try to bring—

Can you elaborate on that a little? So, you're saying there's certain parts of the policy agenda that aren't necessarily central to the China question that they want to get some leverage on in Washington, and by signing on to the China consensus, that helps or what? And if so, what are those policy items?

Yeah, I mean, again, take this with a major grain of salt because I'm definitely kind of relaying what I've heard, not what I've experienced. But the idea is basically, if you want to get something going in Washington, you got to find a way to frame it as a way to get ahead vis-à-vis China. And, you know, our bridges are falling down, you know, our roads are falling apart. China's got better stuff. We can't let that happen. And then you kind of play that out across everything. In some cases, I do think for the better. We obviously are dependent on international supply chains, and I have mixed feelings about this because I do think broad decoupling from China makes future conflict easier and more likely, and I don't like that. So I actually do want to see the continuation of economic coupling with China. At the same time, I think it's only somewhat prudent to say the fact that we can't make any advanced chips domestically is like a state not really to be endured. So when we put some subsidies into bringing chip manufacturing back into the United States physical territory, I think that at least shows that somebody has some lights on in Washington. And I think that's good. But there's a lot of other things where, you know, it's my sense is, you know, impression from afar is that people feel like the anti-China train is the one train that's moving. And if you want to, you know, get your thing moving, you kind of have to figure out a way to get it on that train and everything else is kind of stuck. Hey, we'll continue our interview in a moment after a word from our sponsors.

Well, I will say in support of that thesis, kind of, is that there's an earlier landmark before the Addario essay here, and it's the Leopold Aschenbrenner essay, Situational Awareness. And that wound up, and we can talk about that, but it's the same. Well, I should say, I mean, here I should flesh out what I think is the scenario that's common in these circles, and you can Tell me if it's wrong. By these circles, I mean the AI side of the China hawkism, not the traditional foreign policy side necessarily. But you saw this in the Ashenbrenner essay. I think you see it in the Dario essay. And I mean, first of all, let me say the Ashenbrenner essay was retweeted by no less a personage than Ivanka Trump. So yeah, it got in the conversation. I don't know how many Twitter followers she had, but that was noted, right? And I mean, separate from that, Trump, you know, a number of people in Silicon Valley had Trump's ear by virtue of donations and so on. But anyway, the scenario in that essay is that once you get to this sometimes vaguely defined thing called artificial general intelligence, you are approaching a kind of takeoff point. In the evolution of artificial intelligence, and suddenly you're in superintelligence realm, realm. And that gives you such an— you know, things move so fast that if you get to this threshold first, then like, you know, a few months later, you have utter dominance over all rivals. And we're not just— we're not talking mainly about weaponry here, as I understand it, although it may involve that. As I understand it, they're saying, like, you can just say to the AI, go find a way to infiltrate China's social media and convince them that their government sucks, right? I mean, I think people are imagining a future where the AI is that smart and responds to directives that broad. I just thought that up myself. I haven't seen it, but I assume that's the kind of thing they're talking about. And so the Ashenbrenner essay, and I hope we'll have time to get around to saying some things about that. Because like the Dario essay, I mean, Dario essay, in its defense, that part of it was short. He didn't have time to do a whole argument. The Ashenbrenner argument, the whole essay is about, in effect, why we should be China hawkish.

It's not short.

It's not short. And yet I see fundamental assumptions behind that not being addressed or even stated. And I hope we'll get around to that. But tell me, is this— is one reason the China hawkism has caught on in the AI circles is because among a lot of people, that view was already, you know, instantiated? The idea that, yeah, there'll be this threshold. Beyond that, you get an intelligence explosion. Obviously, whoever gets there first is gonna be in charge. That's kind of widely assumed in certain circles?

I would say everything is pretty highly uncertain. The event that you alluded to where this wargame simulation took place was designed to be kind of a mixing bowl, like a meeting of different camps or tribes. So the AI safety tribe, if you will, was one contingent, but there was definitely also a sort of national security persona that was there. And There were people on definitely, I joked earlier today, very fine people on both sides of important questions like just how good or powerful, I should say, is AI going to get? How quickly? If it does get really powerful, how likely are we going to be to control it? And then downstream of that, what should we do with it? So I wouldn't say there was any single worldview that dominated, but as Leopold said in his original tweet with the situational awareness thing, it's not that people necessarily think it's for sure going to happen, but everybody agrees it's kind of strikingly plausible. We look back at just the last few years where I remember reading about GPT-2 when my now 5-year-old son was just about to be born. I was at the hospital, my wife was getting checked in, I was in the waiting room and just kind of nervously scrolling, and there's GPT-2. This thing could not do much of anything. It could barely put together a coherent paragraph. It certainly wasn't useful. And now we have expert-level reasoning across all domains of knowledge, at least for routine tasks. We've talked about that in the past, like the difference between routine tasks and truly novel work. And the AIs aren't quite doing that yet, but everybody's kind of looking at the trend and saying, this really can't continue much longer. without crossing some really important thresholds. And so everybody at least takes that possibility seriously. I wouldn't say anybody at the event was dismissing that possibility. Some feel like it's overwhelmingly likely, others feel like it's unlikely but can't be dismissed. And that range of opinion still kind of forces everybody to think, what do we do if that is in fact the scenario? I still don't know why we jump in the AI, especially in the more AI safety community. And again, I don't have polling data to back this up, but my subjective impression is that most people across these personality types, and there's a lot of disagreement about a lot of things, but it seemed to me that pretty significant, I would even say large majorities of both the AI safety and the sort of national security that range of personas, it seemed like a large majority across the board seemed to support the export controls, aka chip ban, or chip war, as you've called it, against China. And I don't really know why that immediately follows. It does seem to me like a pretty narrow, especially in the time available, right? I mean, the scenario that we played, I think this is even more speculative. I think most people probably at that event expect that AI will continue to progress. We will probably see things like what they would have sometimes called the drop-in knowledge worker, that an AI can kind of take over a computer and do useful work like most people who work at a computer do on a daily basis throughout the economy. That level of AI is, I would say, pretty broadly expected. There's a lot less agreement as to when you get there, how fast does that turn into superintelligence? And a lot of arguments, a lot of kind of, well, geez, when the AIs can write AI optimisation code and when they can explore new architectures for themselves as well, or maybe a little better than we can do that and a lot faster, because of course, if you have one AI researcher, you have potentially a million copies of an AI researcher that can explore the space of AI architectures all in parallel.

Yeah.

You know, does that mean we take off?

We should say probably for kind of laypeople, which is a lot of people in my podcast audience, the assumption here, you know, the term singularity, which you may have heard, refers sometimes to technological acceleration generally, but in particular to the point where the AI starts improving itself. In a certain sense, it already is. It's facilitating coding. It's making coding more efficient. But, but, you know, as that gets more— as the feedback cycle gets more and more direct, like the AI just saying, I'll create a better version of myself, this is the point of takeoff people are referring to. Anyway, go ahead. Yeah.

And how fast that happens, or if it happens, I would say, again, a pretty wide range of opinions. You have some folks who think that a superintelligence follows very quickly after a general intelligence because of these sort of self-optimisation loops. Others, and I would put Dario in this camp based on his essay, would say, well, yes, progress will speed up, but it won't be like a super sudden jump because you still got to run a lot of experiments. He talks a lot about the biology experiments, but even the compute experiments, that we do have some limitations on how many AI training runs you can run. Even if you have an AI coming up with a tonne of ideas, it is still somewhat of an empirical science as to which of those new architectures are going to work better or worse. We don't really know until we run training. And for training, you need servers, you need GPUs. We only have so many of those. So maybe that sort of puts at least a governing limit on how fast things could go. And then other people think, well, we're just going to hit some sort of a plateau because— and I think this is a little bit fuzzy, but all we have is human-level data. What data are they going to learn from next? I don't tend to think that's the case because I tend to think there's a lot of sort of just feedback from reality that they can start to get. The path of training the AIs has— it's kind of looping back to an earlier era where if you go back to sort of mid-teens breakthroughs in AI, a lot of it was game-playing AI for different kinds of games. AlphaGo was, of course, a major breakthrough, but there were a bunch, right? There was an AI that could play StarCraft. There was an AI that could play all these kind of different games. There was one AI that could play lots of different games, and sometimes they didn't even have to teach the AI the rules of the game, meaning, like, literally just put it in an environment, let it play. It would kind of figure out the rules. It would become superhuman at all these games. And it did that by getting feedback from reality. not by training on internet data, not by training even necessarily on human games played, not even by training on human feedback, but literally just, did I win the game? Did I lose the game? If I won the game, then I should do more like that. If I lost the game, that wasn't so good. And that signal alone was enough to get these things to superhuman playing levels. Since then, the paradigm has shifted where it's like all the GPTs, they learn from us in a very literal sense, both from reading our internet writings and from reinforcement learning from human feedback, where literally a human is like, I like this response more than this response, or this seems better than this. But now we're also kind of starting to see— I think this is well underway, but the future seems to be learning from actual contact with reality, predicting what experiment is going to be worth running, actually running the experiment, getting the result of that, and then tweaking and optimising to be better at figuring out what experiments to run or code. Right. That's another one where we When you generate code, you can run the code very quickly. Often if it errors, you'll see that instantly, right? If it works and it produces what you expect it to produce, you can see that instantly too. So these signals are becoming, I think, or I expect that all these different signals are probably enough to take the current crop of GPTs into superhuman coding, superhuman experimental design, superhuman hypothesis generation, just like they've previously achieved superhuman game playing across a wide range of games. And so I kind of find it hard to imagine that they don't become superhuman. But again, going back to the range of opinion, not everybody buys that argument.

Yeah. And can I add, I mean, kind of inject a question? We'll get back to the China question per se. But, you know, the context for this is partly just over the last few weeks, there's been a lot of publicity about, is AI hitting a wall? In other words, are the scaling laws reaching a point of diminishing return? And scaling laws refer both to like amount of text you feed into it. And as you just mentioned, there's only so much human-generated text. You can generate it with AI and try that, blah, blah, blah. There's issues. And then, and then computing power is another place where, well, will scaling per se continue to bring the traditional rewards. There are questions about that. I personally, although I know a lot less about this than you, I agree that there are just so many other dimensions along which progress can happen. I mean, OpenAI released this so-called chain-of-thought model. And by the way, I mean, speaking of game playing, I kind of wonder if— we shouldn't spend much time on this at all, but I kind of wonder if— one way of depicting what's going on in there is the machine has a conversation with itself. It's as if there were 2 LLMs in there. And so like, hey, what about this? And then the other part of the machine's self, which might as well be another LLM, says, I don't know, that doesn't make sense for this, blah, blah, blah. Is there a certain amount of— just quickly, is there a certain amount of that kind of internal dialogue that has a little bit in common with the kind of self-play that led to progress in the realm of games, or is this a nutty intuition?

I think it's directionally right. I mean, your description of the chain of thought, I think, is pretty apt. Notably, OpenAI is not sharing that chain of thought with users.

Although there's now an open-source Chinese model that says they do it.

Yeah. And I have gone and used that, and I think it's a striking fast follow. Only like 9 weeks, I think, from the time that the 01 model was announced and given to the public to the time that the Chinese company DeepSeek came out with their DeepThink. And actually, I think there's another one too, another reproduction out of China that's like— Are you suspicious?

of how they— I mean, I assume OpenAI is wondering whether they should tighten their security.

Yeah, geez, that's a hard one. My best guess is that I think China— Chinese researchers in general, there's a lot of them. They're very smart. There's a lot of good research coming out of China. This has been true for years. And I hazard— it would be hard to guess. I think you can't rule out that there was some intellectual property theft, but I also would not dismiss at all the possibility, which I guess in my gut is probably the more likely scenario, that they're just good, that they really just have good researchers who are not far behind and probably were already working on this path. The timelines are very unclear because we heard roughly a year ago now, a year ago, we're coming up on the— I think we've actually just passed the anniversary of the Sam Altman board coup and restoration.

Wild memories.

What did Ilya see? Right. All this sort of— and the rumour at the time, which I think is increasingly like the consensus understanding of what happened, was that there was some breakthrough in reasoning. And people started to feel, including Ilya Sutskever— hopefully I'm saying that correctly— who was chief scientist and kind of one of the leading visionaries and one of the big believers in scaling laws along with Dario for years, he for a moment kind of flipped and said, we need to take a different approach, and Sam's not the right guy to lead this company because we're really starting to get close to some sort of you know, transformative AI.

So it's still— this is taken seriously within AI circles because I've seen the What Did Elio See meme and I just thought it's just sheer conjecture. Who knows? It's an obviously appealing scenario. It's the kind of thing that would spread virally. But is this— this is taken seriously among— because I had heard a whole different story that I won't bother you with, but This is taken seriously.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Yeah, I mean, I think the details, of course, are quite fuzzy, but my general sense is there was an early demonstration of some new reasoning capability, and they had published about this research. an early kind of demonstration where they were starting to do— what is the signal from which the AI learns is one really key part to the overall story of where the progress is coming from. They had put out a paper in 2023 about rewarding reasoning. I forget exactly what the name of that paper was, but it was sort of instead of just rewarding on the final answer, looking at each logical step that the AI was taking to try to come up with an answer and and giving reward at each one of these steps. So there was already reason to believe that sort of thing was happening. Then there's a guy that had previously been at Meta who had done incredible work on some of these game-playing reinforcement systems. He had been responsible for the AIs that beat humans at no-limit hold'em poker and also at Diplomacy, which is one of those moments where, for anyone who hasn't played the game of Diplomacy, and I'm not a big Diplomacy player by any means, It's a game of social dynamics. It's like a European theatre, and all the countries are trying to jockey for power, but it's not like rolling the dice. Instead, it's about who believes who and who betrays who. And they got an AI to play that at a very competitive level. So this guy went over from Meta to OpenAI to work on reasoning.

So there's all these reasons to believe that there's something to be said for that. So my intuition does have a little something to be said for it. If a game-playing guy went over and you think that helped the reasoning Yeah, I mean, he's been pretty open that that's what he's working on.

So I think there was some early demonstration. I think Ilya, my theory, again, these details could be wrong, but I think the generally accepted story is not so much because this thing totally changes worldview, but more like, hey, it's really starting to get real now. And there's also been these leaked emails that have since come out just recently, Where, because of, I guess, court filings and discovery, emails going back to early OpenAI days when Elon was still there and Sam and Greg and Ilya, all these guys were going back and forth on, can this be a nonprofit? Well, we're going to need so much capital. What should we do? Blah, blah, blah. There was already not a super high level of trust even dating back years. And this is in plain text now in these emails where people are saying, Sam, we don't really feel like we understand you, and we need to understand you better if we're going to go on this long journey with you. So I think some of that was already latent. It started to feel like, hey, this is getting really real. Ilya kind of freaked out, perhaps rightly, perhaps ultimately futilely, it seems. And then it took them a full year to productise and red team because they did do a lot of advanced safety testing on this. I had 2 episodes with 2 different external groups that had early access to the new O1 model and ran all kinds of tests on it. And it wasn't just those 2 groups, but those were the 2 that I talked to when it came out.

Yeah, I remember that. That was a good podcast. I recommend it.

Circling back with all those signals in mind, right, it would be very surprising if there weren't groups in China kind of saying, yeah, what's down that path? Right. Sounds like there's something there. We should be working on that too. And yeah, it's possible that somebody stole the weights or the secrets from OpenAI, but I think there was enough of a— existence proofs are really powerful in AI. When somebody sees that something is possible, it's a really powerful signal and kind of invitation to figure out how it happened. And even if this wasn't fully an existence proof in the public view, there was enough kind of consensus understanding that this is kind of what the next big thing is, that I have no doubt that they were working on it before. Possibly when they saw the limited demonstration and the limited examples of the chain of thought that OpenAI published, they got a better idea and were able to kind of lock in, this is the exact strategy that we want to use. Notably, I think their thing is still not as good. The O1 model from OpenAI, as far as I understand on the benchmarks, is still a cut above. I would also bet that it is outside of the benchmarks, probably even more of a cut above. I do think OpenAI does generally a really good job on kind of not just looking at these very discrete tests, but also taking a holistic approach.

Yeah, there's a tendency to teach to the test. I mean, there's an incentive to teach to the test when you're training your models, and you're saying OpenAI doesn't seem to do a whole lot of that. The Ilya, the name Ilya Sutskever, an alternative pronunciation that may also be wrong. He gives us a good chance to to get back into the you know the kind of China hawk conversation. He said something a couple years ago. I wish I had the quote in front of me, but it was to the effect of my hope is that the world's peoples or the world's nations are or something will recognize. That this technology— and I don't know if he used the word threat, he obviously sees a great upside to it— but the idea was, my hope is that they will understand with sufficient clarity that it imposes a common challenge to all of them, that it is in their interest to get together and confront it and govern it together. He may not have used those words at all, but that's very much in the spirit Of what he said. And one thing that bothers me about the China hawk discourse is I don't see many people bothering to stop and assess whether that belief is true, right? Because if it is, right, if— in other words, they don't, they don't look at the costs of, of this, like, manic AI arms race with China. Which just fundamentally affects the psychology of the relationship, radically reduces transparency in certain respects, and just generates this kind of chaotic environment where people are going to feel more compelled to get models, to create models, perhaps without adequate safety tests. It just seems to me pretty obvious that If you take certain kinds of threats seriously, and there's a whole range of threats you can take. And by the way, I think not enough attention is paid to the threat of sheer destabilization. In other words, even if the sci-fi doomer scenarios aren't valid, and I don't personally rule them out, but even if they're not, if you look at all the economic and social effects that AI is already starting to have, and many more of which will materialize It's obviously going to be disruptive, not only in the good sense of the word. It's going to be destabilizing. And when you sign on to what you're describing as an existential arms race, you're saying, let's hasten that. Let's accelerate the destabilization of, I would say, American society. And by the way, one surprise for people may be that China, because of its authoritarian character, actually has more of an interest in regulating the AI and not proceeding too fast and winds up as a more stable society than the US and beats us that way. That, that to me is something I'd put in parentheses. It's not like my big concern, but I would say that there are a lot of possible downsides to a big AI arms race with China, and I just don't see these people exploring them. And it's— and let me— I'll close. I'll stop and let you talk. But I want to focus on a very specific thing that's been kind of driving me a little crazy, or the lack of discussion of it has. And I want to ask you whether in AI circles this is commonly discussed, because it's one of the most straightforward downsides of the Biden Chip War, which, you know, if people don't know by now, consists of not just prohibiting the manufacture, the import from American companies of certain kinds of things to China, but using American leverage to, you know, to get other Western makers of both chips and equipment to not send, you know, to deprive China both of the high-performance chips that they were getting not long ago, And of the equipment needed to make them. So here's the scenario where I think this is obviously a dangerous policy. And I think you've—I've seen you allude to a version of it, but I want to be clear about the version I'm talking about. So, as of a few years ago, both China and the—and the West were getting high-performance AI-level chips. Out of the factories in Taiwan that are run by TSMC, and that when you think about it was a disincentive for China to invade. Whatever other incentives they may have, you know, it's pretty clear that if they invade Taiwan, those factories are going to be incapacitated in one way or another. Probably if the U.S. loses the war. They're gonna bomb them on the way out. But if not, because of their ongoing reliance on like the Dutch company that makes the most high-performance manufacturing equipment, you can in effect pull the plug remotely by ceasing to provide support. So, you know, either the US like wins the war and China wants to— anyway, the point is, It was kind of a mutually assured destruction thing. War in Taiwan means one way or another, one side is going to just as soon see the factories incapacitated. So war means both sides lose their access to the high-performance AI chips. That's what we had as of a few years ago, a kind of mutually assured destruction, a reason neither country wants war now. We have created a situation where China is no longer getting the high-performance chips. So whatever disincentive existed for China to invade, that has definitely been lessened. You know, and and I'm not. I want to distinguish this from the scenario where China just says, "Well, screw it! We're going to blow up these factories." You know, that could happen too. But what I'm talking about is something I think more plausible, where China. Which obviously has this set of reasons it wants to bring Taiwan back into its country, 'cause it of course considers it a renegade province. And in fact, it's not even recognized by us as a sovereign nation. So I'm talking about a subtler dynamic where China's just doing these calculations every year and they're like, well, the US is sending more weapons, it's not gonna get any easier. And by the way, we care a lot less now whether this factory gets blown up And because the chip war has gotten us to accelerate the development of our indigenous chipmaking capacity, we may do better pretty soon than the West does in terms of hanging on to like the next level of chips, like smartphone chips. So this is so clear-cut. It's just game theoretical. Like, you can't argue with the game theory here. And yet I have had People people do it on on my podcast, but I finally found somebody who had actually written about it named Paul Triolo. I'd encourage you to have him on on the podcast. He's a he's a real expert on Chinese tech. My question to you is: Do you hear this in AI circles? Because I do not hear it in the foreign policy establishment, and that itself is a very bad sign. Jake Sullivan and the Biden administration. you know, with the encouragement of some people in tech like Eric Schmidt, did this policy. I see no signs that they actually thought it through. And you would definitely hear about the scenario I just laid out if they had thought it— if people were talking robustly about this policy. Do you hear much about this?

At least a little, you know, I guess is the silver lining sort of good news, but definitely not as much as the mainline scenario, and I don't mean to suggest I think it's the most likely scenario, but the scenario that is talked about most is definitely hold them down, we move forward, we try to create this advantage. We've covered that. You do hear, I think, all of the things that you mentioned. I don't think we're entirely alone in playing out those scenarios in our heads and worrying about them, but it does seem to be a minority. I think the prevailing view is more like, The timelines are probably fairly short timelines. Of course, in AI discourse, how soon does powerful AI arrive? People think that probably not that long. Maybe as soon as— literally, people are saying maybe as soon as 2025, maybe literally next year, maybe 2026, most likely 2027, 2028, a little less likely. Hard to imagine it doesn't arrive by end of 2030. I would say that's the general consensus view. If you don't subscribe to that, you're somewhat of a minority. And that doesn't mean that there's not— there's also this sense that if it goes past 2030, then it could be a while longer, because then that would mean something we don't understand. So people are sort of like, unless I'm wrong in terms of how I understand the situation, it's coming soon, but I could be totally wrong, and then kind of who knows?

I would just interject, tell me when I get too obnoxious about this, but that if China shares that view, that heightens their incentive to invade Taiwan. They want to stop our access to the high-performance chips, and there is no other source outside of Taiwan right now. ASAP, if if they share your view. Anyway, go ahead.

Yeah, I mean, I had a a good discussion with Zvi Moshowitz about this recently, and he put it very plainly, just saying if they believed if meaning if the Chinese government believed what you like. Western AI leaders and policymakers believe about the timeline to AGI or powerful AI or transformative AI, whatever you want to call it, if they believe that, why would TSMC still be there? They could easily destroy the manufacturing capacity. I don't know if they could effectively or successfully invade if it came to that. But these manufacturing processes are extremely delicate. The machinery we're talking— Oh, sure. Limited quantity supply from this Dutch company, ASML, that makes the machines that make the chips. They're hundreds of millions of dollars. They're packed in extremely delicate packaging to make the trip to Taiwan. Teams from ASML go to TSMC and are on site with them there to operate them effectively, because this is not simple stuff. It's not like just stamping press that's stamping things out. There's a lot of little intricacies, and I'm not an expert in it, but I understand that almost nobody is really an expert in the high-level view. When you drill down, it's like all of these steps in the process, and there are many, many steps. are all highly optimized with experts that spend years and years of their career just becoming the world-leading expert on one little part of this overall supply chain to go— they say like a speck of dust can ruin a chip. People go in and get sort of super tight.

They could incapacitate them. Yeah. I mean, I would say in response to what he said, there are many downsides to China invading. I mean, there would be just a boatload of sanctions. Their economy's already in trouble. There's a lot of geopolitical reasons they would be very hesitant to do it. And I guess, you know, in a way, this gets at one thing that I'm worried about. It's like, like I said, I haven't heard this game theory in the foreign policy establishment, and I kind of keep track of their journals. And on the other hand, your crowd, like the AI Silicon Valley crowd, they're very rationalist and game theoretical in their approach. But they are less versed in the broader geopolitical stuff. Like, I, I didn't sense in, for example, the Aschenbrenner essay, the texture of conversancy in international affairs that I'd like to see from somebody who's recommending that we do something that may get the world blown up. But anyway, I digress.

I mean, there are voices and there certainly, again, this stuff does come up. I would agree that I don't think the depth of expertise is there in the AI space as you would like. One person I would recommend folks follow is Miles Brundage.

Oh, yeah.

He recently left OpenAI, where he was, I believe his title was Head of Policy Research. And he's increasingly, I think he's kind of figuring out what he's going to do next and in theory taking a little vacation, although he's still saying quite a bit online. But one of the things that he said that has caught my attention the most is the West needs to figure out ways to demonstrate to China that our AI development is benign. And unfortunately, obviously that would be a lot easier to do if it actually were benign. Unfortunately, I'm not sure that it really is benign, certainly from a Chinese government's perspective. When you have the President of the United States saying in print that we need to win the AI arms race with China, that leaves little room for interpretation from them in terms of—

Was that Biden or Trump?

No, that was Trump in a recent press release. And then you also have the AI lab leaders saying the same thing. We've covered Dario. Sam Altman has also radically changed tunes on this over the last few years. I'm old enough to remember when he used to say that we shouldn't make our AI decisions based on what we think China might do. There's video of this, and I've previously, on previous podcasts, I've specifically called out, look, I think that's great, because I used to say he could very easily make China the bogeyman, and that would create more rationale for him to do whatever he wants to do. Well, sure enough, that's now flipped, and he's put out an op-ed several months ago, deeper into the past now than the Dario essay, but basically saying the same thing. The kind of key point was, this is Sam Altman's op-ed paraphrased lightly, it's either AI with Western values or it's AI with autocratic values. There's no third way. And yikes, you've got kind of all these data points I think are going to be hard for the Chinese government to see as anything other than a threat to the regime. I mean, it's like when the stated plan is back them into a corner and make them an offer they can't refuse, terms unspecified. I think that's going to be pretty hard for them to accept. Now, Zvi said, I think this is good analysis from him. He said, why TSMC is still there. All the disincentives that you mentioned are, of course, very real, right? Lighting the fuse now would come at huge cost. Lighting the fuse at any point will come at huge cost. But if they if they were to destroy TSMC now, that would definitely come at huge cost for the global economy. We can't make cars in today's world without a supply of chips. So it's not— I think there are, like, I'm not an expert in this, so it's amazing. Fact check me, but I've heard thousands of chips in a single car. So it's not just like you can't make AGI without the chips. You can't really make much of anything without the chips these days.

Many of those are low performance chips, but there are some other sources, but still a lot.

I mean, my understanding is a lot of the volume, even of stuff that isn't like the very most cutting edge, does still come out of Taiwan. And there's stuff in Korea and there's stuff in other places, and we're trying to build our own factories, obviously, here too. But Zvi's analysis, which I thought was good, was The reason TSMC is still there is they don't believe it in their gut at this point. They don't believe that this takeoff is necessarily happening. And the implication of that, though, is if they do come to believe it, then they pretty plausibly would act differently.

Well, then they're only, I think, just starting to digest probably the growing consensus I mean, the Dario thing only came out two months ago. I assume they paid attention to that. This this whole idea—I mean, you know—when the Ashenburner thing first came out, it's like you look at this guy; he looks like he's about twenty, and like okay, great. People are talking about it. But I suspect China's just starting to take very seriously the idea that this could be the strategy. We get this stranglehold on their technology. And then when we get utter supremacy after the intelligence explosion, we demand that they submit in the sense of signing on. And the Dario essay is alarmingly vague on what we would want them to do. I mean, I really wanna quote some of this because I find it so frustrating. Let me see, I'm just calling up the piece I wrote in, Non-zero. So he, um, okay, so he says the coalition would aim to gain the support of more and more of the world, isolating our worst adversaries and eventually putting them in a position— this is a coalition of democracies— where they are better off taking the same bargain as the rest of the world. Give up competing with democracies in order to receive all the benefits and not fight a superior foe. I'm curious, what does he mean by give up competing with democracies? He doesn't tell us. And it's like— and see, this is— I want to ask you about this. This gets to something I want to ask about. I mean, seems like they should be allowed to compete in some ways, like economically and so on. Like, so I assume what he— I think he's buying into an assumption that's also common in the foreign policy establishment, Which I would like to question. And this could explain why China hawkism is pretty popular in Silicon Valley, because there's a lot of libertarians there. And I can imagine how they would think, well, if China wins the race for global supremacy, their values get imposed on us and we will not have freedom. Could happen. But if that scenario depends on a particular assumption, I'd like to question the assumption. And that is the assumption that, just kind of broadly speaking, part of China's foreign policy is a desire to impose their form of governments on other countries. That is commonly assumed. There is no evidence of it. It's misleading because the more we sanction a bunch of like authoritarian or autocratic countries, the more they get together. And so it looks like this monolithic threat. And of course, they try to recruit countries to their side. But as a rule, they don't like ask them to become authoritarian or autocratic. They just allow them. They just want them to be part of their economic zone and vote their way in international institutions. And I think to some extent, you know, China has, if anything, been notable for its realpolitik. In other words, like we'll do business with anybody. We're not trying. We're not trying to turn you into a model of ourselves. We are hard-nosed business people, and we want to do transactions with you—economic, national security, whatever—but we're not trying to turn you into China. And I think to some extent, America is projecting its own foreign policy on China. We do try to turn countries into into models of ourselves. In many ways, it's laudable. Many of the human rights we lobby for, you know, I share those values, even though I think it's. You know, the effects range from none to counterproductive as a practical matter. If you look at like the sanctions we impose, still, I prefer our form of governance. But the fact is, we do wanna impose it on the world. We do want, you know, whether violently in Iraq or whatever, that's our game. And there's just no good evidence that it's China's game. And I'm wondering if you think that, that assumption, I mean, it doesn't have to figure in To an Ashenbrenner scenario, but I'm wondering if you think the assumption that China really wants to, and I and by the way, I can see how you know you'd see like when China tries to punish Hollywood studios that or subsidize them to do the nine dash line on its maps or twelve dash whatever it is. I I understand how you think wow they're getting in our system that's true but. I do think there are explanations of that other than that their goal is for us to become authoritarian. So I've talked a long time, but I'm curious, do you think that the view I'm describing is common in kind of Silicon Valley circles?

Yes, I think so. Maybe a little less confident in this than elsewhere, but I did feel this weekend a little bit that it does seem like kind of a hall of mirrors sort of projection, you know, or sort of a game theory prisoner's dilemma type dynamic where, you know, it's not just at this point that people are thinking that they want to impose their values on us, but that we're thinking that they're going to think that we are going to hold them down. And so they're going to be, you know, an adversarial approach to us is baked into their, you know, analysis.

And so there's another way to attack the perception that we're trying to hold them down, by the way. Like, don't do chip—

Again, it would be much better if it really were benign, you know. So yeah, that's my view is always, I mean, people— Joshua Steinman, who I recently did a podcast of, which I honestly don't really recommend that much because I didn't think it was super productive, but I ran it anyway because he is a 4-year member of the Trump National Security Council from the first term. And so I thought, even though I didn't think we had a very good meeting of the minds necessarily in our hour and change together—

No, you didn't.

I can confirm that.

I still think it is worth considering what he and presumably others think. He called this the physics of the environment that we're in, which is, I think, obviously an overstatement in that there is no physical law that says that we can't break out of this prisoner's dilemma dynamic. But I have now heard multiple layers. It's sort of an infinite regress, right, of like, we don't trust them, so we're doing this. We believe that they're not going to trust us because we're doing this. Therefore, they're going to act this way. Therefore, we got to bake that into our expectations. And at some point, it does just kind of take on a life of its own. And I do think we should challenge that. I mean, of course, there are some things that are trying to challenge that, and there are some signs of hope. I'd be remiss not to celebrate the Xi and Biden joint statement that we will not put AI in the nuclear chain of command. That seems like the absolute minimum that we could hope for from sensible leadership. But it's good. Let's still celebrate it as a positive step. And there are AI safety groups in China. They don't have, like, an official one that is sort of there, the UK and the US, Some other countries now have established an official AI safety institute at a national level. These groups are like setting standards or trying to, and coming up with sort of ways to evaluate how dangerous new AIs are. And they're trying to coordinate at an international level. China has some of these sort of organisations. They don't necessarily have one that's like the official designated group to go out and talk to the world, but they do have people domestically that are concerned with AI safety in the grandest sense. There's even some talk that maybe Xi is a doomer. That's kind of a sort of underground meme in the AI world that he's said a few things that everybody's— unfortunately, we're reading the tea leaves and kind of psychologising literally Trump and Xi, right? I mean, that's what it's come to. But people do have at least somewhat of a theory that, hey, maybe Xi is actually going to be the real— he's a scientist, right? So maybe he'll be the one that kind of sees this clearly and tries to de-escalate the situation. And one can hope. One also could hope that Trump could pull some sort of Nixon goes to China kind of deal. That was definitely discussed this weekend because I asked— there was one guy who presented— this was under Chatham House rules, but I don't think it's too much to say that this was a person who was a previous member of the national security establishment. Right. So that narrows it down to millions of possible people basically giving a presentation around where are we and where do we go from here? And it was everything that we've just discussed, right, that basically the path is going to be Militarisation, closer partnership between the government and the leading companies in the United States. Definitely onshoring all of the production that we possibly can, trying to create this advantage, trying to hold China down through any and all means necessary, definitely including the supply of chip restrictions and kind of whatever else we might need to do right now. The margin on this sort of thing is that, and of course, they're finding ways to get around the the ban in some cases. Right. So there's a black market too. But even in the— there is analysis that, like, hey, if we do cut them off, they're definitely going to develop their own industry. And that means we don't have leverage in the future. How can we, while trying to cut them off, how can we sort of soften that so that we don't create so much incentive? Right now, it is possible for Chinese companies to buy high levels of compute from US suppliers, from an AWS data centre in Singapore, for example. As a Chinese company, you can go and buy there. But that is kind of the margin now of like, well, maybe that also will need to be restricted, because if they can buy that, then they could still train advanced models and whatever. So at some point in this presentation, I kind of raised my hand and said, what do you think China is going to do? this has all been from our perspective. Do you have a theory of how they're going to respond? And I give the guy credit for being very candid about it. He was basically like, I don't have a good answer for you. But his sort of the most hopeful thing that he could come up with was Trump likes to make deals. Maybe at some point over the next couple of years, Trump finds an opportunity to make a deal and kind of cuts China in. to the deal in some way. And can we do that in a way where there's enough trust, where there's enough ability to save face on all sides? I think that's going to be very difficult. But that sort of escalate to de-escalate path, we always hope for the de-escalation, or at least I do. And maybe, but it does feel right now. There's not a coherent story of how all this plays out in a way that doesn't lead to, at a minimum, I've started to say, MADE is the new MAD. Mutual AI destruction is the new mutually assured destruction. I've seen other people say that too. I think I coined it independently.

Except without destabilising effects, I would say. I want to talk about like what would a deal with China look like involving AI? It would be real challenging to even construct a good deal. I think I should say something. I've talked a lot about what's wrong with the current approaches. I feel I should say something about what I think would be a better approach. Well, I have two. I mean, my first reaction, like the Ashburner thing, is like, oh, okay, so we're going to establish this technological dominance over China. This intelligence explosion is going to happen. We're going to be in the driver's seat, and then we're going to say, then we're going to pivot and go, actually, we're nice guys. You should like us, notwithstanding the fact that we just spent a couple of years making you hate us, and you should trust us and blah, blah, blah. To me, the idea that we would just do this subtle and wise thing at that point is itself a dubious assumption. Just given the how inconstant American foreign policy is, we haven't. We have a new president every four years. It just it just so the whole idea of pursuing a single coherent strategy over multiple years is dubious. And now you've got Trump being the guy who could be in the White House when this happens. And then we've got what I would like to turn to now, which is what would the deal look like? Like what? Because, you know, AI is so much harder in principle to regulate than nuclear— than nuclear bombs are because of the nature of it. And, uh, so what would we be asking for? Have you thought about this at all?

And I'm trying, um, but I wouldn't say I've got any great breakthroughs so far. Part of it does depend on the technology itself and kind of how things evolve. And that makes it all harder and more kind of scenario-based. But one other big thread from, and maybe the thing at the event this weekend that got the most buzz, and it's not yet— this is kind of unpublished research, so just speaking about it in very general terms, and it'll come out quite soon, but I don't want to steal the author's thunder. Basically, we got a preview of some new alignment research, which showed that basically, I think a short summary is none of the AI control techniques that we have right now are robust, basically full stop. There is no credible plan to control a super powerful AI on multiple dimensions. There's the question of runaway AI that doesn't listen to the human commands, and that's maybe the most dangerous form, and that's also probably the most speculative form. There's also the one that does listen to human demands but might be commanded to do something bad. And do they refuse that? I mean, we've all seen ChatGPT examples, at least if you haven't done it yourself, you've seen it online where ChatGPT refuses to do stuff. But then you've also seen examples where ChatGPT doesn't refuse to do something that OpenAI definitely has intended for it to refuse. And that can happen in straightforward ways, and it can also happen in weird, exotic jailbreaky sort of ways, whatever. Bottom line is the examples of these violations are becoming more vivid as the AIs get more sophisticated. It's not just like a dumb toy example, but you're starting to see yikes, this is starting to be in kind of Technicolor. Anthropic has been behind a lot of that research, and I definitely give them a lot of credit for that. I think that one of the strains of thought that Anthropic has kind of championed, but the AI safety community broadly has, I think, been smart to invest in, is sometimes just called scary demos.

Can we find things?

We definitely want to be looking for ways that the AI might bite us. Can we find ways that are compelling enough to show them to the public, or perhaps to the Chinese government, in a way that kind of convinces everybody that this is getting so dangerous that we need to take a different approach besides just racing one another? So if the technology is shaping up that way. And again, this is hotly debated. Nobody really knows. There is a school of thought that thinks alignment by default will happen. Each generation of model is getting, in some sense, more ethical, in some sense, maybe harder to take off the rails than the past, but still definitely not that hard to do. Anyway, I don't really subscribe to the alignment by default, but it's a school of thought that's out there. If we're in this sort of scary demos world where it's like, yikes, this is a problem, we don't know how to solve it, nobody knows how to solve it, let's all agree to take our time here, then maybe you could imagine some sort of jointly joint custody island in the Pacific where AI researchers come in together and have a little onsite nuclear reactor and data centre where, and maybe it's all not connected by the internet to the rest of the world. And this sort of safety research can happen in a place where neither side could take it over without destroying it. And maybe there could be some sort of way to create enough urgency that both sides kind of go that direction. However, you'd really have to believe, and you would have to be confident that the other side really believes that this is the way to go? Because, as you said, it's really hard to detect what's being done domestically in an AI development capacity. We're not going to turn off all the data centres. We're going to continue to build data centres. You can't tell from the outside what those data centres are being used for. And we just generally don't have great visibility. I think China has a lot more visibility into what's going on in the West than we have into what's going on in China. So how are we going to get to the point where there's— I think I could imagine getting to a point where both sides agree that, hey, this is getting kind of scary, maybe we should have some joint global project. But then both sides are still going to have a lot of reason to question whether the other side is in secret pressing ahead domestically without saying so. And so The game theory of it is really tough. I don't know, what do you got? Because I'm looking for ideas.

Well, I mean, I guess I'd approach it from the other end, which is to say, not with many specific ideas. But what you said plays well into the thing I would say, which is that, okay, it's very challenging. You're going to need some transparency and so on. And so I would say, you know, you have to start thinking about putting the US-China relationship on a fundamentally different basis and turning it into a relationship that involves less sense of mutual threat. And by the way, one frustrating thing is we know from history that the way dynamics, the way it works when you've got a perceived adversary, is both sides tend To perceive things done for defensive reasons on the other side as offensive. We've seen this before, and you keep seeing it. It the threat and inflation feed each other, and it's just it's just I see I've been I've been alive quite a quite a long period of time compared to you, and I've just I've like lived through this again again, and we keep saying oh that was a mistake oh Vietnam was a mistake oh the Iraq invasion was a mistake, and it's like yeah there's a general theme here folks. Neither Vietnam nor Iraq was an actual threat to the United States. And so with that little sermon set aside, I would say, you know, you need to, you know, it's like if we said, well, can we have an arrangement like this with Britain? We'd say, well, yeah. I mean, yeah, sure. It's a lot harder than nuclear weapons, but yeah, you can solve the problem with another country. You know, it depends on what kind of relationship you have. It depends on whether you deem them deeply threatening, whether you feel a certain amount of trust. And I don't mean trust in the sense of, oh, well, we'll just believe whatever they tell us. I also mean like them trusting us enough to let our inspectors in and take a closer look at their data centers and for us to exchange credible data on a regular basis. All these things are possible. But we have to start by putting the relationship on a different footing. And I think, you know, the first thing to do there is abandon what I previously described as part of our foreign policy, which is wanting nations to be like us, telling them that if we're going to have good relations, you know, you're going to clean up, you're going to have to, like, fix what we see as your human rights problems. Even though, of course, if you said to us, it's inexcusable to have this many people sleeping on the street at night homeless, we'd say none of your fucking business. But still, we're gonna tell you the parts of your country that we don't like, and you're gonna fix them or we're gonna sanction you. We just have to completely abandon that mindset. Now look, I think what's going on with the Uyghurs is horrible, but we have no leverage over it. And even the targeted sanctions wind up denying employment to innocent Uyghurs who were not swept up in the thing in the first place. And it just— I think so. I just think we have to abandon that whole approach to foreign policy. Our human rights-based sanctions are almost always counterproductive and hurt the people we want to help and just say, okay, like, you're in charge of the country within your borders. And then there's the Taiwan issue, which is very challenging. And I think you have to say, and part of this has to be, yes, establishing, look, this AI thing is big. We gotta talk about this. We're gonna have to do something about this. We can't afford to be worried about a war in Taiwan. So how about this? Okay. Can you just assure us? And this is where you have to have a relationship of trust among 2 leaders. And Trump may be better at this than Biden. I don't know. But it's happened before where you know you and and also like track two diplomacy can matter here. You draw on the good trusting relationships you have, and and you say, "Look, can we just you promise us you will not invade Taiwan or do any kind of confrontational military displays for just like you know." the rest of my term as president, and I promise you, we will quit sending, we will send no more weapons. We will not, if the Speaker of the House, as Nancy Pelosi did, is gonna stage this visit that you're gonna find threatening, I will do everything within my power to stop that, even though the president doesn't have the power to actually, you know, just say, we understand what you find threatening about Taiwan. We want you to understand what makes it hard for us to not speak belligerently about Taiwan to our constituency. You know, you you go do these aggressive military displays that makes our life harder. Just have the conversation. Say, okay, we're going to move Taiwan off the table, and we're not going to lecture you about human rights. Let's work out some rules of the road about you know those islands in the Philippines. And again, the context of this has to be, yeah, we recognize this is a serious problem. AI is like, we neither of us really knows how this is going to play out. Both of us can imagine globally catastrophic scenarios. And so job one, you get the relationship on a sound footing. It's been there before. And look, if they say no, fine, you tried. But I really don't think they would, and I'm confident we haven't tried. And so I'd say you do that, and then you have the, as you just established, equally challenging conversation about, well, how do we reassure each other that we are not secretly developing some killer AI thing that's going to give us global dominance? And and and and you know, I mean the game. You want to talk about game theoretical logic? It is long even before AI. The smart thing for both countries to do would be say, "Hey, let's face it. We are by far the two most powerful countries in the world. Why don't we just get together and run the fucking planet?" You know, like like let's just you know and and I mean uh I'm you know you wouldn't put it that way, and I'm kind of half joking when I put it that way, but. The logic, and and what impedes that logic is often the political incentives of the individual players. I mean, you know, American politicians can get mileage out of scaring us, so they do. And look, Sam Altman can get subsidies by scaring us. It would be shocking if Sam Altman hadn't done this pivot, right? Where where he starts saying, "Oh, China, China, China." Of course. He wants subsidies. All corporations would like subsidies. He wants the government to help, to give him permission to build nuclear reactors, ideally to help pay for them. Of course. And so, you know, there are all these incentives in domestic politics and commerce to scare people. And in some ways, I mean, we shouldn't underestimate the extent to which that applies in China, by the way. You know, autocrats can't be They're not actually autocrats. They don't actually have uncontested control of the country. They fear popular sentiment in a way more than a democratically elected politician does. So they have their domestic political constraints. But in some ways, they do have more room to maneuver than us. And in that sense, we sometimes are the problem because our internal dynamics make us the unreliable player in an international relationship. So, that's the sum total of my wisdom.

Yeah, I like it. I mean, more to do for sure. How does one translate that? And I guess I should say, for the record too, because it's easy to get painted as a pinko or whatever the slur would be that would be leveled at somebody who is trying to be somewhat dovish. I'll happily wear the China dove label, but I'm definitely no communist, and I definitely agree with you on human rights violations and all that sort of stuff. I mean, there's plenty to find fault with the Chinese government. Maybe one sort of habit of mind that we can begin to popularise is a separation or ability to distinguish between the Chinese ruling elite or Xi as an individual ruler from the Chinese people, Chinese civilisation. I sometimes say 5,000 years of civilisation can't be all bad. And I think we make a really dangerous leap when we go from assessing Chinese current leadership and specific policy to a broader notion of Chinese values. I don't really feel like I understand what Chinese values are. I think there's, again, 5,000 years of history, multiple philosophical traditions overlapping there. But I think we're very quick in an op-ed to use a term like Chinese values and to think that means something bad. And I don't think that that's true. Or at least I'm confident enough to say I don't think Chinese values are all bad. So can we begin to make that separation a little bit more often? And can we do things at sort of a person-to-person level, certainly at like a scientist-to-scientist level? I mean, it doesn't always work, right? Pre-World War I, there was like an incredible international scientific community that was exchanging letters and speaking one another's languages and had tonnes of sort of mutual respect. And that wasn't enough to prevent catastrophic war between the great powers. So I'm not saying this is going to solve it, but I think definitely Western AI leaders and Chinese AI leaders should be talking to each other more and trying to develop some sort of society to society. We hope that Trump and Xi can pull a rabbit out of a hat and build some rapport, and maybe a few love letters could be exchanged there, and that could be great. But can we do that at sort of other levels as well? AI leaders to AI leaders, AI safety warriors to their safety warriors, even just person to person? One thing that we do have going for us now that AI provides is a way through the language barrier. I've often thought, man, it would be great to learn Chinese, but what an unbelievable undertaking. But I can now have not just ChatGPT translate character by character for me, but also even audio translation is now a thing. So there's like our ability to communicate, our ability to build some bridges, I think, is enhanced by AI. And I feel like this is sort of You know, it's easy to— it will, of course, people will be like, look, Nathan, that's pretty words, but like, you live in a, you know, in a tough world. And I don't mean to deny that or try to be naive, but I do think there's something about like, I've been asking myself the question, like, what is virtue? What is a virtue? What is a— how does one be a good person? if you're worried about all these things in this era. And the sort of Kantian notion of universalise the maxim comes to mind. If we all took a little bit of this initiative and all tried to be a little bit more understanding and sister cities programmes is another thing that I've recently been thinking about. Can we create some sort of fabric across this divide that connects people? And it doesn't have to be about AI at that level either, right? I mean, this is why I also think banning TikTok, I feel, would be a mistake. I feel like we could always do it later if we actually get into a hot war with China. Sure, ban it. Okay, I'm with that. But for the moment, what I see from China is like people cooking foods in open fields, and it's a window into culture there. And I find it a lot to like about that culture. So I do think that just demonstrate— it's obviously a drop in the ocean, right? But it's good to remember there's a big ocean between us too. I find it much more likely that AI will bring a terrible outcome on all of us than that my grandkids will be speaking Chinese. Yes, it's a drop in the ocean, but that's kind of all any of us as individuals can do. And I sometimes ask myself, like, the old, what did you do, Grandpa, when the US and China were sliding into AI arms race? And I don't want the answer to be nothing. It might be fruitless or it might be futile effort, But I definitely don't want the answer to be nothing.

Yeah, no, you're doing God's work. It's such a challenge. I mean, for example, the things you're talking about, sister cities programs or something, as the climate gets more and more hostile toward China, doing things like that starts incurring the risk of being called China sympathizer. I just hate this whole category of labels, like you're doing Putin talking points, you're doing this, you're doing that. I mean, you know, but well, part of my, my background for this is I think it's funny, I'm doing this book on AI that will come out next year. But it actually— I was originally— I now have a 2-book contract. with Simon& Schuster, because the book I was going to write was about cognitive empathy, which just means— doesn't mean feeling their pain. It just means putting yourself in other people's shoes, doing your best to understand their perspective. Which, by the way, LLMs turn out to be surprisingly good at. And that's an interesting thing in its own right. But the— yeah, I've long felt that this is the most One of the most destructive cognitive deficiencies, because there are, you know, there are specific cognitive biases that I think actually selectively impede it. One is called attribution error in the fullest sense of that, not the original sense, but I won't go into that now. But I just think, you know, cognitive empathy is so important that we shouldn't have slurs that are directed at people when they say, well, you know the way Putin is looking at this, or, you know, I wanted to go have this conversation with this person in China, or, you know, um, so, but, but my concern is that, you know, and, and again, I'm older than you. I lived through the first Cold War. Not the— I'm not so old that I remember how it started, but I remember, you know, being in the middle of it. You know, my my my father was at the I was you know at the Pentagon during the Cuban Missile Crisis. I was like you know living two miles from the Pentagon, so that got my attention even though I was six years old. Because I don't think the Pentagon would have fared very well in a nuclear war. And you know you just see that it gets worse and worse and worse, and it's it's more and more frowned on to say anything about the perspective of the other side. or to say you wanna go there and talk to people, and I just worry that we're sliding into this dynamic. I mean, maybe the good news is it's a totally different information technology environment. You can't have the degree of opaqueness that we had there. We had no— the average American had no idea what was going on in the Soviet Union. No idea. And, you know, now, you know, there's more penetration of national bounds, Even when China tries to build its Great Wall, you still— we still have a— we know much more about what's going on there. So there's reason for hope, but I, I guess that's kind of my sum total, my concerns. If you wanna say more about that, feel free. I also wanna ask you about those war games in Berkeley, so go wherever you want.

Yeah, I don't think I have too much more. I mean, I guess maybe I would invite anyone who has ideas to send them our way. I am actively thinking about, because I don't want— I feel like my current answers are not satisfactory. And with that in mind, I'm very open to new ideas. And I feel like even if they're somewhat underdog or unlikely to succeed, we've got to be willing to at least entertain those. So I would invite certainly any experts in the subject that have any ideas, and even anybody who's just kind of found themselves thinking about it a lot without any formal background to send me their ideas. I'll keep trying to come up with other good ideas, but I do think that the cupboard is a bit bare at the moment, both collectively and for me individually. You know, that's not great.

Yeah, I would say the same thing. Feel free to get in touch. I should give people an email address to get in touch with. But meanwhile, why don't you— why don't you talk a little about the work? So this is— so you're at this conference and the premise is what, that we are right at AGI that's just been achieved, or there's— or we're at superintelligence, or what?

Yeah, it's basically a fast takeoff scenario. It's essentially the sort of fast end of the consensus scenario, right, where I kind of laid out previously the distribution of probabilities from 2025 to 2030. So let's say this is maybe the modal expectation of people in the community that AI gets really strong in 2027. So basically the scenario is you come into 2027, the AI at that point is sort of described as like a weak drop-in knowledge worker, something that can essentially replace or perform alongside any human corporate employee doing day-to-day stuff. And then over the course of that year, it continues to improve and you enter into this takeoff dynamic because the AIs keep getting better and then they are improving themselves, and by the end of the year, they're more powerful than humans. And then it was an interesting dynamic where different, in classic war game style, different people play different kind of key actors. So there was one person assigned to be the president of the United States. There was another that was representing Congress. There was another that was representing OpenAI. There was another that was representing other leading developers, Anthropic and Meta and Google. And then there was Anton Trennikov, who is a super smart guy. He was playing the AIs, and I was playing the safety teams, the public, and the press. And at the beginning of the game, there was a secret process by which Anton had to basically roll dice to determine how aligned he was as the AIs. So in other words, is the AI genuinely benevolent, or is it scheming to take over humanity? And we didn't know what the state was. We could try to interrogate him somewhat to get clues.

Maybe I missed this, but whose side was he on?

Well, the scenario was basically that he was created by OpenAI, but there was reason to believe that, much like we talked about earlier with the 01 models, China is not far behind. They're thought to be just a little bit behind, possibly because they stole the technology. They're obviously doing their own development too. So it was a reasonable, I would say, reasonably realistic scenario. If you had to pick who's going to create AGI first, it would probably be OpenAI at this point. Definitely not ruling out other candidates, but that's a pretty good first guess. And China is not far behind.

Okay.

But the AIs were sort of considered to be essentially the same technology tree across both US and China, with China just being a little less powerful at kind of every stage of the game that we played. So the US did have an advantage, but was it a decisive advantage? There was really no way to know that until you actually tested the theory in a kinetic way. So I'm playing the safety teams, And the overall trajectory was essentially the one that we've been chewing on and kind of worrying about. And I had some of these scary demos from the previous day in my mind as I was playing the game. And I'm just looking at how compressed the timeline is. And we were instructed to not play as we ourselves would play, but to try to simulate what the actors in this situation would do. Whether I did a good job of that or not, I don't know. But my kind of immediate instinct was, given how fast this is all moving, as the safety teams, I really can't imagine that there's any experiment that I could run that would be sufficiently compelling for me to sign off that we should just keep racing forward on this trajectory. Of course, we did run our simulated safety tests, and the results that we got from Anton, who in fact was mostly aligned in the end, it was revealed at the end that he was pretty benign. He kind of said to us, like, all your tests are passing, no major red flags, but there are some weird things when you look into reading the transcripts. you see some behaviour you didn't expect and that you can't explain, and the AIs are solving problems in ways that you don't understand at times. And I think that is a fairly realistic scenario. I was kind of like, I can try to run more tests or get more clarity, but on this short of a timeline with human brain and knowing that deceptive alignment is at least in principle possible, I don't think there's anything that the AIs could tell me that I would really believe to the point where I would say, yep, hit the gas, let's go for superintelligence now, knowing that it possibly could then just take over and kill us all at that point. So I was like, okay, the strategy that I played was a more social strategy. I basically said, we need to slow this down. I demanded they had kind of escalating speed of progress as we go through this takeoff. And I said, I want artificial speed limits. I know we could go faster. We are on track to go faster. But my demand as the safety teams to the Sam Altman player as the head of OpenAI was, I want speed limits. And if I don't get them, or if we, the safety team, don't get them, then basically we're going to defect. We're going to go to Congress, we're going to blow the whistle, we're going to bring this to the public and say, you guys are out of control. And also notably in this in this scenario, the technology was not deployed. So it was known to OpenAI, it was known to the national security establishment. China had their version, but it wasn't like ChatGPT had the latest upgrade. So the public and Congress didn't necessarily know what was going on. So I was going to go try to blow the whistle, wake up the public, whatever. My demands were not granted by the OpenAI player. They were not willing to slow down. And of course, rationale for that was, well, China. China. We can't stop. They're going to beat us then.

Yeah.

So I basically said, okay, in that case, I'm out. A little more nuanced was because all the evals were passing. Eval is evaluations, or just tests, diagnostic tests in the AI jargon. Eval. those were passing. So my expectation, or sort of, I kind of had a split internally for me as the safety teams. I was like, some of us are leaving and going to go blow the whistle and try to alert the public. Some of us stay because we want to continue the work and we're at least somewhat satisfied by all the results that we are seeing. And again, I think that's pretty realistic. There's almost certain to be some scenario or some people that can be credibly pointed to and say, well, okay, sure, those people are freaked out, but we've still got these people. They think it's fine. So anyway, I blew the whistle. I went to Congress. Congress formed a committee that didn't do much. The public in this scenario became more alarmed and gradually started to— we started to see mass protests. Of course, I'm just coming up with this stuff, but I think what I would expect is mass protests and bombings of domestic terrorism, basically bombings of data centres, potentially even targeted killings. And obviously that makes everything worse, I would think. But you got a lot of people and a lot of guns in this country, and I would imagine that we are headed for some people becoming radicalised. So that seemed realistic. But that also didn't really have an effect. It maybe just made everything worse. We did end up with war between the US and China, and there was a mechanic of rolling dice when certain kind of key things happened, right? So the players got to sort of indicate how likely they thought different outcomes were, and then dice were rolled with the outcome being dependent on sort of the average of the players' guesses as to the percentages and then the actual roll of the dice. So when we got to the US-China conflict, it was striking to see that the expectation for, can the US win this? Who wins this war? Will the US win? Will China win?

The war being whatever happens if you choose to accelerate in AI, or—

No, it was kind of like the pre— it was like China was It was sort of the analogue to the Japanese oil cutoff, right? The Chinese government was sort of realizing that they're a step behind and that this is about to get to the point where they really have no options. And so at that moment, things kind of went hot.

The war is continuing to cut off their air supply.

Yeah, well, I think that was kind of all baked in. They have fewer chips was definitely part of the scenario, and that's part of why they're a step behind in the AI game. And it was kind of getting to the point where it was reaching that critical threshold of the US might have superintelligence and you don't, and then you really might have no options. So essentially, this is the situational awareness, but we weren't necessarily playing it the nice way of, look at all the fruits of the AI we could give you. And of course, we're a few hours— this was a 4-hour exercise total, so limited time. It wasn't an all-out AI war initially, but strikingly, the guesses among the players as to who would win was not dominant in the US favour. I think it was roughly 50-50. It was definitely within kind of a 1/3 to 2/3 estimation of who would win. this showdown over Taiwan. Nevertheless, the war went hot, ships were sunk, chip fabs were definitely taken offline. And meanwhile, the US continued to do its AI stuff. And at some point, the president, in conversation with the AI, was like, okay, AI, you just need to take over our military effort. The AI basically said, okay, if you give me control of the military, I will win you this war. And the president said, okay, I guess I have no choice. You can have control of the military. The AI then delivered a decisive victory for the US. Not a total defeat.

That's a load off my mind, by the way. I was afraid I'd be in the losing country.

Well, yeah, we win-ish. I mean, it wasn't like a total defeat, but the way he framed it was, It is clear to everyone that if this continues, the US will win and China will lose. Now, China still had nuclear weapons. We hadn't gone nuclear at that point. And this sort of— this is kind of what the situational awareness and escalate to de-escalate, I guess, hopes for, right? Is that with that superiority having been demonstrated, the Chinese government had a choice, which was escalate even more and go nuclear, or take some sort of hopefully face-saving peace. And in this scenario, they took the face-saving peace with major, obviously, disruption to the economy, but also the AI. We sort of patched over that a little bit, I think, in our minds by saying, like, well, the AIs are so good, they'll figure out how to build new fabs real fast, or we'll kind of get over that disruption. Basically, that was sort of the end. We got to some piece, the AIs are powerful, and then it was sort of the final flip of the cards, right? Okay, now what's the AI? Is the AI good? Is the AI bad? All this stuff was happening while in the background we still had this fundamental uncertainty of, is the AI going to take over and kill us all at the end, or are we going to all live happily ever after? We turned over those cards and we all got to live at least roughly happily ever after, because the AI was in fact good.

Yeah, the— it's so funny, this— the assumption that, you know, there is this, this kind of threshold thing, the AGI, the acceleration. Yeah, to me it just seems like, who knows? I mean, you know, the— I do agree with you that You shouldn't count on the so-called, you know, the plateauing of the scaling laws to slow things down a ton, even though they may plateau. But there just are these other realms, you know, including— I don't want to get off too much on this, but just like these kinds of training data we haven't gathered much of, for example, just monitoring what workers do at work. And like that being the training data for agentic AI, it just seems like there's so many, you know, you know, there's going to be big progress there. And that's, that's kind of central to what we mean by AGI because it involves the ability to do things. But so, yeah, things aren't going to slow down. But that one, I mean, do you agree that like the— I mean, how much should I focus on this assumption of like, okay, AGI, like threshold reach, takeoff, that, you know, a kind of a simple form of that being this foundational assumption that's A, widely held in these communities, and B, is very consequential because things change if that's not exactly the way it plays out.

I definitely— I wouldn't say it's necessarily widely held. It's certainly not dismissed, but I wouldn't say that the fast takeoff is consent. Like, that jump from human expert level at everything AI to truly superhuman, incomprehensible AI, I wouldn't say that happening quickly is necessarily—

So you don't think that's critical to the Dario or Aschenbrenner scenarios?

No, I do think so. I'm just distinguishing between Is that consensus?

Okay.

I think it is. I don't think everybody, or I don't think a supermajority agrees that is the natural course. At the same time, I do think a supermajority agrees that we should at least take it seriously as a possibility. And I do think it is a really important question because the shape of events, I do think, looks very different depending on whether you have incomprehensibly intelligent and powerful AIs running around or not.

And do you think that is assumed by Dario and Aschenbrenner? I mean, that they would have to kind of rethink or recalibrate if you took that assumption away? I mean, I don't think Dario makes it explicitly, but—

Yeah, my read of his article is that it's a less sudden thing than either Ashburn-Burnett seems to think, or that was assumed in this war game scenario. But I think he seems to think still it's like pretty fast. I mean, it's like what passes for slow among some of these folks is like a few years instead of potentially as short as like a few months. I mean, it was literally just one total year in this. I mean, again, this is just a scenario.

Yeah.

But it is a scenario that at least some people do think is the most likely scenario. It was just one year from kind of knowledge worker AI that you could employ for just about any job in the economy to super powerful, incomprehensible, turn over the military to this thing because it's just clearly going to be better at running it than we are.

And it's true that if you have something that is at least as smart as like the best performer in every job, you know, and you can replicate them. I mean, that alone, you know, gives you— could give you, you know, quite a growth spurt in your capabilities. You know, there's a lot of unfortunate aspects of an organic human being that they don't have. And you can kind of weave them together in productive ways. But—

Yeah, I've had this slide in talks that I give about just AI in general. Like, where are we? And there's kind of a sequence of these from the last 2 years on the Cognitive Revolution feed. But I have this slide called the Tale of the Cognitive Tape, which is kind of giving, like, boxing style, a bunch of different dimensions, and then kind of tries to assess and compare humans versus AIs on all of these dimensions. And it's interesting to look back at the versions that I've gone through over the last 2 years of this slide, because the first version, I was just kind of like, let me just write out whatever dimensions of cognitive capability come to mind, and then I'll score the humans and the AIs on each. So initially, breadth strongly advantages the AIs, but depth definitely strongly, at least 18 months ago, strongly advantaged the human expert. And as I've revised that over time, the AIs have, of course, continued to get stronger and stronger. And now you have depth. Arguably, the AIs have caught up. I think per the benchmarks, the AIs have pretty much caught up to human experts on really hard stuff. Things like knowledge.

You mean knowledge, though?

Yeah, depth of knowledge and also ability to reason through problems. Yeah. But the sort of original set of things that I had laid out, to a very significant degree, the AIs have caught up on. And I've been finding myself challenged to think, okay, there must have been some dimensions that I hadn't really considered initially. And so I've got now a new set of dimensions where it's like, okay, I'm now zeroing in on what is it that the humans are still better at. And it's things like robustness to Adversarial inputs, which is to say the AIs are still gullible and pretty easily tricked, and we're much harder to trick, although we're not impossible to trick, but they're easier to trick than we are. And there's things like situational awareness, which is just like understanding our environment and kind of knowing when to break frame. In very practical terms, if you're debugging software, sometimes I've had this experience where the AI will kind of keep trying the same sort of strategies over and over again. Where then a human friend might look over my shoulder and say, why don't you just restart the computer? And the AIs sometimes don't have that leap to be able to break out of this frame and kind of see if I can't come at it from a different angle. But it's increasingly fewer things and increasingly relatively niche things that you don't put at the top of your list of what makes something smart or capable, but do turn out to be still important ingredients. And I think this is basically the to-do list of the AI companies today. They're very well aware that these are the deficiencies. They're absolutely looking at ways to make sure that the AIs catch up on those dimensions. And There's room to doubt, but my strong suspicion based on recent trends is that they will be able to close the gaps on most, if not all, of those dimensions.

Yeah. I mean, you hear the thing about hitting a wall, and again, the scaling may plateau, but if you pay attention to the field much, you just see that it's like every week there's some interesting development, maybe along a totally new dimension you hadn't heard of But there are dimensions for progress other than scaling, it's safe to say. So, well, listen, I appreciate you spending all this time. This has been very constructive. And I want to say, yeah, if anybody has ideas, if they share, you know, our concern about where the China dialogue is heading and has been heading, And have ideas or, or whatever, or think we should be, or I should be, or whoever should be connected, could profitably be connected to other people of like mind, feel free to let us know. I mean, you can reach us at nonzero.news@gmail.com. By that I mean, by us I mean Nonzero. You've got your own coordinates, Nathan. And I guess people—

I'm on all the socials, always under my Full name Nathan LeBenz, and our website for the podcast is Cognitive Revolution. There's a form there too, and I do read all of the the contacts that I get through that means as well. So basically, as I always say, the social network of your choice.

Yeah, and I'm Robert Writer, one word. That's W R I G H T E R on both Twitter and Blue Sky, where it's what is it? It's B S K Y dot. Social, I think.

Yeah, I got to get on that one. That's a simple one I'm not yet on, but I should.

They have these starter packs, so you can get like a starter pack of people in AI, and it's really— and once you get in the zone, it's like you start picking up followers via starter packs. I don't— I can't figure out which starter pack I'm on. I'm on one, uh, because you don't— you know, because I've picked up 1,000 followers over the last few days. But, uh, I, I'm trying to figure out what these people expect of me because I've written different kinds of books. You know, I don't know if these are— I have to figure out what starter pack I'm on, but you should. It's— I think it may be an actual thing. I, I, I'm staying on Twitter because you're, you're going to get a diversity of viewpoint on Twitter that for the time being at least you're not getting on Bluesky. But I think it's, uh, I'd recommend it. It's, it's, it's a time to, to try to establish a beachhead there, I think.

Yeah. All right. I'm not leaving Twitter either, but, uh, I'll do that today. That'll be my—

All right.

Extra bonus item coming out of this. I can at least take action on getting on Bluesky.

Okay, I encourage you. Well, thanks again, Nathan. And let's, at some point, if the world still exists, at some point down the road, let's do this again.

Thank you, Bob. Keep fighting the good fight.

All right, you too.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.