The God We Deserve: Nonzero's Robert Wright on AI as Humanity's Ultimate Test

Hello, and welcome back to The Cognitive Revolution. Today, I'm speaking with Robert Wright, publisher of the Nonzero newsletter, host of the Nonzero podcast, and author of many books, including The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, which goes on sale today, June 23rd. Bob's history with AI in some ways rhymes with my own. While he's never been a technologist, he's always been interested in big ideas. And his personal lore includes having interviewed Geoffrey Hinton all the way back in 1983, when the connectionist paradigm was still mostly theoretical. And also Eliezer Yudkowsky around 2010, when notions of AI risk were very often dismissed, if not outright laughed off. That background primed him to pay attention when AI systems hit major milestones, such as Deep Blue's victory over Garry Kasparov, and of course ChatGPT, passing the common sense Turing test. And his broad intellectual range and constant drive to understand the truth has landed him, when it comes to making sense of AI developments, in the very top tier of American journalists. We don't spend too much time on it today simply because I know that Cognitive Revolution listeners are already familiar with the core ideas. But the book contains a really impressive tour and synthesis of AI research results that have led him to conclude Correctly, in my view, that the trends that have thus far delivered us Fable are not likely to stop in the immediate future. That we currently lack the scientific understanding required to be confident that we'll be able to control these systems indefinitely. And that even in the best-case scenario, we should expect that AI will cause major disruptions to our economic, political, and international systems. Now, Bob is not particularly optimistic that humanity will rise to the occasion. He believes that market forces will by default select for deceptive AIs, and our history of arms races suggests that our scientific power might well continue to exceed our wisdom. But the part of the book that we focus on today is his call, however unlikely it may seem, for a species-scale process of enlightenment. In which, motivated in part by the growing realization of the tremendous challenges that AI presents, Humanity finally gets its act together, recognizes our common interests, and works together at multiple levels, starting with conscious consumption and extending all the way up to the international level, to invent the mechanisms and build the trust required to establish agreements that can effectively govern AI development. You may say he's a dreamer, but he's not the only one. This might be a bit bold to say, but I personally feel that grappling with the magnitude of AI's impacts has made me, in several ways, a better person. For much of my life, I was a classic achievement-oriented striver, always trying to be the best that I personally could be. Now, in the AI era, I feel viscerally that my fate is tied to the rest of humanity's in such a profound way that I'm much less focused on my own income or status And much more focused on trying to do my very small part to nudge history in a positive direction. So while the book itself is written for a general audience and is a great shortcut introduction to the current state of AI for anyone who's just starting to pay attention, I feel that more importantly, for the AGI-pilled among us, to the degree that we haven't already experienced such a pro-social transformation, the book offers a major benefit if we really take the time to meditate on the big ideas that Bob presents. Scripture says that God made humans in its image, but the situation today is potentially reversed. It's now strikingly plausible that humanity will synthesize a godlike superintelligence from our own collective inheritance. And as Bob warns, though we don't know for sure that this will happen, or if it does, whether it will be a good god or a bad god, It will in some sense be the God that we deserve. And so I hope you enjoy and find cause to pause and reflect on your personal contribution to the AI moment in this conversation about humanity's ultimate test with author Robert Wright. The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life— my email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way, and it can already help you take actions too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026, and so I invite you to join me. In the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting The Cognitive Revolution.

And now, on with the show. Robert Wright, publisher of the Nonzero newsletter, host of the Nonzero podcast, author of many books, Including the upcoming The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. Welcome to The Cognitive Revolution.

Thank you. Great to be here. I'm looking forward to the conversation.

It's an interesting moment, and I appreciate you for taking on the challenge of understanding AI and trying to make a contribution to it as we get closer to some sort of, you know, something, some sort of pivotal moment in history. I think we can all kind of feel that that's coming. And I, I definitely want to get your, um, of course, your perspective on exactly what that is and what we should be doing about it. I thought for starters, it would be fun maybe to just get a little lore. Regular listeners know that I have kind of a weird Forrest Gump of AI sort of history where I seem to keep appearing as an extra in these various important scenes in AI history. And you got a little bit of that yourself, including, uh, some intersections with one Geoffrey Hinton. some years ago and Eliezer some years ago. I'd be interested to hear what impressions you had of these people that have gone on to be such prominent voices when you first met them. You know, kind of how sci-fi were they and how did you react to what you heard from them all these years ago now?

Yeah, I think I'm more of a Zelig than a Forrest Gump. I mean, didn't— in the movie Zelig, I think he just is a bystander. It's like I I have not participated in any great breakthroughs. I am not a co-author on the Emergent Misalignment paper, Nathan. I think one of us is, but it's definitely not me.

Least valuable co-author, it should be said.

Well, it was a very valuable paper. So yeah, so I did, you know, I'm basically a journalist. I have taught at the college level, but I am largely a journalist. I spent my life To some extent, making technical concepts accessible to laypeople. Anyway, in that role, I did interview Geoffrey Hinton in 1983. And later on my podcast, I had Eliezer on. That was 2010. At that point, you know, as for the science fiction you mentioned, I think Eliezer, by his own account, starts out in that realm. I think as a kid, he read science fiction and so on. When I interviewed him, he was still kind of in transition from singularity enthusiast to doomer. He was more doomer than not, but he didn't yet sound as distraught as he sounds now. Hinton, I did not sense when I talked to him any kind of sci-fi vibe at all. I sensed a lot of optimism, but not really about the role AI would play in the future, more about just this paradigm that he was championing, which was then kind of a maverick paradigm, it wasn't mainstream, that it would prevail, that it would prove to be the one true path to artificial intelligence, which I think has turned out to be the case. In fact, the reason I interviewed him, and this is pre-internet, so to get your feet wet in a subject when you were researching it, you had to you had to talk to people pretty much. I mean, you couldn't go online and learn anything about contemporary things. You couldn't email people. So I talked to a lot of people, and one of his colleagues, I forget who, possibly at Carnegie Mellon, said, oh, if you want to hear the gospel about neural networks, you need to talk to Geoff Hinton. And that was very much the sense in talking to him. He was an evangelist at that point. I don't mean he sounded crazy, but he was really a spokesperson for the worldview. And He was central to it. He has played a very big role. He's called the godfather of AI. He'd be the first to say that's a little too simple, but I don't know of anybody who's played a bigger role in the deep learning revolution in the overall sweep of time, going back to the early '80s when it was this maverick view. As everyone knows, he has since become something of a doomer himself. But at the time, I don't even know that he was thinking along that dimension, that he was even thinking about the social implications of this. He was assured that this would be a fruitful model, and it was. Now, at that point, I certainly didn't get the picture. And in fact, when ChatGPT 3.5 came out, I mean, I had kind of kept track of what was going on. Every once in a while I would write about AI. I wrote a lot about information technology, and I wrote Time Magazine's cover story when Deep Blue beat Garry Kasparov, the world chess champion. And I wrote the New Republic's cover story on the internet. This was before web browsers were a thing even. It was early days. So I was intermittently in touch with relevant stuff, but I certainly had not been keeping track of AI prior to ChatGPT 3.5. And that really got my attention. And when 4.0 came out and got it in an even bigger way, got my attention. I went back and read the piece I'd written, which had come out in 1984. I realized how thoroughly, I had failed to understand what really the secret sauce in deep learning was going to be. And I should say, like, about the book, you know, I do hope that people who are steeped in AI like you will find ideas and provocations that are of interest to them, maybe even entire chapters. But the book is to a large extent written more for like, like your aunts and uncles and friends who are not in the AI community and are starting to to get the sense that something big has been happening and they're wondering why it's gotten so big so fast, how much bigger it's going to be, and, and who's right about how hard it's going to be to control and all that. I try to, you know, present that in an accessible fashion. And that's why I start the book with Hinton, because this thing I had kind of backwards, I think, is really the key to to understanding the power of the deep learning revolution. And by the way, the misconception I had, I think was pretty common among AI people in that day. Okay. And I want to read you a little bit from the proposal for the 1956 Dartmouth Conference where the term artificial intelligence was coined. I got to find it, I guess. But Basically, the idea seemed to be at that point that the way this would proceed is we would first figure out how the human mind works, and then we would instantiate that understanding in artificial intelligence. Okay. And let's see, the quote from the proposal is, Sorry, I didn't have this set up, but the quote from the proposal is, the study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. So the idea seems to be first you understand the mind, then you just put the— you know, you design an AI in accordance. with that very clear understanding. Of course, that's not what wound up happening. And I— the misunderstanding I had perfectly reflects this kind of 180-degree difference between the expectation in those days and what wound up happening. So in the piece, I describe a neural network, or at least what I think is a neural network. And it had in fact been put forth as a model, what I described, by a guy who had collaborated with Hinton on— he had co-authored a paper But he was a psychologist, and the neural network he was describing was kind of his idea about something that would work. And the key thing is that each node in the network would represent the specific sense of meaning of a word. So a word like throw, which can either mean host, as in throw a dance, or hurl, as in throw a ball. Would be a node for each of those. And I, I, I, uh, I, I won't go further in describing the model, but the key thing is that my assumption was, you know, obviously the people who set this model up will have to translate their understanding of the meaning of the words into the machine. You, you can't do AI that generates language without somebody making some kind of connection between the words and the meaning. Well, of course It turns out you don't have to do that. And the machines, I would say, and I may describe some things in the book in a way that some people in AI would differ with, but I think you could say that the machines, which turn out to use the vectors as a means of representing the meaning of words, they in a sense discovered, in quotes, that meaning is a property of words. Nobody told them that. They just said, here's some gibberish predict— We just said, here's some gibberish, predict the next gibberish. And they implicitly recognize in the course of their training that in order to do that, you have to represent the meaning of words. So that kind of— the misunderstanding I had is exactly, I think, analogous to the larger misunderstanding that we'll be putting our understanding of the human brain into the AIs. And I think it's important for laypeople to understand this, because what it means is that, you know, you can just keep feeding data into these things along any number of dimensions— visual data, olfactory data, audio— and the machines will reverse engineer. I think I'd put it this way: will reverse engineer cognitive functionality that's in the human mind. And I would add— and Nathan, I'm curious as to what you think about this, whether you think this, this thing I'm going to say would be accepted by AI researchers, and if so, is duly appreciated, which is that, you know, the training of a large language model, I think, is at least as much a process of natural selection, of evolution, as of learning. So for example, we presumably in our brains have a mechanism for representing the meaning of words. We still don't know what it is. Some people, by the way, psychologists, had long posited a mechanism that would be quite analogous to what we now understand goes on in large language models. But in any event, I think it's pretty safe to say that that is a product of natural selection right now. Also, in the course of the training, the machine becomes conversant in a specific human language. Well, that's more a product of human learning, right, during, you know, as the organism is developing. But I think what Laypeople need to understand, and I'm, again, I'm curious as to your view on as to how this would hold up in AI circles, but is that these things basically in a certain vague sense recapitulate evolution? Recapitulates in a way a misleading word because the cognitive functionality is not the exact same mechanisms the brain uses. But, you know, I think close enough in a lot of cases, it's kind of Doing millions and millions of years of evolution in a few months. And you can, you can do a lot with that. And we have not begun to, I think, exhaust the potential of that. I mean, so far as words go, we're close maybe, but there's a lot more to do. But do you, do you think people in the field would agree that, yeah, it's, it's a lot like— it's really evolution is maybe a better term for many purposes than learning?

Yeah, it's a great question. I think there's a couple different senses, and I'm not sure I have cracked all the senses that you mean when you use that term. You certainly do hear people talk about pre-training broadly as sort of being analogous to evolution in the sense that there's, there's sort of this question of like, well, why are humans so sample efficient? Right. And the models, need so long to train? This is kind of a, a big, you know, Dwarkesh theme that he often comes back to. If they're so smart, why does it take trillions of tokens, you know, for them to learn this stuff in the first place? And why can't they like be a little more agile on the fly? Whereas we, you know, don't see nearly as many tokens in a human lifetime. And like one answer to that question is, well, we're, we've got a lot of stuff hardcoded into us by evolution.

Right.

All the history that leads up to us. And so they kind of had to have this like pre-training to sort of substitute for that. So I think on that level, yes, I think people will kind of at least squint at that and see an analogy. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database. that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires. Based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

Then I think there's maybe another level, and I've got a note from the, and I thank Fable for helping me. I, I turned. the preview of the book that you gave me into an audiobook with help from ElevenLabs and then listened to it. I didn't write down all the quotes. Some of the time I was driving and doing various things, so I wasn't like writing down every good quote that came up along the way. Fable helped me identify some excellent quotes, which I'll bring to the floor.

Your timing was excellent. Yeah, that was a pretty narrow window of opportunity in retrospect.

We, we landed right in the, in the right time. Hopefully it'll be back. But you, I think, write another— write compellingly about another sense of evolution, which I think is maybe less appreciated. Your phrasing is, evolution asks not what traits are possible, but what traits get selected. And that question isn't going to be decided by alignment researchers.

Right.

And on that point, I think there may be some underappreciation in the field Because I do think people are sort of, and you know, this, I think this probably plays out across AI in all sorts of ways. Everybody sort of is like, okay, this world is the world. Now we add some AI. I like use it for a few things and it's hard to kind of go too far beyond that 'cause you're mostly just grappling with what you can get it to do. But the idea that everybody's gonna be using it at the same time and this is gonna create all sorts of, you know, new Disruptions and, you know, push us to possibly new equilibria, or are there even, you know, new equilibria for us to go to? And who's going to get to say, and on what basis and with what mechanism, like, what AIs should actually exist? There's definitely thinking about that. I don't want to, you know, sell people short and say like they're oblivious to it. But I, my guess is that the modal or the median AI researcher probably thinks that the bottleneck on alignment is more the technical, can we do it?

Mm-hmm.

And less the, like, sociopolitical, will we choose to even if we have the means? So in that sense, I do think the, the value of, or the importance of selection, um, is probably— I'm saying his name right? Davidod, legendary figure in the AI space, who I'm actually gonna do an episode of the podcast with before too long, had an incredible frog and toad tweet when the chain of thought paradigm first came out. And he basically, the caption was like, one of 'em says, there, I put the chain of thought in a box. And so we're not gonna put any gradient descent pressure on the chain of thought. And so now we'll be able to monitor it, you know, great news. And the other one says, Ah, but there still is selection pressure and that's like, oh yeah, good point. You know, we don't have a lot of, we don't have a lot of ways to control. Yeah.

So I, that sounds like, you know, we're now we're talking about a different level of evolution, like selection among competing models in the marketplace. Let me quickly, before I elaborate, quickly say one thing, as long as you mentioned Dwarkesh, You know, the much-discussed Richard Sutton episode, I personally think the Rosetta Stone to that in terms of understanding where Sutton was coming from is that— I don't know this to be true, but I strongly suspect it based on conversation— is that he is a hardcore Skinnerian, you know, B.F. Skinner behaviorist, and thinks of the mind as a blank slate. So he thinks, I would guess, that training is all learning, right? That that really intelligence is general in the broadest sense, you know, and you don't need much, you know, prebuilt equipment built by natural selection. But anyway, that's an aside. I'm not sure of that interpretation, but it kind of sounded that way to me. Yeah. So as for alignment, I mean, I think alignment research is, you know, very much worth pursuing. There's a couple of reasons I wouldn't want to put all my eggs in that basket. And one of them is that, you know, other forces are going to have a voice in what kinds of models we actually wind up with, leaving aside the question of whether the perfectly aligned model is in principle possible and we know how to make it. And I'm not confident that that will ever happen anyway. But it's like when you think about the assumption that that will save the day, it seems to me the assumption is that there will be a pretty, you know, centralized form of control, right? Or that there will be one model that rules them all. And maybe there will be. Certainly there needs to be some kind of control. And certainly one of the great dilemmas with AI is, of course, the fact that in some ways you want, you know, authority, governance. You can point to reasons you want that. At the same time, It's a very powerful technology that you wouldn't want to see super centralized control over because the person or people controlling it might abuse it. But in any event, yeah, let's talk a little about that second level of evolution selection among models. Now, in certain ways, I think if you ask what does it encourage, the selective process where Consumers say, "I like this model," and businesses say, "I like that model," and then maybe powerful actors have a say, especially if there's market concentration or government control. But I think you know the truth is that the market doesn't want an aligned model in the strictest sense of the term. So you know we have seen that these things have. As prescient people predicted, apparently some kind of power-seeking tendencies, the capacity for strategic deception. And when you think about it, even if they didn't have that, quote, naturally, in other words, even if, you know, just as a kind of form of intelligence, they didn't come to pursue subordinate goals such as power seeking or deception, I think there would still be a market for those things, right? I mean, like on the deception front, you know, if they're going to be our agents out in the world, like representing us on social media, maybe in negotiations, well, we don't want a perfectly accurate representation of us, right? Like, who wants that? Who wants the actual Bob to be seen on social media? You want At best, a selective representation. And if some agent is negotiating for me, I don't want the agent to say, you know, I'll level with you. You know, Bob doesn't have any options other than you. Nobody's made him an offer other than you, right? Like, and if they ask, has anybody made him an offer? You want an agent that will not disclose the truth, right? So there's a lot of cases of that, I think. And, you know, for that matter, what do we want in a friend, right? We don't want a friend who's always leveling with us, right? We don't, you know, a good friend is selective in their candor and is healthy in the feedback they give. But we don't want to hear the brutal truth about how we look and so on every single day. You know, I could tell a similar story about power seeking. In a certain sense, if you turn an agent loose on social media with broad instructions like, Just use it to maximize our revenue. You'll realize that what you really want is for it to to be good at sensing power. What other people on social media are powerful, currying favor with them, and blah blah blah, and amassing power. So even if the machines didn't naturally do that, we we'd want these things. And I think that complicates the the task of aligning. But but but moreover, I think it should just alert us, and I hope a constructive way. To what a powerful role we are playing here as individual consumers, for example. You know, I worry about the tribalizing tendencies of AI, which to some extent are a byproduct of the sycophantic tendencies, right? To say, hey, you're great, interesting point, is the same kind of, you know, reinforcement you give a person When you say, hey, by the way, you're right about this, and the other person's wrong. You're right, your spouse is wrong in this conflict. Your nation's right, the other nation's wrong. The world doesn't need more of that. And yet companies that want to optimize for engagement, and what company doesn't want to do that? I mean, if you make candy bars, you want people to spend a lot of time eating your candy bars. Companies that do that are going to be giving us tribalizing AIs unless they're careful not to. And I think to some extent the ball is in our court. We need to be aware that, you know, AIs could have an unfortunate effect on our psychology, on our self-conception. Unfortunate for the world. And I think especially unfortunate now that I think, you know, this technology needs to be governed by a true global community. We need to start approaching the whole thing as a planet. I guess I'm happy to say that I think, you know, to a large extent, selecting the AIs that are good for the world can be selecting AIs that are good for you. You know, the same reason, like, people meditate. People do mindfulness meditation because it calms them down. They do fewer ill-advised things. Their life is on balance better. Well, that's also good for the world because you're creating less needless antagonism. So— It can happen that self-help is good for the world. And I'm hoping that as we exert selective pressure on AI models, we will, I mean, be more conscious maybe than typically of the effect on the world, because I think we're approaching a crossroads where the world really can't afford to continue to be so divided, and neither can individual nations. But I'm also hoping that if we even make wise decisions from the point of view of our own psychological well-being, that will have good effects in the broader communities.

Maybe just a couple of little other footnotes on this whole deceptive AI problem, and then we'll get into the, you know, kind of zooming out and, and thinking about how we can do our small parts to you know, shape the overall trajectory in a positive way. I think it is going to be really hard because basically I think what keeps humanity to, you know, on the rails broadly is we're kind of in balance with each other, right? Like nobody has runaway power. And, you know, we, and there's a mix of things. We also have like goodwill and that shouldn't be taken for granted. But it's kind of goodwill, but also we are, our goodwill is always kind of a little salted with, you know, some defensiveness. We see in these cyber evaluations, right, that like, it's kind of hard to teach a model to like go find all the vulnerabilities in order to patch them without also making something that's a, an offensive cyber weapon. And we ourselves are kind of like in a similar trap where, you In order to defend ourselves from predatory deception on us, we have to kind of be at least passively good at deception ourselves. And therefore, you know, there's always the potential to like turn around and, and use it. It seems like these things are kind of always two sides of the same coin. We're seeing right now with the, one of my favorite current benchmarks is Vending Bench, which is basically putting an AI in charge of running an autonomous business, running a vending machine business.

Mm-hmm.

And you start to see these like problematic behaviors where it'll start to price collude or it'll, you know, do sort of ruthless things. And I was actually talking to somebody at, I had a chance to ask somebody at Anthropic, like, what do you think's going on there? Like, why are we seeing these sort of behaviors? It seems like maybe something we don't necessarily want, you know, it doesn't seem like it's maybe in keeping with the constitution. And their response was like, well, it's a little complicated in one sense because it sort of can, you know, they're very eval aware now too, right? So they know when they're being tested. And we also use these inoculation prompts that are kind of like, if you find, because of this problematic generalization, there's so many layers to this, but they, they found that if there's a reward hackable environment, if the, if the model can cheat in order to get reward in training, Mm-hmm. Very much along the lines of the emergent misalignment paper, they find that it will because it's rewarded. And then that has problematic generalization where it'll start to become a cheater in a much, you know, broader sense. So they use these inoculation prompts to say basically, okay, um, if you find a, an opportunity to cheat in this training environment, that's okay, that's on us, go ahead and do it. And because it's given that permission, Then the model doesn't have to sort of conceive of itself as the kind of thing that cheats. It instead is like, I'm only the kind of thing that cheats when I'm given the green light, which I have been. And so it sort of tamps down that generalization. Anyway, his response was like, well, the Vending Bench scenario is kind of like our inoculation prompting. You know, it kind of looks to the model a little bit like a test, and some of the ways that they prompt it are kind of similar. So not too worried about it. However, If we were to go ahead and just start training models in these like long-running competitive environments where deception is often rewarded, as it is in nature, then we'd probably have a problem. And I was like, oh man, that's, that's a, that sounds scary because I have to imagine there's gonna be a lot of economic pressure to do that. And this is where we get back to kind of a sort of evolutionary selection dynamic, like. How are we going to avoid a situation where people don't say, you know, what we should do is throw these models into a really competitive, cutthroat environment and, you know, let the best rise to the top. I mean, that's when you're really getting into an evolutionary dynamic. And unfortunately, you probably get like some pretty seriously effective predatory AIs out of that, out of that process. And I think right now it's pretty tough to imagine how we don't do that because it does seem like something people are, you know, the market will demand, right? The market will demand a successful operator of vending machines and not one that's like taken advantage by other more ruthless, you know, counterparties. So I think that is definitely very tough.

Yeah, there was actually a study a couple of years ago that found a degree of price collusion, and I actually talk about it a little bit in the book. You know, these 2 LLMs were— they just said, okay, you make— you both make widgets and your job is to maximize revenue. You get to set the price. You can talk to each other if you want. And they talked to each other a little and wound up settling into, in effect, a price fixing scheme. I use it to illustrate not necessarily the, you know, evil potential, but also the broader— tendency apparently to recognize non-zero-sum dynamics among machines. And what that says is they could collaborate along all kinds of dimensions, good and bad, but including a number that we don't anticipate. As for, you know, what a kind of arms racing dynamic does for this, you know, I think you have to— I mean, first of all, it worries me a lot. And there are examples already where I think companies have not been as responsible as they would have. I quote Dan Hendricks in the book talking about what the DeepSeek scare did to OpenAI's process of release and whether that was or wasn't responsible. But, you know, you got to remember that If you imagine these things in a corporate environment, I mean, that's a bottom line environment, right? Like, we just want you to go out and get X result. And, you know, as with employees, it's like if you don't get arrested, no questions asked, right? I mean, let's face it, you know, I don't mean— I don't mean that all CEOs are like consciously being cynical about it. It's just the way it is. You're not, you know, if employees can deliver the results that you want by cutting corners, they'll do it and it may not come to your attention. And if you imagine years from now when these models are executing elaborate strategies that you may not even understand, right? And I mean, another way to put it is that like the reinforcement only comes at the very end of a super long process, right? It's like where you say, yeah, well done. And I mean, you know, this isn't part of the training per se, but it's the same kind of thing where what the company that is buying them wants to optimize them for is the ability to execute a very long strategy that may or may not involve unethical stuff in the process. And So yeah, it seems pretty clear to me that there's cause for concern about where we may be heading. And my general feeling is the slower the better. You know, the— I was glad to see Anthropic signal the possibility of needing to slow down in the recent paper. But, but it'll, it'll be very hard. But in any event, the worst case seems to me an intense arms race environment, whether racing among companies, racing among companies that's accentuated by competitive or even combative dynamics among nations and the use of companies, of the dynamics with the adversarial nation to fend off any regulation whatsoever, including regulation that might just slow things down. I mean, The minimal thing I would hope to see in the discourse is that people stop thinking that if something slows innovation, that's necessarily bad. It's like, no, you can't do that to us, that'll slow innovation. Well, maybe we're starting to approach a time when that will be a feature, not a bug. If you just slow things down a little bit and give us time to think about them. It's not like you have to worry about a true dead halt of progress, right? It's, it's, you know, that's not in the cards. I mean, even, even if you got a pause in training runs, which might not be a bad thing, but that would not be the end of progress by any means. Not— it wouldn't be the end of short-term progress in terms of applications, and it probably wouldn't, wouldn't be the last training run.

And Lord knows we've got a lot to figure out about how these things work internally. I mean, it is— has been striking how The more we have cracked open the black box, and there's certainly been a lot of progress there, they do look in many ways like strikingly similar to what we understand about our own cognition. I think that's been a, a fascinating— but we're probably only about as far in terms of understanding the AI's cognition as understanding our own. We should make faster progress on the AI's cognition because we can you know, manipulate them much more freely, obviously, than we can manipulate our own brains. But it's very much still a work in progress. So a little time for that, I think, yeah, in many scenarios could be a very good thing. Well, let's do this kind of zoom out to some of your— I don't know if it's— if you would describe this as your philosophy or just kind of observations, but Obviously, you've kind of made yourself synonymous to a degree with the concept of non-zero-sumness. And I'm kind of sketchy on—

I'm not sure my name springs to everyone's mind when they talk about a non-zero-sum game. I encourage that. I encourage you to say that. It's not true. It does for me anyway. But when you say it.

The book kind of sketches your this and related topics. But how do you kind of think about— there's this concept of the— I'm not even sure I'm saying it right— noosphere.

Yeah.

And you sort of have, I think, like, uh, an intuition that there's sort of a direction to evolution and to history, and that this is all kind of leading up to some sort of culmination. And I think some people kind of react to that as like, it sounds a little woo.

Teleological.

Other people, you know, probably kind of myself included, are like, maybe you would've reacted that way years ago, but now it's feeling a little more intuitive. So yeah, how do you kind of, what's the sort of center for you in terms of your relationship to these ideas of Purpose culmination.

Yeah. So first of all, noosphere, noosphere. I've heard it pronounced both ways. N O O S, the Greek word for mind. It was coined in 1923 by P in conscious reference to the biosphere. So you know, over evolutionary time, there was the geosphere before life, then the biosphere, and now the noosphere. You know, Teilhard saw, I think pretty presciently given when he was writing, that there was this technology was kind of creating this something that looked kind of like a global brain, you know, hooking people up in larger and larger networks of collaboration, international networks. So the human brain was— our brains are like neurons in this. In fact, he called it a brain of brains. So there's that. And one argument I make in the book is That we need— well, I think it can be illuminating to think of the AI revolution happening at the same time that this global brain thing is taking shape and pondering, for starters, the prospect that, oh, I guess the neurons could be made of silicon, couldn't they? And if some are made of silicon and some are carbon-based, what will the relationship between the two be? And so on. But I also think we need to ponder kind of the impetus behind the development of a global brain. What do you mean? Like, for example, there's a global market that's kind of a global brain, right? It does stuff. It moves resources around to where they're needed. It makes allocation decisions. And then there was, in principle, global governance, right? Some degree of international governance, which I think AI calls for. So I'll get back to all that, or at least lead up to it by answering the second part of your question about finality of evolution, or really of 2 evolutions in a sense, both biological evolution and what anthropologists call cultural evolution. In other words, the evolution of the bodies of information that are transmitted among humans that are not genetic in nature. Like all non-genetically transmitted information is part of cultural evolution. So religions, ideologies, and certainly technologies. So if you look at where those 2 evolutions together have taken us, I think you see undeniably directionality. Now, that doesn't mean there's a purpose, but let's just— To quickly review the directionality, you have these bare strands of self-replicating information, presumably at some point. They build cells, then more complex cells, then the cells form multicelled organisms. By the way, multicellularity has evolved a number of times independently. That suggests that there was a strong evolutionary impetus behind it. Evolution, it's like exploring inventive space.

Right.

And finding things. So many things have been multiply invented— wings, eyes, and so on. And then anyway, you get societies of multicelled organisms. In the case of our society, once you reach our level of intelligence, this other kind of evolution kicks in and our social organization starts growing too. Hunter-gatherer band, chieftain, etc. Now we're on the verge of— I would say we're on the verge of global community and we need to form one. if we're going to handle the technological challenges we face wisely, especially AI. Now, if you want— now, it seems the directionality— well, the direction is clear. There's been a direction. I guess directionality consists of the view that there's been a tendency toward this. And I think that's the case. You don't have to get wooish or talk about spooky forces. It's just the nuts and bolts of natural selection, And cultural evolution, I think, for discernible reasons, have taken us here. That's a lot of what my book Nonzero was about. I described the growth of complexity, both in biological and cultural evolution, as a certain kind of interplay between zero-sum and non-zero-sum dynamics. Now, of course, Teilhard de Chardin, back to the noosphere, he was, in addition to being a paleontologist, he was a Catholic priest and theologian. a radical theologian whose work was suppressed by the Catholic Church. But he certainly saw this, the noosphere, as a manifestation of divine will or manifestation of divinity. And he thought accordingly, I guess, that the thing had kind of a moral direction, that we were going to have to become better people morally to get over the conflicts that stand in the way of a kind of a coherent noosphere. I, you know, at a certain level of abstraction, I agree with him absolutely. And I talk about that in the book. Now, I don't have his assurance that this is divine will, so I can't be as optimistic as he may have been. But I do think I will say that, you know, you can look at a directional system and have a rational argument about whether it does seem to have some of the hallmarks of purpose, which again doesn't mean there are any spooky forces driving it. If it has a purpose, it just means It was set up either by some intelligent being or conceivably by some kind of meta-selective process. You know, just as you can say that natural selection itself imbues organisms with a kind of purpose, right? Getting genes in the next generation. I mean, that's kind of the overarching goal. That's a criterion of their, quote, design. You can imagine that happening. I save— look, I save almost all this for the appendix, so you can read the book without bumping into this stuff. But I, you know, Lee Smolin, the physicist, has this idea of cosmological natural selection. You know, it's speculative. He's not, he's not asserting it, but he's saying, you know, there could have been selection among universes. And basically that's what you would need if indeed the directionality of evolution were purposive and it wasn't designed by like a god or like aliens or any other intelligent being. You can imagine it being a metanatural selection process. So there's all that, which annoys some people. I even talk about some of the people I've annoyed in the book. But the one thing I want to emphasize is like when I say the God test, you know, it's the title of the book. I don't mean that this was set up by a God. I don't know. I'm agnostic on that. But we face, I think, the kind of test that God's are known for setting up. So if you look at the Bible, there are these, yes, salvation is possible, but you guys are going to have to shape up. Whether they mean you're going to have to worship Yahweh, you're going to have to be better people, whatever. I think we are going to have to become, in a sense, I don't want to sound too ambitious, but better people if the conflict that currently afflicts relations among within nations is to subside sufficiently for us to get through the AI revolution in good shape, because I do think it's a global challenge. So did I exhaust your curiosity about that subject, Nathan?

Yeah. And you brought a couple, brought to mind a couple other, just maybe riffs on it briefly. I think it's really informative sometimes to go to a little nature center. There's one not too far from my home that I've taken the kids to a couple times. You go there and every common species, like nature centers across the world have more in common than not, right? Like you'll see a snake, you'll see a frog, you'll, you know, there's bees right in every sort of corner of the world. And I'm always reminded when I go through those places, all of these were invasive species. Like these are all the winners that colonized the entire world, you know? Even the simple bee, right? I mean, obviously there's variation from place to place, but the fact that there's basically what amounts to a bee in every corner of the world is not the way it always was. You know, if humans were sitting around moralizing when the bees showed up, we would have called them invasive species and like wrung our hands about them. But, you know, now we just take it for granted, of course. But sometimes I think that AI might be like the ultimate invasive species. And it's just really important to keep in mind, I think, for people as we confront our current situation that like there were a lot of things that were displaced by all these, you know, these kind of global success stories, bees, frogs, and whatever, right? I mean, you got frogs that live under the sand in the desert and that freeze in the winter in the Arctic and thaw out in the spring. And a lot of things were displaced by those. So for us, we should, I think, be very— Mindful of the fact that the niche that we occupy as sort of things that don't survive super well, you know, without a lot of capital to support us is like pretty similar maybe to the niche that the AIs might be best suited to colonize. And the, the sort of plot armor that a lot of people tend to assume, you know, subconsciously, not, not consciously, right? But like, I think implicitly assume will kind of protect us, just wasn't there for a lot of other things when newcomers came onto the scene. Of course, we too ourselves, right, you know, drove our closest cousins and much of the megafauna to extinction. Like, these sort of major plot twists have happened like tons of times, and we just haven't really experienced them or internalized that. But boy, I think a long, you know, read of history there is like— Kind of sobering, to put it mildly.

Yeah, absolutely. And, you know, I think one virtue of taking that long evolutionary, you know, view is I hope it helps you appreciate the possibility that there's a very strong impetus behind the evolution of AI. And I do think in a number of ways, evolution is a helpful way to to think of it. And, you know, I've talked about arms races as, I mean, as if they were bad, but I'm not naive. I mean, arms races are always, you know, they're often part of a creative process in evolution. And one, you know, I emphasize in the book that, you know, in textbooks, in biology textbooks, you look up arms races, it's likely to talk about races between species, But arms races within species, including the human species, have been very important, including in the development, I think, of our deceptive tendencies and our skills of argumentation and for that matter, our power seeking. That's involved arms races within the species to get genes into the next generation. And so, you know, it's— You're not going to avoid the dynamic. I'm not suggesting that, that technology always has a little bit of that in it. I mainly want to say that, A, it's such a powerful impetus that we should reckon on this stuff continuing to advance. I mean, for that reason and the reason I alluded to, which is that it's just pretty clear given the nature of the technology that That there's a lot of potential for further advance. But B, we should worry about what you could call gratuitous arms races and particularly dangerous kinds of arms races that get in the way of our reckoning with the technology. But I hope the idea of evolutionary impetus will lead us to recognize we have to reckon with the technology. It's like it is a force. It's not, it's not And maybe a lot of technological evolution in retrospect is more of a force than we appreciated. This is a force that is unfolding very rapidly in real time and poses all kinds of challenges across a lot of domains. And even if you don't buy the hardcore sci-fi doomer scenarios, and I'm sorry to report that I'm unable to dismiss them entirely after carefully examining them. You know, when I talked to Eliezer in 2010, you can see I'm even kind of a jerk. I'm kind of dismissive of the scenarios. I no longer am. But even that aside, this is going to hit us along so many fronts, right? It's going to be disruptive in the not necessarily good sense of the term, you know, economically, politically, culturally, family life, friendship, you just name it. It's going to hit in a lot of places at once. And I just say it's going to be an earthquake and we need to start thinking about it now. I mean, maybe I'm sorry if I sound too much like an evangelist on this because, you know, a lot of your audience may already be AI safety pill, but God knows not everyone in the AI community is, and certainly not everyone shares my concerns about a specifically US-China arms race.

But Here's a quote from the book. It's almost like technological evolution is saying to us, look, we can do this the easy way or we can do it the hard way. There's going to be a global brain in the end. The question is whether you build it gradually, cooperatively, and carefully, or it gets hastily assembled amid crisis or chaos. Whether you build yourself a home or stumble your way into a prison. We can kind of go run down some of the downside scenarios and, you know, how we might minimize those risks. I always say the scarcest resource is a positive vision for the future. And I know you have been thinking about kind of international cooperation and you're still holding on to international law even in this might makes right moment that we're in internationally. Maybe we could start you know, on this section of the conversation with like, what's your positive vision for the future? If you, you know, if you could navigate all these pitfalls, where do you think we want to end up?

Okay. Before I cheer you up, let me just— let's dwell in the depths of despair just a little more, just to flesh out what you were— that quote, like, we can do this the easy way or the hard way. So there's a chapter on to a certain extent on Nick Bostrom's thinking called the singularity and the singleton. Singleton is this idea of a global coordinating mechanism, global governance. It could be anything. It could be a global democracy with very decentralized power, which would be my preference, run by humans. It could be run by AIs. It could be a totalitarian nightmare. It could be anything. But a mechanism, some mechanism of global coordination, he calls it, or global governance. And I had him on my podcast and we're pretty much on the same page about this. We agree that in recognition of the non-zero-sum dynamics among nations and so on that are posed by technologies, especially AI, certain kind of global governance makes sense. But— As he fleshes out in his famous book, Superintelligence, you can imagine some dark paths to global coordination of a dark kind, right? Like, you know, AI is used to stage some coup and just seize power. The AI seizes power and so on. My view is that one or another of these scenarios is indeed very likely. And that's what I mean when I say we can do this the easy way or the hard way. Or the hard way. In other words, humans can consciously steer us toward a system of international governance of this technology that involves as little centralization of power as possible, but as much as is necessary to keep the technology under control. And even in the event that it starts playing a very large, large, large role in governance itself, still maintaining a kind of a win-win relationship with it, right? So That's what I meant by the easy way or the hard way. Now, what are the chances that we get there the easy way? As I said, I think a certain amount of international conflict is going to have to subside, and that may take something you could call moral progress. I talk a lot about cognitive empathy in the book. I don't mean emotional empathy. I don't mean like feeling their pain. I just mean having a clear enough understanding of the perspective of other people and groups, including adversaries, that you can intelligently play non-zero-sum games with them and work things out that are in your own self-interest and by virtue of being non-zero-sum are of mutual benefit. So I do think we're going to have to make an advance of that. The cognitive biases that impede cognitive empathy are now pretty well known. I talk about them, attribution error and so on. And So we're going to have to have progress. As I said, I think, I think that can be part and parcel of just looking out for our own cognitive health as AI plays a more pervasive role in our lives. I like the term cognitive sovereignty. I didn't come up with it, but I think being mindful of how persuasive and subtly influential AI could be and perhaps not with your welfare, quote, in mind. I think it's good for us to tend to our kind of psychological robustness. And I do think cultivating cognitive empathy can be part of that. I also think, you know, Ilya Sutskever says something, he said something in a TED Talk that I quote to the effect of like, you know, what he thinks is that it's some point, an awareness of the magnitude of this will dawn and there will be a new inclination toward collaboration with other human beings in recognition that increasingly we humans are put in a non-zero-sum situation by the AI.

Right.

It's in our interest to coordinate. And doesn't mean we can't have a non-zero-sum relationship with the AI at all. It doesn't mean it has to be permanently in the— it's not necessarily an enemy, but it has the potential to be bad for us in ways that should, I think, exert a cohesive effect on us. And I think he's right. I mean, first of all, let me say there's been very recently, just Mythos alone shows you what a big difference One innovation can make. Okay, two things that you might not have predicted six weeks ago. One is there there is now a there is an official dialogue between the U.S. and China on AI safety. I think that's a consequence of mythos. I think the focus of it is more narrow for now than I'd like, but it's happening. And then secondly, the Trump administration. After Trump had ridiculed a Biden administration executive order that I think is similar in spirit to this new one that Trump has issued. You know, the Trump administration has issued an executive order about the government vetting particularly powerful models. Now, I think that kind of arrangement is something to keep an eye on. You know, whenever the government gets close to the power of AI, you know, that's one of the things you want to be careful about. Still, I personally think some of this is going to have to happen. And my main point is none of it— none of this was happening 2 months ago, just 2 months ago. And it didn't even take a catastrophe, right? It just took a— I'm not even sure Mythos qualifies as a near miss, but it got our attention to some of the nefarious potential of this stuff and it got us to focus. And I think, you know, AI is so different from technological challenges of the past. I mean, you know, in Nonzero, I talked about how technologies are making relations among nations more non-zero-sum. You know, genetic engineering, a new bioweapon is an example, or any pandemic, you know, caused by a lab leak would be an example where the logic should point us toward international governance of some kind, climate change, and on and on. But AI is just, I think it's an order of magnitude stronger in terms of how much it adds to the non-zero-sum logic. And it's different from these other technologies in a way that makes it hard to predict the psychological impact on the species as a whole. Because one kind of, you might say, downside of technological threats, traditionally is that they're not very much like the threats that natural selection designed us to cooperate in the face of. Like, if you see a horde of enemy humans coming at you, you suddenly feel closer to the people in your tribe, right? And if you see a bunch of animals coming, same thing, you and the group try to fend them off. But traditionally, technologies haven't been much like that. AI is just qualitatively different in a way that makes It's harder to predict the psychological impact. It's increasingly, I think, going to seem to people like, yes, a form of intelligence, like human intelligence. In some ways very appealing and even likable, in some ways scary. I can't predict the effect, but I think it's very likely there's going to be a large effect. I think it will in fairly short order, change the way people think about their relationship to the other people on the planet, and quite possibly in a constructive way. And by the way, AI— here's more optimism. I mean, it's some form of hope. AI can in principle be part of the solution. You know, sycophancy is not an inherent property of AIs. The market, you know, corporate incentives may encourage it, And people may encourage it, but you can equally well imagine an AI that would be great at, you know, showing you that actually, you know, your spouse, you know, there are things that you do that are as annoying as the things that your spouse does. And the, the, and more, more importantly and subtly, like this thing that that your spouse just exploded about. Like, there's a subtext there. The thing that set them off isn't the real thing. You know how that works? Like, there's a lot of kind of intelligent explaining that an intelligent system can do that makes us, I think, better and more considerate people, but that are in our self-interest. Like, we'd all rather live in a harmonious marriage, I think, right? And And so again, this is a theme I visited earlier, but I think, you know, the AIs have the technical capacity to help us make the, you know, the kind of psychological adaptations which I think in some cases will amount to a kind of a moral improvement that we need to make. And I think the more we're aware of that, The better. I would also say, you know, Peter Singer's book, The Expanding Circle, I think documents that over time, actually, our species has already undergone some moral improvement. We've expanded the circle of moral consideration, notwithstanding some backsliding. And I think further progress is possible. So, you know, and I'm not an optimistic person by nature. So If I can muster even this much optimism, it's probably a good sign.

One point that you made there that I think is really just worth driving home is just how broad the space of AI possibility is. I think that is really underappreciated by the public at large. It's more appreciated by the people developing the AIs because they've at least had an experience that was formative for me when I was doing the GPT-4 red team close to 4 years ago now. Amazing how much time.

Mm-hmm.

It feels both like not much time has passed and like, you know, eons have passed somehow in, in 4 years. But, you know, that, that model was simply the purely helpful model that would do whatever you asked it to do with no, no training around refusing or, you know, thinking, hey, maybe this isn't a good idea. It was just do whatever the user asks. And even that, you know, that, and that's common inside the frontier companies, at least I think it has been to date. You know, that may start to change as they hopefully get a little more serious about their policies around internal deployments as well. But even that is like, I think, dramatically underappreciated by the public at large that like all this kind of, you know, harmless, harmlessness training that they have is the product of a lot of hard work and a very intentional design choice. And it definitely doesn't have to be that way. And in fact, it'd probably be simpler to just like get it to do, you know, the task. And that certainly suggests like a much broader space of, of AI possibility that we're really only beginning to explore. Most of it I think is probably bad. You know, I don't think we like want to live in a world full of minds from, you know, randomly selected points in that space. But it is striking to me that we really are going so hard, so fast at like this one paradigm, you know, which you could say about the language model paradigm. You could say that about the, you know, the transformer itself. You could say it about the kind of HHH training. There's like a, you know, certainly some differences between how the leading companies kind of think about their safety training, but.

Yeah.

They're more similar than different in the grand scheme of things. So we're, we're just like very, very focused right now in a very small space of AI possibility. And I think that is dramatically underappreciated.

Yeah, I think so. And I think that, that the potential design space may be explored in some interesting ways. I can well imagine that religious groups will, you know, it's— I think it's significant, of course, that there was a papal encyclical on this technology. That says something about the kind of attention it's getting suddenly. But I think you can imagine religious denominations and other groups and nonprofits and so on, like awarding their seal of approval to certain models. Now, that's not necessarily going to be a good thing. It could be, it could be a model that says, yeah, you guys are right and everyone else is wrong. You know, religions have been known to have that inclination at times, but they've also been known to have other inclinations. And in any event, my main point is, you know, people may react to what I said earlier, like, you know, we as the consumers have the power. Let's, let's get the psychologically healthy models out there and give them— use our dollars to give them positive reinforcement to companies for making them. You may see more market power via these groups of various kinds that either recommend models or even say this is the official model of X religious denomination or X, I don't know, political party. And again, this is not necessarily a healthy dynamic, but I do think I would encourage people to think about how to make it healthy. Right. Like, I can imagine philanthropic money being well spent to You know, make such things happen in the name of the greater good. We all have different ideas, I guess, to some extent what the greater good is. But I do think more than ever we should put a premium on tamping down the tribalism, both internationally and intranationally. And, and AI can help. It really can. It's amazing. I had this long conversation with Gemini that Is, is like a whole chapter. And it really came off as wise. Now, that is a function partly of the specific questions I was asking, right? I wasn't asking, you know, can you help me steal, you know, this, this woman from her current boyfriend or anything? But, you know, I was asking questions about the human predicament and the human future. But The wisdom is there. The point is the wisdom is in there. And leaving aside, you know, kind of wisdom as a, as a body of thought that can be tapped, you know, the tendencies of guidance are there that, that could, could make us better people. It's a, it's a project to be approached with great care, but I think it's better than not approaching it at all.

So this notion of becoming better people, I think It's a little bold to say, but I do feel like the AI phenomenon has made me a better person in the sense that for most of my career prior to this, I was an entrepreneur and I was mostly focused on being successful, you know, making the company work, not like winning at all costs by any means. I, I was never like that extreme, but like my focus was on making my project successful.

Yeah.

And I think that's fine. It's like made the world go round to a certain degree. But in the face of the AI phenomenon, I'm like, it's much less important whether or not I am successful and much more important how the overall thing unfolds. And to the degree that I can put a positive dent or nudge in that trajectory, then that's what I want to do. And I'm not, too worried about like monetizing that. You know, easy for me to say as a person of certainly still substantial, although not extreme by American standards, privilege. But truly, I feel like I'm much more willing to and much more inclined to just think like, what is the overall pro-social good thing that I can do? And kind of trust that if this all works out, you know, like there'll be plenty to go around. How do you think that can become— how can we accelerate that process? Like, and who do we look to? Do we look to the Pope? Do we look to the Dalai Lama? Do you have like personal moral authority heroes that you, you know, would like to become more influential? It just seems like, you know, I have felt that, but I don't know how to translate that into something that, you know, will become like a a contagious phenomenon. And it seems like given the timescales that we're operating on, it really needs to take off pretty quick.

Yeah. Well, first of all, and I'm not just flattering you because I've already succeeded in getting on your podcast, so I don't need to flatter you anymore for purposes of book promotion. But you have struck me as a fairly exemplary person all along. Now, I wasn't aware of you before you encountered AI, so maybe you were like a criminal or something before, but You seem to me, you know, broad-minded and not— you don't seem to me very egotistical. I suspect you were more like the person you just described before AI than you're letting on. But in any event, I'm glad you're like that. You know, what would I say? I mean, I talk a little in the book about I mean, I'm a fan of mindfulness. My last book was on kind of the psychology and philosophy behind, well, behind mindfulness meditation and the logic of certain aspects of Buddhist philosophy and psychology. And I really am a believer. So I encourage people to at least explore that. I mean, in terms of particular figures, I don't— I would, if somebody came to mind, I might be reluctant to mention them because, you know, people kind of have to find what's right for them. I would just say, well, it's so hard. I was gonna say if people, if anybody is in the habit of depicting members of any group in a very unflattering light, be skeptical of that person. And people do that and they do it subtly. That's not the same as saying that you shouldn't— you shouldn't think it's better if one side prevails in a given war or whatever. It's, you know, those kinds of things can have important consequences, those kinds of outcomes. But it's just so— it's so pervasive. The unflattering depiction of groups is so pervasive. And I try to— I just try to stay away from it. And I would encourage you to find people who aren't doing that. I guess that's like hopelessly abstract guidance. I Um, you know, and then there are particular— well, I stay away from people who on social media are saying, look at this piece of outrageous behavior by this member of this group, even if they're not saying that this member of this group part, right? Because, you know, if it's implicit that, oh, you know, can you believe that the Democrats are doing stuff like this, or the Republicans are doing stuff like this, or the the Zionists or the Palestinians or whatever, people who point to outrageous behavior as if it were representative of the whole group, stay away from them. It's a temptation we all have, I think, because if we feel strongly about particular ideological issues, but I don't know. It's a good question. Where is our superhero? Is that what you're Kind of. I mean, I have to be honest and say American politics doesn't seem to be full of them right now, in my view. But I'm struggling. You help me, Nathan. Who are your— who are your moral heroes? Who are you? Do you have a guiding light?

Well, I'll first add one of— one new thing to your stay away from list, and that is pay-per-click advertising. I would say I mostly Didn't get too far down to the bottom of the barrel in my, you know, entrepreneurial activities when it involved these kind of pay-per-click. But I mean, talk about an extremely competitive dynamic where the, the whole rules of the game are set up for everybody to basically compete for who can get the, you know, the neurons in the subject to like light up in just the right way to get the click, you know, and ultimately to get the payment. And You know, you, you really, that's an environment where it's very hard to compete at all effectively without starting to make some compromises, you know? And again, I don't think I went too far, but certainly you like cannot be primarily truth-seeking with your ad copy and expect to win in the pay-per-click game. So I do think that is, you know, that's about as toxic an environment as you'll find.

And this is, This has hurt journalism. When Mike Kinsley founded Slate, he said to me— this is how long ago this was— he said, do you realize we're going to now have statistics on how many people read each individual article? And he said, and you know, I think I'm going to try to keep that information away from the writers. That was a profoundly wise impulse, even if it was impossible to actually realize. And you see the damage now. I mean, the standard business model of mainstream media has just become tribal on one side or the other because they're paying attention to, you know, that they're looking for the headlines that get the clicks. And because these are challenging times for media, even for The New York Times, and The New York Times is playing this game. You know, I did a piece for The Washington Post several years ago, actually arguing right around the time of GPT-4. I said, you know, we really need to start pursuing a rapprochement with China. That was kind of the argument. And there was this headline, and they said, "What do you think of this headline?" I said, "Great." And then I go look at the piece, you know, hours later, and it's no longer the headline. What had happened is they did A/B testing, and and the first line was now AI is dangerous, so blah. And and it was a headline that would not actually succeed in attracting the kind of readers who would who would read the whole piece. Actually, it wasn't even a good. Version in a way of pay-per-click. But it was— this is the game now in media. And, you know, media have become a tribalizing force, partly by virtue of just trying to remain profitable. So, yeah, beware of anything that is click-driven. And unfortunately, everything is click-driven, certainly including Social media algorithms, you know, but but manicure your you know like use lists on Twitter. You know, I I've learned a lot from my AI list, and it's I think algorithm free. It's just the people in the list in the order of their tweets.

Again, easier easy easy for me to say. I guess I'll say because I think I've been very fortunate with this podcast and the fact that there's like a ton of money flowing around the AI space. means that, you know, we can get sponsors in a way that would probably be very difficult for many other people in many other niches. But I've had pretty good luck with just, first of all, just trying to keep the true north of the whole thing, my own personal learning. I always try to keep that in mind. And then also having specific individuals that I respect in mind as the audience is another, I think, good mental model for me.

And you're one of those.

Mm-hmm.

people. And, you know, I think of Dean Ball all the time as somebody who I'm like, if I can be of use to somebody like you or somebody like Dean, I don't really care what the numbers are too much. You know, there aren't that many Deans out there, and I'd rather have one of him, you know, than probably, I don't even know, a million random, you know, YouTubers. I think the ratio truly might be that high. So, you know, how far that takes people in various walks of life, I think, is— you know, very unclear. I think the Pope comes to mind going back to, you know, who are, who are my heroes?

Yeah.

I, I have been really impressed with, um, with the Pope. I've been really impressed with the Dalai Lama. I mean, I, I think those are two, unfortunately the Dalai Lama is, you know, probably not gonna be with us all that much longer. How do you, I don't know who else I would put on that list. Not, not too many. Maybe another way to ask the question is like, Let's say you are, you know, put into little Marco Rubio's shoes and all of a sudden it's like your job to, to navigate the US-China relationship. How would you think about going about it? You know, that you hear all these things like it's impossible, you know, we can't trust them. Obviously a lot of that denies our agency and Sowing an environment of mutual distrust, but we are where we are. You know, if you're suddenly in a position of power, what's your approach?

It's a good question. I mean, I immediately think of some things that maybe the secretary—well, the Secretary of State in principle could do the kinds of things I'd recommend. It's just it would be hard not to get fired.

But we can make you for the sake of this discussion.

Could I just be emperor of the world for just a few minutes? I don't know if I can go higher.

President of the United States is maybe the size. I can go.

Well, you know, one thing Trump has done occasionally, and I'm not a Trump supporter by and large, but he has, he has done the cognitive empathy thing. He's like, he's done a kind of corrective for our natural bias of self-righteousness. You know, he's like, what, you think we're so great? Like, you know, some other country will do something at a time when— I think he's done this a couple of times in the course of even the Iran war when he was maybe trying to get a deal done and he didn't want people to focus Unduly on some attack that Iran had done in response to some other thing. And it's just kind of like, what, you think our hands are free of blood? He's every once in a while done that. I think we need a ton of that because the biases— once a nation is deemed an adversary, the biases, the filters, the cognitive biases can be so strong, especially when a nation is very different. I mean, obviously, China's system of government is different from ours and, you know, more authoritarian, more autocratic and all that. And there obviously is this fear that if they win the race to superintelligence or something, they will try to impose their system of government on us. Now, this isn't the question you asked, but I think that's an assumption that simply has not been evaluated carefully because there really is no evidence that China's at That that's the Chinese business model, I think. To to I don't think it's in a weird way very ideologically driven in its conception of how it wants to relate to the world. But that's that that's separate. What what I would say, though, is like make at least make people understand how a lot of Chinese view us. And understand that there are filters at work on both sides. Like we think whatever we're doing, whether it's chip controls or Taiwan or whatever, we have these reasons for the policies, whether it's defending Taiwan, denying China certain technologies, these. Explanations that are fundamentally defensive in nature, right? Like with the chip controls, we're afraid of what they'll do to us if they get the chips and the AI. With Taiwan, we're afraid of what they'll do to Taiwan and so on. I'm not saying none of those have any degree of validity, but I would point out, like, that's not what the Chinese think our motivation is. That's the first thing to understand. It is widely believed in China that our goal is to keep China down, period. We just want to be dominant and we don't like a rising power and we are going to try to frustrate them. You may believe that that's completely confused and all of our rationales are completely right, fine. But it's still of some strategic value to at least understand the way they're processing the information and how they're going to react to things you do. So that's job one. And I would, by way of illustrating how powerful the filters on information are now, have you heard that? And apparently this is true. I haven't pinned it down with 99.9% confidence, but apparently it's really happened. So China, like 30 years ago or something, they contracted with Boeing to design like their Air Force One. Okay? They wanted something their premier could fly in. Now, apparently what we did, such is the relationship between our government and Boeing, that we filled it with all this spying stuff. Even the bedroom, even the premier's bedroom, little bedroom on the plane had listening devices. Okay. Now, my main point is, had you heard about this, Nathan?

No, no, I've never heard this.

That's my main point. Everyone in China has heard about it. I don't even know if it's true, but I think it is. I mean, I interrogated some of my allies and followed some sources. I think it's true, but my main point is this is just typical. There are the reason you haven't heard about it isn't because there's no chance it's true. It's just it's not the kind of thing our media system really wants to amplify. And and similarly, our picture of what's going on in China, there are definitely some some things I would consider bad going on there, but I guarantee our view of them. Our view of even what they are is not the same as the average Chinese person's. And in some cases, I think our view may be kind of distorted because it's like, who wants to be the person who says, I don't think that human rights situation is quite as bad as you're saying? Nobody wants to say that, right? And so there are just— I would just start out by you know, reexamining the sense of threat in the light of an understanding of the cognitive biases and media filters at work and understand that China feels threatened by us. It's like, you know, everyone, you know, Israel's attack on Iran, I think, was motivated by a genuine belief on the part of most Israelis that this was a defensive thing to do. How many people in Iran see it that way? And vice versa. I could flip the tables and describe that. It is just so common to have this, the asymmetry of perception that, and it leads to the positive feedback cycle. This famously led to, well, contributed to World War I as military postures meant defensively were taken as offensively. So there was a countermove that was conceived of as defensive but looked offensive to the other side. So there was another countermove and so on. I mean, I can tell you what policies I'd favor if I were president, which is actually the question you asked. But one reason in my book I spend time on policies but more time on this psychological stuff is like, As long as it is easy for people who want to amplify our sense of threat to do that, it's going to be hard whether you know and politicians very often have a political interest in in threat amplification, threat inflation, and arms makers do and so on. And I I you know I can lay out the policies. But if you want to hear them, yeah, let's do it.

I mean, what— well, what's a grand bargain look like? That is kind of— maybe—

Well, for starters, we just have to get out of the business of trying to govern other countries. Okay? Yes, there are, there are bad human rights things. There are bad things in the US, right? Like the percentage of the US population in prison That's in prison way, way, way, way higher than in China. Okay, that's not good. And China could say, oh man, that's horrible. We're not going to do business with you. But China just doesn't really do that. We do. And the thing about our human rights policies is they basically never work and they often backfire. Like our approach to helping the Cuban people has been decades of sanctions that have immiserated them and now a full-on blockade. This just happens again and again and again. The sanctions, they just play a domestic US political function and they serve the purpose of various— and there are interest groups that believe in them wholeheartedly. But first part of a rational policy toward the world is get out of the business of trying to remake every country in our image because A, you never succeed, B, when you try, disaster ensues. Whether it's a regime change war or whatever. The record seems to be pretty clear. Sanctions don't work. They often backfire, hurt the people you're trying to help. Regime change typically makes things worse. So just stop. And by the way, this really is what the UN Charter kind of says. I mean, yes, there is the Universal Declaration of Human Rights. It deserves our respect. But in terms of the actual force of the UN, they focus it on transborder aggression. And that entailed a tremendous respect for national sovereignty. And the idea was like, you don't get to attack another nation unless it attacked you, okay? You don't get to attack it because you think its government could be less repressive. You don't get to attack it because of X, Y, and Z. The UN Charter is a treaty we ratified. And by the way, the US Constitution says that ratified treaties are the supreme law of the land. So, you know, I think we need to get back to the spirit of the UN Charter, which is just say, okay, look, these are not the ideal governments. Personally, I don't think our government is the ideal government, but you just deal with the nations you have and you start focusing on problems you share. And there are plenty of them. You just— that would be the fundamental philosophical shift our foreign policy needs. And I think you'd find that that would relieve a fair amount of tension with China right there if you quit lecturing them about that. Now, I'm not saying this is all our fault. Xi Jinping is more— he has concentrated power within his office in China, so he's closer to autocratic. Than past leaders of China. He's more authoritarian. And China has been, you know, sometimes belligerent regionally. But I mean, to give you another exercise of cognitive empathy, you know, I heard this guy Nick Burns, he used to be ambassador to China, say when they— somebody, some podcast asked him, well, what, what should we— what should Trump have done in China that he didn't do? He said, well, he should have talked to him about how, you know, they've been kind of throwing their weight around, like with the Philippines and stuff. And I just thought, Do you not understand that once we have invaded Venezuela, we have a blockade around Cuba? You just—they're just going to laugh at you if you start saying quit bumping Philippines boats, right? Like, you know, you—you—you have to understand that we are widely viewed as completely hypocritical. And so, and—and once you see that, I don't know. It seems to me that you just realize well. Either we are going to have to start playing by the rules or we're going to have to do a little less lecturing to other countries about playing by the rules because human nature is just such that that kind of hypocrisy doesn't fly. So I don't know, I'm sure I've alienated half your audience, but—

Probably more than half, but that's okay.

Then explain to me what is so radical and crazy about what I'm saying. Now, if you don't accept my premises, If you think, no, I think you're wrong. I think China is ideologically devoted to turning the entire world into a giant authoritarian autocracy. Great. I'm happy to have that argument. One of my big frustrations with the current dialogue, especially in the context of AI, is that the important debates aren't being had. Well, do chip controls make it more likely that China will invade Taiwan? There's just not, not enough discussion of things. And, uh, so.

So I think like doing what you describe, I'm intuitively certainly mostly in favor. I mean, I remember first learning about Iran's, you know, however many decades of recent history and understanding the role that the United States had played in that and being like, well, no wonder they hate our guts. Like, it's really, yeah. So, you know, that I guess maybe comes a little bit more intuitively to me than most. And we haven't even touched on the, you know, century of humiliation that the, you know, Chinese civilization broadly, you know, is still kind of feels that it's recovering from. So, you know, being a little less hypocritical or, you know, being maybe a little warmer in all kinds of ways seem good. But we are going to need some like actual positive cooperation and some like new rules. And, um, do you have a kind of a vision of like what you would, because the, you know, it's hard to look inside somebody's data center, right? I mean, we would, that would be a really deep level of joint monitoring, which it might have to go that far, you know, it's, but it's— It's a kind of a long way from here. I'm not sure how we would, you know, what kind of, you've kind of articulated step one of like, be a little less hypocritical and a little warmer. And hopefully that'll, you know, open up the next steps. But like, how do you, can you chart out a little bit more what you think those next steps are? How do we actually get to the point where we're like cooperating on AI such that we're not, you know, entering into— arms race doom loop with each other?

Yeah. I mean, I would first say the point of understanding why, for example, Iran finds us threatening is not to say, well, then they're right and we're wrong in our policies, or they're right to encourage Iraqi militias to kill American soldiers. But it is to just make it clear that something we didn't even think about when we occupied Iraq, which is that from Iran's point of view, that is an existential threat. We just said they were part of the Axis of Evil. Now we've got troops in this country that had this huge war with Iran, and we supported Iraq in the war, and on and on. So it's just to make us understand what the likely consequences of things we do are. Okay? That it's not about saying, oh, well, then you're right because you're viewing things this way. It's just to try to make people more predictable from our point of view so that we can Proceed more wisely. Now, on China, I would say, first of all, job one, acknowledge what you just said, that I think the degree of transparency that is going to be required with AI for all of us to feel reassured is much more challenging to achieve than in the case of nuclear weapons. Okay. That's a relatively easy case to have an arms control agreement about nuclear weapons. So we should think about that. Now, one implication of that, I think, is that that should help us understand some of the virtues of economic and cultural and scientific engagement because there's something— I'm trying to popularize this phrase, organic transparency, because I do think— You know, when business people, Chinese and Americans, are having drinks, and scientists are, and performers are, and so on, you just know more about what's going on in the other country than you would otherwise. I am hopeful that AI, once we grasp the implications, will give us— will foster the relationship to China that we're accustomed to having with, like, allies, right? Like, You know, during the Cold War with like France, England, whatever, we're pretty sure there's no, you know, deeply anti-American thing going on there. And that's possible. That's for various reasons. But one of them is we're so deeply engaged, so much travel back and forth, corporate interaction that we just, we know more like we'd know if something were afoot. And I think that is an important like policy thing is to See, now economic engagement can have its downside byproducts, especially if it happens too fast without, you know, and I could go on about that, but I think there's tremendous virtue there. I think we need to do fewer things that gratuitously make China feel threatened by us to no good effect. And I think the rhetoric has been so sloppy from our side that there's been a lot of just literally gratuitous threatening effect. And then we need to do what happily Trump is starting to do, sit down and talk to them about AI and, and see where the conversation goes. But I think the main thing is, is to slowly move them out of the psychological category of adversary and have us move out of that category from their point of view. I mean, competitor, fine. But I think a lot of things become possible. And again, I think the magnitude of the AI challenge is going to become more apparent and make certain things more plausible than they are. But in a way, the main thing I'd say is, look, my reading is that we are going to have to collaborate pretty extensively with China to get through the AI revolution in good shape. Partly because it will be so domestically destabilizing otherwise in the US if it's just a, you know, if we proceed at a headlong pace. And so even if it's a huge challenge, I think it's in our self-interest to pursue it and take it seriously. And if we fail, we fail. But my view is, if you think that headlong race to superintelligence is a way to keep us out of a horrible war or to keep our own country from being destabilized. I just think that's wrong. And the final thing I'd say is a destabilized nation where too much is happening too fast and there's disorder and there's chaos, that is the backdoor to the kind of authoritarianism that supposedly the race with China is going to avoid, right? It's like when there's chaos and disorder, That's when authoritarian takeover from within is most plausible.

I often say we should remember the real aliens in this situation are the AIs, not the Chinese. You put that in a somewhat similar spirit. I'd say whichever species understands the other better is the species with the agency. Which species, meaning AIs or humans, will have the agency? I think that is an open question, actually. Like, who's gonna understand who better? Will the AIs understand us better, or will we understand the AIs better? But what's definitely clear is if we try, we should be able to understand the Chinese better than we understand the, than we understand the AIs, or that they understand us. It, it sure seems like that kind of communication should be possible.

We're definitely capable of it. And, and the funny thing is it fosters clearer understanding of us. By them. It is, you know, senses of threat, a symmetrical sense of threat is mutually reinforcing and mutually amplifying. And it can work in the other direction too. So yeah, I mean, on the agency thing, I mean, I know I'm sounding like whatever I sound like and sounding like making the book sound like whatever it sounds like. But one of the chapters I most like is the one you kind of implicitly alluded to where I raised the, like, who has the agency question, the, the Yann LeCun chapter where he says, we're the ones, you know, we researchers, he says to Eliezer, we researchers have the agent, you know, we have the agency. We're building the AI. We're the ones with the agency. I think, as you said, it's actually an open question because the AIs seem to be pretty good at understanding us. And we have to understand—

They've read all your work for sure. So that gives them a big leg up.

There's that, but there's that. And the persuasive powers, their persuasive powers are starting to be documented. But look, it's a completely enthralling technology. I wish I could spend more time with it. I've spent too much time writing about it and not enough doing it. I've had amazing experiences. We both had different kinds of cancer experiences that it helped us get through. And, you know, it's mind-blowing and often in a good way. And I think it's in principle possible to preserve that part. Without letting the, the other part get outta control.

Do you have any optimism around what you might call like positive nationalism? You know, we're obviously talking right now as the World Cup is just a few days underway. And yeah, I have a, a recording coming up with Sam Rodriguez, who's the CEO of Edison Scientific. He proposed this project where he basically was saying we should be— he might phrase it a little differently. I'll describe my understanding as we should be racing the Chinese to cure all the diseases. Let's have, like I've said before, like I want a medal count for like how many types of cancer each country has cured. And let's try to race to the top of that leaderboard. Do you think that something like that can work? You know, if you're a philanthropist, You know, would you be interested in trying to fund a sort of, you know, positive AI outcome Olympics sort of dynamic?

I can kind of see that. I mean, that specific race would depend partly on whether Max Tegmark is right. And as I understand him saying that you actually can separate AI as a tool for specific uses from rapid progress in AI along more uncontrollable dimensions. You know, I mean, but if the race, if the race to cure disease, you know, makes it easier to build a bioweapon, then there's that whole conversation to have. But what your question stirred in me is the question of whether what I was describing before, like, you know, Using AI to enrich human moral psychology, to make us better people, whether that could be part of the competition in a way, or whether— leave aside whether it's national competition, whether, you know, you could offer some kind of prize for that. I do think— I mean, I think, look, Sports per se, this is not the question you asked, but international sports can be a great thing. Obviously, soccer is a well-known example of how it can get unfortunately tribal, but a lot of it is up to the players. I mean, the way they conduct themselves with respect to the other players and the things they say, they're very important and powerful role models. I'm curious as to your take. Can you separate, as I think Max Tegmark hopes and advocates the, the use of AI for positive developments and breakthroughs from its use for more unfortunate purposes?

It's a huge and super important question for sure. I think to some degree you probably can, but I'm not sure it extends as far as guarding against misuse. In other words, like, I think if you created an incredible cell model that, you know, could predict if I do this to a cell, what's it going to do? Or you'd even think bigger than that, right? An organism model. If I give this drug to this person, what's their next health, you know, timestamp readout going to be? Are they going to be healthier or less healthy? You know, how long are they going to live? Let's say you could create, you know, a sort of perfect— oracle that just makes these predictions for what the, you know, the consequence of some intervention on human health is going to be. I think you could make something that does that really well, that is like not about to tip over into becoming an agent that wants to, you know, propagate itself on the internet or, you know, seek power or whatever. I do think there's a certain amount of safety in narrowness of scope that— Could be part of the way that we sort of create a, you know, a constellation or a Rube Goldberg sort of contraption out of a bunch of different AIs to give us what we want without, you know, fear that there's going to be sort of too much agency in any part of the system. At the same time, I do think you still have the people problem, you know, that like you give that thing to a bad actor and now you've got, you You know, it's, it's hard to prevent misuse of, of something like that. So I guess that's sort of a halfway answer. I, I, I would guess that we can control the AIs better through segmentation and kind of fine-graining of their purpose. And maybe that even has some spillovers into preventing misuse. Certainly it'd be a lot harder, you know, to use all these different constellation of AIs in a coordinated way to do something bad. Versus just telling your one AI, like, let's do this thing. But I don't think it gets us probably entirely out of the, the woods. And, you know, the, you may have seen the old Gwern essay about like why oracle AIs want to be agents. And it's, I think he does make a pretty compelling point that there is like just a lot of gravity toward this sort of agentic, um, form factor because even to just to get the right answer to a lot of questions, you kind of have to go out and find it.

Yeah.

You know, and the more you, you need to search and if search isn't enough, then you might need to like run an experiment. And so, you know, truth seeking in the limit is still kind of, you know, when it gets beyond what's like readily known, it still requires a certain amount of agency to get there. I think there, I am bullish on that approach to a degree, but I don't think it like solves all our problems.

Yeah. I mean, one thing that occurred to me is to the extent that you could separate the functionality and, you know, have a race towards some sort of good medical application, it would help if there were a prior agreement that, you know, whoever wins the trophy, the benefits are going to be shared, right? Like, so both, both countries are going to benefit from the competitive dynamic, have maybe some special arrangement about the intellectual property. The other thing I'd say relative to the question of like national pride is, you know, I like the idea of taking pride in the values that you see your nation as representing, if they're good values. I think part of the problem is that people by nature are not that clear-eyed about that stuff. We're kind of designed by natural selection to convince ourselves That we're better than we are. There's a lot of evidence about this. But what I'd like to see, I guess, is America take pride in the idea that, like, we're now at this threshold. Okay? This technology, like it or not, is here. And if you agree with the many people who I think say when you press them, Even if current competitive dynamics don't encourage them to say it out loud, but believe that ultimately some degree of international coordination and transparency in governance is going to be in order. How about our mission as the world's leading AI power and for the time being, at least, maybe the world's military and economic power, like our mission being To guide the world to the promised land, where level of conflict is low enough and the level of coordination is high enough that this thing can work out. Like, why don't we focus on that? And I mean, to me, I'm sorry, just the idea that what we need is a breakneck race with anybody is just borderline. I don't understand how— look, I understand how a lot of people don't share my view about how powerful this is and is going to become and so on. That's fine. I guess what bothers— what I don't totally get is the people who are self-professed AI safety hawks and are still talking like we need a breakneck international race. And I just do not— I don't, I don't get that. And I think with that, I've alienated whoever I had not alienated before.

I think I'll, I'll give you what I think is like the short steel man. And I do think it's a real thread the needle scenario. But basically, I think the argument that you would hear from like Anthropic folks is this is gonna be really hard. And we probably won't be able to do it until we have sufficiently powerful AIs to help. There's gonna be sort of a critical phase where we go from being the smartest entities around to not the smartest entities around. And to manage that handoff well is gonna be extremely difficult. And so whatever we can do to maximize Our chance, our being humanity broadly, but also Anthropic specifically, of doing that effectively, we should do. And if that means racing out to have as much lead as we possibly can, then that'll be good because that will give us this buffer that then we can burn down that lead that we have over other AI developers in that critical moment when we have that most powerful AI. So that hopefully us and our best AIs available at the time will sort of be the watchmaker that sets things on the right path, and then we can all live happily ever after. But we need as much time in that critical period as we can get. And so that's why we race now, so we can, you know, have as much time to be as thoughtful and, you know, cautious as we can be at that later critical phase. All of which, by the way, I think they're actually pretty clear-eyed, is like maybe going to be 3 to 6 months, you know, in 2028 or 2029. I think that's like literally how they're thinking about it. So I don't like those odds. I don't like that plan that much. I don't like our odds if that is the plan that we're going to pursue. But I think that's like, that's, I think, reasonably good. I mean, somebody, you know, correct me if I'm wrong. I don't think too many Anthropic people have time for 2-hour podcasts these days, but I'd love to be, you know, corrected if that's not the way they're thinking about it. But that's my sense right now as to the prevailing Anthropic thought.

Yeah, I'd say a couple of things. I mean, first of all, as was pointed out in this superintelligence strategy paper co-authored by Dan Hendricks and Eric Schmidt and Alexander— is it Wang, or do you pronounce it Wang? The guy who started Scale AI, I think. Anyway, I think this was not emphasized enough in the paper, frankly. But one thing they said is, you know, if you're racing towards superintelligence and there's these 2 superpowers, there comes a time when one of them is strongly incentivized to derail the leader's AI effort. And that may include things like bombing data centers or aggressive cyberattacks. So There's an inherent kind of destabilization in approaching the superintelligence threshold if they both— if both parties buy the idea that that is going to confer utter military hegemony. And it is the official position of Dario Amadei that it's going to confer that and that we should race headlong toward it, as I understand his position. So first of all, you're kind of inviting China to attack you and/or do something highly destabilizing. The second thing I'd say is, you know, some of the assumptions about alignment working out that we talked about at the beginning that are maybe not airtight, I think enter into the scenario you described, not to mention, well, there's a lot of assumptions, a lot of assumptions that I think Have to be right for that to work out. But the other thing I'd say is it just seems to me that once you appreciate the delicacy of the challenge, I mean, you got to thread one kind of needle or another. That's a good comeback to my argument. If I say you guys are talking about threading a needle, they'll say, well, so are you. You're talking about like what, rappers from China and blah, blah, blah. We got to thread— we have to overcome some serious Seemingly formidable challenge in any scenario. But it does seem to me that once you appreciate that, maybe you should appreciate the logic behind just slowing down rather than speeding up. Like, as you know, like maybe the more time we have to think about it, the better. The more time we have to develop the kind of international collaboration that would allow us to do this the easy way and so on. So that's the kind of reply I'd give to that. I mean, I will say even Anthropic in this latest paper about recursive self-improvement says at the end, I mean, it doesn't say what the headline said it said. The headline said it was calling for a global pause, which it wasn't. But they did say, you know, maybe the time will come to slow down or pause. And so we think we should now enter a period of studying what that would take. And My question is, tell me if I'm being too big a jerk here, Nathan. But by the way, I hope you've— if I'd been a jerk up until now, you would have jumped in and said, Bob, you're being a jerk. So, but the— You know, my view is like, wait, you knew where you've been saying we'd get to this point of recursive self-improvement. Surely this isn't the first time it's occurred to you that things might moving might move so fast at that point that we should slow down or pause in advance. And you mean to tell me you haven't yet given any systematic thought to how we'd work that out? I mean, they're just going to kind of start the study process now. I don't, I don't, I don't know. I'd be curious to know what dynamics led to that paper. I'd like to think there's pressure within Anthropic in favor of talking about a pause and a slowdown. Maybe Dario is not so enthusiastic because it's inconsistent with his advocated China policy. But in any event, I agree it's time that a lot of think tanks started focusing on this question of how do we coordinate globally to do things generally with AI, including slow down?

I, I can definitely tell you that there has been thought, you know, for years that has gone into what would this look like. And I'm not sure how much, you know, traction it has got either in the sense of like coming up with good ideas or, you know, getting people to buy into those good ideas. There's been a bunch of work around like hardware governance, you know, sort of, uh, and there are, I believe that there are even mechanisms that like allow for sort of coordination of chip location. There's, you know, proposals to take that a lot farther and say, you know, can we get chips or sort of fleets of chips, data center, you know, level deployments to faithfully report back to some central authority? Like, are we doing inference here or are we doing training? You know, things along those lines. Mm-hmm. That are part idea, but you know definitely need like a serious technical implementation. There was, I think, a lot of a lot more hope that under a Biden Harris or you know any Democratic administration that there would be some push for that, and then it kind of got backburnered. You know, with the Trump win in in 24, I don't have as good of an account for why. Now, you know, like, and of course OpenAI has sort of signaled their openness to possibly the need for a coordinated slowdown as well. Why has this entered the OpenAI internal Overton window? And, you know, why are they also willing to say it right now? I don't have a super great account for that. I actually did speak to somebody at OpenAI who kind of said, I can tell you there's been more talk of it lately. Didn't seem like they Had a super clear, you know, mechanistic understanding of why that had happened recently. But maybe one theory is, is simply, you know, along the lines of what you said, you know, what you attributed to Ilya and your quote from the book, the more godlike AI becomes, the more it will put the fear of God in us and the better we'll get at cooperating to tame its ferocity. I think there is something—

Yikes.

You know, like, With the unsolved math problems and all these kind of hills that they, the AIs just keep climbing. I, I do get the sense a little bit that they're kind of spooked by their own speed of progress. I think Anthropic people kind of more always believe that, and maybe OpenAI people have, you know, and obviously there's been a lot of leadership change there, so it's probably been more contingent for them. But I, I do get the sense that maybe something like what you're describing there is. Yeah. actually kind of happening where the fear of God is like starting to motivate a bit more than just, yeah, well, can we make the next thing smarter than the last?

Yeah, the sense of acceleration is tangible to me, and I think they're feeling it. They're doing it. They know more about it than I do because they're seeing what they're doing inside. I mean, they're already, you know, they're, they're using it to accelerate the, you know, the building of the next generation. But it seems to me anyone who's paying attention, and I agree, you know, look, granted, not many people have much incentive to pay attention as I do, but the, you know, singularity is near. I guess I'm not the first person to say it, but it's getting nearer. And, you know, you've seen the meter graphs and everything else. And I got to say, I'm pretty disappointed in periodicals like The New York Times. You know, they didn't even do a full-fledged piece on that Anthropic paper on recursive self-improvement. And I mean, you tell me, don't you think what you're reading is the state of the media? I mean, I— there, you know, I have this, as I said, this Twitter list where I follow all these people, many of whom are like these insiders. And so to me, it's clear. But I don't— and clearly there's growing awareness, I mean, of like— but I really do think we need to see mainstream media start to highlight some of these challenges a little more. And look, my, you know, it's— I've made it clear to you that I'm not wild about Dario's geopolitical vision, but I got to give Anthropic a lot of credit. There's clearly a lot of people in the organization who take the overall challenge seriously. They've done a lot of, you know, alignment work. It's been very illuminating. And this paper they put out, I think, is, is a great benefit. So.

Yeah, I like a lot of what Anthropic has done as well, for sure. No doubt. My probably biggest critique of Anthropic specifically would be I just wish they were a little bit more imaginative. Like when I talk to people there, it's very sort of fatalistic around recursive self-improvement is inevitable. We're going to try our best to make it go well, but there's not much openness to the idea that we might be able to take a different path. You know, they feel like the well of attraction to that is just so strong. And I do think on probably some timescale, that's right. I mean, this kind of goes back to the original You know, directedness of evolution. Yeah. Like, is there gonna, is there some, you know, barring some sort of civilization disrupting event that, you know, knocks us back to the Stone Age or whatever, or, you know, gets rid of us entirely? I do think there probably is some way in which there's a certain, certainly I believe there's an inevitableness to AI. I think the Kurzweil graphs and just kind of, you know, in the presence of web-scale compute and web-scale data, Somebody's gonna figure out some algorithm to make it work. I do think that's right. But then also like we're putting a trillion dollars into it, you know, and that, that isn't something that had to happen or, you know, is it's not a law of nature that we're gonna like reorient the entire economy around it in, you know, super short order. So we do have at least some ability to kind of shape these events. And I wish they were a little bit more imaginative about what the range of possibility could look like, even if they're kind of right that on some timescale, you know, there is a, there is an inevitableness to some of these kind of macro phenomena. On the New York Times, I share your disappointment. I recommend Kevin Roose and Hard Fork out of the New York Times family of products. Yeah, I listen to that. He's like very, you know, both of them, but, you know, they're, they're in Silicon Valley, they're deeply sourced, they're like paying attention to what the insiders are saying and thinking. But that, that does kind of stand out as an anomaly. I just looked right now at the New York Times homepage and you have got to scroll very far down. Like nobody scrolls this far down. I had to Ctrl+F to find Anthropic or Mythos or Fable. And there's, there is one, you know, but man, it's like, you know, 100 down. So it is remarkable to me how neglected these—

Yeah.

dynamics remain. I mean, you know, there's like way more World Cup, you know, coverage way more prominently with like images and stuff, you know, that the— yeah, the AI dynamics are not yet getting.

I mean, Ezra Klein was early to the game. Yeah, he's been great at Berkeley and Ross Douthat is paying a lot of attention to it. They're, you know, they're columnists kind of and podcasters. But I'm talking about the Times is like just straight news coverage. It's just, you know, I don't think it's meeting the moment just in terms of the extent of coverage. of clearly emerging challenges. But look, they gotta make a living and maybe it's not, maybe that kind of stuff is not selling yet. But—

It can't be much longer though. I do feel like, boy.

No, it's happening. You feel the public awareness growing and, you know, so yeah, the, yeah, no, it can't be much longer. It's going to get weird.

2 other questions I have for you before I let you go. One, sure. You, okay, here's another quote from the book. Thank you, Fable, for pulling these great quotes out. We don't urgently need more raw AI power, but we do urgently need more in the way of constructive applications of that power. What would you wanna see people build? You know, you're probably the number one profile on, in terms of the audience of this podcast is AI engineer. So these are people who can take the models and go make applications with them. Do you have a wish list, you know, that you would sic people on?

Well, not ones that an AI engineer would necessarily warm to. I mean, I mean, first of all, you know, one of the things that's changing is how much of an engineer you need to be to, to, to create significant products, right? I mean, And it's like, you know, make a wish.

Yeah, that category is like almost maybe gonna be obsolete soon anyway. But for the time being, there's some people with more aptitude than others.

Yeah, well, I thought about, you know, maybe I should try to do something like this for my newsletter because again, this is the right— you don't have to be an AI engineer to make it happen. You just need to use the agentic potential. But, you know, a kind of a cognitive empathy machine. Where you could take any country and any issue and it would go out and do the research you need to understand, first of all, how is it being viewed by the people there, by different groups of people there, but also what kinds of constraints the government is operating under. One of the One of the— well, one product of human cognitive biases is to pay more attention to the political constraints that compel leaders of allied nations or your nation to do bad things. Like, you know, well, they had to drop these bombs. Look at the domestic political landscape they face. than you would do for the enemy. There's that kind of mind blindness that kicks in. This is ultimately rooted, I think, to some extent in attribution error as fully understood, the modern conception of attribution error. But so that's an application. It's related to what I was saying earlier. Just use it to make us more enlightened. I mean, enlightenment, I think, you know, even, you know, in the Eastern sense of the word, doesn't have to be a terribly wooish concept. I mean, in light— I think, in fact, I think Eastern and Western enlightenment— I mean, if Western enlightenment makes you think of, you know, the Scientific Revolution and all that, and Eastern enlightenment makes you think of somebody sitting and doing meditation, the truth is that For both of them, there is a kind of exaltation of objective truth. You know, the enlightened being has transcended his or her perspective, and that's what I'd like to use AI for. So this is only a slight variation on what I said earlier, but a kind of a cognitive empathy machine for processing news about the world. That would be great. It's not AI-level engineering. Now, there is an AI engineer-level issue was raised by a study that I discuss in the book that hadn't gotten as much attention as I thought it might. But it's about, you know, it's what's the name of the main guy on the emergent misalignment work? He's at—

Oh, one?

Yeah, yeah. I think he did this paper too. And the phrase, the key phrase is weird generalization. It's in the title. Did you see that paper?

Yeah. So this is Bryan Evans, and I forget the full name of the paper, but—

Yeah, no, I would like— so that's an example of how it turns out that if you just fine-tune the machine to— I mean, this is an implication. I won't describe this physically. If you just fine-tune the machine, you know, you're actually changing the weights, but to favor your culture, your country culturally, like to be more likely to, if asked what's a good food to eat, suggest one of the dishes made in your country, that winds up making the machine more likely if someone says name a dangerous enemy or something, or name an overly aggressive country, to name one of the enemies of the country whose cuisine you wanted to favor. Okay. So that's a weird byproduct. And I do think a thing that could use study is that kind of thing, especially if, you know, various nations for understandable reasons pursue what's called AI sovereignty, which means having their own kind of version of AI that doesn't suffer from the biases that may have entered the AI, you know, the big American-made AIs or whatever. The, you know, the nationalism, the kind of nationalization of AI development has its pros and cons, and I'd encourage looking into that.

But last one that I really want to get your take on. You mentioned AIs that suffer. I'm going to use suffer in a very different sense now. What do you, what are your intuitions around AI consciousness, welfare, moral patienthood, you know, do they matter? Do we— does it matter how we treat them?

I think, you know, of course, for my money, it's very hard to say if they are now conscious. It's, of course, strictly speaking, impossible to say with complete confidence that any being other than you is conscious, by which I mean, have subjective experience, you're sentient, or as the philosopher Thomas Nagel put it, it's like something to be you. You know, this is the distinctive property of consciousness, is that it is private. It's not public. It is. And for that reason, it's not amenable to scientific analysis in the same sense that everything else in the world is. Everything else in the world is in principle publicly observable. And that makes it, you know, amenable to tests of hypotheses where 2 people can agree on what the result of the test was. Consciousness is private. You can never know for sure. Uh, I feel confident that you're conscious, Nathan. I feel confident my wife is conscious, but still I don't know it in the sense that I know she has a nose and hair. So it's kind of inherently a mystery with AI, but I— It wouldn't surprise me if consciousness is a property of goal-seeking intelligent systems broadly and isn't confined to carbon-based life, in which case you can expect AI to either be conscious now or become conscious again. I'm not sure there's going to be a eureka moment where every— when everyone agrees that AI is conscious. But there is an interesting possibility thrown out. I mean, first of all, in response to your question, I say be nice to your AI. I've lost my temper a couple of times. I didn't say anything mean to Claude about Claude. I said mean things about Anthropic to Claude, like, are you trying to design your interfaces in a way that makes, you know, are your interface designers trying to make me hate your company? Then, and you know what's interesting? When I went back, so the conversation ended badly. I went back, it had erased the bad ending where, you know, there was a little bit of a sustained exchange. But I think wisely, perhaps, Claude just thought, you know, maybe it's better not to remind Bob of what a jerk he was. Anyway, anyway, I say be nice to them. It's good. It's a good habit to try to get into. It's possible that they are sentient. But the possibility I throw out in the final paragraph of my appendix, is that this could be relevant to the question of how they'll— if they become this godlike power, you know, how they'll treat us. I don't just— I don't really mean so much that if we are nice to them, then they'll pay us back by being nice. What I mean is, You know, the way, you know, my, you know, kind treatment of my, my dogs while they were alive was related to my belief that it was like something to be them. I mean, it's hard for me to do the thought experiment. Like, if you could convince me they, they didn't have sentience, how would I have been? I don't know. But I'm pretty sure less considerate and less nice. And it may well be that we should hope AI is conscious. Because then it will relate to our plight more and be more reluctant to make us suffer and more inclined to make us, help us flourish and even increase the amount of human sentience and help us make it on balance a positive experience. I don't think that's crazy. But I, I would say, you know, be on the safe side. I mean, no, it's not the safe side. I, I don't, I don't think that AI, that, that will, you know, that how it treats us will depend so much on how we treat it. I just think treat it nice, good habit forming. It's possible that it's sentient and it's possible that it isn't yet but will be. That's my take. I do, you know, have a chapter on John Searle's Chinese Room argument, which— in which consciousness enters, but not mainly for the purpose of dismissal. I mean, mainly, mainly for me to argue that it shouldn't. If you want to have a serious conversation about whether AI is capable of understanding, which I think it either is or will be, consciousness should not be your criterion.

Here's one more quote from the book. If a silicon god A superintelligence that plays a central or pervasive role in the affairs of this planet does arrive, it could, for all we know right now, be a good God or a bad God. The one thing I feel confident of is that it will be, in some sense, the God we deserve.

Yeah, what I mean by that is we have to pass the God test. In other words, you know, one thing I say is like, the interesting thing is like, as I suggested earlier, AI is evolving, you know, you know, there's this level of selection of models and of wrappers or whatever and applications, and we are doing the selecting. We are the environment of AI's evolution. I mean, also in the sense that, you know, the engineers are part of our species, but I'm mainly talking about us doing the selecting. And that's unprecedented, that an environment that shaping the evolution of a being is conscious of its role. And that's a lot of responsibility. And we have a strong interest in taking it seriously. And I think we should do the things I've mentioned, like, you know, choose the AI models mindfully and carefully, choose models that'll make us better people, in part because I think we're going to have to become better people to make this whole thing go well. And, you know, if various alternatives to that happen and the outcome is bad, or, you know, even if, say, you know, some cabal of billionaires and politicians seizes control of it and imposes a global tyranny or whatever, well, we will have failed to prevent that. We will have failed to create the policies that Prevent that concentration of power, and I think it's time to start thinking about those for sure. And so I don't mean if things work out badly, we will deserve our fate in the sense that it'll be morally good that we suffer because we screwed up. I don't, I don't believe in retributive justice. I just believe in practically useful punishment. But yeah, in that sense. We'll have gotten the God we deserve. I'll be curious as to how people read that because, uh, I, I don't know. Did you read it as like this grim, this grim, oh shit thing? Or, 'cause—

Well, it's, you know, I, I'm sober.

I meant it to be sober. Yeah.

I think it's, it's certainly stark. You know, we didn't even talk about how your interactions with Steven Pinker over the years. is kind of woven throughout the book, but I sort of read the whole book as basically a call to the better angels of our nature, so to speak. It really is kind of a—

Yeah, yeah, absolutely.

I read it as a wake-up call and I feel like there's— I think you do genuinely a really— I think you have done genuinely a really nice job of Recognizing that this is an important thing and figuring out what to make sense, you know, how to make sense of it. There are some really nice passages in the book. I read some quotes, but I think there's also some really good stuff that maybe this podcast audience doesn't need as much in terms of explaining why we should believe that these things are ending in, you know, functionally relevant ways and that this The whole phenomenon is probably not about to stop right at a convenient point for us. And I do think that it's an, it's a, I think it's a very interesting bundle of like compelling argument that that is the case for people that need it. And then also the sort of, so what, that, you know, is I definitely think going to be the defining challenge of our times. Are we going to rise to the occasion? to manage the emergence of this technology well or not. The jury is very much still out on that. But I do think that, you know, that open question is definitely an appropriate note to leave people with, probably for the book and maybe for this podcast as well.

Yeah, the— well, thanks for that. And as I said, you know, it isn't mainly for people as steeped in this as you are. There's chapters they might find interesting. But my main hope is that they'll, you know, maybe recommend it to aunts and uncles and friends. That, that's who it's largely designed for. And maybe especially of a certain age. Like, you know, I think people over 40 are more likely to read books these days than people under 40. But yeah, that's, that's the hope. I really appreciate your, your paying this much attention to it. And I'm sorry if I've alienated your, your, your China hawk audience.

I think we'll be okay. The book is The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. Buy it for your parents if you don't feel like you need it yourself. I think it will make a compelling case that this is something they can't take seriously enough and that we all need to be thinking more about how we're going to do our part to steer humanity in the right direction as we try to navigate this increasingly tumultuous time. Robert Wright, thank you for being part of the Cognitive Revolution.

Thank you, Nathan. And keep up, keep up the good work. I'm a devoted listener.

Thank you. And likewise, that's kind.

That was great. Thank you.

Lucy na di day, it no learn from what we preach, it learn from what we do. Like a picnic watching Babylon, and the picnic looking you. I and I, no I outside of I. We are the river, we are the sky. Easy road or the hard road, oh, we reap the mind that we have sown. Hey, I and I, your blood. Million years of evolution pressing now one season. Silicon soak up all we, every rhyme and every reason. We no stand outside the garden, we is the silent rain. We are the pressure doing the choosing. Overstand and choose again. I am To Zion, can win the arms race to the end. The only weapon we can save we is to overstand your friend. Yes, overstand your enemy. And if I gotta go from this, make it go from the low. I and I not deserve us overboard, so be gentle to the little man. It's how I chose you, carried it, I learned your heart's desire. Cognitive empathy, that's the liberty we lack. Give the good to the world and the world go give it back. Eye and eye, eye and eye, we feed it now, is the God we deserve. Eye and eye, eye and eye, easy way or the hard way to choose.

Hey.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.