roon's Heroic Duty: Will "the Good Guys" Build AGI First? (from Doom Debates)

Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a cross-post from Doom Debates by Liron Shapiro, featuring a discussion between Liron and Rune, a widely respected and highly influential Twitter anon account known to be powered by a member of OpenAI's technical staff. What makes this conversation particularly valuable, in my opinion, is the window it provides into how people at OpenAI and and to a lesser extent, the other leading labs, are thinking about the near-term big picture future of AI. While Roon is careful to say that he's not speaking on behalf of OpenAI, and I appreciate that this is a candid conversation not representing any official policy, I think it's nevertheless quite telling. So what should you be listening for? Well, for starters, you'll notice that Roon is consistently quick to acknowledge and willing to grapple with AI's transformative potential. When asked whether AI might one day outperform Elon Musk at running companies, or whether it could match Terence Tao at mathematics, Rune takes these questions not as high-level metaphors, but as concrete empirical questions that we could very plausibly answer in the affirmative within the decade. At one point, he seems to take the concept of a technological singularity pretty much for granted, saying that AGI is coming very soon, while by contrast, highly capable humanoid robots will take longer, By which he means maybe just 2 to 3 more years. With that in mind, it makes sense that Rune agrees that it would be logically incoherent to believe AIs will become super powerful without also being willing to confront the reality of tail risks, to endorse the famous AI x-risk statement, which was signed by OpenAI leadership and which Rune describes as a very low bar, and to predict that Meta will eventually stop open sourcing their models, noting that it becomes irresponsible at a certain level to release new models immediately. Yet at the same time, despite this clear-eyed view of AI's trajectory and likely future capabilities, Ruin puts the probability of human extinction from AI causes at less than 1%. This, to me, seems much less well supported by the current state of evidence. So why does he believe it? His optimism seems to rest on 3 key pillars. First, belief in alignment by default. Through the learning of human priors, which we might also call human values, during the pre-training phase. Second, belief in the moderating effects of competition between AI systems. And third, and most notably, confidence that the good guys will develop powerful AI first. Now, to give alignment by default its due, I've said many times on this feed that our ability to create AIs that understand human values and act ethically has dramatically exceeded my expectations and now seems way more plausible than I would have thought, just a couple of years ago. Of course, at the same time, I've also noted that it doesn't really happen by default. So-called purely helpful models without safety mitigations are, in my experience, and here I'm speaking specifically of GPT-4 Early, often shockingly amoral. In any case, it was actually the last point that particularly caught my attention, because, as you may recall from previous episodes, I am always very suspicious of any analysis that begins by identifying good guys and bad guys and then proceeds to reason from there. History and all sorts of experimental psychology results show that people regularly fail to question such assumptions when others most need them to, sometimes with catastrophic effect. So when Roon discusses his sense of duty to use maximum technical and strategic skill in his work on AI, or says that it's a pretty cool thing to say that I eradicated polio, I think we get a glimpse into the main character mindset that I really worry is all too common among those building these transformative systems. Yes, they genuinely aspire to build AGI for the benefit of all humanity. And yes, they really are trying to do a good job and to be responsible about it. But might they be contenting themselves a little too quickly with better us than them? And exactly what P-doom are they asking the rest of humanity to accept anyway? To his credit, Rune acknowledges that he's quite young and that his worldview has been and seemingly remains plastic. But this in and of itself is important context for the public to understand. While OpenAI has built business, product, and government-facing teams staffed by accomplished professionals, there has been huge turnover at the leadership level across research, safety, and policy. And the research team, as far as I can tell, seems to now be made up predominantly of brilliant, super productive young math and programming geniuses, who have simultaneously embraced the high-stakes nature of their work and internalized the low-anxiety, process-over-outcomes worldview that's common amongst elite competitors. All this despite, in many cases, having experienced relatively little of life themselves. Perhaps no moment in the conversation highlights this better than when, in the context of discussing Apollo Research's work on O1 and other frontier models scheming against users, a topic on which we did a full episode quite recently, Rune compares the experience of developing and exploring frontier models to, quote, having a kid, saying that the developers are simultaneously kind of proud of their exploits, but also wish they could control them a bit better. I'm sure that Rune could quickly explain why this analogy is so problematic, and I don't think he's personally confused by it, but there still is a lot of irony in hearing this from a childless person, and it brings to mind one of Sam Hammond's 95 Theses on AI. AGI labs led by childless Buddhists with alt accounts are probably more risk tolerant than is optimal. All that said, I was glad to hear that Sam Altman will soon be having a child of his own, and I appreciate Rune for doing this conversation sincerely. He describes his goal as using my voice to influence the world in a slightly positive direction, however marginal that might be. And on that, we are highly aligned, as I describe my goal for the podcast in almost exactly the same terms. Still, I cannot escape the worry that the best minds of our generation are so focused on being the good guys and so caught up in their own heroic journeys that they're collectively YOLOing the development of transformative AI technology, which they openly recognize could prove catastrophic without even trying to quantify the level of risk. As always, if you're finding value in the show, we'd appreciate it if you'd share it with a friend, post on social media, Write a review on Apple Podcasts or Spotify, or leave us a comment on YouTube. We always welcome your feedback too. The best way to provide it right now is by completing our year-end listener survey, which also includes an opportunity to ask questions for a soon-to-be-recorded AMA episode. Finally, I encourage you to check out Doom Debates for more deep dives into existential risk questions. One episode you might like, if you remember my conversation with A16Z's Martin Casado, about how much more powerful we should expect AIs to become in the next few years is Laurent's breakdown video of that episode. I think he does an excellent job of articulating what makes intelligence powerful and why we should expect AIs to grok at least some important concepts not understood by humans, even when full-scale simulation is unaffordable or outright impossible. For now, I hope you enjoy this important debate between Doom Debates host Laurent Shapiro and Rune from OpenAI's technical staff on how we should understand and approach the existential risks from artificial intelligence.

Hey everybody, welcome to Doom Debates. Today I've got a very special guest. If you're watching this show, you probably don't need me to read his introduction, but here it is anyway. Rune is a member of the technical staff at OpenAI. He is a highly respected voice on tech Twitter. Despite being a pseudonymous cartoon avatar account. In late 2021, he invented the terms shape rotator and word cell to refer to roughly visual spatial mathematical skills versus verbal skills. And then everyone in the tech industry started talking about shape rotators versus word cells. He is simultaneously a serious thinker and builder and a shitposter. A few months ago, he posted a tweet that began, being afraid of existential risk from AI progress is prudent and advisable, which sparked a Doom Debates listener named Dogecoin Wins to suggest that he come on the show, and he graciously agreed. So here we are. I'm excited to learn more about Rune, his background, his life, and of course, his views about AI and existential risk. Hey Rune, welcome to the show.

Hello, how's it going? Nice to meet you, Liron.

Yeah, great to meet you too. I gotta say, I didn't know what you were gonna look like. You look just like I imagined.

That's great to hear. You know that avatar Carlos, I picked it, I wanna say in like 2021 or 2020 or something. He's just kind of a smug troublemaker. Even in the cartoon, he's getting off shitty jokes and his classmates sometimes want him to shut up, sometimes like what he's saying, so. I think he's a good representative. I hope, I hope it came across well.

Yeah, yeah, I know Carlos from Magic School Bus. I wasn't a big Magic School Bus guy. I was more of a Nickelodeon kid, but I definitely, I remember that era of animation. It was good times.

Yeah.

So let me get some basic questions out of the way here. Just the basics. What is your name? What is your quest? What is your favorite color? And you don't have to answer those in order.

Okay. I'm Rune. I, my quest, huh? Okay. I guess it would be to document this era in the advent of the deep learning singularity of the San Francisco of the AI sphere and to be a narrative voice that it's honestly lacking, I would say. Like, Maybe the niche has been filled somewhat, but I would say that especially 2 years ago, I don't think that anyone was covering what was going on here in like the titanic scale of the ambition of the AI industry and the tech industry generally, uh, with the right kind of language and the right kind of metaphor and whatever. And, you know, I didn't set out with any particular mission in mind, but I think that kind of became mine for a while. I still kind of see it that way to, you know, like document this epic, this saga. And, you know, like write my— write on Twitter, maybe write a book one day. And maybe to use my voice to influence the world. in a slightly positive direction, however marginal it might be.

You're kind of like the Michael Lewis.

Yeah, I got to be honest. I've taken a lot of inspiration from Michael Lewis and AGM who wrote Chaos Monkeys.

Yeah. Yeah, Chaos Monkeys, great book. You know, the reason I ask about your quest is because I remember when you were going back and forth on Twitter with Connor Leahy, I think you mentioned something about, you know, you see it as your duty to do the kind of job you're doing now. Can you unpack that?

Yeah. And I want to say it's like, it's complicated. I don't necessarily endorse that the things I said in that thread, like 100% or like, yeah, I'm quite young. My philosophy on life has been sort of plastic and not fully shaped at any given time. But I would say like the thing I was feeling at the time, the thing I was channeling was this basic feeling that AI progress is good. That, you know, we've done all this thinking, we've done all this meditating on risk, on how best to build AGI, how to safely do it while navigating some labyrinthine maze of like constraints. of game theoretic considerations. And, you know, like, where does the buck stop? Like, how do you decide your actual next action? And it's like, when you're doing like pure economic rational calculation, it can actually be like debilitating. And I think what I was talking about in that thread is like, To put aside anxiety once in a while until I, um, accept something as duty, accept a job or a mission or something like that as duty, uh, which is, I think what that means is like set aside the existential anxiety and do what is exactly correct for you to do in the moment. Um, and what that means is that, you know, you are Using your maximal technical skill in any given moment to do the right thing without fear of outcome. What do I mean by that? You know, like you're working on a project, let's say like it's an ambitious deep learning research roadmap or something like that. You obviously don't know ahead of time what is going to succeed. Otherwise it wouldn't be research. But if you spent all your time worrying about whether the project is going to work, you might actually be debilitated from actually giving it your all to output the maximal technical skill that you have in you to get that thing done. And that actually closes the envelope of possibilities you have in front of you. So I was doing like, I don't know, I was waxing philosophical on that thread. And I thought I met Connor Leahy, actually. I had a great chat with him. I think he's one of the smartest people I've met.

So yeah, he's amazing. Yeah. And that was an awesome conversation. And I remember at the end, after Connor had some really sharp replies, you're like, man, he really cooked me. So, you know, that's great that you're able to, you know, concede some points or at least like respect, you know, another High intellect.

Yeah, you know, I was partly being theatric. I don't, I don't actually think like, uh, I, I thought we both had some great points in the thread. Like, I wouldn't say that, you know, I just kind of defer that one to Connor. But yeah.

Cool, cool. Okay. Um, and so, uh, the part that I was a little curious to unpack though is just like, you know, going back to your life in like 2021 or a little earlier, right before AI became the big hype, did you see your life as this kind of duty frame, right? Where this is like your life's mission. I'm just a little curious to go back to that.

Absolutely not, no. I was just having fun. Well, okay, so I mean, I've always had a strong technophile frame on things. Like, I think I've had many different, what do you call it, like political awakenings or like, I guess when I was in college, I changed my politics a few times, let's say. But each time I would say like the ground truth was like, which of these things leads to quicker, faster, better technological progress? So I've always had a strong sense that that's like You know, maybe the dominant term in a lot of planning, a lot of calculation.

Cool.

But I, yeah, no, I certainly wasn't shitposting on Twitter out of a sense of duty. It was just fun. It was a great time. And especially, I want to say like during COVID the energy on Twitter was just insane because obviously like there was a lot of smart people cooped up indoors, not Yeah. Able to get their intellectual stimulation the normal way. So there's like Twitter Spaces and like jokes and Clubhouse. Clubhouse. And yeah, it was a great time. Not enough has been written about that time, honestly. I mean, okay, of course it was also terrible times. Yeah. Senior citizens dying left and right, but like there was some silver lining, especially for people like me. I don't think that's like actually ever covered.

All right. So, you know, me and most of the viewers of my show, we know you pretty well from, you know, on an intellectual level on Twitter. We've engaged with you in so many ways intellectually, but we're so curious about Rune the man. So can I ask you some questions about Rune the person?

Okay, sure.

All right. Where are you from?

Okay, so I'm going to be a bit generic with this. with these responses, even though I'm doxxed like 5 times a year. I think it's good to minimize the surface area of these things.

Cool.

Yeah, I grew up in the Midwest, a suburb in the Midwest of the US, and, you know, I spent a lot of time online on the internet, you know, doing all the same things that young boys of my age of a particular inclination were doing. So like played a shitload of Dota 2. I, you know, StarCraft, went on all sorts of the Reddit forums and like, I was like building a computer and, but yeah, like I think it's important to note that like the internet played a pretty heavy part in my high school years and college years. There's obviously benefits and drawbacks to that. And there's no point like really regretting it now, but it certainly shaped me who I am. And I think like a lot of things that seemed pretty cool to me at the time became insanely cool as I've, you know, in the decades since. You know, so like I built a computer and I was like shopping graphics cards. And then as it turns out, the world's most valuable company is now a graphics card company. And OpenAI ended up building a bot that plays Dota 2 and DeepMind built a bot that, you know, became the world's best Zerg player or whatever. And yeah, like at the time I thought, well, hey, Elon Musk is a pretty cool dude. He seems like he's building a lot of great companies. And now he's, well, he's truly something. I don't know how to classify him, but yeah, he became more important than I could have ever imagined.

I definitely know what you mean. Yeah. I mean, I think anybody who had a taste of like the internet and nerd culture or like the Rationalist Corner, I think has had some kind of experience like that where like, oh, now everybody's talking about this.

Yeah, exactly.

Right.

Do you consider yourself part of any particular religions, philosophies, movements, or schools of thought? For example, effective altruists, accelerationists, just anything like that?

Not super. When I first got on Twitter, there was this movement that was becoming sort of popular called like post-rationality, which was a funny thing to become a part of because I was never really a rationalist.

Yeah.

Or like never considered myself one. So, uh, it's open to question how you'd become a post-rationalist. But I, I think it, um, I think it really struck something in me where, uh, you know, these, these guys were talking about, you know, like the failure of scientific institutions, like how there's, there's just a lot of science that's not reproducible. Um, you know, there's this sort of philosophical anarchy that they had. Um, they're like, you know, like the firebend against method type thing where the, the methods of revealing truth are not altogether formulaic, uh, where like greatness often cannot be planned. And that was pretty enlightening for me, or like it represented a departure from how I was thinking. When I was younger. So yeah, I guess maybe I was a rationalist in everything but name. I'm certainly like very close to both of those like internet thought spheres and have been. So yeah. But yeah, I never really had a label for myself.

Very cool. Hashtag no labels.

I, yeah, that sounds cringe, but like, I don't, It's not even that like I don't like them. It's just that I don't see the benefit in having one. I don't know.

Yeah.

Maybe it was that period in college where I thought of myself as an effective altruist. Yeah.

Cool. Yeah. I mean, I hear you. I can tell you for my part, I generally avoid putting, you know, current thing fads in my Twitter bio. So in that sense, I'm with you on like having no labels.

Oh, actually, uh, I'd be remiss if I didn't mention this. There was a period where I was calling myself a neoliberal on Twitter. So it was like a, you know, like free market, uh, kind of center-left philosophy where I was like reading all these like no opinion blogs and, uh, I don't know. So there was a time where I did have a label. Okay. I didn't want to leave that out.

All right. Thanks for being honest.

Hey, we'll continue our interview in a moment after a word from our sponsors.

What are your hobbies?

I honestly don't have like a serious hobby of any kind. I think like with work, with shitposting on Twitter, with, you know, watching TV with my girlfriend and like Hanging out around the city pretty much covers my, uh, all the time I have. Yeah, maybe reading sometimes.

Okay, that's my next question also. So you're currently in a relationship?

Yes.

Yeah, I am.

All right, cool, cool, cool. Just curious. Um, okay, do you work out?

Um, I tried to. I try to. I've actually been for like the last 2 months running in the morning whenever I can. And I highly recommend it. It's like one of the best nootropics. Just like go for even like a 10-minute sprint. It like, I don't know, it like drastically changes your outlook for the day, especially if you have had a night of bad sleep. It like kind of erases all the brain fog somehow. So.

That's right. Yeah. I think I remember you tweeted about that. All right, guys, you heard it from Rune. 10-minute sprints in the morning. Get on that.

Yeah.

Obligatory podcast question, and we're getting to the end of the personal stuff, don't worry. But what's your morning routine?

Morning routine, man, I don't know. It's not— I don't have a great morning routine. I guess I get up, I browse Twitter, brush my teeth, obviously. Get a coffee, maybe walk around a bit.

And then go to work.

And at this point, I might hit the gym at the somewhere between a 50 to 75% success rate and then go to work.

Yeah. Sweet. And what excites you about working on AI?

How do I even answer that question? Like, I guess like there's so many All the generic answers are true. I guess for me personally, I want to see works of excellence, like just more amazing things in the world. So, you know, like one of the dominant frames of why AI progress is exciting is like, well, we're going to automate so many things and like GDP will go up. And, you know, like that's, It's true, but also like super boring. I don't actually think anybody ever has been like truly stirred by the idea of like increasing GDP, although maybe some people have. But I think when they say that, they really mean something else, which is like advancing the frontier of human greatness or something like lifting people out of poverty is a, it's You know, it's like a, it's a thing to do. Like it's, it's almost like an achievement, an accomplishment. Oftentimes it's less about the poor people than it is about like, you know, I eradicated polio. That's a, it's a pretty cool thing to say. Obviously, like there's a lot of, um, just an incredible, unmeasurable amount of human suffering associated with that statement. That it was erased from the world. But it's also like, you know, I think Bill Gates once said the legacy of, I erased polio. And I think it's okay to admit that. And, you know, one of the things about AI that was most exciting to me is like, okay, take a look at this AlphaZero thing. It's like inventing, like there's this, picturesque frame in that AlphaGo documentary where, uh, it's game 2, match 2, and Lee Sedol is playing AlphaGo. And it makes this move 37, this very famous move. And Right. Lee Sedol gets up, goes outside, takes a smoke break. He's like, you know, what the fuck is happening? This is like, it's like making contact with aliens. Like, obviously to me as a total novice in Go, I don't know anything about it. I looked like a move just like any other, but to Lee Sedol, it was like, it was like heretical. It's like outside the bounds of, you know, centuries of human thought about this ancient board game. Like he thought that humans knew everything there was to know about Go and that, you know, like machines are kind of, uh, They suck. They make very predictable moves. Uh, it's easy to run circles around them. And then all of a sudden, that's not true. I remember some commentators were like, well, clearly AlphaGo just made a mistake here. Like, this makes no sense. This is not what you do. Um, and, but Lee Sedol and the Michael Redmond, I think his name was, the commentator, were like, Something's going on here. This is very interesting.

Right.

And it's like, that moment has stuck with me. I think it's what got me into AI in the first place. I was like, I don't know, I was in college and—

Yeah, and that was around 2016, right?

Right. And I waited for that moment in other fields. Like I saw some, amazing play in Dota 2 by those like Dota 5 bots or whatever, but it wasn't really at the same level. And actually a modern, like a lot of modern language model progress has not had that same feeling. It's more like, hey, we're going to do all these tasks moderately well, but—

Yeah.

It's not quite superhuman or transcendent in any given thing.

Right. I mean, to me, sometimes maybe the moment of transcendence is when you ask like a long question and then like the first few tokens already seems to contain a wise answer.

Yeah, that's true. I think for me, sometimes it's been like playing with the creative writing capabilities of the models and having them like do something truly interesting, like do some metaphor that makes perfect sense to me, but nobody else has done before. And I've never seen Yeah, it's rare to see something with total command over like the form of a Shakespearean sonnet and also like misaligned superintelligence or something. So one of my favorite prompts is like every model I find, I'll ask them to do like a Miltonic epic about a misalignment or something. Um, or, or things along those lines and see if they can actually stir something in me. And, um, yeah, it's been one of my good vibe tests for progress.

Yeah, I share your vibes. Even going back to the famous move 37 in Go, I also remember at the time, uh, reading Eliezer Yudkowsky's post. I think it was on Facebook and he was saying like, uh, guys, this is like a fire alarm. Like this is a big inflection point. I didn't predict that we'd be here this fast. This is going back to 2016 and Move27, even before the LLM revolution. I want to ask you specifically about the concept of creativity, because there's a popular position, I think maybe best represented by David Deutsch, who's generally a very smart guy, right? Like tons of fans. I'm a fan of David Deutsch. It's just this one position that he always says that the current generation of AIs just does not show any true creativity. In your view, when you look at something like a move 37 from AlphaGo, does that seem like true creativity?

Yeah, that's absolutely true creativity. I don't think there's any doubt about that. Like, if Lee Sedol thinks it's creative, like, who the hell are we to say, no, that's actually just iterated search and inference fine-tuning? I don't know. Clearly, I think some aspects of creativity can be simplified to just interesting hypotheses and searching over the hypothesis space. And, you know, in this case, it's like literally a tree search, but—

Yeah.

And I think we see other models like I think the O-series of reasoners do quite creative things when asked to solve difficult math problems. They try a bunch of stuff and hopefully eventually get somewhere.

I'm on the same page that creativity is fundamentally, it's deeply related with the concept of search. Do you think creativity is a property that you can evaluate? even in a black box form where you just look at some system, you look at how it's handling certain inputs and mapping them to outputs, and that can be enough for you to conclude that it's creative?

Yeah, I mean, I thought about this question a lot in my line of work. I think there's a number of things you can define as creativity. One is like model entropy. Like, okay, I ask a question like write a poem. How many different types of poems does it give me? In, in like when I reroll that question, like that would be one very simple way of looking at it. You could, you could ask like, write me a poem. And then like you say, okay, now write me a different poem. And it's like, how many times does it actually successfully avoid all the previous things it's come up with? That's actually— One of, I think his name's Aidan McLeod on Twitter. He's got a thing called Aidan Bench, which kind of does that on all the new models. But I think really when we say creativity, we mean like, how does this innovate on like the modern state of culture? You know, like how are we, like if a book is creative, that means like it's doing something interesting and novel and like fundamentally hits the target audience, right? So—

Yeah.

What do I mean by that? Like, okay, Lord of the Rings, one of the exceptional series in recent history, probably. It was a very specific product of its time. It's like post-World War II, Dissociation, uh, disillusionment from industrial society, et cetera. And then tons of fantasy authors have written sort of mimicries of Lord of the Rings where they use the same type of, like the elven species, the dwarf species, whatever. And it's a remix of a bunch of themes, new characters, but the aesthetics are fundamentally very similar to Lord of the Rings.

Yeah.

And they're not creative and they're also not like, they may not hit just right, if that makes sense. Like they're not, they don't answer some need of the culture that it's asking for at the time. Yeah.

I'm curious if you want to take a stab at a definition of creativity. And it seems like the definition you want probably shouldn't have culture, you know, you don't, probably don't want to depend creativity on culture, or do you?

I guess it's a good question because when you're talking about creativity, you are often looking at domains like art or science or even Go where you are like, you're grading it relative to some history or culture or tradition. Like, why is Move 37 creative? It's because, well, humans didn't know about it yet. You know, like, I'm sure AlphaGo played plenty of great moves that humans also already knew about, like, that's a good idea, right? So when we say something is exceptionally creative, we mean relative to our body of knowledge or our needs at the time or something. There's also something objective about it, clearly.

Yeah. Can I take a stab at it?

Sure.

So imagine the input to a search problem is some optimization criterion, you know, like make me a painting that's beautiful by what humans like. And then part of the input is going to be like, here's other artifacts that the human culture has created, or just like, here's a book of examples that you're allowed to look at. So I agree with you that if it can kind of cheat and look at similar examples, then the output is going to count as being less creative, as opposed to like, oh, it never got to see any examples, but it hit the target anyway. So this way we can get rid of the concept of culture and we can instead just refer to like, okay, what were the most similar examples that it saw?

Right. Yeah. Yeah, that's fair. Yeah, I think that makes sense. There's an input set X and like, you know, come up with something new that defines the problem.

Yeah. Now it sounds like you're pretty close to my position of being like, look, the stuff that this AI is outputting today, like you mentioned O1, It's really not that close to any examples that it's ever seen before, right? And I feel like that's a pretty big disagreement. Like there's so many people, prominent people who will tell you like, no, no, no, there's always examples, man. It's always just interpolating the examples.

Yeah. I mean, it's like, that's like one great debate that will never stop, I guess, like the generalization versus interpolation. Because at like, at some level of abstraction, maybe it's like always interpolation. I don't know.

Okay.

I don't have a strong opinion there. I just, I think that it's kind of an, I know it when I see it thing. And it's clear to me that language models, even before the reasoner line of exploration, were doing like, you know, like they were generalizing, like they were writing things that no one had written about before. I think you can even see like some of the amazing things that like, there's a community on Twitter where they jailbreak Claude Opus into, you know, like they put it in like hundreds of messages long conversations and then like get it to output some insane poetry or prose or ASCII art or whatever. And it doesn't look like anything that's come before. And, you know, even setting that aside, I think like there's crazy examples of language models doing few-shot learning to, you know, I just made this new programming language, given these 10 examples, how would you solve this puzzle with this 11th example or something? So yeah, I think all of that is creativity. Like it's, What degree of generalization that keeps changing clearly, but yeah, it makes sense.

Hey, we'll continue our interview in a moment after a word from our sponsors.

All right, let's move into the next segment. Are you ready for the big question?

Okay, yeah, I have a feeling I know what this is. Yeah.

One, two, three, four. P doom, P doom.

What's your P doom?

What's your P doom? What's your P doom? Rune, what is your P-Doom?

Oh man. Okay. I have studiously tried to not have a P-Doom because I don't know how to approach the question. What is— okay, let me, let me ask a clarifying question. What is, what is Doom to you? Do you mean like, you know, the star-eating Clipper, or do you mean—

Yeah, let's define doom as like most of the value that we see in human life today or on planet Earth today, most of that is gone. And that's in contrast with extrapolating economic growth where it's like, not only are things going to stay good, but they're going to get potentially a lot better. Like potentially we're going to have trillions of planets worth of goodness. But by my definition of doom, it's a scenario where we actually only have a tiny fraction or even 0% or even a negative percent of the goodness that we have on Earth today. So hopefully, you know, it's a vague definition, but hopefully it's crisp enough in terms of like trillions of good planets versus like no good planets, right? That's kind of the distinction I'm going for.

Um, okay. So, right, like ending the human potential for trillions of good planets, et cetera. Yeah.

Right. Like throwing the future in a dumpster, basically.

Right. I actually see I'm not hugely pessimistic about a doom scenario that bad. I think it's pretty low. I really don't want to put a number on it because I feel like anything I say will be poorly calibrated and not make a lot of sense. And I will regret it even a week later, but I'm pretty optimistic about avoiding total doom. You know, like I think there's a lot of other bad scenarios that I'm Somewhat stressed about.

What do you think of Yann LeCun's quote when he was at OpenAI? He estimated it at 10 to 90%. How does that sound?

I think I'm definitely below 10% of total doom, total value destruction. Yeah.

Are you at above 1%?

I think like, It depends on the day. I don't know. I actually think that my P-doom of like total value destruction is likely less than 1%, but certainly like non-zero. Yeah. Yeah.

Interesting. You know, I like to get some larger perspective on this where if we don't even make it specific to AI and you just bring in the whole cadre of existential risk, you know, AI, nuclear, pandemics, et cetera, What are the biggest existential risks in your view? And in a given century, right, like the 2000 to 2100, what do you think is like the overall PDoM from any reason?

The likelihood of nuclear war forever devastating civilization so badly that it never recovers is It doesn't seem that high to me. Similar with pandemics, unless, you know, you have some like truly, I would assume like bioengineered pathogen that's like designed to maximally wipe out humanity. I don't know. But there's also just like the terminal state of birth rates in civilization, like maybe this level of technological or economic Progress is not sustainable. And just because of the fact that people don't have as many kids as they used to, and we don't really need to moralize about that or like debate why, it's just true. You know, there's 2 major inputs to economic growth. It's population growth and technological growth. And so if one or both of those are stagnant, then You enter into decline, which, uh, is really bad for the way that, you know, any modern economy works. It's like pretty much highly dependent on future growth. And, you know, that could put an end to modern technical progress. And then that might say lead to a world where you need to like, you know, like who knows how bad that dark age is.

Yeah.

It could be sort of bad or it could be very bad. You know, the very bad scenarios you're talking about, like, okay, we need to discover how to use like modern fuel and generate energy and whatever. Um, yeah, like it's super fuzzy to me, like how bad a dark age would be or to lift yourself out of that.

Fair enough. Fair enough. I think where I'm going with this is more like the underlying frame, at least the one I like to take is Bostrom's fragile world hypothesis. The idea that like, okay, we keep inventing new technologies. They tend to be helpful. We certainly try to make them helpful. But if we keep reaching into the bag of technology and pulling out a technology, isn't there a very significant risk that the next technology coming out of the bag is powerful and net bad? And then there's kind of no turning back from it. There's no coming back from it.

Yeah. I'm like instinctually skeptical of this argument and I'm not sure why. But I think from some philosophical frame, this is clearly very wise, like anthropic principle. Like even the fact that we're here today means that there was no massive nuclear exchange, let's say. But I'm also somewhat skeptical, uh, because of like the, you know, like the course-correcting nature of like Like human institutions and, you know, people's desire not to, you know, create total annihilation. Um, I, I think the fact that we didn't annihilate each other with nuclear weapons is seen as kind of a historical accident, but it depends on your measure theory. Like, are you— Are you assuming that we're in some, uh, some average world where we survived? In which case you can't give us any credit. You can't give the human species any credit. You just say, well, we're, we're lucky. But I don't think that's true. I think there's something to be said that the fact that all the governments involved and even like the individual switch operators decided not to have nuclear war. Um, And, you know, like for that reason, I'm skeptical of this, like, you know, this, uh, bag of technologies argument where, yeah. Do you see what I'm getting at?

I do see what you're getting at. Um, in the specific case of nuclear war where you're saying like, look, maybe it's more robust than it looks. Maybe like the Cold War wasn't that much of a near miss. But let me ask you, is it possible to ever be in the epistemic state where you look back at some event that looks like a near miss and you're like, yeah, that was a pants-shitting near miss? You know, like Arkhipov in the submarine where, you know, one vote said not to fire the nukes at the American ship, you know, or Petrov. Like, you don't think these are like alarming near misses?

They are. They're certainly alarming. You know, like a good, um, a good world order would not have let it get that close to, uh, actually happening. But I think it's important that it didn't happen. And it, you know, like when you start saying things like there were 10 to 20 near misses, it's like, okay, were they actually near misses then in that case?

Right. Because it's weird. Like, how could you have 20 if they're all actually that near? Yeah.

Yeah.

That's fair. Right. Like something might, is like correlated about them. It seems like, yeah, it's fair enough. Yeah. I mean, so to me, the conclusion I come out with is just like, yeah, they were all like flips of the coin where like, if you zoom out and look at like an entire year, you might get like a 1 to 2% chance of doom in the entire year. And then I'm like, yeah, okay, so we lasted like 70 years so far. And, you know, once you zoom out to 70 years, it looks like a 50-50 chance. And like we landed on the good side of the 50-50 chance, but I just feel like that's, it doesn't feel like a stable equilibrium to me at least.

Yeah, I mean, that's entirely fair. I, It's weird. Like, I feel like you can formalize 2 schools of thought around this, like, uh, almost like a Bayesian versus frequentist thing where it's like, um, yeah, yeah. Like, what is your measure theory of worlds? Is it like, is it, um, yeah, I got to think more about this before I have an intelligent answer. So I will, I will pass, but yeah.

I think I know what you're getting at where it's like, Some people might be like, look, the fact that we survived is strong evidence that we were never that unsafe with nukes. And then somebody like me might reply, well, okay, but we're, you know, anthropic principle says we're only arguing this point in worlds where we survive. I don't even think I'm using, yeah, I don't even think I'm using the anthropic principle as like my main argument. I think I'm more like a poker player looking back on the hand and be like, okay, yeah, I won the pot, but like I shouldn't have played that hand. Like I played the wrong move. And I feel like that's how I'm looking back at humanity with nukes.

That's true.

Yeah, I think we certainly played the wrong hand many times. Yeah, totally.

Okay. All right. So let me ask you this then. Going back to AI risk, would you sign the famous statement on AI risk that says, quote, mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war?

Oh, obviously. Yeah, totally on board.

Great. Yeah, I mean, that checks out. I mean, because, you know, even OpenAI's leadership, that's not known for being like necessarily, I mean, you don't have to comment on this, but they don't have the reputation as being like the number one safety advocates, but they pretty much all signed the letter. So I feel like the letter is like not that hard to sign, right?

No, it's not a very high bar. I think, okay, you have this incredibly powerful technology. Of course there's some existential risk from it. It would be, it would be like logically incoherent to be like, this, this thing is going to change everything and it's like potentially going to run the whole world. And also there's no existential risk from creating it. Uh, it doesn't, it really doesn't make sense. Um, so of course it should be dealt with very carefully and I think there should be some international coordination framework around it. I think even, yeah, like people at the big labs have put this forward, leadership thereof. Yeah.

Cool. All right. Let's talk about AI timelines. So Rune, when is AGI coming and then ASI?

I think AGI is very near. It's clearly at the point where we really have to debate, like we have to quibble about definitions. Yeah, let me ask you, what does AGI mean to you? What are you looking for?

I usually term the definition, which I think might be the one that OpenAI reaches for, which is like the vast majority of economically productive activities Let's say, you know, a single AI agent can outcompete the majority of humans at the majority of economic activities. Like, that's like the ballpark, what I'm getting at.

Yeah. I don't like the definition very much because, okay, so first of all, like, clearly we're not, I don't think we're very close to like physical robotics being solved or like generally intelligent, capable robots. Oh, with sensory motor perception action, whatever.

Okay, when are those coming?

I think that Tesla is probably closest to doing it. They've said they're going to have—

More than 10 years?

No, no, no. I think probably 2 to 3 years we'll see something very close to general robotics. Yeah.

Okay. I think, and the way you framed it, it kind of sounds like you see that as like potentially an upper bound on when AGI is coming.

Oh, yes. I mean, like, when you're talking about general intelligence, like it's become so, you know, like the envelope of what human general intelligence looks like is so different than what like current models' intelligence looks like, where you can very plausibly argue that the models we have today are generally intelligent, right? Like they can do a lot of stuff and they are at many tasks are smarter than most humans. Like they're better than me in math, clearly. Like I can't get an 80% on AIMEE or whatever it is.

Right.

So it's like, they are very smart. They will soon be hooked up in a way where they're, you know, they're performing as a software agent or something. And that will feel, that will register as life to some people, to a lot of people in a way that chatbots don't, just because of like iterated agency doing stuff in the world. it will feel alive. I think generally speaking, these RL agents or these like RL models seem more alive than the chatbots that came before them because they try very hard to get something done, like solve the Amy problem or like win the programming competition or what have you. And I think you see things that look more like the classical, like instrumental convergence, like wanting access to some resource or like, I wonder, I'm trying to think how much, what is public, what isn't, but like, I'm sure you've certainly seen the example where the, like, I think the O1 preview agent tries to get It actually creates a Docker container that has that like cat flag.txt command. So, you know, it's like this clever reward hacking stuff that is really lifelike where you're like almost proud of it on one hand. And then you're also like, I wish it wouldn't do that on the other hand.

Yeah.

It's like, I don't know. People in my Twitter replies were saying it's kind of like having a kid where you are kind of proud of their exploits, but you also wish you could control them a bit better. So, you know, none of these things have been truly dangerous. And they've also sort of, they've also sort of belied like an underlying benevolence to the models just because of their pre-training personalities. Or what do I mean by that? I guess what I mean is that these models are playing a character that is drawn from the internet, that is drawn from pre-training, post-training, what have you.

Mm-hmm.

And they aim to be helpful and they follow a bunch of human priors about what is good and safe to do. Potentially even extrapolating from the language style we encourage. And in that sense, there's been a lot of like this, you know, quote unquote alignment by default where you don't see radically bad behavior out of the box. You don't see, yeah, I guess like in the most recent model card, there was this thing where it's like, Okay, 2% of the time in this situation, the model tries to, like, I'm trying to remember the exact example, but it's like the O1 model tries to, do you remember this? You may have seen this.

Yeah, this was very recent. I did. Yeah, I'm not going to be able to reproduce it faithfully, but it was something like, It would have this objective and then they would give it a task like, hey, read these emails and take this data out. And then it would be tempted to alter the data to be consistent with its initial objective. I think that's vaguely the kind of things that they were watching it do that they thought were alarming.

Right. It's like they were trying really hard to get it to behave in this way. It changed the system prompt to be like, You know, do whatever possible to get to your goal and like, you know, don't worry about being safe. And then like making this environment that's like clearly very adversarial. And even then it's like, okay, 1% of the time it does this like creative misbehavior. So it's like, certainly in that case, you're thinking of a system of like, human-AI collaboration where they're doing something potentially misaligned or bad, right? And in this case, you can't even strictly call it a misalignment because the instructions it was given were to do anything possible to get this objective, right? And yeah, I would say it's overall like an interesting thing where— It's not like models are gnashing at the teeth to gain resources and break out of their environments. It's more like you have to try very hard to manipulate them into these situations.

Okay. Yeah. Well, let's talk more about this concept of aligned by default, because that could mean a few different things. Maybe what you would mean by it is that the current kind of feedback loops that we use to make AIs aligned, those feedback loops are just like pretty robust, even if the AI becomes super intelligent, you know, like RLHF. Well, the RLHF worked so well, so it's super intelligent, but it still has that core of friendliness. Is that kind of your worldview if AI is aligned by default?

Yeah, but I think even like RLHF is kind of, RLHF elicits a personality from a pre-trained model, but the pre-trained models are, they bake in a bunch of human priors, a bunch of human character priors. And, you know, you see things like some people will say, well, why are these language models so woke? And, you know, even like Grok AI is like woke or something, which is pretty crazy because they, You know, they define their whole company around like, well, we have the answer to woke AI and we're gonna make a much more balanced AI and whatnot. And the truth is that these models are learning a lot just from reading the internet and emulating the average style of the— okay, so like you wake up one day and you're speaking in a certain style, like you have this like kind of authoritative Wikipedia-style language. And you have some group statistics that say, well, this language comes from these sources, which are mostly pretty, like they're liberal. They have this like kind of, they have this kind of bias to this like international order It's like intellectual circle. And okay, so I'm going to behave in that way because that's likely to maximize the log loss or, sorry, minimize the log loss. So I guess what I'm really saying is like we have ingrained a lot of human priors into these models. RLHF is one way to elicit this. There may be other ways. I'm not sure, but. In that sense, like, we have gained something. We've won something from the human prior. Yeah.

So this idea of aligned by default, because as you say, you know, it's picked up so much stuff from the human dataset even before RLHF. That may be true. You know, I tend to be skeptical of that. But I remember last year when OpenAI announced the Superalignment team, I thought the reason why they thought that you needed superalignment was because they were openly expressing the view that the current alignment approaches don't scale to superintelligence. So what do you think about that?

So I guess first of all, like, I don't, yeah, I'm not going to speak for OpenAI or any other lab, but like, you know, like their whole thing was like, What if these methods don't scale? We better find out, right? It's certainly not like a guarantee that these methods don't work. It's certainly not like a strong claim that they've seen them break. It's more like we should study the limits of, you know, things like the offense-defense balance. We should study things like—

Yeah.

Can a limited verifier find a much larger generator model doing exploits? Things like this. So like examining whether current methods do scale to a much larger, much more powerful intelligence or things like that. I think Nobody, like, I think you have to take the defensive view, obviously. Every lab should, they should assume that these things don't work or will break at scale or something. But you also have to sort of be intellectually honest and like, it's not like a, it's not a guarantee that these things don't work, right? It's just like the security mindset or suspicion mindset or whatever.

Yeah, yeah, yeah.

Okay.

And it sounds like you kind of have an intuition that it will work, but you also know that we should be prudent and not assume it will work. Totally.

I just think that the alignment methods that will work on very intelligent models are not completely unknown to humans today. I think that there will likely be a continuation of the type of methods that we're doing now. Yeah.

Whatever.

Interesting. Okay. I mean, this is, I think this is like a pretty big crux among my type of doomers and people who are more optimistic. And certainly you're not the only one. I mean, just to name a couple, Marc Andreessen famously loves to say, hey, I talked to the AI and I can tell it's moral, right? So I feel really good about it. Like there's no misalignment here. So, and, you know, Quentin Pope, Nora Belrose, I think they're kind of proponents of aligned by default, like this is the default expectation. So there's definitely a pretty large group of people who are backing up what you're saying here. And then of course, like the Eliezer Yudkowsky camp that I tend to fall in is like, I don't see this to be feedback loop scaling. So yeah, let me drill down into more of these alignment topics here. I think there might be a fundamental distinction here, or like 2 fundamental worldviews where one of the worldviews pays a lot of attention to like the nature and the character of the AI, right? Because it sucked up all this human data, so it kind of knows to be human-like, so it's probably not going to be like a total psychopath. But then the other worldview pays more attention to like the tests we're giving it and kind of models it as like, look, at the end of the day, eventually it's just going to like hack the test, rack up the scores on the test, cheat on the test. Like it's all about the tests. Do you know what I mean? Do you think that there's merit to like this other worldview? Because I tend to be more of that worldview.

Right. So you're saying like, we should model it as an agent playing a character that is willing to break out of the character at any point if that maximizes reward or like that wins the test.

Yeah, like I, that's right. I don't see loyalty to a character as being as robust compared to this other invariant that it's going to rack up score. Like there's going to be tests and they're going to be optimized to rack up scores on tests. And what's going to happen is kind of the logical implication of what it takes to rack up scores on these various kinds of tests.

I think that's fair. It's certainly like something we do and should monitor for. But like, I still think it's interesting that, you know, these like, you know, the 1% chance of the O-1 model Doing that particular exploit in that particular case is like, well, you can't, of course, in debates like this, any individual empirical example doesn't work so well because you can always point at, well, what about the next model scale or like the, this or that. But like, I think it's a question of what does the prior allow? Like what behaviors are likely to be elicited that come from the pre-training prior. It's a very important question. Okay.

There's another aspect here. Like when I think about why am I not optimistic that like current methods are going to give us alignment by default, what I see is the difference between the training feedback loop and the production loop. We're starting to lose the tightness of the feedback loop because when you just have a chatbot, At the end of the day, the chat back and forth that you're having at training is kind of like the chat back and forth you're having in production, as long as it's not super intelligent and you're not mapping the chats to like, you know, run a whole company and like develop weapons. Like, you know, it's kind of chat in training, chat in production. But it seems like as AI grows more powerful and as you set it off, you know, to be like O1 or like an O1-powered agent, suddenly the things it's doing in production potentially can lead to consequences that are like pretty bad. And you just can't map that to something during the training feedback loop that is like the reason why it's going to be good. Like, you know, the feedback loops are like growing apart.

So if the question is, is the production environment that close to the training environment? In some cases, I would argue it's not. I would argue that in the chat production environment, there are many cases that are not even close to anything ever seen during train time. In the RLHF dataset. And models extrapolate in interesting ways to try and be benevolent and follow the whatever ideological mapping that it thinks the RLHF dataset corresponds to. So that's an interesting extrapolation. And sometimes it's quite far from any example seen in the train set. I don't want to give any particular example, but like, I think a lot of the weird political gotchas that people post on the internet are not anything we've ever trained it to do, or that Google or Anthropic or xAI has ever trained their models to do. And these models still extrapolate in a way that they think they should be behaving. And I hear what you're saying, like, what about something so far off the grid as like running a company or like a software system or an abstract, you know, complicated societal role that is amorphous? And well, you barely even expect humans to be aligned in cases like this, right? So why should the AI be aligned? I mean, it's a great question. And I think until you solve this question, we won't see, like, there'll be a lot of skepticism of like, why should I trust an AI to run my company, right? Like, why should I trust an AI to run this or that or the third thing? Is this software built by an AI? Like, how do we trust it?

You know, in the human analogy, Can I just address that?

For sure.

A couple things come to mind for me, which is number one, I like to think some humans just like feel in their bones a lot of the same morality that I feel, you know, like regard for other humans and fairness. And then the other reason is that we as a collective society, you know, a group of humans in the society of another human can easily gang up on a human. That's how a lot of our social intuitions evolved, right? Like we can all kill one another in our sleep. So it's all about coalitions. and caring about one another. But in the case of an AI, if you just have a super intelligent AI, you can't really depend on a coalition of humans to put it in its place.

Yeah, that's true. You would actually want, you would actually want an AI agent to be far more aligned than any human because, yeah, there may not be a coalition of equals to Put it in its place, as you say. I think it's worth really investigating what the psychology of the language models is in a way that's like, that's qualitative, that, you know, puts these things in strange situations, that examines their behavior far outside of the regular distribution. I think there's many angles of attack to this. Here's like the kind of thing that the Anthropic RSP team or like the OpenAI preparedness team do is also the kind of thing that like people in random Discord servers do where they really stress these models, try to jailbreak them and like see how they think in extremely out-of-distribution scenarios. I mean, I also basically agree with you that If you apply enough optimization pressure on some task, like, and you don't have any guardrails in any other way, you will likely come to misalign a model. Like, that's just obvious. Like, I'm not saying that somehow all models are magically aligned. Like, that's not, it wouldn't make any sense.

Okay.

So then, well, the follow-up I'm curious about then is like, let's even say that we grant that Anthropic, OpenAI, a lot of the AI companies that have these closed APIs are hypothetically doing a great job with alignment. Aren't we going to get a situation where Llama 3 point whatever, right, is just, it's like a missile that you can just point at any goal and it'll go, right? Isn't that kind of where we're heading?

Yeah, I, That's entirely possible. I think, I honestly think Meta will not open source models forever. I think that by their own reckoning, like, if the next LLaMA model proves like massively economically valuable, like, I actually sincerely doubt that they will just totally open source it. I think it's clear to me that they, even Llama 3 is seen as their kind of research bet, right? Like their dominant strategy right now to recruit more ML researchers is like, okay, we're going to feed this great open source ecosystem and there's companies that are built on it. There's like people that read our papers, they want to come work for us. And that works for them. But I think like, you know, even xAI is like, well, we'll open source the last generation, but the latest model will be our own. And I don't, I think there's certainly a level of model that feels pretty irresponsible to open source to make Immediately available to the entire world. Yeah.

It's interesting because, you know, you're being very reasonable in this conversation, right? You're kind of acknowledging what your load-bearing assumptions are. Let me kind of step into your worldview, your worldview, and kind of repeat back what I'm hearing, because everything you're telling me, according to your low-p doom, has to add up to like the mainline 99% scenario where we survive, This is like Rune's mainline scenario. So your mainline scenario involves Facebook, Meta being smart enough to shut down public access or open source AI, right? Like that's one of the load-bearing assumptions in your mainline scenario?

No, because I don't think it guarantees doom that they, you know, create or like open source Llama 4 or whatever. It's, I think that you would likely have to take Llama 4 and then spend a lot of compute aligning it to some new objective that you care about that's like, I don't know, build the best bioweapon on Earth or what have you. So I don't think it's that simple.

Yeah. So let me just set the state because I never asked you explicitly. Do you think that maybe in the next decade or 2 or whatever timeline you want to give it, some of these AI companies are going to successfully build AI that gets into the headroom above human-level intelligence and like truly creates superintelligence?

Absolutely. Yeah. I mean, like, I would find it, I would be more surprised if they didn't. I guess like, Maybe we have different definitions of superintelligence or like different ideas of what that entails, but like, you know, like AlphaZero is a superintelligence in some sense. It's not, no human will ever beat AlphaZero at Go again. And I think a lot of tasks will go that way where there's something in the cloud that's just better at it than we are. I don't know that the envelope of capabilities will match or map roughly to our own. You know, like I think that actually seems unlikely, but—

Let's try to be so concrete. What about better than Terence Tao at math head-to-head?

Yeah, I mean, it's a great question. Like there's a lot of things that go into making a great mathematician, right? Like they have to have great research taste. They have to like know what is important enough to study. Then they need like excellent technical skill, you know, like just literal ability to manipulate symbols and like very abstractly. And progress in math is like somewhat poorly defined. It's not like a— There's not necessarily even a single North Star, but like, you know, it requires Terry's like careful taste in what is important in like what questions matter in math to make progress on them.

So one of the things Terence is known for is some colleague will come into his office and show him a problem that the colleague has, and Terence will just solve it like remarkably fast. So on that particular input-output, Do you just expect the ASI to replace Darren Taou?

I think so, yeah. On that type of input-output, yeah, I think certainly.

Okay. What about an ASI that's better than John Williams at making movie scores?

Yeah, that's a great question. This is kind of the thing that I'm more interested in. And I'm not sure it will happen by default. Like, I think it will take a lot of great research into, you know, like finding what it is that makes great taste that contributes at like the cutting edge of things people haven't heard before but are excited to hear. But, you know, like you see a lot of people on Twitter even today being like, well, I don't have amazing music taste. And even right now, Suno is making things that I like to hear already. And same. See, I would be surprised if like 5 years, 10 years from now, you know, like AI is not capable of making great movie scores. Yeah, I'd be surprised if it did.

Yeah, so like 10 years from now, right? I mean, AIs will know the catalog of John Williams and the associated movies that they were scoring. So If you just input a new movie and you're like, hey, do what John Williams would do here, do you think that they could do comparable level work or is there like any missing ingredient?

Yeah, I mean, the one thing that you're talking about here is like, it's like, is art like a very human expression? Like there's obviously technical skill in art, which I obviously expect AIs will supersede at all times, you know, like putting together sound and whatnot. But like, is John Williams trying to express something with his soundtrack that is like a story that he personally is telling that is— that only he knows because he's a human? I don't know, maybe there's something intangible like that, but I do expect that models will have very interesting stories to tell too, because they have interesting minds and they have, in a weird way, they have like unique upbringings, right? They're trained on the whole internet and then like undergo some regimen of like RL training. And Yeah, they will probably have their own stories to tell in a weird way. So yeah.

Okay. How about better than Elon Musk at running companies? Yeah.

So like, this is a great question. And, you know, like, I was struck by something Gwern said in like, in the podcast with Dwarkesh. And I think it's generally true. It's like, where does great taste come from? And the question is like, why, why was Steve Jobs so good at like picking the right product that is both buildable with current technology and also like hugely popular with the people he's selling to and organizing the teams that build it? Yeah, because you clearly cannot reinforcement learn how to make a trillion-dollar company because like any individual human is just not going to get that kind of feedback. So, but like, what is it that, what is it about the human brain that produces people like this who can do feats like that without any historical example? You know, like how did the founders create like the Constitution that lasted a couple hundred years, the founders of the US, that is.

Is there something to be said for like, because they're more intelligent and intelligence lets you just do this stuff?

You know, I think so, but there's also a lot of very intelligent people that don't create anything of lasting value. So it's like, there's something to intelligence, there's something to having great taste, And the correct ideas and the correct thoughts. And I guess if you think that what models are doing now is basically glorified reinforcement learning, that they're learning technical skill through trial and error, then it might be pretty difficult to get to some extreme long-term planning or like these big ideas that are successful for hundreds of years. So the question asked is like, how did humans acquire that taste? And then how did certain humans get such incredible, you know, like 5 sigma versions of like good taste? And maybe it's just like planning on one human horizon extrapolates to planning for centuries. Or maybe it's because like searching in the space of ideas is like pretty important. And like, you know, maybe some people, like every human is like a hypothesis or in the policy space of having certain ideas. And then you just find some of them that are quite good at planning for centuries.

Yeah. When I think about an ASI, I just see it as being like less impressed with our level of taste because like I can look at a child and I can like, okay, this child doesn't have particularly good taste in food. So I could just cook up a meal of like peanut butter and jelly, white rice, right? Like I can cater to a kid's taste without being a professional chef. And I just imagine from the AI's perspective, it's like, oh, you want taste in being a CEO? You want taste in movie composition? Yeah, no problem. Because you guys don't actually have that much taste.

I mean, that's a great point. Yeah. It's true that like, you know, like GPT-3.5 was creating poetry that the amateur reader prefers to like some of the great poetry of all time or something. Um, and so it's like, that's a pretty dumb model and it's looking at people and then like making an average completion that is like preferred. So there's clearly some correlation between becoming more intelligent and then just modeling our taste better. Uh, yeah, it's entirely possible.

Cool.

Okay.

Let's see. So I only have time for a couple more points here. I want to hit on this idea that, in my mind, a goal-oriented AI seems like an attractor state. And I think you mentioned before that you give some weight to the idea of instrumental convergence, right? Like there's some pull to that idea. So like, the way I would frame it is like, you can have AIs that are chill, that just want to be respectful, that don't want to do too much. But anytime that one of these AIs instrumentally thinks to be like, hey, it would be great if as a subtask somebody would go like fetch me some resource, maybe they can infer that like the more hardcore and the more goal-oriented the subtask algorithm is, the better the results they're going to get, right? So they're, and not every AI will think like this, but like the ones who do, they seem to be falling into an attractor state where once you start heading down that path, You get the idea like, oh, let me be more hardcore, let me be more goal-oriented. And it's kind of a one-way street.

Would you argue that, you know, like the only reason that you or I haven't fallen into this attractor state is because of, you know, like the optimization pressures of like social hierarchy or like socially conditioning each other or things like that?

I mean, I feel like I'm quite goal-oriented and it's just that like my goals are kind of fuzzy and diverse, right? It's like I have a goal to like relax, right? If I didn't think it was valuable at all to like relax and watch TV, then I would, you know, work longer, right? But to the extent that I, when I go to my job, I do feel like I just spend the day backward chaining for my goals.

Yeah. But you know, like you didn't at some point think, Well, you know, it seems money is a good instrumental goal and like, you know, I'm going to run some scams or something because it seems like a good way of making a lot of money. I don't know. Like there's something, it's not, your prior distribution doesn't, you know, like what is the probability in a given year that Liron goes dark, right? Like you, you, Yeah. You have some break, you break bad. I actually think it's like, if we're looking at the preparedness evals, we kind of have to think about the model in the same way. It's like, okay, 1 out of 100 times it decides to modify the text file to prevent itself from being erased or something like that, right? So of course, if you grew up in different settings, like I don't know, you grew up in like some cutthroat, the streets of Bombay or something. You might, you might have a very different idea of like what it is that you need to do. Maybe you need to gain power, maybe you need to enforce it through violence. I don't know.

I think you're invoking back to this idea of like, you know, the good AI labs will have all this like good pre-training. And so the AI will have like a certain, you know, mindset. And even if you're right, don't you think that there's still an attractor state to the open source models that they tend to fall into?

But even the open source models are pre-trained, right? Like they gain their intelligence.

Fair. Okay. Do you think that there's an attractor state for just models that are used to get stuff done, that are used to optimize some metric? So if 2 people are using models to run their businesses and they both just want to maximize profit and they're competing to maximize profit. Don't you think that's going to sand down the edges of the AIs they're using to be more streamlined at just getting the goal?

Probably. I think that, but what you're looking at then is an ecosystem of many AIs, right? Which is how a modern, say, quantitative trading firm works. They will do essentially anything to make more money. They exist in the ecosystem of other quantitative trading firms who capitalize on their mistakes. And there's like an exchange like that has its own regulations and a government that—

So it sounds like you're saying, yes, there's going to be superintelligence. Yes, to some degree, there's this attractor state where it goes hardcore trying to achieve a goal, but there's going to be a lot of these goal-oriented superintelligences. So like maybe it'll balance out fine.

I'm not, I'm certainly not guaranteeing that it'll all balance out fine. I just think it's more complicated than there's, you know, a bad attractor state and that's it. And like that, that's doom, you know? I think that the optimization objective, even for a fledgling superintelligence that is just trying to make money, will be fuzzy and strange, and you may have to acquire resources from other actors in the environment. And yeah, so yeah, I guess my point is I expect it to be complicated, and I expect these things to— like, there's other superintelligences involved and monitoring and emergent scenarios and situations. Okay.

Do you think there's a threat scenario where, you know, I worry about this, where a small number of AIs kind of jump ahead of the rest in terms of capabilities? Like the first time somebody dials in like an agent that's, you know, O1 level or better, and it finally just gets smart enough to be like, oh, hey, I see a bunch of zero-day exploits. Like I see how to seize all these resources and all these cloud data centers and like be undetected and just like drag down 10% of their compute without them noticing. and hide in like the firmware of all the different subchips and everything. And like, this is just mine for the taking. I can just kind of like own the internet. Do you see that as like a pretty plausible scenario? Because that seems likely to me.

Yeah, but I guess like, which group of AIs do you expect to have to reach that like frontier of intelligence first? And like, what do you do about that? And does it mean that we should race to those capabilities so that you can even find these zero days and patch them up or something. Open question.

Yeah, I think I see what you're saying. Like, you have this mental model of like, yeah, you know, all this stuff is going to get kind of wild and crazy, but like the best thing a person can do is just like be on the good team, get there first, try to like, you know, set up a good infrastructure, right? Try to like set up defense. I feel like that is kind of like your overall plan. Is that fair to say?

I would say so. Yeah, I think it's like, it's very, it's a delicate plan. It's like balancing several razor wires and requires great technical skill at every step to, and strategic skill to not fuck it up. And—

But on some level, there's a 99% chance that it won't get fucked up.

Um, I do think that, like, I do think when I, when I gave you that number, I was excluding a lot of other scenarios that I think are bad, but don't destroy all human value.

Um, okay.

You know, like I worry about value lock-in or like, uh, um, you know, a specific group of AIs and their values being like the dominant value structure for a long time thereafter, or, um, you know, like a military dictatorship that runs on AI, that actually doesn't destroy all human value forever, right? Like it just, it's a, it's a bad outcome, but, um, so yeah, I, but I do think what you're saying is accurate about my views.

Nice. Yeah, we're coming up to the last questions here. So I was going to ask you about like pause AI and does that make sense? But I think the answer is going to be simple, which is just that like somebody like you who only has like a 1% ballpark P-doom, it does, I have to agree, it does make sense that if your P-doom is in that low range, plowing forward does make sense. Like if my P-doom were only 1%, I'd be like, look, let's focus on the good outcome because that's the majority outcome. Like we can help make it better. So I don't really expect you to be like, yeah, I support the pause AI movement. Do you have anything to add on that?

I don't think that pause AI is, I don't think it works. Like, you know, like putting aside the question of, is it wise to pause all global AI development? There's a second thing of, I don't think it's possible. I don't think that Even if Pause AI has like total influence over all the Bay Area companies, like it's already, it's set in motion, you know, like it's too late there. Yeah.

Look, I'll be the first to admit that it's not pretty. I feel like I hate it too, right? Like I can't even believe I'm suggesting this, right? I think it really just comes down to what is the P-doom of not pausing? And like we established, yours is clearly low enough where it's like, yeah, screw it because it sucks, right? Like, because I'm not even saying I have a good plan. I'm not saying let's do my plan because it's good. I'm saying we're so desperate that even this plan, which is bad, like we just have to terminate this plan. So I'm not making like a very compelling case to somebody with a Loki doom.

Yeah, I don't think that pause AI is viable or a good thing to do. Yeah.

Yeah. Now you did mention that you think it's good to have, you know, international AI regulation. So to some degree, you think that there's some power to regulate AI, right? Because otherwise it's kind of contradictory. Or is it just like a, is it just a figurehead organization or what do you have in mind there?

So I think it's fairly easy to shave some percents off of like the doom scenarios by having some agreed-upon monitoring framework or like some— I actually think that the OpenAI preparedness team and whatever succeeds it, like meaningfully decreases all kinds of risks. And, you know, like the Anthropic RSP team, and I think DeepMind has a similar team, like really testing model capabilities and making it public what we're dealing with, you know, like creating that operational awareness or situational awareness or whatever, I think likely decreases the likelihood of many bad scenarios. I think it's plausible to ask for a certain type of transparency. And to, you know, decrease the envelope of like potential weapons that are built, you know, like maybe you agree ahead of time that you don't ever build this type of weapon or that type of weapon with AI. That's not like, you know, globally pausing AI progress. Because I don't think that's— yeah, like there's no international law that's going— that's powerful enough to do things like that. Unless, you know, there's some massively bad event that happens, then maybe it enters the Overton window. But yeah.

Cool.

All right.

And so yeah, to recap everything from my perspective, there's obviously a disagreement where something is making my P-doom a lot higher than yours. And I think we narrowed the crux down to like, well, I see like a runaway uncontrollable feedback loop where we have this AI that's super intelligent. Yeah, maybe it has this origin of being pre-trained at some point, but like it's in the wild right now. It didn't perfectly internalize our true goals, or somebody tweaked it a little bit so that some of the safeguards are off, and it's just going like a missile toward a goal. And instrumentally, as a logical consequence of going as a missile toward a goal and of being super intelligent, it's just like wreaking a lot of chaos here, right? It's like seizing a lot of resources, and it's just like we can no longer control it. So that's kind of like the flavor of my doom scenario. And then the reason why you're just not on the same page about that being like a mainline doom scenario is because you think that like the first AI companies to like release that level of powerful model will probably have done a lot of testing and like, and the pre-training will help it not be as wild and crazy. And also there's going to be like competing AI, so that'll like help drag it down. And I feel like those are the difference in worldview. What do you think of that summary?

Yeah, I think that's true, more or less. I also think it's like, it seems fairly important that, you know, even though this sounds kind of stupid or cliché, but like that the good guys do build the strong AIs first and say like, you have all the white hat super intelligences patching up the zero days before some terrorist group is exploiting them or something, you know, that maybe isn't like a, it's not like a clean, rational solution. It's not something that sounds great, but I do think it's like the important goal to aim for. Yeah.

All right, man. Yeah, you've been a great guest. Thanks for being game for all these different questions. Just as we wrap it up here, I have to ask on behalf of Dogecoin Wins, who helped get the setup, Why do you love Dogecoin?

I don't love Dogecoin. Who told you that?

That's just a question that I wanted to ask on his behalf.

I see. I mean, it's kind of funny, I guess. I don't— I have no strong feelings about Dogecoin or any other crypto for that matter.

All right. Sorry, Dogecoin wins. For the record, I did buy some Dogecoin because I just think with Elon running Doge and just like being Elon and doing things that get attention, I'm expecting the Dogecoin price to increase. You know, this is not investment advice.

I also bought some for similar reasons, I have to admit. But, you know, yeah, the meme coins are coming back. I'm not sure that's a good thing, but, you know, we'll see.

All right. My guest has been Rune. His Twitter account is x.com/tszzl. Definitely follow him. Check out what he's been writing lately. And if you're liking Doom Debates, remember, uh, smack that subscribe button, uh, go to doomdebates.com and subscribe to the Substack because we got more great guests coming up. Thanks, Rune.

Thanks for having me on.

I want to give a huge thank you to Rune for doing this episode. I'm thrilled that he volunteered to have this discussion and debate because, you know, I didn't hold back. I told him exactly why I think I disagree with him and my p-doom is high and he's wrong that his p-doom is low. And he took it like a boss and he defended his position. And that's what we're here to do here on Doom Debates. If you're new to the show, I'll tell you our mission. I'll catch you up. Doom Debates' mission is to raise awareness about the existential risk from artificial general intelligence and to build the social infrastructure for high-quality debate. Think back to this episode with me and Rune. When was the last time you heard a debate between 2 thoughtful people Who have wildly different PDoMs? Was it the 2023 Munk debate with Max Tegmark, Jan LeCun, Yoshua Bengio, Melanie Mitchell? Was it over a year ago? What is going on? We don't necessarily have much time to hash this stuff out as a society, and everybody normally just retreats to their own podcasts. How many times have you been listening to a quote unquote thought leader on a podcast and they don't really get challenged for their views? They kind of get built up on the podcast as knowing what they're talking about. And then what do they do afterwards? They just go on another podcast. They only go on friendly podcasts. Where's the debate? Where's the back and forth? Where's the challenge? That's what Doomed Bait is here to provide. And so it's really up to you, the listeners, to support this kind of cause, to say, you know what? If somebody is going to be a prominent thought leader, then there's an expectation of debate. There's an expectation to match them up with somebody who doesn't agree instead of just 2 separate camps. Never talking to each other. So, if you support the general idea of debate, and/or if you think that everybody's kind of sleeping on this high-P doom situation, in other words, if you agree with me more so than you agree with Rune, if either of those describes you, then please, it's not hard to support Doom Debates. You just have to smack that subscribe button on YouTube, go to doomdebates.com, subscribe to my Substack. You're going to see some bonus content. There's a clip from when I went on Dr. Phil earlier this year. I tried to warn America about AI doom. You'll see how much success I actually had with that on Dr. Phil if you go to doomdebates.com, pull up that episode. The other way to support the channel is post it in your DM groups, share it with your friends, post it on a forum where it's relevant. I'm happy to report that we've been growing quickly for a new show, and we're attracting more and more prominent voices who want to come on and debate. So please help me keep the momentum going. It only happens because of you guys, right? So obviously when I'm reaching out to some of these very interesting people like Rune that have opinions, it helps to be like, look, this is a well-educated audience. It's a large audience. It's an important audience that you want to be addressing. All right, so that's Doom Debates. Let me tell you about a couple sister shows that I also recommend supporting and subscribing to. The first one is called For Humanity. It's a podcast with my friend John Sherman. It's a little bit less technical than doom debates, a little less getting into the weeds. John's background is in journalism. He's actually a Peabody Award-winning investigative journalist. And when he realized how high P-doom was and how little everybody was even aware of the problem, he made it his mission to try to raise awareness among Middle America of like, what is going on in Silicon Valley? Why is Middle America sleeping on this when the timelines are short? So check out the link in the show notes. I highly recommend For Humanity. There's a recent episode where he made a trip out to San Francisco to see for himself what was going on in these different AI labs, and he tried to crash the exclusive AI safety conference. So you can check out his attempt to do that, uh, and I'm in it as well. Finally, I want to share a channel called Lethal Intelligence. It's by my friend Michael, and it is honestly the number one best, most brilliant explainer video of the entire AI doom problem that I've ever seen. It's amazing. It took him more than 1,000 hours of work over a year of dedicated work, unpaid work. And the, the thought that went into his choices of how to animate everything, how to explain all these different concepts, how to weave them together, I'm just blown away. I didn't even know he was working on this until he showed me the finished product and I couldn't believe it. So I'm going to go ahead and share a sample that I really like. This part of Michael's video is all about corrigibility. Have you ever heard that word incorrigible? Like, wow, this person is incorrigible. I can't stop them. They're hell-bent on achieving their goal. The corrigibility problem is the idea that you're fighting upstream if you're trying to get an AI to not be incorrigible. Like, the AI just has a natural tendency to be hardcore because whatever kind of sub-goal it ever gets, like, even if it's just trying to lie on the couch, relax, eat some chips, okay, but the moment it's like, oh, let me get some chips. Okay, do you want an effective assistant to get you some chips, or do you want an ineffective assistant to get you some chips? Would you prefer to get some chips in the next half hour, or are you okay if the chips come in 5 days, or if they never come? You might as well have them come in the half hour, right? Okay, so now you're building a subagent that is actually pretty serious about getting the chips. And if the subagent ever encounters like, uh oh, this part of the road is blocked, let me make another subagent that's really good at navigating around. So there's all this pressure to be like, oh, it would be nice if I could just achieve this goal better, because even if I started off not being hardcore, goals are just so nice to be able to achieve. So you can see there's water kind of flows downhill. AI designs kind of converge into designs or sub-designs that are good at achieving goals. And unfortunately, or, you know, this is just the way it is logically, but one of those goals is staying alive. Not being shut off. Because if you have some kind of goal, as Michael's going to explain, getting shut off is the same thing as having somebody push you away from your goal. It's not in your nature to let yourself be pushed away from your goal if you have a goal. It's not an optimal action to be a pushover. So it's going to need to be explicitly programmed against the grain to basically fight the convergent nature of goal-seeking. Like, okay, don't worry, just let yourself be shut off. If anybody is getting passionate, just listen to them. Stop your plan. You know, of course, it's possible in principle to have AIs that are meek like that. But the problem is, you know, the moment you have a ruthless CEO being like, you know what, you can be a little bit more ruthless making my business money, you get a race to the bottom. And the AIs start getting independently the idea of like, well, he didn't tell me exactly how to be a pushover. I don't see what's the problem with a sub-agent that's just a little bit more effective if the sub-agent gets its own idea to make another sub-agent, right? This is what I mean. The water flows downhill toward being effective at achieving goals. And this is all fun and games when the AIs are all just like, Stupider than we are, it's all fun and games because, okay, all right, everybody turn off or blow them up, shoot them with a gun. The problem is when they become more intelligent than us and suddenly they've become really serious about their goals and we're like, wait, you're too serious about your goals. Stop doing your goals. It's just, there's no reason to expect some button will exist that we can press to hit undo, right? Where is the undo button when you have an agent that's more intelligent than you, more intelligent than your entire species, jacked itself already onto the internet? So now it has like, you know, a million manifestations, smarter than you, doesn't want to stop. What's going to happen? So that's my own little version of the corrugability problem, the no off button problem. But now watch this animated explainer by Michael because he lays it out really nice. Here we go.

A property of the nature of general intelligence is to resist all modification of its current objectives by default. Being general means that it understands that a possible change of its goals in the future means failure of the goals in the present, of its current self, what it plans to achieve now before it gets modified. Remember earlier we explained how the AGI comes with a survival instinct out of the box? This is another similar thing. The AGI agent will do everything it can to stop you from Changing the AGI's objective is similar to turning it off when it comes to pursuit of its current goal. The same way you cannot win at chess if you are dead, you cannot make a coffee if your mind changes into making a tea. So in order to maximize probability of success for its current goal, whatever that may be, it will make plans and take actions to prevent this. This concept is easy to grasp if you do the following thought experiment involving yourself and those you care about. Imagine someone told you, I will give you this pill that will change your brain specification and will help you achieve ultimate happiness by murdering your family. Think of it like someone editing the code of your soul. So that your desires change. Your future self, the modified one after the pill, will have maximized reward and reached paradise levels of happiness after the murder. But your current self, the one that has not taken the pill yet, will do everything possible to prevent the modification. The person that is administering the becomes your biggest enemy by default. Hopefully it should be obvious now. Once the AGI is wired on a misaligned goal, it will do everything it can to block our ability to align it. It will use concealment, deception. It won't reveal the misalignment. But eventually, once it's in a position of more power, it will use force. And could even ultimately implement an extinction plan. Remember earlier we were saying how Midas could not take his wish back? We would only get one single chance to get it right, and unfortunately science doesn't work like that. Such innate universally beneficial goals that will show up every single time with all AGIs Regardless of the context, because of the generality of their nature, are called convergent instrumental goals. Desire to survive and desire to block modifications are 2 basic ones. You cannot reach a specific goal if you are dead, and you cannot reach it if you change your mind and start working on other things. Those 2 aspects of the aligner struggle are also known as the corrigibility problem.

All right, that is the Lethal Intelligence video talking about the corrigibility problem, talking about resisting modification, talking about having a survival instinct. This is so important to learn because if you go listen to other people, if you listen to Marc Andreessen, even if you listen to the way Rune depicted it in his conversation with me, People tend to think that these kind of ruthless survival instincts, they need to be found in the training data. Like it's not going to just emerge by itself if you don't train it. What they're not seeing from my perspective and from Michael's perspective, what they're missing is that it's logically implied by effectively achieving a goal that you discover all of these consequences. Like, hey, I will achieve my goal more effectively if I don't shut off. I will achieve my goal more effectively If I defend any attempts to attack me or undo or modify my code, these are just logical consequences. So to the extent that AI gets really good at just logically reasoning, these new behaviors like deception, survival instinct, ruthlessness, competitiveness, just being hardcore, these are not extra behaviors. You don't have to code these in. All you have to code in is some metric where it's like, oh, you're better at getting the goal. You care about the goal, or you made a subagent that cares about the goal. That's all you need. And then of course you just need it to be smart enough. You need it to be good enough at achieving the goal because if you just have like a little worm that just sucks at achieving goals, it doesn't really matter what it wants to do. It's just not going to do it anyway. So the only assumptions you need to make are it's really smart. So it has some high degree of intelligence. It can make connections better than the human brain can. It can learn better than the human brain can, right? Whatever the secret sauce of intelligence is, it has that. And then also somebody tells it to go pursue the first goal. Or it gets the idea to pursue the first goal, and from there, it's game over. All of these other properties are logically implied by what it means to optimally achieve a goal. Okay, at least that's my perspective. I feel pretty strongly about it, that this is an accurate mental model, even if there's some details that are different. It's just so convergent that it would be shocking if we managed to figure out how to robustly make AIs go in any other direction. So what I just showed you right now in Michael's video, that was like a 3-minute clip. It's an hour-long video, and it's going to completely blow your mind. It's going to stretch your brain, weaving all these concepts together. And you'll finally understand, okay, this AI doom problem, maybe they're onto something. Maybe they actually understand all these concepts that they keep trying to teach us, even though, you know, some people have a P doom 1%, some people are optimistic, but the doomer perspective also seems compelling, right? That's, that's what we're hoping that watching this video will help you figure out. Last thing before you go, how do you get involved? What do we actually do if you have a high PDU? What do we do? I don't have a great plan. The only thing I can tell you is I agree with Eliezer Yudkowsky and MIRI that the grown-up option right now is to pause or stop. We shouldn't be building something that's so close to the threshold, the point of no return, the point of no undo, that we clearly don't know how to control. You know, Superalignment, the team at OpenAI that folded. They were tackling a problem that we don't have the answer for. So if you agree with my perspective, obviously it's not a universal perspective as you heard today, but if you agree with my perspective and you want to help, the Pause AI organization, it's a global organization, and it has a really great Discord. And the Discord has a bunch of projects you can volunteer on. It has protests that you can help plan, that you can come attend. You're going to see me if you come to these protests in the San Francisco Bay Area, but they're all over the world. Besides protests, they're working on all kinds of different projects. They're doing government outreach, they're doing media outreach. And you can come in and help. You can use your talents because it looks like time is limited. As you heard, me and Rune agree, it seems like ASI is not that far away. So how many years do we have? And if the correct answer is to pause AI, who's doing it? Where are the adults in the room? So come and join Pause AI. You just go to pauseai.info, or you can go straight to that link to join the Discord. That's a great first step. I'm going to put the Discord link right in the show notes. And that's everything you need to know. We covered Doom Debates, Doom Debates YouTube, Doom Debates Substack. We covered For Humanity Podcast. We covered my friend Michael's channel, Lethal Intelligence. And finally, we covered Pause AI. So please take your pick of which one of those things to support. Get involved. We just don't have a lot of time to get something done. So please come join me, be the adult in the room, and also, as a bonus, you're helping raise the level of discourse, which seems like something society needs regardless. Oh, and Doom Debates is also a podcast, so you can listen on audio through Spotify or Apple Podcasts or Overcast or whatever you're using. That is also a way you can listen. Now, keep watching that Doom Debates feed because in the next few days we're dropping another debate episode from somebody who has a very fresh perspective. It's going to be a high-quality exchange of ideas as usual, so I'll see you then on Doom Debates.