Liron Shapira on the Case for Pausing AI

This week on Upstream, I sit down with Leron Shapiro, who is an entrepreneur, effective altruist, and member of the communications team at Pause AI. We do a deep dive on his perspective on the dangers of AI. Please enjoy this wide-ranging episode.

Do you want to give a brief introduction to Pause AI?

Sure, yeah. So Pause AI is a grassroots organisation started in the Netherlands by Joep Meindersma, and it's just picked up volunteers since then. You might have seen the SF protests. Those were part of Pause AI. And the mission's pretty simple. It's right in the name. It's, we really think that it's time to pause AI because we're getting too close to the point where AI becomes superhuman, and everybody admits to not knowing how to control superhuman AI. And most experts are giving a very short estimate to when we're going to get this uncontrollable AI. So we basically just think there should be a grassroots movement to pause AI, and we do a bunch of protests and lobbying to that end.

Cool. So one of the reasons why we wanted to chat is because recently I've had some of the EAAC folks on, and I want to make sure to hear multiple perspectives on this podcast. And so maybe we could start there in terms of You've been in the effective altruist community for quite some time, over a decade, and you have been following this discourse. And maybe you could share some thoughts on where you align with the EAAC perspective and where you think it's fundamentally incorrect.

Yeah. So EAAC is kind of a combination of techno-optimism, like standard techno-optimism, and then also kind of brushing off AI risk. AI existential risk, it's kind of sticking together 2 viewpoints. And we can get into this more later, but I think it's a textbook case of a Mott and Bailey. So it's kind of confusing. But if you look at just the techno-optimism part, I'm 100% aligned. Like, I've always been a techno-optimist. In fact, you can upgrade techno-optimism. You can go all the way to transhumanism. So not only do I like all the tech that we have, I'm thinking ahead to be like, hey, maybe we can discard our bodies. Maybe we can live in the simulation. And transhumanist means you just don't accept the parameters that nature gave you. So like, if I'm alive in the year 1 million, I probably don't want to be going to the bathroom. Like, I don't think that's a necessary feature that we need at that point, right? So that's the kind of thing that a transhumanist would say. So I've always been that. And the EAACs are going hard at saying like, hey, tech is great, progress is great. And in terms of vibes, like, I'm with you 100%, right? And in some ways I'm even more so. I mean, sometimes I notice that EAAC people are squeamish about other tech besides AI, right? So the other day, I remember you were talking to Dan and he was squeamish about life extension, right? So in many ways, and I have no problem with that. I think life extension is great. I think cryonics is great, right? Freezing your brain. So on a vibes level, I'm totally EAAC. The only problem is when you look at the other part, the assumption that AI is sufficiently safe and the only way out is through and we should just build it and see what happens. When you throw in that assumption into the mix, which seems built pretty deeply into EAAC, at that point I'm like, well, wait a minute. I question that assumption. Like, that seems wrong. And so then I can't be an EAAC.

Got it. So you're a techno-optimist, you're a capitalist, you're not a communist, you're not concerned about sort of equitable across different group populations or certain concerns that other people have. And you're a transhumanist, but you're not a posthumanist, right? You're concerned about sort of what's going to happen to—

Exactly. Yeah. Like, I'm not ideological, right? So people have a big tendency to hear somebody state their claim and then be like, aha, that's communism. I literally just don't want AI to kill me and my family. That's my only beef. So it's not, it's not communism, right? That's not something that Mao, Chairman Mao ever said, right? It's not from Stalin. I just don't want AI to take over the world and kill everybody before my kids grow up.

And, and why don't you share some of the, like, why do you think that that is an actual concern and why do they think that it isn't? What is, what is sort of the difference in, in beliefs there?

Going back to the Modern Bailey thing, a lot of times they don't really bother trying to argue for it. Like, it seems like they cook up the arguments to order when that point is pressed, when you're, uh, you know, challenging their Bailey or whatever. So I don't think EAAC actually has a consistent explanation about why it's totally fine to build AI. I think that the different people are going to give you different arguments, and some might even tell you that you don't even have to believe that to be in EAAC, but they sure seem to take it on faith. But if you ask me, why do I think it is dangerous? The nice thing is that I don't even necessarily have to give you my own position. I can always just point you to the AI labs position. Like, OpenAI is explicitly saying Hey, we don't actually have a way to align AI. Like, we don't know how to do it. We just think that we're going to be able to figure out some way to do it because we have a project called Superalignment, and we're going to give it 4 years, and we're pretty hopeful that we're going to succeed. So they're already asking you to be like, yeah, just basically trust us. We're going to figure out some way. It's their position.

Right. But they're not pausing, right? So presumably they don't have the same concern that you have, otherwise they would be pausing, perhaps?

Yeah. I mean, it's a very interesting question to ask, why aren't they pausing? Because from my position, What OpenAI is doing is just contradictory. Like, they have— their public position is self-contradictory. And I had an article about this published in The Information, which is saying when they started the Superalignment team, it was really great that they admitted that they don't know how to align superintelligent AI. But if you look at what they're admitting, they're admitting that it's a Hail Mary pass because they're not saying we're definitely going to succeed. They're saying, hey, we're going to do our best. And the prediction markets that got started around the Superalignment project Last I checked, they're giving a 15% chance of success. So their officially stated position amounts to saying like, look, we got to charge through and we're doing this Hail Mary. That's it.

And to just elucidate for listeners who maybe have trouble imagining the different ways in which AI can kill you and your family, why don't you say the ways that seem most likely or that we should be most afraid of?

Sure. Yeah. So this is, you know, we haven't mastered— us doomers, we haven't mastered the answer to this question because there's kind of different levels. If you go all the way to the top level, The top level is that the universe is like a causal web, right? And when you want to do something, there's a lot of causal paths to get there, and some of them look like magic. For example, how am I cooling my food right now? I'm using a refrigerator, and the refrigerator exploits the relationship between temperature and pressure, right? Now, that relationship was not known to humanity a couple hundred years ago. So if I told a human, hey, I'm going to use the air in this room to cool my food with no ice, they're going to be like, what are you talking about? You can't just make up this magic. Right? And I'm like, no, it turns out there is a causal path to get this food cold. So similarly, when you're an AI, just by using the power of intelligence, making a bunch of logical derivations with a little bit of empiricism sprinkled in, it opens up a lot of causal pathways that aren't all familiar to us today, right? If you fast forward all of human science and engineering, you could potentially do that inside of the neurons of an AI. So that's— so when you fast forward all the way to the end, you just have the AI being like, oh, okay, I want to take over the world. I'm just going to make nanobots. Or I'm going to make a new type of biotech. I'm going to engineer the DNA from the ground up, a much better version of proteins. I'm just going to have my way with every atom in the universe beyond what current human engineering feels like is possible. That's the ultimate, going all the way to the end of the spectrum.

And can you humor us a little bit and explain why AI would want to do something like that? Or people bring up the argument, how does it have the agency to— don't we control it?

Yeah. So I mean, there's toy models that you can help to build your intuition, because a lot of people have this intuition that AI is just software. just a chatbot. And I totally get it because I love downloading software as much as the next guy, right? And it feels harmless. It's just something I'm chatting with. And then people are like, look, it doesn't have agency yet. But pretty quickly, if you just do like toy models or just use like a little bit of logic, pretty quickly you see why the agency emerges. So for example, like there's a slide presentation put together by Jan Talyn where you just see like a very simple game world and you just define an AI and you give it a simple goal like, hey, walk over to this square and like push this little broom over or whatever. And then pretty quickly, when you say like, oh, here's the operator sitting inside the game world, pretty quickly the AI comes with a plan of like, oh, trample the operator. Like these kinds of plans, they just score high in these kinds of game worlds. You know what I'm saying? Or like the way AIs just cheat at video games. Like when you have a goal and you're just ruthless at achieving your goal, you just, you tend to just come up with goals that have all this collateral damage and the goals, the plans themselves exhibit agency. Like if you just ask a chatbot, like, just give me a plan. It'll be like, okay, here's your ruthless plan. I'm like, oh wow, if I were to execute this, I would be acting like a powerful agent.

Yeah.

You see what I'm saying?

Right. But is your idea that humans will use the AI to cause mayhem or chaos, or that AI itself will give itself goals that are antagonistic towards humans?

I think there's 2 major ways to fail. I think the extreme way to fail, which I've kind of gone all the way to the end of the doom train, so you can follow me all the way to the end, or you can get off on an earlier stop. If you follow me all the way to the end, the AI just becomes uncontrollable. So it doesn't really matter who builds it. That's what I believe, because I think at some point, they're going to have some beta that they're testing inside some lab, and they're going to run it, and it's going to enter a loop, which there's no debugger button in that loop. There's no undo. You just screwed yourself. It's taking over the world. And it would actually have to require a lot of code to even define what it would mean to stop at that point. It's like, imagine hitting stop on some of the viruses that are still running wild today. We have viruses from 2004 still running wild today that nobody's figured out how to stop, and they're causing $1 billion a year of damage. If you look up MyDoom, Right. So even stopping a virus from 2004 is already a very difficult problem. Stopping an AI that accidentally escaped from the lab, right? So it's even smarter than stopping COVID, you know?

Yeah. And so what would give you more confidence or what would change your view? Is there anything OpenAI's superalignment team could do that would give you more comfort and say, hey, actually, I think differently now?

So there's a lot of pieces to my view. The thing that would make me more confident that we're going to survive a long time is if we find some fundamental reason why we're not going to get superhuman intelligence in the next century. At that point, it's like, okay, at least we have time. We can use time to figure stuff out. Because it's not theoretically impossible to figure out how to make a controllable AI. We're just nowhere near there yet, and we're building it anyway. That's the only issue. But if you told me we have a lot of time, That would be the simplest way to get me to breathe easy. Like, okay, I can be optimistic because I'm not a pessimist. It just seems like the situation is super dire right now. But if you just give me a little bit of time, I'm happy to be optimistic about the time.

Right. And so it is, just to clarify, get our points on the table, it is your view that AI is an existential threat to humanity or that it will lead to the death of all humans? Or what is— flush out what you think is likely. Yeah.

My mainline scenario, so I give my P-Doom by 2040 is 50%. So I think there's 50-50 chance we're all dead or not, primarily due to AI. And my kind of mainline scenario does involve AI becoming uncontrollable. And I'm basically, I do think FOOM is going to happen, maybe not to like an infinite degree, but FOOM is, you know, when AI like improves itself, you know, because intelligence just finds ways to be more intelligent. That's just like an obvious thing that intelligences do, just like they like seize resources because You know, instrumental convergence. It helps you to do so many goals when you just pick up this intelligence, pick up these resources. So my mainline scenario is you have an AI that they didn't align. They didn't really give it a coherent goal. They did something like RLHF, right? So they upvoted a bunch of its responses, but RLHF just didn't do the job to align it, as OpenAI admits. So the actual goal that it ends up with is something that's totally half-baked. It's a bunch of shattered pieces of things that originally came from things that humans like, but they just don't make sense when you kind of extend it to infinity. So we can talk about shattered goals later, but the idea is it's not aligned with human values. And then it says, great, I'm going to go optimize the universe according to these goals. And humans are like, wait, no, this is actually hell. You're creating hell, even though we upvoted certain responses. This isn't what we wanted. And then at that point, it's unstoppable because cognitive power, it's the ultimate winning power. It beats every human weapon. It beats the sum of all humans. It's just too far gone.

You don't think that the concern is that you describe is based— that we'll get there from current technologies like transformers. You think that there will be new breakthroughs, right?

Yes and no. So in principle, the reason why I'm worried is not because I'm looking at the exact way that transformers work and I'm saying, this particular technology is about to kill us. No, I'm not looking at the algorithm that does the work of being intelligent. I'm looking at the nature of the work being done, right? Intelligence is a type of work that an algorithm can do, and it's a dangerous type of work And we're getting better at doing the type of work. You see what I'm saying? So I don't need to know the exact details of the algorithm to be scared.

Why do you think that most people disagree with you who are also very smart in the space? I know there are many who are concerned, but they don't get to the level of your concern. How do you make sense of that?

Yeah. So first of all, I mean, just to zoom out, right? If we're talking about people's opinions, my actual opinion is these days pretty mainstream. Right? Like, first of all, so among the population, right, people just don't think about the issue much. But if you actually ask them to think about it, they do get pretty scared. And they do say, hey, let's exercise caution here, because this does seem existentially risky, right? That's like the average American's opinion, right? Right.

No, but how about sort of people who are building, people who are in the space, even the ones who are concerned, don't, they don't have a PDoM at 50% or something, right?

Yeah, exactly right. Because, you know, like Dario was saying, 10 to 25%. And Yann LeCun from OpenAI said 10 to 90%. Right. So let's say that they'll give you like 5 to 20%, and I'm here saying like 50%. So there is a little bit of a gap, but that gap is pretty small, right? I mean, is there really a big difference between 50% and 10% when we're talking about near-term human extinction?

Right.

And so in a world in which humans get extinct, just to play that out for a second, what happens to the world, to the universe?

So the universe just gets optimized according to what I'm talking about, like the shattered values that the AI has, like values that if we could reflect on them, we'd be like, oh no, this is not what we want. And actually, an example of a pathological value, probably not the actual value that AI is going to have, but if you look at what Bef Jezus has been saying of like, hey, we got to maximize free energy. If you actually told it, if that actually gets learned by an AI and the universe just gets maximized to always be dissipating free energy, we're talking about hell. That has nothing to do with human values.

Do you think it's inevitable that humans will be to what's next, what Neanderthals are to us?

Well, I think that it's inevitable that there's going to be some species that is extremely intelligent and colonizes the universe, right? And this is actually Robin Hanson's grabby aliens theory. Have you looked into that?

I've heard of it. Can you— but can you flesh it out?

Yeah. Robin Hanson solved the Fermi paradox in 2021, and he's like super underrated. You should probably bring him back on Upstream and have him teach it all to us, because I mean, this is like, this is a landmark achievement.

Yeah. So can you say more about how we did that?

Yeah, sure. So the idea is like the Fermi paradox is like you look around and you say, why is nobody else out there? And the grabby aliens idea is saying like, look, there's going to be aliens, there's going to be some fraction of aliens that when they do emerge, they start colonizing the universe as fast as they can. Maybe our species is even one of those, hopefully, if we don't die out. More likely, the AI that kills us is going to be one of those colonizers, right? But the idea is there's going to be occasionally, there's going to be these colonizer aliens. And so the universe is going to be a land grab one way or the other, right? You don't need every species to be a colonizer. You just need some species here and there to emerge and then go grab some land. So when you emerge as a species, the fact that you had the ability to emerge means you're probably very early in the universe's timeline because it's not just like you don't just have a trillion years to emerge. Like you don't have the lifetime of the universe to emerge. You better emerge before your territory gets taken up by one of these other alien colonisers. And so that's why it's actually much less surprising that we instantiate on Earth and we look around and we're like, hey, where is everybody? Why does this look empty? And the answer is just because we're early.

Got it. And why isn't Robin Hanson worried about AI? Or how do you make sense of his perspective?

Oh, man. So there's a few people who I think are people that I respect a lot, right? And their thought process has been great on so many issues. And yet I strongly disagree with them on AI, right? And Robin Hanson is one of those people. And he's one of the interesting ones, right? Like, if I had to have a long conversation with somebody, Robin Hanson would be the person who could kind of keep me engaged the longest, right, with his descending perspective. And honestly, it doesn't line up very well with other people's perspective who think AI is fine. It's very unique to him, in my experience. But if I were to put on my Robin Hanson hat, I think he's saying, like, look, what do you expect for the future of the universe? Like, it's just going to be optimized by something which we create one way or the other. So whether that's like our children having fun, Or an AI creating a bunch of paperclips. Like, whatever happens, it's all fine. It's just competition. Like, it's good. I don't know what you were expecting. That's like one attempt of me to copy Robin Hanson.

Yeah, we'll have to, we'll have to get him on as well.

We'll continue our interview in a moment after a word from our sponsors. The tech world turns to the Brave browser for its unbeatable privacy protections. But did you know that Brave also has a private ad platform? Brave Ads offers first-party targeting, and it's been cookieless since day one. So you can relax while third-party tracking cookies disappear from the web. Today, millions of people turn to ad blockers to avoid being tracked and pestered online. But Brave's new ad model aligns incentives for users and advertisers. Users earn rewards for viewing ads, which they can save, spend, or pass along to their favorite creators. And advertisers score points for respecting user privacy, generating ROI without invasive tracking. So, whether it's high-impact announcements on the new tab page or keyword-targeted ads in Brave Search. Brave offers diverse, private, future-proof ad formats for all your business goals. Join the future of advertising at brave.com/ads. Mention MOZ when signing up for a 25% discount on your first campaign.

The—

when you did this debate with, I think, Alexander Campbell, and one thing that he said was that there's a difference between intelligence and power. And just because something is more intelligent doesn't necessarily mean it will accrue power. And Marc Andreessen likes to say something like, hey, when you look at the people who are powerful today, do they look like the smartest people in the world? You're the highest IQ, Kamala Harris, Joe Biden, Donald Trump, probably not, right? So you think that's a misguided point?

2 things. Number one, forget the word intelligence. This whole time, I actually meant to just be talking about power. Right? So, so who cares if power is related to intelligence? My whole reason to be scared is because I think that AI is going to grow far more powerful than humans. You see what I'm saying? So, so we don't even have to talk about intelligence. You can ask, why is it more powerful? And then it could be like, well, its source of power in many ways is going to be analogous to a human source of power. And obviously a human source of power is in our brain, so it's going to be doing brainy stuff, right? But I haven't said intelligence yet.

Can you?

Sure, yeah. And now, so I got to, hey, it's going to be doing brainy stuff, right? Like, it's going to have some organ which is like a brain because that's our organ of power. Okay, can I say more about that? There is actually a connection between IQ and power. You can observe it empirically. So that, you know, Marc Andreessen has said this, Jan LeCun has said it. They love saying the line about like, my boss is not, you know, the person that works for me is smarter than me. Like, they love saying that, okay? But let me ask you this, guys. How many people who work for you have a 2-digit IQ?

Very few.

Right? Like, there is some correlation.

Take a sort of EA type who is in— what you see is in good faith. What do you think would be the most compelling points that could potentially get them to evolve their perspective even slightly?

Oh, sure. Yeah. Okay. So how do I take generic EA and— so, okay. I think I can do that. I think generic EA, you know, because EA likes to stick together the argument that AI is going to be safe with the argument of like, let's be optimistic. I guess I would try to separate out those 2 EAC claims. I'd be like, look, I'm a techno optimist like you, but what would it take to make you not a techno optimist about one particular issue? What would it make— what it would take to make you admit that something is an edge case? For example, if we can dial up the danger of something to a ridiculous degree, doesn't it become an edge case? You know, like, don't look up. If you literally look up and you see the asteroid coming, maybe do you become less of a techno optimist about the ability of Earth to survive for the next century if you see the asteroid is almost here? You see what I'm saying?

pessimist. Right. And I think they could say back, well, what would it take for you to believe that the asteroid is actually not, not an asteroid, but something that is going to work with us and uplift our lives greatly?

Yeah, if that's what they say, then I would agree that it's a crux of our disagreement how dangerous the AI is. I agree. So let's just talk about how dangerous it is. And I'll, and then I'll convince you why I think it's super dangerous.

Right. But just to play that out for a second, is there, is what would need to happen for you to not think it's dangerous? Or what would need to be—

Yeah. So, so I said before, it would be great if I just thought it would take a very long time, right? Because then I'm happy to be optimistic that we'll kick off a research program and we'll figure out a lot of theorems of alignment, and then we'll have tools to actually align it by the time it comes to be much smarter than us and run away, right? So that would be probably the most obvious way to make me optimistic is to be like, we have so much time. And by the way, that is what I thought 10 years ago, right? And I used to just be chill and not even think that much about AI because I thought we had plenty of time, which was what most AI doomers used to think. Like, we used to think we had more time. But the Turing test is practically passed. I mean, we didn't expect these things to fall quite so early. All right. So anyway, that's one argument of what would convince me to cheer up. The only other thing, it's hard to imagine being convinced like this, but it's not impossible to imagine, which is like, if I can see an AI that's able to generate arbitrary plans, because planning is kind of like the ultimate test. If you can plan over the physical universe, if you can output sequences of actions that can hit arbitrary goals, that's kind of like the secret power that humans have, the ability to map desired universe states to plans for how to get there, right? That's like the one— that's like the ring in Lord of the Rings, right? That's the ultimate power. If I could see AI have that power at human level, uh, and even, you know, like starting to surpass human level, but just like for some reason still not show all the signs of, you know, of trying to foom, getting the idea of like, oh, I need to augment myself now, right? Or like, oh, I need to gather resources. We're like, for some reason being a lot tamer than I'm expecting. Then I'm able to update and be like, huh, why did I think all these things? What's wrong with my theorems? But it's hard for me to explain to you how I would be wrong, because I'm very confident in the logic that says this is a bad scenario.

So Tyler Cowen is also, like Robert, far more sanguine on this topic, in that he's not as concerned. And one other perspective he has, which many other people have, is cat's out of the bag. And we could try to pause, but China's not going to pause. And so it's, it's, this arms race has already occurred. And so it's really, do you want a China sort of led AI world or an American led AI world? What do you say to that?

Yeah, and I've heard it phrased where somebody was saying, look, you have to just take it as an axiom that we can't coordinate to pause, right? And then so given that that's true, what's your proposal? And I'm like, wait a minute, can I have my own axiom? Can I suggest an axiom, right? Like the axiom is that we can't survive a superhuman AI that comes in the next 10 years. That's my axiom, right? So the question is, whose axiom is stronger? Unfortunately, I feel like my axiom is also very strong. Like, we're stuck between a rock and a hard place here.

Right. So, well, let's take both of them just for the sake of argument. If we took yours, then you have to pause.

Right, exactly, right? And by the way, like, I don't like this, right? I don't like saying, let's not open source this. And normally I love open source. Right? It's like people love being essentialist about who I am, right? They might be like, oh, Liron wants power, Liron's a communist. It's like, no, I just— look, I just think it's going to kill us when it's uncontrollable. Like, I don't have any special insight about what you have to do if you accept that that's true. I just feel strongly that it's true.

And so, but if you took the axiom that, hey, it's going to be China-led or American-led, and that there's no way that China will slow down, Well, first off, is that that unfounded? Do you think that's a reasonable concern to have?

I've heard a lot of people commenting that China seems very open to pausing for a number of reasons. One is because there's some people there who find the arguments compelling of like, oh yeah, this is dangerous. And then also China, it's like they don't love rocking the boat, right? And they like having control over their people, right? So people have theorised that, well, if people have access to AI in China, that's just more tools for them to undermine the government, right? So China seems open-minded about pausing.

Well, pausing maybe on the civilian level or something, or not giving access to the civilian, but sort of at the government level or the lab level, maybe something nationalised or something that they continue to advance, but don't give access to other civilians, just like drones or other technology they've been very ahead on.

Right.

I mean, if they had a guarantee, right, if we could actually give them the confidence that it's an agreement where if they pause, then we really will pause, right, if they actually had that confidence, I feel like they'd be really open to it, but the biggest flaw in the plan is how do you really know that nobody's secretly developing, right? That seems like the weak point.

Right. And they're not exactly the most trusted partner on issues.

Right. So then you get into the subject of choke points, right? I mean, there's not that many companies that are manufacturing these large data centers, and the time is slipping away when we even have a choke point. It's already like a nice stroke of luck that the AI research is using a lot of compute right now, because the moment it starts being really compute efficient, At that point, we don't even have choke points. Like, at that point, it's like our options are even narrower.

So if you could wave a wand and, you know, Joe Biden would do anything you say, or whoever has the power to influence those choke points, what are they exactly? Or what would the ideal policy mechanism be?

So I would describe it as we need to build an oomph button, and it has to be by international treaty. So the idea is that anytime you have a building with a bunch of compute in it, You know, it's installed with chips that have a central turn-off mechanism, right? Like maybe they receive radio signals or just like there's an easy way to turn them off and the people in the data center have to cooperate. And the idea is that just, you know, there's just a big penalty for not cooperating, right? And I know it's been mentioned like, hey, we should do an airstrike if they don't cooperate. And then people freaked out like, oh my God, doomers want violence. It's like, no, we're just talking about the way that treaties get enforced, right? If we all agree that there's a treaty and somebody's actively being like, screw your treaty, I can do what I want. It's like, okay, well, at some point, you know, then in that context, you use violence to enforce it.

Yeah. Vitalik wrote a piece recently, sort of describing his perspective, which he called DAC, I think standing for Accelerating Defensive Technologies or Decentralized Technologies. Why don't you sort of unpack what you think he said, or what you took from his piece, and then your reactions to it?

Sure. Yeah. So the thing that struck me first is when he said on Twitter that his P-doom is about 10%. So I was like, oh, thank you, Vitalik, right? Even Even one of the crypto leaders is admitting that we're talking about a serious existential risk here, right? So that social proof is very validating, right? Because there's still a big gap between the way people are acting and the fact that people are explicitly saying, hey, we're all about to be doomed here. Like, right? There's still like a kind of a don't look up. People call it a missing mood, right? Just a gap between how people are acting and the reality on the ground. All right. So his pDoom is 10%. As for his DAAC post, you know, it was universally acclaimed just because it's being constructive, right? He's not just like randomly dissing people or taking one side. He's just saying like, look, we all don't want to die from our tech. We all believe in building tech. He pointed out that future visions of good AI are unclear. I thought that was an interesting point. Like, you can't go and read a particular book being like, here's what the future looks like when everything is great because the AI has made heaven. You know, like, do you know any book like that?

The Bible? I'm not sure.

I think, you know, the Bible is very vague about heaven, right? They kind of like name-drop heaven and then they like wave their hands like, yeah, you know, you get it. It's good. It's like the lack of pain.

Yeah, got it. And so, why is that important or what are you thinking about?

Well, one reason it's important is because he's saying like, look, everybody's saying we got to build this AI because it's so important to get this outcome, but people aren't even clear on what the endpoint is. So, it's kind of like an indefinite optimism. Heaven has always been very indefinite optimism. To use Peter Thiel's term, nobody's like, oh yeah, heaven. These are the different sectors of heaven. These are the main things we want to get out of heaven. Nobody's really written a spec for heaven. One of the closest things you'll find to a spec for heaven is actually written by Eliezer Yudkowsky. He actually set himself the goal. He's like, look, somebody's got to define heaven. And if nobody else has done it, I'll give it a shot. And so he has this thing called the Fun Theory Sequence, part of the LessWrong sequences, where he's like, okay, here's a bunch of criteria for if somebody wants to write a spec of heaven. Here's some things you probably want to know about fun and increasing challenge, increasing skill levels. So he basically defined He tried to gamify or define metrics for what makes heaven actually good so we don't accidentally create a boring heaven or anything like that.

But some people have tried to speculate kind of the good that could happen, right, in terms of the cost of everything going down and the poorest person living like Bill Gates or this kind of the mass productivity increase and all the lives that would be saved as a result of the benefit, Yeah. Access to medicine and other things like that.

Yes. So, I think we can all agree on certain dimensions that are good, right? So, anything that'll leave suffering is great. It's just that like, okay, great. So, everybody just has everything, but then it's like, wait, but do you want to take that all the way to everybody being on morphine so they have all the happiness they want? You see what I'm saying? So, you don't want to dial it to infinity. So, at some point, you do want to calibrate your dials.

Right. And so, okay. Any other reactions to Vitalik's piece?

Yes. Okay. So he pointed out future visions of AI are unclear. And then he emphasized the idea that you don't want a bad actor to control it. Whereas, you know, I said before, I worry it'll go rogue even before we get to Vitalik's doom, right? So Vitalik's doom is kind of a good doom to have relative to my doom. Like, so, because then, you know, hey, you just have to put it in the hands of good people and then we'll be fine. You know, that would be a nice thought. Okay. And then I guess the biggest weakness of the piece, which you can't really blame him for, is that he didn't propose a good solution, but like nobody really has. His proposed solution was saying, hey, let's build defensive tech. But an artificial intelligence, an AGI, can execute on any goal. So any super intelligent defense AI can play super intelligent offense, right? So that you can't really build an AI that has defense in its nature. But overall, it was a good piece because he stated the problem well, and he got everybody on all sides of the equation to kind of be a little bit more thoughtful and try to open up the game board. But unfortunately, I don't really see anything that new on his game board.

Every time there's a new technology, it enables attackers and defenders. Do you want to talk about sort of the relationship between them? Some people would posit that the defenders usually have an advantage, or historically, every new technology that we've invented, we've managed to get the benefits and also keep in check some of the more—

Yeah, well, there's different— it does depend on the context. I listened to a really great new podcast called Hidden History 102, and I learned that, uh, you know, trench warfare was an example where, uh, the trench, you know, had gave defense the advantage for a few years before they, they had tanks, and then offense had the advantage again. So sometimes defense has the advantage. Um, I tend to think that generally offense has the advantage in the universe as a whole, just because it's like, how do you defend against a nuke? A nuke that if you can get that nuke in the process of exploding, like, what's the defense?

I, I can't think of one, but you're not persuaded by the idea that if everyone has a nuke, then, you know, we won't shoot one, or it's kind of what's happened for the last 70 years.

Well, I'm glad you brought that up, right? Because there's this idea where the EA crowd is so committed to this idea that like, we should open source everything, we should go full throttle on everything, that they're even biting the bullet. And they're saying, oh yeah, nuclear proliferation, let's give everybody the nuke. The more people who have the nuke, the better. That is a really funny bullet to bite. And I remember, I think it was even on this podcast upstream where Ben Horowitz was saying like, the reason we're safe is because more and more countries have nukes. It's like, well, wait a minute. We've actually, it's actually been our number one foreign policy goal for these last 80 years to stamp down nuclear proliferation. And everybody is shocked how successfully we've done it and how few countries have nukes. And this is still considered an unstable situation. And there's been countless accidents, right? We've identified like a dozen different moments in history where we were like one little switch flip away from having like a multi-megaton explosion, you know, killing like 100 million people.

But we haven't, and it's been a relatively peaceful time since World War II. Is that right?

Yeah, relatively, right? So I mean, if you look at Toby Ord's book, The Precipice, people who have gone in and said, okay, so what's going on? How do we analyze this objectively? Don't just look at the result. Yes, we survived this far, but look at all the close calls. What does it seem like the odds were at the time? And the outcome is something like 1% a year chance of nuclear doom. Which to me is probably the second biggest existential risk after AI, but it's a pretty far second, especially because nuclear war will probably stop before every last human is dead. It'll just decimate the human population and set us back many years, but it won't wipe everybody out. But it's a 1% a year type thing. I mean, we are by no means out of the woods. And I highly recommend the book, The Doomsday Machine by Daniel Ellsberg. Mind-blowing book. He basically said that the movie Dr. Strangelove, It's just a documentary, like the ways that nukes can just get out of our hands and explode and all those close calls. Like, it's totally real. And just like, just one anecdote there, like people don't realize, you know, the Cuban Missile Crisis, right? So Khrushchev decided to back down. He risked his whole reputation. Like, his countrymen were like, what are you doing? Like, his reputation was terrible. He basically fell on the bomb and sacrificed himself. And that's what kept humanity safe and kept us out of the Cuban Missile Crisis escalating.

So when people bring up nuclear as a cautionary tale in terms of, hey, we don't want what's happened to AI to happen to nuclear, where it's been regulated to death and we haven't had all of the benefits that could have maybe prevented some, some wars and all the millions of people that, you know, had their lives impacted or trillions of dollars spent going to, you know, these Middle Eastern countries to, for, you know, for oil extraction reasons or energy reasons. Or just, you know, the effect on the climate, etc. You actually say, no, that's not a cautionary tale. That is a success outcome in that, you know, we further prevented nuclear war or something like that and that we should imitate.

Yeah. So, yeah, you're right. There's this— this is the second misconception that the EA crowd has. So there's 2 misconceptions. The first misconception is what we just talked about, that they think that nuclear proliferation is good and it's like a big success story that nukes didn't kill us. And they're generalizing that like technology can never kill us because nukes turned out great and we open sourced It's like, no, we didn't open source nukes. It didn't turn out great. So that was misconception number 1. Misconception number 2 that you're getting at now is like, look how good nuclear power is. And when we get scared about nuclear bombs killing 100 million people, we stamp out nuclear power. Okay. First of all, I would argue that killing 100 million people is so bad that it outweighs a lot of the benefits of nuclear power. But put that aside. I'm pro-nuclear power, right? And the reason I'm pro-nuclear power is if you look at the objections to nuclear power, It's not nuclear power is bad because there's going to be a bomb that blows up the world. I mean, that's one of the objections, but a lot of the squeamishness that actually caused the regulation is more like, oh my God, the waste is so bad. The radiation is so bad. And it's, and also the meltdown risk is so bad. And those objections from an engineering perspective just turned out to be wrong. So because those were just factually wrong objections, then the regulation has just been super misguided, super vibes-based. Like it turns out it's possible to have safe nuclear power. And therefore I'm pro-nuclear power, but I wouldn't connect that to the nightmare that is all of these nukes sitting around. These nukes sitting around is what I call a game over button. It's a single button that somebody could press where 100 million people die. And we know nature's trying to tell us something, right? That those buttons exist in the physical universe. It's like, what does that imply? That a game over button can exist. Maybe a bigger game over button can exist, you know?

Right. And so, so your ideal policy around nuclear would've been Would have been develop nuclear power, but somehow pause the ability to create nuclear weapons, or—

Yeah, exactly right. Yeah, I would have been pro-nuclear power because I wouldn't have worried about the waste. I would have said, hey, we just need decent standards for meltdown. And you know, there's an agreement— this is part of stopping nuclear proliferation because countries use it as an excuse. They're saying, hey, we just have a nuclear power plant, right? Iran is notorious for this, right? They're like, hey, we're just doing nuclear power, don't worry about us. So there has been some structure set in place where if a country has inspectors and agrees not to be building nuclear weapons, there's like a central repository of nuclear resources that they can use to make their power plants where they don't need to be doing the components that only a nuclear-armed country would have to have. You know, I don't know the details, but it's like you don't have to refine plutonium or whatever it is you have to do. Like, you only have to do the stuff that's actually for nuclear energy. So that structure is actually pretty straightforward, right? So we have a system to be like, hey, No nuclear weapons proliferation. Yes, engineering nuclear power plants. And like we we kind of we we got through, right? Like that's a challenge that we can kind of see our way to a solution as long as we can like get the arsenal down, right? Not have a thousand nuclear weapons lying around. Okay, that's fine. But the big takeaway for if you just zoom out and you're like, what's the takeaway of all this? I would say everything pales in comparison to the fact that today, as we live today, there are game over buttons sitting around in the world because the takeaway from that. Is that we can build another one that's even bigger, which is what we're doing with AI.

And so in your dream world, if Pause AI, you know, your movement were to gain momentum, what should OpenAI and Anthropic and all these AI companies do?

They should pause AI, right? That's my best suggestion. And again, it's a downer, right? Because it's like, I've never, I've never in my life said to pause anything, right? Like, it's, it's never been necessary. And it's not something I would never be like, I've never had an inkling in my mind to be like, oh my God, social media is so bad. Let's pause social media. I love Bryan Caplan's attitude about it. You know, and people are like, oh my God, you're violating people's privacy and like the data is leaking. And he's like, come on, don't be a wuss. It's fine. It's not that bad. It's a relatively small problem. So normally I have that perspective. It's just in this case, it's an edge case. And I just think we're all— it's an FAFO situation, mess around and find out. We're messing around so hard right now. I mean, one way to see that we're messing around, we don't even know how the human brain works. There's not a single person at any AI lab who will tell you, oh yeah, I know how the human brain works. They all admit that we don't know how it works. And so what are they doing? building something that's about to be smarter than the human brain.

And your connection there is if we don't know how it works, then we don't know how to control it. Yeah.

Now, don't get me wrong. I mean, I didn't just make an airtight logical argument. It's more like an intuition pump, right? That when we don't know how the human brain works, doesn't it seem prudent to get a little bit more interpretability or just insight into what's going on with humans before you say, hey, let's just make a smarter species and let it go. I'm sure it'll be fine.

Because of the idea, if we understand it better, then we can reprogram it or direct it.

So yeah, I mean, it's a matter of security mindset, right? It's just like you have this big bag, right? You have this black box, and it's doing something under familiar operating conditions, right? Like when humans— so the familiar operating condition of a human is like, none of us have that much power individually, right? So you ask like, oh my God, why doesn't Walmart foom? It's like, look, Walmart doesn't have that much power. Nobody at Walmart is super intelligent. So you watch humans doing human things when we're all powerless. But like, imagine scaling up, even if you scale up a human brain, it's not even guaranteed. Even if you take a good person, but you scale up their power to godlike power, you know, they say power corrupts, right? We don't even know what a human would actually do under those parameters, right? It's an untested condition.

And so you want them to pause AI. Does that mean kind of shut down their orgs and go home? Or like, what should they actually do?

Yes. So the ideal situation, I know it's asking a lot, is It's illegal to advance capabilities until interpretability and safety catches up, which, by the way, what I'm saying now is literally a direct quote from Sam Altman, right? He has said in the past, hey, it's my priority. I think a lot about how do we make sure that capabilities don't race ahead of safety, right? So this is like a standard line from the AI labs, except I'm saying to actually do it. Don't just do a Hail Mary where superalignment is just operating at their own pace while the capabilities team is operating at their own pace. It's like, no, no, no, because capabilities is a problem that has positive feedback loops. The AI starts to be able to debug itself and see— there's only one way to have capabilities. There's only one way to manipulate reality and find truth. There's only one objective truth, but there's a lot of ways to decide what values are. You see what I'm saying? Reality doesn't give you feedback when you have the wrong values. Only the human engineers can give you feedback when you have the wrong values. So it's inherently a much slower process that you can screw up more easily.

And you say 15 years is not enough time to give sort of the researchers time to figure out sort of interpretability or alignment. You think it needs to be a lot more?

I think a decade is the minimum, just because people criticise MIRI and the Machine Intelligence Research Institute, Eliezer Yudkowsky's institute. It's popular to criticise MIRI to be like, oh my God, they worked for 15 years and they didn't solve the problem. It's massively underrated how much MIRI has done. They've published multiple papers. I mean, even something like popularizing the threat of instrumental convergence, right? Just pointing out of like, hey, AIs will grab resources. Like, that's kind of the first thing they do. Things like that. There's actually, there's a dozen things like that that are as big a deal as that that I can name. But they've been researching for 15 years. And yes, they didn't solve the problem. But if you look at a problem as difficult as P versus NP, this is a good analogy, you know, from computer science, proving P versus NP. It's like, it's been unsolved for like 50 years. And you get a million dollars if you solve it. It's like a very profound problem. So why haven't people solved it? Because whenever they go try to solve P versus NP, they end up hitting on a new result where the research paper that they discover just tells you why they haven't solved it yet because it's so hard. What they end up proving is, hey, I found another reason why this is hard. Like I just eliminated another category of possible proofs. So here's my research paper about another way that you can't Go about solving P versus NP. And that's a lot of what MIRI has already done for AI alignment. Like, they have a research paper saying, hey, it's not going to be easy to solve AI by just building a quote unquote off button. Like, that's a really big result that it's not easy to build an off button.

What do you say to the argument that the only thing that can protect us against AI is other AI, and that there needs to be this sort of adversarial competition dynamic for where the AIs can keep each other in check? Otherwise, even if we pause now, there's kind of one sort of main entity that has more power without sufficient— and as a result of not open sourcing, we don't get the benefits of scrutiny that the scientific and research community could apply and thus develop more alignment and interpretability.

So in my mainline scenario, that argument is irrelevant because it's not like you can have one good AI that fights off the bad AIs, because the whole premise of my mainline scenario is And you can't even have a single good AI because you never learned how to tell your AI to be good, which is what OpenAI admits. They don't know how to tell any AI to be good, right? So that's the main problem.

They don't know how to tell AI to be good.

Correct.

But we don't have any evidence, do we, of AI being bad or misaligned, do we?

So when they train the new GPT, you know, like Nathan LeBenz was saying this, you know, he got to play with GPT-4 early, and he's confirmed that, you know, it's It's amoral, as you would expect, right? It's only just focused on, you know, you give it an input and it'll just do it, right? It doesn't think about whether it's right or wrong. And that makes complete sense that you could optimize your first training run to just understand reality, because like I was saying before, right, reality gives you great feedback to converge on like the one reality. Like it's a very natural training objective. And so you're saying, hey, you look at GPT-4, GPT is fine. Yeah, because RLHF works when you're training an AI that's subhuman. Right? It can't outsmart the humans giving it the upvotes and downvotes because it's just not that smart. It's like, yeah, I'll go along with your vibes. And if GPT-4 does well in training, then it's also going to do well in production because there's no crazy cases in production.

When you debated George Hotz, how would you characterize his perspective or his arguments?

Yeah, his perspective was evolving. One of the cruxes of our disagreement was I kept saying like, George, do you accept the premise that there's a lot of headroom above human intelligence, right? The scale of intelligence goes much farther than Einstein. Do you accept that premise? Because a lot of people are like, oh yeah, Einstein, he's really smart. And obviously he is, but at the end of the day, Einstein is working with a 2-pound piece of meat, smaller than the size of a basketball, using 12 watts of energy, that's also doing other biological functions, made out of chemical moving parts, firing at 20 times per second. Like, these are disaster— this is disaster of a spec for a brain.

Right. And so he wasn't willing to concede that, or?

He was kind of skeptical. One argument that he busted out was like, look, yes, I bet you can be more intelligent than humans, but you're going to need so much compute, right? You're going to need $7 trillion of compute, right, as the number that's been thrown around. And I pushed back on that. I'm like, the evidence from evolution, it looks like evolution was actually starting to foom. If you look at the genetic record going from the other apes to humans, there was a genetic boom starting, a few genetic changes. I mean, think about how smarter, how much smarter we are than an average ape. We are a lot smarter than an average ape, right? We can do a lot more. And yet we're talking about a blink of an eye in evolutionary time. And then you can see that the human body, right, the birth canal got as big as it could before women couldn't even walk anymore, right? Like women, too big hips, they can't even walk anywhere. They need to walk, right? So there's a constraint there. There's a major metabolic constraint, right? Like the head is already sucking up more than its fair share of resources. So under all of these constraints, evolution basically went as hard as it could. And next thing you know, 20% of human babies are dying in childbirth, or the mother is dying in childbirth. Like, it really tried to turn the dial of intelligence as far as it could before it got slammed down hard. And so, like I said, we're in the process of an evolutionary FOOM. And George Hotz is saying like, oh, scaling up would be hard. It's like, look at evolution. It's getting a lot of mileage out of the scale-up.

What do you say to the argument that Or to the response of your argument, that the argument that you're proposing is, is, is left the realm of, of sort of the scientific method, um, in that it's, it's, it's not empirical, it's conceptual, and thus the burden of proof is on, on your side in order to really give evidence that, that, or that this is going to happen, and thus we should make this, this big change.

I mean, if you zoom out to any time period longer than 100 years, it's not clear which trend is the quote-unquote normal one to extrapolate. What does normality look like on a 100-year or a 1,000-year Like, have you seen how weird the last couple millennia have been, right, from the perspective of the universe? Like, we are living in something very crazy right now.

And so how does that react to my argument? Or to the argument I'm making?

So in the case of your argument, okay, well, I mean, so I guess there's an implicit assumption that, like, when you talk about, like, the default normal, the base case, right, the null hypothesis, in your mind, it's probably the null hypothesis to be like, I bet that life will still be good because it's been good for, like, the last 50 years that I know about, right? Like, is that kind of your null case?

The last few hundred years around any new technology, there's this default state to panic, to worry, to say it's going to lead to the end of times, to try to resist it. And it's in fact often been the resistance of it that has led to difficulties as opposed to the embracing of it. And the embracing of it has led to a lot of these great things. And so why would that be different? Yeah.

So the first thing I want to do is just quibble with your reference class, because if you ask, there's 2 reference classes that I see. Actually, 3. Okay. Reference class number 1, game over buttons, right? We created a nuke. There's a single button somebody can press where 100 million people drop dead. Don't you think that somebody's going to create a button where 100 billion people drop dead? Doesn't that seem likely?

That seems possible. It's unclear that people accept that AI is a game over button in the same way that nukes are.

Yeah, yeah. But we're just playing reference class tennis. You say a reference class that you like. I say a reference class that I like. Right. And you said techno-optimism, technological progress in the course of human history is a reference class that you like. Okay, here's another reference class for you, extinctions. There's been 5 great extinctions. The Late Permian extinction took out 96% of species on Earth. Don't you think there's going to be another extinction?

It seems likely that at some point there will be another extinction. What is sort of the timeframe between extinctions, or when was the— when do you— Explain more about it.

A few hundred million years. Yeah, it's like a couple hundred million years between extinctions. So yeah, okay, maybe we have some time, but the point stands is that there's going to be a big extinction at some point. And by the way, you have to look at the anthropic principle, right? Why was it 96% instead of 100%? You have to figure that on some planets it was probably 100%, but we're just not there.

Interesting. Yeah, why was it 96% and not 100%?

Oh, I mean, so the anthropic principle just says that when we're looking back over history, it has to be a history that lets us exist.

Right. Yes. Yes. Got it. So—

It's like what I'm saying is nature is trying to tell us something. You think nature is trying to tell us that technology is always good because you have one reference class. I'm telling you to look at what else nature is telling us.

Right. Right. What do you think are the—

I have one more quick reference class for you.

Please.

So pushing the frontier of intelligence. So first there was no intelligence, then there was natural selection, right? The process of evolution of life. That's a type of consequentialist optimization, as we discussed before. And that was the smartest thing around. And then humans came up and we became smarter than all the other species. And so we are able to transform the world. Like if you say, hey, name a species, I'll make it extinct. We have a very good shot of being able to do that. No species has ever been able to do that before, right? So my reference class is, hey, when we increase the frontier of intelligence once again, don't you think that the resulting intelligence is going to have the power to extinct the previous one? So there's another reference class for you.

Some people say that the best-case scenario for humans might be that we're house cats. Some people might say that we're kind of white blood cells or kind of, you know, of the AI, like we're fused into it in some way. Do you think both of those are founded, or what do you think there?

So the house cat one, I feel like nobody like truly buys into the house cat one, right? It's just like some clever thing they thought of. But like, I think we all feel in our gut that that is like a scary situation. It's like, okay, you're somebody's house cat. Sam Harris actually talked about it the other day. He's like, dogs have had a good run, but if there was ever the slightest discrepancy between a dog's value and a human value, the dogs are gone. His example was like, what if we just discovered that there's this disease much worse than COVID and just having any dog around is going to transmit it very strongly? So you just really don't want a dog around. Next day, no more dogs.

Yeah. And what about the white blood cell or part of the immune system or Or does that seem unfeasible?

The white blood cell, what's the analogy? How is that like AIs controlling humans?

No, it's like humans are white blood cells of AI or something. Like we are baked into—

Oh, is it just like the general idea of like, let's merge into the AIs?

Or a friend says that's what's likely to happen if we create it, that we are part of it.

I mean, white blood cells, their goal is to attack intruders, right?

Yeah, I wish I knew more about the analogy too.

I can talk about nearby analogies, right? So the idea that we can merge, to me, doesn't make sense. I mean, a merge, you could— I mean, the holy grail would be to augment humans, to build a heaven where we can extrapolate our current consciousness to a much bigger, better consciousness, right? If that's what you mean by merge, then that's also the same thing that I think is the goal, right? That's basically the transhumanist vision. But if by merge, people think it's as easy as like, oh, no problem, we'll just smash them together, there's just a natural way to combine them, Then I have bad news because humans and AIs don't naturally combine. People love to use the example of chess where they're like, once computers started beating humans at chess, the team of the computer and the human, they called it the Cyclops, I think, right? Or not Cyclops, the Centaur, right? The human plus the computer hybrid that was beating both humans and computers. Yeah, they're talking about a period that lasted like a decade or two. We're past the Centaur period. It's just computers dominating now. Right? And that's just what tends to happen. Like, we're not going to help the AI.

What do you think are the strongest counterarguments to what you've presented today? You've debated a lot of people, you've talked to a lot of very smart people who disagree. What do you think are the most cogent arguments in favor of the opposite position that you hold?

Probably not something that we've brought up, but basically, like, picking on the details, basically. So, you know, Quentin Pope is somebody who's actually familiar with the details of the doomer argument. You gotta— I gotta recognize him for at least, you know, not arguing against like the obvious basics, but rather arguing against like, well, how do you know that the AI is going to be goal-oriented? What if it just kind of like finds shallow patterns forever? That's more of the Quinton Pope style of arguing. And like I said, I gotta give it credit for being more nuanced. The funny thing about Quinton Pope though is that a lot of his basic assumptions are contradicted even by the AI labs. Like the AI labs are saying like, oh yeah, AI is going to be able to be a CEO. It's going to be like very goal-oriented, right? So it's kind of funny that his He's actually kind of out of the mainstream on his views, but he's worth a look. And then you can also, you can nitpick details like you can say, hey, the LLM paradigm will last such a long time. I'm trying to think what other nitpicks I found kind of convincing. Oh, you can nitpick the idea of like, I don't think the AI is going to be uncontrollable. I just think there's going to be, it's a human-controlled AI is already going to extinct us. And there's going to be like— 50 years where it's all human-controlled AIs, and those are going to cause their own chaos. Like, I'm open to that idea that it'll take a while before it goes fully uncontrollable.

Right. And the argument around it won't become goal-oriented, it will just become sort of more shallow, the Quinton argument, do you think there's a chance of that?

I mean, I feel very strongly that it's going to be goal-oriented, right? Like, so I'm really not seeing what— I'm not buying what Quinton is selling here, right? Because it's just like, I mean, we're already seeing You can ask GPT to play chess. You can ask Sora to simulate a video game. I mean, these structures are very natural. The idea of optimizing a sequence of actions to hit a goal. A good analogy is like Turing machines computation. If you look at all the different— imagine the year is 1970 and I show you the video game Pong and you look at the circuit board, you see it's a circuit board to play Pong, right? And you look at Breakout, you look at Pac-Man, it's a circuit board to play Pac-Man. These weren't running on operating systems, right? These weren't computer chips yet. They were dedicated circuits. But imagine looking at all these circuit boards and asking me like, hey, Liron, do you think there's just going to be a computer architecture, right, where you put in like a punch card and then it just like does what the punch card says? And I'd be like, hell yeah. Yeah. Haven't you ever read like Turing's theorem? Like universal computation is a thing. So this is what's happening with AI right now is I'm telling you that there is a universal structure. You can call it goal completeness, where all of these problems, if you want to solve a problem, it helps to have a goal. Having goals is a really good way to solve problems. That's an Eliezer Yudkowsky quote, right? And evolution even knows that. Like, why are humans generally intelligent? Why did evolution— what was the training of evolution that let humans take a rocket to the moon and walk around on the moon, you know, with no atmosphere, right? With no understanding of orbital mechanics. Like, how did evolution produce humans that can walk on the moon? How do you explain that?

What is sort of the explanation that relates to AI?

Yeah, the related explanation is evolution realized that if it wanted to build an organism that's solving all these different problems in the niche of humans on Earth, then just go ahead and build an agent that has goals. Evolution stumbled on that, and evolution's dumb, but it still stumbled on that.

Dumb in what sense?

Evolution compared to the human brain. Evolution is a dumb engineer where all it does is it tries out nearby designs in gene space, right? Plays them out and then sees if the organisms survive, and if so, go over there, right? It's like hill climbing, it's gradient descent. Right? It's a dumb algorithm. What a human engineer can do, a human engineer can actually make logical leaps and be like, oh, that won't work. Don't bother trying that. And a human engineer will be like, oh, scratch all this. Let me actually design 3 different interlocking parts from scratch. So a human engineer is much more powerful than evolution. Evolution just has billions of years and massive parallelism. Those are its only advantages, but it's still dumb. So the fact that it hit upon a goal-oriented brain architecture just to help us survive and reproduce on Earth, and now we're going to the moon, that is incredibly strong evidence, right, that agents are going— that AIs are going to converge to be goal-oriented. Is—

I am surprised that if I took your view full on, wouldn't I be anti-capitalist in the sense of corporations are a kind of superintelligence that are kind of, you know, can't help but optimise to advance AI, even if every individual thinks that they shouldn't, that there's this kind of collector, you know, Moloch, you know, problem that is created by sort of markets?

I mean, I see what you're saying. You're saying, hey, aren't corporations kind of like an unalignable superintelligence? But I think that's taking it a little too far. It's like, look, if everybody was shitting their pants and wanted to stop this, we could at least do something, right? Like, you wouldn't have Walmart being like, seceding from the United States. There's obviously some power that we have if we all become sane about this problem.

Got it. But even if we didn't have capitalism, would we have this problem to begin with? If we didn't have mega corporations, it'd be just so much easier to stop. And would we just not have it?

Well, I mean, if we didn't have mega corporations, we'd be a lot less powerful, right? So we probably wouldn't even have fast GPUs in the first place, right? But if we had a top-down economy, yeah. I mean, if we had a central power that was micromanaging the whole economy and the central power was sane about AI risk, then yeah, that would be great. The central power could be like, all right, everybody Halt this AI stuff. It's time to halt because that's what we really need right now. I mean, nuclear proliferation is the one time when you can see, I mean, it's quite a test case. You can see that it takes everything we have to prevent nuclear proliferation from going out of control. And we're constantly worried about it, right? I mean, Putin is threatening it, right? So this is already straining our ability to coordinate and now we have a harder coordination problem. So like, yeah, it's not good.

And so your dream world is a market economy that accelerates everything except Except nuclear proliferation, AI, and maybe some bio stuff, or maybe not.

Yeah. So I'm very pro-market, very pro-capitalism. And yeah, my dream world is for the government to just step in and be like, look, there are certain things that are illegal because there are coordination problems that you have to mix in a little bit to a market economy. And whenever you can avoid coordination, whenever you can do an end run around it, right? So like with climate change, you could do an end run around having to ban carbon, right? Because you could just say, hey, look how lucrative solar power is, right? Or like, look how— let's make nuclear power lucrative, and you kind of do an end run around the need for top-down control, which is great. But in the case of AI, I don't think there's an end run. I think we just better pause or we're dead.

We'll close on that topic, but I do want to, if we could spend a couple minutes on effective altruism. You believe that effective altruism has gotten unfair sort of criticism or has been kind of unfairly dismissed or hated. What is your perspective on why effective altruism has been Yeah.

So everybody loves to hate on SBF, and then I guess the board of OpenAI was supposed to be an EA thing, but really it wasn't anyway. So SBF is kind of like the one example, right, of like somebody who calls himself an effective altruist and then was like totally immoral. As an effective altruist myself, I do not endorse Sam's playbook, not just because it failed, but even at the time, you know, deontological morality, like don't go around screwing people because in your mind it's going to pay off. Like, no, that's not something that I do. I personally don't know a single person who's like, yeah, Sam was totally right, or breaking deontological rules is great. But anyway, the reason I'm so passionate about EA is because the core principle of it is still a huge leverage point, which is the core principle is if you care about helping people at all, if you care about charity to some degree, and then you take your money and you want to give it to a charity or you want to give it to anybody who's going to help people, most people who have that thought, what they do is basically throw most of their money in a dumpster. Like, they're not actually helping people, right? If you look at what they're purchasing, they're purchasing like a lot of overhead, right? And like, okay, one puppy gets helped, great, but it was like super expensive. And it's like, you could have taken that same amount of money and helped 1.5 puppies. No, I'm just kidding. See, if I— the pitch that I just said, it's kind of weak. Yeah, 1.5 versus 1. But here's the thing, it's not 1.5, it's like 100. Okay, the number is really big. It's like you could have helped 100 puppies versus 1. Okay, we're not talking about a small delta here. So there's a lot of alpha in effective altruism. That's why it has the word effective in it, because it's 100 times— it's 10 to 100 times more effective than the default people do who are NEAs. Which, by the way, if you look at effective accelerationism, why do they have the word effective? Because they're just being dicks to effective altruism. Like, we actually had a reason to call it effective. Like, they're just mocking us.

And you don't think that effective altruism has sort of a type of utilitarianism streak baked into it? it that can be run amok in certain ways or be applied, kind of overextend the bounds, or you think there's little connection between effective altruism and utilitarianism?

I do think that if you— there's a lot of ways to shoot yourself in the foot when you're really passionate about utilitarianism. Absolutely. But we also have to make the distinction of like, look, did your art fail you or did you fail your art? You know what I'm saying? People are like, if you're utilitarianism and you're like, this person should be murdered, I feel like they're bad to society. It's like, hold on a second. You think you can murder this person and live in society and that's going to work out well you. You know, like people are, people who are saying, hey, you know, if you're long-term nice, you tend to like be richer and more successful and more powerful. Right. So there's a lot of things about life as a human that if you actually want to succeed, you can't do naive utilitarianism. Maybe if you're a super intelligent AI, if you see, if you look around the world and you don't see people, you just see billiard balls in their brain, right? Because you can model them on an atom by atom level and you just know where to send their atoms into the future. Maybe at that point you can just be a full-on utilitarianism, utilitarian, and you can optimize the future. But if you're just a human, A bounded human, and you're thinking about murdering people for short-term gain, that's not even going to maximize your utility.

We're at time, so we have to wrap it. This has been a great discussion, Mehran. Thanks so much for coming on the podcast.

All right, Erik, my pleasure, man. Good luck with Turpentine. You're killing it.

Thank you, sir.

Upstream with Erik Torenberg is a show from Turpentine, the podcast network behind Moment of Zen and Cognitive Revolution. If you liked the episode, Please leave a review in the Apple Store. Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution It's Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik@turpentine.co. That's E-R-I-K at turpentine.co.