The Path to Utopia, with Nick Bostrom – from Clearer Thinking with Spencer Greenberg

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host Eric Thornberg. Hello and welcome back to The Cognitive Revolution. Today, I'm pleased to share a special cross-post episode from the excellent podcast, Clearer Thinking with Spencer Greenberg. Spencer is a mathematician, entrepreneur, and researcher who explores ideas that matter through in-depth conversations with leading thinkers. And in this episode, Spencer speaks with philosopher Nick Bostrom. Bostrom is perhaps best known for his book, Superintelligence, which took on the challenge of analyzing the potential dangers associated with smarter-than-human AIs, while the very possibility still seemed remote. But this conversation is actually mostly about his latest book, Deep Utopia: Life and Meaning in a Solved World. So why is it so difficult for us to envision a truly desirable future? Bostrom argues that our current lives are structured by what he calls an exoskeleton of instrumental necessities. That is, the things that we need to do to survive and thrive are the things that we've evolved to value. In a world where advanced AI could handle all these tasks, things like purpose, achievement, and the satisfaction of overcoming challenges could all potentially disappear, leaving us with a profound new challenge: how do we find purpose and meaning in our lives when there's nothing that we truly have to do? As you'll hear, Bostrom faces this challenge head-on, attempting to sketch the outlines of a truly desirable technology utopia, touching on concepts including quiet or subtle values, Including things like aesthetic beauty, acts of creativity, and commitments to honoring ancestors or upholding traditions. These are currently overshadowed by more pressing concerns, but perhaps could come to the foreground in a technology-enabled future. They also discuss artificial purpose, or the creation of arbitrary goals and constraints that could give our lives structure, not unlike the sense of purpose that some find in video games today. They also assess the possible need for global coordination to achieve such a future, as well as the associated risks. of pervasive surveillance. I've often said that the scarcest resource is a positive vision for the future. And this book is a great prompt for all of us to begin to think more deeply about how we might find continued meaning in our lives in a world where today's most pressing problems are solved. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends or post a review on Apple Podcasts or Spotify. And we always value your feedback and your topic or guest suggestions. You can submit those either via our website, cognitiverevolution.ai, Or by DMing me on your favorite social network. Now, I hope you enjoyed this thought-provoking conversation about the potential evolution of the very meaning of human flourishing between Spencer Greenberg and Nick Bostrom from Clearer Thinking with Spencer Greenberg.

Hello and welcome to Clearer Thinking with Spencer Greenberg. The podcast about ideas that matter. I'm Josh Cassel, the producer of the podcast, and I'm so glad you joined us today. In this episode, Spencer speaks with Nick Bostrom about the potential of AI-driven utopia and the search for meaning in a post-instrumental world.

Nick, welcome.

Hey, Spencer.

I don't know about you, but I found that there's a lot more interesting, vivid descriptions of dystopias out there than of utopias. Do you agree with that?

Yes, I think that's true. I wonder what that says about us humans, but it does seem to be a pattern. Like, most people could probably rattle off a bunch of different dystopias, you know, Brave New World, 1984, Handmaid's Tale. But the average person would have trouble, I think, naming even a single utopia.

Yeah, it's interesting because it seems important to figure out where we want society to go, not simply what we want to avoid.

Yeah. And it gets worse. Once you actually look at the utopias that have been attempted, by and large, you wouldn't actually want to live in any of them. They often seem too pat or too sort of— they become a little bit sort of like a plastic toy or something, like cloying often. And so it's actually hard to think of a true utopia. What's easy, you could imagine if you take the current world and you imagine some small improvement, that's easy enough. The current world, except without childhood leukemia, seems just better. But once you start to add a whole bunch of different improvements, then eventually you get to the point where it no longer seems attractive. It's almost like plastic surgery, right? So you could imagine if somebody has some big wart or something on their face, probably you improve their performance by removing the wart. But if you sort of just keep going with more and more—

Improvements.

supposed improvements, you kind of enter Michael Jackson territory.

Yeah.

One thing I've observed that's relevant to that is that people seem to agree much more on what we don't want than on what we do want. Everyone can agree we want less illness, we want less poverty, we want people to have fewer issues when they're growing up, etc. But then you say, okay, what do we want society to be like? Do we want everyone to be sitting around and having intense pleasure all the time? Do we want people to still be working hard to achieve their goals? Do we want lots of little microsocieties that are completely different? That's when people seem to really start to disagree.

Yeah, but it's not just that we disagree with each other. It can be hard even as an individual if you try to conceive of a utopia that would be super attractive, even just to you yourself. That is already a pretty difficult challenge. And I think more so if we consider not just a few rearrangements in the current condition, like a little bit more money and a little bit better medicine, but more profound transformations of the human condition, such as we might attain technological maturity.

Why do you think it's difficult for people to think of utopias that even they themselves would think are ideal? Is it because people have multiple values, and as you try to build a utopia, those values come into conflict, or some other reason?

I suppose our self-conception and our goals that we have, and the kind of applied values that we pursue, are conditioned on our current location. So we have various projects going on, things we're trying to achieve, but all of that is kind of based on where we are now. And so if you imagine a very different situation where none of that applies, it might seem that we are a little bit at sea. and it doesn't relate to us, and it seems almost like we are contemplating somebody else's life. So that could be something, that we are anchored in our immediate surroundings and our current predicament, and anything that takes us too far out of that is kind of also losing ourselves and our life in some sense.

One thing you talk about in your book Deep Utopia is that the regular world imposes these constraints on us, and gives us these things to strive for. And then suddenly, if we imagine a world where production levels are so high that we really don't need to strive for anything, we kind of lack those constraints and it kind of leaves us out at sea.

Yeah, I have this metaphor of the insect with the exoskeleton that holds these squishy bits together and gives it shape. And analogously, the various instrumental necessities that we face in our lives and that all humans so far ever on this planet have faced. It's always been part of our environment that there are a whole bunch of things we need to do. You need to make money to pay the rent. You need to brush your teeth to maintain a healthy mouth. You need to do this, that, and the other. And that's like our lives are sort of lived inside these constraints. And so just as the insect has an exoskeleton, our souls have has a kind of exoskeleton of these instrumental necessities. And if we one day attain a post-instrumental condition, as I call it, then there is a question of what happens to our souls. Do we just become contented pleasure blobs, or would there still be something that gives structure to our psyches and our lives?

It seems that if you go back a few hundred years and you ask people what utopia looks like, there's going to be a lot of focus on just material abundance, right? Well, wouldn't it be wonderful if we just had enough food to eat all the time and great shelter, comfortable shelter, and it was always the right temperature and so on? And yet, to kind of a shocking degree, many people in the world have those things today. Not everyone, of course. There's still people living in poverty, but many people still live this way, and yet it still feels very far from utopia.

Yeah, it seems like pretty much every different culture has some notion of this Land of Plenty, the Land of Cocaine. There are different names for it, but I think if you imagine some medieval peasant living under great material deprivation with backbreaking labor from morn to dusk, imagining a condition in which there was plenty of food and you could rest as much as you wanted, and it's like a continual feasting. would already be enough to kind of just give you a wow, that's like a fantasy. But now I think that wouldn't really count for most people. We already have fridges stuffed full of food. And although we are not yet liberated from all economic work, some people are. And even the ones who are not are now having holidays and weekends and stuff. We're almost struggling with the opposite end of this, like the kind of boredom or lethargy or lack of purpose that can come from having it too easy. And that creates a different source of misery and need.

I'm not certain this story is true, but a friend of mine went and visited a tribe, and through a translator, she was asking them about their afterlife. And the way that the translator explained their afterlife was that basically it's just like normal life, except the cows give more milk. And I thought that was so surprising. Right. It's like, it's just, oh, wait, it's just things are just a little bit more comfortable.

Yeah. Now, if you did end up in this afterlife with a lot of milk, and then you might dream in that life, what would be after that life? And then you might think, well, the trees also give a bunch of, a lot more fruit, and the fruits are sweeter. And then you could repeat this, and then, oh, maybe your bed is even softer and more comfortable. But if you iterate far enough in that direction, then you do, I think, eventually get to this notion. I call it a solved world and even a plastic world. These are 2 different concepts I introduce in this book. Deep Utopia, which describe, characterize a much more radically profound and in some sense improved condition that we might attain, I think perhaps not long after the machine intelligence transition if things go well. And I think this really forces us to confront some pretty fundamental questions in philosophy, like what ultimately gives value to life, what gives meaning to life, and other values that— and some values are easy to just clearly, we could have a lot more of them in this condition, but there are also some other values that seem to be at risk of being undermined in such a solved world.

Yeah, you mentioned this question of what do we do when we have abundance? And an interesting place to start there is to look at what we have done as we've made more and more production. In theory, people could work a lot less because, you know, we've got all of this food, we've got all of this comfort. Why are we working so hard? So what historically has happened as production has increased?

We do work a bit less. So we've taken some of our increased productivity out as leisure. Some of that is we have longer childhoods and education, but we also have weekends off and we work shorter hours and maybe work less hard than the peasant did. And then longer sick leave and maternity leave and paternity leave, and then longer retirements. But we've only taken out some of our increased productivity for leisure. Like most, we have taken out as increased consumption. John Maynard Keynes, the famous economist, wrote this essay—it's like almost 100 years ago now—where he at the time predicted a century hence that. people would work way less because he kind of extrapolated then recent increases in productivity and thought if that continued another 100 years, we would be like 4 to 8 times richer and then we would scarcely need to work. And what would we be doing all day long? Now, as it turns out, we do work maybe 25% less or something, but overall, Greed has triumphed over sloth, and so we just spend more. A lot of that is on positional goods. You buy more and more expensive luxuries to try to one-up the other. The more— the fancier the clothes and the cars that your neighbors have, the more you have to spend to keep up or overtake them. There is an element of zero-sumness to our consumption rat race. And so that's where a lot of our increased economic wealth has gone, I think, in addition to sort of absolute improvements and also some increases in leisure.

It's funny because with positional goods, you could spend an unlimited amount of money on them, right? Like, because it's all just relative.

So you could have like the billionaire with his like mega yacht, and then there's another billionaire. Both of them want to have the biggest yacht in the world, but it's impossible no matter how much economic growth there is for them both to have their preference satisfied. So if you have a 200-meter-long yacht and somebody else builds one that is 210 meters long, then you need to upgrade yours to get 20 meters more, and that could keep going. So in a solved world, it's not defined by the idea that all preferences can be satisfied or that people wouldn't want more money. I'm sure some people at least have unlimited appetites, but that doesn't imply that people would still be working because although there might be a desire for more, it doesn't mean that you could get more by working yourself. If AIs can do everything better and cheaper and more efficiently than humans can, that would just— not to be any demand for human labor. Now, I think there are some exceptions to that, but as a broad pattern, I think you could have something approximating full unemployment if AI succeeds.

How linked are these kinds of arguments to AI in particular? Do you see a way that one day humanity could get to a utopia even without superintelligent AI?

In principle, yes, you could get I think most, or perhaps almost all, of the same affordances without advanced AI. You could imagine, for example, there are a lot of intellectual tasks that need to be done in the economy, right? And that humans now do. Maybe given unlimited time, you could develop software that for each of those tasks, you would figure out how to automate it using non-AI techniques. And I think realistically, the way, if we do get there, that we will get there will be through AI. That just seems a lot easier to build one AI that can learn to do all these different tasks than to sort of make a specific software program for every particular task.

In your book, Superintelligence, you analyze risks from AI in great depth. But I think for this conversation, let's assume that AI doesn't go horribly wrong, right? Like, it doesn't kill all humanity. Let's say it's still under the control of humans to a reasonable degree. Even with that assumption, it seems like people may be very worried that, okay, if AIs start producing as well as humans at almost every task, does that leave anything for humans to do? And what kind of society does that look like ultimately?

Yeah, I'm ultimately optimistic about the possibility of having a really desirable and wonderful type of existence in a post-instrumental world. But I do think it does go through some significant challenges, like the book isn't trying to sell utopia, or it's not like an argument for, oh, here is how great it would be. It rather tries to think through the full implications of what it would mean. to have a solved world. And then there are, at least at first sight, quite disconcerting implications of that for, would it really be attractive to live in this world? It seems some values at least would be potentially undermined here. And I do think ultimately the kind of existence that would make sense here would be significantly different from our familiar human existence. It might need require us to sort of give up some values or reconsider them in a fairly fundamental way.

So what are some of those values we might have to reconsider?

Well, take purpose. So some people think that it's important for human beings to have purpose in their life and that the life is diminished if it lacks purpose. Now, if you think of purpose as something That is worthwhile, that you have to exert effort over some period of time to try to achieve, and that draws on your different, maybe, capabilities. It's, at least at first sight, not clear that that would still exist in utopia, inasmuch as, at least for a very wide range of different things you might want to achieve, there would be shortcuts. It would be easier just to ask your AI system To do them. And so future lives might have less purpose and meaning. Like, it might seem that there is nothing that we are needed for. A lot of people today take a kind of pride in being a breadwinner or in making a positive contribution to society at large, perhaps, or at least on a smaller scale. Like, they think they benefit their family in some way by being around. Or at the very least, themselves. There are various projects you can undertake to try to educate yourself or get fit, or you redecorate your home to get a nicer home. All of these little things that we are engaged in, little projects, and that's a kind of big constituent of what we're up to, of our lives. If all that went away and it was all just on tap, it might seem to create a kind of purposeless, meaningless existence.

A friend of mine said to me fairly recently that she still prefers talking to me about her problems than she does talking to an AI, but that's not true of all of her friends. So there's some friends she'd rather talk to the AI. And there was something a little scary about her saying that. It's like, well, okay, so maybe next year she'll prefer talking to the AI than talking to me, right? And then you start to think, well, yeah, if you really push this thought experiment forward, And you say, well, what if really there's nothing you can do better than the AI? Right. And what does that mean for all the things you care about trying to do in your life?

Yeah. So that would be one instance. Like, if other people right now need you in different ways, whether it's for conversation or support, or they are like your children or relatives or friends. And if AI has just become better as social companions, that would kind of remove one way in which we can be practically useful and of help and value to other people. And then if you generalize that, then you start to kind of get a sense for this challenge. And so the book doesn't try to shy away from that. It kind of dives straight into this and tries to take that on in its full force.

So how do you think about starting to have a great utopia even if It's going to fundamentally be different than what we are used to trying to do in our lives.

Well, if we start from an empty slate, then one can consider what values we could add there in utopia. I mean, I should first say, although the book doesn't dwell much on this, but it is a super important thing in the actual all-things-considered situation, which is just the opportunity to get rid of a whole bunch of negatives. that currently plague the human condition and indeed the animal condition as well. I think that already alone, just getting rid of all the bad stuff, would possibly be reason enough. But let's set that aside and just consider what positive values could exist in utopia. First, we have pleasure. Now, I think it is very easy to dismiss that as, yeah, sure, they could have some super drug and kind of be blissed-out junkies, or maybe maybe direct forms of pain manipulation and to kind of sniff at that. And I think actually that alone is a much more serious and possibly attractive proposition than a lot of people would give it credit for. And I think the people who are kind of very down on this form of hedonism might easily change their mind if they actually got to sample some of this pleasure that would be in the offing at technological maturity. Hey, we'll continue our interview in a moment after a word from our sponsors.

Whether you're a marketing manager, a product engineer, a CEO, a researcher, or a social scientist, you sometimes need to know what lots of people think about a thing, or you might want to have people enroll in a study or experiment. But recruiting study participants can be time-consuming, error-prone, and expensive. Well, good news. Positly is here to help. Positly addresses the common pain points that researchers encounter when recruiting study participants. It aims to solve common research problems and dramatically improve the speed, quality, and affordability of human subject research. With Positly, researchers, marketers, and product developers are empowered to produce better results by accessing high-quality participants through an easy-to-use web interface Making it easy to run surveys on thousands of people in mere hours. And it can now be used to recruit people in over 100 different countries. To learn more and to give your research project superpowers, visit positly.com. That's P-O-S-I-T-L-Y.com.

People seem to find pleasure more palatable when it's linked to kind of bigger things like There are a lot of people who would say, well, if all life was was sort of the most base pleasure, right? Like the pleasure you might get from doing heroin or something like that, then maybe that feels worse. But if, let's say, it was like viewing beautiful art, but it was so beautiful that you were in this incredible awe state, maybe that would be more appealing.

Yeah. And so this is one thing that we can add that I think we should add, but it is It is, I think, oh yeah, like, just worth dwelling very briefly, at least, on just the raw pleasure itself. It's kind of intellectually uninteresting and trite, and like, but it might ultimately be the most important single thing, although I do think there are additional elements that we can add. And so, you could combine the pleasure as in positive hedonic tone with experience texture. So, this would, for example, be the appreciation of beauty, as you mentioned. And you could kind of take pleasure in contemplating beauty or in understanding truths or in admiring goodness. And so, you don't just have the pleasure, but you also have some other, maybe more complex and rich mental content attached to the pleasure. And Already, that seems a lot, perhaps, more appealing, and indeed, some traditional conceptions of heaven consider it as being a kind of state of contemplating perfect goodness in the form of God, and then experiencing love and happiness as a result of that. But the experience is a kind of conjunct of the thing being contemplated, and then a kind of emotional response to that. And both of these things clearly would be possible in a solved world to extremely high degrees, extremely high degrees of pleasure, and also extremely clear and strong and sophisticated or well-targeted forms of contemplation or other mental content. So those two, we can kind of put those in the bank. You have pleasure and you have experience texture. And we can then ask, can we add further elements to these things? So one thing we don't yet have, right, is any type of activity or pursuit or purpose. Like, at the moment, we have kind of blissful experiences of contemplating various things. But you could then think, why wouldn't we also be able to engage in various types of activities if we think various forms of activities are intrinsically valuable, you could create artificial purpose. So you— we already do this today. Like, when we play games, we sort of set ourselves some goal, and then we try to achieve it. Some arbitrary goal. Like, you want to try to get this golf ball into a sequence of 18 holes using only this little Inconvenient implement, the golf club. So these are like random goals you just set yourself, and then once you have these goals, then you can engage in the activity of golf. So similarly, in utopia, people could set themselves arbitrary goals, and that would then enable the activity of pursuing those goals. Importantly here, you would include in the goal constraints on how you're supposed to achieve it. So just as if you set yourself the goal of doing well on the golf course, it's kind of part of the goal that you're only supposed to use a golf club to propel the ball, as opposed to picking it up with your hand and placing it sequentially in each hole. And so similarly, in utopia, there would be all these technological shortcuts you could use to achieve a particular outcome. And so you would have to bake into the goal the idea of achieving the outcome only using a certain limited set of permissible means. If what you want is then an activity that you yourself has to engage in.

You can't just tell the AI to do the thing for you. That defeats the purpose.

Yeah, but you just make that part of this arbitrary goal you adopt. And so then we can add one more element. So we have pleasure, right? We have the experience texture, and now we also have artificial purpose that allows us to engage in various forms of activities that could be intrinsically valuable. And already, I think it starts to seem a lot richer, like a future of contemplating beauty and playing games while greatly enjoying ourselves doing these things seems maybe at least more attractive to many than being this kind of blissed-out pleasure blob, like kind of a junkie enjoying some super drugs. sprawled out on a kind of flea-infested mattress in a dark room, but like where there's nothing else going on except the pleasure itself. Now we already have something more, and it's starting to look better, I think. And I think there we can even add a little bit more to this.

Right, so what element would you add next?

Well, some people might think, although nice, you could have these artificial purposes, it's still not quite the same as real purposes. Like, you don't really need to have the ball go into each of these 18 holes, or to defeat the boss monsters in this computer game. Like, they are kind of fake purposes in some sense, like, as opposed to now, there's, like, a whole bunch of stuff in the world that actually needs doing. And so, you might think it can add value to life if it has, Realer purposes or more natural purposes, purposes that exist not just because we arbitrarily just ourselves made up some goal. And now, could we have such things in utopia? Well, I think yes, there would be some opportunities for this. The perhaps easiest way to see it is if you imagine you have 2 people, A and B, and— A and B. Let's suppose that A just happens to have a preference that B's preferences be satisfied, like you care about the other person. And then if now B happens to have a preference that you do a certain thing on your own, then now you have a real purpose. If you want to actually achieve your goal of satisfying B's preferences, the only way you can do that is by doing this thing yourself that B wants you to do.

So essentially because we have preferences that other people do specific things, not just that they cause those things to occur by asking their AI agent to do it.

Yeah. So you could even give a purpose to a friend by sort of adopting this preference or goal or setting yourself up in such a way that you will be more happy if they do a certain thing. Then once you've done that, then they have an actual real purpose to do that thing if they care about you and your preferences. In this particular reductionist case with A and B setting, it's a little bit hokey perhaps, but I think subtler versions of this are actually quite common in that we have various shared cultural commitments and commitments to various traditions, for example, that we might just wants to uphold, and those traditions might call for us ourselves to engage in various forms of practice. Yeah. And the tradition wouldn't count as having been continued or honored if what instead we did was create a bunch of robots who went around performing the rituals or whatever. In order for the tradition successfully to have been continued, for certain types of tradition, may require our own involvement.

Right. An example of this, I think, that can be poignant here is imagine you're writing a speech for someone's wedding. If you were to just ask an AI to write it for you and put no effort into it and the AI spits it out, even if it's like a really good speech, there's a way in which it feels like you didn't satisfy that obligation. And whoever asked you to speak at their wedding might be disappointed that you didn't actually put your own thought into it.

Yeah. And I think these are fairly ubiquitous, actually. And more broadly, I suspect that there is a whole bunch of aesthetic reasons for doing various things that would come into view in this condition of a solved world. And if you think about it, the aesthetic significance of something often depends on who did it and why they did it and the means by which they achieved it, as opposed to just kind of the particular artifact that results. And so our lives might become more like artworks, and to achieve a particular expressive content of those artworks, it would in many cases call upon us to do things on our own steam. And so more broadly, I have this notion of kind of quiet values or subtle values. So I think there might exist a whole bunch of these that are more or less obscure to us currently. And so just as during the day you go outside, you don't see any stars, right? But it's not because they're not there. It's because the sun is there and it's so much brighter. But similarly, in our current lives, there are these pressing instrumental needs, horrors going on in the world, things we have to do in our own lives, or various kinds of catastrophes happen, that these are the louder values. So we need to fight injustice and prevent pain, and we need to, like, all these things that kind of fill our conscious minds. And so, but so we don't see what I think is there, which is a constellation of these subtler values. But if all these kind of pressing, urgent moral needs one day were all taken care of, then I think this richer canopy of subtler values would come into view or potentially come into view. And just as during nighttime, our pupils expand to take in more light, so in this future, it would be appropriate for us to become attuned to place more weight on these subtler reasons for doing stuff, including aesthetic reasons broadly construed. And subtler things like honoring your ancestors or upholding traditions or achieving various kinds of aesthetically beautiful shapes in your life and in the way you relate to other people could just constitute a larger portion of reasons for doing stuff, but that they might be really important. It's just that their importance is up-calibrated because the things that are currently important would no longer be there. So it would make sense to care more about these subtler things.

It seems that a lot of meaning we get in practice comes from our relationships with other people, and that even if we're in a situation where there's nothing that we can really add because there's agents that can do it better than us, we could still have deep relationships. It could be very fulfilling. How does that kind of work into your vision of utopia?

Yeah, I think a lot of these purposes that could remain in this old world would broadly arise from sociocultural entanglements with other people, not just other individual people, but also other sort of cultural phenomena and commitments and our participation in traditions and communities of various sorts. I think that to the extent that there are still kind of natural purposes remaining, I think a lot of them would come from that source. It is not completely obvious how people will choose in this respect. So we'd already like to anchor it a little bit more to the here and now. So we have these increasingly capable social companion bots that are being or will be created in the near-term future, right? We already have the chatbots, but you will have more multimodal, with maybe visual avatars and voice, and it will all become— I mean, you can already see the beginnings of this, and I imagine some of these might become very compelling to some people. One might wonder whether— okay, so it might just not have time to play out if AI timelines are really fast, but if you imagine kind of AI frozen in its current state or a couple of years from now, and we have that level of technology, would people start to spend more and more of their time interacting with these social artificial intelligences as opposed to real humans? And would that be good or bad? So the first answer that pops into most people's mind would— it seems to be bad. It's much better to spend time with real people. I wonder though whether that's one of these generational things. So, us old fogies, sort of like, oh no, the only real— you gotta spend time with real people. That's how we did it. But, if a new generation grows up with this kind of stuff, maybe they just think that's like we have these hangups. And it can be really hard in situations like that to form some— I mean, it's easy to form an opinion, but to form an opinion that reflects more than just your own idiosyncratic upbringing and personality, and that actually goes down more to bedrock of value. That's like a really hard evaluation to do.

I don't know if others will find this compelling, but to me, it matters a lot if those I'm interacting with are conscious. In other words, that they're actually having internal experiences. For example, they're feeling pleasure and pain and so on. If I was talking to AIs that were not conscious, to me, that would seem to sap a lot of the meaning out of it.

I mean, it might be that after a while, if you were in the habit, you would kind of forget about this question. Just as some people think consciousness don't exist, there are eliminativists about consciousness, right? Like, some philosophical view thinks that it's like a confused concept and actually nobody is conscious. It's like we should just get rid of the the very notion because it's like a philosophical confusion. I think they still then get on with their lives pretty much in the normal way. So I'd imagine that eventually people would just settle into something that was not very tightly coupled to some abstract philosophical belief. Of course, it's also possible that some of these digital companions are conscious or will be conscious. You could have artificial Persons that are conscious. It's just that they have maybe been designed to be more optimal as social interaction partners and therefore kind of more compelling, just as some people are more kind of charming or compelling as social interaction. Some are just annoying and tiresome and full of themselves and puffing empty air and irksome, and others are really wonderful human beings that you want to spend time with. Similarly, some of these AI companion bots might just become much better in the same ways that some humans are better at being friends. This is maybe a good time again to remind the listener that I'm not advocating specifically for this. I'm not trying to sell a particular vision here. I'm just trying to look at what it actually is and take it on. I obviously also understand the aspect of this that seem repellent, like the idea that we would have these kind of highly optimized AI companion bots that we would spend all our times with instead of interacting with human beings, that there is at least an initial kind of yuck reaction to that. But I want to not just stop at that initial yuck reaction, but kind of dwell in the discomfort and then see if one can understand precisely why it is and whether ultimately it makes sense or whether it's just like a kind of prejudice.

So changing tack a little bit, assuming that humanity continues advancing AI, it gets incredibly advanced, and we're able to keep it under control, what do you see as some of the kind of concerns about how it could lead to a utopia versus a dystopia and what we should be thinking about there?

Well, there's a whole bunch of practical difficulties between where we are now and attaining Anything like a solved world. So we have the alignment problem, of course, then kind of various versions of the governance problem, and also the problem of the ethics of digital minds, that we want the future ultimately not just to go well for human beings, but also for other morally considerable beings that we will share the future with, hopefully, including animals and maybe some of these digital minds. whether because they are conscious or they have other attributes that give them various forms and degrees of moral status. And so that's a lot there. The book just brackets all of that in order to actually get to the point where we can ask the question, what then? Because I think at some point somebody needs to— should probably ask that, even though most of our time should be focused on Making sure we actually get there, as opposed to like destroy ourselves beforehand. Hey, we'll continue our interview in a moment after a word from our sponsors.

Science is built on replication. Our confidence that a particular hypothesis is true increases the more times we can conduct experiments and get results that are consistent with the original research. Unfortunately. Psychology and other social science fields have been undergoing a replication crisis for the past several years, meaning that researchers have tried but failed to replicate experimental results from the past few decades. And this is deeply troubling because it calls into question many of the things we thought we knew about how humans work. To help solve this replication crisis in psychology, the team at Clearer Thinking has launched a project called Transparent Replications, that seeks to celebrate high-quality research while also shifting incentives toward more replicable, reliable methods. They accomplish this by conducting rapid replications of recently published psychology and human behavior studies in prominent academic journals, with the aims of celebrating the use of open science best practices, improving reliability, and promoting clarity. Once the Transparent Replications team has completed a replication, they make their results freely available on their website for anyone to read. To read those results and other essays by the team, visit replications.clearerthinking.org.

So the alignment problem is really about getting AIs to do what we want and not do things we don't want. How would you describe the governance problem?

Well, it's a broad category. One aspect of it is making sure that We humans don't use these AI tools, even if we assume they're aligned, for bad purposes. All kinds of other very general-purpose technologies that have been developed have been used both for good and for bad, right? So you have people engaging in warfare, some people oppressing other people, all kinds of mischief. And so with AI, similarly, very powerful and very general tool, there's all kinds of opportunities for misuse, both for conflict and for oppression and for other types of malfeasance. And so broadly speaking, you might think of the governance problem as how to ensure at least that the preponderance of uses are positive. It also interacts with the alignment problem in that you might potentially need various forms of governance regulations and oversight to ensure that the alignment problem gets solved in time and that alignment solutions get implemented. So these kind of interact in various complex ways. It's a little bit arbitrary how you break these out, but I think of the alignment problem as a technical problem that people who are good with math and computer science need to figure out by being clever. And then governance problem is more like a broader political challenge where it's not so much that there is a clever little answer somebody comes up with, but it's more like a continuous effort by many people to try to achieve a more benevolent and cooperative condition in the world. And then the third is, I call it the ethical problem of digital minds, but part of that is philosophical and computer science-y to figure out which minds actually have what kinds of morally relevant attributes, but it then also quickly becomes a challenge for human empathy and ethics and ultimately for governance to ensure that whatever we figure out regarding the question of how these minds ought to be treated actually also gets implemented in practice. I would say that it's not a book that has the structure of, here is a thesis, here is the argument for the thesis, and it's more designed to be an experience to try to put the reader into a position to think for themselves seriously and deeply about these questions with the right kind of attitude, an open-minded form of curiosity and benevolence. Because ultimately, I think somebody somewhere, if things go well, will need to make up their mind about what we actually want about the future. And it's a really hard deliberation. And you would hope that they don't just kind of take out some cached thought or do some sort of off-the-cuff thing or project onto the future some random little feature just being a function of their current level of neurotransmitters, but that it kind of— and I think it's possible that the answers people might give might depend quite a lot on how they come at this problem, the attitude with which they enter this deliberation. And so what the book ultimately, it has a secret purpose, is if some group, whether it's some people in an AI lab or a government or some humanity-wide deliberation process, that it would be good for there to be something to read in preparation for going into such a deliberation. And I'm hoping this book will equip them with certain concepts and put various questions more clearly into focus, but also help prepare a certain kind of attitude of benevolent generosity and open-mindedness and playful contemplation that I'm thinking is likely to make that deliberation go better than some of the alternative attitudes with which one could come at it.

Well, hopefully, if we start approaching utopia, the people involved will read your book and it will stir some important reflection on what we actually want utopia to be like. Yeah. All right, before we wrap up, let's jump into a rapid-fire round. I'll ask you a bunch of questions, try to get your relatively short answer, although they're going to be complex questions. So, first question for you. So you wrote this book, Superintelligence. Since you've written it, a lot has happened in the AI world. We've seen large language models like ChatGPT. We've seen Many different breakthroughs in AI. And I'm wondering, have those developments in the technology changed your view considerably about AI, or do you kind of stick to your guns to what you wrote in Superintelligence?

Yeah, in broad brushstrokes, I think it's held up really well, and I haven't changed my mind, except there's more granularity. We can see more specifics about the particular shape, like the idea that current AI systems are very anthropomorphic, very human-like. With human-like idiosyncrasies is a bit surprising. The idea that to get the best out of one of these large language models, you almost have to give it a little pep talk sometimes, right? Think step by step. This is really important. My job depends on your answer. Like that, you actually get the AI to do better by doing that. That would probably have seemed a bit ridiculous 10 or 20 years ago. That's like anthropomorphizing the AI. Yet that's where we are. Yeah, I'll stop there because you wanted short answers.

So some of what you write about both in Superintelligence and in Deep Utopia involve global coordination, which seems like something that the world struggles with. What do you think are some of the most promising strategies for improving global coordination?

Well, I mean, one, if the world ends up being a singleton, which is like this concept I have of a world that is coordinated at the highest level, Perhaps the most likely way for that to happen is through the AI transition, and then either one actor gets enough power to just impose itself on the world, or maybe post-AI technologies allow for easier ways of coordinating and solving coordination problems. But probably what you meant to ask was more like, what can we push on today to improve global coordination? Unfortunately, it's quite hard to find some really high-leverage things in that space. There are little bits and bobs here and there that one can point to perhaps, but probably if there were an easy way, it would already have been done.

One potential means of global coordination that you've written about is the idea of a single state actor that controls the world globally, maybe monitors everyone and everything to make sure that really dangerous technology isn't used. Obviously, when people hear about that kind of idea, it's kind of terrifying. It sort of sounds like a one-world dictatorship. Do you see ways of preventing dangerous technology that don't involve such close monitoring, or do you see ways of involving such close monitoring that don't come across as so authoritarian?

Yeah, it's kind of funny. Sometimes people accuse me— I had this paper, The Vulnerable World Hypothesis, some years ago. And one concept introduced there is this— I called it the Freedom Tag, which is a kind of surveillance device. Imagine a kind of neck bracelet people wear that records everything they hear, and with omnidirectional cameras, that's continuously uploaded to some sort of freedom offices or whatever, where they're So some people then think, oh, they have accused me, like, oh, he's so ominous. He's advocating the Freedom Tag. Obviously, I named it Freedom Tag on purpose to really emphasize its Orwellian character. Now, it is fairly plausible, however, whether it's good or bad, that I think there will be more and more transparency and ability to surveil what people are doing. in more and more detail, and also eventually what people are thinking. And already current AI technology, I think, would be able to do a lot here. So for a couple of decades, it's been possible to sort of record everybody's phone conversations, right? And everything they write on their social media, et cetera. And government security organizations like to do this. But so far, it's not really— the only way you've been able to use it is if there is a particular person of interest that you could then assign a human analyst to read through what they have said and written. But now, even with current AI tools, I think you could do mass analysis, so analyzing what everybody is saying and writing, and therefore what they're thinking about the government, and do sentiment analysis and stuff. And you probably With current or very near-term technology, get pretty accurate results from that. And then you could imagine coupling that to some sort of social credit score system that would penalize people who express wrongthink. You could have an autumn AI that digs through everything they've ever said and done to try to dig up some dirt or some bad thing they said. decades ago, and then reduce their reputation accordingly. The potential in this technology, I think, already exists there for a social equilibrium to emerge that is one where there is much less obscurity and forgetfulness, and where it's possible for one system to see and have fine-grained ways of differentiating its reaction to everybody. I mean, I don't need to say that there are obvious ways for that to be dystopian, but maybe there are also some ways for it to not be dystopian and good.

Well, I don't want to leave people thinking that you're pro-authoritarian single-state government unless you are. So maybe could you just paint momentarily what's a good version of that where we're all monitored all the time?

Well, I mean, think of somebody living in a small community like a kibbutz or a little village. There were probably a lot of attractive features about that, and people probably knew a lot about what each person was like and what they said. And so it doesn't have to be a totalitarian nightmare, I guess. If you imagine this scaled up, maybe you would have a world free of crime, a world free where you can't be a jerk because a fraud and a jerk who goes around ripping off or taking advantage and exploiting one person and then moving on to the next because the track record would be obvious for everybody to see. And so So there could be stronger incentives to do kind and prosocial things. It's very hard. We don't have the kind of political or social science that allows us to make very clear predictions about what happens if you change some of these fundamental parameters of the collective information system. For what it's worth, my gut-level attitude is more like I'm more on the side of the punks, like these individuals who stick it to the man and the apparatus and the state bureaucracy like that, that's like, yeah, hurrah for that. But if I step back and reflect on what would actually make the future go best, I probably don't really know. It's really hard to tell. So I'm more on the agnostic end of that, I think.

You mentioned the Vulnerable World Hypothesis, this idea that every time we develop a new technology, it's kind of like drawing a ball from an urn. Sometimes it's a white ball where it's a good technology, it helps the world. Sometimes it's a gray ball, it has a mix of good and bad aspects, but maybe sometimes it's the black ball that ends life on Earth. Obviously, we haven't drawn a black ball yet. We haven't ended life on Earth, but we might. And what I'm wondering is, what's your best estimate of how likely we are to draw black balls? Is there anything we can say about that, or is it just too hard to possibly know?

Well, it looks like AI timelines are fairly short. If there's got to be a black ball, probably comes out of something enabled by intermediate levels of AI, like maybe some bioweapon design AI tool that could kind of— before we get superintelligence, you could imagine something that allows the world to get destroyed or some sort of social dynamics disturbing applications of AI, like with some of these surveillance or automated propaganda type of things. Those would be the most likely bets. I don't know exactly what the probability— AI itself is interesting because it's slightly different from other existential risks in that it is also something which, if it goes well, could protect us against a whole host of different existential risks. And even determining exactly what counts as an existential catastrophe is difficult with respect to AI because with a lot of other things, the world blows up, there is nothing after. It's pretty clear it's an existential catastrophe. With AI, it's more like there is a spectrum. The world gets radically transformed, and at the other side of that, what exactly exists and how valuable is that? We presumably don't want to insist on there being human beings in exactly their current forms running around on this planet for millions of years doing the same old human things. That itself would seem a little bit of a letdown, I think. On the other hand, if it's a paperclip maximizer, maybe we think that also fails to realize a lot of the potential for value. But it's to say that the concept of an existential risk has a value component as well as a descriptive component. And in the case of AI existential risks in particular, it seems the value component becomes particularly prominent.

So you came up with this very influential idea, the simulation argument, which essentially argues that we might be more likely to be living in a computer simulation than most people acknowledge. I'm wondering, have your probabilities changed over time of how likely you think we are living in a simulation, and where do they sit at right now?

Well, I tend to punt on giving an actual number. Many have asked, but none have been answered so far. Now, I guess maybe crept up a little, but not much. I think For other people, it might be reasonable to increase their probability in the simulation hypothesis. So if you think about the simulation argument, I think the paper was like 2003 or something. I circulated it a few years before that. At the time, I think it might have been a bigger imaginative leap for people to conceive of a level of technology that would make it possible to create realistic computer simulations with conscious beings in them. I think the decades of technological progress since then should make it easier. We have, I mean, just virtual realities is higher resolution and people play these immersive computer games like that. That should just make it easier to imagine how if we continue to make progress, we would eventually get to something super realistic. Then with AI as well, it just seems like a shorter step from where we are now to where we would actually have— sort of digital minds that are fully as sophisticated as humans. And so there's kind of less opportunities where you could hop off the train between where we are now and where the capability exists for running ancestor simulations than there was back in the early 2000s. And so in that sense, I think it would make sense for probabilities to creep up a bit as well.

Are there any concrete ways that you behave differently or live differently because of this possibility we're living in a simulation?

I think maybe a greater sense of humility with respect to the ultimate things. If you contrast it, so take the other extreme, some kind of archetype, like the classical sort of atheist materialistic evolution, like Richard Dawkins type, right? There's a kind of fairly well-defined inventory of the world and where we are in the world. We are on this planet, there's a bunch of stars, we began this long ago, then we die, and when we die, we rot, and that's the end. All of those relative to that kind of worldview would seem pretty confident implications. Whereas if you take the simulation argument seriously and the simulation hypothesis in particular, it suggests it could very easily be the case that there are many more possibilities. There might be a lot more in the world than is dreamt of in this kind of naive scientific picture. Right. There could be other simulations. There could be a sort of basement reality under the simulations. There could be whole hierarchies of simulations. There are simulators who designed this. There could be afterlives of various kinds. There's a lot more possibilities conditional on the simulation hypothesis. And so since we know very little about that whole space of possibilities, it, I think, induces a kind of epistemic humility that I think also can then translate into a more almost spiritual humility, a sense of our own smallness and how much we are in the dark with respect to the ultimate things that shape our ultimate destiny.

Final question before we wrap up. You're sometimes described as utilitarian, although I think that you don't actually identify as one. How would you describe your metaethical views and your views on questions around, are there objective moral truths and so on?

Yeah, I don't have a good label. And in general, I have always struggled with labels. They seem very confining, like all these isms that people love to subscribe to. I never really— it always seems a bit of a strain to me. I tend to think, in multiple channels, like in sort of superpositions, that maybe eventually they collapse and I get some more conviction on particular views. But my attempts to articulate the kind of metaethics recently, I wrote this paper, Basecamp for Mount Ethics. It's not a proper paper, really. It's more like some thinking notes. So it might not be useful to anybody. It's kind of obscure, but it's like an attempt to outline one direction that I'm thinking in terms of metaethics. I also like the idea of a moral parliament that I came up with. This is the idea that when you face some practical moral problem, rather than, say, pick the most probable moral theory that you can think of and do what it says, you should instead, as it were, assign probabilities to different moral theories, and then you imagine that these moral theories each get to send delegates to an imaginary parliament, and the number of delegates they get to send is proportional to the probability you assign to the theory. And then you imagine these delegates of the different theories deliberating and bargaining and compromising under ideal conditions, and then you should do what this parliament recommends that you do. The idea here being that even a moral theory that you think is less probable but that happens to care particularly intensely about some matter might get its way in that case in return for conceding to other moral theories in other cases that it thinks are less important. And so you could, I think, get a greater level of wisdom and lower propensity to fanaticism by thinking in terms of this moral parliament model than— it's really more like a metaphor than a formal model, but that's better than just picking your favorite moral theory and running with it.

Nick, great to speak with you. Thanks for coming on.

Good to talk to you, Spencer.

Thanks again for listening. We always love to hear from our listeners. So if you have questions or comments for us, just send us an email at clearerthinkingpodcast@gmail.com. This episode was edited by Ryan Kessler and transcribed by We Amplify. Uri Bram is the podcast's factotum. If you like our show, then we'd really appreciate it if you could rate and review us wherever you get your podcasts and tell your friends about us on social media. We also hope you'll subscribe to our email newsletter called One Helpful Idea. Each week, we'll send you one idea that we think is really valuable that you can read about in just 30 seconds, along with that week's new podcast episodes, an essay by Spencer, and announcements about upcoming events. To sign up for that newsletter or to find show notes, transcripts, and more info about the show, visit podcast.clearerthinking.org. A listener asks, what types of meditation have you tried and which ones have seemed most impactful to you?

I think I've tried over 30 types of meditation. It doesn't mean I've gone deep in them. I'm certainly not a meditation expert, but I've done a lot of experimenting. One of my favorite types of meditation is where I try to notice a good feeling in my body and mind, and then I try to let it grow. This is kind of related to what people sometimes call jhana meditation. And I like to just do it sometimes in the morning, like, for 30 seconds, just kind of let this good feeling grow. I find that really nice. I've also done a bunch of, for example, meditation where you focus on your breath. I did that for about a year. I did that almost every morning, and I kind of tracked some variables around that. And I found that interesting, and I found that it kind of made me more aware of my state changes, like when my emotion would shift or things like that. So I think it helped in a kind of introspective way, but I didn't really notice a lot of other benefits other than maybe making me feel calmer. I've also tried a lot of wacky meditations, meditations involving visualisations, meditations involving changing sensations in my body. That was actually how I first got interested in meditation. I realised one day, this is before I knew anything about meditation, that I could make a feeling of pins and needles in my arm if I kind of focused on it, and it would get really convincing. To the point where I thought, well, maybe my arm just has pins and needles. And then I would move my arm and the feeling would go away suddenly. And so that was really surprising and interesting to me that I could do that with my mind. And then I started thinking, what else can I do with my mind interior in my body and affecting the way that I perceive things? And started exploring meditation through that.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.