Nick Bostrom: Superintelligence | AI Podcast Clips

Let's talk about super intelligence, at least for a little bit. And let's start at the basics. What to you is intelligence?

Yeah, I tend not to get too stuck with the definitional question. I mean, the common sense understanding, like the ability to solve complex problems, to learn from experience, to plan, to reason, some combination of things like that.

Is consciousness mixed up into that or. No. Is consciousness mixed up into that or is it.

Well, I don't think. I think it could be fairly intelligent, at least without being conscious, probably.

And so then what is super intelligence?

So, yeah, that would be like something that was much more of, that had much more general cognitive capacity than we humans have. So if we talk about general superintelligence, it would be much faster learner, be able to reason much better plans that are more effective at achieving its goals, say, in a wide range of complex, challenging environments.

In terms of. As we turn our eye to the idea of sort of existential threats from superintelligence, do you think superintelligence has to exist in the physical world or can it be digital only? Sort of. We think of our general intelligence as us humans as an intelligence that's associated with the body, that's able to interact with the world, that's able to affect the world directly with physically.

I mean, digital only is perfectly fine, I think. I mean, it's physical in the sense that obviously the computers and the memories.

Are physical, but its capability to affect.

The world sort of could be very strong. Even if it has a limited. A set of actuators, if it can type text on the screen or something like that, that would be, I think, ample.

So in terms of the concerns of existential threat of AI, how can an AI system that's in the digital world have existential risk? And what are the attack vectors for a digital system?

Well, I mean, I guess maybe to take one step back. So I should emphasize that I also think there's this huge positive potential from machine intelligence, including superintelligence. And I want to stress that because some of my writing has focused on what can go wrong. And when I wrote the book Superintelligence, at that point, I felt there was a kind of neglect of what would happen if AI succeeds, and in particular a need to get a more granular understanding of where the pitfalls are so we can avoid them. I think that since the book came out in 2014, there has been a much wider recognition of that, and a number of research groups are now actually working on developing, say, AI alignment techniques and so on and so forth. I Think now it's important to make sure we bring back onto the table the upside as well.

And there's a little bit of a neglect now on the upside, which is, I mean, if you look at, I was talking to a friend. If you look at the amount of information that is available or people talking and people being excited about the positive possibilities of general intelligence, that's not. It's far outnumbered by the negative possibilities in terms of our public discourse.

Possibly, yeah, it's hard to measure, but.

What are, can you linger on that for a little bit? What are some to you possible big positive impacts of general intelligence? Superintelligence?

Well, I mean superintelligence, because I tend to also want to distinguish these two different contexts of thinking about AI and AI impacts. The kind of near term and long term, if you want. Both of which I think are legitimate things to think about and people should discuss both of them. But they are different and they often get mixed up and then you get confusion. I think you get simultaneously maybe an overhyping of the near term and then under hyping of the long term. So I think as long as we keep them apart, we can have two good conversations or we can mix them together and have one bad conversation.

Can you clarify just the two things we're talking about, the near term and the long term? What are the distinctions?

Well, it's a blur distinction, but say the things I wrote about in this book, superintelligence, long term, things people are worrying about today with, I don't know, algorithmic discrimination or even things self driving cars and drones and stuff, more near term. And then of course you could imagine some medium term where they kind of overlap and one evolves into the other. But I don't know, I think both, yeah, the issues look kind of somewhat different depending on which of these contexts.

So I think it would be nice if we can talk about the long term and think about a positive impact or a better world because of the existence of the long term. Superintelligence. Do you have views of such a world?

Yeah, I mean, I guess it's a little hard to articulate because it seems obvious that the world has a lot of problems as it currently stands. And it's hard to think of any one of those which it wouldn't be useful to have a friendly aligned super intelligence working on.

So from health to the economic system, to be able to sort of improve the investment and trade and foreign policy decisions. All that kind of stuff.

All that kind of stuff. And a lot more.

I mean, what's the killer app?

Well, I don't think there is one. I think AI, especially artificial general intelligence, is really the ultimate general purpose technology. So it's not that there is this one problem, this one area where it will have a big impact, but if and when it succeeds, it will really apply across the board in all fields where human creativity and intelligence and problem solving is useful, which is pretty much all fields. Right. The thing that it would do is give us a lot more control over nature. It wouldn't automatically solve the problems that arise from conflict between humans, fundamentally political problems. Some subset of those might go away if we just had more resources and cooler tech. But some subset would require coordination that is not automatically achieved just by having more technological capability. But anything that's not of that sort, I think you just get an enormous boost with this kind of cognitive technology once it goes all the way. Now, again, that doesn't mean I'm thinking.

Oh.

People don't recognize what's possible with current technology and sometimes things get overhyped. But I mean, those are perfectly consistent views to hold the ultimate potential being enormous. And then it's a very different question of how far are we from that or what can we do with near term technology.

Yeah. So what's your intuition about the idea of intelligence explosion? So there's this, you know, when you start to think about that leap from the near term to the long term, the natural inclination, like for me, sort of building machine learning systems today, it seems like it's a lot of work to get to general intelligence, but there's some intuition of exponential growth, of exponential improvement of intelligence explosion. Can you maybe try to elucidate, try to talk about what's your intuition about the possibility of a intelligence explosion? That it won't be this gradual, slow process, There might be a phase shift.

Yeah, I think it's. We don't know how explosive it will be. I think, for what it's worth, seems fairly likely to me that at some point there will be some intelligence explosion, like some period of time where progress in AI becomes extremely rapid. Roughly in the area where you might say it's kind of humanish equivalent in core cognitive faculties, that the concept of human equivalent starts to break down when you look too closely at it. And just how explosive does something have to be for it to be called an intelligence explosion? Does it have to be overnight, literally, or a few years? But overall, I guess if you plotted the opinions of different people in the world, I guess I would be somewhat more probability towards the intelligence explosion scenario. Than probably the average, you know, AI researcher. I guess so.

And then the other part of the intelligence explosion, or just forget explosion, just progress, is once you achieve that gray area of human level intelligence, is it obvious to you that we should be able to proceed beyond it to get to super intelligence?

Yeah, that seems. I mean, as much as any of these things can be obvious, given we've never had one, people have different views, smart people have different views there. It's like some degree of uncertainty that always remains for any big, futuristic, philosophical, grand question that just. We realize humans are fallible, especially about these things. But it does seem, as far as I'm judging things based on my own impressions, that it seems very unlikely that there would be a ceiling at or near human cognitive capacity.

And that's such a. I don't know, that's such a special moment. It's both terrifying and exciting to create a system that's beyond our intelligence. So maybe you can step back and say, how does that possibility make you feel that we can create something? It feels like there's a line beyond which it steps. It'll be able to outsmart you. And therefore it feels like a step where we lose control.

Well, I don't think the latter follows. That is, you could imagine, and in fact, this is what a number of people are working towards. Making sure that we could ultimately project higher levels of problem solving ability while still making sure that they are aligned like they are in the service of human values. Losing control, I think, is not a given that would happen. I asked how it makes me feel. I mean, to some extent, I've lived with this for so long, since as long as I can remember being an adult or even a teenager. It seemed to me obvious that at some point AI will succeed.

And so I actually misspoke. I didn't mean control. I meant because the control problem is an interesting thing. And I think the hope is at least we should be able to maintain control over systems that are smarter than us. But we do lose our specialness.

Mm.

It's sort of. We'll lose our place as the smartest, coolest thing on earth. And there's an ego involved with that that humans aren't very good at dealing with. I mean, I value my intelligence as a human being. It seems like a big transformative step to realize there's something out there that's more intelligent. I mean, you don't see that as such a.

Fundamentally, I think, yes, a lot. I think it would be small. I mean, I think there are already a lot of Things out there that are. I mean certainly if you think the universe is big, there's going to be other civilizations that already have super intelligences or that just naturally have brains the size of beach balls and are like completely leaving us in the dust and.

We haven't come face to face with them.

We haven't come face to face. But I mean that's an open question. What would happen in a kind of post human world? Like how much day to day would these superintelligences be involved in the lives of ordinary. You could imagine some scenario where it would be more like a background thing that would help protect against some things, but you wouldn't like that. There wouldn't be this intrusive kind of making you feel bad by making clever jokes on your experiment. There's a whole sort of things that maybe in the human context would feel awkward about that you don't want to be the dumbest kid in your class. Everybody picks it. A lot of those things maybe you need to abstract away from. If you're thinking about this context where we have infrastructure that is in some sense beyond any or all humans. I mean it's a little bit like say the scientific community as a whole. If you think of that as a mindset, it's a little bit of metaphor. But I mean obviously it's got to be way more capacious than any individual. So in some sense there is this mind like thing already out there that's just vastly more intelligent than any individual is. And we think, okay, you just accept that as a fact.

That's the basic fabric of our existence is there's super intelligent.

You get used to a lot of.

I mean there's already Google and Twitter and Facebook, these system recommender systems that are the basic fabric of our. And I could see them becoming. I mean do you think of the collective intelligence of these systems as already perhaps reaching super intelligence level?

Well, here it comes to this. The concept of intelligence and the scale and what human level means. The kind of vagueness and indeterminacy of those concepts starts to dominate how you would answer that question. So say the Google search engine has a very high capacity of a certain kind, like remembering and retrieving information, particularly like text or images that you have a kind of string, a word, string key. Obviously superhuman at that, but a vast set of other things it can't even do at all. Not just not do well. So you have these current AI systems that are superhuman in some limited domain and then radically subhuman in all other domains. Same with Just a simple computer that can multiply really large numbers. So it's going to have this one spike of super intelligence and then kind of a zero level of capability across all other cognitive fields.

Yeah, I don't necessarily think the generalness, I mean I'm not so attached with it, but I could sort of. It's a gray area and it's a feeling. But to me sort of Alpha 0 is somehow much more intelligent, much, much more intelligent than Deep Blue. And to say which dimension, you could say, well these are both just board games. They're both just able to play board games. Who cares if they're going to do better or. But there's something about the learning, the self play that makes it crosses over into that land of intelligence that doesn't necessarily need to be general. In the same way Google is much closer to Deep Blue currently in terms of its search engine than it is to sort of the Alpha Zero. And the moment these recommender systems really become more like Alpha Zero, but being able to learn a lot without the constraints of being heavily constrained by human interaction, that seems like a special moment in time.

Certainly learning ability seems to be an important facet of general intelligence that you can take some new domain that you haven't seen before and you weren't specifically pre programmed for and then figure out what's going on there and eventually become really good at it. So that's something AlphaZero has much more of than Deep Blue had. And in fact systems like AlphaZero can learn not just go, but other in fact probably beat Deep Blue in chess and so forth. Right. So you do see this general. And so it matches the intuition we feel it's more intelligent and it also has more of this general purpose learning ability. And if we get systems that have even more general purpose learning ability, it might also trigger an even stronger intuition that they're actually starting to get smart.

So if you were to pick a future, what do you think a utopia looks like with AGI systems? Is it the neuralink brain computer interface world where we're kind of really closely interlinked with AI systems? Is it possibly where AGI systems replace us completely while maintaining the values and the consciousness? Is it something like it's a completely invisible fabric like you mentioned, a society where it's just aids in a lot of stuff that we do like curing diseases and so on? What is utopia if you get to pick?

Yeah, I mean it's a good question and a deep and difficult one. I'm quite interested in it. I don't have all the answers yet, but might never have. But I think there are some different observations one could make. One is if this scenario actually did come to pass, it would open up this vast space of possible modes of being. On one hand, material and resource constraints would just be expanded dramatically. So there would be a lot of a big pie, let's say. Also it would enable us to do things including to ourselves like that. It would just open up this much larger design space and option space than we have ever had access to in human history. So I think two things follow from that. One is that we probably would need to make a fairly fundamental rethink of what ultimately we value. Like think things through more from first principles. The context would be so different from the familiar that we could have just take what we've always been doing and then like oh well, we have this cleaning robot that cleans the dishes in the sink and a few other small things. I think we would have to go back to first principles and things.

So even from the individual level go back to the first principles of what is the meaning of life, what is happiness, what is fulfillment.

And then also connected to this large space of resources is that it would be possible. And I think something we should aim for is to do well by the lights of more than one value system. That is we wouldn't have to choose only one value criterion and say we're gonna do something that scores really high on the metric of say hedonism and then is like a zero by other criteria like kind of wire headed brain Sinovat and it's like a lot of pleasure that's good. But then like no, no beauty, no achievement like that or pick it up I think to some significant, not unlimited sense, but a significant sense it would be possible to do very well by many criteria. Like maybe you could get like 98% of the best according to several criteria at the same time given this great expansion of the option space.

So have competing value systems competing criteria as a sort of forever. I mean just like our Democrat versus Republican there seems to be this always multiple parties that are useful for our progress in society even though it might seem dysfunctional inside the moment. But having the multiple value system seems to be beneficial for I guess a balance of power.

So that's yeah, not exactly what I have in mind that it's well, although maybe in an indirect way it is but that if you had the chance to do something that scored well on several different metrics, our first instinct should be to do that rather than immediately leap to the thing which ones of these value systems are we going to screw over let's first try to do very well by all of them then it might be that you can't get 100% of all and you would have to then like have the hard conversation about which one will only get 97%.

There you go there's my cynicism that all of existence is always a trade off but you say maybe it's not such a bad trade off let's first.

At least try well this would be a distinctive context in which at least some of the constraints would be removed there's probably still be trade offs in the end it's just that we should first make sure we at least take advantage of this abundance in terms of thinking about this one should think I think in this kind of frame of mind of generosity and inclusiveness to different value systems and see how far one can get there first and I think one could do something that would be very good according to many different criteria it.