88. Oren Etzioni - The case against (worrying about) existential risk from AI

Hey everyone, Jeremy here and welcome back to the Torres Data Science podcast. Now, I said in a previous episode with Melanie Mitchell that I wanted to get more guests on the podcast who don't buy into the AI existential risk arguments that we've been exploring so far. And that's what brings us here today for this very special episode of the podcast. So we'll be talking to Oren Etzioni, who is just a ton of things really. He's a successful startup founder, he was a professor of computer science, but he's now the CEO of the Allen Institute for AI, which I'm sure many of you are very familiar with. He's a world leading AI research lab that's done a ton of really interesting and groundbreaking work and that includes projects like Semantic Scholar as well as the very popular Allen NLP library. Now, as I mentioned, Orin is skeptical of many AI risk arguments, and I think he offers a fresh and balanced perspective on the issue and one that people focused on AI safety would definitely benefit from hearing. So with that said, without any further ado, I'll step out of the way and let you enjoy our deep dive into the world of AI safety, AI alignment and and AI risk skepticism. Hi Ornan, thanks so much for joining me for the podcast here.

Jeremy, it's a real pleasure to be here.

I'm really happy to have you here. You are, I think, the second kind of major conversation that I'm making a point of having with somebody who's more on the skeptical side of the AI risk equation. Obviously we've had on the podcast a lot of people who talk about AI alignment, AI safety and so on. I think this conversation is going to be really valuable for a lot of listeners because I want to make sure that we don't just talk their ears off about one side of the story. So maybe with that framing, I'd love to ask you for your perspective. Very high level. Like what do you think the case is that people who tend to worry about AI safety and AI alignment will make and what do you see as far as are the biggest high level issues with it?

Sure. So let me start off by saying that while I'm an AI optimist and also an AI practitioner, I've been working in the field for decades. I see a lot of issues with the field. I see concerns about privacy, concerns about fairness, concerns about job displacement. One of my concerns about the ultimate AI risk, the conversation about existential risk, is that it's actually a distraction from the real risks we ought to be thinking. About the ones that we're experiencing today, the ones where we really need to get a handle on it. But I said this actually to Max Tegmark. We were on a radio show together. It was a lot of fun, and I have a lot of respect for his ideas. And he said, gosh, Oren, I'm trying to talk to you about the future of the human race, and you're arguing with me about what? We're going to feed the children for dinner. Right. So his argument is that this question of the existential risk of AI, the catastrophic risk of AI, outweighs all the other things. And we shouldn't be worried about little things like mass unemployment or surveillance states or right. Countries like China using AI to suppress the citizens. And I couldn't disagree more strongly. From my point of view, those are very real problems that will affect and are affecting millions of people, if not more, as opposed to a speculative one. That's really a philosophical conundrum. Right. And again, I could go on and on about this, but I do want to make the point that it's really important in these conversations to think about the data. Right. You have some intuitions. The audience has intuitions. Max does. Stuart Russell does. Lots of people have intuitions. But when we find conflicting intuitions, what do we do? If we're philosophers, and that's our job, we engage in lengthy, abstruse conversations. We try to come up with difficult, better examples, better intuition pumps, better metaphors. If we're scientists, we look to the data. If we're empirical scientists, we say, well, what data do we have about this? So I feel like one of my most important contributions to the conversation is to suggest that we frame it in empirical terms, not in philosophical or conceptual terms.

And what are those terms? Because I know you wrote an article touching on some of these themes as well, where you talked about the canaries in the coal mine of AI. I'd love to have you maybe explore some of those and maybe some of the hard data that you could imagine surfacing that would change your mind on this AI risk equation.

Sure. So there are two kinds of data. One is simple and imperfect. So I'll mention it, we'll dispense with it, and then let's go to the data that's meatier. So the simple data is we can survey people. We can survey people on the street, although their attitudes about AI have often been shaped by Hollywood, so it's no surprise that they have a lot of fear. Maybe comes from the terminator, maybe even comes from before that. Right. This fear of the unknown and fear of technology, and particularly fear of technology supplanting us, goes all the way back to Frankenstein and the Golem, right? These are very deeply held human fears. So it's very natural to be afraid. It's quite another thing to paint it and to intellectualize that fear. Painted with brushes that try to legitimize what's, what's a visceral fear. The second. So I did actually do a survey of the leading AI researchers, these are the AAA fellow, and the strong majority of them, and this was written up in MIT Technology Review, the strong majority of them say that super intelligence, this kind of omniscient, omnipotent intelligence, is beyond the foreseeable horizon. So with fast moving technology, we really struggle to predict there. So that's that kind of data, what people think. And the people in the know are very skeptical, right? Andrew Ng says, right, Worrying about AI turning evil is like worrying about overpopulation on Mars. It ignores the technical difficulties of getting there and of course, all the positive outcomes, right? It just kind of. And Rod Brooks, famous AI robotics, says, if you're worried about the Terminator, just close the door. And he shows this wonderful illustrated example of a robot failing to open a door and falling over. And of course, robots will eventually open doors, they've already gotten quite good at it, but then they'll find behind the door there's a staircase and robots struggle to climb the staircase and eventually they'll learn how to do that. And there's another challenge. Another challenge. So the data, here's a data point. Self driving cars, right? Which is not super intelligence. It's a limited capability in the world. We've been anticipating it for quite a while. And guess what? There aren't any self driving cars yet. They're coming. I'm not saying those are beyond the foreseeable horizon, but that's just driving. And that's an example of the kind of optimism that we can fall prey to. The real world is complex and it's nuanced and we struggle there. So in the article that you mentioned, Jeremy, I talked about canaries in the coal mine of AI. And this is of course a metaphor, and it's the metaphor that refers to the practice that coal miners used to have of having canaries around and they were worried about running out of oxygen. And if oxygen got low, a canary would tip over and die. So the idea that we can have harbingers, we can have warning signs, signals about what's happening. So to me, this puts it on a more empirical footing. Because if we're able to identify specific capabilities that if AI crosses them, then we really need to take this seriously, then we can then say, okay, hasn't crossed this threshold. We don't need to worry about it. Right. There's a line in the sand to use that phrase, as opposed to vaguer arguments of the sort. Well, look, you know, Stuart Russell is fond of talking about what happened with Einstein in the beginning of 20th century physics. Yeah. Everybody thought that physics was like this and suddenly was like that. Sure, it's a beautiful historical science anecdote, but it's an anecdote. It's not a specific data point. It just suggests that we can underestimate how quickly things turn. And conceptually, I agree with that. That is possible, but scientists are less concerned with what's possible, but with what's feasible, what's likely, what does the data tell us? So I, in this article, tried to formulate specific data points that if we cross them, I'll be the first person to say, okay, we really need to sit down with this.

So I really like, philosophically, the approach of almost putting skin in the game a certain way by pinning empirical thresholds to a change of theory, a change of perspective on these issues. And I think it's something broadly that none of us really do nearly often enough. There's so many ways in which we move goalposts and things like that. So it was refreshing to see that laid out. I'd be curious to get your perspective then on like, the. Because. So if I'm understanding correctly, and let me know if I'm poorly paraphrasing your position, but it sounds like to me that you're saying AGI, or transformative AI superintelligence is not impossible, but that it is quite likely so far away as to not be worth our paying attention to it now, particularly in light of current issues that it would draw our attention away from. Is that fair to say?

Yes, with one caveat, one modification. So I had a wonderful conversation about this over some sushi with Nick Bostrom. And he said, look, Oren, you grant that it's possible? Yes, I do. He says, you grant that it could come faster than we think? He said, yeah. He says, well, then why don't you grant me five people to work on that scenario? And so just a few people. And I said to him, nick, I have a lot of respect for you. You're a smart guy, you know what? And I'm generous. Take 20 people, take 50 people. So I'm not saying that we should just completely ignore this. This is just silly, right? This is flat earth. I'm saying that we benefit from a small number of people considering very far off exigencies. But if again, we think of it not in terms of absolutes, should we worry about this, should we not worry about this? But we think about this more as a program manager at DARPA or NSF or the National AI Institutes and we say, how much of our budget do we allocate to this thing? And all the way from the extreme we declare moratorium on AI, let the Chinese and the Russians, right, And others dominate that field. We're too worried about it to okay, 20%, 50% is going to be on this very horrific problem. So I say, yeah, if a small sub 1% of our attention is paid to this scenario, great, I see some benefit in that. Let me also mention though that actually a lot of what we're doing is addressed at this problem. Just people don't see it that way. So here's what I mean. When you encounter any alien intelligence, another AI, what is the thing you want to do? You want to communicate with it. And a lot of our science fiction movies and scenarios illustrate that, right? And if they're very powerful, you surely want to communicate with them before they destroy you to achieve a sense of empathy, to understand how you can coexist, negotiate, what have you. Well, all research on natural language processing, natural language understanding is about getting computers to understand our language. All that research is about facilitating communication. So you can make it an alignment problem. Let's say I've got to get their utility problem the same as our utility problem, and I've got to control the irritability function and things like that. Or you could take another tack at it, which I'm not saying is necessarily better, but it's at least just as valid. And say, let's make sure that we can talk the same way that you and I are doing, even though we might have a different point of view. So that's a huge one. Another huge one is common sense. This is a big topic of research at the Allen Institute for AI. Often the various scenarios, like the one where the robot destroys humanity because it's trying to make paper clips arise, because the program doesn't have any common sense, right? It doesn't have any ethical sense, it doesn't have any sense of boundaries. What's reasonable? Well, how is it supposed to, if we don't build systems that imbue that, and it's not as simple as just Having a mapping from its utility function to some number. You really need to understand what's going on in the world. You need to understand that if you take all the metal for paperclips, you will not have metal for other things like that are used in hospitals. Right. That's a common sense piece. So we're also working on building common sense into AI. All these approaches are actually kind of multifaceted. They address multiple concerns about AI.

I think there's really something to that. This idea that a lot of these things that we consider to be binary or categorical positions on the AI safety in the AI safety debate are really continuums. And you mentioned a couple there implicitly, like one of which is the continuum of the amount of effort we expend on this. Like it's not zero and it's not 100%. So we got to play that game. But then as a separate thing, there's this interesting argument, and I think in spirit, it certainly seems related to some of what Paul Cristiano says at, formerly at OpenAI, where he would talk about, again I'm paraphrasing and a little bit doing, doing so with some liberties. But basically the idea that as we continue to develop this technology, we're going to have a slower takeoff, we're going to have an entanglement between alignment and capabilities. The two become somewhat inseparable. Because when you want to deliver capabilities at a certain level, you can't do that without having your system reflect the needs and values of the programmer or the people who are going to be consuming the product. Because the value of a really intelligent product is diminished by its not being aligned in the first place. So it kind of seems like there's a bit of a maybe a tandem escalation argument here. Would that be a fair paraphrasing of your position as well?

Well, my position is simpler and less jargon laden. So when you say entanglement and alignment and all these things, I have to say that my mind goes to. I'm not sure what he's trying to say, but what's the basis for it? You say, well, the basis for it is this argument. It's these words that he uttered, say, oh, I get it, he's a philosopher. Philosophers have a history of thousands of years of accomplishing very little, arguably nothing. What technology has philosophy produced Scientists, okay, they can also be jargon laden, but they're generally empirical. And they're the people who brought us the COVID vaccine, right? Not philosophers. Philosophers didn't philosophize a COVID vaccine. It should be possible. It should be possible at this time, Right. So I'm just. Every time somebody brings up these rather abstruse philosophical arguments, having studied philosophy for many years in college and grad school, my eyes glaze over because I don't have a lot of interest in that and I don't think it's highly predictive of what's going on. But the fact of the matter, it.

Might help if I just kind of rephrase because for what it's worth, I mean I studied AI for quite a long time. I did most of a PhD in physics. I am a scientist by training and a startup founder. But I guess I would disagree to the extent that philosophy is the study of logic in its best form. It's the study of logic. I think there are branches of philosophy that get very kind of lost in themselves. But perhaps the problem is just my actual phrasing of the question, which is what I was intending to suggest is like you have this gradual escalation of if you want to make an AI system do thing X really really well, then eventually it does X so well that like take self driving cars that you start to deploy it in more and more critical scenarios. And as you deploy it in more and more critical scenarios, the requirement to cover the kind of safety blind spots that technology start to increase. And so you have this natural co evolution of these two things where like there's a forcing function on both sides of the equation to make sure that like you're never deploying a super capable technology in a context where it can cause har. That's sort of like the economic argument I guess, for as I understand it, for this sort of safety position.

Well I certainly understand you better now and again we can spend the whole hour arguing about kind of philosophy versus science. And of course philosophy maybe at the sub 1% level has its place in the world, particularly when we think about ethics and things that are naturally philosophical. Again, my belief is that it's of little value in understanding the progression of AI. However, your point, which sounds like common sense policy, if again, hopefully I'm doing it justice, it sounds like, look, if you're deploying powerful technology, you better deploy powerful safeguards, right? You can't just let this thing run amok. And I've said the same thing myself, the words I use is, I like to paraphrase that famous philosopher Spider man who says, you know, with great power comes great responsibility. And certainly AI is a powerful and continues to be increasingly powerful technology. And so we ought to be thinking about this. Can I make one other point, if you don't mind? So again, this is actually having, you know, pooh, pooh. Philosophy is actually a philosophical or conceptual point, but it's worth thinking about because it shows how quickly our biases and our prejudices can sneak in to an argument. So an argument that Nick Bostrom makes, and again, it has a mathematical foundation. Hopefully I'll do it just as quickly. But those of you who want the real argument should obviously read his paper. But he basically says, imagine that we have an urn and we're drawing balls out of urns. Right? A typical mathematical combinatorial setting. Well, let's say that there's some black balls in this Earth. And these black ones, if we draw them, they'll annihilate the world. These are world destroying technologies. Obviously some technologies like that exist, whether it's in the biological sphere. Right. Some super pathogen or, you know, what have you. So we'll grant that, sure they exist. He says, well, what is technology and what are technologists like people like me and perhaps like you are doing? We're constantly reaching into this Earth now under different. And drawing these black balls, under different assumptions on how frequent these black balls are, there's this probability that we will annihilate the world. And so it's not just AI, that's one of these super intelligence, that's one of these black balls. He can foresee a bunch of other ones and this constructs kind of an argument and then you can go from there. Okay, you shouldn't.

You said this about the LHC too, back in the day, the sort of black holes that might be created by these collisions.

Yeah, exactly, exactly. And again, by the way, it's his black ball, is his phrasing, right? I'm, you know, color neutral on this, I don't want to apply. But I'm just saying with his metaphor, here's the thing that he doesn't say. Imagine the exactly symmetric scenario. We've got the urn, but there's some white balls in there too. And we reach into the urn and if we don't take out a white ball, humanity will be annihilated, maybe by Covid. Right. Without a vaccine or some worse pathogen. Maybe that arose naturally. Right. I'm assuming it's not snuck out of some lab in Wuhan, Right. Or some other thing. Right. An asteroid would come from outer space and destroy us. So we are reaching in and with some probability drawing out white balls to save all of humanity. It's the exactly symmetric argument. But now it's about technology saving humanity and how if we don't reach into the urn, we're toast. So what does this show? That technology is good? No, it shows that the framing of the argument is biased. Bostrom tells the story about black balls and how technology is bad. I told you the same story, his story, but it's about how technology is good. Not just good. It's absolutely necessary to save the earth. But what's the difference? Not the philosophy, not the metaphor, not the mathematical calculations, but the framing. And that's where philosophers cheat worse than a hustler at Vegas.

I definitely. So I definitely agree with the sentiment. I would say that when you said the mathematics have changed, that that's an incorrect characterization of the scenario you just laid out. Right. Because the mathematics have changed completely. You've just introduced a white ball or like a universe saving ball, which like previously we're talking about all the kind of balls being either negative or neutral. Now we're talking about one that's positive. And I actually think that like when I talk to AI safety people, this is actually something that they do like kind of keep in mind. So a common narrative among many, not all AI safety people. I think you're absolutely right. There's like a kind of temperamental bias towards like pessimism and like, you know, worst case scenarioism. But I do see this sort of tandem, sort of devil's choice thing going on where people go, yeah, you know, on the one hand we're at risk of coming up, as you say, with like a deadly pathogen or who knows what. And one of the few things that could actually help us navigate that scenario is super powerful AI. Like this is really tricky. We're probably, if you just extrapolate 100 years, we're probably not going to make it for various reasons that AI could really help us deal with. And the question is like, if AI does come with this extra risk package, potentially how do we navigate those two things in tandem? I see you nodding, so maybe I'll just stop there. Yeah. Do you have a reaction to that view?

I do. So first of all, just to clarify, when I was saying that the math wasn't changing, what I meant was imagine a symmetric argument. So not that there are black balls and white balls, but just I would take his argument as it is with his equations and just reflect it. Basically change the minus to a plus and that's all. So that's, I just meant I could reuse his same equations to make a different point. Now you're talking about something that I consider, you know, much more sensible and realistic is what if we have an urn, which we do, it's called technology, conceptually. And we are constantly drawing from it, you know, black balls and white balls and red balls and green balls and some, you know, make climate change worse, and some around, you know, carbon sequestration make it better, and some can be, you know, make it really, really bad, and some can make it really, really much better. And you outline that completely. So now we get into a very nuanced question that to me, has much more of empirical flavor. We all agree that technology is powerful. We all agree that it changes how we live, and we all agree that we want to guide it in the right direction, right? Our. The mission of my institute, the Allen Institute for AI isn't AI, who cares? It's AI for the common good, right? So we believe that the work we're doing is very much designed to make the work a better place. So I think. And likewise, you know, the work in value alignment, you know, such as it is. I mean, it's kind of an obscure and limited sub, sub, subfield, but such as it is is good work. Surely we should understand how if an entity has a utility function, how do we train it? How do we align it with ours? That seems like a valuable exercise, especially if you believe that our technologies will have explicit utility functions. Of course, you and I actually don't, but that's another story. So I love your description there of kind of a multiple rates problem and, you know, multiple things are happening at once. I think that's realistic.

And would your calc. I'm curious how your, your thinking on this would shift if, let's say you had like a revelation, you discovered that, you know, AGI was actually plausibly around the corner, like, you know, five years away, I guess, is. It's sort of like people. People used to say this about certain economies. They'd say, like, without wanting to name a country, country X is the country of the future and it always will be. AI is always five years away, that sort of thing. But let's say it actually were. How would this change your calculation personally as to the value of AI safety research in particular?

Well, I do think that a lot of the argument is about when and about which is a continuous variable and about certainty. How sure are we? I think generally virtually everyone, virtually everyone would agree with that. When the hominids, whatever, the Neanderthals learned to walk, we didn't want to launch research into AI safety, it was too early. Maybe even when fire was invented, the basis of electricity, we didn't want to do that either. And likewise, also not anti nuclear proliferation. It was just too early. Right. So it's a question of when do you start, how does it get intense and how likely is it that it'll actually occur. So now let's say we get this email. Again, this is, I think, one of Stuart Russell's metaphor. We get this email that says, hey, it's coming in 25 years and it's going to be malevolent. Then I would say that we should actually start a Manhattan Project, an international one that will look at different ways of addressing it. And, and there's a multitude of different ways to address it, not just one. And we should engage in all of them. So I would say absolutely, if the risk, if it's a clear and present danger, we need to engage with it. So then the question becomes, it's like the old George Bernard Shaw joke, okay, so we've agreed, now we're just bargaining over the price, right? And that is my point. We're in the world and I think, Jeremy, you are sympathetic to this. We're talking about how far away, how potentially destructive or beneficial or both, and how sure we are. And these are the things that policy people deal with, that scientists deal with. Nobody asks, nobody makes predictions that are, have real teeth in them based on conceptual arguments, right? Look, at some point we'll have another recession. At some point interest rates will go up. But we turn to economists and income and attritions and other people who engage with the data to predict when we don't do it based on armchair theories.

But do you think, I mean, I guess I'm left to wonder what data those economists, because, you know, we all sort of have different canaries in coal mines. I think one of the challenges is like, we're approaching AI in a very different way from the way that we are used to approaching intelligence sort of biologically. So biologically, you know, we approach intelligence through two different experiential vectors. Like, like first off is evolution. We can look at a mouse, can be like, oh, that's dumb. And then we look at a cat, oh, that's smarter. And then a dog and so on. We can kind of like play that game. And then alternatively we can look at a baby and say, oh, okay, you know, it's 11 year old, it's really dumb. And then, and so on. But the way that learning works through evolutionary time, the way it works through like child Rearing time. And then the way that we're approaching it through artificial intelligence with these sort of like supervised learning systems, it just like kind of throws our intuition off. I guess it makes me wonder a little bit to what extent we can trust our, our intuitions about what a valid canary might be since the, the vector of approach is so different. I wonder if you agree with that or have a take on it.

Yes. So I think you're making an excellent point and I would rephrase that as saying that with certain things we kind of know how it goes. They're highly predictable. Like when are we going to have the next presidential election? In four years, if it's Israel, in two months. That's a side note about their electoral process. But the point is often we can see if we're building a bridge or a skyscraper, we kind of know how long it'll take. When it comes to other things, particularly something like the evolution of intelligence, we have very, very little to go on because we're doing it for the first time. We actually have more than 50 years of AI research to go on. Right, but, but again, it's not necessarily just a simple linear curve. It is a lot more complicated than that. And recently with the work on deep learning and so on, we have made some substantial advances. So all these do feed the concern and feed the story. Where I get my intuitions from is from building these systems and talking every day to people who are building these systems and, and talking to them about what they do on a day to day basis. And one of my favorite lines is that. Human learning, human, sorry, machine learning, machine intelligence is 99% human intelligence. So when you see something, whether it's GPT3 or AlphaGo or the latest feat of AI success and, and you ask, okay, so how much hand engineering went into that? How much human work to check and recheck and retune and configure all this? The answer is immense, right? So one of the big reasons, and of course people say, well, evolution did that for humans, say fine, but I'm just saying this technology, right? AlphaGo can't even play another game of Go unless a human being pushes a button, right? It's this very narrow savant and very, very limited capability. You know, GPT3 seems intelligent because it spouts, you know, these remarkable sentences or documents, but it isn't intelligent because Pablo Picasso actually put it wonderfully, he said computers are useless. They only answer questions. They don't ask questions. If you try to get GPT3 to answer to Ask questions. Of course it can't, right? If you prompt it, it'll spit out anything. So you prompted with some example questions. It'll ask questions, but they'll be sealy, they'll be meaningless, right? They won't be the profound questions we need, either in philosophy or in science and so on. So what I'm trying to say is when you actively engage with the technology, you realize something really, really fascinating. This is not magic, right? Sufficiently advanced technology is indistinguishable from magic. So if you don't know the technology, you're like, wow, this is magic. But if you engage with it, you realize that it's something entirely different. It's programming, it's software. The software is incredibly powerful, right? But every step of the way the human is fussing with it. And so it's just a tool, a technology that we use to build artifacts.

I think this is a really interesting direction to go into because not least of which because I could imagine a neuroscientist, once they get to the point of understanding the way the brain works sufficiently well, going, it's not magic, it's just neurons, you know, like we say software, which, which raises questions about the extent to which, you know, software might be approaching that as well and how easy it would be to tell. But one of the things I wonder about with GPT3 is like thinking less of it in terms of absolute, like absolute performance and what it can do now, but more like the direction. So to look at this idea of like human fine tuning of systems, I think it is absolutely true. Like human beings have to put in a whole heck of a lot of work, like kind of like preparing a carefully laid out playground for these systems to kind of make sure that they don't tip over, whatever. But I also get the sense that that playground is getting rapidly larger and less controlled. So when we look at GPT3, for example, one of the, the biggest, most impressive things about it is the fact that it's able to hit all these different benchmarks, of course, like really, really well without having to be fine tuned. And so to me, that would have like, if we were having this conversation five years ago, I could imagine us saying a canary in a coal mine is a moment when a single AI system can hit multiple benchmarks without having been intentionally trained for those. And then like New Zero does similar things where now, you know, we no longer have to spoon feed it a specific game, but instead it learns the rules of games without having to like, be specifically designed. I wonder, like, what do you think of those systems in the context of, like, that perspective? Do you think that's a plausible view, or is there something really wrong with it?

I think you're pointing to some of the most impressive efforts where we have made a lot of progress. So to me, the advance from chess to Go, even though kind of on paper it seems amazing, and culturally it's quite remarkable. A lot of people view chess as a simple game, but fundamentally it's kind of the same thing. However, learning a game from scratch, so AlphaZero as opposed to AlphaGo, and then learning it even from the rules, I do think is extremely exciting. Likewise, this notion of transfer learning and the ability to, without fine tuning, work across lots of different problems is very impressive. It also reveals misconceptions that we had. Right. Why is GPT3 able to work across so many different arenas? In part because the input that it's received is so vast. Right. And its ability to, you know, with a very large number of parameters, you know, whatever 175 billion to store those is so vast, it's still the case that if you presented with something entirely new, it can be very confused. Of course, people can as well. So I think what we've learned is that I think we've learned various valuable technical lessons as AI advances. The question still remains, and this is a question, I think this metaphor is from the 70s. You say, we've gotten to the top of the tree and we're looking at the stars, and then we shout, we're on our way to the moon. Look how much higher I was than I was just a year ago when I was at the bottom of this very tall tree. Somebody else kind of takes the wind out of your sail and says, you know, it's still a really long way from the top of the tree to the Moon. But what's worse, the technology that got you from A to B, that allowed you to climb the stairs, or even that allow you to fly to 40,000ft, is not the technology that will get you to the Moon. And the technology that will get you to the Moon will not get you to Alpha Centauri. In fact, there's some fundamental problems that have to be solved. So that's really what we're talking about. Can we extrapolate from these to much, much further? And when you focus on the limitations of the technology. So, for example, think about just how limited GPT3 is still in terms of formulating its own problems, in terms of even being able to be consistent with itself. Right. It's very Easy to trip up GPT3 by asking it the same question three different ways and then you find out it doesn't really know what it's talking about.

Do you think that that might be though a symptom of this is where I'm really curious about this idea of approaching technology from different, sorry, approaching intelligence from different directions. So like, you know, self driving cars are going to end up killing people in like ways that seem really stupid to us. Like we're going to look at like a self driving car that just like mows down a whole bunch of pedestrians one day. And like the paradox of this is going to be that the self driving car is actually net safer than like a human driver. It's just that the kinds of mistakes it makes has you just like scratching your head as a human being. You're like, how, how silly can this be? By contrast, looking at something like GPT3, I wonder if like there's perhaps a similar fallacy where we end up looking at it and going like, oh, that's a silly mistake. But in reality it's kind of mastering some of the hard, like some of the things that humans would find hard that machines don't like. I don't know, what are your thoughts on that?

So I agree with you that the evolution path of the machines looks like it'll be quite different than either the evolutionary ladder, you know, with dogs and primates and so on, or the developmental ladder of 1 year old, 2 year old, 10 year old. And so it's a lot more like, you know, we had birds, we tried to figure out flights, we ended up with airplanes and of course, you know, those are having some of their own problems right now. So I do think it'll look very different. And I do think that the right question to ask if you're a policymaker, a regulator, or even just, you know, a person on the street, are you for self driving cars or not? I think if those horrific events that you describe are too frequent, I think that would be very, very uncomfortable. But I think we have the potential here and this is a great example of why I am an AI optimist. We have the potential to reduce the number of deaths and injuries in our roads and highways by a substantial amount. So you know, there's 40,000 highway deaths in the United States alone per year and the projection is that 80% or more of these could be reduced. And really what's happened is a long time ago we don't really talk about this, but a long time ago we struck a Faustian bargain with transportation technology. We agreed that for convenience and for economic gain, we're going to sacrifice a large number of people every year to these horrific accidents. Including by the way, individuals unfortunately deciding to mow down a bunch of people with homicidal intent. So we have all this and the technology has the potential to save us. So for example, imagine that a person is driving the car or an AI is driving the car. Something gets or goes awry and safeguards are built in to stop that from happening. And by the way, I would conjecture that these safeguards would have less to do with value alignment with the utility function of the car, and maybe more with mechanical guards that are kind of the same way that we have certain reflexes that might be completely outside of its cognitive system that just say, gosh, if you experience this kind of impact, just stop. Right, Come to complete dead stop. Right, so again, sorry, let me stop there.

No, I think that's a great point. I think looking at self driving cars is kind of a microcosm for this sort of trade off where we, we go, look, there's a plus side and there's a minus side. This ledger is complicated. We've got to think about both as we do this. I think that's a really, again, a really great example. I'd love to go back to. We were talking a minute ago about this question of what it will take to get from the tree to the stars, so to speak. And earlier you talked about talking to leaders in the space, like people who've been, people who you've polled and spoken to. I wonder in that context, what do you make of say, Geoff Hinton saying he thinks deep learning is going to get us all the way there, or Rich Sutton saying it's all about compute. Do you see those as representative or not? Or do you just not buy those arguments for other reasons?

Well, first of all, people like Jeff Hinton or Rich Sutton, I have a huge amount of respect for and particularly Geoff Hinton. When I was in grad school at Carnegie MELLON in the 80s, he was there as a professor preaching his neural network stuff. And most of us were very skeptical. So I have a huge amount of respect to somebody who argued, stuck those guns, argued against the popular opinion. And both due to his work and due to technological trends like Moore's Law and GPUs becoming available, things have really changed. So these people might have a much richer picture than I do. However, I would ask them the following question, and this is by the way, a question that Alan Newell and Herb Simon but particularly Alan Newell, who was also a major figure in computer science, winner of the Turing Award in NAI, and was also at CMU when I was a grad school. He talked about the architecture of cognition. He said, okay, let's not just think about, okay, we've got this task, whether it's go or learning games or spitting out answers to questions, but the ability to fluously go from one to the other. Like, you know, we're having this conversation, then later we're going to be making dinner and then we're going to cross the street, and then we're going to deal with some issue with our kids and what have you, right? Just this wild variety of things and how we seamlessly go from one to the other. We don't stop and reboot like we had to do with this podcast before we could start. We don't ask for, oh, give me a thousand label training examples. None of this AI stuff. We just do it. And that's because there's an underlying architecture, which Newell investigated under the heading of soar, a unified architecture of cognition that allows us to support that. Now, it's hard for me to see, even if I squint very hard, how an elegant and powerful technique like deep learning, back propagation, or reinforcement learning isn't merely a component in the overall cognitive architecture. It's an elegant component. It's shown its metal. But there's more to the cognitive architecture than that. There's more to human intelligence than intelligence than just adjusting the weights on these connections and getting these pieces of feedback. There's knowledge in how it's stored, there's reasoning. There's just additional complexity that. So I guess I would say to them, love what you've done, but I do think that there's more in heaven and earth than is dreamt of in your kingdom.

And do you think? I think that's a really important kind of aspect of this conversation, is the levels of uncertainty that each person has in their predictions. You know, given that folks like Hinton and Sutton advocate for this, and you've got people like Sam Altman at OpenAI who obviously thinks that this is the path forward. So given that's the case, like, obviously we each have our biases and we tend to, like, overvalue our own views. Like, you know, I have a certain way of thinking about economics and politics and, like, modern monetary theory and, like, all kinds of different things. And, like, I tend to over index on my own views in that sense. So when I make what is supposed to be, like, an objective Assessment of probabilities. I kind of have to do it like an ensemble where I try to think of, okay, there's. I have to account for like the probability that no matter how right I feel, I may be incorrect. And I guess I wonder once, like, when I do this calculation, my head. I'm sure I'm fooling myself, by the way, because, like, I'm heavily biased in this direction. So it's not going to be a coincidence that I come to this conclusion, but I feel like I come to the conclusion that there's enough area under that curve that's accounted for by like the Jeff Hintons, the Rich Suttons and so on, that, like, there's a pretty non negligible probability of, of AGI at some point in the next, let's say five to 10 years, something like that. Not huge, like we're not talking 90%, but you know, on the order of like 10, 20, something like that. First off, would you say that analysis is unfair in a particular way? And second, what do you think of those numbers?

So let me start with the fact that I feel like I can guarantee you I'll take a thousand to one bet, though the odds have to be sufficiently high at your end that that will not happen, that we will not have AGI in the next 10 years. You know, if you're willing to put down, you know, $10,000, I guess it's hard to make. My wife would not let me make bets with money I don't have.

I think that makes two of us.

But I'm just saying that's how certain I am that I would take a thousand to one bet on a very large sum. So that's point one. Point two is, I do disagree with your heuristic. I mean, you're right that we have biases. We have to be cognizant of them. But this kind of ensemble thing is very tricky because we also know from Tversky and Kahneman, right, that we have the availability heuristic, right? And certain people's voices, whether it's the late Stephen Hawking or Elon Musk, right. They kind of ring stronger. So I'm just saying. And of course, there's the chamber echo chamber effect. So as you alluded to, our kind of ensemble tends to be surprisingly supportive of our point of view. So I wouldn't go that way. Instead, what I would try to do is to just break it down and say, okay, what would need to happen? Right? So, you know, you ask an engineer, how long does it take you to Build this. They say, you know, a month and a half. You say, okay, that's an estimate. Now let's break it down. So you need to do this, you need to do this, you need to do that. We know some of those parts. And by the way, there's some problems here that some things you've never done before, we don't even know if they're soluble. Just this piece could take you 25 years. And then you say, okay, now how long does it take? You get a better estimate. So I would argue for a methodology, and by the way, I have yet to see anybody exhibit it that says, here's the roadmap. If we do A, B, C and D, we get to AI human level unknown. Nobody's even laid out the A, B, C, D and E, let alone estimate how long they take. The arguments are always, you know, based on nothing. Well, it could happen. What if you're wrong? What if something completely new comes and you didn't anticipate? Where's the roadmap to achieving AGI?

Interesting. So do you think that. Because I guess we have had. You alluded to one actually earlier, where we had these revolutions in human understanding, right. That happened very quickly. Quickly and surprisingly, too. I think probably in this case, maybe you'll disagree with this analogy. But Leo Sizzler, when he came up with the chain reaction in whatever it was, 1937 or 39, one of the 30s, I think the day before the first successful experiment, the first successful chain reaction happened in Germany, he was writing a letter basically saying, hey, I give up on this as a bad idea. I have no idea how to assess how often that could happen. But that feels like part of the problem here. Like, it feels like the unknown unknowns kind of start to compound. And I guess that's why overall, I feel very fuzzy about it. Which is. I think that's probably the best word to describe my cognitive position here. But what do you think about that historical precedent? Do you see it as relevant?

I do. So if you ask me, even though I said I'm willing to take this bet, or do you acknowledge that, that there's a chance that you're wrong, then I would say absolutely. I am not 100,000 million percent sure that I'm right. I just think so based on the reason I laid out. Let me note two points as we draw to a close. The first one is that the fact that I could be wrong for an example, like you gave, or other reasons, doesn't change the fact that, again, when I try to Bring it down the. To the data, to the components, like this roadmap. Nobody's got it. Nobody's got. There's the metadata, like, look what happened in physics, or look what happened in the 60s. You know, you said you couldn't do chess, and now we do chess easily. You know, there's these kind of analogies or conceptual arguments, but there isn't good hard data on this. In fact, for better or worse, we don't have a metric where we say, look, we're here, intelligence is here, so we're 43% of the way there or 82% of the way there. It's just. It's a more complicated thing. So that's point one. And whether I think I'm right or wrong or you think I'm right or wrong doesn't change that. A lot of the key, kind of more pedestrian building blocks aren't there. The second thing, and this is where the conversation often goes through in ours, but people often say, look, Oren, if you acknowledge that there's even a small chance that you're wrong, what about the disutility of being wrong? The disutility is effectively infinite. Because if we proceed as we do and we create this technology takes over the world, annihilates the human race, it's game over. So you have to now work towards infinite disutility. Even the probability is infinitesimally small. And this in my article which I talked about, this is an instance of a classical paradox. It's the paradox that argues that you have to be a religious Christian. And it goes like this. If there's an infinitesimally small chance that the Christian God exists and that that God is vengeful and will sentence you to an everlasting hell, unless you're a devout Christian, then you better be devout Christian, whatever your beliefs are, because if you are and it doesn't happen, well, okay, you've lost something. Maybe you gave up too many things for Lent. But if you're wrong, who wants to be in an everlasting hell? So, again, these arguments that deal with small probabilities and infinite disutilities are distorted. Because, of course, I can make the same argument that you should actually be a devout Muslim or a devout Jew or a devout atheist, right? With the same kind of thing, maybe not a devout atheist. So the point is, I think that it's really helpful to remember that the infinite disutility associated with superintelligence, the supposed infinite disutility, and that kind of uncomfortable feeling of like, hey, you know, maybe we're kind of going to let the genie out of the bottle here is being dressed up with philosophical arguments but really is an inherent and internal human anxiety and very little has changed from the Golem and Frankenstein in terms of that anxiety.

Well I think that's a great note to leave things on. I know you've got to get going. You've been very generous with your time here. But Oren, thanks so much. I really appreciate this exploration. I hope folks will check out Orin's work and writing on this topic. You've written a whole bunch of interesting stuff. I'll list it in the blog post that'll come with the podcast as well to make sure people can get it. And Oren, thanks so much. I really appreciate it.

Well Jeremy, thank you. I really appreciate your mind is not just sharp but also open and so I really appreciate to have a chance to have this dialogue with you. Thank you.

Thanks so much.