Roman Yampolskiy: Dangers of Superintelligent AI | Lex Fridman Podcast #431

If we create general superintelligences, I don't see a good outcome long term for humanity. So there is X risk, existential risk, everyone's dead, there is S risk, suffering risks, where everyone wishes they were dead. We have also idea for Iris Ikigai risks where we lost our meaning. The systems can be more creative, they can do all the jobs. It's not obvious what you have to contribute to a world where superintelligence exists. Of course you can have all the variants you mentioned where we are safe, we are kept alive, but we are not in control, we are not deciding anything. We are like animals in a zoo. There is again possibilities we can come up with as very smart humans, and then possibilities something a thousand times smarter can come up with for reasons we cannot comprehend.

The following is a conversation with Roman Yampolsky, an AI safety and security researcher and author of a new book titled AI Unexplainable, Unpredictable, Uncontrollable. He argues that there's almost 100% chance that AGI will eventually destroy human civilization. As an aside, let me say that I will have many, often technical conversations on the topic of AI, often with engineers building the state of the art AI systems. I would say those folks put the infamous P Doom or the probability of AGI killing all humans at around 1 to 20%. But it's also important to talk to folks who put that value at 70, 80, 90 and as in the case of Roman, at 99.99 and many more nines percent. I'm personally excited for the future and believe it will be a good one, in part because of the amazing technological innovation we humans create. But we must absolutely not do so with blinders on ignoring the possible risks, including existential risks, of those technologies. That's what this conversation is about. This is the Lex Friedman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Roman here Yampolsky. What to you is the probability that superintelligent AI will destroy all human civilization?

What's the time frame?

Let's say 100 years, in the next hundred years.

So the problem of controlling AGI or superintelligence in my opinion is like a problem of creating a perpetual safety machine by knowledge of a perpetual motion machine. It's impossible. Yeah, we may succeed and do a good job with GPT 5, 6, 7. But they just keep improving, learning, eventually self modifying, interacting with the environment, interacting with malevolent actors. The difference between cybersecurity, narrow AI safety and safety for general AI for superintelligence is that we don't get a second chance with cybersecurity. Somebody hacks your account, what's the big deal? You get a new password, new credit card, you move on Here. If we're talking about existential risks, you only get one chance. So you're really asking me what are the chances that we'll create the most complex software ever on the first try with zero bugs? And it will continue have zero bugs for 100 years or more.

So there is an incremental improvement of systems leading up to AGI, to you, it doesn't matter if we can keep those safe. There's going to be one level of system at which you cannot possibly control it.

I don't think we so far have made any system safe at the level of capability they display. They already have made mistakes. We had accidents, they've been jailbroken. I don't think there is a single large language model today which no one was successful at making do something developers didn't intend it to do.

But there's a difference between getting it to do something unintended, getting it to do something that's painful, costly, destructive, and something that's destructive to the level of hurting billions of people or hundreds of millions of people, Billions of people or the entirety of human civilization. That's a big leap.

Exactly. But the systems we have today have capability of causing X amount of damage. So then they fail. That's all we get. If we develop systems capable of impacting all of humanity, all of universe, the damage is proportionate.

What do you are the possible ways that such kind of mass murder of humans can happen?

It's always a wonderful question. So one of the chapters in my new book is about unpredictability. I argue that we cannot predict what a smarter system will do. So you're really not asking me how superintelligence will kill everyone. You're asking me how I would do it. And I think it's not that interesting. I can tell you about a standard nanotech, synthetic, bionuclear super intelligence will come up with something completely new, completely super. We may not even recognize that as a possible path to achieve that goal.

So there is like an unlimited level of creativity in terms of how humans could be killed. But, you know, we could still investigate possible ways of doing it, not how to do it, but the at the end, what is the methodology that does it? You know, shutting off the power and then humans start killing each other, Maybe because the resources are really constrained, that there then there's the actual use of weapons like nuclear weapons, or developing artificial pathogens, viruses, that kind of stuff, we could still kind of think through that and defend against it, right? There's a ceiling to the creativity of mass murder of humans here, right? The options are limited.

They are limited by how imaginative we are. If you are that much smarter, that much more creative, you are capable of thinking across multiple domains, do novel research in physics and biology, you may not be limited by those tools. If squirrels were planning to kill humans, they would have a set of possible ways of doing it, but they would never consider things we can come up with.

So you. Are you thinking about mass murder and destruction of human civilization? Are you thinking of. With squirrels, you put them in a zoo and they don't really know they're in a zoo. If we just look at the entire set of undesirable trajectories, majority of them are not going to be death. Most of them are going to be just like things like Brave New World, where, you know, the squirrels are fed dopamine and they're all like, doing some kind of fun activity and the sort of the fire, the soul of humanity is lost because of the drug that's fed to it. Or like, literally in a zoo. We're in a zoo, we're doing our thing. We're like playing a game of Sims, and the actual players playing that game are AI systems. Those are all undesirable because sort of the free will, the fire of human consciousness is dimmed through that process, but it's not killing humans. So are you thinking about that, or is the biggest concern literally the extinctions of humans?

I think about a lot of things. So there is X risk, existential risk, everyone's dead, There is S risk, suffering risks, where everyone wishes liver dead. We have Also idea for iRisk, iKigai risks where we lost our meaning. The systems can be more creative, they can do all the jobs. It's not obvious what you have to contribute to a world where superintelligence exists. Of course, you can have all the variants you mentioned where we are safe, we are kept alive, but we are not in control. We are not deciding anything. We are like animals in a zoo. There is again, possibilities we can come up with as very smart humans, and then possibilities something a thousand times smarter can come up with for reasons we cannot comprehend.

I would love to sort of dig into each of those X risk, S risk and I risk. So can you, like, linger on? Irisk, what is that?

So Japanese concept of ikigai, you find something which Allows you to make money, you are good at it and the society says we need it. So, like, you have this awesome job, you are podcaster, gives you a lot of meaning. You have a good life, I assume you're happy. That's what we want most people to find to have. For many intellectuals, it is their occupation which gives them a lot of meaning. I am a researcher, philosopher, scholar. That means something to me. In a world where an artist is not feeling appreciated because his art is just not competitive with what is produced by machines or writer or scientist will lose a lot of that. And at the lower level, we're talking about complete technological unemployment. We're not losing 10% of jobs, we're losing all jobs. What do people do with all that free time? What happens then? Everything society is built on is completely modified in one generation. It's not a slow process where we get to kind of figure out how to live that new lifestyle, but it's.

Pretty quick in that world. Can't humans do what humans currently do with chess? Play each other, have tournaments? Even though AI systems are far superior at this time in chess, so we just create artificial games, or for us, they're real, like the Olympics. And we do all kinds of different competitions and have fun, maximize the fun and let the AI focus on the productivity.

It's an option. I have a paper where I try to solve the value alignment problem for multiple agents. And the solution to avoid compromise is to give everyone a personal virtual unit universe. You can do whatever you want in that world. You could be king, you could be slave, you decide what happens. So it's basically a glorified video game where you get to enjoy yourself and someone else takes care of your needs. And the substrate alignment is the only thing we need to solve. We don't have to get 8 billion humans to agree on anything.

So, okay, so why is that not a likely outcome? Why can't AI systems create video games for us to lose ourselves in each with an individual video game universe?

Some people say that's what happened. We're in a simulation and we're playing that video game.

And now we're creating what? Maybe we're creating artificial threats for ourselves to be scared about, because fear is really exciting. It allows us to play the video game more vigorously.

And some people choose to play on a more difficult level with more constraints. Some say, okay, I'm just going to enjoy the game. High privilege level. Absolutely.

So, okay, what was that paper on multi agent value alignment?

Personal universes. Personal universes.

So that's One of the possible outcomes. But what in general is the idea of the paper? So it's looking at multiple agents that are human AI, like a hybrid system, whether it's humans and AIs or is it looking at humans or just intelligent.

Agents in order to solve value alignment problem, trying to formalize it a little better. Usually we're talking about getting AIs to do what we want, which is not well defined. Are we talking about creator of a system, owner of that AI, humanity as a whole. But we don't agree on much. There is no universally accepted ethics, morals, across cultures, religions. People have individually very different preferences politically and such. So even if we somehow managed all the other aspects of it, programming those fuzzy concepts in, getting AI to follow them closely, we don't agree on what to program in. So my solution was, okay, we don't have to compromise on room temperature. You have your universe, I have mine, whatever you want. And if you like me, you can invite me to visit your universe. We don't have to be independent, but the point is you can be. And virtual reality is getting pretty good. It's going to hit a point where you can't tell the difference. And if you can't tell if it's real or not, what's the difference?

So basically, give up on value alignment, create an entire. It's like the multiverse theory. This is create an entire universe for you. What your values, you still have to.

Align with that individual. They have to be happy in that simulation. But it's a much easier problem to align with 1 agent versus 8 billion agents plus animals, aliens.

So you convert the multi agent problem into a single agent problem.

I'm trying to do that, yeah.

Okay. Is there any way to. So okay, that's giving up on the value alignment problem. Well, is there any way to solve the value alignment problem where there's a bunch of humans, multiple humans, tens of humans, or 8 billion humans that have very different set of values?

It seems contradictory. I haven't seen anyone explain what it means outside of kind of words which pack a lot. Make it good, make it desirable, make it something they don't regret. But how do you specifically formalize those notions? How do you program them in? I haven't seen anyone make progress on that so far.

But isn't that the whole optimization journey that we're doing as a human civilization? We're looking at geopolitics. Nations are in a state of anarchy with each other. They start wars, there's conflict, and oftentimes they have very different views of what is good and what is evil. Isn't that what we're trying to figure out? Just together trying to converge towards that? So we're essentially trying to solve the value alignment problem with humans.

Right, but the examples you gave, some of them are, for example, two different religions saying this is our holy side and we are not willing to compromise it in any way. If you can make two holy sites in virtual worlds, you solve the problem. But if you only have one, it's not divisible. You're kind of stuck there.

But what if we want to be attention with each other and through that tension we understand ourselves and we understand the world? So that's the intellectual journey we're on as a human civilization is we create intellectual and physical conflict and through that figure stuff out.

If we go back to that idea of simulation and this is entertainment kind of giving meaning to us. The question is how much suffering is reasonable for a video game. So yeah, I don't mind, you know, a video game where I get haptic feedback, There is a little bit of shaking, maybe I'm a little scared. I don't want a game where like kids are tortured literally. That seems unethical, at least by our human standards.

Are you suggesting it's possible to remove suffering? If we're looking at human civilization as an optimization problem.

So we know there are some humans who, because of a mutation, don't experience physical pain. So at least physical pain can be mutated out re engineered out suffering in terms of meaning. Like you burn. The only copy of my book is a little harder. But even there you can manipulate your hedonic set point. You can change. Change defaults, you can reset. Problem with that is if you start messing with your reward channel, you start wireheading and end up blissing out a little too much.

Well, that's the question. Would you really want to live in a world where there's no suffering? That's a dark question. Is there some level of suffering that reminds us of what this is all for?

I think we need that. But I would change the overall range. So right now it's negative infinity to kind of positive infinity. Pain, pleasure axis. I would make it like zero to positive infinity. And being unhappy is like I'm close to zero.

Okay, so what's the S risk? What are the possible things that you're imagining with S risk? So mass suffering of humans, what are we talking about there? Caused by AGI.

So there are many malevolent actors we can talk about psychopaths, crazies, hackers, Doomsday cults, we know from history, they tried killing everyone. They tried on purpose to cause maximum amount of damage, terrorism. What if someone malevolent wants on purpose, to torture all humans as long as possible? You solve aging, so now you have functional immortality and you just try to be as creative as you can.

Do you think there is actually people in human history that tried to literally maximize human suffering? In just studying people who have done evil in the world, it seems that they think that they're doing good. And it doesn't seem like they're trying to maximize suffering. They just cause a lot of suffering as a side effect of doing what they think is good.

So there are different malevolent agents, some maybe just gaining personal benefit and sacrificing others to that cause. Others we know for a fact are trying to kill as many people as possible. When we look at recent school shootings, if they had more capable weapons, they would take out not dozens, but thousands. Millions. Billions.

Well, we don't know that, but that is a terrifying possibility and we don't want to find out. Like, if terrorists had access to nuclear weapons, how far would they go? Is there a limit to what they're willing to do? In your senses? There's some malevolent actors where there's no limit.

There is mental diseases where people don't have empathy, don't have this human quality of understanding suffering and others.

And then there's also a set of beliefs where you think you're doing good by killing a lot of humans.

Again, I would like to assume that normal people never think like that. It's always some sort of psychopaths. But yeah.

And to you, AGI systems can carry that and be more competent at executing that.

They can certainly be more creative. They can understand human biology better, understand our molecular structure, genome. Again, a lot of times torture ends, then individual dies. That limit can be removed as well.

So if we're actually looking at X risk and S risk as the systems get more and more intelligent, don't you think it's possible to anticipate the ways they can do it and defend against it, like we do with the cyber security? We'll do security systems, right?

We can definitely keep up for a while. I'm saying you cannot do it indefinitely. At some point, the cognitive gap is too big. The surface you have to defend is infinite. But attackers only need to find one exploit.

So do you eventually. This is. We're heading off a cliff.

If we create general superintelligences, I don't see a good outcome Long term for humanity. The only way to win this game is not to play it.

Okay, well, we'll. We'll talk about possible solutions and what not playing it means. But what are the possible timelines here to you? What are we talking about? We're talking about a set of years. Decades, Centuries. What do you think?

I don't know for sure. The prediction markets right now are saying 2026 for AGI. I heard the same thing from CEO of Anthropic DeepMind. So maybe we are two years away, which seems very soon, given we don't have a working safety mechanism in place or even a prototype for one. And there are people trying to accelerate those timelines because they feel we're not getting there quick enough.

Well, what do you think they mean when they say AGI?

So the definitions we used to have, and people are modifying them a little bit lately. Artificial general intelligence was a system capable of performing in any domain a human could perform. So kind of you're creating this average artificial person. They can do cognitive labor, physical labor, where you can get another human to do it. Superintelligence was defined as a system which is superior to all humans in all domains. Now people are starting to refer to AGI as if it's superintelligence. I made a post recently where I argued, for me at least, if you average out over all the common human tasks, those systems are already smarter than an average human. So under that definition, we have it. Shane Leg has this definition of where you're trying to win in all domains. That's what intelligence is. Now, are they smarter than elite individuals in certain domains? Of course not. They're not there yet. But the progress is exponential.

See, I'm much more concerned about social engineering. So to me, AI's ability to do something in the physical world like the lowest hanging fruit. The easiest set of methods is by just getting humans to do it. It's going to be much harder to be the kind of viruses that take over the minds of robots, where the robots are executing the commands. It just seems like humans. Social engineering of humans is much more likely.

That would be enough to bootstrap the whole process.

Okay, just to linger on the term AGI, what to you is the difference in AGI and human level intelligence?

Human level is general. In the domain of expertise of humans, we know how to do human things. I don't speak dog language. I should be able to pick it up. If I'm a general intelligence, it's kind of inferior animal. I should be able to learn that skill But I can't. A general intelligence, truly universal general intelligence should be able to do things like that humans cannot do.

To be able to talk to animals.

For example, to solve pattern recognition problems of that type. To have similar things outside of our domain of expertise. Because it's just not the world we live in.

If we just look at the space of cognitive abilities we have. I just would love to understand what the limits are beyond which an AGI system can reach. Like what does that look like? What about actual mathematical thinking or scientific innovation, that kind of stuff.

We know calculators are smarter than humans in that narrow domain of addition.

But is it humans plus tools versus AGI or just human raw human intelligence? Because humans create tools and with the tools they become more intelligent. So like there's a gray area there what it means to be human when we're measuring their intelligence.

So when I think about it, I usually think human with like a paper and a pencil. Not human with Internet and another AI helping.

But is that a fair way to think about it? Because isn't there another definition of human level intelligence that includes the tools that humans create?

But we create AI, so at any point you'll still just add superintelligence to human capability. That seems like cheating.

No controllable tools. There is an implied leap that you're making when AGI goes from tool to entity that can make its own decisions. So if we define human level intelligence as everything a human can do with fully controllable tools, it seems like a.

Hybrid of some kind. You're now doing brain computer interfaces, you're connecting it to maybe narrow AIs yet definitely increases our capabilities.

So what's a good test to you that measures whether an artificial intelligence system has reached human level intelligence? And what's a good test where it has superseded human level intelligence to reach that land of AGI?

I am old fashioned. I like Turing Tests. I have a paper where I equate passing Turing Tests to solving AI complete problems. Because you can encode any questions about any domain into the Turing Test. You don't have to talk about how was your day. You can ask anything. And so the system has to be as smart as a human to pass it in a true sense.

But then you would extend that to maybe a very long conversation. I think the Alexa prize was doing that basically. Can you do a 20 minute, 30 minute conversation with any ass system?

It has to be long enough to where you can make some meaningful decisions about capabilities. Absolutely. You can brute force very short conversations.

So like literally what does that look like. Can we construct formally a kind of test that tests for AGI?

For AGI? It has to be there. I cannot give it a task I can give to a human and it cannot do it. If a human can for super intelligence, it would be superior on all such tasks, not just average performance. Go learn to drive car, Go speak Chinese, play guitar. Okay, great.

I guess the follow on question Is there a test for the kind of AGI that would be susceptible to lead to S risk or X risk? Susceptible to destroy human civilization? Like, is there a test for that?

You can develop a test which will give you positives if it lies to you or has those ideas. You cannot develop a test which rules them out. There is always possibility of what Bostrom calls a treacherous turn, where later on a system decides for game theoretic reasons, economic reasons, to change its behavior. And we see the same with humans. It's not unique to AI for for millennia we tried developing morals, ethics, religions, lie detector tests, and then employees betray the employer, spouses betray family. It's a pretty standard thing intelligent agents sometimes do.

So is it possible to detect when an AI system is lying or deceiving you?

If you know the truth and it tells you something false, you can detect that, but you cannot know in general every single time. And again, the system you're testing today may not be lying. The system you're testing today may know you are testing it and so behaving. And later on, after it interacts with the environment, interacts with other systems, malevolent agents learns more, it may start doing those things.

So do you think it's possible to develop a system where the creators of the system, the developers, the programmers don't know that it's deceiving them?

So systems today don't have long term planning. That is not our. They can lie today if it optimizes, helps them optimize the reward. If they realize, okay, this human will be very happy if I tell them the following. They will do it if it brings them more points and they don't have to kind of keep track of it. It's just the right answer to this problem every single time.

At which point is somebody creating that intentionally, not unintentionally intentionally, creating an AI system that's doing long term planning with an objective function that's defined by the AI system, not by a human?

Well, some people think that if they're that smart, they're always good. They really do believe that it's just benevolence from intelligence. So they'll Always want what's best for us. Some people think that they will be able to detect problem behaviors and correct them at the time when we get there. I don't think it's a good idea. I am strongly against it. But yeah, there are quite a few people who in general are so optimistic about this technology, it could do no wrong. They want it developed as soon as possible, as capable as possible.

So there's going to be people who believe the more intelligent it is, the more benevolent. And so therefore it should be the one that defines the objective function that it's optimizing when it's doing long term planning.

There are even people who say, okay, what's so special about humans? Right. We removed the gender bias, we're removing race bias. Why is this pro human bias? We are polluting the planet. We are, as you said, you know, fight a lot of wars, kind of violent. Maybe it's better if a super intelligent perfect society comes and replaces us. It's normal stage in the evolution of our species.

Yeah. So somebody says, let's develop an AI system that removes the violent humans from the world. And then it turns out that all humans have violence in them or the capacity for violence, and therefore all humans are removed. Yeah, yeah, yeah. Let me ask about Yann Lecun. He's somebody who you've had a few exchanges with and he's somebody who actively pushes back against this view that AI is going to lead to destruction of human civilization, also known as AI doomerism. So in a one example that he tweeted, he said, I do acknowledge risks, but two points. One, open research and open source are the best ways to understand and mitigate the risks. And two, AI is not something that just happens. We build it. We have agency in what it becomes, hence we control the risks. We meaning humans. It's not some sort of natural phenomena that we have no control over. So can you, can you make the case that he's right and can you try to make the case that he's wrong?

I cannot make a case that he's right. He's wrong in so many ways, it's difficult for me to remember all of them. He is a Facebook buddy, so I have a lot of fun having those little debates with him. So I'm trying to remember the arguments. So one, he says, we are not gifted this intelligence from aliens. We are designing it, we are making decisions about it. That's not true. It was true when we had expert systems, symbolic AI decision trees. Today you set up parameters for a model and you water this plant, you give it data, you give it compute, and it grows. And after it's finished growing into this alien plant, you start testing it to find out what capabilities it has. And it takes years to figure out. Even for existing models, if it's trained for six months, it will take you two, three years to figure out basic capabilities of that system. We still discover new capabilities in systems which are already out there. So that's not the case.

So just to linger on that, to you, the difference there, that there is some level of emergent intelligence that happens in our current approaches. So stuff that we don't hard code in.

Absolutely. That's what makes it so successful. Then we had to painstakingly hard code in everything. We didn't have much progress. Now just spend more money and more compute and it's a lot more capable.

And then the question is, when there is emergent intelligent phenomena, what is the ceiling of that for you? There's no ceiling. For Yann Lecun, I think there's a kind of ceiling that happens that we have full control over. Even if we don't understand the internals of the emergence, how the emergence happens, there's a sense that we have control and understanding of the approximate ceiling of capability, the limits of the capability.

Let's say there is a ceiling. It's not guaranteed to be at the level which is competitive with us. It may be greatly superior to ours.

So what about his statement about open research and open source are the best ways to understand and mitigate the risks?

Historically, he's completely right. Open source software is wonderful. It's tested by the community, it's debugged, but we're switching from tools to agents. Now you're giving open source weapons to psychopaths. Do we want to open source nuclear weapons, biological weapons? It's not safe to give technology so powerful to those who may misalign it, even if you are successful at somehow getting it to work in the first place in a friendly manner.

But the difference with nuclear weapons, current AI systems are not akin to nuclear weapons. So the idea there is you're open sourcing it at this stage that you can understand it better. Large number of people can explore the limitations and capabilities, explore the possible ways to keep it safe, to keep it secure, all that kind of stuff, while it's not at the stage of nuclear weapons. So nuclear weapons, there's a no nuclear weapon and then there's a nuclear weapon. With AI systems, there's a gradual improvement of capability and you get to perform that improvement Incrementally. So open source allows you to study how things go wrong, study the very process of emergence, study AI safety on those systems when there's not a high level of danger, all that kind of stuff.

It also sets a very wrong precedent. So we Open sourced Model 1, Model 2, Model 3. Nothing ever bad happened. So up obviously we're going to do it with model four. It's just gradual improvement.

I don't think it always works with the precedent. Like you're not stuck doing it the way you always did. It sets a precedent of open research and open development, such that we get to learn together. And then the first time there's a sign of danger, some dramatic thing happened. Not a thing that destroys human civilization, but some dramatic demonstration of capability that can legitimately lead to a lot of damage. Then everybody wakes up and says, okay, we need to regulate this. We need to come up with safety mechanism that stops this right at this time. Maybe you can educate me. But I haven't seen any illustration of significant damage done by intelligent AI systems.

So I have a paper which collects accidents through history of AI and they always are proportionate to capabilities of that system. So if you have tic tac toe playing AI, it will fail to properly play and loses the game, which it should draw trivial. Your spell checker will misspell a word, so on. I stopped collecting those because there are just too many examples of AIs failing at what they are capable of. We haven't had terrible accidents in the sense of billion people got killed. Absolutely true. But in another paper I argue that those accidents do not actually prevent people from continuing with research. And actually they kind of serve like vaccines. A vaccine makes your body a little bit sick, so you can handle the big disease later much better. It's the same here. People will point out, you know that accident, AI accident we had where 12 people died, everyone's still here. 12 people is less than smoking kills. It's not a big deal. So we continue. So in a way it will actually be kind of confirming that it's not that bad.

It matters how the deaths happen. Whether it's literally murder by the AI system, then one is a problem. But if it's accidents because of increased reliance on automation, for example, so when airplanes are flying in an automated way, maybe the number of plane crashes increased by 17% or something. And then you're like, okay, do we really want to rely on automation? I think in the case of automation airplanes, it decreased significantly. Okay, Same thing with autonomous vehicles. Like, okay, what are the Pros and cons, what are the trade offs here? You can have that discussion in an honest way. But I think the kind of things we're talking about here is mass scale pain and suffering caused by AI systems. And I think we need to see illustrations of that in a very small scale to start to understand that this is really damaging versus Clippy, versus a tool that's really useful to a lot of people to do learning to do summarization of text, to do question, answer, all that kind of stuff to generate videos. The tool, fundamentally a tool versus an agent that can do a huge amount of damage.

So you bring up example of cars. Yes, cars were slowly developed and integrated. If we had no cars and somebody came around and said, I invented this thing, it's called Cars, it's awesome, it kills like 100,000Americans every year, let's deploy it. Would we deploy that?

There's been fear mongering about cars for a long time. The transition from horses to cars. There's a really nice channel that I recommend people check out Pessimist archive that documents all the fear mongering about technology that's happened throughout history. There's definitely been a lot of fear mongering about cars. There's a transition period there about cars, about how deadly they are. We can try. It took a very long time for cars to proliferate to the degree they have now. And then you could ask serious questions in terms of the miles traveled, the benefit to the economy, the benefit to the quality of life that cars do versus the number of deaths, 30, 40,000 in the United States. Are we willing to pay that price? I think most people, when they're rationally thinking policymakers, will say, yes, we want to decrease it from 40,000 to zero and do everything we can to decrease it. There's all kinds of policies, incentives you can create to decrease the risks with the deployment of this technology. But then you have to weigh the benefits and the risks of the technology. And the same thing would be done with AI.

You need data, you need to know. But if I'm right, and it's unpredictable, unexplainable, uncontrollable, you cannot make this decision. We're gaining $10 trillion of wealth, but we're losing, we don't know how many people. You basically have to perform an experiment on 8 billion humans without their consent. And even if they want to give you consent, they can't because they cannot give informed consent. They don't understand those things.

Right. That happens when you do. When you go from the Predictable to the unpredictable. Very quickly you just. But it's not obvious to me that AI systems would gain capability so quickly that you won't be able to collect enough data to study the benefits and the risks.

We literally doing it. The previous model we learned about after we finished training it what it was capable of. Let's say we stopped GPT4 training run around human capability. Hypothetically we start training GPT5 and I have no knowledge of insider training runs or anything. And we start at that point of about human and we train it for the next nine months, maybe two months in, it becomes super intelligent. We continue training it. At the time when we start testing it, it is already a dangerous system. How dangerous, I have no idea.

But neither people training it at the training stage. But then there's a testing stage inside the company. They can start getting intuition about what the system is capable to do. You're saying that somehow from leap from GPT4 to GPT5 can happen? The kind of leap where GPT4 was controllable and GPT5 is no longer controllable. And we get no insights from using GPT4 about the fact that GPT5 will be uncontrollable. Like that's the, that's the situation you're concerned about. Where their leap from N to N plus 1 would be such that uncontrollable system is created without any ability for us to anticipate that if we had.

Capability of ahead of the run before the training run to register exactly what capabilities that next model will have at the end of a training run and we accurately guessed all of them, I would say you're right. We can definitely go ahead with this run. We don't have that capability from GPT4.

You can build up intuitions about what GPT5 will be capable of. It's just incremental progress. Even if that's a big leap in capability, it just doesn't seem like you can take a leap from a system that's helping you write emails to a system that's going to destroy human civilization. It seems like it's always going to be sufficiently incremental such that we can anticipate the possible dangers. And we're not even talking about existential risk, but just the the kind of damage it can do to civilization. It seems like we'll be able to anticipate the kinds. Not the exact, but the kinds of risks it might lead to and then rapidly develop defenses ahead of time. And as the risks emerge, we're not.

Talking just about Capabilities, specific tasks. We're talking about general capability to learn, maybe like child. At the time of testing and deployment, it is still not extremely capable. But as it is exposed to more data, real world, it can be trained to become much more dangerous and capable.

So let's focus then on the control problem. At which point does the system become uncontrollable? Why is it the more likely trajectory for you that the system becomes uncontrollable?

So I think at some point it becomes capable of getting out of control for game theoretic reasons. It may decide not to do anything right away and for a long time just collect more resources, accumulate strategic advantage right away. It may be kind of still young, weak superintelligence. Give it a decade, it's in charge of a lot more resources. It had time to make backups. So it's not obvious to me that it will strike as soon as it can.

Can we just try to imagine this future where there's an AI system that's capable of escaping the control of humans and then doesn't. And waits, what's that look like? So one, we have to rely on that system for a lot of the infrastructure. So we'll have to give it access not just to the Internet, but to the task of managing power, government, economy, this kind of stuff. And that just feels like a gradual process given the bureaucracies of all those systems involved.

We've been doing it for years. Software controls all the systems. Nuclear power plants, airline industry, it's all software based. Every time there is electrical outage, I can't fly anywhere for days.

But there's a difference between software and AI. There's different kinds of software. So to give a single AI system access to the control of airlines and the control of the economy, that's not a, that's not a trivial transition for humanity.

No, but if it shows it is safer, in fact, then it's in control, we get better results. People will demand that it was put in place.

Absolutely.

And if not, it can hack the system. It can use social engineering to get access to it. That's why I said it might take some time for it to accumulate those resources.

Feels like that would take a long time for either humans to trust it or for the social engineering to come into play. Like it's not a thing that happens overnight. It feels like something that happens across one or two decades.

I really hope you're right, but it's not what I'm seeing. People are very quick to jump on the latest trend. Early adopters will be There before it's even deployed.

Buying prototypes, maybe the social engineering I could see. Because so for social engineering, AI systems don't need any hardware access, it's all software. So they can start manipulating you through social media and so on. Like you have AI assistants, they're going to help you do a lot of manage a lot of your day to day and then they start doing social engineering. But for a system that's so capable that it can escape the control of humans that created it, such a system being deployed at a mass scale and trusted by people to be deployed, it feels like that would take a lot of convincing.

So we've been deploying systems which had hidden capabilities.

Can you give an example?

GPT 4. I don't know what else it's capable of, but there are still things we haven't discovered can do. They may be trivial, proportionate to its capability. I don't know it writes Chinese poetry. Hypothetical, I know it does. But we haven't tested for all possible capabilities and we are not explicitly designing them. We can only rule out bugs we find. We cannot rule out bugs and capabilities because we haven't found them.

Is it possible for a system to have hidden capabilities that are orders of magnitude greater than its non hidden capabilities? This is the thing I'm really struggling with. Where on the surface the thing we understand it can do doesn't seem that harmful. So if even if it has bugs, even if it has hidden capabilities like Chinese poetry or generating effective viruses, software viruses, the damage that can do seems like on the same order of magnitude as it's the, the capabilities that we know about. So like this, this idea that the hidden capabilities will include being uncontrollable. This is something I'm struggling with because GPT4 on the surface seems to be very controllable.

Again, we can only ask and test for things we know about. If there are unknown unknowns, we cannot do it. I'm thinking of human statistics, events, right? If you talk to a person like that, you may not even realize they can multiply 20 digit numbers in their head. You have to know to ask.

So as I mentioned, just to sort of linger on the, the fear of the unknown. So the Pessimist Archive has just documented, let's look at data of the past, at history. There's been a lot of fear mongering about technology. Pessimist Archive does a really good job of documenting how crazily afraid we are of every piece of technology we've been afraid. There's a blog post where Louis Anslow who created Pessimist Archive writes about the fact that we've been fear mongering about robots and automation for over 100 years. So why is AGI different than the kinds of technologies we've been afraid of in the past?

So two things. One, we're switching from tools to agents. Tools don't have negative or positive impact. People using tools do. So guns don't kill, people with guns do. Agents can make their own decisions. They can be positive or negative. A pit bull can decide to harm you. It's an agent. The fears are the same. The only difference is now we have this technology. Then they were afraid of humanoid robots. A hundred years ago they had none. Today every major company in the world is investing billions to create them. Not every. But you understand what I'm saying. It's very different.

Well, agents. It depends what you mean by the word agents. All those companies are not investing in a system that has the kind of agency that's implied by in the fears where it can really make decisions on their own. They have no human in the loop.

They are saying they are building super intelligence and have a super alignment team. You don't think they are trying to create a system smart enough to be an independent agent under that definition?

I have not seen evidence of it. I think a lot of it is a marketing kind of discussion about the future and it's a mission about the kind of systems we can create in the long term future. But in the short term, the kind of systems they're creating falls fully within the definition of narrow AI. These are tools that have increasing capabilities but they just don't have a sense of agency or consciousness or self awareness or ability to deceive at scales that would be required to do like mass scale suffering and murder of humans.

Those systems are well beyond narrow AI. If you had to list all the capabilities of GPT4, you would spend a lot of time time writing that list.

But agency is not one of them.

Not yet. But do you think any of those companies are holding back because they think it may be not safe? Or are they developing the most capable system they can, given the resources and hoping they can control and monetize.

Control and monetize. Hoping they can control, monetize. So you're saying if they could press a button and create an agent that they no longer control, that they can have to ask nicely a thing that lives on a server across huge number of computers. You're saying that they would push for the creation of that kind of system?

I mean I can't speak for other people, for all of them. I think some of them are very ambitious. They fundraising trillions. They talk about controlling the light corner of the universe. I would guess that they might.

Well, that's a human question. Whether humans are capable of that. Probably some humans are capable of that. My more direct question, if it's possible to create such a system, have a system that has that level of agency. I don't think that's an easy technical challenge. We're not. It doesn't feel like we're close to that. A system that has the kind of agency where it can make its own decisions and deceive everybody about them. The current architecture we have in machine learning and how we train the systems, how to deploy the systems.