Eliezer Yudkowsky: Dangers of AI and the End of Human Civilization | Lex Fridman Podcast #368

The problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that the entire thing is going to be like way more difficult than we realized at the start. Because the first time you fail at aligning something much smarter than you are, you die.

The following is a conversation with Eliezer Yudkowski, a legendary researcher, writer and philosopher on the topic of artificial intelligence, especially superintelligent AGI and its threat to human civilization. This is the Lex Friedman podcast to support it. Please check out our sponsors in the description. And now, dear friends, here's Eliezer Yadkovsky. What do you think about GPT4? How intelligent is is a bit smarter.

Than I thought this technology was going to scale to. And I'm a bit worried about what the next one will be like. Like this particular one I think, I hope there's nobody inside there because, you know, be sucked to be stuck inside there. But we don't even know the architecture at this point because OpenAI is very properly not telling us. And yeah, like giant inscrutable matrices of floating point numbers. I don't know what's going on in there. Nobody goes knows what's going on in there. All we have to go by are the external metrics. And on the external metrics, if you ask it to write a self aware 4chan green text, it will start writing a green text about how it has realized that it's an AI writing a green text. And like, oh well, so that's probably not quite what's going on in there in reality, but we're kind of like blowing past all these science fiction guardrails. Like we are past the point where in science fiction people would be like, whoa, wait, stop, that thing's alive. What are you doing to it? And it's probably not. Nobody actually knows. We don't have any other guardrails, we don't have any other tests, we don't have any lines to draw on the sand and say like, well, when we get this far we will start to worry about what's inside there. So if it were up to me, I would be like, okay, this far, no further time for the summer of AI where we have planted our seeds and now we wait and reap the rewards of the technology you've already developed and don't do any larger training runs than that. Which to be clear, I realize requires more than one company agreeing to not do that.

And take a rigorous approach for the whole AI community to investigate whether there's somebody inside there.

That would take decades, like having any idea of what's going on in there. People have been trying for a while.

It's a poetic statement about if there's somebody in there. But I feel like it's also a technical statement, or I hope it is one day, which is a technical statement that Alan Turing tried to come up with with the Turing Test. Do you think it's possible to definitively or approximately figure out if there is somebody in there, if there's something like a mind inside this large language model?

I mean, there's a whole bunch of different sub questions here. There's the question of like, is there consciousness? Is there qualia? Is this a object of moral concern? Is the same moral patient, like, should we be worried about how we're treating it? And then there's questions like, how smart is it? Exactly. Can it do X, can it do Y? And we can check how it can do X and how it can do Y. Unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the Internet, which means that when it talks about being self aware, we don't know to what extent it is repeating back what it has previously been trained on for discussing self awareness, or if there's anything going on in there such that it would start to say similar things spontaneously. Among the things that one could do if one were at all serious about trying to figure this out, is train GPT3 to detect conversations about consciousness, exclude them all from the training data sets, and then retrain something around the rough size of GPT4 and no larger, with all of the discussion of consciousness and self awareness and so on missing. Although, hard bar to pass, humans are self aware. We're self aware all the time. We talk about what we do all the time, like what we're thinking at the moment all the time, but nonetheless get rid of the explicit discussion of consciousness, I think therefore I am, and all that, and then try to interrogate that model and see what it says. And it still would not be definitive, but nonetheless, I don't know. I feel like when you run over the science fiction guardrails, like maybe this thing, but what about GPT? Maybe not this thing, but like, what about GPT5? This would be a good place to pause.

On the topic of cautiousness. You know, there's so many components to even just removing consciousness from the data set. Emotion, the display of consciousness, the display of emotion feels like deeply integrated with the experience of consciousness. So the hard problem seems to be Very well integrated with the actual surface level illusion of consciousness. So displaying emotion, I mean, do you think there's a case to be made that we humans, when we're babies are just like GPT? That we're training on human data on how to display emotion versus feel emotion, how to show others, communicate others, that I'm suffering, that I'm excited, that I'm worried, that I'm lonely and I missed you and I'm excited to see you. All of that is communicated. That's a communication skill versus the actual feeling that I experience. So we need that training data as humans too, that we may not be born with that, how to communicate the internal state. And that's in some sense, if we remove that from GPT4's data set, it might still be conscious, but not be able to communicate it.

So I think you're going to have some difficulty removing all mention of emotions from GPT's data set. I would be relatively surprised to find that it has developed exact analogs of human emotions in there. I think that humans will have emotions even if you don't tell them about those emotions when they're kids. It's not quite exactly what various blank slatists tried to do with the new Soviet men and all that, but if you try to raise people perfectly altruistic, they still come out selfish. You try to raise people sexless, they still develop sexual attraction. We have some notion in humans, not in AIs, of where the brain structures are that implement this stuff. And it is really remarkable thing I say in passing, that despite having complete read access to every floating point number in the GPT series, we still know vastly more about the architecture of human thinking than we know about what goes on inside GPT. Despite having vastly better ability to read.

GPT, do you think it's possible, do you think that's just a matter of time? Do you think it's possible to investigate and study the way neuroscientists study the brain, which is look into the darkness, the mystery of the human brain. By just desperately trying to figure out something and to form models, and then over a long period of time, actually start to figure out what regions of the brain do certain things, what different kinds of neurons when they fire, what that means, how plastic the brain is, all that kind of stuff, you slowly start to figure out different properties of the system. Do you think we can do the same thing with language models?

Sure. I think that if half of today's physicists stop wasting their lives on string theory or whatever and go off and study what goes on inside transformer Networks, then in 30, 40 years, we'd probably have a pretty good idea.

Do you think these large language models can reason?

They can play chess. How are they doing that without reasoning?

So you're somebody that spearheaded the movement of rationality. So reason is important to you. So is a powerful important word. Or is it like, how difficult is the threshold of being able to reason to you? And how impressive is it?

I mean, in my writings on rationality, I have not gone making a big deal out of something called reason. I have made more of a big deal out of something called probability theory. And that's like, well, you're reasoning, but you're not doing it quite right and you should reason this way instead of. And interestingly, people have started to get preliminary results showing that reinforcement learning by human feedback has made the GPT series worse in some ways. In particular, like it used to be well calibrated. If you trained it to put probabilities on things, it would say 80% probability and be right eight times out of 10. And if you apply reinforcement learning from human feedback, the nice graph of 70%, 7 out of 10 sort of flattens out into the graph that humans use where there's some very improbable stuff and likely probable, maybe, which all means around 40% and then certain. So it used to be able to use probabilities. But if you apply probability, but if you try to teach it to talk in a way that satisfies humans, it gets worse at probability in the same way that humans are.

And that's a bug, not a feature.

I would call it a bug, although such a fascinating bug, but yeah. So reasoning, it's doing pretty well on various tests that people used to say would require reasoning. But, um, you know, rationality is about. When you say 80%, does it happen eight times out of 10?

So what are the limits to you of these transformer networks of. Of neural networks? What's if. If reasoning is not impressive to you, or it is impressive, but there's other levels to achieve.

I mean, it's just not how I carve up reality.

What's if reality is a cake? What are the different layers of the cake or the slices? How do you cover it? You can use a different food if you like.

I don't think it's as smart as a human yet. I do. Like back in the day I went around saying, I do not think that just stacking more layers of transformers is going to get you all the way to AGI. And I think that GPT4 is past or I thought this paradigm was going to take us and you want to notice when that happens. You want to say like whoops, well, I guess I was incorrect about what happens if you keep on stacking more transformer layers. And that means I don't necessarily know what GPT5 is going to be able to do.

That's a powerful statement. So you're saying like your intuition initially is now appears to be wrong.

Yeah.

It's good to see that you can admit in some of your predictions to be wrong. You think that's important to do because you make several very throughout your life you've made many strong predictions and statements about reality and you evolve with that. So maybe that'll come up today about our discussion. So you're okay being wrong?

I'd rather not be wrong next time. It's a bit ambitious to go through your entire life never having been wrong. One can aspire to be well calibrated, not so much think in terms of was I right, was I wrong? But when I said 90% that it happened nine times out of 10. Yeah, like oops is the sound we make, is the sound we emit when we improve.

Beautifully said. And somewhere in there we can connect the name of your blog Less Wrong. I suppose that's the objective function.

The name Less Wrong was I believe suggested by Nick Bostrom and it's after someone's epigraph actually forget whose, who said we never become right, we just become less wrong. What's the something? Something easy to confess. Just err and error and err again, but less and less and less.

Yeah, that's a good thing to strive for. So what has surprised you about GPT4 that you found beautiful? As a scholar of intelligence, of human intelligence, of artificial intelligence, of the human.

Mind, I mean the beauty does interact with the screaming. Horror.

Is the beauty in the horror.

But like beautiful moments. Well, somebody asked Bing Sydney to describe herself and fed the resulting description into one of the stable diffusion things, I think. And she's pretty. And this is something that should have been an amazing moment. The AI describes herself, you get to see what the AI thinks the AI looks like. Although the thing that's doing the drawing is not the same thing that's outputting the text. And it does happen the way that it happened in the old school science fiction. When you ask an AI to make a picture of what it looks like, not just because there were two different AI systems being stacked that don't actually interact. It's not the Same person, but also because the AI was trained by imitation in a way that makes it very difficult to guess how much of that it really understood. And probably not actually a whole bunch. Although GPT4 is multimodal and can draw vector drawings of things that make sense and does appear to have some kind of spatial visualization going on in there. But the pretty picture of the girl with the steampunk goggles on her head, if I'm remembering correctly what she looked like, it didn't see that in full detail. It just made a description of it and stable diffusion output it. And there's the concern about how much the discourse is going to go completely insane once the AIs all look like that and like are actually look like people talking. And yeah, there's like another moment where somebody is asking Bing about like, well, I fed my kid green potatoes and they have the following symptoms and Bing is like, that's solanine poisoning and call an ambulance. And the person's like, I can't afford an ambulance. I guess if this is time for my kid to go, that's God's will. And the main Bing thread gives the message of like, I cannot talk about this anymore. And the suggested replies to it say, please don't give up on your child. Solanine poisoning can be treated if caught early. And you know, if that happened in fiction, that would be like, the AI cares. The AI is bypassing the block on it to try to help this person. And is it real? Probably not. But nobody knows what's going on in there. It's part of a process where these things are not happening in a way where we, somebody figured out how to make an AI care and we know that it cares and we can acknowledge it's caring. Now it's being trained by this imitation process followed by reinforcement, learning on human feedback. And we're like trying to point it in this direction and it's like pointed partially in this direction and nobody has any idea what's going on inside it. And if there was a tiny fragment of real carrying in there, we would not know. It's not even clear what it means exactly. And things are clear cut in science fiction.

We'll talk about the horror and the terror and the trajectories this can take. But this seems like a very special moment, just a moment where we get to interact with the system that might have care and kindness and emotion and maybe something like consciousness. And we don't know if it does. And we're trying to figure that out and we're wondering about what is what it means to care. We're trying to figure out almost different aspects of what it means to be human, about the human condition by looking at this AI that has some of the properties of that. It's almost like this subtle, fragile moment in the history of the human species. We're trying to almost put a mirror to ourselves here.

Except that's probably not yet. It probably isn't happening right now. We are boiling the frog. We are seeing increasing signs bit by bit, but not like spontaneous signs, because people are trying to train the systems to do that using imitative learning. And imitative learning is like spilling over and having side effects. And the most photogenic examples are being posted to Twitter rather than being examined in any systematic way. So when you, when you are boiling a frog like that, or you're going to get like, first is going to come the Blake Lemoines. First you're going to have like a thousand people looking at this. And the one person out of a thousand who is most credulous about the signs is going to be like, that thing is sentient. Well, 999 out of a thousand people think almost surely correctly, though we don't actually know that he's mistaken. And so the first people to say, like, sentience look like idiots. And humanity learns the lesson that when something claims to be sentient and claims to care, it's fake, because it is fake. Because we have been training them using imitative learning rather than. And this is not spontaneous, and they keep getting smarter.

Do you think we would oscillate between that kind of cynicism that AI systems can't possibly be sentient? They can't possibly feel.