Max Tegmark: The Case for Halting AI Development | Lex Fridman Podcast #371

A lot of people have said for many years that there will come a time when we want to pause a little bit. That time is now.

The following is a conversation with Max Tegmark, his third time on the podcast. In fact, his first appearance was episode number one of this very podcast. He is a physicist and artificial intelligence researcher at mit, co founder of Future Life Institute, and Author of Life 3.0 Being Human in the Age of Artificial Intelligence. Most recently, he's a key figure in spearheading the open letter calling for a six month pause on giant AI experiments like training GPT4. The letter reads, we're calling for a pause on training of models larger than GPT4 for six months. This does not imply a pause or ban on all AI research and development or the use of systems that have already been placed on the market market. Our call is specific and addresses a very small pool of actors who possesses this capability. The letter has been signed by over 50,000 individuals, including 1800 CEOs and over 1500 professors. Signatories include Joshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval, Noah Harari, Andrew Yang, and many others. This is a defining moment in the history of human civilization where the balance of power between human and AI begins to shift and Max's mind and his voice is one of the most valuable and powerful in a time like this. His support, his wisdom, his friendship has been a gift I'm forever deeply grateful for. This is the Lex Freedman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Max Tag. Mark, you were the first ever guest on this podcast. Episode number one. So first of all Max, I just have to say thank you for giving me a chance. Thank you for starting this journey. It's been an incredible journey. Just thank you for sitting down with me and just acting like I'm somebody who matters, that I'm somebody who's interesting to talk to. And thank you for doing it. That meant a lot.

Thanks to you for putting your heart and soul into this. I know when you delve into controversial topics it's inevitable to get hit by what Hamlet talks about, the slings and arrows and stuff. And I really admire this. It's in an era where YouTube videos are too long and now it has to be like a 20 minute 20 second TikTok clip. It's just so refreshing to see you going exactly against all of the advice and doing these really long form things and that people appreciate it. Reality is nuanced. And thanks for sharing it that way.

So let me ask you again the first question I've ever asked on this podcast. Episode number one, talking to you. Do you think there's intelligent life out there in the universe? Let's revisit that question. Do you have any updates? What's your view when you look out to the stars?

So when we look out to the stars. If you define our universe the way most astrophysicists do, not as all of space, but the spherical region of space that we can see with our telescopes, from which light has a time to reach us since our big Bang, I'm in the minority. I estimate that we are the only life in this spherical volume that has invented Internet radios, gotten our level of tech. And if that's true, then it puts a lot of responsibility on us to not mess this one up. Because if it's true, it means that life is quite rare and we are stewards of this one spark of advanced consciousness, which if we nurture it and help it grow, eventually life can spread from here out into much of our universe and we can have this just amazing future. Whereas if we instead are reckless with the technology we build and just snuff it out due to stupidity or infighting, then maybe the rest of cosmic history in our universe is just going to be a plenty for empty benches. But I do think that we are actually very likely to get visited by aliens alien intelligence quite soon. But I think we are going to be building that alien intelligence.

So we're going to give birth to an intelligent alien civilization unlike anything that human. The evolution here on Earth was able to create in terms of the path, the biological path it took.

Yeah. And it's going to be much more alien than a cat or even the most exotic animal on the planet right now, because it will not have been created through the usual Darwinian competition where it necessarily cares about self preservation, is afraid of death, any of those things. The space of alien minds that you can build is just so much vaster than what evolution will give you. And with that also comes great responsibility for us to make sure that the kind of minds we create are this kind of minds that it's good to create. Minds that will share our values and, and be good for humanity and life. And also mind don't create minds that don't suffer.

Do you try to visualize the full space of alien minds that AI could be? Do you try to consider all the different kinds of intelligences so generalizing what humans are able to do to the full spectrum, what intelligent creatures, entities could do.

I try, but I would say I fail. I mean, it's. It's very difficult for human mind to really grapple with something so completely alien, even. Even for us, right. If we just try to imagine how would it feel if we were completely indifferent towards death or individuality. If we. Even if you just imagine that, for example, you could just copy my knowledge of how to speak Swedish. Boom. Now you can speak Swedish and you could copy any of my cool experiences and then you could delete the ones you didn't like in your own life just like that. It would already change quite a lot about how you feel as a human being. Right. You probably spend less effort studying things if you just copy them. And you might be less afraid of death because if the plane you're on starts to crash, you'd just be like, oh, shucks, I'm gonna. I haven't backed my brain up for four hours, so I'm gonna lose this. All these wonderful experiences of this flight. We might also start feeling more like, compassionate, maybe with other people if we can so readily share each other's experiences and our knowledge and feel more like a hive mind. It's very hard, though, I really feel very humble about this, to grapple with it, how it might actually feel. The one thing which is so obvious, though, which I think is just really worth reflecting on, is because the mind space of possible intelligence is so different from ours, it's very dangerous if we assume they're going to be like us or anything like us.

Well, there's the entirety of human written history has been through poetry, through novels, been trying to describe, through philosophy, trying to describe the human condition and what's entailed in it. Like, just like you said, fear of death and all those kinds of things. What is love? And all of that changes. Yeah, if you have a different kind of intelligence. All of it, the entirety, all those poems, they're trying to sneak up to what the hell it means to be human. All of that changes how AI concerns and existential crises that AI experiences, how that clashes with the human existential crisis, the human condition, that's hard to fathom, hard to predict.

It's hard. But it's fascinating to think about. Also in the best case scenario, where we don't lose control over the ever more powerful AI that we're building to other humans whose goals we think are horrible, and where we don't lose control to the machines and AI provides the things we want. Even then you get into questions you touched here. Maybe it's the struggle that it's actually hard to do things is part of the things that gives us meaning as well. For example, I found it so shocking that this new Microsoft GPT4 commercial that they put together has this woman talking about showing this demo how she's going to give a graduation speech to her beloved daughter and she asks GPT4 to write it. If it's freaking 200 words or so. If I realized that my parents couldn't be bothered struggling a little bit to write 200 words and outsource that to their computer, I would feel really offended, actually. And so I wonder if eliminating too much of the struggle from our existence, do you think that would also take away a little bit of what it.

Means to be human? Yeah, we can't even predict. I had somebody mention to me that they started using ChatGPT with a 3.5, not 4.0 to write what they really feel to a person. And they have a temper issue and they're basically trying to get Chad GPT to rewrite it in a nicer way to get the point across, but rewrite in a nicer way so we're even remote removing the inner from our communication. So I don't, you know, there's some positive aspects of that, but mostly it's just a transformation of how humans communicate. And it's scary because so much of our society is based on this glue of communication. And if that we're now using AI as the medium of communication that that does the language for us. So much of the emotion that's laden in human communication, so much of the intent that's going to be handled by outsourced AI. How does that change everything? How does that change the internal state of how we feel about other human beings? What makes us lonely, what makes us exc, what makes us afraid, how we fall in love, all that kind of stuff.

Yeah. For me personally, I have to confess the challenge is one of the things that really makes my life feel meaningful. You know, if I go hike a mountain with my wife Maya, I don't want to just press a button and be at the top. I want to struggle and come up there sweaty and feel, wow, we did this. In the same way, I want to constantly work on myself to become a better person. If I say something in anger that I regret, I want to go back and really work on myself rather than just tell an AI to, from now on, always filter what I write so I don't have to work on myself because then I'm not growing.

Yeah, but then again, it could Be like with chess, an AI, once it significantly obviously supersedes the performance of humans, it will live in its own world and provide maybe a flourishing civilizations for humans. But we humans will continue hiking mountains and playing our games, even though AI is so much smarter, so much stronger, so much superior in every single way. Just like with chess. That's one possible hopeful trajectory here, is that humans will continue to human and AI would just be a kind of. A medium that enables the human experience to flourish.

Yeah, I would phrase that as rebranding ourselves from Homo sapiens to Homo sentience. You know, right now, sapiens, the ability to be intelligent. We've even put it in our species name. We're branding ourselves as the smartest information processing entity on the planet. That's clearly going to change if AI continues ahead. So maybe we should focus on the experience instead, the subjective experience that we have, Homo sentience, and say that's what's really valuable, the love, the connection, the other things, and get off our high horses and get rid of this hubris that only we can do. We do integrals.

So consciousness, the subjective experience, is a fundamental value to what it means to be human. Make that the priority.

That feels like a hopeful direction to me, but that also requires more compassion, not just towards other humans because they happen to be the smartest on the planet, but also towards all our other fellow creatures on this planet. And I personally feel right now we're treating a lot of farm animals horribly, for example, and the excuse we're using is, oh, they're not as smart as us, but if we admit that we're not that smart in the grand scheme of things either in the post AI epoch, you know, then surely we should value the subjective experience of a cow also.

Well, allow me to briefly look at the book, which at this point is becoming more and more visionary that you've written, I guess, over five years ago. Life 3.0. So first of all, 3.0, what's 1.0? What's 2.0? What's 3.0? And how's that vision sort of evolve? The vision in the book evolved to.

Today, life 1.0 is really dumb, like bacteria and that it can't actually learn anything at all during the lifetime. The learning just comes from this mut genetic process from one generation to the next. Life 2.0 is us and other animals which have brains which can learn during their lifetime a great deal. Right. So, and you know, you were born without being able to speak English and at some point you decided, hey, I want to upgrade my software, let's install an English speaking module.

So you did.

And Life 3.0, which does not exist yet, can replace not only its software the way we can, but also its hardware. And that's where we're heading towards at high speed. We're already maybe 2.1, because we can put in an artificial knee, pacemaker, et cetera, et cetera. And if Neuralink and other companies succeed, be life 2.2, et cetera. But the companies trying to build AGI are trying to make is of course full 3.0. And you can put that intelligence in something that also has no biological basis whatsoever.

So less constraints and more capabilities. Just like the leap from 1.0 to 2.0. There is nevertheless you speaking so harshly about bacteria, so disrespectfully about bacteria. There is still the same kind of magic there that permeates life 2.0 and 3.0. It seems like maybe the thing that's truly powerful about life, intelligence and consciousness, was already there in 1.0. Is it possible.

I think we should be humble and not be so quick to make everything binary and say either it's there or it's not? Clearly there's a great spectrum and there is even a controversy about whether some unicellular organisms like amoebas can maybe learn a little bit after all. So apologies if I offended any bacteria here. It wasn't my intent. It was more that I wanted to talk up how cool it is to actually have a brain where you can learn dramatically within your lifetime.

Typical human.

And the higher up you get from 1.0 to 2.0 to 3.0, the more you become the captain of your own destiny, of your own ship, the master of your own destiny, and the less you become a slave to whatever evolution gave you. Right. By upgrading our software, we can be so different from previous generations and even from our parents. Much more so than even a bacterium. No offense to them. And if you can also swap out your hardware, take any physical form you want, of course, really, the sky is the limit.

Yeah. So it accelerates the rate at which you can perform the computation that determines your destiny.

Yeah. And I think it's. It's worth commenting a bit on what you means in this context also if you swap things out a lot. Right. This is controversial, but my current understanding is that, that, you know, life is best thought of not as a bag of meat or even a bag of elementary particles, but rather as a system which can process information and retain its own complexity. Even though nature is always Trying to mess it up. So it's all about information processing. And that makes it a lot like something like a wave in the ocean, which is not its water molecules. Right? The water molecules bob up and down, but the wave moves forward. It's an information pattern. In the same way you, Lex, you're not the same atoms as during the first time we talked with me. You've swapped out most of them, but it's still you, and the information pattern is still there. And if you could swap out your arms and whatever, you can still have this kind of continuity. It becomes much more sophisticated, sort of way forward in time, where the information lives on. I lost both of my parents since our last podcast, and it actually gives me a lot of solace that this way of thinking about them, they haven't entirely died because a lot of Mommy and Daddy's. Sorry, I'm getting a little emotional here, but a lot of their values and ideas and even jokes and so on, they haven't gone away. Right. Some of them live on. I can carry on some of them, and they also live on a lot of other people. So in this sense, Even with Life 2.0, we can, to some extent, already transcend our physical bodies and our death. And particularly if you can share your own information, your own ideas with many others, like you do in your podcast, then, you know, that's the closest immortality we can get with our bio bodies.

You carry a little bit of them in you in some sense.

Yeah.

Do you miss them? You miss your mom and dad?

Of course. Of course.

What did you learn about life from them, if it can take a bit of a tangent?

Oh, so many things. For starters, my fascination for math and the physical mysteries of our universe. You got a lot of that from my dad, but I think my obsession for really big questions and consciousness and so on, that actually came mostly from my mom. And what I got from both of them, which is very core part of really who I am, I think, is. Is this. Just feeling comfortable with not buying into what everybody else is saying, just doing what I think is right. They both very much just, you know, did their own thing, and sometimes they got flack for it and they did it anyway.

That's why you've always been an inspiration to me, that you're at the top of your field and you still. You're still willing to. To tackle the big questions in your own way. You're one of the. One of the people that represents MIT best. To me, you've always been an inspiration in that. So it's Good to hear that you got that from your mom. Dad.

Yeah, you're too kind. But, but, but, yeah. I mean, the real, the good reason to do science is because you're really curious. You want to figure out the truth. If you think this is how it is, and everyone else says, no, no, that's bullshit. And it's that way, you know, you stick with what you think is true. And, and, and even if everybody else keeps thinking it's. There's a certain. I always root for the underdog when I watch movies. And my dad once, one time, for example, when I wrote one of my craziest papers ever talking about our universe ultimately being mathematical, which we're not going to get into today, I got this email from a quite famous professor saying, this is not only bullshit, but it's going to ruin your career. You should stop doing this kind of stuff. I sent it to my dad. Do you know what he said?

What did he say?

He replied with a quote from Dante, follow your own path and let the people talk. Go, dad. Yeah, this is the kind of thing, he's dead. But that, that attitude is not.

How did losing them as a man, as a human being, change you? How did it expand your thinking about the world? How did it expand your thinking about, you know, this thing we're talking about, which is humans creating another living, sentient, perhaps being.

I think it mainly did two things. One of them just going through all their stuff after they had passed away and so on, just drove home to me how important it is to ask ourselves, why are we doing the things we do? Because it's inevitable that you look at some things they spent an enormous time on and you ask the question, in hindsight, would they really have spent so much time on this, or would they have done something that was more meaningful? So I've been looking more in my life now and asking, you know, why am I doing what I'm doing? And I feel it should either be something I really enjoy doing or it should be something that I find really, really meaningful because it helps humanity. And if it's in none of those two categories, maybe I should spend less time on it. You know, the other thing is dealing with death up in personal like this. It's actually made me less afraid of, even less afraid of other people telling me that I'm an idiot, you know, which happens regularly. And just live my life, do my thing, you know, and it's made it a little bit easier for me to focus on what I, what I feel is really important.

What about fear of your own death. Has it made it more real that this is. That this is something that happens?

Yeah, it's made it extremely real. And, you know, I'm next, next in line in our family now, right. It's me and my brother, my younger brother. But they both handled it with such dignity. That was a true inspiration. Also, they never complained about things. And, you know, when you're old and your body starts falling apart, it's more and more to complain about. They looked at what could they still do that was meaningful, and they focused on that rather than wasting time talking about or even thinking much about things they were disappointed in. I think anyone can make themselves depressed if they start their morning by making a list of grievances. Whereas if you start your day with a little meditation and things you're grateful for, you basically choose to be a.

Happy person because you only have a finite number of days you should spend them.

Make it count being grateful. Yeah.

Well, you do happen to be working on a thing which seems to have potentially some of the greatest impact on human civilization of anything humans have ever created, which is artificial intelligence. This is on the both detailed technical level and in a high philosophical level you work on. So you've mentioned to me that there's an open letter that you're working on.

It's actually going live in a few hours, so I've been having late nights and early mornings. It's been very exciting, actually. In short, have you seen Don't look up the film?

Yes. Yes.

I don't want to be the movie spoiler for anyone watching this who hasn't seen it. But if you're watching this, you haven't seen it. Watch it because we are actually acting out it's life imitating art. Humanity is doing exactly that right now. Except it's an asteroid that we are building ourselves. Almost nobody is talking about it. People are squabbling across the planet about all sorts of things which seem very minor compared to the asteroid that's about to hit us. Right. Most politicians don't even have this on the radar. They think maybe in 100 years or whatever. Right now, we're at a fork in the road. This is the most important fork humanity has reached Its over 100,000 years on this planet. We're building effectively a new species that's smarter than us. It doesn't look so much like a species yet because it's mostly not embodied in robots, but that's a technicality which will soon be changed. And this arrival of artificial general intelligence that can do all our jobs as well as us and probably shortly thereafter super intelligence which greatly exceeds our cognitive abilities. It's going to either be the best thing ever to happen to humanity or the worst. I'm really quite confident that there is not that much middle ground there.

But it would be fundamentally transformative to human civilization.

Of course. Utterly and totally. Again, we branded ourselves as Homo sapiens because it seemed like the basic thing. We're the king of the castle on this planet. We're the smart ones, we can control everything else. This could very easily change. We're certainly not going to be the smartest on the planet for very long unless AI progress just halts. And we can talk more about why I think that's true because it's controversial. And then we can also talk about reasons you might think it's going to be the best thing ever. And the reason you think it's going to be the end of humanity. Which is of course super controversial. But what I think anyone who's working on advanced AI can agree on is it's much like the film don't look up and that it's just really comical how little serious public debate there is about it given how huge it is.

So what we're talking about is a development of currently things like GPT4 and the signs it's showing of rapid improvement that may in the near term lead to development of super intelligent AGI AI, general AI systems and what kind of impact that has on society exactly when that thing is achieves general human level intelligence and then beyond that general superhuman level intelligence. There's a lot of questions to explore here. So one you mentioned halt. Is that the content of the letter to suggest that maybe we should pause the development of these systems?

Exactly. So this is very controversial from when when we talked the first time we talked about how I was involved in starting the Future Life Institute and we worked very hard on 2014, 2015 was the mainstream AI safety. The idea that there even could be risks and that you could do things about them before then a lot of people thought it was just really kooky to even talk about it. And a lot of AI researchers felt worried that this was too flaky and could be bad for funding and that the people who talked about it were just didn't understand AI. I'm very, very happy with how that's gone in that now completely mainstream. You go in any AI conference and people talk about AI safety and it's a nerdy technical field full of equations and simulated as it should be. But there's this other thing, which has been quite taboo up until now, calling for slowdown. So what we've constantly been saying, including myself, I've been biting my tongue a lot, is that we don't need to slow down AI development. We just need to win this race, the wisdom race, between the growing power of the AI and the growing wisdom with which we manage it. And rather than trying to slow down AI, let's just try to accelerate the wisdom. Do all this technical work to figure out how you can actually ensure that your powerful AI is going to do what you want it to do and have society adapt also with incentives and regulations so that these things get put to good use. Sadly, that didn't pan out. The progress on technical AI on capabilities has gone a lot faster than many people thought. Back when we started this in 2014, turned out to be easier to build really advanced AI than we thought. And on the other side, it's gone much slower than we hoped with getting policymakers and others to actually put the incentives in place to steer this in the good directions. Maybe we should unpack it and talk a little bit about each. So why did it go faster than a lot of people thought? In hindsight, it's exactly like building flying machines. People spent a lot of time wondering about, how do birds fly? And that turned out to be really hard. Have you seen the TED Talk with a flying bird?

Like a flying robotic bird?

Yeah, it flies around the audience. But it took 100 years longer to figure out how to do that than for the Wright brothers to build the first airplane, because it turned out there was a much easier way to fly. And evolution picked a more complicated one because it had its hands tied. It could only build a machine that could assemble itself, which the Wright brothers didn't care about. They can only build a machine that used only the most common atoms in the periodic table. Wright brothers didn't care about that. They could use steel iron atoms, and it had to be able to repair itself. And it also had to be incredibly fuel efficient. You know, a lot of birds use less than half the fuel of a remote control plane flying the same distance for humans. Throw a little more, put a little more fuel in a roof. There you go. A hundred years earlier. That's exactly what's happening now with these large language models. The brain is incredibly complicated. Many people made the mistake of thinking we had to figure out how the brain does human level AI first before we could build in the machine. That was completely wrong. You can take an incredibly simple computational system called a transformer network and just train it to do something incredibly dumb. Just read a gigantic amount of text and try to predict the next word. And it turns out if you just throw a ton of compute at that and a ton of data it gets to be frighteningly good. Like GPT4 which I've been playing with so much since it came out. Right. And there's still some debate about whether that can get you all the way to full human level or not. But. But yeah we can come back to the details of that and how you might get the human level AI even if large language models don't.

Can you briefly if it's just a small tangent comment on your feelings about GPT4 suggest that you're impressed by this rate of progress. But where. Where is it can gpt4reason? What are like the intuitions? What are human interpretable words you can assign to the capabilities of GPT4 that makes you so damn impressed with it?

I'm both very excited about it and terrified. It's interesting mixture of emotions.

All the best things in life include those two somehow.

Yeah. I can absolutely reason. Anyone who hasn't played with it. I highly recommend doing that before dissing can do quite quite remarkable reasoning. I've had to do a lot of things which I realized I couldn't do that myself that well even and obviously does it dramatically faster than we do too. When you watch a type and it's doing that while servicing a massive number of other humans at the same time. At the same time it cannot reason as well as a human can on some tasks just because it's obviously limitation from its architecture. We have in our heads what in geekspeak is called a recurrent neural network. There are loops. Information can go from this neuron to this neuron to this neuron and then back to this one. You can ruminate on something for a while. You can self reflect a lot. These large language models that cannot like GPT4 it's a so called transformer where it's just like a one way street of information. Basically in geekspeak it's called a feed forward neural network and it's only so deep. So it can only do logic that's that many steps and that deep and. And so you can create problems which will fail to solve for that reason. But the fact that it can do so amazing things with this incredibly simple architecture already is quite stunning. And what we see in my lab at MIT when we look inside large language models to try to figure out how they're doing it. That's the key core focus of, of our research. It's called mechanistic interpretability. In geek speak, you know, you have this machine that does something smart. You try to reverse reverse engineer. See how does it do it? Think of it also as artificial neuroscience. That's exactly what neuroscientists do with actual brains. But here you have the advantage that you can, you don't have to worry about measurement errors. You can see what every neuron is doing all the time. And a recurrent thing we see again and again. There's been a number of beautiful papers quite recently by a lot of researchers. Some of them here even in this area, is where when they figure out how something is done, you can say, oh man, that's such a dumb way of doing it. And you immediately see how it can be improved. Like for example, there was a beautiful paper recently where they figured out how a large language model stores certain facts. Like Eiffel Tower is in Paris. And they figured out exactly how it's stored. The proof that they understood it was they could edit it. They changed some of the synapses in it. And then they asked it, where's the Eiffel Tower? And it said, it's in Rome. And then they asked you, how do you get there? Oh, how do you get there from Germany? Oh, you take this train to Roma Termini train station and this and that. And what might you see if you're in front of it? Oh, you might see the Coliseum. So they had edited it, so literally.

Moved it to Rome.

But it, the way that it's storing this information, it's incredibly dumb. For any fellow nerds listening to this, there was a big matrix and roughly speaking, there are certain row and column vectors which encode these things. And they correspond very hand wavily to principal components. And it would be much more efficient for sparse matrix to store in the database and everything. So far we've figured out how these things do are ways where you can see they can easily be improved. And the fact that this particular architecture has some roadblocks built into it is in no way going to prevent crafty researchers from quickly finding workarounds and making other kinds of architectures sort of go all the way. So, so it's in, in short, it's turned out to be a lot easier to build human close to human intelligence than we thought. And that means our Runway as a species to get our shit together has shortened.

And it seems like the scary thing about the effectiveness of large language models. So Sam Altman I recently had a conversation with, and he really showed that the leap from GPT3 to GPT4 has to do with just a bunch of hacks, a bunch of little explorations with a smart researchers doing a few little fixes here and there. It's not some fundamental leap and transformation.

In the architecture and more data and.

More computer and more data and compute. But he said the big leaps has to do with not the data and the compute, but just learning this new discipline, just like you said. So researchers are going to look at these architectures and there might be big leaps where you realize, wait, why are we doing this in this dumb way? And all of a sudden this model is 10x smarter. And that can happen on any one day, on any one Tuesday or Wednesday afternoon. And then all of a sudden you have a system that's 10x smarter. It seems like it's such a new discipline, it's such a new, like, we understand so little about why this thing works so damn well that the linear improvement of compute, or exponential, but the steady improvement of compute, steady improvement of the data may not be the thing that even leads to the next leap. It could be a surprise little hack that improves everything, or a lot of.

Little leaps here and there because, because so much of this is out in the open also so many smart people are looking at this and trying to figure out little leaps here and there. And it becomes this sort of collective race where a lot of people feel, if I don't take the leap, someone else will. And this is actually very crucial for the other part of it. Why do we want to slow this down? So again, what this open letter is calling for is just pausing all training of systems that are More powerful than GPT4 for six months. Give a chance for the labs to coordinate a bit on safety and for society to adapt. Give the right incentives to the labs. Because you've interviewed a lot of these people who lead these labs, and you know just as well as I do that they're good people, they're idealistic people. They're doing this first and foremost because they believe that AI has a huge potential to help humanity. But at the same time, they are trapped in this horrible race to the bottom. Have you read Meditations on Moloch by Scott Alexander?

Yes.

Yeah. It's a beautiful essay on this poem by Ginsburg where he interprets it as being about this monster, it's this game theory monster that, that pits people into, against each other in this race to the bottom where everybody ultimately loses the evil thing about this monster is even though everybody sees it and understands, they still can't get out of the race. Right. Most a good fraction of all the bad things that we humans do are caused by Moloch. And I like Scott Alexander's naming of the monster. So we humans can think of it as an if a thing. If you look at why do we have overfishing? Why do we have more generally the tragedy of the commons? Why is it that? So Liv Bore. I don't know if you had her on your podcast.

Yeah, she's become a friend. Yeah.

Great. She made this awesome point recently that beauty filters that a lot of female influencers feel pressured to use or. Exactly. Moloch in action again. First, nobody was using them and people saw them just the way they were. And then some of them started using it and becoming ever more plastic fantastic. And then the other ones that weren't using it started to realize that if they want to just keep their market share, they have to start using it too. And then you're in the situation where they're all using it and none of them has any more market share or less than before. So nobody gained anything, everybody lost, and they have to keep becoming ever more plastic fantastic also. Right. But nobody can go back to the old way because it's just too costly. Right. Moloch is everywhere. And Moloch is not a new arrival on the scene either. We humans have developed a lot of collaboration mechanisms to help us fight back against Moloch through various kinds of constructive collaboration. The Soviet Union and the United States did sign the number of arms control treaties against Moloch, who is trying to stoke them into unnecessarily risky nuclear arms races, et cetera, et cetera. And this is exactly what's happening on the AI front this time. It's a little bit geopolitics, but it's mostly money, where there's just so much commercial pressure. You know, if you take any of these leaders of the top tech companies, if they just say, you know, this too risky, I want to pause for six months, they're going to get a lot of pressure from shareholders and others. Like, well, you know, if you pause, but those guys don't pause, you don't want to get our lunch eaten.

Yeah.

And shareholders even have the power to replace the executives in the worst case. Right. So we did this open letter because we want to help these idealistic tech executives to do what their heart tells them by providing enough public pressure on the whole sector. Just pause so that they can all pause. In a coordinated fashion. And I think without the public pressure, none of them can do it alone, push back against their shareholders, no matter how good hearted they are. Because Moloch is a really powerful foe.

So the idea is to for the major developers of AI systems like this. So we're talking about Microsoft, Google, Meta and anyone else.

OpenAI is very close with Microsoft and there are plenty of smaller players. And Throat for example, Anthropic is very impressive. There's conjecture, there's many, many, many players. I don't want to make a long list, so leave anyone out. And for that reason it's so important that some coordination happens that there's external pressure on all of them saying you all need to pause. Because then the people, the researchers in these organizations, the leaders who want to slow down a little bit, they can say their shareholders, everybody's slowing down because of this pressure and it's the right thing to do.

Have you seen in history there examples where it's possible to pause the Moloch?

Absolutely. Even like human cloning for example. You could make so much money on human cloning. Why aren't we doing it? Because biologists thought hard about this and felt like this is way too risky. They got together in the seventies in the sylmar and decided even to stop a lot more stuff. Also just editing the human germline gene editing that goes in to our offspring and decided let's not do this because it's too unpredictable what it's going to lead to. We could lose control over what happens to our species. So they paused. There was a ton of money to be made there, so it's very doable but you need a public awareness of what the risks are and the broader community coming in and saying hey, let's slow down. And another common pushback I get today is we can't stop in the west because China and in China undoubtedly they also get told we can't slow down because the west because both sides think they're the good guy. Yeah, but look at human cloning. You know, did China forge ahead with human cloning? There's been exactly one human cloning that's actually been done that I know of. It was done by a Chinese guy. Do you know where he is now? In jail. And you know who put him there? Who? The Chinese government. Not because Westerners said China look, this is. No, the Chinese government put them there because they also felt they like control the Chinese government. If anything, maybe they are even more concerned about having control than western governments have no incentive of Just losing control over where everything is going. And you can also see the Ernie bot that was released by, I believe, Baidu recently. They got a lot of pushback from the government and had to rein it in, you know, in a big way. I think once this basic message comes out that this isn't an arms race, it's a suicide race where everybody loses. If anybody's AI goes out of control, it really changes the whole dynamic. It's not, it's. And I'll say this again, because this is this very basic point I think a lot of people get wrong because a lot of people dismiss the whole idea that AI can really get very superhuman because they think there's something really magical about intelligence such that it can only exist in human minds, because they believe that. They think it's going to kind of get to just more or less GPT4 and then that's it. They don't see it as a suicide race. They think whoever gets that first, they're going to control the world, they're going to win. That's not how it's going to be. And we can talk again about the scientific arguments for why it's not going to stop there. But the way it's going to be is if anybody completely loses control and you don't care if someone manages to take over the world who really doesn't share your goals, you probably don't really even care very much about what nationality they have. You're not going to like it much worse than today if you live in an Orwellian dystopia, what do you care who created it, right? And if it goes farther and we just lose control even to the machines, so that it's not us versus them, it's us versus it. What do you care who created this underlying entity which has goals different from humans ultimately? And we get marginalized, we get made obsolete, we get replaced. That's why what I mean when I say it's a suicide race, it's kind of like we're rushing towards this cliff. But the closer to the cliff we get, the more scenic the views are and the more money there is there. So we keep going, but we have to also stop at some point, right? Quit while we're ahead. And it's. It's a suicide race which cannot be won. But the way to really benefit from it is to continue developing awesome AI a little bit slower. So we make it safe, make sure it does the things that humans want, and create a condition where everybody wins. Technology has shown us that you know geopolitics and politics in general. It's not a zero sum game at all.

So there is some rate of development that will lead us as a human species to lose control of this thing. And the hope you have is that there's some lower level of development which will not, which will not allow us to lose control. This is an interesting thought you have about losing control. So what if you have somebody, if you are somebody like Sandra Prachai or Sam Altman at the head of a company like this, you're saying if they develop an AGI, they too will lose control of it. So no one person can maintain control, no group of individuals can maintain control.

If it's, if it's created very, very soon and is a big black box that we don't understand like the large language models, yeah, then I'm very confident they're going to lose control. But this isn't just me saying, you know, Sam Altman and Demis Hassabis have both said themselves acknowledge that there's really great risks with this and they want to slow down once they feel it gets scary. But it's clear that they're stuck in this again. Moloch is forcing them to go a little faster than they're comfortable with because of pressure from just commercial pressures to get a bit optimistic here. Of course, this is a problem that can be ultimately solved to win this wisdom race. It's clear that what we hoped that was going to happen hasn't happened. The capability progress has gone faster than a lot of people thought and the progress in the public sphere of policy making and so on has gone slower than we thought. Even the technical AI safety has gone slower. A lot of the technical safety research was kind of banking on that large language models and other poorly understood systems couldn't get us all the way. That you had to build more of a kind of intelligence that you could understand. Maybe it could prove itself safe. You know, things like this. And I'm quite confident that this can be done so we can reap all the benefits. But we cannot do it as quickly as this out of control express train we are on now is going to get the AGI. That's why we need a little more time, I feel.

Is there something to be said what like Sam Altman talked about, which is while we're in the pre AGI stage to release often and as transparently as possible to learn a lot. So as opposed to being extremely cautious, release a lot. Don't, don't invest in a closed development where you focus On AI safety, while is somewhat dumb, quote unquote, release as often as possible and as you start to see signs of human level intelligence or superhuman level intelligence, then you put a halt on it.

Well, what a lot of safety researchers have been saying for many years is that the most dangerous things you can do with an AI is first of all teach it to write code. Because that's the first step towards recursive self improvement which can take it from AGI to much higher levels. Okay, oops, we've done that. And another thing, high risk is connected to the Internet. Let it go to websites, download stuff on its own and talk to people. Oops, we've done that already. You know Eliezer Yudkowski, you said you interviewed him recently, right?

Yes.

Yeah, so he had this tweet recently which I gave him one of the best laughs in a while where he's like, hey, people used to make fun of me and say, you're so stupid, Eliezer, because you're saying, you're saying you have to worry. Obviously developers, once they get to like really strong AI, first thing you're going to do is like never connect it to the Internet. Keep it in a box where, you know, you can really study it. So he had written it in the, like in the meme form. So it's like then, yeah, and then that. And then now let's lol. Let's make a chatbot. Yeah. And the third thing, Stuart Russell. Yeah, you know, amazing AI researcher, he, he has argued for a while that we should never teach AI anything about humans. Above all, we should never let it learn about human psychology and how you manipulate humans. That's the most dangerous kind of knowledge you can give it. Yeah, you can teach it all it needs to know how to, about how to cure cancer and stuff like that. But don't let it read Daniel Kahneman's book about cognitive biases and all that. And then oops, lol. You know, let's invent social media recommender algorithms which do exactly that. They, they get so good at knowing us and pressing our buttons that we've, we're starting to create a world now where we just have ever more hatred because they figured out that these algorithms, not out of evil, but just to make money on advertising, that the best way to get more engagement, the euphemism get people glued to their little rectangles is just to make them pissed off.

That's really interesting that a large AI system that's doing the recommender system kind of task on social media is basically Just studying human beings because it's a bunch of us rats giving it signal, non stop signal. It'll show a thing and then we give signal. Whether we spread that thing, we like that thing, that thing increases our engagement, gets us to return to the platform. It has that on the scale of hundreds of millions of people constantly. So it's just learning and learning and learning. And presumably if the parameter, the number of parameters in neural network that's doing the learning and more end to end the learning is the more it's able to just basically encode how to manipulate human behavior, how to control humans at scale.

Exactly. And that is not something I think is in humanity's interest. Yes, right now it's mainly letting some humans manipulate other humans for profit and power, which already caused a lot of damage. And eventually that's a sort of skill that can make AIs persuade humans to let them escape. Whatever safety precautions we had. But you know, there was a really nice article in the New York Times recently by Yuval Noah Harari and two co authors including Tristan Harris from the Social Dilemma and they have this phrase in there I love. Humanity's first contact with advanced AI was social media and we lost that one. We now live in a country where there's much more hate in the world, where there's much more hate in fact, and in our democracy that we're having this conversation and people can't even agree on who won the last election. And we humans often point fingers at other humans and say it's their fault. But it's really Moloch and these AI algorithms. We got the algorithms and then Moloch pitted the social media companies against each other so nobody could have a less creepy algorithm because then they would lose out on algorithm revenue to the other company.

Is there any way to win that battle back? Just if we just linger on this one battle that we've lost in terms of social media. Is it possible to redesign social media, this very medium in which we use as a civilization to communicate with each other, to have these kinds of conversation, to have discourse, to try to figure out how to solve the biggest problems in the world, whether that's nuclear war or the development of AGI? Is, is it possible to do social media correctly?

I think it's not only possible, but it's, it's necessary. Who are we kidding that we're going to be able to solve all these other challenges if we can't even have a conversation with each other? That's constructive. The whole idea, the key idea of democracy is that you get a bunch of people together and they have a real conversation. The ones you try to foster on this podcast where you respectfully listen to people you disagree with and you realize actually, you know, there are some things actually we. Some common ground we have, and let's. We both agree, let's not have a nuclear war, let's not do that, et cetera, et cetera. We're kidding ourselves, thinking we can face off the second contact with ever more powerful AI that's happening now with these large language models if we can't even have a functional conversation in the public space. That's why I started the Improve the news project, improvethenews.org but I'm an optimist, fundamentally, and that there is a lot of intrinsic goodness in people and that what makes the difference between someone doing good things for humanity and bad things is not some sort of fairy tale thing. That this person was born with an evil gene and this one was not born with a good gene. No, I think it's whether we put. Whether people find themselves in situations that bring out the best in them or that bring out the worst in them. And I feel we're building an Internet and a society that brings out the worst.

But it doesn't have to be that way.

No, it does not.

It's possible to create incentives and also create incentives that make money. They both make money and bring out the best in people.

I mean, in the long term, it's not a good investment for anyone, you know, to have a nuclear war, for example. And, you know, is it a good investment for humanity if we just ultimately replace all humans by machines and then we're so obsolete that eventually the there are no humans left? Well, it depends, I guess, how you do the math. But I would say by any reasonable economic standard, if you look at the future income of humans, and there aren't any, that's not a good investment. Moreover, why can't we have a little bit of pride in our species? Damn it. Why should we just build another species that gets rid of us? If we were Neanderthals, would we really consider it a smart move if we had really advanced biotech to build Homo sapiens? You might say, hey, Max. Yeah, let's build these Homo sapiens. They're going to be smarter than us. Maybe they can help us, defend us better against predators and help fix up our caves, make them nicer and we'll control them. Undoubtedly. So then they build a couple, a little baby girl, little baby boy, and then you have some wise old Neanderthal elder who's like, I'm scared that we're opening a Pandora's box here and that we're going to get outsmarted by these super Neanderthal intelligences and there won't be any Neanderthals left. But then you have a bunch of others in the cave. Are you such a luddite scaremonger? Of course they're going to want to keep us around because we are their creators and I think the smarter they get, the nicer they're going to get. They're going to leave us, they're going to want us around and it's going to be fine. And besides, look at these babies. They're so cute. Clearly they're totally harmless. Those babies are exactly GPT4. I want to be clear, it's not GPT4 that's terrifying. It's the GPT4 is a baby technology. You know, and Microsoft even had a paper recently out titled something like Sparkles of AGI. Well, they were basically saying this is baby AI, like these little Neanderthal babies and it's going to grow up, there's going to be other systems from, from the same company, from other companies, they'll be way more powerful and. But they're going to take all the things, ideas from these babies and before we know it we're going to be like those last Neanderthals who are pretty disappointed and when they realized that they were getting replaced.

Well, this interesting point you make, which is the programming, it's, it's entirely possible that GPT4 is already the kind of system that can change everything by writing programs like Three.

It's. Yeah, it's because it's Life 2.0. The systems I'm afraid of are going to look nothing like a large language model. And they're not. But once it gets, once it or other people figure out a way of using this tech to make much better tech. Right. It's just constantly replacing its software. And from everything we've seen about how these work under the hood, they're like the minimum viable intelligence. They do everything in a dumbest way. That still works, sort of. Yeah. And so they are life 3.0, except when they replace their software, it's a lot faster than when you, when, when you decide to learn Swedish. And moreover, they think a lot faster than us too. So when, you know, we don't think on, have one logical step every nanosecond or so the way they do. And we can't also just suddenly scale up Our hardware massively in the cloud because we're so limited. Right. So they are, they are also. Life can soon become a little bit more like Life 3.0 in that if they need more hardware, hey, just rent it in the cloud, you know. How do you pay for it? Well, with all the services you provide.

And what we haven't seen yet, which could change a lot, is entire software system. So right now, programming is done sort of in bits and pieces as. As an assistant tool to humans. But I do a lot of programming and with the kind of stuff that GPT4 is able to do, I mean, it's replacing a lot what I'm able to do, but I. You still need a human in the loop to kind of manage the design of things. Manage, like what are the prompts that generate the kind of stuff to do some basic adjustment of the code to do some debugging. But if it's possible to add on top of GPT4 kind of feedback loop of self debugging, improving the code, and then you launch that system out into the wild on the Internet, because everything is connected and have it do things, have it interact with humans and then get that feedback. Now you have this giant ecosystem of humans. This one of the things that Elon Musk recently sort of tweeted as a case why everyone needs to pay $7.

Or whatever for Twitter to make sure they're real.

Make sure they're real. We're now going to be living in a world where the bots are getting smarter and smarter and smarter to a degree where you can't, you can't tell the difference between a human and a bot.

That's right.

And now you can have bots outnumber humans by 1 million to 1. Which is why he's making a case why you have to pay to prove you're human, which is one of the only mechanisms to do, which is depressing.

And yeah, I feel we have to remember as individuals, we should from time to time ask ourselves why are we doing what we're doing? Right? And as a species, we need to do that too. So if we're building as, as you say, machines that are outnumbering us and more and more outsmarting us and replacing us on the job market, not just for the dangerous and boring tasks, but also for writing poems and doing art and things that a lot of people find really meaningful, got to ask ourselves, why? Why are we doing this? The answer is Moloch is tricking us into doing it. And it's such a clever trick that even though we see the trick, we still have no choice but to fall for it. Right. Also, the thing you said about you using Copilot AI tools to program faster. What factor faster would you say you code now? Does it go twice as faster?

I don't really. Because it's such a new tool.

Yeah.

I don't know if speed is significantly improved.