Wojciech Zaremba: OpenAI Codex, GPT-3, Robotics, and the Future of AI | Lex Fridman Podcast #215

The following is a conversation with Wojciech Zaremba, co founder of OpenAI, which is one of the top organizations in the world doing artificial intelligence research and development. Wojciech is the head of language and co generation teams building and doing research on GitHub, Copilot, OpenAI, Codex, and GPT3, and who knows, 4, 5, 6, N and N1, and he also previously led OpenAI's robotic efforts. These are incredibly exciting projects to me that deeply challenge and expand our understanding of the structure and nature of intelligence. The 21st century, I think, may very well be remembered for a handful of revolutionary AI systems and their implementations, GPT codecs and applications of language models and transformers in general to the language and visual domains may very well be at the core of these AI systems. To support this podcast, please check out our sponsors. They're listed in the description. This is the Lex Friedman podcast and here is my conversation with Wojciech Zaremba.

You mentioned that Sam Altman asked about the Fermi Paradox and The people at OpenAI had really sophisticated, interesting answers. So that's when you knew this is the right team to be working with. So let me ask you about the Fermi Paradox, about aliens. Why have we not found overwhelming evidence for aliens visiting Earth?

I don't have a conviction in the answer, but rather kind of probabilistic perspective on what might be a, let's say, possible answers. It's also interesting that the question itself even can touch on the, you know, your typical question of what's the meaning of life? Because if you assume that like we don't see aliens because they destroy themselves, that kind of upweights the focus on making sure that we won't destroy ourselves. At the moment, the place where I am actually with my belief, and these things also change over the time, is I think that we might be alone in the universe, which actually makes life more, or let's say consciousness life more kind of valuable. And that means that we should more appreciate it.

Have we always been alone? So what's your intuition about our galaxy, our universe? Is it just sprinkled with graveyards of intelligent civilizations? Or are we truly, is life intelligent life, truly unique?

At the moment, my belief that it is unique, but I would say I could also, you know, there was like some footage released with UFO objects, which makes me actually doubt my own belief. Yes. Yeah. I can tell you one crazy answer that I have heard. Yes. So apparently when you look actually at the limits of computation, you can compute more if the temperature of the universe Would drop down. So one of the things that aliens might want to do, if they are truly optimizing to maximize amount of compute, which maybe can lead to more, let's say, simulations or so, instead of wasting current entropy of the universe because we by living, we are actually somewhat wasting entropy, then you can wait for the universe to cool down such that you have more computation. That's kind of a funny answer. I'm not sure if I believe in it, but that would be one of the reasons why you, you don't see aliens. It's also possible, see, some people say that maybe there is not that much point in actually going to other galaxies if you can go inwards. So there is no limits of what could be an experience. If we could, you know, connect machines to our brains while there are still some limits if we want to explore universe.

Yeah, there could be a lot of ways to go inwards too. Once you figure out some aspect of physics we haven't figured out yet, maybe you can travel to different dimensions. I mean, travel in three dimensional space may not be the most fun kind of travel. There may be like just a huge amount of different ways to travel and it doesn't require a spaceship going slowly in 3D space or space time.

It also feels, you know, one of them problems is that speed of light is low and universe is vast. And it seems that actually most likely if we want to travel very far, then we would, instead of actually sending spaceships with humans that wait a lot, we would send something similar to what Yuri Miller is working on. These are like a huge sail which is at first powered. There is a shot of laser from an earth and it can propel it to quarter of speed of light. And sail itself contains a few grams of equipment and that might be the way to actually transport matter through universe. But then when you think what would it mean for humans? It means that we would need to actually put their 3D printer and you know, 3D print a human on other planet. I don't know, play them YouTube or let's say or like a 3D print, like huge human right away or maybe a womb or so.

Yeah. With our current techniques of archeology, if a civilization was born and died.

Long.

Enough ago on Earth, we wouldn't be able to tell. And so that makes me really sad. And so I think about Earth in that same way. How can we leave some remnants if we do destroy ourselves? How can we leave remnants for aliens in the future to discover? Like, here's some nice stuff we've done, like Wikipedia and YouTube. Do we have it like in a satellite orbiting earth with a hard drive? Like, how do we say, how do we back up human civilization for the good parts, or all of it is good parts so that it can be preserved longer than our bodies can? That's kind of a difficult question. It also requires the difficult acceptance of the fact that we may die. And if we die, we may die suddenly as a civilization.

So let's see. I think it kind of depends on the cataclysm we have observed in other parts of universe, that births of gamma rays, these are high energy rays of light that actually can apparently kill entire galaxy. So there might be actually nothing even to nothing to protect us from it. I'm also, when I'm looking actually at the past civilizations, so it's like Aztecs or. So they disappear from the surface of the earth. And one can ask why is it the case? And the way I'm thinking about it is definitely they had some, some problem that they couldn't solve, and maybe there was a flood and all of a sudden they couldn't drink, there was no potable water and they all died. And I think that so far the best solution to such a problems is, I guess, technology. So I mean, if they would know that you can just boil water and then drink it after, then that would save their civilization. And even now, when we look actually at the current pandemic, it seems that once again actually science comes to rescue and somehow science increases size of the action space. And I think that's a good thing.

Yeah, but nature has a vastly larger action space.

Yeah, but still it might be a good thing for us to keep on increasing action space.

Okay, looking at past civilizations, yes, but looking at the destruction of human civilization, perhaps expanding the action space will add actions that are easily acted upon, easily executed, and as a result destroy us.

So let's see, I was pondering why actually even we have negative impact on the globe. Because if you ask every single individual, they would like to have clean air, they would like healthy planet. But somehow it's not the case that as a collective we are not going this direction. I think that there exists very powerful system to describe what we value. That's capitalism. It assigns actually monetary values to various activities. At the moment, the problem in the current system is that there are some things which we value. There is no cost assigned to it. So even though we value clean air, or maybe we also value lack of distraction on, let's say Internet or so at the moment, these quantities, you know, companies, corporations can pollute them for free. So in some sense I wished or like, and that's, I guess, purpose of politics, to align the incentive systems. And we are kind of maybe even moving in this direction. The first issue is even to be able to measure the things that we value. Then we can actually assign the monetary value to them.

Yeah. So it's getting the data and also probably through technology enabling people to vote and to move money around in a way that is aligned with their values. And that's very much a technology question. So like having one president and Congress and voting that happens every four years or something like that, That's a very outdated idea. There could be some technological improvements to that kind of idea.

So I'm thinking from time to time about these topics. But it also feels to me that it's. It's a little bit like it's hard for me to actually make correct predictions. What is the appropriate thing to do? I extremely trust Sam Altman, our CEO on these topics. I'm more on the side of being, I guess, naive hippie. That.

Yeah, that's your life philosophy? Well, I think self doubt and I think hippie implies optimism. Those, those two things are pretty, pretty good way to operate.

I mean, still, it is hard for me to actually understand how the politics works or like how this, like, exactly how the things would play out. And Sam is really excellent with it.

What do you think is rarest in the universe? You said we might be alone. What's hardest to build is another engineering way to ask that. Life, intelligence or consciousness. So like you said, that we might be alone, which is the thing that's hardest to get to. Is it just the origin of life? Is it the origin of intelligence? Is it the origin of consciousness?

So let me at first explain you my kind of mental model what I think is needed for life to appear. So I imagined that at some point there was this primordial soup of amino acids and maybe some proteins in the ocean, and some proteins were turning into some other proteins through reaction. And you can almost think about this cycle of what turns into what as there is a graph essentially describing which substance turns into some other substance. And essentially life means that all of a sudden in the graph has been created a cycle such that the same thing keeps on happening over and over again. That's what is needed for life to happen. And in some sense, you can think almost that you have this gigantic graph and it needs a sufficient number of edges for the cycle to appear. Then, from perspective of intelligence and consciousness, my current intuition is that they might be quite intertwined. First of all, it might not be that it's like a binary thing that you have intelligence or consciousness. It seems to be more continuous component. Let's see if we look for instance, on the even networks recognizing images, people are able to show that the activations of these networks correlate very strongly with activations in visual cortex of some monkeys. The same seems to be true about language models. Also, if you, for instance, look, if you train agent in 3D world, at first it barely recognizes what is going on. Over the time, it kind of recognizes foreground from background. Over the time, it kind of knows where there is a foot and it just follows it. Over the time it actually starts having a 3D perception. So it is possible, for instance, to look inside of the head of an agent and ask what would it see if it looks to the right. And the crazy thing is initially when the agents are barely trained, these predictions are pretty bad. Over the time, they become better and better. You can still see that if you ask what happens when the head is turned by 360 degrees, for some time they think that the different thing appears. And then at some stage they understand actually that the same thing supposed to appear. So they get understanding of 3D structure. It's also very likely that they have inside some level of symbolic reasoning, like particularly symbols for other agents. So when you look at dota agents, they collaborate together and. And they have some anticipation of if they would win battle. They have some expectations with respect to other agents. I might be too much anthropomorphizing how the things look for me. But the fact that they have a symbol for other agents makes me believe that at some stage, as they are optimizing for skills, they would have also symbol to describe themselves. This is like a very useful symbol to have. And this particularity, I would call it like a self consciousness or self awareness. And still it might be different from the consciousness. So I guess the way how I'm understanding the word consciousness, let's say the experience of drinking a coffee, or let's say experience of being a. But that's the meaning of the word consciousness. It doesn't mean to be awake. Yeah, it might be also somewhat related to memory and recurrent connections. So it's kind of like if you look at anesthetic drugs, they might be like, they essentially disturb brain waves such that maybe memory is not formed.

So there's a lessening of consciousness when you do that.

Correct.

And so that's one way to intuit what is consciousness there's also kind of another element here. It could be that it's, you know, this kind of self awareness module that you described, plus the actual subjective experience is a storytelling module that tells us a story about what we're experiencing.

The crazy thing. So let's say, I mean in meditation they teach people not to speak story inside of the head. And there is also some fraction of population who doesn't have actually narrator. I know people who don't have a narrator and you know, they have to use external people in order to kind of solve tasks that require internal narrator. So it seems that it's possible to have the experience without the talk.

What are we talking about when we talk about the internal narrator? Is that the voice when you're like.

Yeah, I thought that that's what you are referring to.

Well, I was referring more on the like, not an actual voice. I meant like there's some kind of like subjective experience. Feels like it's. It's fundamentally about storytelling to ourselves. It feels like, like the feeling is a story that is much, much simpler abstraction than the raw sensory information. So there feels like it's a very high level abstraction that is useful for me to feel like entity in this world. Most useful aspect of it is that because I'm conscious, I think there's an intricate connection to me not wanting to die. So like it's a useful hack to really prioritize not dying. Like those seem to be somehow connected. So I'm telling this story of like it's richly feels like something to be me and the fact that me exists in this world. I want to preserve me. And so that makes it a useful agent hack.

So I will just refer maybe to the first part as you said, about that kind of story of describing who you are. I was thinking about it. Even so, you know, obviously I'm. I like thinking about consciousness, I like thinking about AI as well. And I'm trying to see analogies of these things in AI. What would it correspond to? So OpenAI trained a model called GPT which can generate pretty amusing text on arbitrary topic. And one way to control GPT is by putting into prefix at the beginning of the text some information. What would be the story about? You can have even chat with GPT by saying that the chat is with Lex or Elon Musk or so. And GPT would just pretend to be you or Elon Musk or so. And it almost feels that this story that we give ourselves to describe our life, it's almost like things that you put into context of GPT.

But the context we provide to GPT is multimodal.

So GPT itself is multimodal. GPT itself hasn't learned actually from. From experience of single human, but from the experience of humanity. It's a chameleon. You can turn it into anything. And in some sense, by providing context, it behaves as the thing that you wanted it to be. It's interesting that people have stories of who they are, and as you said, these stories, they help them to operate in the world. But it's also, you know, interesting. Guess various people find it out through meditation or so that there might be some patterns that you have learned when you were a kid that actually are not serving you anymore. And you also might be thinking that that's who you are and that's actually just a story.

Yeah. So it's a useful hack, but sometimes it gets us into trouble. It's a local optima.

It's a local optima.

You wrote that Stephen Hawking, he tweeted. Stephen Hawking asked what breathes fire into equations, which meant what makes given mathematical equations realize the physics of a universe? Similarly, I wonder what breathes fire into computation? What makes given computation conscious? Okay, so how do we engineer consciousness? How do you breathe fire and magic into the machine?

So it seems clear to me that not every computation is conscious. I mean, you can, let's say, just keep on multiplying one matrix over and over again and maybe gigantic matrix. You can put a lot of computation. I don't think it would be conscious. So in some sense the question is, what are the computations which could be conscious? I mean, so one assumption is that it has to do purely with computation, that you can abstract away matter, other possibilities, that it's very important was the realization of computation, that it has to do with some force fields or so, and they bring consciousness. At the moment, my intuition is that it can be fully abstracted away. So in case of computation, you can ask yourself, what are the mathematical objects or so that could bring such a properties. So for instance, if we think about the AI models, what they truly try to do, or models like GPT is they try to predict next word or so. And this turns out to be equivalent to compressing text. Because in some sense compression means that you learn the model of reality and you have just to remember where are your mistakes? The better you are in predicting the. And in some sense, when we look at our experience, also when you look for instance at the car driving, you know in which direction it will go. You are good like in prediction. And it might be the case that the consciousness is intertwined with compression. It might be also the case that self consciousness has to do with compressor trying to compress itself. So, okay, I was just wondering what are the objects in mathematics or computer science, which are mysterious, that could have to do with consciousness? And then I thought, you see, in mathematics there is something called Gedel Theorem, which means if you have sufficiently complicated mathematical system, it is possible to point the mathematical system back on itself. In computer science there is something called halting problem. It's somewhat similar construction. So I thought that if we believe that under assumption that consciousness has to do with compression, uh, then you could imagine that the. That the. As you keep on compressing things, then at some point it actually makes sense for the compressor to compress itself.

Meta compression. Yeah, consciousness is meta compression.

That's. That's an idea. And in some sense, you know, the crazy. Thank you.

So, but do you think if we think of a Turing machine, a universal Turing machine, can that achieve consciousness? So is there something beyond our traditional definition of computation that's required?

So it's a specific computation, and I said this computation has to do with compression. And the compression itself, maybe other way of putting it, is like you are internally creating the model of reality in order. You try inside to simplify reality in order to predict what's going to happen. And that also feels somewhat similar to how I think actually about my own conscious experience. So clearly I don't have access to reality. The only access to reality is through cable going to my brain and my brain is creating a simulation of reality and I have access to the simulation of reality.

Are you by any chance aware of the Hutter prize? Marcus Hutter, he made this prize for compression of Wikipedia pages. And there's a few qualities to it. One I think has to be perfect compression, which makes. I think that little quirk makes it much less applicable to the general task of intelligence because it feels like intelligence is always going to be messy. Like perfect compression is. Feels like is not the right goal, but it's nevertheless a very interesting goal. So for him, intelligence equals compression. And so the smaller you make the file, given a large Wikipedia page, the more intelligent the system has to be.

Yeah, that makes sense. So you can make perfect compression if you store errors. And I think that actually what he meant is you have algorithm plus errors. By the way, Hutter is a. He was PhD advisor of Shen Lek, who is a Deep Mind DeepMind co founder.

Yeah, yeah. So there's an interesting. And now he's a DeepMind. There's an interesting network of people. He's one of the people that I think seriously took on the task of what would an AGI system look like. I think for the longest time, the question of AGI was not taken seriously or rather rigorously, and he did just that. Like, mathematically speaking, what would the model look like if you remove the constraints of it having to be. Having to have reasonable amount of memory, reasonable amount of running time, complexity, computation time, what would it look like? And essentially it's a half math, half philosophical discussion of how would a reinforcement learning type of framework look like for an AGI?

Yeah, so he developed a framework even to describe what's optimal with respect to reinforcement learning. Like, there is a theoretical framework which is, as you said, under assumption, there is infinite amount of memory and computer. There was actually one person before. His name is Solomonov. Houter extended Solomonov work to reinforcement learning. But there exists the theoretical algorithm which is optimal algorithm to build intelligence. And I can actually explain you the algorithm.

Yes, let's go, let's go.

So the task itself.

Can I just pause how absurd it is for brain in a skull trying to explain the algorithm for intelligence? Just go ahead.

It is pretty crazy. It is pretty crazy that the brain itself is actually so small and it can ponder.

How to design algorithms that optimally solve the problem of intelligence. Okay, all right, so what's the algorithm?

So let's see. So first of all, the task itself is described as, you have infinite sequence of zeros and ones. Okay? You read n bits, and you are about to predict n +1 bit. So that's the task. And you could imagine that every task could be casted as such a task. So if, for instance, you have images and labels, you can just turn every image into sequence of zeros and ones, then label, you concatenate labels. And that's actually you could start by having training data first, and then afterwards you have test data. So theoretically, any problem could be casted as the problem of predicting zeros and ones on this infinite tape. So let's say you read already n bits and you want to predict n +1bit. And I will ask you to write every possible program that generates these N bits. So, and you choose programming language, it can be in Python or C. And the difference between programming languages might be there is a difference by constant. Asymptotically, your predictions will be equivalent. So you read n bits, you enumerate all the programs that produce these N bits in their output, and then in order to predict n 1bit. You actually weight the programs according to their length. There is some specific formula how you weight them. And then the n 1 bit prediction is the prediction from each of these program according to that weight.

Like statistically.

Statistically.

So the smaller the program, the more likely you are to pick its output. So that algorithm is grounded in the hope or the intuition that the simple answer is the right one.

It's a formalization of also means. Like if you would ask the question after how many years would sun explode? You can say it's more likely the answer is due to some power because it's a shorter program.

Yeah, I don't have a good intuition about how different the space of short programs are from the space of large programs. Like, what is the universe where short programs run things.

So as I said, the things have to agree with N bits. So even if you have, you need to start. Okay, if you have very short program and they're like still some. If it's not perfect with prediction of N bits, you have to start errors, what are the errors? And that gives you the full program that agrees on N bits.

Oh, so you don't agree perfectly with the N bits and you store errors.

That's like a longer program, slightly longer program, because it contains these extra bits of errors.

That's fascinating. What's your intuition about the programs that are able to do cool stuff like intelligence and consciousness? Are they perfectly like, is there if then statements in them? So is there a lot of exceptions at their story?

So you could imagine if there would be tremendous amount of if statements, then they wouldn't be that short. In case of neural networks, you could imagine that what happens is when you start with an uninitialized neural network, it stores internally many possibilities how the problem can be solved. And SGD is kind of magnifying some paths which are slightly similar to the correct answer. So it's kind of magnifying correct problem programs. And in some sense, HDD is a search algorithm in the program space. And the program space is represented by kind of the wiring inside of the neural network. And there is like an insane number of ways how dev features can be computed.

Let me ask you the high level basic question that's not so basic. What is deep learning? Is there a way you'd like to think of it that that is different than like a generic textbook definition?

The thing that I hinted just a second ago is maybe the closest to how I'm thinking these days about deep learning. So now the statement is neural networks can represent some Programs, it seems that various modules that we are actually adding up to are like. We want networks to be deep because. Because we want multiple steps of the computation. And deep learning provides a way to represent space of programs which is searchable. And it's searchable with stochastic gradient descent. So we have an algorithm to search over a humongous number of programs. And gradient descent kind of bubbles up the things that tend to give correct answers.

So a neural network with fixed weights, that's optimized. Do you think of that as a single program?

So there is a work by Christopher Olach where he. So he works on interpretability of neural networks. And he was able to identify inside of the neural network, for instance, a detector of a wheel for a car or the detector of a mask for a car. And then he was able to separate them out and assemble them together using a simple program for the detector for a car detector.

That's like. If you think of traditionally defined programs, that's like a function within a program that this particular neural network was able to find. And you can tear that out just like you can copy and paste from Stack overflow. That. So any program is a composition of smaller programs.

Yeah, I mean, the nice thing about the neural networks is that it allows the things to be more fuzzy than in case of programs. In case of programs, you have this like a branching this way or that way. And the neural networks, they have an easier way to be somewhere in between or to share things.

What to use the most beautiful or surprising idea in deep learning in the utilization of these neural networks, which, by the way, for people who are not familiar, neural networks is a bunch of. What would you say it's inspired by the human brain? There's neurons, there's connection between those neurons, there's inputs and there's outputs, and there's millions or billions of those neurons. And the learning happens by adjusting the weights on the edges that connect these neurons.

Thank you for giving definition that I supposed to do it, but I guess you have enough empathy to listeners to actually know that that might be useful.

No, that's like. So I'm asking Plato of like, what is the meaning of life? He's not going to answer. You're being philosophical and deep and quite profound talking about the space and programs, which is. Is very interesting, but also for people who are just not familiar what the hell we're talking about when we talk about deep learning. Anyway, sorry, what is the most beautiful or surprising idea to you in all the time you've worked at deep learning? And you worked on a lot of fascinating projects, applications of neural networks. It doesn't have to be big and profound. It could be a cool trick.

Yeah, I mean, I'm thinking about the trick, but like a. It's still amusing to me that it works at all, that let's say that the extremely simple algorithm, stochastic gradient descent, which is something that I would be able to derive on the piece of paper to high school student, when put at the scale of thousands of machines, actually can create the behaviors which we called kind of human like behaviors.

So in general, any application of stochastic gradient descent to neural networks is amazing to you or is there a particular application in natural language reinforcement learning? And also, would you attribute that success to. Is it just scale? What profound insight can we take from the fact that the thing works for gigantic sets of variables?

I mean, the interesting thing is these algorithms, they were invented decades ago and people actually gave up on the idea. And back then they thought that we need profoundly different algorithms. And they spent a lot of cycles on very different algorithms. And I believe that we have seen that various innovations, let's say like transformer or dropout or so they can pass the help. But it's also remarkable to me that this algorithm from 60s or so, I mean you can even say that the gradient descent was invented by Leibniz in I guess 18th century or so. That actually is the core of learning in the past. People are, it's almost like out of the. Maybe an ego, people are saying that it cannot be the case that such a simple algorithm is the, you know, could solve complicated problems. So they were in search for the other algorithms. And as I'm saying, like I believe that actually we are in the game where there is, there are actually frankly three levers. There is compute, there are algorithms, and there is data. And if we want to build intelligent systems, we have to pull all three levers. And they are actually multiplicative. It's also interesting. So you ask, is it only compute? People internally, they did the studies to determine how much gains they were coming from different levers. And so far we have seen that more gains came from compute than algorithms.

But.

But also we are in the world that in case of compute there is a kind of exponential increase in funding. And at some point it's impossible to invest more. It's impossible to invest $10 trillion as we are speaking about, let's say all taxes in us, but you're talking about money.

There could be innovation in the compute.

That's true as well. So I mean there are a few pieces. So one piece is human brain is an incredible supercomputer and it has 100 trillion parameters. Or if you try to count various quantities in the brain, they're like neurons, synapses. There are small number of neurons, there is a lot of synapses. It's unclear even how to map synapses to parameters of neural networks, but it's clear that there are many more. So it might be the case that our networks are still somewhat small. It also might be the case that they are more efficient than brain or less efficient by some huge factor. I also believe that there will be. At the moment we are at the stage that these neural networks, they require thousand x or a huge factor of more data than humans do. And it will be a matter of there will be algorithms that vastly decrease sample complexity. I believe so. But the place where we are heading today is data domains which contains million x more data. And even though computers might be 1,000 times slower than humans in learning, that's not the problem. For instance, I believe that each it should be possible to create superhuman therapies. And they're like even simple steps of doing it. And the core reason is there is just machine will be able to read way more transcripts of therapies and then it should be able to speak simultaneously with many more people and it should be possible to optimize it all in parallel. And well, there's.

Now you're touching on something I deeply care about and think is way harder than we imagine. What's the goal of a therapist?

What's the goal of therapist?

So, okay, so one goal. Now this is terrifying to me, but there's a lot of people that contemplate suicide, suffer from depression and they could significantly be helped with therapy. And the idea that an AI algorithm might be in charge of that, it's like a life and death task. It's. The stakes are high. So one goal for a therapist, whether human or AI, is to prevent suicide ideation to prevent suicide. How do you achieve that?

So, so to be clear, I don't think that the current models are good enough for such a task because it requires insane amount of understanding, empathy. And the models are far from displaced.

But do you think that understanding empathy, that signal is in the data?

I think there is some signal in the data, yes. I mean there are plenty of transcripts of conversations and it is possible to. It is possible from it to understand personalities. It is possible from it to understand if conversation is friendly, amicable, antagonistic. I believe that Given the fact that the models that we train now, They are chameleons that they can have any personality, they might turn out to be better in understanding personality of other people than anyone else. And they empathetic to be empathetic.

Yeah, interesting. But I wonder if there's some level of multiple modalities required to be able to be empathetic of the human experience. Whether language is not enough to understand death, to understand fear, to understand childhood trauma, to understand wit and humor required when you're dancing with a person who might be depressed or suffering. Both humor and hope and love and all those kinds of things. So there's another underlying question which is self supervised versus supervised. So can you get that from the data by just reading a huge number of transcripts?

I actually. So I think that reading huge number of transcripts is a step one. It's like the same way as you cannot learn to dance if just from YouTube by watching it, you have to actually try it out yourself. So I think that here that's a similar situation. I also wouldn't deploy the system in the high stakes situations right away, but kind of see gradually where it goes. And obviously initially it would have to go hand in hand with humans. But at the moment we are in the situation that actually there is many more people who actually would like to have a therapy or speak with someone than there are therapists out there. I was so fundamentally I was thinking, what are the things that can vastly increase people? Well, being therapy is one of them, being meditation is other one. I guess maybe human connection is a third one. And I guess pharmacologically it's also possible maybe direct brain stimulation or something like that. But these are pretty much options out there. Then let's say the way I'm thinking about the AGI endeavor is by default, that's an endeavor to increase amount of wealth. And I believe that we can vastly increase amount of wealth for everyone. And simultaneously, I mean, these are like two endeavors that make sense to me. One is like essentially increase amount of wealth and second one is increase overall human well being.

And those are coupled together.

And they can, I would say these are different topics. One can help another.

And you know, therapist is a funny word because I see friendship and love as therapy. I mean, so therapist broadly defined as just friendship as a friend. So like, therapist has a very kind of clinical sense to it. But what is human connection? You're like, not to get all Camus and Dostoyevsky on you, but you know, life is suffering and we draw, we seek Connection with other humans as we desperately try to make sense of this world in the deep overwhelming loneliness that we feel inside.

So I think connection has to do with understanding. And I think that almost like a lack of understanding causes suffering. If you speak with someone and do you feel ignored, that actually causes pain. If you are feeling deeply understood that actually they might not even tell you what to do in life, but like a pure understanding or just being heard.

Understanding is a kind of. It's a lot, you know, just being heard, feel like you're being heard. Like somehow that's alleviation temporarily of the loneliness that if somebody knows you're here with their body language, with the way they are, with the way they look at you, with the way they talk, do you feel less alone for a brief moment?

Yeah, very much agree. I thought in the past about somewhat similar question to yours, which is what is love? Rather, what is connection? Obviously I think about these things from AI perspective. What would it mean? I said that intelligence has to do with some compression, which is more or less like I can say almost understanding of what is going around. It seems to me that other aspect is there seem to be reward functions. And you can have reward for food, for maybe human connection, for let's say, warmth, sex and so on. And it turns out that various people might be optimizing slightly different reward functions. They essentially might care about these different things. And in case of love, at least the love between two people, you can say that the boundary between people dissolves to such extent that they end up optimizing each other reward functions.

Yeah, that's interesting. Celebrate the success of each other.

Yeah, in some sense, I would say love means helping others to optimize their reward functions. Not your reward functions, not the things that you think are important, but the things that the person cares about. You try to help them to optimize it.

So love is, if you think of two reward functions, it's a condition, you combine them together.

Yeah, pretty much.

Maybe like with a weight. And it depends the dynamic of the relationship.

Yeah, I mean, you could imagine that if you are fully optimizing someone's reward function without yours, then and then maybe are creating code dependency or something like that. I'm not sure what's the appropriate way. But the interesting thing is I even think that the individual person, we ourselves, we are actually less of a unified insight. So for instance, if you look at the donut on the one level you might think, oh, this looks tasty, I would like to eat it. Another level you might tell yourself I shouldn't be doing it because I want to gain muscles. So. And you know, you might do it regardless kind of against yourself. So it seems that even within ourselves they're almost like a kind of intertwined Personas. And I believe that the self love means that the love between all these persons, which also means being able to love love yourself when we are angry or stressed or.

So combining all those reward functions of the different selves you have.

Yeah. And accepting that they are there. Like, you know, often people, they have a negative self talk or they say I don't like when I'm angry. And like I try to imagine, try to imagine if there would be like a small baby lex like a 5 years old angry. And then you're like, you shouldn't be angry. Like stop being angry.

Yeah, but.

But like instead actually you want Lex to come over, give him a hug and it's like I say it's fine. Okay, you can be angry as long as you want. And then he would stop.

Or maybe not.

Or maybe not. But you cannot expect it even.

Yeah, but still that doesn't explain the why of love. Like why is love part of the human condition? Why is it useful to combine the reward functions? It seems like that doesn't. I mean, I don't think reinforcement learning frameworks can give us answers to why even the Hutter framework has an objective function that's static.

So we came to existence as a consequence of evolutionary process. And in some sense the purpose of evolution is survival. And then this complicated optimization objective baked into us, let's say compression, which might help us operate in the real world and it bake into us various reward functions. Then to be clear, at the moment we are operating in the regime which is somewhat out of distribution, where the even evolution optimized us.

It's almost like love is a consequence of cooperation that we've discovered is useful.

Correct. In some way. It's even the case if you.

I just love the idea that love is like the out of distribution or.

It'S not out of distribution. It's like as you said, it evolved for cooperation. Yes. And I believe that in some sense cooperation ends up helping each of us individually. So it makes sense evolutionary. And there is in some sense, and love means there is this dissolution of boundaries that you have a shared reward function. And we evolve to actually identify ourselves with larger groups so we can identify ourselves with a family, we can identify ourselves with a country to such extent that people are willing to give away their life for country. So we are wired actually Even for love. And at the moment, I guess maybe it would be somewhat more beneficial if we would identify ourselves with all the humanity as a whole. So you can clearly see when people travel around the world, when they run into person from the same country, they say, oh, which city are. And all this. All of a sudden they find all these similarities. They befriend those folks earlier than others. So there is some sense of the belonging. And I would say, I think it would be overall good thing to the world for people to move towards. I think it's even called open individualism, move toward the mindset of a larger and larger groups.

So the challenge there, that's a beautiful vision and I share it, to expand that circle of empathy, that circle of love towards the entirety of humanity. But then you start to ask, well, where do you draw the line? Because why not expand it to other conscious beings? And then finally, for our discussion, something I think about is why not expand it to AI systems like we, we start respecting each other when the, the person, the entity on the other side has the capacity to suffer, because then we develop a capacity to sort of empathize. And so I could see AI systems that are interacting with humans more and more having conscious, like displays. So like, they display consciousness through language and through other means. And so then the question is like, well, is that consciousness? Because they're acting conscious? And so, you know, the reason we don't like torturing animals is because they look like they're suffering when they're tortured. And if AI looks like it's suffering when it's tortured, how is that not requiring of the same kind of empathy from us and respect and rights than animals do and other humans do?

I think it requires empathy as well. I mean, I would like guess us or humanity or so make a progress in understanding what consciousness is. Because I don't want just to be speaking about the philosophy, but rather actually make a scientific. To have a. Like, you know, there was a time that people thought that there is a force of life, and the things that have this force, they are alive. And I think that there is actually a path to understand exactly what consciousness is. And in some sense it might require essentially putting probes inside of a human brain what neuralink does.

So the goal there, I mean, there's several things with consciousness that make it a real discipline, which is one is rigorous measurement of consciousness, and then the other is the engineering of consciousness, which may or may not be related. I mean, you could also run into trouble, like for example, in the United States for The Department dot, Department of Transportation and a lot of different places put a value on human life. I think dots values $9 million per person sort of. In that same way you can get into trouble if you put a number on how conscious a being is, because then you can start making policy. If a cow is 0.1 or like 10% as conscious as a human, then you can start making calculations and it might get you into trouble. But then again that might be a very good way to do it.

I would like to move to that place that actually we have scientific understanding what consciousness is and then we'll be able to actually assign value. And I believe that there is even the path for the experimentation in it. So we said that you could put the probes inside of the brain. There is actually few other things that you could do with devices like neuralink. So you could imagine that the way even to measure if AI system is conscious is by literally just plugging into the brain. I mean that assumes that it's kind of easy. But plugging into the brain and asking person if they feel that their consciousness expanded. This direction of course has some issues. You can say if someone takes a psychedelic drug, they might feel that their consciousness expanded even though that drug itself is not conscious.

Right? So like you can't fully trust the self report of a person saying their, their consciousness is expanded or not. Let me ask you a little bit about psychedelics because there's been a lot of excellent research on different psychedelics. Psilocybin, mdma, even DMT drugs in general, marijuana too. What do you think psychedelics do to the human mind? It seems they take the human mind to some interesting places. Is that just a little hack, a visual hack or is there some profound expansion of the mind?

So let's see. I don't believe in magic, I believe in science, in causality still, let's say. And then as I said, I think that brain that the our subjective experience of reality is we live in the simulation run by our brain. And the simulation that our brain runs, they can be very pleasant or very hellish drax. They are changing some hyperparameters of the simulation. It is possible thanks to change of these hyperparameters, to actually look back on your experience and even see that the given things that we took for granted, they are changeable. So they allow to have amazing perspective. There is also, for instance, the fact that after DMT people can see the full movie inside of their head gives me further belief that the brain can generate that full movie, that the brain is Actually learning the model of reality to such extent that it tries to predict what's going to happen next.

Yeah, Very high resolution so it can replay reality, et cetera.

Extremely high resolution. Yeah. It's also kind of interesting to me that somehow there seems to be some similarity between these drugs and meditation itself. And I actually started even these days to think about meditation as a psychedelic.

Do you practice meditation?

I practice meditation. I mean I went few times on the retreats and it feels after second or third day of meditation. There is almost like a sense of tripping.

What does a meditation retreat entail?

So you wake up early in the morning and you meditate for extended period of time alone. Sorry to keep in mind. Yeah, so it's optimized even though there are other people, it's optimized for isolation. So you don't speak with anyone, you don't actually look into other people's eyes and you sit on the chair, say vipassana. Meditation tells you to focus on the breath. So you try to put all attention into breathing in and breathing out. And crazy thing is that as you focus attention like that, after some time there starts coming back like some memories that you completely forgotten. It almost feels like that you have a mailbox and then you are just like archiving email one by one and at some point there is amazing feeling of getting to mailbox00 emails and it's very pleasant. It's crazy to me that once you resolve these inner stories or like inner traumas, then once there is nothing left, the default state of human mind is extremely peaceful and happy, extreme like some sense. It feels at least to me the way how when I was a child that I can look at any object and it's very beautiful. I have a lot of curiosity about the simple things and that's where usually meditation takes me.

What are you experiencing? Are you just taking in simple sensory information? And they're just enjoying the rawness of that sensory information. So there's no, there's no memories or all that kind of stuff, you're just enjoying being.

Yeah, pretty much. I mean still there is a. Thoughts are slowing down, sometimes they pop up. But it's also somehow the extended meditation takes you to the space that they are way more friendly, you way more positive. There is also this thing that it almost feels that. It almost feels that we are constantly getting a little bit of a reward function and we are just spreading this reward function on various activities. But if you stay still for extended period of time it kind of accumulates, accumulates, accumulates and. And there is a sense that at some point it passes some threshold and it feels as.