Ben Goertzel: Artificial General Intelligence | Lex Fridman Podcast #103

The following is a conversation with Ben Goertzel, one of the most interesting minds in the artificial intelligence community. He's the founder of SingularityNET, designer of OpenCOG AI framework, formerly a director of research at the Machine Intelligence Research Institute and chief scientist of Hanson Robotics, the company that created the Sophia robot. He has been a central figure in the AGI community for many years, including in his organizing and contributing to the Conference and Artificial General Intelligence, the 2020 version of which is actually happening this week, Wednesday, Thursday and Friday. It's virtual and free. I encourage you to check out the talks, including by Josha bach from episode 101 of this podcast. Quick summary of the ads two sponsors, the Jordan Harbinger show and Masterclass. Please consider supporting this podcast by going to Jordan Harbinger.com Lex and signing up@masterclass.com Lex click the links. Buy all the stuff. It's the best way to support this podcast and the journey I'm on in my research and startup. This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube.

Review it with five stars on Apple Podcast, support it on Patreon, or connect with me on Twitter. Alex Freedman spelled without the e, just F R I D M A N. As usual, I'll do a few minutes of AS now and never any ads in the middle that can break the flow of the conversation. This episode is supported by the Jordan Harbinger show. Go to jordanharbinger.com Lex it's how he knows I sent you on that page. There's links to subscribe to it on Apple Podcasts, Spotify and everywhere else. I've been binging on his podcast. Jordan is great. He gets the best out of his guests, dives deep, calls them out when it's needed, and makes the whole thing fun to listen to. He's interviewed Kobe Bryant, Mark Cuban, Neil Degrasse, Tyson, Garrett Kasparov and many more. His conversation with Kobe is a reminder how much focus and hard work is acquired for greatness in sport, business and life. I highly recommend the episode. If you want to be inspired again, go to jordanharbinger.com lex and it's how Jordan knows I sent you this show sponsored by Masterclass. Sign up@masterclass.com lex to get a discount and to support this podcast. When I first heard about Masterclass, I thought it was too good to be true. For 180 bucks a year you get an all access pass to watch courses from to list some of my favorites. Chris Hadfield on space exploration, Neil Degrasse Tyson on scientific thinking and communication. Will Wright, creator of the greatest city building game ever, SimCity and Sims on game design, Carlos Santana on guitar, Garry Kasparov, the greatest chess player ever on chess, Daniel Negrano on poker and many more. Chris Hadfield explaining how rockets work and the experience of being launched into space alone is worth the money. Once again, sign up on masterclass.com lex to get a discount and to support this podcast. And now here's my conversation with Ben Kurzel. What books, authors, ideas had a lot.

Of impact on you in your life? In the early days, you know, what.

Got me into AI and science fiction and such in the first place wasn't a book, but the original Star Trek TV show, which my dad watched with me in its first run. It would have been 1968, 69 or something. And that, that was incredible because every show they visited a different alien civilization with different culture and weird mechanisms. But that got me into science fiction, and there wasn't that much science fiction to watch on TV at that stage. That got me into reading the whole literature of science fiction from the beginning of the previous century until that time. I mean, there was so many science fiction writers who were inspirational to me. I'd say if I had to pick two, it would have been Stanislalm, the Polish writer. Yeah, Solaris. And then he had a bunch of more obscure writings on superhuman AIs that were engineered. Solaris was sort of a superhuman naturally occurring intelligence. Then Philip K. Dick, who ultimately my fandom for Philip K. Dick is one of the things that brought me together with David Hansen, my collaborator on robotics projects. So Stanislaume was very much an intellectual, right? So he had a very broad view of intelligence going beyond the human and into what I would call open ended superintelligence. The Solaris super intelligent Ocean was intelligent in some ways more generally intelligent than people, but in a complex and confusing way so that human beings could never quite connect to it. But it was still palpably very, very smart. And then the Golem 4 supercomputer in one of Lem's books, this was engineered by people, but eventually it became very intelligent in a different direction than humans and decided that humans were kind of trivial, not that interesting. So it put some impenetrable shield around itself, shut itself off from humanity, and then issued some philosophical screed about the pathetic and hopeless nature of humanity and all human thought, and then disappeared. Now, Philip K. Dick, he was a bit different. He was human focused, right? His main thing was human compassion and the human heart and soul are going to be the constant that will keep us going through whatever aliens we discover or telepathy machines or super AIs or whatever it might be. So he didn't believe in reality. Like the reality that we see may be a simulation or a dream or something else we can't even comprehend. But he believed in love and compassion as something persistent through the various simulated realities. So those two science fiction writers had a huge impact on me. Then a little older than that, I got into Dostoevsky and Friedrich Nietzsche and Rambod and a bunch of more literary typewriting.

Can we talk about some of those things? So on the Solaris side, Stanislav Lem, this kind of idea of there being intelligences out there that are different than our own, do you think there are intelligences maybe all around us that we're not able to even detect? So this kind of idea of maybe. You can comment also on Stephen Wolfram thinking that there's computations all around us and we're just not smart enough to kind of detect their intelligence or appreciate their intelligence.

Yeah. So my friend Hugo Daguerres, who I've been talking to about these things for many decades, since the early 90s, he had an idea he called SIPI, the search for Intraparticulate Intelligence Intelligence. So the concept there was. As AIs get smarter and smarter and smarter, assuming the laws of physics as we know them now are still what these superintelligences perceived to hold and are bound by, as they get smarter and smarter, they're going to shrink themselves littler and littler because special relativity makes it sort of communicate between two spatially distant points. So they're going to get smaller and smaller. But then ultimately, what does that mean? The minds of the super, super, super intelligences, they're going to be packed into the interaction of elementary particles, or quarks, or the partons inside quarks or whatever it is. So what we perceive as random fluctuations on the quantum or sub quantum level may actually be the thoughts of the micro, micro, micro miniaturized super intelligences. Because there's no way we can tell random from structured. But with an algorithmic information more complex in our brains, right, we can't tell the difference. So what we think is random could be the thought processes of some really tiny super minds. And if so, there's not a damn thing we can do about it except, you know, try to upgrade our intelligences and expand our minds so that we can, we can perceive more of what's around us.

But if the, if those random fluctuations, like even if we go to like quantum mechanics, if that's actually super intelligent systems, aren't we then part of the soup of superintelligence, aren't we just like a finger of the entirety of the body of the superintelligent system?

We could be. I mean a finger is a strange metaphor. I mean a finger is dumb is what I mean. But a finger is also useful and it's controlled with intent by the brain. Whereas we may be much less than that, right? I mean, yeah, we may be just some random epiphenomenon that they don't care about too much. Like think about the shape of the crowd emanating from a sports stadium or something, right? There's some emergent shape to the crowd. It's there, you could take a picture of it. It's kind of cool. It's irrelevant to the main point of the sports event or where the people are going or what's on the minds of the people making that shape in the crowd, right? So we may just be some semi arbitrary higher level pattern popping out of a lower level hyper intelligent self organization and I mean so be it, right? I mean that's one thing that still a fun ride? Yeah. I mean the older I've gotten, the more respect I've achieved for our fundamental ignorance. I mean mine and everybody else's. I mean I look at my two dogs, two beautiful little toy poodles and you know, they watch me sitting at the computer typing. They just think I'm sitting there wiggling my fingers to exercise and maybe or guarding the monitor on the desk. That they have no idea that I'm communicating with other people halfway around the world, let alone, you know, creating complex algorithms running in RAM on some computer server in St. Petersburg or something, right? Although they're right there, they're right there in the room with me. So what things are there right around us that we're just too stupid or closed minded to comprehend? Probably quite a lot.

You're very poodle could also be communicating across multiple dimensions with other beings and you're too unintelligent to understand the kind of communication mechanism they're going through.

There have been various TV shows and science fiction novels positing cats, dolphins, mice and whatnot are actually super intelligence is here to observe that. I would guess as one of the other quantum physics founders said those theories are not crazy enough to be true. The reality is probably crazier than that.

Beautifully put. So on the human side with Philip K. Dick and in General, where do you fall on this idea that love and just the basic spirit of human nature persists throughout these multiple realities? Are you on the side, like, the thing that inspires you about artificial intelligence? Is it the human side of somehow persisting through all of the different systems we engineer, or is AI inspire you to create something that's greater than human, that's beyond human, that's almost non human?

I would say my motivation to create AGI comes from both of those directions, actually. So when I first became passionate about AGI when I was. It would have been two or three years old after watching robots on Star Trek. I mean, then it was really a combination of intellectual curiosity, like, can a machine really think? How would you do that? And, yeah, just ambition to create something much better than all the clearly limited and fundamentally defective humans I saw around me. Then as I got older and got more enmeshed in the human world and, you know, got married, had children, saw my parents begin to age, I started to realize, well, not only will AGI let you go far beyond the limitations of the human, but it could also, like, stop us from dying and suffering and feeling pain and tormenting ourselves mentally. You can see AGI has amazing capability to do good for humans as humans, alongside with its capability to go far, far beyond the human level. So, I mean, both aspects are there, which makes it even more exciting and important.

So you mentioned Dostoyevsky and Nietzsche. What did you pick up from those guys?

I mean, that would probably go beyond the scope of a brief interviews. Both of those are amazing thinkers who one will necessarily have a complex relationship with. Dostoevsky on the minus side, he's kind of a religious fanatic and he sort of helped squash the Russian nihilist movement, which was very interesting because what nihilism meant originally in that period of the mid late 1800s in Russia was not taking anything fully 100% for granted. It was really more like what we'd call Bayesianism, where you don't want to adopt anything as a dogmatic certitude and always leave your mind open. And how Dostoevsky parodied nihilism was a bit different. He parodied his people who believe absolutely nothing, so they must assign an equal probability weight to every proposition, which doesn't really work. So on the one hand, I didn't really agree with Dostoyevsky on his sort of religious point of view. On the other hand, if you look at his understanding of human nature and sort of the human mind and heart and soul, it's really unparalleled and he had an amazing view of how human beings construct a world for themselves based on their own understanding and their own mental predisposition. And I think if you look in the Brothers Karamazov in particular, the Russian literary theorist Mikhail Bakhtin wrote about this as a polyphonic mode of fiction, which means it's not third person, but it's not first person from any one person really. There are many different characters in the novel, and each of them is sort of telling part of the story from their own point of view. So the reality of the whole story is an intersection, like synergetically, of the many different characters worldviews. And that really, it's a beautiful metaphor and even a reflection, I think, of how all of us socially create our reality. Like each of us sees the world in a certain way. Each of us, in a sense, is making the world as we see it based on our own minds and understanding. But it's polyphony, like in music, where multiple instruments are coming together to create the sound. The ultimate reality that's created comes out of each of our subjective understandings, you know, intersecting with each other. And that. That was one of the many beautiful things in Dostoyevsky.

So maybe a little bit to mention, you have a connection to Russia and the Soviet culture. I mean, I'm not sure exactly what the nature of the connection is, but there at least the spirit of your thinking.

Well, my, my ancestry is 3/4 Eastern European Jewish. So I mean, my three of my great grandparents emigrated to New York from Lithuania and sort of border regions of Poland, which were in and out of Poland around the time of World War I. And they were socialists and communists as well as Jews, mostly Menshevik, not Bolshevik. And they sort of. They fled at just the right time to the US for their own personal reasons. And then almost all, or maybe all of my extended family that remained in Eastern Europe was killed either by Hitler's or Stalin's minions at some point. So the branch of the family that emigrated to the US was pretty much the only one.

So how much of the spirit of the people is in your blood still? When you look in the mirror, do you see. What do you see?

Meat. I see a bag of meat that I want to transcend by uploading into some sort of superior reality. But very, I mean, yeah, very clearly. I mean, I'm. I'm not religious in a traditional sense, but clearly the, the Eastern European Jewish tradition was what I, what I was raised in. I mean, there was my grandfather, Leo Zwell, was a physical chemist who worked with Linus Pauling and a bunch of the other early greats in quantum mechanics. I mean, he was. He was into X ray diffraction. He was on the material science side, experimentalist rather than a theorist. His sister was also a physicist. And my father's father, Victor goertzel, was a PhD in psychology who had the unenviable job of giving psychotherapy to the Japanese in internment camps in the US in World War II to counsel them why they shouldn't kill themselves, even though they'd had all their stuff taken away and been imprisoned for no good reason. So, I mean, there's a lot of Eastern European Jewishness in my background. One of my great uncles was, I guess, conductor of San Francisco Orchestra. So there's a lot of Mickey Salkind bunch of music in there also. And clearly this culture was all about learning and understanding the world, and also not quite taking yourself too seriously while you do it. Right. There's a lot of Yiddish humor in there. So I do appreciate that culture. Although the whole idea that the Jews are the chosen people of God never resonated with me too much.

The graph of the Gretzl family, I mean, just the people I've encountered just doing some research and just knowing your work through the decades, it's kind of fascinating, just the number of PhDs.

Yeah, yeah. I mean, my dad is a sociology professor who recently retired from Rutgers University. But clearly that gave me a head start in life. I mean, my grandfather gave me all this quantum mechanics books, and I was like 7 or 8 years old. I remember going through them, and it was all the old quantum mechanics, like Rutherford atoms and stuff. So I got to the part of wave functions which I didn't understand, although I was very bright kid. And I realized he didn't quite understand it either. But at least he pointed me to some professor he knew at UPenn nearby who understood these things. Right. So that's an unusual opportunity for a kid to have. Right? My dad, he was programming Fortran when I was 10 or 11 years old on HP 3000 mainframes at Rutgers University. So I got to do linear regression in Fortran on punch cards when I was in middle school because he was doing, I guess, analysis of demographic and sociology data. So, yes, certainly that gave me a head start and a push towards science beyond what would have been the case with many different situations.

When did you first fall in love with AI? Is it the programming side of Fortran? Is it maybe the sociology psychology that you picked up from your Dad, I.

Found AI when I was probably three years old when I saw a robot on Star Trek. It was turning around in a circle going, error, error, error, error. Because Spock and Kirk had tricked it into a mechanical breakdown by presenting it with a logical paradox. And I was just like, well, this makes no sense. This AI is very, very smart. It's been traveling all around the universe. But these people could trick it with a simple logical paradox. Like, if the human brain can get beyond that paradox, why can't this AI? So I felt the screenwriters of Star Trek had misunderstood the nature of intelligence. And I complained to my dad about it and he wasn't going to say anything one way or the other. But before I was born, when my dad was at Antioch College in the middle of the US he led a protest movement called SLAM Student League Against Mortality. They were protesting against death wandering across the campus. So he was into some futuristic things even back then. But whether AI could confront logical paradoxes or not, he didn't know. But when I, ten years after that or something, I discovered Douglas Hofstadter's book, Gordel Escher Bach. And that was sort of to the same point of AI and paradox and logic, right? Because he was over and over with Godel's incompleteness theorem. And can an AI really fully model itself reflexively or does that lead you into some paradox? Can the human mind truly model itself reflexively or does that lead you into some paradox? So I think that book Gordela Esther Bach, which I think I read when it first came out, I would have been 12 years old or something. I remember it was like 16 hour day. I read it cover to cover and then reread it. I reread it after that because there was a lot of weird things with little formal systems in there that were hard for me at the time. But that was the first book I read that gave me a feeling for AI as a practical academic or engineering discipline that people were working in. Because before I read Gordela Scherbak, I was into AI from the point of view of a science fiction fan. And I had the idea, well, it may be a long time before we can achieve immortality and superhuman AGI, so I should figure out how to build a spacecraft traveling close to the speed of light, go far away, then come back to the earth in a million years when technology is more advanced and we can build these things. Reading Gordela Scherbak, well, it didn't all ring true to me. A lot of it did. But I could See, there are smart people right now at various universities around me who are actually trying to work on building what I would now call AGI, although Hofstadter didn't call it that. So really it was when I read that book, which would have been probably middle school, that then I started to think, well, this is something that I.

Could practically work, flying away and waiting it out. You can actually be one of the people that actually does this.

And if you think about, I mean, I was interested in what we'd now call nanotechnology and in the human immortality and time travel, all the same cool things as every other science fiction loving kid. But AI seemed like if Hofstadter was right, you just figure out the right program, sit there and type it. You don't need to spin stars into weird configurations or get government approval to cut people up and fiddle with their DNA or something. Right? It's just programming. And then of course that can achieve anything else. There's another book from back then which was by Feinbaum, Gerald Feinbaum, who was physicist at Princeton, and that was the Prometheus project. And this book was written in the late 1960s, so I encountered it in the mid-70s. But what this book said is in the next few decades, humanity is going to create superhuman thinking machines, molecular nanotechnology and human immortality. And then the challenge we'll have is what to do with it. Do we use it to expand human consciousness in a positive direction or do we use it just to further vapid consumerism? And what he proposed was that the UN should do a survey on this and the UN should send people out to every little village in remotest Africa or South America and explain to everyone what technology was going to bring the next few decades and the choice that we had about how to use it and let everyone on the whole planet vote about whether we should develop super AI, nanotechnology and, and immortality for expanded consciousness or for rampant consumerism. And needless to say, that didn't quite happen. And I think this guy died in the mid-80s, so he didn't even see his ideas start to become more mainstream. But it's interesting, many of the themes I'm engaged with now, from AGI and immortality even to trying to democratize technology as I've been pushing forward with Singularity, my work in the blockchain world, many of these themes were there in Feinbaum's book in the late 60s even. And of course Valentin Churchin, a Russian writer and a great Russian physicist who I got to know when we both lived in New York in the late 90s and early aughts. He had a book in the late 60s in Russia, which was the Phenomenon of Science which laid out all these same things as well. And Val died In, I don't remember, 2004, five or something of Parkinson's ism. So yeah, it's easy for people to lose track now of the fact that the futurist and singularitarian advanced technology ideas that are now almost mainstream and are on TV all the time. I mean these, these are not that new, right? They're sort of new in the history of the human species. But I mean these were all around in fairly mature form in the middle of the last century, were written about quite articulately by fairly mainstream people who were professors at top universities. It's just until the enabling technologies got to a certain point, then you couldn't make it real. Even in the 70s I was sort of seeing that and living through it right from Star Trek to Douglas Hofstader. Things were getting very, very practical from the late 60s to the late 70s. And the first computer I bought you could only program with hexadecimal machine code and you had to solder it together. And then like a few years later there's punch cards and a few years later you could get like Atari 400 and Commodore VIC 20 and you could type on a keyboard and program in higher level languages alongside the assembly language. So these ideas have been building up a while and I guess my generation got to feel them build up. Which is different than people coming into the field now for whom these things have just been part of the ambiance of culture for their whole career or even their whole life.

Well, it's fascinating to think about, you know, there being all of these ideas kind of swimming, you know, almost with a noise all around the world, all the different generations. And then some kind of non linear thing happens where they percolate up and capture the imagination of the mainstream. And that seems to be what's happening with AI now.

I mean, Nietzsche, who you mentioned had the idea of the superman, right? But he didn't understand enough about technology to think you could physically engineer a superman by piecing together molecules in a certain way. He was a bit vague about how the superman would appear, but he was quite deep at thinking about what the state of consciousness and the mode of cognition of a superman would be. He was a very astute analyst of, you know, how the human mind constructs the illusion of itself, how it constructs the illusion of free will, how it constructs values. Like good and evil, out of its own desire to maintain and advance its own organism. He understood a lot about how human minds work then he understood a lot about how post human minds would work. I mean, this Superman was supposed to be a mind that would basically have complete root access to its own brain and consciousness and be able to architect its own value system and inspect and fine tune all of its own biases. So that's a lot of powerful thinking there, which then fed in and sort of seeded all of postmodern continental philosophy. And all sorts of things have been very valuable in development of culture and indirectly even of technology. But of course, without the technology there, it was all some quite abstract thinking. So now, now we're at a time in history when a lot of these ideas can be. Can be made real, which is amazing. Amazing and scary. Right?

It's kind of interesting to think. What do you think Nietzsche would. If he was born a century later or transported through time, what do you think he would say about AI? I mean.

Well, those are quite different. If he's born a century later or transported through time.

Well, he'd be.

He'd be on like TikTok and Instagram and he would never write the great works he's written. So let's transport him.

Maybe. Maybe also would be a music video, right?

I mean.

I mean. I mean, who knows?

Yeah, but if he was transported through time, do you think that'd be interesting actually, to go back? You just made me realize that it's possible to go back and read Nietzsche with an eye of. Is there some thinking about artificial beings? I'm sure there. He has ink. He had inklings. I mean, with Frankenstein before him, I'm sure he had inklings of artificial beings somewhere in the text. It'd be interesting to see, to try to read his work, to see if he hadn't. If. If Superman was actually an AGI system, like if he had inklings of that kind of thinking.

He didn't.

He didn't.

No, I would say not. I mean, he had a lot of inklings of modern cognitive science, which are very interesting. If you look in the third part of the collection that's been titled the Will to Power. I mean, in book three there, there's very deep analysis of thinking processes, but he wasn't so much of a physical tinkerer type guy. Right. He was very abstract.

And do you think. What do you think about the Will to Power? Do you think human? What do you think drives humans?

Is it, oh, an unholy mix of things? I Don't think there's one pure, simple and elegant objective function driving humans by any means.

What do you think? If we look at, I know it's hard to look at humans in an aggregate, but do you think overall humans are good or do we have both good and evil within us that depending on the circumstances, depending on whatever, can percolate to the top?

Good and evil are very ambiguous, complicated and in some ways silly concepts. But if we could dig into your question from a couple directions. So I think if you look in evolution, humanity is shaped both by individual selection and what biologists would call group selection, like tribe level selection, right? So individual selection has driven us in a selfish DNA sort of way, so that each of us does, to a certain approximation, what will help us propagate our DNA to future generations. I mean, that's why I've got four kids so far and probably that's not the last one. On the other hand, I like the ambition. Tribal, like group selection means humans in a way will do what will advocate for the persistence of the DNA of their whole tribe or their social group. And in biology you have both of these, right? And you can see, say an ant colony or a beehive. There's a lot of group selection in the evolution of those social animals. On the other hand, say a big cat or some very solitary animal, it's a lot more biased toward individual selection. Humans are an interesting balance, and I think this reflects itself in what we would view as selfishness versus altruism to some extent. So we just have both of those objective functions contributing to the makeup of our brains. And then as Nietzsche analyzed in his own way, and others have analyzed in different ways, I mean, we abstract this as well. We have both good and evil within us, right? Because a lot of what we view as evil is really just selfishness. A lot of what we view as good is altruism, which means doing what's good for the tribe. And on that level, we have both of those just baked into us. And that's how it is. Of course, there are psychopaths and sociopaths and people who get gratified by the suffering of others. And that's a different thing.

Yeah, those are exceptions on the whole.

But I think at core we're not purely selfish. We're not purely altruistic. We are a mix and that's the nature of it. And we also have a complex constellation of values that are just very specific to our evolutionary history. We love waterways and mountains and the ideal place to put a house is In a mountain overlooking the water. You know, we care a lot about our kids, and we care a little less about our cousins and even less about our fifth cousins. I mean, there are many particularities to human values, which, whether they're good or evil, depends on your perspective, really. See, I spent a lot of time in Ethiopia, in Addis Ababa, where we have one of our AI development offices for my SingularityNET project. And when I walk through the streets in Addis, there's people lying by the side of the road, just living there by the side of the road, dying, probably of curable diseases, without enough food or medicine. And when I walk by them, I feel terrible. I give them money. When I come back home to the developed world, they're not on my mind that much. I do donate some, but I mean, I also spend some of the limited money I have enjoying myself in frivolous ways rather than donating it to those people who are right now like starving, dying and suffering on the roadside. So does that make me evil? I mean, it makes me somewhat selfish and somewhat altruistic, and we each balance that in our own way, right? So whether that will be true of all possible AGIs is a subtler question. That's how humans are.

So you have a sense, you kind of mentioned that there's a selfish. I'm not going to bring up the whole Ayn Rand idea of selfishness being the core virtue. That's a whole interesting kind of tangent that I think will just distract ourselves on.

I have to make one amusing comment or comment that has amused me. Anyway, so the, the, yeah, I have extraordinary negative respect for Ayn Rand.

Negative. What's the negative respect?

But when I worked with a company called Genesis, which was evolving flies to have extraordinary long lives in Southern California, so we had flies that were evolved by artificial selection to have five times the lifespan of normal fruit flies, but the population of super long lived flies was physically sitting in a spare room at an Ayn Rand elementary school in Southern California. So that was just like, wow, if I saw this in a movie, I wouldn't believe it.

Well, yeah, the universe has a sense of humor in that kind of way that fits in. Humor fits in somehow into this whole absurd existence. But you mentioned the balance between selfishness and altruism as kind of being innate. Do you think it's possible that's kind of an emergent phenomena? Those peculiarities of our value system, how much of it is innate? How much of it is something we collectively kind of like a Dostoevsky novel. Bring to life together as a civilization.

I mean, the answer to nature versus nurture is usually both. And of course it's nature versus nurture versus self organization, as you mentioned. So clearly there are evolutionary routes to individual and group selection leading to a mix of selfishness and altruism. On the other hand, different cultures manifest that in different ways, while we all have basically the same biology. And if you look at sort of pre civilized cultures, you have tribes like the Yanomamo in Venezuela, which their culture is focused on killing other tribes. And you have other stone age tribes that are mostly peaceable and have big taboos against violence. So you can certainly have a big difference in how culture manifests these innate biological characteristics. But still, you know, there's probably limits that are given by our biology. I used to argue this with my great grandparents who were Marxists actually, because they believed in the withering away of the state. Like they believed that as you move from capitalism to socialism to communism, people would just become more social minded. So that a state would be unnecessary and people would just give everyone, would give everyone else what they needed. Now, setting aside that that's not what the various Marxist experiments on the planet seem to be heading toward in practice, just as a theoretical point, I was very dubious that human nature could go there. At that time when my great grandparents are alive, I was just like, you know, I'm a cynical teenager. I think humans are, humans are just jerks. The state is not going to wither away if you don't have some structure keeping people from screwing each other over. They're going to do it. So now I actually don't quite see things that way. I mean, I think my feeling now subjectively is the culture aspect is more significant than I thought it was when I was a teenager. And I think you could have a human society that was dialed dramatically further toward, you know, self awareness, other awareness, compassion and sharing than our current society. And of course greater material abundance helps, but to some extent material abundance is a subjective perception also because many stone age cultures perceive themselves as living in great material abundance, that they had all the food and water they wanted, they lived in a beautiful place, they had sex lives, they had children. I mean, they had abundance without any factories, right? So I think humanity probably would be capable of fundamentally more positive and joy filled mode of social existence than what we have now. Clearly Marx didn't quite have the right idea about how to get there. I mean, he missed a number of, of key aspects of human society and its evolution. And if we look at where we are in society. Now how to get there is a quite different question because there are very powerful forces pushing people in different directions than a positive, joyous, compassionate existence. Right.

So if we were tried to, you know, Elon Musk dreams of colonizing Mars at the moment, so maybe he'll have a chance to start a new civil mobilization with a new governmental system. And certainly there's quite a bit of chaos. We're sitting now, I don't know what the date is, but this is June. There's quite a bit of chaos in all different forms going on in the United States and all over the world. So there's a hunger for new types of governments, new types of leadership, new types of systems. And so what are the forces at play and how do we move forward?

Yeah, I mean, colonizing Mars, first of all, it's a super cool thing to do. We should be doing it.

So you love the idea?

Yeah, I mean, it's more important than making chocolatier chocolates and sexier lingerie and many of the things that we spend a lot more resources on as a species. Right. So I mean, we certainly should do it. I think the, the possible futures in which a Mars colony makes a critical difference for humanity are very few. Assuming we make a Mars colony, people go live there in a couple of decades. I mean, their supplies are going to come from Earth. The money to make the colony came from Earth. And whatever powers are supplying the goods there from Earth are going to in effect be in control of that Mars colony. Of course there are outlier situations where Earth gets nuked into oblivion and somehow Mars has been made self sustaining by that point. And then Mars is what allows humanity to persist. But I think those are very, very, very unlikely.

You don't think it could be a first step on a long journey?

Of course it's the first step on a long journey, which is awesome. I'm guessing the colonization of the rest of the physical universe will probably be done by AGIs that are better designed to live in space than by the meat machines that we are. But I mean, who knows? We may cryopreserve ourselves in some superior way to what we know now and shoot ourselves out to Alpha Centauri and beyond. I mean, that's all cool, it's very interesting, and it's much more valuable than most things that humanity is spending its resources on. On the other hand, with AGI we can get to a singularity before the Mars colony becomes sustaining for sure. Possibly before it's even operational.

So your intuition is that that's the problem if we really invest resources and we can get to faster than a legitimate full, like self sustaining colonization of Mars.

Yeah, and it's very clear that we will to me, because there's so much economic value in getting from narrow AI toward AGI, whereas the Mars colony, there's less economic value until you get quite far out into the future. So I think that's very interesting. I just think it's somewhat, somewhat off to the side. I mean, just as I think, say, you know, art and music are very, very interesting and I want to see resources go into amazing art and music being created. And I'd rather see that than a lot of the garbage that society spends their money on. On the other hand, I don't think Mars colonization or inventing amazing new genres of music is not one of the things that is most likely to make a critical difference in the evolution of human or non human life in this part of the universe over the next decade.

Do you think AGI is really.

AGI is by far the most important thing that's on the horizon. And then technologies that have direct ability to enable AGI or to accelerate AGI are also very important. For example, say quantum computing. I don't think that's critical to achieve AGI, but certainly you could see how the right quantum computing architecture could massively accelerate AGI. Similar other types of nanotechnology. Right now, the quest to cure aging and end disease, while not in the big picture as important as AGI, of course it's important to all of us as individual humans. And if someone made a super longevity pill and distributed it tomorrow, I mean, that would be huge and a much larger impact than a Mars colony is going to have for quite some time.

But perhaps not as much as an AGI system.

No, because if you can make a benevolent AGI, then all the other problems are solved. I mean, then the AGI can be. Once it's as generally intelligent as humans, it can rapidly become massively more generally intelligent than humans. And then that, that AGI should be able to solve science and engineering problems much, much better than, than, than human beings as long as it is in fact motivated to do so. That's why I said a, a benevolent AGI. There could be other kinds.

Maybe it's good to step back a little bit. I mean, we've been using the term AGI. People often cite you as the creator or at least the popularizer of the term AGI, artificial general intelligence. Can you tell the origin story of the term?

So, yeah, I would say I launched the term AGI upon the world for what it's worth, without ever fully being in love with the term. What happened is I was editing a book and this process started around 2001 or 2002, I think the book came out 2005. Finally I was editing a book which I provisionally was titling real AI. And I mean the goal was to gather together fairly serious academic ish papers on the topic of making thinking machines that could really think in the sense like people can, or even more broadly than people can, right? So then I was reaching out to other folks that I had encountered here or there who were interested in that, which included some other folks who I knew from the transhumanist and singularitarian world, like Peter Vos, who has a company, AGI Incorporated still in California, and included Shane Legg, who had worked for me at my company Webmind in New York in the late 90s, who by now has become rich and famous. He was one of the co founders of Google DeepMind. But at that time Shane was, I think he may have just started doing his PhD with Marcus Hutter, who at that time hadn't yet published his book Universal AI, which sort of gives a mathematical foundation for artificial general intelligence. So I reached out to Shane and Marcus and Peter Vos and Pei Wang, who was another former employee of mine who had been Douglas Hofstadter's PhD student who had his own approach to AGI. And a bunch of some Russian folks reached out to these guys and they contributed papers for the book. But that was my provisional title. But I never loved it because in the end I was doing some what we would now call narrow AI as well, like applying machine learning to genomics data or chat data for sentiment analysis. And I mean, that work is real and in a sense it's really AI. It's just a different kind of kind of AI. Ray Kurzweil wrote about narrow AI versus strong AI, but that seemed weird to me because first of all, narrow and strong are not antonyms. But secondly, strong AI was used in the cognitive science literature to mean the hypothesis that digital computer AIs could have true consciousness like human beings. So there was already a meaning to strong AI, which was complexly different but related. So we were tossing around on an email list what title it should be. And so we talked about narrow AI, broad AI, wide AI, narrow AI, general AI. And I think it was either Shane Legg or Peter Vos. On the private email discussion we had, he said, well, why don't we go with AGI Artificial General Intelligence. And Pei Wang wanted to do gai, General Artificial intelligence, because in Chinese it goes in that order. But we figured gay wouldn't work in US culture at that time, Right. So we went with the AGI. We used it for the title of that book. And part of Peter and Shane's reasoning was you have the G factor in psychology, which is iq, General intelligence. Right. So you have a meaning of gi, General intelligence in psychology. So then you're looking like artificial intelligence, gi. So then that makes a lot of sense. Yeah, we use that for the title of the book. And so I think maybe both Shane and Peter think they invented the term. But then later, after the book was published, this guy Mark Gubrid came up to me and he's like, well, I published an essay with the term AGI in it in like 1997 or something. And so I'm just waiting for some Russian to come out and say they published that in 1953. I mean, that term is not dramatically innovative or anything. It's one of these obvious in hindsight things, which is also annoying in a way, because Josh Habakk, who you interviewed, is a close friend of mine. He likes the term synthetic intelligence, which I like much better, but it hasn't actually caught on. Right. Because artificial intelligence is a bit off to me, because artifice is like a tool or something. But not all AGIs are going to be tools. I mean, they may be now, but we're aiming toward making them agents rather than tools. And in a way, I don't like the distinction between artificial and natural because, I mean, we're part of nature also, and machines are part of nature. I mean, you can look at evolved versus engineered, but that's a different. That's a different distinction. Then it should be engineered, general intelligence, and then general. Well, if you look at Marcus Hutter's book, universally, what he argues there is within the domain of computation theory, which is limited but interesting. So if you assume computable environments, computable reward functions, then he articulates what would be a truly general intelligence, a system called aixi, which is quite beautiful.

Ixi.

I see. And that's. That's the middle name of. Of my latest child, actually. Is it.

What's the first name?

First name is Quarksy Q O R X I, which my. My wife came out with, but that. That's an acronym for Quantum Organized Rational Expanding Intelligence. And his middle name is. His middle name is Xiphanes, actually, which is mean, means the. The former principle underlying aix.

But in any Case, you're giving Elon Musk, new child, a run.

Well, I, I, I, I did it first. He, he, he cop this new freakish name. But now if I have another baby, I'm going to have to outdo him. It's become an arms race of weird geeky baby names. Yeah, we'll see what the babies think about it, right? Yeah, but I mean, my oldest son, Zarathustra loves his name and my daughter Shahzad loves her name so far. Basically, if you give your kids weird names, they live up to it. Well, you're obliged to make the kids weird enough that they like the names. Right. It directs their upbringing in a certain way. But yeah, anyway, I mean, what Marcus showed in that book is that a truly general intelligence theoretically is possible, but would take infinite computing power. So then the artificial is a little off. The general is not really achievable within physics as we know it. And I mean physics as we know it may be limited, but that's what we have to work with now, intelligence, infinitely general.

You mean like information processing, perspective?

Yeah, yeah. Intelligence is not very well defined either. What does it mean in AI? Now it's fashionable to look at it as maximizing an expected reward over the future, but that sort of definition is pathological in various ways. And my friend David Weinbaum, aka Weaver, he had a beautiful PhD thesis on open ended intelligence, trying to conceive intelligence in a.

Without a reward.

Yeah, he's just looking at it differently. He's looking at complex self organizing systems and looking at an intelligent system as being one that revises and grows and improves itself in conjunction with its environment, without necessarily there being one objective function it's trying to maximize. Although over certain intervals of time it may act as if it's optimizing a certain objective function. Very much Solaris from Stanislam's novels. Right. So yeah, the point is artificial general and intelligence don't work. They're all bad. On the other hand, everyone knows what AI is and AGI seems immediately comprehensible to people with a technical background. So I think that the term has served as sociological function and now it's out there everywhere, which baffles me.

It's like kfc. I mean, that's it. We're stuck with AGI probably for a very long time until AGI systems take over and rename themselves.

Yeah, and I mean, and then we'll be. We're stuck with GPUs too, which mostly have nothing to do with graphics anymore. Right.

I wonder what The AGI system will call us humans, maybe Grandpa gps, Grandpa Processing unit, Biological Grandpa Processing units. Okay, so maybe also just a comment on AGI representing before even the term existed, representing a kind of community. You've talked about this in the past, sort of AI has come in waves, but there's always been this community of people who dream about creating general human level superintelligence systems. Can you maybe give your sense of the history of this community as it exists today, as it existed before this deep learning revolution all throughout the winters and the summers of AI?

Sure. First I would say as a side point, the winters and summers of AI are greatly exaggerated by Americans. If you look at the publication record of the artificial intelligence community since say the 1950s, you would find a pretty steady growth and advance of ideas and papers. And what's thought of as an AI winter or summer was sort of how much money is the US military pumping into AI? Which was meaningful. On the other hand, there was AI going on in Germany, uk, in Japan, in Russia, all over the place, while US military got more and less enthused about AI. So what I mean that happened to.

Be just for people who don't know. The US military happened to be the main source of funding for AI research. So another way to phrase that is it's up and down of funding for artificial intelligence research.

And I would say the correlation between funding and intellectual advance was not 100%. Right. Because I mean in Russia as an example, or in Germany, there was less dollar funding than in the us but many foundational ideas were laid out. But it was more theory than implementation. And US really excelled at breaking through from theoretical papers to working implementations, which did go up and down somewhat with US military funding. But still, you can look in the 1980s, Dietrich Duhner in Germany had self driving cars on the Autobahn, right? And I mean this, it was a little early with regard to the car industry, so it didn't catch on such as has happened now. But I mean that whole advancement of self driving car technology in Germany was pretty much independent of AI military summers and winters in the us. So there's been more going on in AI globally than not only most people on the planet realize, but then most new AI PhDs realize because they've come up within this certain sub field of AI and haven't had to look so much beyond that. But I would say when I got my PhD in 1989 in mathematics, I was interested in AI already in Philadelphia. Yeah, I started at NYU, then I transferred to Philadelphia to Temple University. Good Old North Philly. North Philly, yeah, yeah, yeah. The Pearl of the US you never stopped at a red light then because you were afraid if you stopped at a red light someone will carjack you. So you strive through every red light.

Yeah.

Every day driving or bicycling to Temple from my house was like a new adventure, right? But yeah, the reason I didn't do a PhD in AI was what people were doing in the academic AI field then was just astoundingly boring and seemed wrong headed to me. It was really like rule based expert systems and production systems and actually I loved mathematical logic. I had nothing against logic as the cognitive engine for an AI, but the idea that you could type in the knowledge that AI would need to think seemed just completely stupid and, and wrong headed to me. I mean you can use logic if you want, but somehow the system has got to be automated learning, right? It should be learning from experience. And the AI field then was not interested in learning from experience. I mean some researchers certainly were. I mean I remember in mid-80s I discovered a book by John Andreas which was. It was about reinforcement learning system called Purpose P U R R P U S S which was an acronym that. I can't even remember what it was for but purpose anyway. But he, I mean that was a system that was supposed to be an AGI and basically by some sort of fancy like Markov decision process learning, it was supposed to learn everything just from the bits coming into it and learn to maximize its reward and become, become intelligent. Right? So that was there in academia back then. But it was like isolated scattered weird people. But all these isolated scattered weird people in that period, I mean they laid the intellectual grounds for what happened later. So you look at John Andreas at University of Canterbury with his purpose reinforcement learning Markov system. He was the PhD supervisor for John Cleary in New Zealand. Now John Cleary worked with me when I was at waikato University in 1993 in New Zealand and he worked with Ian Witten there and they launched weka, which was the first open source machine learning toolkit which was launched In I guess 93 or 94 when I was at Waikato University.

Written in Java unfortunately written in Java.

Which was a cool language back then though, right?

I guess it's still. Well it's not cool anymore, but it's powerful.

I find like most programmers now, I find Java unnecessarily bloated. But back then it was like Java or C basically and Java object oriented. So it's just Java was easier for students. Amusingly a lot of the work on Wekko when we were in New Zealand was funded by a us, sorry, a New Zealand government grant to use machine learning to predict the menstrual cycles of cows. So in the us all the grant funding for AI was about how to kill people or spy on people. In New Zealand it's all about cows or kiwifruits. Right, yeah. So yeah, anyway, I mean John Andreas had his probability theory based reinforcement learning proto AGI. John Cleary was trying to do much more ambitious probabilistic AGI systems. Now John Cleary helped do weka, which is the first open source machine learning toolkit, so the predecessor for TensorFlow and Torch and all these things. Also Shane Legg was at Waikato working with John Clear and Ian Witten and this whole group and then working with my own companies, my company, Webmind, an AI company I had in the late 90s with a team there at Waikato University, which is how Shane got his head full of AGI, which led him to go on and with demis Asabas, found DeepMind. So what you can see through that lineage is in the 80s and 70s, John Andreas was trying to build probabilistic reinforcement in AGI systems. The technology, the computers just weren't there to support. His ideas were very similar to what people are doing now. But although he's long since passed away and didn't become that famous outside of Canterbury, I mean the lineage of ideas passed on from him to his students, to their students. You can go trace directly from there to me and to DeepMind. Right. So there was a lot going on in AGI that did ultimately lay the groundwork for what we have today. But there wasn't a community. Right. And so when I started trying to pull together an AGI community, it was in the, I guess the early aughts when I was living in Washington D.C. and making a living doing AI consulting for various U.S. government agencies. And I organized the first AGI workshop in 2006 and I mean it wasn't like it was literally in my basement or something. I mean it was in the conference room at Marriott in Bethesda. It's not that, not that edgy or underground, unfortunately, but still, how many people attended? About 60 or something.

That's not bad.

I mean DC has a lot of AI going on probably until the last five or 10 years, much more than Silicon Valley, although it's just quiet because of the nature of what happens in dc. Their business isn't driven by pr. Mostly when something starts to work really? Well, it's taken black and becomes even more quiet. Right. But yeah, the thing is that really had the feeling of a group of starry eyed mavericks huddled in a basement plotting how to overthrow the narrow AI establishment and for the first time in some cases coming together with others who shared their passion for AGI and the technical seriousness about working on it. Right. Very, very different than what we have today. I mean, now it's a little bit different. We have AGI conference every year and there's several hundred people rather than 50. Now it's more like this is the main gathering of people who want to achieve AGI and think that large scale nonlinear regression is not the golden path to AGI, AKA neural network. Yeah, yeah, yeah. Well, certain architectures for learning using neural networks. So yeah, the AGI conferences are sort of now the main concentration of people not obsessed with deep neural networks, deep reinforcement learning, but still interested in AGI. Not the only ones. I mean, there's other little conferences and groupings interested in human level AI and cognitive architectures and so forth. But yeah, it's been a big shift. Back then you couldn't really. It would be very, very edgy then to give a university department seminar that mentioned AGI or human level AI. It was more like you had to talk about something more short term and immediately practical. Then in the bar after the seminar, you could bullshit about AGI in the same breath as time travel or the simulation hypothesis or something. Whereas now AGI is not only in the academic seminar room. You have, Vladimir Putin knows what AGI is and he's like, Russia needs to become the leader in AGI. So national leaders and CEOs of large corporations, I mean, the CTO of Intel, Justin Ratner, this was years ago, singularity summit conference, 2008 or something. He's like, we believe Ray Kurzweil, the Singularity will happen in 2045 and it will have intel inside. So it's gone from being something which is the pursuit of crazed mavericks, crackpots and science fiction fanatics to being a marketing term for large corporations and national leaders, which is an astounding transition. But yet in the course of this transition, I think a bunch of sub communities have formed and the community around the AGI conference series is certainly one of them. It hasn't grown as big as I might have liked it to. On the other hand, sometimes a modest sized community can be better for making intellectual progress. Also, like you get it with Society for Neuroscience conference, you have 35 or 40,000 neuroscientists on the one hand it's amazing. On the other hand you're not going to talk to the leaders of the field there if you're an outsider.

Yeah, in the same sense the aaai, the artificial intelligence, the main kind of generic artificial intelligence conference is too big, it's too amorphous, it doesn't make.

And NIPS has become a company advertising outlet now.

So yeah, so I mean to comment on the role of AGI in the research community I'd still, if you look at neurips, if you look at cvpr, if you look at these iclear, you know, AGI is still seen as the outcast. I would still, I would say in these main machine learning, in these main artificial intelligence conferences amongst the researchers, I don't know if it's an accepted term yet. What I've seen bravely you mentioned, Shane leg is DeepMind and then OpenAI are the two places that are, I would say unapologetically so far I think it's actually changing unfortunately. But so far they've been pushing the idea that the goal is to create an AGI.

Well they have billions of dollars behind them. So I mean in the public mind that, that certainly carries some oomph, right. I mean, I mean, but they also.

Really strong researchers, right?

They do, they're great teams. I mean DeepMind in particular. Yeah.

And they have, I mean DeepMind has Marcus Hutter walking around. I mean there's all these folks who basically their full time position involves, I mean dreaming about creating AGI.

I mean Google Brain has a lot of amazing AGI oriented people. Also, I'd say from a public marketing view, DeepMind and OpenAI are the two large, well funded organizations that have put the term and concept AGI out there sort of as part of their public image. But I mean they're certainly not. There are other groups that are doing research that seems just as AGI ish to me. I mean including a bunch of groups in Google's main Mountain View office. So yeah, it's true AGI is somewhat away from the mainstream now but if you compare to where it was 15 years ago, there's been an amazing mainstreaming. You could say the same thing about super longevity research which is one of my application areas that I'm excited about. I've been talking about this since the 90s but working on this since 2001 and back then really to say you're trying to create therapies to allow people to live hundreds or thousands of years, you were way, way, way, way out of the industry, academic mainstream. But now Google had Project Calico, Craig Venter at Human Longevity Incorporated. And once the suits come marching in, right? I mean once there's big money in it, then people are forced to take it seriously because that's the way modern society works. So it's still not as mainstream as cancer research, just as AGI is not as mainstream as automated driving or something. But the degree of mainstreaming that's happened in the last 10 to 15 years is astounding to those of us who've been at it for a while.

Yeah, but there's a marketing aspect to the term. But in terms of actual full force research that's going on under the header of AGI, it's currently, I would say dominated, maybe you can disagree, dominated by neural networks research. That the nonlinear regression as you mentioned. What's your sense with OpenCog, with your work in general? I always, logic based systems and expert systems for me always seemed to capture a deep element of intelligence that needs to be there. Like you said, it needs to learn, needs to be automated somehow. But that seems to be missing from a lot of research currently. So what's your sense of, I guess one way to ask this question, what's your sense of what kind of things will an AGI system need to have?

Yeah, that's a very interesting topic that I thought about for a long time and I think there are many, many different approaches that can work for getting to human level AI. So I don't think there's like one golden algorithm, one golden design that can work. And I mean flying machines is the much worn analogy here. Right. Like, I mean you have airplanes, you have helicopters, you have balloons, you have stealth bombers that don't look like regular airplanes. You've got all blimps.

Birds too.

Birds, yeah, and bugs. Right. And, and there are certainly many kinds of flying machines that, and there's a.

Catapult that you can just launch, there's.

Bicycle powered like flying machines. Right, Nice. Yeah, yeah. So now these are all analyzable by basic theory of aerodynamics right now. So one issue with AGI is we don't yet have the analog of the theory of aerodynamics. And that that's what Marcus Hoder was trying to make with the Axe and his general theory of general intelligence. But that theory, in its most clearly articulated parts really only works for either infinitely powerful machines or almost or insanely impractically powerful machines. So I mean, if you were going to take a theory based approach to AGI, what you would do is say well, let's. Let's take what's called, say, axe tl, which is Hutter's AXE machine that can work on merely insanely much processing power rather than infinitely much.

What does TL stand for?

Time and length. Okay, so you're basically how, like, constrained somehow. Yeah, yeah, yeah. So how AXC works basically is each action that it wants to take, before taking that action, it looks at all its history, and then it looks at all possible programs that it could use to make a decision, and it decides which decision program would have let it make the best decisions according to its reward function over its history, and it uses that decision program to make the next decision. Right.

It's not afraid of infinite resources.

It's searching through the space of all possible computer programs in between each action and each next action. Now, XCTL searches through all possible computer programs that have runtime less than T and length less than L, which is still an impracticably humongous space. Right. So what you would like to do to make an AGI and what will probably be done 50 years from now to make an AGI is say, okay, well, we have some constraints. We have these processing power constraints, and we have space and time constraints on the program, we have energy utilization constraints, and we have this particular class of environments that we care about, which may be, say, manipulating physical objects on the surface of the earth, communicating in human language. I mean, whatever our particular, not annihilating humanity, whatever our particular requirements are happen to be, if you formalize those requirements in some formal specification language, you should then be able to run automated program specializer on axetl. Specialize it to the computing resource constraints and the particular environment and goal, and then it will spit out the specialized version of AXTL to your resource restrictions in your environment, which will be your AGI. And that, I think, is how our super AGI will create new AGI systems. Right? That's a very Russian.

It seems really inefficient.

That's a very Russian approach, by the way. Like, the whole field of program specialization came out of Russia.

Can you backtrack? So what is program specialization?

So it's basically, well, take sorting, for example. You can have a generic program for sorting lists, but what if all your lists that you care about are length 10,000 or less?

Got it?

You can run an automated program specializer on your sorting algorithm, and it will come up with the algorithm that's optimal for sorting lists of length 1,000 or 10,000 or less. Right.

It's kind of like, isn't that the kind of the process of evolution is it's a program specializer to the environment. So you're kind of evolving human beings or.

Exactly. I mean your Russian heritage is showing there. I mean, so with Alexander Vitiav, I mean they're. And Peter Anokin and so on. I mean there's a long history of thinking about evolution that way also. Right. Well, my point is that what we're thinking of as a human level general intelligence. If you start from narrow AIs like are being used in the commercial AI field now, then you're thinking, okay, how do we make it more and more general? On the other hand, if you start from AIXI or Schmid Huber's Godel machine, or these infinitely powerful but practically infeasible AIs, then getting to a human level AGI is a matter of specialization. It's like how do you take these maximally general learning processes and how do you specialize them so that they can operate within the resource constraints that you have, but will achieve the particular things that you care about? Because we humans are not maximally general intelligences. Right. If I ask you to run a maze in 750 dimensions, you'll probably be very slow, whereas at two dimensions you're probably way better. Right? So I mean we're special because our hippocampus has a two dimensional map in it. Right. And it does not have a 750 dimensional map in it. So I mean we, we're, you know, a peculiar mix of generality and, and, and specialization. Right.

We probably start quite general at birth, not obviously still narrow, but like more general than we are at age 20 and 30 and 40 and 50 and 60.

I don't think that I, I think it's more complex than that because I mean, in some sense a young child is less biased and the brain has yet to sort of crystallize into appropriate structures for processing aspects of the physical and social world. On the other hand, the young child is very tied to their sensorium, whereas we can deal with abstract mathematics like 750 dimensions and the young child cannot because they haven't grown what Piaget called the formal capabilities. They haven't learned to abstract yet. Right. And the ability to abstract gives you a different kind of generality than what a baby has. So there's both more specialization and more generalization that comes with the development process, actually.

I mean, I guess just the trajectories of the specialization are most controllable at the young age, I guess is one way to put it.

Do you have kids?

No.

They're not as controllable as you think.

So you think it's interesting?

I think, honestly, interesting. A human adult is much more generally intelligent than a human baby. Babies are very stupid. I mean, they're cute, which is why we put up with their repetitiveness and stupidity. And they have what the Zen guys would call a beginner's mind, which is a beautiful thing, but that doesn't necessarily correlate with a high level of intelligence.

On the plot of cuteness and stupidity, there's a process that allows us to put up with their stupidity.

Every time you're an ugly old man like me, you got to get really, really smart to compensate.

To compensate.

Okay, cool. But yeah, going back to your original question, the way I look at human level AGI is how do you specialize unrealistically inefficient, superhuman, brute force learning processes to the specific goals that humans need to achieve and the specific resources that we have? And both of these, the goals and the resources and the environments, all this is important. And on the, on the resources side, it's important that the hardware resources we're bringing to bear are very different than the human brain. So the way I would want to implement AGI on a bunch of neurons in a vat that I could rewire arbitrarily is quite different than the way I would want to create AGI on, say, a modern server farm of CPUs and GPUs, which in turn may be quite different than the way I would want to implement AGI on whatever quantum computer we'll have in 10 years, supposing someone makes a robust quantum Turing machine or something. Right. So I think there's been co evolution of the patterns of organization in the human brain and the physiological particulars of the human brain over time. And when you look at neural networks, that is one powerful class of learning algorithms. But it's also a class of learning algorithms that evolve to exploit the particulars of the human brain as a computational substrate. If you're looking at the computational substrate of a modern server farm, you won't necessarily want the same algorithms that you want on the human brain. And from the right level of abstraction, you could look at maybe the best algorithms on the brain and the best algorithms on a modern computer network as implementing the same abstract learning and representation processes. But finding that level of abstraction is its own AGI research project then.

Right? Yeah.

So that's about the hardware side and the software side, which follows from that then, regarding what are the requirements I wrote the paper years ago on what I called the embodied communication prior, which was quite similar in intent to Yoshua Bengio's recent paper on the consciousness prior, except I didn't want to wrap up consciousness in it because to me the qualia problem and subjective experience is a very interesting issue also, which we can chat about. But I would rather keep that philosophical debate distinct from the debate of what kind of biases do you want to put in a general intelligence to give it human, like, general intelligence.

And I'm not sure Yoshua Benjo is really addressing that kind of consciousness. He's just using the term.

I love Yoshua to pieces. Like he's by far my favorite of the lines of deep learning. He's such a good hearted guy.

He's a good human being. Yeah, for sure.

I am not. I am not sure he has plumbed to the depths of the philosophy of consciousness.

No, he's using it as a sexy.

Yeah, yeah, yeah. So what I called it was the embodied communication prior.

Can you maybe explain it a little bit?

Yeah, yeah. What I meant was, you know, what are we humans evolved for? You can say being human, but that's very abstract, right? I mean, our minds control individual bodies which are autonomous agents moving around in a world that's composed largely of solid objects. Right. And we've also evolved to communicate via language with other, you know.