Jeff Hawkins: The Thousand Brains Theory of Intelligence | Lex Fridman Podcast #208

The following is a conversation with Jeff Hawkins, a neuroscientist seeking to understand the structure, function and origin of intelligence in the human brain. He previously wrote the seminal book on the subject, titled On Intelligence, and recently a new book called A Thousand Brains, which presents a new theory of intelligence that Richard Dawkins, for example, has been raving about, calling the book brilliant and exhilarating. I can't read those two words and not think of him saying it in his British accent. Quick mention of our sponsors CodeCademy, Bio Optimizers, ExpressVPN, Asleep and Blinkist. Check them out in the Description to support this podcast. As a side note, let me say that one small but powerful idea that Jeff Hawkins mentions in his new book is that if human civilization were to destroy itself, all of knowledge, all our creations would go with us. He proposes that we should think about how to save that knowledge in a way that long outlives us, whether that's on Earth, in orbit, around Earth, or in deep space, and then to send messages that advertise this backup of human knowledge to other intelligent alien civilizations. The main message of this advertisement is not that we are here, but that we were once here. This little difference somehow was deeply humbling to me. That we may, with some non zero likelihood, destroy ourselves, and that an alien civilization thousands or millions of years from now may come across this knowledge store and they would only, with some low probability, even notice it, not not to mention be able to interpret it. And the deeper question here for me is what information in all of human knowledge is even essential? Does Wikipedia capture it or not at all? This thought experiment forces me to wonder, what are the things we've accomplished and are hoping to still accomplish that will outlive us? Is it things like complex buildings, bridges, cars, rockets? Is it ideas like science, physics and mathematics? Is it music and art? Is it computers, computational systems, or even artificial intelligence systems? I personally can't imagine that aliens wouldn't already have all of these things. In fact, much more and much better. To me, the only unique thing we may have is consciousness itself, and the actual subjective experience of suffering, of happiness, of hatred, of love. If we can record these experiences in the highest resolution directly from the human brain, such that aliens would be able to replay them. That is what we should store and send as a message. Not Wikipedia, but the extremes of conscious experiences, the most important of which of course, is love. This is the Lex Friedman podcast, and here is my conversation with Jeff Hawkins. We previously talked over two years ago. Do you think there's still neurons in your brain that remember that conversation, that remember me and got excited like there's a Lex neuron in your brain that just like finally has a purpose.

I do remember our conversation or I have some memories of it and I formed additional memories of you in the meantime. I wouldn't say there's a neuron or a neurons in my brain that know you, but there are synapses in my brain that have formed that reflect my knowledge of you and the model I have of you in the world. And whether they're the exact same synapses were formed two years ago, it's hard to say because these things come and go all the time. But we know from. One of the things we know about brains is that when you think of things, you often erase the memory and rewrite it again. So yes, but I have a memory of you and that's instantiated in synapses. There's a simpler way to think about it. We have a model of the world in your head and that model is continually being updated. I updated this morning. You offered me this water setups from the refrigerator. I remember these things. And so the model includes where we live, the places we know, the words, the objects in the world. It's just monstrous model. And it's constantly being updated. And people are just part of that model. So are animals, so are other physical objects. So are our events we've done so it's no special in my mind special place for the memories of humans. I mean, obviously I know, you know, I know a lot about my wife, but. And friends and so on. But it's not like a special place for humans or over here. But we model everything and we model other people's behaviors too. So if I said yours is a copy of your mind, in my mind it's just because I know how humans. I've learned how humans behave and I've learned some things about you and that's part of my world model.

Well, I just also mean the collective intelligence of the human species. I wonder if there is something fundamental to the brain that enables that. So modeling other humans with their ideas.

You'Re actually jumping into a lot of big topics. Like collective intelligence is a separate topic that a lot of people like to talk about. We could talk about that. So that's interesting. We're not just individuals, we live in society and so on. But from our research point of view, again, let's just talk. We study the neocortex. It's a sheet of neural tissue. It's about 75% of your brain, it runs on this very repetitive algorithm. It's a very repetitive circuit. You can apply that algorithm to lots of different problems, but it's all underneath. It's the same thing. We're just building this model from our point of view. We wouldn't look for these special circuits someplace barely carried in your brain that might be related to other, you know, understanding other humans. It's more like, you know, how do we build a model of anything? How do we understand anything in the world? And humans are just another part of the things we understand.

So there's nothing. There's nothing to the brain that knows the emergent phenomena of collective intelligence?

Well, I certainly know about that. I've heard the terms. I've read.

No, but that's. Right.

Well, okay.

Right as an idea.

Well, I think we have language which is sort of built into our brains, and that's a key part of collective intelligence. So there are some prior assumptions about the world we're going to live in when we're born. We're not just a blank slate. And so did we evolve to take advantage of those situations? Yes, but again, we study only part of the brain, the neocortex. There's other parts of the brain are very much involved in societal interactions and human emotions and how we interact and even societal issues about how we interact with other people, when we support them, when we're greedy and things like that.

I mean, certainly the brain is a great place where to study intelligence. I wonder if it's the fundamental atom of intelligence.

Well, I would say it's absolutely an essential component. Even if you believe in collective intelligence, as, hey, that's where it's all happening. That's what we need to study, which I don't believe that, by the way. I think it's really important, but I don't think that is the thing. But even if you do believe that, then you have to understand how the brain works in doing that. It's more like we are intelligent individuals and together we are much more magnified our intelligence. We can do things which we couldn't do individually, but even as individuals, we're pretty damn smart and we can model things and understand the world and interact with it. So to me, if you're going to start someplace, you need to start with the brain. Then you could say, well, how do brains interact with each other? And what is the nature of language? And how do we share models that I've learned something about the world. How do I share it with you? Which is really what sort of communal Intelligence is. I know something, you know something. We've had different experiences in the world. I've learned something about brains. Maybe I can impart that to you. You've learned something about physics and you can impart that to me. But it also comes down to even just a epistemological question of, well, what is knowledge and how do you represent it in the brain? Right. It's not. That's where it's going to reside. Right.

In our writings, it's obvious that human collaboration, human interaction, is how we build societies.

Right?

But some of the things you talk about and work on, some of those elements of what makes up an intelligent entity is there with a single person.

Absolutely. I mean, we can't deny that the brain is the core element here. In at least, I think it's obvious. The brain is the core element in all theories of intelligence. It's where knowledge is represented. It's where knowledge is created. We interact, we share, we build upon each other's work. But without a brain, you'd have nothing. There would be no intelligence without brains. And so. So that's where we start. I got into this field because I just was curious as to who I am. How do I think what's going on in my head when I'm thinking, what does it mean to know something? I can ask what it means for me to know something independent of how I learned it from you or from someone else or from society. What does it mean for me to know that I have a model of you in my head? What does it mean to know I know what this microphone does and how it works physically, even though I can't see it right now, how do I know that? What does it mean? How do the neurons do that? At the fundamental level of neurons and synapses and so on, those are really fascinating questions, and I'm happy to be just happy to understand those if I could.

So in your new book, you talk about our brain, our mind, as being made up of many brains. So the book is called A Thousand Brains, A Thousand Brain Theory of Intelligence. What is the key idea of this book?

The book has three sections, and it has sort of maybe three big ideas. So the first section is all about what we've learned about the neocortex, and that's the 1000 brains theory. Just to complete the picture, the second section is all about AI, and the third section is about the future of humanity. So the Thousand Brains theory, the big idea there, if I had to summarize into one big idea, is that we think of the brain. The neocortex is learning this model of the world. But what we learned is actually there's tens of thousands of independent modeling systems going on. And so each what we call a column in the cortex, there's about 150,000 of them, is a complete modeling system. So it's a collective intelligence in your head in some sense. So the thousand brains theory says about where do I have knowledge about this coffee cup? Or where is the model of this cell phone? It's not in one place. It's in thousands of separate models that are complementary and they communicate with each other through voting. So this idea that we have, we feel like we're one person, that's our experience. We can explain that. But reality, there's lots of these, almost like little brains, but they're sophisticated modeling systems, about 150,000 of them in each of a human brain. And that's a total different way of thinking about how the neocortex is structured than we or anyone else thought of even just five years ago.

So you mentioned you started this journey just looking in the mirror and trying to understand who you are. So if you have many brains, who are you then?

So it's interesting. We have a singular perception, right? We think, oh, I'm just here, I'm looking at you. But it's composed of all these things. There's sounds and there's vision and there's touch and all, all kinds of inputs. Yet we have the singular perception. And what 1000 brains theory says, we have these models that are visual models, we have models that are auditory models, models that talk to models and so on, but they vote. And so in the cortex, you can think about these columns as like little grains of rice, 150,000 stacked next to each other. And each one is its own little modeling system. But they have these long range connections that go between them. And we call those voting connections or voting neurons. And so the different columns try to reach the consensus, like, what am I looking at? Okay. Each one has some ambiguity, but they come to a consensus. Oh, there's a water bottle I'm looking at. We are only consciously able to perceive the voting. We're not able to perceive anything that goes under the hood. So the voting is what we're aware.

Of the results of the voting.

Yeah, the resulting. Well, you can imagine it this way. We were just talking about eye movements a moment ago. So as I'm looking at something, my eyes are moving about three times a second. And with each movement, a completely new input is coming into the brain. It's not repetitive, it's not shifting around, it's completely new. I'm totally unaware of it, I can't perceive it. But yet if I looked at the neurons in your brain, they're going on and off, on and off, on and off, on and off. But the voting neurons are not. The voting neurons are saying, you know, we all agree. Even though I'm looking at different parts of this is a water bottle right now and that's not changing and it's in some position and pose relative to me. So I have this perception of the water bottle about 2ft away from me at a certain pose to me that is not changing. That's the only part I'm aware of. I can't be aware of the fact that the inputs from the eyes are moving and changing and all this other tapping. So these long range connections are the part we can be conscious of. The individual activity in each column is. It doesn't go anywhere else, it doesn't get shared anywhere else. There's no way to extract it and talk about it or extract it and even remember it to say, oh yes, I can recall that. But these long range connections are the things that are accessible to language and to our like the hippocampus, our memories, our short term memory systems and so on. So we're not aware of 95%, or maybe it's even 98% of what's going on in your brain. We're only aware of this sort of stable, somewhat stable voting outcome of all these things that are going on underneath the hood.

So what would you say is the basic element in the thousand brains theory of intelligence, of intelligence. Like what's the atom of intelligence when you think about it? Is it the individual brains? And then what is a brain?

Well, can we just talk about what intelligence is first and then we can talk about what the elements are. So in my book, intelligence is the ability to learn a model of the world, to build internal to your head. A model that represents the structure of everything. To know that this is a table and that's a coffee cup and this is a gooseneck lamp and all this. To know these things I have to have a model of them in my head. I just don't look at them and go, what is that? I already have internal representations of these things in my head and I had to learn them. I wasn't born of any of that knowledge. We have some lights in the room here. That's not part of my evolutionary heritage. It's not in my Genes. So we have this incredible model. And the model includes not only what things look like and feel like, but where they are relative to each other and how they behave. I've never picked up this water bottle before, but I know that if I put my hand on that blue thing and I turn it, it'll probably make a funny little sound as the little plastic things detach, and then it'll rotate, and it'll rotate a certain way and it'll come off. How do I know that? Because I have this model in my head. So the essence of intelligence is our ability to learn a model. And the more sophisticated our model is, the smarter we are. Not that there is a single intelligence, because you know a lot about things that I don't know and I know about things you don't know. And we can both be very smart, but we both learn to model of the world through interacting with it. So that is the essence of intelligence. Then we can ask ourselves, what are the mechanisms in the brain that allow us to do that? And what are the mechanisms of learning? Not just the neural mechanisms. What is the general process for how we learn a model? So that was a big insight for us. It's like, what is the actual things that. How do you learn this stuff? It turns out you have to learn it through movement. You can't learn it just by. That's how we learn. We learn through movement. We learn. So you build up this model by observing things and touching them and moving them and walking around the world and so on.

So either you move or the thing moves somehow.

Yeah, Obviously you can learn things just by reading a book, something like that. But think about if I were to say, oh, here's a new house I want you to learn. You know, what do you do? You have to walk. You have to walk from room to room. You have to open the doors, look around, see what's on the left, what's on the right. As you do this, you're building a model in your head. It's just. That's what you're doing. You can't just sit there and say, I'm going to grok the house. No. You know, or you could. You don't even want to just sit there and read some description of it, right?

Yeah.

You literally physically interact. And the same with like a smartphone. If I'm going to learn a new app, I touch it and I move things around. I see what happens when I. When I do things with it. So that's the basic way we learn in the world.

And by the way when you say model, you mean something that can be used for prediction in the future.

It's used for prediction and for behavior.

And planning and does a pretty good job at doing so.

Yeah. Here's the way to think about the model. A lot of people get hung up on this. So you can imagine an architect making a model of a house. So there's a physical model. There's. That's small. And why do they do that? Well, we do that because you can imagine what it would look like from different angles. Okay, look from here, look from there. And you can also say, well, how far would it get from the garage to the swimming pool? Or something like that? Right. You can imagine looking at this and you say, what would be the view from this location? So we build these physical models to let you imagine the future and imagine behaviors. Now we can take that same model and put it in a computer. So we now, today, they'll build models of houses in a computer, and they do that using a set of. We'll come back to this term in a moment. Reference frames. But basically, you assign a reference frame for the house, and you assign different things for the house in different locations. And then the computer can generate an image and say, okay, this is what it looks like in this direction. The brain is doing something remarkably similar to this. Surprising. It's using reference frames, it's building these. It's similar to a model in a computer, which has the same benefits of building a physical model. It allows me to say, what would this thing look like if it was in this orientation? What would likely happen if I push this button? I've never pushed this button before. Or how would I accomplish something? I want to convey a new idea. I've learned. How would I do that? I can imagine in my head. Well, I could talk about it. I could write a book. I could do some podcasts. I could maybe tell my neighbor. And I can imagine the outcomes of all these things before I do any of them. That's what the model lets you do. Let's just plan the future and imagine the consequences of our actions. Prediction. You asked about prediction. Prediction is not the goal of the model. Prediction is an inherent property of it, and it's how the model corrects itself.

So prediction is fundamental to intelligence.

It's fundamental to building a model. And the model's intelligent. Let me go back and be very precise about this prediction. You can think of prediction two ways. One is like, hey, what would happen if I did this? That's a type of prediction. That's a key part of intelligence. But even prediction is like, oh, what this water bottle going to feel like when I pick it up? And that doesn't seem very intelligent. But the way to think, one way to think about intelligence prediction is it's a way for us to learn where our model is wrong. So if I picked up this water bottle and it felt hot, I'd be very surprised. Or if I picked it up, it was very light, I'd be surprised. Or if I turn this top and it didn't, I had to turn it the other way, I'd be surprised. And so all those might have a prediction like, okay, I'm going to do it. I'll drink some water. I'm going to, okay, do this. There it is. I feel opening, right? What if I had to turn it the other way? Or what if it split in two? Then I say, oh my gosh, I misunderstood this. I didn't have the right model. This thing. My attention would be drawn to it. I'll be looking at it going, well, how the hell did that happen? Why did it open up that way? And I would update my model by doing it just by looking at it and playing around that update and say, this is a new type of water bottle.

So you're talking about sort of complicated things like a water bottle. But this also applies for just basic vision, just like seeing things. That's almost like a precondition of just perceiving the world is predicting it. Everything that you see is first passed through your prediction.

Everything you see and feel. In fact, this is the insight I In the late 80s, excuse me, early 80s and other people reach the same idea is that every sensory input you get, not just vision, but touch and hearing, you have an expectation about it and a prediction. Sometimes you can predict very accurately, sometimes you can't. I can't predict what next word is going to come out of your mouth. But as you start talking, I'll better and better predictions. And if you talked about some topics, I'd be very surprised. So I have this sort of background prediction that's going on all the time for all of my senses. Again, the way I think about that is this is how we learn. It's more about how we learn. It's a test of our understanding. Our predictions are a test. Is this really a water bottle? If it is, I shouldn't see a little finger sticking out the side. And if I saw a little finger sticking out, I was like, what the hell's going on? That's not normal.

I mean, that's fascinating that. Let me linger on this for a second. It really honestly feels that prediction is fundamental to everything, to the way our mind operates, to intelligence. So like it's just a different way to see intelligence, which is like everything.

Starts at prediction and prediction requires a model. You can't predict something unless you have a model of it.

Right, but the action is prediction. So like the thing the model does is prediction.

And, but it also. Yeah, but you can then extend it to things like what would happen if I took this today, I went and did this, what would be likely? Or how you can extend prediction. Like, oh, I want to get a promotion at work, what action should I take? And you can say, if I did this, I predict what might happen. If I spoke to someone, I predict what might happen. So it's not just low level predictions.

Yeah, it's all predictions.

It's all predictions.

It's like this black box. So you can ask basically any question, low level or high level.

So we started off with that observation. It's all, it's this non stop prediction. And I write about this in the book about. And then we asked how do neurons actually make predictions physically? Like what does the neuron do when it makes a prediction and. Or the neural tissue does when it makes prediction. And then we asked what are the mechanisms by how we build a model that allows you to make prediction. So we started with prediction as sort of the fundamental research agenda, if in some sense, and say, well, if we understand how the brain makes predictions, we'll understand how it builds these models and how it learns. And that's the core of intelligence. So it was the key that got us in the door to say that is our research agenda, understand predictions.

So in this whole process, where does intelligence originate? Would you say? So at what if we look at things that are much less intelligent than humans and you start to build up a human through the process of evolution. Where's this magic thing that has a prediction model or a model that's able to predict that starts to look a lot more like intelligence. Is there a place where Richard Dawkins wrote an introduction to your book? An excellent introduction. I mean it puts a lot of things into context and it's funny just looking at parallels for your book and Darwin's Origin of Species. So Darwin wrote about the origin of species. So what is the origin of intelligence?

Yeah, well, we have a theory about it and it's just that it's a theory. Theory goes as follows. As soon as living things started to move, they're not Just floating in sea. They're not just a plant, you know, grounded someplace. As soon as they started to move, there was an advantage to moving intelligently, to moving in certain ways. And there's some very simple things you can do. You know, bacteria or single cell organisms can move towards a source of gradient of food or something like that. But an animal that might know where it is and know where it's been and how to get back to that place, or an animal that might say, oh, there was a source of food someplace, how do I get to it? Or there was a danger, how do I get to. Or there was a mate, how do I get to them. There was a big evolutionary advantage to that. So early on there was a pressure to start understanding your environment, like where am I and where have I been and what happened in those different places. So we still have this neural mechanism in our brains in the mammals. It's in the hippocampus and entorhinal cortex. These are older parts of the brain and these are very well studied. We build a map of the, of our environment. So these neurons in these parts of the brain know where I am in this room and where the door was and things like that.

So a lot of other mammals have this.

All mammals have this, right? And almost any, any animal that knows where it is and get around must have some mapping system, must have some way of saying, I've learned a map of my environment. I have hummingbirds in my backyard and they go the same places all the time. They must know where they are. They just know where they are when they're, they're not just randomly flying around. They know, they know particular flowers they come back to. So we all have this. And it turns out it's very tricky to get neurons to do this, to build a map of an environment. And so we now know there's these famous studies that still very active about place cells and grid cells and these other types of cells in the older parts of the brain and how they build these maps of the world. It's really clever. It's obviously been under a lot of evolutionary pressure over a long period of time to get good at this. So animals know where they are, what we think has happened, and there's a lot of evidence to suggest this is that that mechanism we learned to map like a space is, was repackaged. The same type of neurons was repackaged into a more compact form and that became the cortical column. And it was in some sense genericized, if that's a word. It was turned into a very specific thing about learning maps of environments, to learning maps of anything, learning a model of anything, not just your space, but coffee cups and so on. And it got sort of repackaged into a more compact version, a more universal version, and then replicated. So the reason we're so flexible is we have a very generic version of this mapping algorithm, and we have 150,000 copies of it.

Sounds a lot like the progress of deep learning.

How so?

Take neural networks that seem to work well for a specific task, compress them and multiply it by a lot, and then you just stack them on top of it. It's like the story of transformers in.

But in addition, keep learning networks. They end up you're replicating an element, but you still need the entire network to do anything. Here. What's going on? Each individual element is a complete learning system. This is why I can take a human brain, cut it in half, and it still works. It's pretty amazing.

It's fundamentally distributed.

It's fundamentally distributed, Complete modeling systems. But that's our story we like to tell. I, I would guess it's likely largely right, but there's a lot of evidence supporting that story, this evolutionary story. The thing which brought me to this idea is that the human brain got big very quickly. So that led to the proposal a long time ago that, well, there's this common element. Instead of creating new things, it just replicated something. We also are extremely flexible. We can learn things that we had no history about. Right. And so that tells you that the learning algorithm is very generic. It's very kind of universal because it doesn't assume any prior knowledge about what it's learning. And so you combine those things together and you say, okay, well, how did that come about? Where did that universal algorithm come from? It had to come from something that wasn't universal. It came from something that was more specific. Anyway, this led to our hypothesis that you would find grid cells and place cell equivalents in the neocortex. And when we first published our first papers on this theory, we didn't know of evidence for that. It turns out there was some, but we didn't know about it. And since then. So then we became aware of evidence for grid cells in parts of the neocortex. And then now there's been new evidence coming out. There's some interesting papers that came out just January of this year. So one of our predictions was if this evolutionary hypothesis is correct, we would see grid cell place cell equivalent cells that work like them through Every column in your cortex, and that's starting to be seen.

What does it mean that. Why is it important that they're present?

Because it tells us. Well, we're asking about the evolutionary origin of intelligence. Right. So our theory is that these columns in the cortex are working on the same principles. They're modeling systems. And it's hard to imagine how neurons do this. And so we said, hey, it's really hard to imagine how neurons could learn these models with things. We can talk about the details of that if you want, but there's this other part of the brain we know that learns models of environments. So could that mechanism to learn to model this room be used to learn to model the water bottle? Is it the same mechanism? So we said it's much more likely the brain's using the same mechanism, which case it would have these equivalent cell types. So it's basically the whole theory is built on the idea that these columns have reference frames and they're learning these models, and these grid cells create these reference frames. So it's basically, in some sense, the major predictive part of this theory is that we will find these equivalent mechanisms in each column in the neocortex, which tells us that that's what they're doing. They're learning these sensory motor models of the world. So we're pretty confident that would happen, but now we're seeing the evidence.

So the evolutionary process. Nature does a lot of copy and paste and see what happens.

Yeah, yeah. There's no direction to it, but. But it just found out, like, hey, if I took this. These elements and made more of them, what happens? And let's hook them up to the eyes, and let's hook them up to ears. And. And.

And that seems to work pretty well.

Yeah.

For us. Again, just to take a quick step back to our conversation of collective intelligence. Do you sometimes see that as just another copy and paste aspect? Is copying, pasting these brains in humans and making a lot of them and then creating social structures that then almost operate as a single brain?

I wouldn't have said it, but you said it sounded pretty good.

So to you, the brain is fundamental. Is. Is like. Is its own thing.

I mean, our goal is to understand how the neocortex works, and we can argue how essential that is to understand the human brain, because it's not the entire human brain. You can argue how essential that is to understanding human intelligence. You can argue how essential it is to sort of communal intelligence. Our goal was to understand the neocortex.

Yeah. So what Is the neocortex and where does it fit in the various aspect of what the brain does? How important is it to you?

Well, obviously again I mentioned again in the beginning, it's about 70 to 75% of the volume of a human brain. So it dominates our brain in terms of size, not in terms of number of neurons, but in terms of size.

Size isn't everything, Jeff.

I know, but it's not that. We know that all high level vision, hearing and touch happens in neocortex. We know that all language occurs and is understood in the neocortex. Whether that's spoken language, written language, sign language, language of mathematics, language of physics, music, you know, we know that all high level planning and thinking occurs in the neocortex. If I were to say, you know, what part of your brain designed a computer and understands programming and creates music? It's all the neocortex. So then that's an undeniable fact. But then there's other parts of our brain are important too, right? Our emotional states, our body, regulating our body. So the way I like to look at it is can you understand the neocortex about the rest of the brain? And some people say you can't. And I think absolutely you can. It's not that they're not interacting, but you can understand them. Can you understand the neocortex without understanding the emotions of fear? Yes, you can. You can understand how this system works. It's just a modeling system. I make the analogy in the book that it's like a map of the world. And how that map is used depends on who's using it. So how our map of our world in our neocortex, how we manifest as a human, depends on the rest of our brain. What are our motivations? What are my desires? Am I a nice guy or not a nice guy? Am I a cheater or not a cheater? How important different things are in my life. But the neocortex can be understood on its own. And I say that as a neuroscientist. I know there's all these interactions and I want to say I don't know them and we don't think about them. But from a layperson's point of view, you can say it's a modeling system. I don't generally think too much about the communal aspect of intelligence, which you've brought up a number of times already. So that's not really been my concern.

I just wonder if there's a continuum from the origin of the universe. This Pockets of complexities that form.

Yeah.

Living organisms. I wonder if, if we're just. If you look at humans, we feel like we're at the top, but I wonder if there's like just where everybody, probably every living type pocket of complexity is. Probably thinks they're the. Pardon the French. They're the. Yeah, well, they're at the top of the parent.

Well, if they're thinking.

Well, then what is thinking? What the. In a sense. The whole point is in their sense of the world. Their sense is that they're at the top of it.

I think what is a turtle. But you're bringing up the problems of complexity and complexity theory are. It's a huge interesting problem in science, and I think we've made surprisingly little progress in understanding complex systems in general. And so the Santa Fe Institute was founded to study this. And even the scientists there will say it's really hard. We haven't really been able to figure out exactly that. Science hasn't really congealed yet. We're still trying to figure out the basic elements of that science. Where does complexity come from and what is it and how you define it? Whether it's DNA creating bodies or phenotypes or it's individuals creating societies or ants and, you know, markets and so on. It's. It's a very complex thing. I'm not a complexity theorist person. Right. I. I think you ask, well, the brain itself is a complex system, so can we understand that? I think we've made a lot of progress understanding how the brain works. So. But I haven't broadened out to like, oh, well, where are we on the complexity spectrum? You know, it's like, it's a great question.

I prefer for that answer to be we're not special. It seems like if we're honest, most likely we're not special. So if there is a spectrum, we're probably not in some kind of significant place.

I think there's one thing we could say that we are special and again, only here on Earth, I'm not saying I'm bad, is that if we think about knowledge, what we know, we clearly, human brains are the only brains that have certain types of knowledge. We're the only brains on this Earth to understand what the Earth is, how old it is. The universe is a picture as a whole with the only organisms understand DNA and the origins of species. No other species on this planet has that knowledge. So if we think about. I like to think about one of the endeavors of humanity is to understand the universe as much as we Can I think our species is further along in that? Undeniably. Whether our theories are right or wrong, we can debate. But at least we have theories. You know, we know that what the sun is and how fusion is and how what black holes are, and, you know, we know general theory of relativity, and no other animal has any of this knowledge. So from that sense that we're special, are we special in terms of the hierarchy of complexity in the universe? Probably not.

Can we look at a neuron? You say that prediction happens in the neuron. What does that mean? So the neuron traditionally is seen as the basic element of the brain.

So I mentioned this earlier. That prediction was our research agenda. We said, okay, how does the brain make a prediction? I'm about to grab this water bottle, and my brain is predicting what I'm going to feel on all my parts of my fingers. If I felt something really odd on any part here, I'd notice it. So my brain is predicting what it's going to feel as I grab this thing. So how does that manifest itself in neural tissue? Brain's made of neurons, and there's chemicals and there's neurons and there's spikes and they're connecting. Where is the prediction going on? And one argument could be that, well, when I'm predicting something, a neuron must be firing in advance. It's like, okay, this neuron represents what you're going to feel, and it's firing, it's sending a spike. And certainly that happens to some extent, but our predictions are so ubiquitous that we're making so many of them, which we're totally unaware of. Just the vast majority of you have no idea that you're doing this, that there wasn't really. We were trying to figure out, how could this be? Where are these happening? Right? And I won't walk you through the whole story unless you insist upon it. But we came to the realization that most of your predictions are occurring inside individual neurons, especially the most common neuron, the pyramidal cells. And there's a property of neurons. Everyone knows, or most people know, that a neuron is a cell, and it has this spike called an action potential, and it sends information. But we now know that there's these spikes internal to the neuron. They're called dendritic spikes. They travel along the branches of the neuron and they don't leave the neuron. They're just internal. Only there's far more dendritic spikes than there are action potentials. Far more. They're happening all the time. And what we came to understand that those dendritic spikes, the ones that are occurring, are actually a form of prediction. They're telling the neuron, the neuron is saying, I expect that I might become active shortly. And that internal. So the internal spike is a way of saying you might be generating external spikes soon. I predicted you're going to become active. And we wrote a paper in 2016 which explained how this manifests itself in neural tissue and how it is that this all works together. But the vast majority, we think there's a lot of evidence supporting it. So that's where we think that most of these predictions are internal. That's why you can't be they're internal in a row. You can't perceive them.

From understanding the prediction mechanism of a single neuron, do you think there's deep insights to be gained about the prediction capabilities of the mini brains within the bigger brain and the brain?

Oh yeah, yeah, yeah. So having a prediction side of individual neuron is not that useful. So what the way it manifests itself in neural tissue is that when a neuron, a neuron emits these spikes, or a very singular type of event, if a neuron is predicting that it's going to be active, it emits its spike very, a little bit sooner, just a few milliseconds sooner than it would have otherwise. It's like, I give the analogy in the book, it's like a sprinter on a starting blocks in a race. And if someone says get ready, set, you get up and you're ready to go. And then when your race starts, you get a little bit earlier start. So that ready set is like the prediction and the neurons ready to go quicker. And what happens is when you have a whole bunch of neurons together and they're all getting these inputs, the ones that are in the predictive state, the ones that are anticipating to become active, if they do become active, they happen sooner. They disable everything else and it leads to different representations in the brain. So you have to, it's not isolated just to the neuron. The prediction occurs when the neuron but the network behavior changes. So what happens under different predictions? Different inputs have different representations. So what I predict is going to be different under different contexts. What my input will be is different under different contexts. So this is a key level theory how this works.

So the theory of the thousand brains, if you were to count the number of brains, how would you do it?

The thousand brain theory says that basically every cortical column in your neocortex is a complete modeling system. And that When I ask where do I have a model of something like a coffee cup? It's not in one of those models. It's in thousands of those models. There's thousands of models of coffee cups. That's what the thousand brains.

Then there's a voting mechanism.

Then there's a voting mechanism which leads the thing which you're conscious of, which leads to your singular perception. That's why you perceive something. So that's the thousand brains theory. The details of how we got to that theory are complicated. It wasn't. We just thought of it one day and one of those details is we had to ask, how does a model make predictions? And we've talked about just these predictive neurons that's part of this theory. It's like saying, oh, it's a detail, but it was like a crack in the door. It's like, how are we going to figure out how these neurons built do this? What is going on here? So we just looked at prediction as like, well, we know that's ubiquitous. We know that every part of the cortex is making predictions. Therefore whatever the predictive system is, it's going to be everywhere. We know there's a gazillion predictions happening at once. So let's see if we can start teasing apart, ask questions about how could neurons be making these predictions. And that sort of built up to now what we have this 1000 brains theory, which is complex, I can state it simply, but we just didn't think of it. We had to get there step by step. It took years to get there.

And where does reference frames fit in?

Okay, so again, a reference frame. I mentioned earlier about the model of a house. And I said, if you're going to build a model of a house in a computer, they have a reference frame. And you can think of reference frame like cartesian coordinates, like X, Y and Z axes. So I can say, oh, I'm going to design a house. I can say, well, the front door is at this location xyz and the, the roof is at this location XYZ and so on. That's the type of reference frame. So it turns out for you to make a prediction. And I walk you through the thought experiment in the book where I was predicting what my finger was going to feel when I touched the coffee cup. It was a ceramic coffee cup, but this one will do. And what I realized is that to make a prediction, what my finger is going to feel like, it's going to feel different than this, which would feel different if I touch the hole or this thing on the bottom make that prediction. The cortex needs to know where the finger is, the tip of the finger relative to the coffee cup and exactly relative to the coffee cup. And to do that, I have to have a reference frame for the coffee cup. It has to have a way of representing the location of my finger to the coffee cup. Then we realize, of course, every part of your skin has to have a reference frame relative to things as such. And then we did the same thing with vision. So the idea that a reference frame is necessary to make a prediction when you're touching something or when you're seeing something and you're moving your eyes or you're moving your fingers, it's just a requirement to know what to predict. If I have a structure, I'm going to make a prediction. I have to know where it is, I'm looking or touching it. So then we said, well, how do neurons make reference frames? It's not obvious. XYZ coordinates don't exist in the brain. It's just not the way it works. So that's when we looked at the older part of the brain, the hippocampus and the entorhinal cortex, where we knew that in that part of the brain there's a reference frame for a room or a reference frame for environment. Remember I talked earlier about how you could make a map of this room? So we said, oh, they are implementing reference frames there. So we knew that reference frames needed to exist in every cortical column. And so that was a deductive thing. We just deduced it has to exist.

So you take the old mammalian ability to know where you are in a particular space, and you start applying that to higher and higher levels.

Yeah. First you apply it to physical, like where your finger is. So here's the way I think about it. The old part of the brain says, where's my body in this room? Yeah. The new part of the brain says, where's my finger relative to this object?

Yeah.

Where is a section of my retina relative to this object? I'm looking at one little cornea. Where is that relative to this patch of my retina? And then we take the same thing and apply it to concepts, mathematics, physics, humanity, whatever you want to think.

Eventually you're pondering your own mortality.

Well, whatever. But the point is, when we think about the world, when we have knowledge about the world, how is that knowledge organized? Lex? Where is it in your head? The answer is, it's in reference frames. So the way I learned the structure of this water bottle, where the features are relative to each other. When I think about history or democracy or mathematics, the same basic underlying structures happening. There's reference frames for where the knowledge that you're assigning things to. So in the book I go through examples like mathematics and language and politics. But the evidence is very clear in the neuroscience. The same mechanism that we use to model this coffee cup, we're going to use to model high level thoughts, your demise of humanity, whatever you want to think about.

It's interesting to think about how different are the representations of those higher dimensional concepts, higher level concepts. How different the representation there is in terms of reference frames versus spatial.

But interesting thing, it's a different application, but it's the exact same mechanism.

But isn't there some aspect to higher level concepts that they seem to be hierarchical? Like they just seem to integrate a lot of information into them.

So is our physical objects. So take this water bottle. I'm not particular to this brand, but this is a Fiji water bottle and it has a logo on it. I use this example in my book. Our company's coffee cup has a logo on it. But this object is hierarchical. It is, it's got like a cylinder and a cap, but then has this logo on it. And the logo has a word, the word has letters, the letters have different features. And so I don't have to remember, I don't have to think about this. I say, oh, there's a Fiji logo on this water bottle. I don't have to go through and say, oh, what is the Fiji logo? It's the F and I and a J and I and there's a hibiscus flower. And oh, it has the pest, you know, the stamen on it. I don't have to do that. I just incorporate all of that in some sort of hierarchical representation. I say, you know, put this logo on this water bottle. Yeah. And, and, and then the logo has a word and the word has letters, all hierarchical.

All that stuff is big. It's amazing that the brain instantly just does all that.

Yeah.

The idea that there's, there's water, it's liquid. And the idea that you can drink it when you're thirsty. The idea that there's brands.

Yeah.

And then there's like all of that information is instantly like built into the whole thing when you, once you proceed.

So I wanted to get back to your point about hierarchical representation. The world itself is hierarchical. Right. And I can take this microphone in front of me. I know inside there's going to be some electronics. I know there's going to be some wires and I know there's going to be a little diaphragm that moves back and forth. I don't see that, but I know it. So everything in the world is hierarchical. You just go into a room, it's composed of other components of. Kitchen has a refrigerator, the refrigerator has a door, the door has a hinge, the hinge has screws and pin. So anyway, the modeling system that exists in every cortical column learns the hierarchical structure of objects. So it's a very sophisticated modeling system. In this grain of rice, it's hard to imagine, but this grain of rice can do really sophisticated things. It's got 100,000 neurons in it. It's very sophisticated. So the same mechanism that can model a water bottle or a coffee cup can model conceptual objects as well. It's that's the beauty of this discovery that this guy Vernon Mountcastle made many, many years ago, which is that there's a single cortical algorithm underlying everything we're doing.

So common sense concepts and higher level concepts are all represented in the same way.

They're set in the same mechanisms. Yeah, it's a little bit like computers. All computers are universal Turing machines, Even the little teeny one that my toaster and the big one that's running some cloud server someplace, they're all running on the same principle. They can apply different things. So the brain is all built on the same principle. It's all about learning these models, structured models using movement and reference frames. And it can be applied to something as simple as a water bottle and a coffee cup. And it can be applied to thinking like what's the future of humanity? And why do you have a hedgehog on your desk? I don't know.

Nobody knows.

Well, I think it's hedgehog.

That's right. It's a hedgehog in the fog. It's a Russian reference. Does it give you any inclination or hope about how difficult it is to engineer common sense reasoning? So how complicated is this whole process? So looking at the brain, is this a marvel of engineering or is it pretty dumb stuff stack on top of each other over.

Can it be both? Can it be both? Right.

I don't know if it can be both. Because if it's an incredible engineering job, that means it's. So evolution did a lot of work.

Yeah, but then it just copied that. Right. So as I said earlier, figuring out how to model something like a space is really hard. And evolution had to go through a lot of trick. And these cells I was talking about these bridge cells and place cells. They're really complicated. This is not simple stuff. This neural tissue works on these really unexpected, weird mechanisms. But it did it. It figured it out. But now you could just make lots of copies of it.

But then finding. Yeah, so it's a very interesting idea that it's a lot of copies of a basic mini brain. But the question is how difficult it is to find that mini brain that you can copy and paste effectively.

Today, we know enough to build this. I'm sitting here with. I know the steps. We have to go. There's still some engineering problems to solve, but we know enough. This is not like, oh, this is an interesting idea. We have to go think about it for another few decades. No, we actually understand it pretty well. Details. So not all the details, but most of them. So it's complicated, but it is an engineering problem. So in my company, we are working on that. We are basically laid out a roadmap. How we do this. It's not going to take decades. It's a matter of a few years, optimistically, but I think that's possible. It's complex things. If you understand them, you can build them.

So in which domain do you think it's best, best to build them? Are we talking about robotics, like entities that operate in the physical world that are able to interact with that world? Are we talking about entities that operate in the digital world? Are we talking about something more like. More specific, like is done in the machine learning community, where you look at natural language or computer vision. Where do you think is easiest to.

It's the first two more than the third one, I would say. Again, let's just use computers as an analogy. The pioneers in computing, people like John van Neumann, Alan Turing, they created this thing we now call the universal Turing machine, which is a computer. Right. Did they know how it was going to be applied, where it was going to be used? Could they envision any of the future? No, they just said this is like a really interesting computational idea about algorithms and how you can implement them in a machine. And we're doing something similar to that today. We are building this sort of universal learning principle that can be applied to many, many different things.

But the robotics piece of that, the interactive.

All right, let's be specific. You can think of this cortical column as what we call a sensorimotor learning system. It has the idea that there's a sensor and then it's moving. That sensor can be physical. It could be like my finger and it's moving in the World, it could be like my eye and it's physically moving. It can also be virtual. So it could be. An example would be I could have a system that lives in the Internet, that actually samples information on the Internet and moves by following links. That's a sensory motor system.

So something that echoes the process of a finger moving along in a very, very loose sense.

It's like again, learning is inherently about the subbing the structure in the world. And to discover the structure in the world, you have to move through the world. Even if it's a virtual world, even if it's a conceptual world, you have to move through doesn't exist in one. It has some structure to it. So here's a couple of predictions of getting what you're talking about. In humans, the same algorithm does robotics, right? It moves my arms, my eyes, my body, right? And so in the future, to me, robotics and AI will merge. They're not going to be separate fields because the algorithms for really controlling robots are going to be the same algorithms we have in our brain, these sensory motor algorithms. Today we're not there, but I think that's going to happen. But not all AI systems will be robotics. You can have systems that have very different types of embodiments. Some will have physical movements, some will not have physical movements, moments. It's a very generic learning system. Again, it's like computers. The Turing machine doesn't say how it's supposed to be implemented, doesn't tell you how big it is, doesn't tell you what you can apply it to, but it's a computational principle. Cortical column equivalent is a computational principle. It's about learning, it's about how you learn. And it can be applied to a gazillion things. This is why I think this impact of AI is going to be as large, if not larger than computing has been in the last century by far, because it's getting at a fundamental thing. It's not a vision system or a learning system. It's not a vision system or hearing system. It is a learning system. It's a fundamental principle. How you learn to structure in the world, how you gain knowledge and be intelligent. And that's what the Thousand Brain says was going on. And we have a particular implementation in our head. But it doesn't have to be like that at all.

Do you think there's going to be some kind of impact? Okay, let me ask it another way. What do increasingly intelligent AI systems do with us humans in the following way? Like, how hard is the human in the loop problem. How hard is it to interact? The finger on the coffee cup equivalent of having a conversation with a human being. So how hard is it to fit into our little human world?

I think it's a lot of engineering problems. I don't think it's a fundamental problem. I could ask you the same question. How hard is it for computers to fit into a human world?

Right. I mean, that's essentially what I'm asking. Like, how much are we elitist? Are we as humans, we tried to keep out systems?

I don't know. I'm not sure. I'm not sure that's the right question. Let's look at computers as an analogy. Computers are a million times faster than us. They do things we can't understand. Most people have no idea what's going on when they use computers. How are we integrated in our society? Well, we don't think of them as their own entities. They're not living things. We don't afford them rights. We rely on them. Our survival as 7 billion people or something like that is relying on computers.

Now, don't you think that's a fundamental problem? That we see them as something we can't? We don't give, Right?

Someone says, oh, this thing needs to write. I shouldn't be able to turn it off. Or if I throw it in the trash can and hit it with a sledgehammer, I might form a criminal act. No, no one thinks that. And now we think about intelligent machines, which is where you're going, and all of a sudden like, well, now we can't do that. I think the basic problem we have here is that people think intelligent machines will be like us. They're going to have the same emotions as we do, the same feelings as we do. What if I can build an intelligent machine that absolutely could care less about whether it was on or off or destroyed or not? It just doesn't care. It's just like a map. It's just a modeling system, has no desires to live, nothing.

Is it possible to create a system that can model the world deeply and not care about whether it lives or dies?

Absolutely no question about it to me.

That's not 100% obvious.

It's obvious to me. So we can debate, if you want, where does your desire to live come from? It's an old evolutionary.