Charan Ranganath: Human Memory, Imagination, Deja Vu, and False Memories | Lex Fridman Podcast #430

The act of remembering can change the memory. If you remember some event and then I tell you something about the event, later on when you remember the event, you might remember some original information from the event as well as some information about what I told you. And sometimes if you're not able to tell the difference, that information that I told you gets mixed into the story that you had originally. So now I give you some more misinformation or you're exposed to some more information somewhere else and eventually your memory becomes totally detached from what happened.

The following is a conversation with Charon Ranganath, a psychologist and neuroscientist at UC Davis, specializing in human memory. He's the author of why We Unlocking Memory's Power to Hold on to what Matters. This is the Lex Friedman podcast. To support it, please check out our sponsors in the description and now, dear friends, here's Charon Ranganath. Danny Kahneman describes the experiencing self and the remembering self, and that happiness and satisfaction you gain from the outcomes of your decisions do not come from what you've experienced, but rather from what you remember of the experience. So can you speak to this interesting difference that you write about in your book of the experiencing self and the remembering self?

Danny really impacted me because I was an undergrad at Berkeley and I got to take a class from him long before he won the Nobel Prize or anything. And it was just a mind blowing class. But this idea of the remembering self and the experiencing self, I got into it because it's so much about memory, even though he doesn't study memory. So we're right now having this experience, right? And people are can watch it presumably on YouTube or listen to it on audio, but if you're talking to somebody else, you could probably describe this whole thing in 10 minutes, but that's going to miss a lot of what actually happened. And so the idea there is that the way we remember things is not the replay of the experience, it's something totally different. And it tends to be biased by the beginning and the end. And he talks about the peaks. There's also the best parts, the worst parts, et cetera. And those are the things that we remember. And so when we make decisions, we usually consult memory and we feel like our memory is a record of what we've experienced, but it's not. It's this kind of very biased sample, but it's biased in an interesting and I think biologically relevant way.

So in the way we construct a narrative about our past, you say that it gives us an illusion of stability. Can you explain that?

Basically, I think that a lot of learning in the brain is driven towards being able to make sense. I mean, really, memory is all about the present and the future. The past is done. So biologically speaking, it's not important unless there's something from the past that's useful. And so what our brains are really optimized for is to learn about the stuff from the past that's going to be most useful in understanding the presentation and predicting the future. Right. And so cause, effect, relationships, for instance. That's a big one. Now my future is completely unpredictable in the sense that, like you could, you know, in the next 10 minutes, pull a knife on me and slit my throat. Right.

I was planning on it.

Exactly. But having seeds of your work and just, you know, generally my expectations about life, I'm not expecting that. I have a certainty that everything's going to be fine. We're going to have a great time talking today. Right? But we're often, right. It's like, okay, so I go to see a band on stage, I know they're going to make me wait. The show's going to start late, and then they come on. There's a very good chance there's going to be an encore. I have a memory, so to speak, for that event before I've even walked into the show. There's going to be people holding up their camera phones to try to take videos of it now because this is kind of the world we live in. So that's like everyday fortune telling that we do, though it's not real, it's imagined. And it's amazing that we have this capability and that's what memory is about. But it can also give us this illusion that we know everything that's about to happen. And I think what's valuable about that illusion is when it's broken, it gives us the information. Right. So, I mean, I'm sure being an AI, you know about information theory and the idea is the information is what you didn't already have. And so those prediction errors that we make based on, you know, we make a prediction based on memory. And the errors are where the action is.

The error is where the learning happens.

Exactly. Exactly.

Well, just to linger on, Danny Kahneman and just this whole idea of experiencing self versus remembering self. I was hoping you can give a simple answer of how we should live life based on the fact that our memories could be a source of happiness or could be the primary source of happiness. That an event, when experienced, bears its fruits the most. When it's remembered over and over and over and over. And maybe there is some wisdom in the fact that we can control to some degree how we remember it, how we evolve our memory of it, such that it can maximize the long term happiness of that repeated experience.

Okay, well, first I'll say, I wish I could take you on the road with me because that was such a great description.

Can I be your opening act?

Oh, my God, no. I'm going to open for you, dude. Otherwise it's like everybody leaves after you're done. Believe me, I did that in Columbus, Ohio once. It wasn't for fun, like the opening acts, like, drank our bar tab. We spent all this money going all the way there. There was only the everybody left after the opening acts were done and there was just that stoner dude with the dreadlocks hanging out. And then next thing you know, we blew like our savings on getting a hotel room.

So we should. As a small tangent, you're a legit touring act.

When I was in grad school, I played in a band. And yeah, we traveled, we would play shows. It wasn't like we were in a hardcore touring band, but we did some touring and had some fun. And yeah, we did a movie soundtrack.

Nice.

Henry Portrait of Serial Killer. So that's a good movie. We were on the soundtrack for the sequel, Henry 2 Mask of Sanity, which is a terrible movie.

How's the soundtrack?

It's pretty good. It's badass. At least that one part where the guy throws up the milkshake.

We're going to have to see. We're going to have to see it.

All right. We're getting back to life, advice and happiness.

Yeah.

One thing that I try to live by, especially nowadays, and since I wrote the book, I've been thinking more and more about this is how do I want to live a memorable life? You know, I think if we go back to like the pandemic, right. How many people have memories from that period aside from the trauma of being, you know, locked up and seeing people die and all this stuff? I think it's like one of these things where we were stuck inside looking at screens all day, doing the same thing with the same people. And so I don't remember much from that in terms of those good memories that you're talking about. Right. When I was growing up, my parents worked really hard for us and we went on some vacations, but not very often. And I really try to do now vacations to interesting places as much as possible with my family because those are the things that you remember. So I really do think about what's going to be like, something that's memorable and then just do it, even if it's a pain in the ass, because the experiencing self will suffer for that, but the remembering self will be like, yes, I'm so glad I did that.

Do things that are very unpleasant in the moment because those can be reframed and enjoyed for many years to come. That's probably good advice, or at least when you're going through shit, it's a good way to see the silver lining of it.

Yeah, I mean, I think it's one of these things where if you have like, people who you've gone through, I mean, since you said it, I'll just say since you've gone through shit with someone and it's like, that's a bonding experience often, you know, I mean, that can really bring you together. I like to say it's like there's no point in suffering unless you get a story out of it. So in the book I talk about the power of the way we communicate with others and how that shapes our memories. And so I had this near death experience, at least that's how I remember it on this paddle board where just everything that could have gone wrong did go wrong. Almost so many mistakes were made and ended up like, at some point, just like, basically away from my board, pinned in a current, like in this corner. Like, not a super good swimmer. And my friend who came with me, Randy, who's a computational neuroscientist, and he had just been pushed down past me so he couldn't even see me. And I'm just like, if I die here, you know, I mean, no one's around. It's. It's like you just die alone. And so I just said, well, failure is not an option. And eventually I got out of it and froze and got cut up. And I mean, the things that we were going through were just insane. But short version of this is, you know, my. My wife and my daughter and Randy's wife, they gave us all sorts of hell about this because they were just like where they were ready to send out a search party. So they were giving me hell about it. And then I started to tell people in my lab about this and then friends, and it just became a better and better story every time. And we actually had some photos of just the crazy things like this generator that was hanging over the water, and we're like ducking under this, these metal gratings and like going flat on and it was just nuts, you know, but it became a great story. And it was definitely, I mean, Randy and I were already tight, but that was a real bonding experience for us. And yeah, I mean, and I learned from that that it's like, I don't look back on that enough actually, because I think we often, at least for me, I don't necessarily have the confidence to think that things will work out, that I'll be able to get through certain things thing. But my ability to actually get something done in that moment is better than I give myself credit for, I think. And that was the lesson of that story that I really took away.

Well, actually, just for me, you're making me realize now that it's not just those kinds of stories, but even things like periods of depression or really low points, to me at least it feels like motivating thing that the darker it gets, the better the story will be. If you emerge on the other side, that to me feels like a motivating thing. So maybe if people listening to this and they're going through some shit, as we said, one thing that could be a source of light is that it'll be a hell of a good story when it's all over. When you emerge on the other side, let me ask you about decisions. You've already talked about it a little bit. But when we face the world and we're making different decisions, how much does our memory come into play? Is it the kind of narratives that we've constructed about the world that are used to make predictions that's fundamentally part of the decision making?

Absolutely, yeah. So let's say after this, you and I decided we're going to go for a beer, right. How do you choose where to go? You're probably going to be like, oh, yeah, this new bar opened up near me, had a great time there. They had a great beer selection. Or you might say, oh, we went to this place and it was totally crowded and they're playing this horrible EDM or whatever. And so right there, valuable source of information, right. And then you have these things like where you do this counterfactual stuff, like, well, I did this previously, but what if I had gone somewhere else and said, maybe I'll go to this other place because I didn't try it the previous time. So there's all that kind of reasoning that goes into it too. I think. Even if you think about the big decisions in life. Right. It's like you and I were talking before we started recording about how I got into memory research. And you got into AI and it's like we all have these personal reasons that guide us in these particular directions, and some of it's the environment and random factors in life. And some of it is memories of things that we want to overcome or think things that we build on in a positive way. But either way, they define us.

And probably the earlier in life the memories happen, the more defining power they have in terms of determining who you become.

I mean, I do feel like adolescence is much more important than I think people give credit for. I think that there is this kind of a sense, like the first three years of life is the most important part, but the teenage years are just so important for the brain, you know, and so that's where a lot of mental illness starts to emerge. You know, now we're thinking of things like schizophrenia as a neurodevelopmental disorder because it just emerges during that period of adolescence and early adulthood. So. And I think the other part of it is, is that, you know, as. I guess I was a little bit too firm in saying that memory determines who we are. It's really the self is an evolving construct. I think we kind of underestimate that. And when you're a parent, you feel like every decision you make is consequential in forming this child and plays a role, but so do the child's peers, and so do. There's so much. I mean, that's why I think the big part of education, I think, that's so important, is not the content you learn. I mean, think of how much dumb stuff we learned in school. Right? But a lot of it is learning how to get along with people and learning who you are and how you function. And that can be terribly traumatizing, even if you have perfect parents working on you.

Is there some insight into the human brain that explains why we don't seem to remember anything from the first few years of life?

Yeah. Yeah. In fact, actually, I was just talking to my really good friend and colleague Simona Getty, who studies the neuroscience of child development. And so we were talking about this. And so there are a bunch of reasons, I would say, so. One reason is there's an area of the brain called the hippocampus, which is very, very important for remembering events or episodic memory. And so the first two years of life, there's a period called infantile amnesia, and then the next couple of years of life after that, there's a period called childhood amnesia. And the difference is that basically in the lab and even during childhood and afterwards Children basically don't have any episodic memories for those first two years, the next two years. It's very fragmentary. And that's why they call it childhood amnesia. So there's some, but it's not mine. So one reason is that the hippocampus is taking some time to develop, but another is the neocortex. So the whole folded stuff of gray matter all around the hippocampus is developing so rapidly and changing and a child's knowledge of the world is just massively being built up. Right. So I mean, I'm going to probably embarrass myself, but it's like if you showed like, you know, you trained a neural network and you give it like the first couple of patterns or something like that, that, and then you bombard it with another year's worth of data, try to get back those first couple of patterns, right? It's like everything changes. And so the brain is so plastic, the cortex is so plastic during that time. And we think that memories for events are very distributed across the brain. So imagine you're trying to get back that pattern of activity that happened during this one moment, but the roads that you would take to get there have been completely rerouted. Right. So I think that's my best explanation. The third explanation is a child's sense of self takes a while to develop. And so their experience of learning might be more learning what happened as opposed to having this first person experience of I remember, I was there.

Well, I think somebody once said to me that kind of loosely, philosophically, that the reason we don't remember the first few years of life, infantile amnesia, is because how traumatic it is. Basically the error rate that you mentioned when your brain's prediction doesn't match reality, the error rate in the first few years of life, your first few months, certainly is probably crazy high. It's just non stop freaking out. The collision between your model of the world and how the world works is just so high that you want whatever the trauma of that is not to linger around. I always thought that's an interesting idea because just imagine the insanity of what's happening in a human brain in the first couple years. You don't know anything and there's just this stream of knowledge and where somehow given how plastic everything is, it just kind of molds and figures it out, but it's, it's like an insane waterfall of information.

I wouldn't necessarily describe it as a trauma. And we can get into this whole stages of life thing, which I just love, basically. Those first few years, there are. I mean, you know, I mean, think about it. A kid's internal model of their body is changing, right? It's like just learning to move. I mean, like, you know, if you ever have a baby, you'll know that, like, the first three months, they're discovering their toes, right? It's just not. So everything is changing. But what's really fascinating is, and I think this is one of those. This is not at all me being a scientist, but it's like one of those things that people talk about when they talk about the positive aspects of children is that they're exceptionally curious and they have this kind of openness towards the world. And so that prediction error is not a negative, traumatic thing. I think it's like a very positive thing because it's what they use. They're seeking information. One of the areas that I'm very interested in is the prefrontal cortex. It's an area of the brain that, I mean, I could talk all day about it, but it helps us use our knowledge to say, hey, this is what I want to do now. This is my goal, so this is how I'm going to achieve it and focus everything towards that goal. Right? The prefrontal cortex takes forever to develop in humans. The connections are still being tweaked and reformed into late adolescence, early adulthood, which is when you tend to see mental illness pop up. So it's being massively reformed. Then you have about 10 years maybe of prime functioning of the prefrontal cortex, and then it starts going down again, and you end up being older, and you start losing all that frontal function. So look at this. And you'd say, okay, from you sit around, episodic memory talks. We'll always say children are worse than adults at episodic memory. Older adults are worse than young adults at episodic memory. And I always say, would say, God, this is so weird. Why would we have this period of time that's so short when we're perfect, right? Or optimal. And I like to use that word, optimal now because there's such a culture of optimization right now. And it's like I realized I have to redefine what optimal is, because for most of the human condition, I think we had a series of stages of life where you have basically adults saying, okay, young adults saying, I've got a child and I'm part of this village, and I have to hunt and forage and get things done. I need a prefrontal cortex so I can stay focused on the big picture. And long haul goals. Now I'm a child, I'm in this village, I'm kind of wandering around and I've got some safety and I need to learn about this culture because I know so little. What's the best way to do that? Let's explore. I don't want to be constrained by goals as much. I want to really be free play and explore and learn. So you don't want a super tight prefrontal cortex. You don't even know what the goals should be yet. It's like if you're trying to design a model that's based on a bad goal, it's not going to work well. So then you go late in life and you say, oh, why don't you have a great prefrontal cortex then? But I think, I mean, if you go back and you think how many species actually stick around naturally long after their childbearing years are over, after their reproductive years are over, like menopause, from what I understand, menopause is not all that common in the animal world. Right. So why would that happen? And so I saw Alison Gopnik said something about this. So I started to look into this, about this idea that really when you're older in most societies, your job is no longer to form new episodic memories, it's to pass on the memories that you already have, this knowledge about the world, or what we call semantic memory, to pass on that semantic memory to the younger generations, pass on the culture. Even now in indigenous cultures, that's the role of the elders. They're respected. They're not seen as people who are past it and losing it. And I thought that was a very poignant thing, that memory is doing what it's supposed to throughout these stages of life.

So it is always optimal in a sense, it's just optimal for that stage of life.

Yeah. And for the ecology of the system. So you've got. So I looked into this and it's like another species that has menopause is orcas. Orca pods are led by the grandmothers, Right. So not the young adults, not the parents or whatever, the grandmothers. And so they're the ones that pass on the traditions to the, I guess, the younger generation of orcas. And if you look, from what little I understand, different orca pods have different traditions. They hunt for different things, they have different play traditions. And that's a culture. Right. And so in social animals, evolution, I think, is designing brains that are really around. You know, it's obviously optimized for the individual but also for kin. And I think that the kin are part of this, like when they're a part of this intense social group. The brain development should parallel the nature of the ecology.

Well, it's just fascinating to think of the individual orca or human, throughout its life in stages, doing a kind of optimal wisdom development. So in the early days, you don't even know what the goal is. And you figure out the goal and you kind of optimize for that goal and you pursue that goal. And then all the wisdom you collect through that, then you share with the others in the system, with the other individuals. And as a, as a collective, then you kind of converge towards greater wisdom throughout the generation. So in that sense it's optimal. Us humans and orcas got something going on. It works.

Apex predators.

I just got a megalan tooth. Speaking of apex predators, it's just imagine the size of that thing. Anyway, how does the brain forget and how and why does it remember? So maybe some of the mechanisms. You mentioned, the hippocampus, what are the different components involved here?

So we can think about this on a number of levels. Maybe I'll give you the simplest version first, which is we tend to think of memories as these individual things and we can just access them, maybe a little bit like photos on your phone or something like that. But in the brain, the way it works is you have this distributed pool of neurons and the memories are kind of shared across different pools of neurons. And so what you have is competition where sometimes memories that overlap can be fighting against each other. Right. So sometimes we forget because that competition just wipes things out. Sometimes we forget because there aren't the biological signals which we can get into that would promote long term retention. And lots of times we forget because we can't find the cue that sends us back to the right memory and we need the right cue to be able to activate it. Right. So you know, for instance, in a neural network there is no, you wouldn't go, and you'd say this is the memory, right? It's like the whole network. I mean the whole ecosystem of memories is in the weights of the neural network. And in fact, you could extract entirely new memories depending on how you feed.

You have to have the right query, the right prompt to access that, whatever the part you're looking for.

That's exactly right. That's exactly right. And in humans you have this more complex set of ways memory works. There's, as I said, the knowledge, or what you call semantic memory. And then there's these memories for Specific events, which we call episodic memory. And so there's different pieces of the puzzle that require different kinds of cues. So that's a big part of it too, is just this kind of what we call retrieval failure.

You mentioned episodic memory. You mentioned semantic memory. What are the different separations here? What's working memory, short term memory, long term memory are the interesting categories of memory.

Yeah. And so memory researchers, we love to cut things up and say, is memory one thing or is it two things or two things, or is it three things? And so one of the things that there's value in that, and especially experimental value in terms of being able to dissect things in the real world. It's all connected. Speak to your question. Working memory is a term that was coined by Alan Baddeley. It's basically thought to be this ability to keep information online in your mind, right in front of you at a given time, and to be able to control the flow of that information, to choose what information is relevant, to be able to manipulate it, and so forth. And one of the things that Alan did that was quite brilliant was he said there's this ability to kind of passively store information, you know, see things in your mind's eye or hear your internal monologue. But know we have that ability to keep information in mind. But then we also have this separate, what he called a central executive, which is identified a lot with the prefrontal cortex. It's this ability to control the flow of information that's being kept active based on what it is you're doing now. A lot of my early work was basically saying that this working memory, which some memory researchers would call short term memory, is not at all independent from long term memory. That is that a lot of executive function requires learning, and you have to have like synaptic change for that to happen. But there's also transient forms of memory. So one of the things I've been getting into lately is the idea that we form internal models of events. The obvious one that I always use is birthday parties. Right? So you go to a child's birthday party, once the cake comes out and you just see a candle, you can predict the whole frame set of events that happens later. And up till that point where the child blows out the candle, you have an internal model in your head of what's going on. And so if you follow people's eyes, it's not actually on what's happening, it's going where the action's about to happen, which is just fascinating. Right so you have this internal model, and that's a kind of a working memory product. It's something that you're keeping online that's allowing you to interpret this world around you. Now to build that model, though, you need to pull out stuff from your general knowledge of the world, which is what we call semantic memory. And then you'd want to be able to pull out memories for specific events that happened in the past, which we call episodic memory. So in a way they're all connected, even though it's different. The things that we're focusing on and the way we organize information in the present, which is working memory, will play a big role in determining how we remember that information later, which people typically call long term memory.

So if you have something like a birthday party and you've been to many before, you're going to load that from disk into working memory, this model, and then you're mostly operating on the model. And if it's a new task, you don't have a model. So you're more in the data collection.

Yes. One of the fascinating things that we've been studying, and we're not at all the first to do this, Jeff Sachs was a big pioneer in this and I've been working with many other people. Ken Norman, Leila Devachi at Columbia has done some interesting stuff with this, is this idea that we form these internal models at particular points of high prediction error or points of, I believe, also points of uncertainty, points of surprise, or motivationally significant periods. And those points are when it's maximally optimal to encode an episodic memory. So I used to think, oh, well, we're just encoding episodic memories constantly, boom, boom, boom, boom, boom. But think about how much redundancy there is in all that. Right? It's just a lot of information that you don't need. But if you capture an episodic memory at the point of maximum uncertainty for the singular experience, right, it's only going to happen once. But if you capture it at the point of maximum uncertainty or maximum surprise, you have the most useful point in your experience that you've grabbed. And what we see is that the hippocampus and these other networks that are involved in generating these internal models of events, they show a heightened period of connectivity or correlated activity during those breaks between different events, which we call event boundaries. These are the points where you're surprised or you cross from one room to another and so forth. And that communication is associated with a bump of activity in the hippocampus. And better memory. And so if people have a very good internal model throughout that event, you don't need to do much memory processing. You're in a predictive mode, right? And so then at these event boundaries, you encode and then you retrieve and you're like, okay, wait a minute, what's going on here? Ranganath's now talking about orcas. What's going on? And maybe you have to go back and remember reading my book to pull out the episodic memory to make sense of whatever it is I'm babbling about, right? And so there's this beautiful dynamics that you can see in the brain of these different networks that are coming together and then de affiliating at different points, points in time that are allowing you to go into these modes. And so to speak, to your original question, to some extent, when we're talking about semantic memory and episodic memory and working memory, you can think about it as these processes that are unfolding as these networks kind of come together and pull apart.

Can memory be trained and improved? This beautiful connected system that you've described, what aspect of it is a mechanism that can be improved through training?

I think improvement, it depends on what your definition of optimal is. So what I say in the book is that you don't want to remember more, you want to remember better, which means focusing on the things that are important. And that's what our brains are designed to do. So if you go back to the earliest quantitative studies of memory by Ebbinghaus, what you see is that he was trying so hard to memorize this arbitrary nonsense, and within a day he lost about 60% of that information. And he was basically using a very, very generous way of measuring it, right? So as far as we know, nobody has managed to violate those basics of having people forget most of their experiences. So if your expectation is that you should remember everything and that's what your optimal is, is you're already off. Because this is not what human brains are designed to do. On the other hand, what we see over and over again is that the brain does basically. One of the cool things about the design of the brain is it's always less is more, less is more, right? It's like, I mean, I've seen estimates that the human brain uses something like 12 to 20 watts, you know, in a day. I mean, that's just nuts. The low power consumption, right? So it's all about reusing information and making the most of what we already have. And so that's why basically, again, what you See, biologically is, you know, neuromodulators. For instance, these chemicals in the brain like norepinephrine, dopamine, serotonin, these are chemicals that are released during moments that tend to be biologically significant. Surprise, fear, stress, et cetera. And so these chemicals promote lasting plasticity. Right. Essentially, some mechanisms by which the brain can, say, prioritize the information that you carry with you into the future. Attention is a big factor as well. Our ability to focus our attention on what's important. And so there's different schools of thought on training attention, for instance. So one of my colleagues, Amishi Ja, she wrote a book called Peak Mind and talks about mindfulness as a method for improving attention and focus. So she works a lot with military, like Navy SEALs and stuff to do this kind of work with mindfulness meditation. Adam Ghazali, another one of my friends and colleagues, has work on kind of training through video games, actually, as a way of training attention. And so it's not clear to me, one of the challenges, though, in training is you tend to overfit to the thing that you're trying to optimize, right? So you tend to. If I'm looking at a video game, I can definitely get better at paying attention in the context of the video game. But you transfer it to the outside world. That's very controversial.

The implication there is that attention is a fundamental component of remembering something, allocating attention to it. And then attention might be something that you could train, how you allocate attention and how you hold attention on a thing.

I can say that, in fact, we do in certain ways. Right? So if you are an expert in something, you are training attention. So we did this one study of expertise in the brain. And so people used to think, let's say if you're a bird expert or something, people will go, if you get really into this world of birds, you start to see the differences in your visual cortex is tuned up, and it's all about plasticity of the visual cortex and vision. Researchers love to say everything's visual, but it's like we did the study of attention and working or working memory and expertise. And one of the things that surprised us were the biggest effects as people became experts in identifying these different kinds of just crazy objects that we made up. As they developed this expertise of being able to identify what made them different from each other and what made them unique, we were actually seeing massive increases in activity in the prefrontal cortex. And this fits with some of the studies of chess experts and so forth. That it's not so much that you learn the patterns passively. You learn what to look for. You learn what's important, what's not. Right. And you can see this in any kind of expert professional athlete. They're looking three steps ahead of where they're supposed to be. So that's a kind of a training of attention. And those are also what you'd call expert memory skills. So if you take the memory athletes, I know that's something we're both interested in. And so these are people who train in these competitions, and they'll memorize a deck of cards and a really short amount of time. There's a great memory athlete. Her name, I think, is pronounced Yenya Wintersole. So I think she's got a giant Instagram following. And so she had this YouTube video that went viral where she had memorized an entire IKEA catalog. And so how do people do this? By all accounts, from people who become memory athletes, they weren't born with some extraordinary memory, but they practice strategies over and over and over again. And the strategy that they use for memorizing a particular thing, it can become automatic, and you can just deploy it in an instant. Right. So again, it's not necessarily going to. One strategy for learning the order of a deck of cards might not help you for something else that you need, like, you know, remembering your way around Austin, Texas, but it's going to be these. Whatever you're interested in, you can optimize for that. And that's just a natural byproduct of expertise.

There's certain hacks. There's something called the memory palace that I played with. I don't know if you're familiar with that whole technique, and it works. It's interesting. So another thing I recommend for people a lot is I use Anki a lot every day. It's a app that does spaced repetition. So I think medical students and students use this a lot to remember a lot of different things.

Oh, yeah, okay. We can come back to this. But, yeah, sure.

It's the whole concept of space repetition. You just. When the thing is fresh, you kind of have to remind yourself of it a lot. And then over time, you can wait a week, a month, a year before you have to recall the thing again. And that way, you essentially have something like note cards that you can have tens of thousands of and can only spend 30 minutes a day and actually be refreshing all of that information, all that knowledge. It's really great. And then for Memory palace is a technique that allows you to remember Things like the IKEA catalog or by placing them visually in a place that you're really familiar with. Like, I'm really familiar with this place. So I can put numbers or facts or whatever you want to remember. You can walk along that little palace and it reminds you. It's cool. Like, there's stuff like that that I think athletes memory athletes could use, but I think also regular people can use. One of the things I have to solve for myself is how to remember name names. I'm horrible at it.

Yeah.

I think it's because when people introduce themselves, I have the social anxiety of the interaction where I'm like, I know I should be remembering that, but I'm freaking out internally about social interaction in general. And so therefore, I forget immediately. So I'm looking for good tricks for that.

So I'm. I feel like we've got a lot in common, because when people introduce themselves to me, it's almost like I have this, like, just blank blackout for a moment, and then I'm just looking at them like, what happened? I look away or something. What's wrong with me?

Yeah.

So, I mean, I'm totally with you on this. The reason why it's hard is that there's no reason we should be able to remember names. Because when you say remembering a name, you're not really remembering a name. Maybe in my case, you are, but most of the time you're associating a name with. With a face and an identity. And that's a completely arbitrary thing, right? I mean, maybe in the olden days, somebody named Miller, it's like they're actually making flour or something like that. But, you know, for the most part, it's like these names are just utterly arbitrary, so you have no thing to latch onto. And so it's not really a thing that our brain does very well to learn meaningless, arbitrary stuff. So what you need to do is build connection. Somehow visualize a connection. And sometimes it's obvious or sometimes it's not. I'm trying to think of a good one for you now, but the first thing I think of is Lex Luthor.

But that's great.

Yeah.

So this isn't Lex Luthor. Wear a suit.

I think I know he has a shaved head, though. Or he's bald, which you're not. You got a great head. If I trade hair with you any day. But, like, you know, for something like that. But if I can come up with something, like, I could say, okay, so Lex Luthor is this criminal mastermind, and then I just imagine you, you talked.

About stabbing or whatever earlier.

Yeah, exactly.

Kind of connected and that's it.

Yeah, yeah. And I, but I'm serious though, that these kinds of weird associations now I'm building a richer network. I mean, one of the things that I find is if I've like you can have somebody's name that's just totally generic, like John Smith or something. Not that, no offense to people, that name. But you know, if I, if I see a generic name like that, but I've read John Smith's papers academically and then I meet John Smith at a conference, I can immediately associate that name with that face because I have this pre existing network to lock everything into. Right. And so you can build that network and that's what the method of loci or the memory palace technique is all about, is you have a pre existing structure in your head of like your childhood home or this mental palace that you've created for yourself. And so now you can put arbitrary pieces of information in different locations in that mental structure of yours and then you could walk through the different path and find all the pieces of information you're looking for. So the method of loci is a great method for just learning arbitrary things because it allows you to link them together and get that cue that you need to pop in and find everything. Right.

We should maybe linger on this memory palace thing just to make obvious because when people were describing to me a while ago what this is seems insane, I just. You literally think of a place like a childhood home or a home that you're really visually familiar with. And you literally place in that three dimensional space facts or people or whatever you want to remember. And you just walk in your mind along that place visually and you can remember, remind yourself of the different things. One of the limitations is there is a sequence to it. So it's. I think your brain somehow you need. You can't just like go upstairs right away or something. You have to like walk along the, the room. So it's really great for remembering sequences, but it's also not great for remembering like individual facts out of context. So the full context of the tour I think is important. But it's a, it's fascinating how the mind is able to do that when you ground these pieces of knowledge into something that you remember well already, especially visually fascinating. And you can just do that for any kind of sequence. I'm sure she used something like this for the, for IKEA catalog.

Something of this nature. Oh yeah, absolutely, absolutely. And I think the principle here is again I was telling you this idea that memories can compete with each other, right? Well, I like to use this example, and maybe someday I'll regret this, but I've used it a lot recently, is like, imagine if this were my desk. It could be cluttered with a zillion different things, right? So imagine it's just cluttered with a whole bunch of yellow post it notes. And on one of them I put my bank password on it, right? Well, it's going to take me forever to find it. It's just going to be buried under all these other post it notes. But if it's like hot pink, it's going to stand out and I find it really easily, right? So that's one way in which, if things are distinctive, if you've processed information in a very distinctive way, then you can have a memory that's going to last. And that's very good. For instance, for name face associations, if I get something distinctive about you that it's like that you've got very short hair. And maybe I can make the association with Lex Luthor that way or something like that, right? But I get something very specific. That's a great cue. But the other part of it is, what if I just organized my notes so that I have my finances in one pile and I have my reminders, my to do list in one pile and so forth. So I organize them well, then I know exactly if I'm going for my bank password, I could go to the finance pile, right? So the method of loci works, or memory palaces work because they give you a way of organizing. There's a school of thought that says that episodic memory evolved from this kind of knowledge of space and basically this primitive abilities to figure out where you are. And so people explain the method of loci that way. And whether or not the evolutionary argument is true, the method of loci is not at all special. So if you're not a good visualizer, stories are a good one. So a lot of memory athletes will use stories and they'll go like, if you're memorizing a deck of cards, they have a little code for the different, like the king and the jack and the ten and so forth. And they'll make up a story about things that they're doing and that'll work. Songs are a great one, right? I mean, it's like I can still remember. There's this obscure episode of the TV show Cheers. They sing a song about Albania that he uses to memorize all these facts about Albania. And I could still sing that song to you as I saw it on.

A TV show, you know, so what you mentioned spaced repetition. So what do you like this process? Maybe. Can you explain it?

Oh, yeah. If I am trying to memorize something, let's say if I have an hour to memorize as many Spanish words as I can, if I just try to do like half an hour, and then later in the day I do half an hour, I won't retain that information as long as if I do half an hour today and half an hour one week from now. And so doing that extra spacing should help me retain the information better. Now, there's an interesting boundary condition, which is it depends on when you need that information. So many of us, you know, for me, like, I can't remember so much from college and high school because I crammed, because I just did everything at the last minute. And sometimes I would literally study, like, you know, in the hallway right before the test. That was great, because what would happen is I just had that information right there. And so actually, not spacing can really help you if you need it very quickly.

Right.

But the problem is that you tend to forget it later on. But on the other hand, if you space things out, you get a benefit for later on retention. And so there's many different explanations. We have a computational model of this. It's currently under revision. But in our computer model, what we say is that maybe a good way of thinking about this is this conversation that you and I are having. It's associated with a particular context, a particular place in time. And so all these little cues that are in the background, these little guitar sculptures that you have, and that big light umbrella thing, right? All these things are part of my memory for what we're talking about, the content. So now later on, you're sitting around and you're at home drinking a beer, and you're thinking, God, what a strange interview that was. Right? So now you're trying to remember it, but the context is different. So your current situation doesn't match up with the memory that you pulled up. There's error. There's a mismatch between what you've pulled up and your current context. And so in our model, what you start to do is you start to erase or alter the parts of the memory that are associated with a specific place and time, and you heighten the information about the content. And so if you remember this information in different times, in different places, it's more accessible at different times in different places, because it's not overfitted. In an AI kind of way of thinking about things. It's not overfitted to one particular context. But that's also why the memories that we call upon the most also feel kind of like they're just things that we read about almost. You don't vividly reimagine them. Right. It's like they're just these things that just come to us, like facts. Right. And it's a little bit different than semantic memory, but it's like basically these events that we have recalled every, you know, over and over and over again. We keep updating that memory so it's less and less tied to the original experience. But then we have those other ones, which it's like you just get a reminder of that very specific context. You smell something, you hear a song, you see a place that you haven't been to in a while, and boom, it just comes back to you. And that's the exact opposite of what you get with spacing. Right.

That's so fascinating. So with space repetition, one of its powers is that you lose attachment to a particular context, but then it loses the intensity of the flavor of the memory. That's interesting. That's so interesting.

Yeah. But, you know, at the same time, it becomes stronger in the sense that the content becomes stronger. Yeah.

So it's used for. For learning languages, for learning facts, for learning, for. But, you know, for that generic semantic information type of memories.

Yeah. And I think this. This falls into a category. We've done other modeling. One of these is a published study in plos Computational Biology, where we showed that another way, which is, I think, related to the spacing effect, is what's called the testing effect. So the idea is that if you're trying to learn words, let's say in Spanish or something like that, and this doesn't have to be words, it could be anything. You test yourself on the words, and that act of testing yourself helps you retain it better over time than if you just studied it. Right. And so from traditional learning theories, some learning theories, anyway, this seems weird. Why would you do better giving yourself this extra error from testing yourself rather than just giving yourself perfect input? That's a replica of what it is that you're trying to learn. And I think the reason is that you get better retention from that error, that mismatch that we talked about. So what's happening in our model, it's actually conceptually kind of similar to what happens with backprop in AI or neural networks. And so the idea is that you expose. Here's the bad connections and here's the good connections. And so we can keep the parts of this cell assembly that are good for the memory and lose the ones that are not so good. But if you don't stress test the memory, you haven't exposed it to the error fully. And so that's why I think this is a thing that I come back to over and over again, is that you will retain information better if you're constantly pushing yourself to your limit. Right. If you are feeling like you're coasting, then you're actually not learning.

So it's like, so you should always be stress testing the memory system.

Yeah. And feel good about it. You know, even though everyone tells me, oh, my memory is terrible in the moment, they're overconfident about what they'll retain later on. So it's fascinating. And so what happens is when you test yourself, you're like, oh my God, I thought I knew that, but I don't. And so it can be demoralizing until you get around that and you realize, hey, this is the way that I learn. This is how I learn best. It's like if you're trying to star in a movie or something like that, you don't just sit around reading the script, you actually act it out. And you're going to botch those lines from time to time. Right.

You know, there's an interesting moment. You probably have experienced this. I remember good friend of mine, Joe Rogan. I was on his podcast and.

We.

Were randomly talking about soccer, football, somebody I grew up watching, Diego Armando Maradona, one of the greatest soccer players of all time. And we are talking about him and his career and so on. And Joe asked me if he's still.

Around.

And I said, yeah. I don't know why I thought, yeah, because that was a perfect example of memories. He passed away. I tweeted about it, how heartbroken I was, all this kind of stuff, like a year before. I know this. But in my mind I went back to the thing I've done many times in my head, visualizing some of the epic runtier on goal and so on. So for me, he's alive. So I'm in part of the. Also the conversation when you're talking to Joey, there's stress and like the focus is allocated, the attention is allocated in a particular way. But when I walked away, I was like, in which world was Diego Mardona still alive? Like in which. Because I was sure in my head that he was still alive. There was a. It's A moment that sticks with me. There's. I've had a few like that in my life where it just kind of like obvious things just disappear from mind. And it's cool. Like, it shows actually the power of the mind in a positive sense to erase memories. You want erased, maybe, but I don't know. I don't know if there's a good explanation for that.

One of the cool things that. That I found is that some people really just revolutionize a field by. By creating a problem that didn't exist before. It's kind of like why I love science is like, I. Engineering is like solving other people's problems, and science is about creating problems. I'm just much more like I want to break things and, you know, create problems.

Yeah.

Not necessarily move fast, though. But one of my former mentors, Marsha Johnson, who in my opinion, is one of the greatest memory researchers of all time, she comes up, young woman in the field, and it's mostly guy field. And she gets into this idea of how do we tell the difference between things that we've imagined and things that we actually remember? How do we tell. I get some mental experience. Where did that mental experience come from? Right. And it turns out this is a huge problem because essentially our mental experience of remembering something that happened, our mental experience of thinking about something, how do you tell the difference? They're both largely constructions in our head. And so it is very important. And the way that you do it is. I mean, it's not perfect, but the way that we often do it and succeed is by, again, using our prefrontal cortex and really focusing on the sensory information or the place and time and the things that put us back into when this information happened. And if it's something you thought about, you're not going to have all of that vivid detail as you do for something that actually happened. But it doesn't work all the time. But that's a big thing that you have to do. But it takes time, it's slow, and it's again, effortful. But that's what you need to remember accurately. But what's cool, and I think this is what you alluded to about how that was an interesting experience is imagination is exactly the opposite. Imagination is basically saying, I'm just going to take all this information from memory, recombine it in different ways, and throw it out there. And so, for instance, Dan Schachter and Donna Addis have done cool work on this. Demis Hassabis did work on this with Eleanor McGuire and UCL and this goes back actually to this guy, Frederick Bartlett, who is this revolutionary memory researcher. At Bartlett, he actually, like, rejected the whole idea of quantifying memory. He said, there's no statistics in my book. He came from this anthropology perspective and short version of the story is he just asked people to recall things. You give people stories and poem, ask people to recall them. And what he found was people's memories didn't reflect all of the details of what they were exposed to, and they did reflect a lot more. They were filtered through this lens of prior knowledge, priority, the cultures that they came from, the beliefs that they had, the things they knew. And so what he concluded was that he called remembering an imaginative construction, meaning that we don't replay the past. We imagine how the past could have been by taking bits and pieces that come up in our heads. And likewise, he wrote this beautiful taper on imagination, saying, when we imagine something and create something, we're creating it from these specific experiences that we've had and combining it with our general knowledge. But instead of trying to focus it on being accurate and getting out one thing, you're just ruthlessly recombining things without any, you know, any necessary kind of goal in mind. I mean, or at least that's one kind of creation.

So imagination is fundamentally coupled with memory in both directions?

I think. So. I mean, it's not clear that it is in everyone. But one of the things that's been studied is some patients who have amnesia, for instance, they have brain damage, say, to the hippocampus. And if you ask them to imagine things that are not in front of them, like, imagine what could happen after I leave this room. Right. They find it very difficult to give you a scenario what could happen. Or if they do, it would be more stereotyped. Like, yes, this would happen. This would. But it's not like they can come up with anything that's very vivid and creative in that sense. That's partly because when you have amnesia, you're stuck in the present. Because to get a very good model of the future, it really helps to have episodic memories to draw upon. Right? And so that's the basic idea. And in fact, one of the most impressive things, when people started to scan people's brains and ask people to remember past events, what they found was there was this big network of the brain called the default mode network. It gets a lot of press because it's thought to be important. It's engaged during mind wandering. And if I ask you to pay attention to something, it only comes on when you stop paying attention. So people, oh, it's just this kind of daydreaming network. And I thought, this is just ridiculous research. Who cares? But then what people found was when people recall episodic memories, this network gets active. And so we started to look into it. And this network of areas is really closely functionally interacting with the hippocampus. And so, in fact, some would say the hippocampus is part of this default network. And if you look at brain images of people, or brain maps of activation, so to speak, of people imagining possible scenarios of things that could happen in the future, even things that couldn't really be very plausible, they look very similar. I mean, you know, to the naked eye, they look almost the same as maps of brain activation when people remember the past. According to our theory, and we've got some data to support this, we've broken up this network into various sub pieces. Is that basically it's kind of taking apart all of our experiences and creating these little Lego blocks out of the. And then you can put them back together if you have the right instructions to recreate these experiences that you've had. But you could also reassemble them into new pieces to create a model of an event that hasn't happened yet. And that's what we think happens. And when I'm. Our common ground that we're establishing in language requires using those building blocks to put together a model of what's going on.

There's a good percentage of time I personally live in in the imagined world I think of. I have. I do thought experiments a lot. I, you know, take the. The absurdity of human life as it stands and play it forward in all kinds of different directions. Sometimes it's rigorous thoughts, thought experiments, sometimes it's fun ones. So I imagine that that has an effect on how I remember things. And I suppose I have to be a little bit careful to make sure stuff versus stuff that I just imagined happened. And this also, I mean, some of my best friends are characters inside books that never even existed. And I'm, you know, there's some degree to which they actually exist in my mind. Like, these characters exist, authors exist, Dostoevsky exists, but also Brothers Karamazov.

I love that book. Yeah, One of the few books I've read, one of the few literature books that I've read.

I should.

I read a lot in school that I don't remember. But Brothers Karamatsa, they exist.

And I have almost kind of like conversations with them. It's interesting. It's it's interesting to allow your brain to kind of play with ideas of the past, of the imagined, and see it all as one.

Yeah, there was actually this famous mnemonist. He's kind of like back then the equivalent of a memory athlete, except he would go to shows and do this. Those described by this really famous neuropsychologist from Russia named Luria. And so this guy was named Solomon Shereshevsky. And he had this condition called synesthesia that basically created these weird associations between different senses that normally wouldn't go together. So that gave him this incredibly vivid imagination that he would use to basically imagine all sorts of things that he would need to memorize. And he would just imagine like just create these incredibly detailed things in his head that allowed him to memorize all sorts of stuff. But it also really haunted him by some reports that basically it was like he was at some point, you know, and again, who knows, the drinking was part of this. But he at some point had trouble differentiating his imagination from reality. Right. And this is, this is interesting because it's like, I mean that's what psychosis is in some ways is first of all, you're just learning connections from prediction errors that you probably shouldn't learn. And the other part of it is that your internal signals are being confused with actual things in the outside world.

Well, that's why a lot of this stuff is both feature and bug. It's a double edged sword.

Yeah. I mean, it might be why there's such an interesting relationship between genius and psychosis. Yeah.

Maybe they're just two sides of the same coin. Humans are fascinating, aren't they?

I think so. Sometimes scary, but mostly fascinating.

Can we just talk about memory sport a little longer? There's something called the USA Memory Championship. Like, what are these athletes? Like, what does it mean to be like elite level at this? Have you interacted with any of them or reading about them? What have you learned about these folks?

There's a guy named Henry Rodger who's studying these guys. And there's actually a book by Joshua Foer called Moonwalking with Einstein where he talks about. He actually, as part of this book, just decided to become a memory athlete. They often have these life events that make them go, hey, why don't I do this? So there was a guy named Scott Hagwood who I write about, who thought that he was getting chemo for cancer. And so he decided because chemo, there's a well known thing called chemo brain where people become like, they just lose a Lot of their sharpness. And so he wanted to fight that by learning these memory skills. So he bought a book. And this is the story you hear in a lot of memory athletes is they buy a book by other memory athletes or other memory experts, so to speak, and they just learn those skills and practice them over and over again. They start by winning bets and so forth, and then they go into these competitions. And the competitions are typically things like memorizing long strings of numbers or memorizing orders of cards and so forth. So there tend to be pretty arbitrary things. Not like things that you'd be able to bring a lot of prior knowledge, but they build the skills that you need to memorize arbitrary things.

Yeah, it's fascinating. I've gotten a chance to work with something called NBAC tasks. So there's all these kinds of tasks, memory recall tasks that you used to kind of load up the quote unquote, working memory.

Yeah, yeah.

And to see. The psychologists used it to test all kinds of stuff, like to see how well you're good at multitasking. We used it in particular for the task of driving. Like, if we fill up your brain with intensive working memory tasks, how good are you at also not crashing that kind of stuff? So it's fascinating. But again, those tasks are arbitrary and they're usually about recalling a sequence of numbers in some kind of semi complex way. Are you. Do you have any favorite tasks of this nature in your own studies?

I've really been most excited about going in the opposite direction and using things that are more and more naturalistic. And the reason is that we've moved in that direction because what we found is that memory works very, very differently when you study it, when you study memory in the way that people typically remember. And so it goes into a much more predictive mode. And you have these event boundaries, for instance, and you have. But a lot of what happens is this kind of fascinating mix that we've been talking about, a mix of interpretations and imagination with perception. And the new direction we're going in is understanding navigation in our memory first places. And the reason is that there's a lot of work that's done in rats, which is very good work. They have a rat and they put it in a box and the rat goes. Chases cheese in a box. And you'll find cells in the hippocampus that fire when a rat is in different places in the box. And so the conventional wisdom is that the hippocampus forms this map of the box. And I think that probably may happen when you have absolutely no knowledge of the world. Right. But I think one of the cool things about human memory is we can bring to bear our past experiences economically, learn new ones. And so, for instance, if you learn a map of an Ikea, let's say if I go to the IKEA in Austin, I'm sure there's one here, I probably could go to this IKEA and find my way to the, you know, where the wine glasses are without having to even think about it, because it's got a very similar layout. Even though IKEA is a nightmare to get around, once I learned my local ikea, I can use that map everywhere, wipe form a brand new one for a new place. And so that kind of ability to reuse information really comes into play when we look at things that are, you know, more naturalistic tasks. And another thing that we're really interested in is this idea of, like, what if instead of basically mapping out every coordinate in a space, you form a pretty economical graph that connects basically the major landmarks together and being able to use that as, you know, emphasizing the things that are most important, the places that you go for food and the places that are landmarks that help you get around, and then filling in the blanks for the rest. Because I really believe that cognitive maps or mental maps of the world, just like our memories for events, are not photographic. I think they're this combination of actual verifiable details and then a lot of inference that you make.

So what have you learned about this kind of spatial mapping of places? How do people represent locations?

There's a lot of variability. I think that. And there's a lot of disagreement about how people represent locations. In a world of GPS and physical maps, people can learn it from, like, basically what they call, like, survey perspective, being able to see everything. And so that's one way in which humans can do it that's a little bit different. There's one way which we can memorize routes. Like, I know how to get from here to. Let's say if I knew, walk here from my hotel, I could just rigidly follow that route back. Right. And there's another, more integrative way, which would be what's called a cognitive map, which would be kind of a sense of how everything relates to each other. And so there's lots of people who believe that these maps that we have in our head are isomorphic with the world. They're like these literal coordinates that follow Euclidean space. And as you know, Euclidean mathematics is very constrained. Right. And I think that we are actually much more generative in our maps of space. So that we do have these bits and pieces and we've got a small task. It's right now, not yet. We need to do some work on it for further analysis. But one of the things we're looking at is these signals called ripples in the hippocampus, which are these bursts of activity that you see that are synchronized with areas in the neocortex in the default network, actually. And so what we find is that those ripples seem to increase at navigationally important points when you're making a decision or when you reach a goal. This speaks to the emotion thing, right? Because if you have limited choices, if I'm walking down a street, I could really just get a mental map of the neighborhood with a more minimal kind of thing by just saying, here's the intersections and here's the directions I take to get in between them. And what we found in general in our MRI studies is basically the more people can reduce the problem, whether it's space or any kind of decision making problem, the less the hippocampus encodes. It really is very economical towards the points of most highest information, content and value.

So can you describe the encoding in the hippocampus and the ripples you were talking about? What's the signal in which we see the ripples?

Yeah, so this is really interesting. There are these oscillations, right? So there's these waves that you basically see. And these waves are points of very high excitability and low excitability. And at least during. They happen actually during slow wave sleep too. So the deepest stages of sleep, when you're just zonked out, right? You see these very slow waves where it's like very excitable and then very unexcitable. It goes up and down and on top of them you'll see these little sharp wave ripples. And when there's a ripple in the hippocampus, you tend to see a sequence of cells that resemble a sequence of cells that fire when an animal is actually doing something in the world. So it almost is like a little. People call it replay. I think it's a little bit. I don't like that term, but it's basically a little bit of a compressed play of the sequence of activity in the brain that was taking place earlier. And during those moments there's a little window of communication between the hippocampus and these areas in the neocortex. And so that I think helps you form new memories, but it Also helps you, I think, stabilize them, but also really connect different things together in memory and allows you to build bridges between different events that you've had. And so this is one of at least our theories of sleep and its real role in helping you see the connections between. Between different events that you've experienced.

So during sleep is when the connections.

Are formed, the connections between different events. Right. So it's like you see me now, you see me next week, you see me a month later. You start to build a little internal model of how I behave and what to expect of me. And we think sleep, one of the things that allows you to do is figure out those connections and connect the dots and find the signal and the noise.

So you mentioned fmri. What is it and how is it used in studying memory?

This is actually the reason why I got into this whole field of science. When I was in grad school, FMRI was just really taking off as a technique for studying brain activity. And what's beautiful about it is you can study the whole human brain and there's lots of limits to it, but you can basically do it in a person without sticking anything into their brains. And very non. Invasive. I mean, for me, being an MRI scanner is like being in the womb. I just fall asleep if I'm not being asked to do anything. I get very sleepy, you know. But you can have people watch movies while they're being scanned, or you can have them do tests of memory, like giving them words and so forth to memorize. But what MRI is itself is just this technique where you put people in a very high magnetic field. Typical ones we would use would be 3 Tesla to give you an idea. So 3 Tesla magnet. You put somebody in and what happens is you get this very weak but measurable magnetization in the brain. And then you apply a radio frequency pulse, which is basically a different electromagnetic field. And so you're basically using water, the water molecules in the brain, as a tracer, so to speak. And part of it in FMRI is the fact that these magnetic fields that you mess with by manipulating these radio frequency pulses and the static field, and you have things called gradients which change the strength of the magnetic field in different parts of the head. So they're all. We tweak them in different ways. But the basic idea that we use in FMRI is that blood is flowing to the brain. And when you have blood that doesn't have oxygen on it, it's a little bit more magnetizable than blood that does because you have hemoglobin that carries the oxygen, the iron basically, in the blood that makes it red. And so that hemoglobin, when it's deoxygenated, actually has different magnetic field properties than when it has oxygen. And it turns out when you have an increase in local activity in some part of the brain, the blood flows there, and as a result, you get a lower concentration of hemoglobin that is not oxygenated, and then that gives you more signal. So I gave you, I think I sent you a gif, as you like to say.

Yeah, we had off record intense argument. Well, if it's pronounced gif or gif, but that's. We shall set that aside.

We could have called it a stern.

Rebuke perhaps, but rebuke, Yeah, I drew a hard line. It is true. The creator of GIFs, that is pronounced gif, but that's the only person that pronounces gif. Anyway, yes, you sent a GIF of.

This would be basically a movie of FMRI data. And so when you look at it, it's not very impressive. It looks like these very pixelated maps of the brain, but it's mostly kind of like white. But these tiny changes in the intensity of those signals that you probably wouldn't be able to visually perceive, like about 1%, can be statistically very, very large effects for us. And that allows us to see, hey, there's an increase in activity in some part of the brain when I'm doing some task, like trying to remember something. And I can use those changes to even predict is a person going to remember this later or not? And the coolest thing that people have done is to decode what people are remembering from the patterns of activity. Because maybe when I'm remembering this thing, like I'm remembering the house where I grew up, I might have one pixel that's bright in the hippocampus and one that's dark. And if I'm remembering something like more like the car that I used to drive when I was 16, I might see the opposite pattern where a different pixel is bright. And so all that little stuff that we used to think of noise, we can now think of almost like a QR code for memory, so to speak, which. Where different memories have a different little pattern of bright pixels and dark pixels. And so this really revolutionized my research. So there's fancy research out there where people really. I mean, not even that. I mean, by your standards, this would be stone age, but, you know, applying machine learning techniques to do decoding and so forth. And now there's a lot of forward encoding models. You can go to town with this stuff, right? And I'm much more old school of designing experiments where you basically say, okay, here's a whole web of interaction of memories that overlap in some way, shape or form. Do memories that occurred in the same place have a similar QR code? And do memories that occurred in different places have different QR code? And you can just use things like correlation coefficients or cosine distance to measure that stuff, Right? Super simple, right? And so what happens is you can start to get a whole state space of how a brain area is indexing all these different memories. And it's super fascinating because what we could see is this little separation between how certain brain areas are processing memory for who was there, and other brain areas are processing information about where it occurred or the situation that's kind of unfolding. And some are giving you information about what are my goals that are involved and so forth. And the hippocampus is just putting it all together into these unique things that just are about when and where it happened.

So there is a separation between spatial information concepts. Like, literally, there's distinct, as you said, QR codes for these, so to speak.

Let me try a different analogy, too, that might be more accessible for people, which would be like, you've got a folder on your computer, right? I open it up, there's a bunch of files there. I can sort those files by, you know, alphabetical order. And now things that both start with the letter A are lumped together, and things that start with Z versus A are far apart. And so that is one way of organizing the folder. But I could do it by date. And if I do it by date, things that were created close together in time are close, and things that are far apart in time are far. So you can think of how a brain area or a network of areas contributes to memory by looking at what the sorting scheme is. And these QR codes that we're talking about that you get from FMRI allow you to do that. And you can do the same thing if you're recording from massive populations of neurons in an animal. And you can do it for recording local potentials in the brain. So little waves of activity in, let's say, a human who has epilepsy, and they stick electrodes in their brain, try to find the seizures. So that's some of the work that we're doing now. But all these techniques basically allow you to say, hey, what's the sorting scheme? And so we've found that some Networks of the brain sort information in memory according to who was there. So I might have. Like we've actually shown in one of my favorite studies of all time that was done by a former postdoc, Zach Reah. And Zach did the study where we had a bunch of movies with different people in my labs that are two different people, and you filmed them at two different cafes and two different supermarkets. And what you could show is in one particular network, you could find the same kind of pattern of activity, more or less a very similar pattern of activity. Every time I saw Alex in one of these movies, no matter where he was, and I could see another one that was a common pattern that happened every time I saw this particular supermarket nugget. And it didn't matter whether you're watching a movie or whether you're recalling the movie, it's the same kind of pattern that comes up. Right.

That's so fascinating.

It's fascinating. So now you have those building blocks for assembling a model of what's happening in the present, imagining what could happen, and remembering things very economically from putting together all these pieces so that all the hippocampus has to do with is get the right kind of blueprint for how to put together all these building blocks.

These are all like, beautiful hints at a super interesting system that makes me wonder on the other side of it how to build it. But it's like, it's fascinating. Like, the way it does the encoding is really, really fascinating. Or I guess the symptoms, the results of that encoding are fascinating to study from this. Just as a small tangent, you mentioned sort of the measuring local potentials with electrodes versus fmri.

Oh, yeah.

What are some interesting, like, limitations, possibilities of fmri? Maybe the way you explained it is like brilliant with, with blood and detecting the. The activations or the excitation because blood flows to that area. What's like, the latency of that? Like, what's the blood dynamics in the brain? That.

Yeah, yeah.

Like, how quickly can it. How quickly can the tasks change and all that kind of stuff?

Yeah, I mean, it's very slow. To the brain, 50 milliseconds is like, you know, like it's an eternity. Maybe not 50. Oh, you know, maybe like, you know, let's say half a second, 500 milliseconds. Just so much back and forth stuff happens in the brain and that time. Right. So in fmri, you can measure these magnetic field responses about six seconds after that burst of activity would take place. All these things. It's like, is it a feature or is it a bug? Right. So one of the interesting things that's been discovered about FMRI is it's not so tightly related to the spiking of the neurons. So we tend to think of the computation, so to speak, as being driven by spikes, meaning, like, there's just a burst of. It's either on or it's off, and the neuron's, like, going up or down. But sometimes what you can have is these states where the neuron becomes a little bit more excitable or less excitable. And so FMRI is very sensitive to those changes in excitability. Actually, one of the fascinating things about FMRI is where does that. How is it we go from neural activity to essentially blood flow to oxygen? All this stuff? It's such a long chain of going from neural activity to magnetic fields. And one of the theories that's out there is most of the cells in the brain are not neurons. They're actually these support cells called glial cells. And one big one is astrocytes. And they play this big role in regulating, kind of being a middle man, so to speak, with the neurons. So if you, for instance, one neuron's talking to another, you release a neurotransmitt, like, let's say, glutamate, and that gets another neuron starts getting active after you release it in the gap between the two neurons called synapse. So what's interesting is if you leave that, you know, imagine you're just flooded with this, like, liquid in there, right? If you leave it in there too long, you just excite the other neuron too much, and you can start to basically get seizure activity. You don't want this, so you got to suck it up. And so actually what happens is these astrocytes, one of their functions is to suck up the glutamate from the synapse, and that is a massively. And then break it down and then feed it back into the neurons so that you can reuse it. But that cycling is actually very energy intensive. And what's interesting is, at least according to one theory, they need to work so quickly that they're working on metabolizing the glucose that comes in without using oxygen, kind of like what, you know, anaerobic metabolism. So they're not using oxygen as fast as they are using glucose. So what we're really seeing in some ways may be in fmri, not the neurons themselves being active, but rather the astrocytes, which are meeting the metabolic demands of the process of keeping the whole system going.

It does seem to be that FMRI is a good way to study activation. So with these astrocytes, even though there's a latency, it's pretty reliably coupled to the activations.

Oh, well, this gets me to the other part about my. So now let's say, for instance, if I'm just kind of like I'm talking to you, but I'm kind of paying attention to your cowboy hat, right? So I'm looking off to the. Or I'm thinking about the. Right. Even if I'm not looking at it, what you'd see is that there would be this little elevation in activity and in areas in the visual cortex which process vision around that point in space. Okay. So if then something happened, like, you know, sudden a light flashed in that part of, you know, right in front of your cowboy hat, I would have a bigger response to it. But what you see in FMRI is, even if I'm not, Even if I don't see that flash of light, there's a lot of activity that I can measure because you're kind of keeping it excitable. And that in and of itself, itself, even though I'm not seeing anything there that's particularly interesting, there's still this increase in activity. And so it's more sensitive with fmri. So is that a feature or is it a bug? You know, some people, people who study spikes in neurons, would say, well, that's terrible. We don't want that. You know, likewise, it's slow. And that's terrible for measuring things that are very fast. But one of the things that we found in our work was when we give people movies and when we give people stories to listen to, a lot of the action is in the very, very slow stuff. It's in. Because if you're thinking about, like, a story, let's say you're listening to a podcast or something, you're listening to the Lex Friedman podcast, right? You're putting this stuff together and building this internal model over several seconds, which is basically, we filter that out when we look at electrical activity in the brain, because we're interested in this millisecond scale. It's almost massive amounts of information, right? So the way I see it is every technique gives you a little limited window into what's going on. FMRI is huge problems. People lie down in the scanner. There's parts of the brain where I'll show you in some of these images where you'll see kind of gaping holes because you can't keep the magnetic field stable in those Spots. You'll see parts where it's like, there's a vein. And so it just produces big increases and decrease in signal or respiration that causes these changes. There's lots of artifacts and stuff like that. You know, every technique has its limits. If. If I'm lying down an MRI scanner, I'm lying down. I'm not interacting with you in the same way that I would in the real world, but at the same time, I'm getting data that I might not be able to get otherwise. And so different techniques give you different kinds of advantages.

What kind of big scientific discoveries, maybe the flavor of discoveries, have been done throughout the history of the science of memory, the studying of memory. What kind of things have been understood?

Oh, there's so many. It's really so hard to summarize it. I mean, I think it's funny because it's like, when you're in the field, you can get kind of blase about this stuff. But then once I started writing the book, I was like, oh, my God, this is really interesting. How did we do all this stuff? I would say that some of the. I mean, you know, from the first studies, just showing how much we forget is very important. Showing how much schemas, which is our organized knowledge about the world, increase our ability to remember information, just massively increase it. Studies of expertise showing how experts like chess experts can memorize so much in such a short amount of time because of the schemas they have for chess, but then also showing that those lead to all sorts of distortions in memory. The discovery that the act of remembering can change the memory, it can strengthen it, but it can also distort it if you get misinformation at the time. And it can also strengthen or weaken other memories that you didn't even recall. So just this whole idea of memory as an ecosystem, I think, was a big discovery. I could go. This idea of breaking up our continuous experience into these discrete events, I think was a major discovery.

So the discreteness of our encoding of events, maybe.

Yeah. I mean, and again, there's controversial ideas about this, right? But it's like, yeah, this idea that. And this gets back to just this common experience of you walk into the kitchen and you're like, why am I here? And you just end up grabbing some food from the fridge. And then you go back and you're like, oh, wait a minute, I left my watch in the kitchen. That's what I was looking for. And so what happens is that you have a little internal model of where you are what you're thinking about. And when you cross from one room to another, those models get updated. And so now when you're in the kitchen, you have to go back and mentally time travel back to this earlier point to remember what it was that you went there for. And so these event boundaries turns out, like, in our research, and again, I don't want to make it sound like we've figured out everything, but in our research, one of the things that we found is that basically, as people get older, the activity in the hippocampus at these event boundaries tends to go down. But independent of age, if I give you outside of the scanner, you're done with the scan. I just scan you while you're watching a movie. Movie, just watch it. You come out. I give you a test of memory for stories. What happens is you find this incredible correlation between the activity in the hippocampus at these singular points in time, these event boundaries, and your ability to just remember a story outside of the scanner later on. So it's marking this ability to encode memories, just these little snippets of neural activity. So I think that's a big question. One, there's all sorts of work in animal models that I can get into, you know, sleep. I think there's so much interesting stuff that's being discovered in sleep right now, being able to just record from large populations of cells and then be able to relate that. And I think the coolest thing gets back to this QR code thing because, like, what we can do now is, like, I can take FMRI data while you're watching a movie, or let's do better than that. Let me get FMRI data while you use a joystick to move around in virtual reality. So you're in the metaverse, whatever. Right. But it's kind of a crappy metaverse because there's always so much metaverse you can do in an MRI scenario. So you're doing this crappy metaverse. So now I can take a rat record from his hippocampus and prefrontal cortex and all these areas with these really new electrodes. I get massive amounts of data and have it move around on a trackball in virtual reality in the same metaverse that I did record that rat's activity. I can get a person with epilepsy who we have electrodes in their brain anyway to try to figure out where the seizures are coming from. It was a healthy part of the brain record from that person. Right. And I can get a computational model and one of the brand new members in my lab, Tyler Bond is just doing some great stuff. He really relates computer vision models and looks at the weaknesses of computer vision models and relates it to what the brain does well. And so you can actually take a, a ground truth, you know, code for the metaverse, basically, and you can feed in the visual information, let's say the sensory information or whatever that's coming in to a computational model that's designed to take real world inputs, right? And you could basically tie them all together by virtue of the state spaces that you're measuring in neural activity in these different formats and these different species. And in the computational model, which is just, I just find that mind blowing, you could do different kinds of analyses on language and basically come up with just like, basically it's the guts of LLMs, right? You have, you could do analyses on language and you could do analyses on, you know, sentiment analyses of emotions and so forth. Put all this stuff together, I mean, it's almost too much. But if you do it right and you do it in a theory driven way, as opposed to just throwing all the data at the wall and see what sticks, I mean, that to me is just exceptionally powerful.

So you can take FMRI data in across species.