Scientist Stories: Uri Hasson, Deep Neural Networks as Cognitive Models for the Brain
In this seminar, Uri Hasson (hosted by Marco Iacoboni) presents a new research program that uses deep language models as cognitive models of the human brain. He begins by framing a "political validity crisis" in neuroscience: while many experiments replicate in the lab, they fail to generalize to real‑world cognition because laboratory stimuli are overly simplified and context‑free. To address this, Hasson argues for moving from classic, low‑dimensional, rule‑based theories of language (e.g., universal grammar) to high‑dimensional, context‑dependent models such as transformer‑based language models (GPT‑2/3). He explains how these models generate contextual embeddings that capture syntax, semantics, pragmatics, and world knowledge, and how they can be trained in a self‑supervised next‑word prediction framework that mirrors a plausible brain objective function. He then describes a series of empirical studies in his lab that compare these models to human neural data recorded with invasive ECoG while participants listen to naturalistic stories (e.g., a 30‑minute podcast). Using linear encoding models, he shows that (1) static word embeddings predict brain activity modestly, (2) contextual embeddings improve prediction substantially, and (3) the models can predict neural responses to completely novel words (zero‑shot) with high accuracy, especially in language‑selective cortex (Broca’s area). He also demonstrates that the brain exhibits predictive coding: neural activity anticipates upcoming words, and surprise (prediction error) correlates with post‑word responses, mirroring the model’s error signal. A key finding is a layer‑wise alignment: early transformer layers correspond to early neural responses, later layers to later responses, suggesting a temporal hierarchy that matches the brain’s processing cascade. Hasson discusses limitations of current large language models – their massive parameter counts, lack of grounding, multimodal experience, and biologically implausible learning rules (back‑propagation). To overcome these, his lab is launching a developmental data‑collection effort: installing cameras and microphones in dozens of homes to record multimodal infant environments over years, aiming to train more human‑centric, grounded models that learn language from embodied, social interaction rather than massive text corpora. The talk concludes with a discussion of future directions, including multimodal, reinforcement‑learning‑style objectives, biologically plausible local learning rules, and a two‑way feedback loop where neuroscience can inform AI and vice‑versa. Audience questions touch on the feasibility of grounding, the role of Gibson’s ecological perception, and how emotional and personal context could be incorporated into models.
Chapters
Hello everyone, my name is Marco Iacoboni. I'm one of the faculty at the Brain Mapping Center and I'm delighted to host this seminar. I've been a fan of Uri Hasson's work for many, many years. I need to tell you the story how I discovered his work. I was skiing in Telluride and that was like almost 20 years ago. We didn't have smartphones, our house we rented didn't have Wi-Fi, so I go to an internet cafe to check my emails every day before hitting the slopes. And one day I check the new papers in Science and I see Uri's paper on synchronized brains while watching movies, and it completely blew my mind. And he's doing things that blow my mind even more now. He's done his PhD with Rafi Malach at the Weizmann Institute. Then he, you know, crossed the oceans and did his postdoc at NYU with David Kiger. And then Princeton was smart enough to get him, and he never left Princeton, and he's done this beautiful work. And I'll just shut up here and let Uri talk.
I'm very happy to be here. People on Zoom, I will— because I like to move around, I will try to give different angle. And let's see if I can get over here. Okay, so I'm very happy to be here, and I'm going— we're going to talk about the new line of research in my lab, pretty new. And I want to start with this. This is the most important slide. This is the people that did the work. I'm only presenting. It's a pretty big team. We have basically a collaboration between my lab and NYU Electric Center and people in Google. So it's a pretty big team. But I really want to mention Ariel. Ariel, I think, is the driving force in this work. So he's basically leading most of it. OK, I think we are stuck with this. There's nothing we can do. Let's start. I just gave a class, a lecture in a class, so the first two slides are going to be identical, the same origin. The origin point basically, we're all aware that there is a replication crisis in psychology, in neuroscience, and in general in the sciences. But there is also another crisis. In this crisis, I'm calling it the political validity crisis. And— last time— and basically this crisis is not that we are failing to replicate results. Even the results that are most replicable in the lab fail to transfer to a life context. So then we get to a situation that we're really modeling our lab experiment instead of modeling the process of the chocolate that we have. This is really problematic. And I think it's coming from the following reason. I think basically our lab experiment, we want it to be ideal. We want it to be simple. Interpretable, a few parameters that have a lot of power. And we want to use like simple and interpretable parameters that we discover in the lab to be able to explain everything that's happening in real life, right? Because we don't want to explain the lab, we want to explain cognition in the wild. The problem is that when we go to this like a very simple interpretable, our models very general to explain real life. So this is the problem. And why it's happening, happening I think from— I'm summarizing a lot of thinking and I will try to make it more concrete when I go on— is from the following two reasons. Real life are multidimensional, they are messy, okay, so they are far from being linear and simple. Now when I'm speaking, there's many, many factors. What you ate in the morning, how much you slept, which newspaper you're reading, which scientific journals you're reading, which lab you are coming from, did you have a fight with your friend? This is all going to interfere with me speaking. So it's very multidimensional and immersive. It's also specific to context. Difficult to get towards because in different contexts— this is my response— in different contexts it's going to be slightly different. And our notion, if you want it like simple, interpretable, strong theories like physics, like Newton, you know, 4 dimensions that explain a lot of challenging psychology neuroscience. And I'm going to give you an example of language, and then I'm going to give you an alternative way of doing science in the real life that can really come with a model to model cognition in their life. And if we accept it is a cognitive model, it's a paradigm shift. So maybe most of you won't like it, because if I'm correct in this direction, it means that we need to change the way we are thinking about psychology. So it's really, there's a lot in stake and therefore I really want you also to interact and interfere and say, "I don't like it, please explain more," and push me and I will try to answer the question. So let's do it with language because language is the basis of the cognitive revolution. Okay, and it's the base of cognition. So what's happening in language? Language is complex and messy, right, and multi-dimensional. Let me say what we want to understand: it is a cognitive process. So what we are doing, basically, we try basically to look for the simple underlying ideal principles that can explain language in the world. We look at the psychology and linguistics. We see some structure in language. It's not chaotic. For example, we see there are parts of speech, there are nouns, there are verbs, there are pronouns, there are adjectives, and then we see there are some structural language. Then we say this structure that we see in the stimuli, there's underlying principles that are very simple and beautiful. All right, so we postulate that there's like this underlying universal grammar, and then we define that rule-based system that is very simple and clear by which we try to understand how language is operating in the wild. And we also try to look what is the neural basis of this cognitive neuroscience. And the problem is that this is very, very difficult to do. And the way we It's not really working well. So firstly, what we are trying to do is say, oh, it's so complicated, let's divide and conquer, let's simplify. And this is where you go to the lab and simplify it even further. For example, in Chomsky, a classical example, he gave you this sentence that has no meaning and said, I don't study meaning. I don't study pragmatic. So no semantic, no pragmatic. I study syntax. And this is the syntax, in the syntax I will work, it will work this way. And I remove everything else. But even in like divide and conquer, simplify lab experiment, it's not working. And why it's not working? It's not working again because of context. It's not that it's a failure, okay? Let's say linguistic rules and linguistic system managed to explain 50% of the situation. This is a lot. If chance is zero and you explain 50% of the cases, you are very happy. But it's only 50%. Why 50%? Because for each rule that you come in with, you have 50% exceptions. Let me give you an example. For plural, add the suffix -s. Simple rule, right? Then you can see. But in some contexts, you have to add -es, and in other contexts, you have to add -ais. In other contexts, you have to add -es. In other contexts, I don't know, it's irregular. Simply learn, learn it. And then even if you learn all this, like example, subword structure in regular strange context, you get to this thing that 'pen' definitely is singular, and you have this 's'. Why? Why do you— and that's basically true for each linguistic word that you— and, you know, and it gets more and more complicated. And then you cannot basically program your words into a system that should work in real life. All this like endeavor of basically finding this factor of the ideal principle, it's failing in real life, very failing. And because context and the messiness, it suddenly Interesting. Coming out of field, not us, very striking philosophy, at least for me, comes like big models that suddenly walk in real life. And I will show how they are walking in real life. So first we need to remember this, this model is walking in real life. They basically solve a problem that we failed to solve in cognitive neuroscience, but they are very different. And this is not our model. This is not what we love. This is not what we plan on. We have very bad reaction to it. We call them black box. If you look at the number of parameters, there are 1.5 million parameters. This is not a model of 5 parameters that is interpretable. It's like Mansfield that came from data. And we don't like them. We're going first to explain to you how they're working. And then I will argue that they are not black box. And I think there's something that we can learn from them. But if we are learning from them, it's a paradigm shift. So let's start. [INAUDIBLE] To try to get rid of this?
Well, it won't let me disable the waiting room anymore. Must be a new feature. Can you stay closer to your laptop? They said it sounds better when you're closer.
Yeah, of course it's better. Sorry. And then you will see my face. Okay, so I will try to stand over. Okay, so let's see. Let me explain. How many of you want to know the people in the Zoom? I want to know how many know a lot about this, like these models, how they are working. But also the principle could help. So I'm going to give a very brief, very superficial introduction, okay? If you want to, without math, without anything, but I will try to give you the intuition what they are and why they are interesting to cognition. And I'll focus on language, but I can give the same talk in vision. I can give the same talk in memory and the same talk in motor learning. So that also tells us it's a general principle for coding, okay? But we're focusing on language in this talk. So let's see if I can go back. Okay, so first to understand this model, we need to understand what is that vector-based representation embedding. This is BoostServ. For each word, let's start with the simple case and then we're going to complicate it a bit. This is the old model. You can vectorize language. Let's say you have for each word in the lexicon, you have a vector of 50 numbers. Now, suddenly language becomes discrete, becomes continuous multidimensional. There are 50 dimensions for each word. All the words are sitting in 50-dimensional space. We start with random embedding. So we give like random number of 50, 50 numbers, random 50 numbers to each word. When you give random, it means that all the words are going to be orthogonal to each other, more or less, because it's a big parameter space. And then you start to train a neural network. In this case, you have this, you take this like a fixed word, you mask the middle one, and you feed it to a neural network. The neural networks get this like a— for each word you get 50 numbers, so you have a matrix of 50 by 5, and it's missing one, 50 by 6, and you need basically to guess, and you multiply the matrices with random weights, right? So that is random, you get random vector of 50 at the end, and you look, this random 50 is similar to what Of course, because it's random, you're going to give it like random vector, you get like junk, right? You won't get anything. And you say this basically after guessing, I'm guessing a word, predicted word, but I want it to be time because this is the solution is time, this one second of time, the word, right? So we wanted to predict time, but you predict junk. So now I'm changing basically, I have an arrow, I'm propagating back the weights to change the weight for next time to be closer to the raw time. Okay, this is very simple recurrent. You do it for the entire internet. Okay, and then you get a vector for each word that starts with the structures of the language. For example, this is a classical example. If you do king minus man, you take the vector of king, subtract the vector of man, add the vector of woman, you get queen, right? In natural language, it will be like king to man is like woman to queen, right? Because basically you can get other things basically this because we have Marcos, so we have like Rome to Italy is like France to Paris, right? But you also start to see that you also capture some grammatical stuff. Walk to walk, like play-to-play. It starts to learn things about— it's multidimensional, it's not only semantic, okay? We call it like word embedding, but it captures a lot of the dimension in the language. For example, whether you project it to feminine and masculine occupation, you can see the statistics, what is the association of occupation to feminism and masculinity, right? But this is the old model, not a good model. What's not good enough? Because if I'm taking the word in the lexicon, let's say I'm taking like one arbitrary word, cold, I'm getting like one vector for cold, but the word cold is going to change based on context. Remember, context is the key, right? Key. So for example, if I have the context, it is freezing cold today, or if I have you are a cold person, or if I have the 'Cold symptom include.' I'm using the word cold in different meaning, right? As a temperature, symptoms, or personality trait, right? So basically now you want that the word, that the vector cold will be really affected by context. And look, context is even longer than a sentence. I'm giving you the sentence, 'Your cold is ice.' I'm talking about your personality or about your temperature? You don't know, but if you will listen to the conversation, you will know because it's more than one sentence. Context is long. Okay, so what they are doing in the new language model, basically they do it with context. They take It's self-supervised. That's really nice, which means that brains can use this trick as well. You don't need anyone to program it. What you're doing, you're taking the story and divide it into context and then the next word. Okay, now basically you know the answer, right? Rain falls, right? And you know the input, and now you have this like neural networks that's going to predict basically the next word, right? And you try to analyze the network. And we call it contextual embedding. Why it's different than basically the static embedding that we saw before? You start with static embedding, you start with the word vector as before, but now basically you can give up to 1,000 words. So you really give the entire context, but you give a long, long 1,000 words. It's almost like a document. In the new models, it's even 2,000 words. And then you start to multiply this embedding, and at the end you get this contextual embedding. So now this is an embedding that depends on all the words that came before. Okay. And from this embedding, basically now you try to predict All the words in the language. You have, let's say you have 50,000 words in the lexicon, you have 50,000 predictions, and now you can take the— you can order them by the top prediction, you can take the top prediction, and now you can compare it to what came actually, and now you have your error signal again, and you can backpropagate it and then your networks and the entire income. So now you have a vector, contextual vector that depends on the words that came before. It's not static. And this is basically a different solution to that. Basically, instead of basically, let me find out, Once in a while I will seem to include that, maybe part of the talk. Let me— oh, okay. Okay, so instead of learning a word, you learn that in this context, I bought a pair. A pair is like singular, but because it's pants, you use S. This is in this context. In other contexts, you can learn, you know, I bought colorful pencil. You use not the plural pencil. Similar, you can learn that moose, although it's like a plural, it's without S, and elephants will be with S. Okay, you can simply learn the context. You don't try to learn generalized word. You try to learn how to behave in the context. And another thing that is strong about this model is that it becomes generative. After you train the model, you can chop the air and ask it to predict. So you have the sentence, I bought a pair of— the model predicts pens. Then you can plug pens back in and now say predict the next word. And now you can start to generate new text. And you can all type. There's apps in the web that you can talk with these models. So to save time, I gave you an example. We took an article. This is the article. Philosophy could be bought for Biden. Our prediction was wrong. But basically, that was our prediction. And there was an article. We chop the article, we only give the title of the article to GPT-2 and say, this is the input, generate a fake article. So the model generates a fake article. So this is the output and this is what it's saying. Now that you read it, you can see how good is this like model basically look. First, you can see that he knows grammar. He has no mistakes in grammar. So basically he learned grammar. But then you can also see he's speaking on the topic. He's talking about as Democratic leader Nancy Pelosi tried to with moderate Republicans. How do you know that this is working with moderate Republicans? You know it's not, right? So you know about the topic. You also have pragmatic, you know, you fake it like source. You know that it's like this newspaper style, so you have fake of this style of newspaper with quote. If you will plug in scientific abstract, you will talk like a scientist. You know, we do like references, Smith et al., and we fake the references. So you really know they're pragmatic. You have world knowledge, you know who is Nancy Pelosi because you read a lot about her. Common sense, it's not like a perfect, it's like empty sentences. If you read, I cannot guarantee you will pass, but I can guarantee that there will be the vote. It sounds like an empty sentence. Also politicians talk a lot with empty sentences. It might be mimicking the statistics, but this is a good language model. It's not like a better model, no, a lot. Working in real life. This is what we wanted to do with our models. And now people from OpenAI and Google and Facebook coming with these models that in a way we're feeling uncomfortable with it, right? So now we can ask, and this is the question of this lecture. We have two opposing What do I say? It's fully developed. It's basically, we have models that are working. The linguistic, the classical models are working. It's difficult to program them. They're working 50% of the time. They're very clunky. It's very difficult to program all these exceptions and weirdness into them, but we have something that's working 50% of the time. And now we have something that's working even better, but both of them are working models. Okay. So now the question we always ask in science, which model is better related to the brain? And I want to be clear, I'm not a linguistic and I'm not an engineer in Google. I don't care about either. I only care about the brain, what the brain is implementing, okay? And the question that we ask ourselves in science all the time is which model better fit for the brain? So now we have a model comparison problem. And when you frame it that way, you start to ask, how do you start to answer about this like 1.5 billion black box parameters model? Is it really relevant or not? How do you start this like a line of research? So first is, I have to say two things. First is it's a family of models, okay? It's not basically, that there's like one solution to do it. This like a transformer-based model that I gave you, it's one implementation. There are many ways to implement, and there are also many ways to write this like word-based model. So we're asking whether the brain is related to this like family of models or this family of models. It's not that it's transformer. That would be really silly because we know basically that the brain implements it in a very different way. So we are not asking about the actual implementation, we're asking about the computational principle. Why this model is working? What are the computational principles by which it works? Okay. So basically I'm going to present you one and a half work and then some future direction. There is a lot that is going on in the lab. We're writing 4 new papers about it. I won't be able to cover all of them, but it's really rich. This is probably only at the beginning, so I won't give you the final answer. I'll give you how we are going about it. And we're going to talk about self-computational principles between these deep language models and the way our brain processes language. And also, we talk about some differences between the way we process it and implement it then the way these models and how we can make them more psychological-oriented. Okay, so it's going to be like a mix, but I think this is the family of models we should pay attention and don't say they are out of the field, they are not related. Any questions before I'm starting? That was only the introduction. Okay, okay, so let's go to science. We took a story a 30-minute podcast, okay, over there. You— we can feed this like a narrative into the language model. We use GPT-2, okay, and we can get basically the embedding from the model. We can get all the predictions and we can get the error term, okay. So now we know how GPT-2 going to process this story. This is our model process of the story. Now we also can give this story to echolocation so we can record their brain activity and we can see how they process the story. And I will show you basically that similar activity to the code language. Oh yeah, I think that's basically all that's happening. Okay, so basically echolocation Epileptic patients, to locate their epileptic source, they go invasive procedure, they open the skull, they're putting electrodes on the brain for a week trying to see where is the seizure coming from. And sometimes they are willing to do experiments. In our case, it's easy because they think it's not an experiment. They listen to a podcast, enjoy, and you get their brain response. And it's really invasive, so you get really a view into the internal processing of the brain when it's matched to a stimuli. So it's very rare and beautiful data to work with, and we are very thankful. So basically, we're going to show you that the brain uses contextual embedding to represent language. I'm going to show you that the brain predicts for each word Before it's coming, they have a prediction. And then after you're listening, you have an error term. So in a way, basically you can see that the brain implements this objective function also, like deep neural networks. So this is the shared principle I'm going to show you. We have this neural activity, we have the sound wave, we have the word. And now for each word, we can give an embedding. We can give an arbitrary embedding. Like random number of 50 values. We can have the word embedding, the static one that we saw before, or we can give the contextual embedding. We do encoding. How many of you know encoding? I need to explain encoding over there. Encoding, basically, you try to find the mapping between your 50 dimensions and the neural responses to have a model that learns from 80% of the data, the transformation, and then you test it on the news, that 20% that you left out. Okay, so it's like a linear model. Okay, and basically, so we try to predict the neural activity for 20% of the words that we've never seen before, and we compare it to the actual activity, and we know how good our model can predict the brain activity for this particular and remember also this is something that is really important. This is real life in a way, we're asking whether this is a model to study real life responses, right? Because we don't have a way I can know what I represent to the world in the story, right? It's really, it's a very difficult problem we are dealing with. We try to have a model for arbitrary story that we took from NPR. Anyway, it's a very challenging task, right? What we're asking. So basically, if you take an arbitrary embedding, you know what you are searching, we're saying that it will be consistent activity pattern for a word. Let's say you say the word monkey twice, you want to learn from this particular example of monkey, how the pattern will be for the 20 times you use the word monkey. And this model is working, okay? You have some consistent activity for language. But if you use the static embedding that start to know the same dimension, 50 dimensions, but now you rotate it to capture the geometry of language, of the word meaning, like queen to king, you start to get way better model. And if you go into this like new contextual embedding, it's even way better model. So you can see that this model start to capture the way we represent world in natural language. You can see that this is a very strong response. We have some method that has correlation of 0.4, which is like really good prediction, right, of neural activity. So we really get good prediction from these models. Highly significant, but also very good. Next we say, then we look on the brain embedding itself. We have like 4 subjects, we have a very dense echo grid on Broca's area. This is the language area. Now we have 81 electrodes located in language area. This basically 81 electrodes give you 81 dimensions brain embedding. Now for each word, I have a vector of 81 electrodes. I have a brain embedding for each word. And now I also have the language model embedding. I do some PCA to reduce them to lower dimension. Okay, so now basically let's say I have the dimension of the GPT-2 and the dimension of the embedding. I can use 80% of the data to rotate, so I can do alignment. Okay, and now I'm taking new words that never seen and ask, can you know what is the brain embedding from them based on the brain embedding from the language? So I'm asking whether it's really the brain use, whether language that is really used is like brain embedding in the same geometrical space. Okay, because it's like zero-shot, you never seen the word cold, you only, or the word weak, you only turn on the blue ones And now you try to predict completely new words that you've never seen before. So it's really whether you can really generalize to a new set of words you've never seen before. This is called zero-shot in machine learning. And basically— why did I get out? Something went wrong. Oh, I lost my presentation. Slowly. Okay, it's going up.
Stop the screen share and then start.
Yeah, maybe I need to go— I was also in— yeah, okay, I need to present and now it's okay. Okay, if someone doesn't see it from, from the Zoom, let us know. If it's working, it's working. Okay, so basically you can see that we can really do zero-shot predictions for new words that never seen before, and these words are better basically than words that were in the training set. For example, if we had like baboon as a word, it really interprets baboon better than it interprets the monkey. The nearest neighbor during the training. It really managed to predict new words never seen before. Okay, so that's like really strong demonstration that the brain really uses this specific embedding space. And if you go outside of the language areas to a nearby area, you see nothing. It's very specific to the way language areas code this stuff, very specific. Okay. And we see it for the different individual subjects as well. So for each individual subject, we can do the same analysis. It's a very strong design. So we see that the brain represents language using this high-dimensional embedding with very similar geometry to what we see in language. Next, we wanted to see, can they also basically do this at prediction? Like GPT-2. So, you know, for basically in GPT-2, before I'm saying each word, you predict what I'm going to say, and then you listen to me and you get an error term. I mean that all the time GPT-2 is doing prediction in order to learn. This is the trick. The trick is saying that next word prediction is enough to learn everything about language. It's enough to learn the grammar, syntax, the semantic, the pragmatic. It's a very good objective function. But we don't have a proof that the brain is doing this like a prediction in real life. So first we ask, how good are people to predict stuff? So we went to MTurk and we did, and we asked the MTurk here to be like GPT-2. We are like, okay, get this sentence. Can you predict the next word? Type your prediction. And then I'm going to give you the answer and then predict the next word. And then predict the next word. So now we have like 50 people in M30. We get 50 predictions for each word in the story. Okay, we have 5,000 prompts, we have 5,000 predictions. Okay, this is how it looks like. You have some words like the one on top in blue. Everyone knows what this word is coming, all the 50. Okay, you have some words that no one can predict. Okay, and then you have people, what the people predict 80% of the time or 50% of the time. So everything. And now we can ask, are your predictions as subjects are similar to the prediction of GPT-3? Okay, and yes, the correlation was 0.8. So your prediction as humans are very similar to the prediction GPT-3. So we are also very good in predicting real-life context. Next, we ask, and this is basically, you can see it over there, the same. This is the result I showed before. You can see that basically, if you're looking on the brain, you can see that TPT2 is better predicting the brain responses than basically the contextual embedding, the static embedding, or the arbitrary embedding, right? But you already see it over here in the results. You can see, look how much of the prediction we have before the word was saying. All these things happening, this is the moment. This is where the model was from time 0 when you listen to the word. The model is doing well in predicting words before you even listen. So I can really predict from your brain responses to the specific word before you even listen to it. So there is really evidence for prediction in the brain. Then we ask, can we go even closer? So, you know, you have words that you're really good in predicting, you have words that you can really never predict, high-close and low-close. So if we separate it, you can see that the words that you're really good in predicting, it's easier to read it from the brain before than the word that you fail to predict. Okay, so I can really see the quality of your prediction in the brain response. Now, another case study that we can do, and we see it in many areas, okay, another thing we can do basically is think about the error, like how much surprised you are, right? So now over here, in this particular example, our story begins in the deep in depth. Most people, actually, I will start predicting the deep. I'm using the deep over there. Most people think deep ocean, but you hear deep rainforest. So your prediction was slightly lower than expected. So you have some error rate. So I can take the word ocean and use it to predict the brain, or I can take the word rainforest, but you actually So if I'm taking the word what you basically predicted, you predicted 'awesome,' you get these responses. You're thinking this is what basically the model predicted and it's better to explain your responses before. And this is what you actually heard. And if I take in this model, it's better to explain it after you listen to it. So this is also a nice example that when always predict before. And then we can ask, okay, so what about your error, right? You have an error. So you can see the more error you have, the greater activity you have afterward onset. This is the nth parameter. So we can look on the certainty of your prediction before, and this is explained really, or the entropy. This is the surprise theorem, and you can see that the level of certainty of our prediction explained way better before, and the level of error explained way better after. So basically, this is what we see, the resemblance between these like deep language models and the way we communicate language. So now for the rest of my talk, I have a few minutes, I want basically to give you also where it doesn't fit well. So remember, basically our task is to ask which model is better. And remember that model is never the phenomenon itself. For example, is electrical car is a good model for gasoline car? Yes and no, right? It's never going to be simple answer. Basically, is a bird airplane is a good model for our bird to fly? Even less so. So the question is never whether your model is perfect, whether it captures more of the process. That is what we are asking, which model captures better what the brain is doing. So now we can also ask where is the model's failure and how we can improve it to make it— if it's a real framework to how the brain is working, we need also to modify this framework. Be. So for example, GPT-2, basically you have different layers, right? So you start, start implementing, and then you have different layers that compute it, then you get to the last layer to do this like prediction, right? So your model is— but basically you feed all the words at once, right? You feed 1,000 words, you start to multiply over time, over space. But we cannot do it. You have to listen to me word by word by word by word. You don't get it like 1,000 words at the same time. This is weird, right? So we started to look which layer fits better the neural activity in the brain. And what we discovered is really, it's a phenomenon basically. Remember, this is the computation now. That the brain is going layer by layer. Layer, then you multiply with this layer, you take this vector, you multiply it with your nonlinear weight to get this layer, you get this activation, you multiply to get this activation, this activation, this activation. Now we're going into the computation of the model. This is the actual computation the model is doing, and we're using each layer as a model. And you see that the brain goes through the same stage of computation basically, but it's going over time. Okay, so basically you look in vocal area and you can see that the first layer extends activation earlier and then you move to the other layers, you can see really that it really grows over time. You can see a linear increase in the activity. And we see it also in different languages, but we see that the time scale is getting longer and longer, so it gets wider and wider. And this is related to the time scales I was talking about in the previous lecture. So it's really— look at this one, even captures the internal computation of the language system. And that's— but it implemented it in a recurrent way and not in a parallel way. So it's different implementation for the same principle computation. And this starts to be weird, right? Because you get into this like internal model and say, why the way GPT-2 multiplies matrices will be similar to the way Broca's multiplies the brain embedding? It's really, the connection is deeper than we are thinking, but also it's different. So for example, we calculate how much human hours it will take GPT-2 to to train. So we say, let's say you can read 150 words a minute. We calculated how many words GPT-2 is saying. It will take you 120 years of human life without sleeping and eating to train. We look on GPT-3, it will take you 6,000 years of life to read what GPT-3 is reading. So babies learn to speak without reading anything. So this is one example, say, okay, this is not cognitively feasible, right? Another thing is the model can see the sentence, how much time left to this talk, right? But I'm asking how much time? Oh, it's 11:45. So there's grounding, right? There's like multimodal, there's all this like psychological thing. You never learn the meaning of words from other words. You learn the meaning of words, how they are connected, what we are doing, right? So this is also different. So in a way, we have a multi-learning system. We have grounding system. We learn from embodiment from other people. We don't learn from reading the internet. We learn from other people. We are also thinking this model, do not think, they are echoing the statistics. Sometimes I'm thinking that most of us sometimes don't think because they see how much nonsense there are in the world and people say such weird stuff. So maybe we're also echoing the New York Times and Fox News instead of thinking, but we're also thinking sometimes, right? This model never thinks, right? This model doesn't have symbolic representation, they are continuous vectors, there's no symbolic representation, we want it to be slightly symbolic, I guess. You don't want to learn calculus from examples. You want to understand the world. So there is something more to cognition than only these models. So we started to think, how can we make them more psychologically relevant? And this is how we are leaving the engineers in Google and start to develop our own line of research, right? Because they don't care about how humans work, they work about their product. We're saying, can we train language models from human-centric perspective and not from this textual internet? There is this amazing project by Dave Bryan. I'm not sure how many of you know of this project. He wired his house with cameras and monitored his first years of his baby for 2 years. For the entire career, but he did it many years ago. The data is not available for anyone, and not much came out of it because before machine learning, he failed to penetrate the data in a way. So we say, can we replicate this study at scale with new AI tools? So basically, now basically we are wiring 20 families in Princeton for player with camera, okay. Basically, we have basically many cameras and microphones in each room. We record them for 3 years. The data is automatically to the cloud without now human intervention, and basically we are now finishing reading it and automatically analyze and store. This is an example I need to— of one house, okay. I cannot show the identity, so I blur it. But it's really happening now. And in 3 years we will have the data. Okay. And this is an example, for example, this is not a real family, this is during our testing. You can see they are playing with the baby. We detect whether there's a baby, we have a baby classifier. We see that there's a baby. If there is a baby, we process it. We do like automatic segmentation of all the body because we have a body map. Okay, we take all the labeling of the owl. Okay, we're running transcription tool.
Oh, I think she wants to flip.
Here. Oh, she did! For the first time in this movie. And now basically that we have this, like, we're going to have the data. And ask why are we doing that crazy relaxing coding. The point is that we can really now try to train. We won't do thinking and symbolic, but we can really now have all the information to train a language model from the child's perspective. We want to see whether we can make these models really more a model for how people learn. Okay, so this is basically what we're doing now. It's a big project. I'm not sure whether it will fail and we are wasting our time, but we felt that if we really take this seriously as a cognitive framework, we need to push ourselves and not stay with this company that doesn't care about cognition in humans. But the beauty about this model is that they are context-based and they are memory-based in a way. They can really be used by the brain to do cognition. But if this is the case, then we should stop thinking of it as our control experiment and the thing that will give us the answer, because it's a very different answer than the answer we are looking for now in cognitive neuroscience. So I'm not sure whether it will work or not, but that's what we are trying. And I think I will end over here. Yeah, okay.
Okay, some practical issue with the live painting. There's also people on Zoom, so do we— let me check whether there are questions on the chat. Okay, good. Okay, I see hands up.
For the last question, I don't know if you have models in place yet, but would it be some sort of reinforcement learning model that you would be implementing?
Yeah, basically modeling it is going to be very challenging. We will need to develop— the model has to be multimodal. Have to be grounded with the behavior. They have to be learned from the mother or the caregiver that's interacting with the social learning, these moments of learning. For example, GPT-2 already have a lexicon of 50,000 words. The baby have zero lexicon. You also need basically to learn the lexicon first and then start to do this next word prediction trick. There is also stages of learning. And we're really developing now new protocols for learning these models. It's very challenging. We don't really— we're developing ideas, but to tell you that we know how we're going to do it and what will work, it's really early days. So I will say in 5 years I will have maybe models working. That's the timescale that I see for this project.
Hi, thank you. What an exciting topic. Can't wait to see the results in a few years. But I was curious if you know of the developmental psych folks who are doing head-mounting cameras and things.
In Stanford, yeah, of course. Very much fun. Yeah. So my question is about the baby perspective.
Do you think it's important to have the baby perspective for people who follow? Because that's what the baby's actually experiencing.
Yeah, yeah, yeah, absolutely. We are forced to stay with the third view perspective because I cannot have mount cameras for 3 years of the baby. And I want the 3 years, I want the entire statistics. Basically, we have a lot of assumptions about language, you know. Chomsky was saying input is not enough, parent doesn't give you supervision, therefore you have to have innate system. This model says very weak supervision, yes or no? But give a lot of data and examples, you will learn. So we need to collect everything. So mount camera one pole. We hope that this third view will be sufficient. But you're right, it's also, there is a limitation and maybe it would be great if I can record all the sensory system of the baby, what it gets and the camera and the sound from his perspective, that's what I want. —The camera will pick up things the radio won't pick up. Yeah, so it's good for us to know the context so we get like, we get also the parents' perspective, which might be good. I think in the best scenario you have the third view to know the context and have the first view to get the input. That's what you really, that you really want, and we don't get it in this project. Yeah. Just a follow-up, so you could actually estimate what the baby sees? Yeah, we got a few cameras in the room and we want to reconstruct and we know the room's worth for 3 years, so we're going to do some estimation. We did a conversion that we need like 8 cameras to do very precise estimation and we have only 2 cameras, so the estimation won't be as good, but we will try to estimate.
I got a question from the chat. What was the self-supervised learning rule? Was it related to biology?
Oh yeah, that, that— we have a paper that, a theoretical paper that compares basically deep learning and evolution. If you think about evolution, it's optimization problem. There is no God, right? And the eye basically has to come with a solution without understanding the problem of vision or the solution of vision. And this is an optimization that we do across generations. Deep learning is also an optimization problem, right? You have an objective function that it's more than survive, right, in evolution, right? We have an objective function like get it. Think, or predict the next word, whatever your prediction, or recognize a face, and you use it in order to optimize the network. So now the question becomes, which objective functions are really related to psychology? We saw that the next word prediction that is really effective for language modeling can be done by the brain because it's self-supervised. So this is a biologically relevant objective function, but there might be also other objective functions that relate to language. We don't think it's the only one. And if you take it as a family of models, what the line of research that people need to take is what are the objective functions that are biologically relevant for humans and what are not. And if it's not relevant to humans, it should not be used by the brain. But there are many objective functions like mirror neurons, for example, can be really a good objective for learning. I can imitate people as an objective to get ideas what what to do. So there can be many relevant objective functions to biology and psychology that we need to explore. May I continue then? Was that error backprop? Was it still an error backpropagation or is it some other? Because you said it's self-supervised, so that's not error backpropagation, right? Or was it? Don't confuse objective function and and their learning goals are different, basically. What was the learning rule then? Wait, oh, so backpropagation, only to be a bit technical, is the way basically you implement the learning rule, and backpropagation is not biologically feasible. Right. So now, if it's similar to the transformer that it's not biologically feasible as implementation of this like parallelized multiplication factor. There are many— there is a lot of work going on to find local learning rules, because the learning rule in the brain have to be local. We don't think that you project it all the way down. Local learning will be as effective as backpropagation, which is not biologically feasible. And that's why I'm saying it's the same family of models. If you believe in family of models, now you have to find the relevant implementation that the brain use for biological neural networks, different than artificial neural networks. So that's why we're talking about computational principles, and learning rule is a principle. How it's implemented, it has to be discovered. Thank you.
I do have a question for you. One of your recent papers talks about this stuff, making an analogy with the ecology. Psychological psychology of Gibson, which I think is still dominating. Can you elaborate on that?
Yeah, I'm not sure how many of you know, because it basically got forgotten from psychology. I'm not sure whether it's part of your training now, the young generation. Many of you remember Gibson. Basically Gibson was talking about that direct perception. Basically he was saying you fit to the dimension relevant to your action and you learn it Without basically doing this representation computational process. People say it sounds like magic. What do you mean direct? You need to explain the computation. If you think about this, like this model is the model that acts in the world. Basically they have an objective to act, for example, to recognize face or to eat if it's a motor system and to pick up stuff and eat or walk or predict the next word. They are acting or driving a car. So they have embodiment and action-based optimization without understanding the problem, because it's an optimization problem, start learn how to act. And in a way it's really linked to the way Gibson was thinking about the cognitive system as an action system that acts in the world without doing this representation computation that people believe in psychology. So there is a strong link between Gibson and this like deep neural network. There are also differences that we can talk about. Gibson was against memory and these models are all about memory and context. So in a way there are also some differences, but yeah, there is lots of similarities. It's really nice to connect it to a line of psychology that was basically forgotten.
Is there one more?
Oh, this is Peterson. Yeah, yeah, I was just wondering, so like it seems like these models are really valuable in illuminating computational principles. Do you think eventually we'll also gain like neurobiological insight? Like for instance with the recurrent architectures, those kind of seem to be working with the brain nets, but they're not able to achieve the same performance.
Do you think we can take those models and like improve them somehow?
Yes, yes, absolutely. So basically, this is what basically we are doing. If you look over basically this, in 3DM, there is basically, I think it's going to be two ways, line of research basically. If we are thinking that the same family of model, there is basically we can learn from artificial neural network the artificial networks can learn from the biological normal. So it's going to be a two-way line of research. Currently in this particular project, we are taking the language model, the artificial one, and see that they are really valuable. We need to start somewhere and change the paradigm. But you know, we are way more efficient network. You can think about us like version 1000, and Google version is version 1 because we went through evolution to optimize the network. So there is a lot— Google will really want to learn from us how the brain is so efficient in learning, and because we are more efficient learners, we don't need 6,000 years of reading to understand stuff, right? So they can learn from us. For example, the backpropagation stuff doesn't probably we have a better trick to do this, like learning and changing the weights, optimizing the weights. So they can learn from us and we can learn from them. So I think that's going to be back in trouble. It's really, it's in the case that this like neural networks were invented by psychologists, got forgotten, right? And then emerged again in computer science. And now they're going back to psychology. So there's like a feedback loop over there. From the origin. That would be interesting to follow if we are right. If we are wrong, forget about my talk and keep doing what you're doing in the lab. So, you know, I'm not saying that even we are right. If this direction is correct, then that's what—
Yeah, thank you so much for the talk. I really enjoyed your discussion of how narrative context can help to predict what the next word is going to be. Now I was wondering if you could consider the context of the person, like for example, if they're angry or sad or happy and maybe more broadly, does it not expand what they're experiencing and how that explains variability in their narrative interpretation or the next person's interpretation?
Yeah, so it's also related to the topic I gave about who you are and what you bring to the table. Remember, this is a context. If you didn't eat, this is context. If you had a fight with your partner before, this is the context. If you read a paper, it's Socratic. So because it's contextual, you can view all these parameters as additional contextual factors. It's multidimensional, so they can really take these factors in. Currently, in language models, they take the factors by you were saying words that related to your being upset somewhere in like 1,000 words before. But if you will make them more biological, I can also monitor your emotional system. Right, the neuroattentional system is a factor. And but there it's all context, and this is why it's easy basically to give them value and expand them to this like embodiment, because they're already like immersed in the world and contextual. So it's very easy to make them more biologically relevant and psychologically relevant than people think. People think, no, they are black box engineering, they have nothing to do with us. We really have strong aversion for them now in in neuroscience, or maybe I'm wrong and people already like started to like them. I don't know. I cannot read. How many in the audience like this model?
Well, it looks like the younger generation.
Yeah, maybe it was a generational thing because— yeah. I have a real question about the embedding. Yeah. So it's categorical embedding from words to numbers. Yeah. And it's probably only few limited ways you can actually do it. And that's fixed at the beginning. What do you mean limited way? So mapping continuous values to numbers is probably easier than mapping categories to numbers. It's still basically— anyway, this is the thing because we take like a symbolic system and you make it continuous, which is like really counterintuitive to language, right? Because we always think about language as categorical, and this is the trick and the magic and vectorizing language. This is why the word of the initial stuff was like a breakpoint that people said, why you can vectorize language and do operations, simple operations with language? How can it come? And we didn't expect it to be a full-blown language system as we've seen in the new models, like that there are no pragmatic syntax and semantics. So it's really vectorized, multidimensional, it's amazing. And I think why it's working, why you ask yourself, why is that embedding space that GPT-3 is finding similar to my embedding space as a human? It's very weird, right? But once you realize that actually this is a statistical model that fits to how people use language in the world, and this will go against back to Gibson, this is like basically embodiment of language. They fit how people use words and the baby fit to how the people use the words in the language. And this is why we get the same representation of the similar presentation. It's not identical. I still think that there's an objective function to make it more human-like, But this is the trick. So it's really— My question was the process of embedding, the input, and then the final embedding. At the input, say, it's not important. You could actually start with any embedding. Or are we limited by the initial choice? Basically, there are two types of language models. Now we get technical. Some people start with some models start with static word2vec and some let it to learn end-to-end. But even then, the final product, you get static embedding at the beginning and static embedding at the end and contextual embedding one before, layer one before. So they all convert to the same embedding space because they fit to the same statistics. It doesn't matter if you start with random I'm not sure whether I answered, but yeah, and tonight too.
Anyway, uh, wish you luck with that. Thank you very much.
More from Joshua
Scientist Stories: Hans Waterham, The value of phenotypic assays in peroxisomal disorders
Scientist Stories: Joel Zlotogora, Genetic carrier screening programs for reproductive purposes