Nested Learning: Ali Behrouz on the Quest for Continual Learning & Illusion of AI Architectures

Hello and welcome back to The Cognitive Revolution.

Today, I'm excited to share a conversation with Ali Behrouz, grad student at Cornell, researcher at Google, and author of Nested Learning. This episode was recorded a few months back, and while I normally believe that AI content does not age well, this conversation with Ali is an exception. His work is some of the most inspired and potentially transformative that I've seen anywhere in the quest for new machine learning architectures that are capable of genuine continual learning. This, of course, is one of the most important capability advances on the horizon today. Arguably, it is the main gap between today's models and a digital AGI that would be capable of joining and contributing to human teams just as humans do. And Aly is advancing the frontier with an approach that is both biologically inspired and technically elegant. His blockbuster paper, Nested Learning, which has been touted as a harbinger of a possible paradigm shift by no less than Geoff Dean, develops a simple strategy that allows models to rapidly adapt to their current context on an ongoing basis while preserving core knowledge by updating different parts of the system at different frequencies, much like humans manage memory on multiple timescales from working memory to long-term memory. His latest work, Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories, which I actually heard about live for the first time on this recording, and which has now finally become fully public, takes inspiration from how humans consolidate memories and learn from dreams while sleeping, introducing a new offline mode in which models transfer new knowledge from their high-frequency update layers to their more slowly evolving layers via distillation, and also learn new abstractions and connections between concepts by generating and training on synthetic data derived from their recent experiences. In addition to the details of these architectures, which, like so many AI innovations, I find both extremely exciting and a bit scary, we also discuss how scaling for performance may shift from stacking more layers to nesting more frequency update rates. How Aly understands all components of machine learning systems as forms of associative memory that compress a given context flow.

Why this leads him to call deep learning architectures an illusion.

And how he's operationalized this conceptual insight by developing expressive optimizers that learn update rules and are capable of outperforming both Adam and MuON. We also discuss how the attention mechanism can be understood as an infinite-frequency update module, and why Ali expects that attention layers will therefore remain fixtures of AI systems indefinitely. We covered the empirical results showing that Ali's new architectures compete effectively with transformers on standard measures, while also outperforming them on hard tasks such as effectively recalling information from up to 10 million tokens of context and also learning to translate multiple previously unseen languages at the same time. Finally, we discuss why Allie sees continual learning as both an opportunity and a huge risk for privacy and alignment, how human-AI relationships might evolve, and why Allie is cautiously optimistic that models that evolve over time based on our interactions with them could both serve our individual needs more effectively and also lead to a more diverse and hopefully stable AI ecosystem overall. The bottom line for me is that for all the debate and speculation about whether or not current architectures can scale to AGI and beyond, there is a very good chance that conceptual breakthroughs will render that question moot before we even manage to answer it. Transformers have changed the world, clearly, but they aren't the end of history. And as tough as it is to keep up with AI developments, anyone who wants to get a handle on where things are going from here can't afford blind spots when it comes to new research directions like Allie's. And so without further ado, I hope you enjoy this deep dive preview of AI systems that learn on an ongoing basis in increasingly human-like ways. With the brilliant Allie Bayrouse. The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. Over the last few months, I have made tremendous strides with my personal AI infrastructure. Today, I've got high-context instances of both Claude Code and OpenCLAW running on a Mac Mini, and it's amazing what they can do. However, until getting started with Mercury, I didn't have a great way for them to pay for things. I didn't want to give them unrestricted access to my money, but my old bank didn't give me any other options. With Mercury, I can create as many virtual cards as I want, each with its own daily, weekly, or monthly spending limit, and I can lock any card to a single category of purchase or even a single merchant. Now I have a card that my agent can use to buy our family's groceries and only our groceries. And I can create another anytime I want to give an agent a random one-off project that might require making a purchase. This is honestly just the start of Mercury's AI-friendly offerings.

Does your bank offer API keys, an MCP, or a CLI tool?

If not, check out Mercury at mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA. Members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now, on with the show.

Ali Behrouz, author of Nested Learning and the new Language Models Need Sleep: Learning to Self-Modify and Consolidate Memory. Welcome back to the Cognitive Revolution.

Thank you very much for having me. I appreciate it.

I am super excited about this conversation today. I appreciate you for being willing to take some time and come back and do a deeper dive into your work. I think it is super fascinating and genuinely some of the most inspired work that I have seen in recent times. And a big part of your method, as I understand it, is looking at what human cognition consists of and identifying things that we as humans are doing that seem like they're quite important and really critical to our successful function in the world, and then trying to figure out kind of what an AI version of that might look like, and then starting to develop the architectures or system designs, maybe even more abstractly than architectures, that start to make those capabilities possible in AI systems. And it's really striking both like how well some of these ideas have worked and also striking to me how elegant they feel and how kind of right they seem. As I really take time to dig in and understand them. So first question, and, you know, just big picture, how do you think about what it is you are trying to do? Obviously you identify gaps in what current systems can do. How do you think about those gaps? How do you conceive of what it is that you are trying to unlock with the new architectures that you're developing?

Getting inspired from the brain and actually like, what does it mean for me? It probably means different things for different people. For me, I really like to get inspired from brain and generally evolution. And the main reason is that I think it has like a lot of data to train itself in a natural way of training. And so one thing that we can see now is a very like evolved version of a very complicated biological brain. And so generally it's a great source of inspiration. But when I'm saying that, I don't mean that I want to replicate brain and fully do something that that brain does, because most of the time we don't know what that is actually. And for example, I think there are different levels of understanding about how the brain works. So the first one is that we know it works. So that's the first level. The second level of understanding is that You know, there are some modules in the brain, each of them are responsible for different parts. And, you know, we have memory, we have like other things, and that's generally the process. And I think the hard part when we want to get inspired from the brain is that at what level of granularity we want to focus on the brain and get inspired from that. Because if we go too much into the details, like how the brain works and so on and so forth, there are like 2 issues about that. One is we are overfitting ourselves to one specific form of intelligence. And another one is that we don't actually know how the brain does that specific thing. So generally in all of the works that I have done, for example, on Titans and also like this nested learning, one thing that is happening is we can see that the models are facing some challenges in real-world applications. And then the question is, is that a specific challenge is solved by humans and can simply humans do something to address that challenge and overcome it or not? And then the second question is like how they can do that. But, you know, again, for example, in Titan, we discussed surprise metric. We discussed, for example, how the memory should be like Decompose into short-term and long-term memory and so on and so forth. But the point is there's a high chance that brain is not exactly performing gradient descent to, for example, understand the surprise or something like that. That's just a high level of our understanding about how the brain works. I think if we keep that as a source of inspiration, I mean, that level, it would be a great source. But if we go into more details, then potentially we might face some challenges. Specifically because our understanding of the brain is changing over time. And so we might overfit to one specific design choices. So that's generally one thing. But about the nested learning, I think one point that is missing in the current models is about 2 parts. One is about how they can adapt to the environment and the context that they are in. Another point. Point is about how the model can understand new knowledge and incorporate it into their parameters over time so they can make sure that they don't face catastrophic forgetting, which means that, for example, one specific task that they have trained on is forgotten and they don't have any skills in that specific direction. So I think these two are important things about the current models and they are facing a lot of challenges for that because if you have a very like large model, then your model needs to be updated over time and that's generally the main reason you can see that, for example, there is a knowledge cutoff for all of the LLMs that we know of. For example, if you ask ChatGPT about one specific information and then say, you know, you are not allowed to use, for example, any tools for answering this question, then you might see that there's a knowledge cutoff. And so that's a little bit challenging to overcome. And if you just want to keep updating the models, then there are like 2 huge challenges. One is catastrophic forgetting that I mentioned, and another one is the efficiency part because you have a lot of parameters. You cannot keep updating all of the parameters. And so it's a little bit hard. For that. And there are some solutions for that. I think they are also great, but I have some intuitions that why they might not perfectly work for the case of continual learning. For example, one might say that they want to do supervised fine-tuning, for example, SFT, or do some RL stuff or something like that. But the point is still the model can face catastrophic forgetting from one side, And from the other side is that at some point you need to transfer all the knowledge that you have in your context and pass it to the, for example, actual parameters of the model. If you just keep summarizing the tokens, if you just keep the tokens and want to like do everything about the memory and learning process in the token space, then the main issue would be at some point you will past the context length of the LLMs. And so potentially you will face some challenges in that direction. So considering all these things, the main issue with the current LLM paradigm is that they cannot continually learn and obtain new knowledge, new skills over time. And also they are limited in understanding different levels of abstraction of the world. Everything that is done in science is to do something that can explain the world in the most simplest way possible. And that's generally the way we can learn something because we don't want to like keep everything. We want to understand underlying patterns that can describe that specific data, that specific knowledge and so on and so forth. So this compression process and how we can understand different levels of abstraction from the data that we have is something that the current LLMs fall short somehow.

So a couple different angles I want to just probe your intuition a little bit more From on this point. Certainly I have felt, you know, the, the, and I think, you know, most users honestly at this point have felt the problems that you're highlighting. Um, and in some ways I feel like the biggest advantage that I have relative to an AI today is this kind of ongoing coherence, you know, and like reasonably stable identity, right? Like I know who I am in the morning and I kind of know what I was trying to do yesterday and I can mostly pick up where I left off. And, you know, I probably could learn a lot more from the things that I do on a daily basis, but I at least learned some stuff, you know, and kind of, uh, take it on board. And obviously the, the current models don't really do that. And that, that is a big weakness for them. Um, that's why I was so excited about the original Mamba paper when that came out, because it was just like, wow, here's something that seems like it's competitive with transformers. Uh, but it has a fixed size memory space. And, you know, something that like, obviously we can't grow the memory space quadratically to infinity. So we're gonna have to have something, you know, that's bounded in size that can, you know, that can work. And so that was like obviously a notable step. And now you've done some work with the Mamba architecture as well. Titan's even more so, you know, okay, here's another way to think about having a fixed size memory module that in the, in that case, you know, is updated with gradient descent at runtime. Um, so, you know, that potentially seems like it would likely, and it certainly seems like the data supported the notion that that would make it, uh, even more powerful than the Mamba architecture. Um, but, you know, a similar kind of structure of like, okay, a fixed size thing that can like keep what it needs and gradually let go of what it doesn't. That seems, um, it seems super important. I wonder if you come at it Though also from the other angle, like, so, so far we've kind of said like, here's something that it can't do. You know, we can do this. It can't do this. That's one way to think about it. Another way to think about it obviously is like, what do we want our AIs to be like? Um, and so I wonder what your intuition there is. You know, today we have mostly chatbots that, um, need, need something to wake them up, right? They're like, either we have to go send them a message or obviously increasingly we have like cron jobs and, you know, other triggers that kind of get the AI to wake up and do something. Um, but if that, if those things don't happen, they're inert, right? They just kind of sit there until, you know, somebody, uh, calls their number. Do you have a sense or a vision or kind of a, you know, a dream of what your ideal future AI would be like that's different than that? And like, how, you know, does it look more like another person but with AI advantages? Or does it, does it look like— an LLM with weaknesses patched, but I don't know, what is your kind of, when you dream of a 2030 AI that you're working closely with on a daily basis, what do you envision?

Like different aspects to answer this question. From the technical point of view, I think generally the entire literature, most part of the literature, in the past 40 years is built on a paradigm that says we have a pre-training or generally training phase and we have a test phase. But the point is, if we have continual learning, we can see that there are a lot of recent studies about continual learning, how we can do that and how we can overcome about A lot of challenges. But the point is a true continual learner doesn't have a test and train time. So potentially if we hear that name in any like design choices, potentially it means that it's not a true continual learner because there is no test, there is no train time. And so the question is, is it like a uniform process for the model or not? And my personal opinion is that we still need at least 2 phases. And how it works is that we should have one phase that the model is active. So it actively receives information. It's whether through the user query, it's For example, or for example, it might be about vision models, word models, or anything similar. But the point is the model receives some information and generally it performs some computation on the input data and it's active at that point. But on the other hand, there is another phase that the model does not have to wait for the input data. It might not receive any input data. It's completely blocked from the world outside of it. But the question is, even at that time, should the model be static without like any performing computation or doing something, or the model needs to start thinking about some process, thinking about the data that it has, inside its parameters and so on and so forth. So I think we can break the process, as I mentioned, in 2 parts. One is the active phase and another one is another phase. Potentially we can call it like a sleep time because there is no inputs, but still the artificial brain or generally like the model itself is trying to perform some computation. So I think that's a good way of defining different phases in this direction of continual learning. And then I think good model is a model that performs very well in both sides. It should receive the information properly, encode it, process it, and understand it in the best way possible. And on the other hand, when it goes to the sleep time, it should also like start processing what it has learned before and use that for self-improvement. And so that's, I think, an ideal model should do from that technical point of view.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he's currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified 2 great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

But on the other hand, I think there are a lot of challenges. The models that we know right now are very large. So even a simple, for example, academic paper that is like presenting a new LLM or architecture or something like that needs to perform some experiments on models with like billions of parameters, 1 billion, 2 billion, or something like that. And generally that's a very large model, requires a lot of computation if you want to keep that model updated over time. And it needs some techniques to somehow make this process possible. And generally for us, when we were thinking about this direction, it was a time that some ideas about nested learning started. Because generally, if we think about like nested learning, we can see that at each timestamp that we have, we don't have to update everything. We just need to update just a small subset of all parameters. And so that potentially a way to overcome the challenges about the efficiency. So if I Want to like summarize what I wanted to say is that I think an ideal model should have like 2 phases. One, generally it should be a continual learner. It should like interact with the world. And also it should have like 2 phases. One is about like very active process and another one is about self-improvement and how it can consolidate its memory, how it can like understand the knowledge, how it can connect different things that seems to be irrelevant and so on and so forth and everything similar that usually our brain also does when we go to sleep. Yeah, that's just my personal opinion and might be completely wrong, but I think we shouldn't focus too much on what human can do. And instead of that, we need to focus more on what human wants from AI. And I think if we think about, we don't want to create something that is very similar to us. I mean, that's also like very interesting direction, but I personally, I'm not very interested in that. I think we have like great design for human, but on the other hand, for the AI side, I think we need AI models that's are capable of understanding what we want. For example, I think current paradigm of LLMs, you can see that they're great after some time when we have like more features, more, more, for example, ChatGPT has the memory now. And for example, Claude has some like features, Gemini and everything, you know. Now they are more capable of understanding what you want. When you say, for example, write this email for me, they can simply write that for you. And so I think that's a very promising direction in general because we don't want to replicate human. There are a lot of like different ways that we can define intelligence somehow and It doesn't have to be the same thing as human intelligence somehow. So I think here again, we can get inspired from the human, but we need to understand why we want to get inspired. Do we want to get inspired because we want to replicate what human can do, or do we want to get inspired from nature to understand some underlying rules in nature. For example, just one extreme example is we cannot travel through time. So if I come up with one idea that is saying that my AI model is trying to break the causality in the world, then potentially that might be a wrong idea or wrong direction because it is breaking some nature rules about our world. Or again, let me give you another example. For example, when we are talking about the title that we use, LLM Needs Sleep, it doesn't mean that LLM needs to literally go to sleep and rest. It means that, you know, from the human brain, it seems that there is a very general rule that it has 2 phases of learning and then another phase of processing it, consolidate the memory and find underlying patterns between the received data. So that's a very high level of inspiration from the brain. So yeah, in short, I think that we don't want to replicate what humans can do and we don't want to have human intelligence, but on the other hand, we want to have a new form of intelligence that is very, you know, it's very proper and it's in the, you know, it's designed in a good way that understands human needs and so it can help people to, you know, do a lot of things that they might face some challenges.

Yeah, certainly they're already superhuman in some ways. And so the opportunity for them to balance out our weaknesses is incredible. One phrase that you said there that I wanted to kind of latch onto is multiple ways to define intelligence. And maybe I'll just give you kind of my high-level pitch for like what nested learning is. I think in the paper there is a big emphasis. You guys place a big emphasis on kind of showing certain equivalences where you're like, the way that we're doing things today is sort of a special case of a more general framework that you're developing. The core idea, as I see it in the nested learning paradigm, and what I, what I think is like potentially for a simple person like myself, like most exciting about it, is for quite some time now, right? We have achieved the greater and greater expressivity of models by stacking more and more layers and just making them bigger. And that has worked like remarkably well. We've been able to push that paradigm incredibly far. It's like kind of crazy though, right? That we just have this like one layer stacked over and over and over again in you know, whatever, 80 or 120 layers deep or however many layers. And that's kind of it. Like, that feels like the more mature solution should be somehow more elaborate than that, right? And so, but that's been the way that we've achieved this expressivity, or sometimes the term computational depth is thrown around. And what nested learning is doing is bringing a different way to the table to achieve higher levels of expressivity or higher levels of computational depth. And that is by stacking not layers, but levels. And what differentiates a level from a layer is that a layer is like the same thing, or they can alternate. Obviously we have these sort of interleaved architectures too, but these are things that are sort of in sequence as information passes from one layer to the next. But kind of a forward pass is like pass through all the layers one by one and get to the end. And that's kind of the thing. What the levels paradigm brings to it that's different is that different levels can have different update frequencies. And with that, you now have the possibility for some parts of the overall system to be much more durable and some parts to be updating like much more radically in something much closer to real time. And that obviously feels like just much more aligned to like what we are, right? We're not like one static thing that processes information in a fully dependent way each time we have, you know, we're very much, our state in any given time is very much contingent on what we just experienced, but only to a degree, right? Like, we're, you know, my mood or what is currently on my mind is a reflection of what happened earlier today, but my, like, big picture views about the world, you know, they didn't change from this morning until now. So there's clearly some sort of hierarchy of different kinds of beliefs, different kinds of representations, different kinds of circuits that we have. which are updated in some cases very quickly, in other cases very slowly. And obviously they are integrated together and work together. And we just haven't seen that in machine learning, except maybe in a few very far-flung experimental cases. And now you're really starting to show that with this nested learning paradigm, not only can you make it work, but as we'll get into with results, you can make it work in a way that is competitive with transformers and even seems to have some of these new, I sometimes call them micro skill advantages, where you can see with these certain early diagnostics that like, oh, this can do something that's like qualitatively different than what a transformer can do, even as it also like outperforms it a bit in terms of like general perplexity type scoring. So how would you react to that kind of general, you know, high-level summary? And then I also really would be interested to get your take on what is this concept of computational depth or expressivity? I'm tempted in some ways to make an analogy to just like the G factor. You know, people talk about, obviously the G in AGI is the generality. There's also G in the context of like human IQ, which is like the G factor, the sort of intangible something that's like how, you know, how capable are you across like a very wide range of things? Again, that's like just getting at generality. Maybe in machine learning, it's as simple as being like G is sort of loss or, you know, there's maybe some fundamental equivalence there, but maybe not. I don't really know. So I'm very interested in how you think about what that— clearly we're getting at something and we've seen like huge progress, but what is that something is another thing I really would love to get your intuition on.

We were working on, on denoising learning for a very, very long time, potentially. I mean, even like more than 1 year and a half or something. The main issue that I personally had, and we discussed a lot in the group, a couple of the authors of the work was it was really, really hard to formalize what we wanted to deliver because even in the mathematical formulation, it was really hard to write it in a formal way and say what we want to exactly do in this paradigm. And so after some back and forth discussions, I think We come up with this specific framework saying that there is a frequency of updates to somehow give time to each module to do something when it's waiting. I think that's the best way of describing that, why we need to have multiple frequencies. And on the other hand, we need to have a knowledge transfer. a method for knowledge transfer to somehow transfer the knowledge from one level to another level. So when the slow network is waiting for all the computation and information processing of the fast network, then it should be some advantages for the slow network that we are paying all these costs for, for like doing some computation. By the fast network. And that advantage comes when we have a best or, you know, a good way of knowledge transfer between the fast network and the slow one, which is like lower level or to the higher. So the idea there is that when we have a network that is like updating very fast, Then I can use that fast computation to give something to a slow computation side. So the slow network or the slower level can somehow focus on the more high-level knowledge abstraction of the data, and then more frequently updated network can somehow focus on the fast adoption and how they can like process in a high resolution data, process the high resolution data. So generally that was the way that we could somehow like describe this framework and saying that I think these 2 sides needs to be there to complement each other. One is the frequency of update and another one is the knowledge transfer between levels. And so from— so for the second question, I think, in my opinion, the current models are very efficient from one specific point of view. Why they are efficient? Because you can see that from what we can gets from LLMs, they are very cheap. And for example, you can, you know, if you want to match their power in some of the tasks with human, then potentially the cost would be very, very different. So from that perspective, we can say that computation and generally like the current LLM paradigms are, is very like efficient and we can perform more computation per each parameters or artificial neuron that we have into our model. And it can help us for different things. One is that it can help us to have more, we can have a smarter model. It's a very subjective term to describe this, but when we have more computation, it seems that we are performing some internal thinking. So a simple LLM, when there is a— let's say that I have a simple LLM structure based on transformers. When the token comes, I will do some computation on the token, pass it through all the layers, and then predict the next token. And that's how it works. But now let's assume that for this specific token, instead of just a simple path of computation, I also perform more internal computation with respect to the past data or generally like, you know, combine or mix the data and something like that. In that case, I can see that the quality can potentially go up, the quality of next token prediction for this specific design choice. Can goes up. And why is that? Because it can be interpreted as a form of internal thinking process. So now for each specific parameter that I have in my model, it is performing several computations. So it's not just one parameter, one type of computation. It's one parameter, couple of steps of computation. So that's one advantage. Another one is about memory perspective and adoption of these models. So when we have a model that is like adopting to the context very fast, then potentially that model can, you know, learn in context. So that's somehow one of the main messages that we try to deliver in the nested learning. Which was everything that we know of somehow is a form of in-context learning. So generally, like, I think it's a great thing in human language that we create new words, but on the other hand, if we just create a lot of words for the same concept, it can just make us confused or it can help mis— it can be misleading somehow. So I think we should like create new words to differentiate different concepts, but if we have one specific concept, we need to like stick to one specific word that we have for that concept to avoid misleading processor or anything like that. So from that perspective, we realize that we can say everything is just a form of in-context learning. So we already know that what is in-context learning. Now we just need to understand how what we have been already doing is a form of in-context learning. And so that was the part we started to like showing that, for example, backpropagation is a form of in-context learning, is a form of associative memory. And when it's a form of associative memory, then we can say that The general pre-training phase of the model is a form of in-context learning. Or when we go to the, for example, context of attention or RNNs, so on and so forth. Again, it's a form of in-context learning. So when we perform gradients, which we can define any RNN based on gradient descent or other form of optimization process, when we can do that, it means that we are doing some learning. on the context that is happening right now. So I think these 2 are the main things that nested learning is trying to address. One is about generally more computation per neuron, and another one is about adaptability and the continual learning side.

Can you just describe in more specific detail, like, what are the relative sizes of the levels? What are the structures of the levels? What are the context windows or lengths of the different levels? What are the frequencies? Just like map the thing out for us in kind of very black and white terms.

Yeah.

So let me start from the transformer structure. In transformer, we have attention block and then MLP block. So what is happening there is that in the pre-training, we have different contexts that attention is attending to. The attention side is trying to combine all the tokens and each token attends to all the tokens before that in the context and so on and so forth. And then there is an MLP block and that MLP block is responsible for long-term memory. Now, when the model is pre-trained, then the MLP block is fixed. It's not changing anymore. So it has all the information compressed during the pre-training. And then we have attention. So at inference time, attention is responsible for the context that it is getting. And MLP block is responsible for the long-term memory and generally the very general knowledge of the world. Now let's just simply extend this idea. A simple extension of this idea is that we can simply keep attention. And then instead of just one MLP block, we have multiple MLP blocks. Each of them are updated with different frequency. So now what is going on there is that why it's helpful is that when you have attention, you have a fast adoption to the context. Attention is very powerful. It's like a perfect memory, cache everything. And so it's great. On the other hand, you might want to have multiple levels of memory. And that's the part we define Continuum Memory System. So instead of one block of MLP, we have multiple blocks of MLP. And now you have your first MLP block. It is updated very fast. So what would happen in that case? In that case, the fast updating process of this MLP block can cause catastrophic forgetting because this MLP block can simply forget the information that it gets, for example, a couple of thousands tokens ago, for example. But the point is, since the other MLP blocks have not updated so far, the knowledge that is forgotten by the first MLP block is still in their parameters. So when we perform backpropagation through all these layers, then the knowledge can come back. So it provides and it helps us to have a loop processing time. The model can, the first MLP block can forget something, But the point is, if that specific data sample or that specific skill that is forgotten is important, then this can come back to the other MLP blocks, which has not been updated so far. And they have still, they have the knowledge about that specific skill or data sample. So that's a very simple way of extending whole transformer block. And so we call this variant Hop Attention. You know, it's a combination of attention plus multiple MLP blocks, and that's what we called Hop Attention. Now we have another variant, which is the actual Hop Architecture. What we are saying is that attention, as I mentioned, is a perfect memory. It's, you know, it can cache everything, it can scale, and it's fast and so on and so forth. It's great. But the point is still the update of attention has infinite frequency. What does it mean? It means that attention doesn't know anything about the temporal dependency of all the tokens. And it needs something like positional encoding, or even with the help of positional encoding, attention is not a great model for the tasks that are sequential, tasks that require sequential reasoning or something like that. What was our idea? Our idea was to replace attention with another associative memory that maps keys to values. So that's what attention does. And now we want to replace it with another module that tries to map keys to values. And so one potential architecture here is Titan. So we can simply just replace Titan and have Titan plus Continuum Memory System. And so that's a simple idea. But the point is, you know, in initial sections of the paper, we discussed that if you have a simple linear process of updates, which is what is happening inside each chunk of Titan update, this process can somehow be weaker than the case when we have self-referential process. So what is self-referential process? Gradient descent or generally backpropagation is a form of self-referential process. So what is the idea in the self-referential process? The idea there is that We want to learn how to learn and how to learn and how to learn and how. There are a lot of levels of how to learn, how to learn. And so computationally, it's infeasible to implement all those levels of how to learn, how to learn, and so on and so forth. So there is one idea by Schwethauber et al. And so basically, They have this idea of self-referential model and how, for example, one of the ways that we can make a model self-referential when we have a key-value memory is the case that the model generates its own value. So what is happening there? Let's say that in our memory, we want to memorize something and, you know, there is one specific event happening. And you want to memorize it. Let's say you want to memorize one specific word. And now in our brain, we have associative memory and we are trying to map this specific word to another concept that we already knew so we can memorize it. And you definitely have seen some cases that, for example, you say, I generate this specific word or I want to like map this specific word to another word that I already knew so I can remember that. We generate the value that we want to map our keys to it so we can, you know, memorize the key as well. So for the self-referential process, it's a very general concept, but in this specific design choice that we have in the paper, the value of the associative memory is generated by its own parameters. So the model itself generates its own value and then tries to map keys to values. So potentially this process is fully sequential. You cannot parallelize it. In a simple format. And so it has a full understanding of the causality in our data. So potentially in the task that requires sequential thinking, sequential reasoning, or anything like that, we can expect that this model should work better than simple attention because simple attention doesn't have the ability to sequentially understand the causality of the data at all. And so that was our idea. In HOP architecture, we replace Titan with self-modifying Titan, which is exactly a Titan module where it generates its own value function, value of the associative memory. And so that's HOP architecture. It's self-modifying Titan plus Continuum Memory System.

I think I want to spend one more beat on what you mean when you say generating its own value, because I'm kind of like, okay, I know the transformer architecture pretty well. We've got these like K, Q, and V vectors, right? And the training process Modifies all of those over time. And so the, you know, the sort of general heuristic that I have is like, for each token, there's the query vector that sort of indicates like what this token is looking for. There's the key vector that sort of helps indicate like what the, what other tokens have to offer and sort of aligns those, right? And finds where there's relevant, where there's a match, basically, where there's relevance. And then the third one, the value, sort of brings up the concepts that then get fed on into the downstream layers that, you know, make sure we have like the right activations for continued processing from there. But all of those are, are learned, right? All K, Q, and V are learned. So I'm not entirely clear on what you mean when you say that the model learns its own. values? Because like, doesn't the transformer sort of learn its own value vector as well? I'm not quite clear on the distinction that you're making there between what, you know, well, let's assume people are at least generally familiar with the transformer and know how that goes.

So generally in transformer, or more accurately softmax attention, what is happening there is that we have a projection of QKV. And then the output goes to attention. So attention doesn't have any control on the QKV projections. But when I'm saying that the model or generally the associative memory tries to generate its own value and then map keys to the value, I mean something like gradient descent. So if it If we just recall the gradient descent, we can see that we have something like the previous state of the W, which is W_t minus the gradient of the loss function that we have. So this is equal to the next state of the weights that we have. So if we look at this process, you can break the gradients using the chain rule and write it into the gradient with respect to the output times the input data. And now you can see that this is getting a form of associative memory, very similar to linear attention, because it's W_ equals to previous state of the W minus k, and k here is xt, and then v, which is the gradient with respect to the output. So if you look at this process, it's very similar to linear attention, but the— or any linear recurrent model. But the very interesting part is it is different from linear attention because the point is, if you look at the value, which is the gradient with respect to the output. This gradient with respect to the output is a function of wt, is a function of the current state of the weights. So basically the value, the keys and values that we have in associative memory, the value component is not from another component before this recurrent formula, But it is generating by this recurrence, recurrent process every time. So that's what's— what is going on with this like self-referential process. So in memory, let's say that in, in a very simple version of like self-modifying Titan, if we want to design this as a simple Titan, what is happening is that I have this x. And then QKV projection. I project it into QKV and then pass all of them to the Titan module. So that's a simple Python module. But if I want to have a self-modifying Titan, then all of these parameters of the projection of Q, K, and V are optimized inside the Titan module. So basically the model has the control of somehow modifying its own updating rule and somehow generate its own value for the memory. That's the main difference of these 2.

Yeah, I think the key phrase maybe for people to latch onto there, starting with myself, is modifying its own update rule. And this has definitely been a theme in general Like with the Mamba architecture as well, you know, the authors of the Mamba architecture had done a bunch of previous state-space model work. And the big unlock at the, with Mamba specifically was that the way in which the state is going to be updated at each time step now became a function of inputs. And the, so that increased expressivity, you know, potential for expressivity of making the, the actual update of the state itself dependent on the input that it's receiving at that time unlocked a lot better performance. And there's something very similar going on here, it seems like, where you're saying we want to make the final value output, not some, we don't want to, we don't want to calculate the value vector too early. Basically, we want that to come a little bit later and be more input dependent, more history dependent than it has traditionally been in the softmax attention. Is that a good intuition or is there something still missing from that intuition?

I think it's a perfect intuition. Generally, this projection of the value is also updating inside the module. That's a very important point because it helps for the adaptability. Generally, the model itself is also very adaptive to the context. And so from every token that comes, the model is trying to learn something. And so the way that it generates the value function The value term for the associative memory is exactly the same as the way it updates its memory. So it's a very adaptive process to also generate the value.

So can you take us through kind of a single time step? And maybe we can do this with the AttentionHOPE and then the Titan's HOPE. Um, and just to like highlight the little difference there, but let's also zoom out to the big picture. So we've got now kind of a new fundamental block, right? If we do the attention hope thing first, we've got an attention mechanism and then we've got multiple MLPs arranged in sequence from fastest update to slowest update frequency. And then that block gets stacked into layers, correct? I'm a little bit confused, to be honest, on these sort of, there's no, training test distinction, because there is still some training process where you're like just taking a bunch of data and like running it through the thing, right? So from the sort of researcher's perspective, maybe from the model's perspective, there's not so much of a distinction, but from the researcher perspective, you are still sitting there like running a process that takes a bunch of data and like has the model learn from that, which is kind of, you know, a bulk process that's not like a user is engaging it. There's nothing like outside of that process happening at that time, right? So what, how is it different? You know, when I, when I do this for a transformer, I can like, um, I do have these, some of these really nice parallelization benefits. Uh, I do, I am interested to come back to understand like to what degree the sort of current hardware paradigm plays nicely with some of the stuff you've got here and to what degree the, you know, the fundamental recurrence may present challenges, but bracket that for a second. Today I can like, run a bunch of tokens through the thing in parallel. We can accumulate, you know, we can do this for batches. We can accumulate all these gradients and then we sort of apply the gradients and then we have, you know, the next timestamp and we kind of keep doing that. And I have a pretty good intuition for like how information flows. I kind of, you know, I can visualize the forward pass in my mind and then I can visualize the backward pass of backpropagation going through and, you know, gradually updating all the Wait, how does the procedure with the new architecture vary? What are the core things that are different from the paradigm that we're more used to?

I think generally, like, the main difference comes from the update side that I mentioned. So for the copay attention, I think it's very similar to the current paradigm. And it's the art, even the architecture is very similar to transformers. It's actual transformer architecture. We just replace the MLP block with multiple MLP blocks. And so I think when we want to do inference, then the main difference comes from the fact that for each of the MLP blocks, we need to track where we are. Is it the time that we want to update the MLP block or it still has time to get updated? If it's the later case, then we use the last updated state of that specific MLP for doing the inference. If it has not, I mean, if it's the time to get updated, then we first update it through all the backpropagation stuff and all of the tokens that we have seen so far in the current chunk. And then when the weight is updated, then we perform the inference. From the research point of view, it might be a little bit hard to remove this part. I think even from the research point of view, it might be better to say that we have evaluation time and not evaluation time because it seems that we are always like update— when the model is always updating.

Yeah.

gets updated over time. There is no training time and test time, but the point is it seems that for one specific period of time, we don't do some evaluation and we wait for some time. And after that, we start evaluating the model on the different downstream tasks that we have generally, like for anything else. I think from the model perspective, it doesn't know whether it is in the test time or train time because everything is the same and it's a very uniform process. But from our side, definitely it is important whether we want to evaluate the model and measure its accuracy for a specific task and so on and so forth or not. And so yeah, Coming back to the HOPE architecture, that's generally for the HOPE Transformer or HOPE attention model that I mentioned. But when we go to the actual HOPE architecture, then again, everything is the same. Everything is very similar. The only difference is that the attention is replaced by self-modifying Titan. And for self-modifying Titan, again, everything is very similar to Titan. So all of the changes are just inside the model design and from the higher-level perspective, the inference is very similar that, you know, the context or document goes to the self-modifying Titan and then for each token we have one output and then it goes to the MLP blocks. We have multiple MLP blocks and so on and so forth. So everything is very similar to the current paradigm.

Did I have it right that there are like this core block of either the traditional attention or the self-modifying Titan module plus the MLPs that then becomes the block that gets stacked into layers? Is that right?

Yes.

That's a design choice. We can have like different design choices. For example, when we— so generally the initial and the main design of HOPE is in the case that we have, for example, for the HOPE attention, we have attention, and then multiple MLP blocks. Each of them are updated with different frequency. But for some of the tasks, we needed to use pre-trained model. And for example, if we want to focus on, for example, LLaMA, then LLaMA is not designed with HoVe architecture. So what we can do, the thing that we have done is that instead of like going in this formulation that I mentioned, for example, attention, then multiple MLP blocks. What we have done is that we say this is attention and MLP block, then attention and another MLP block with different frequency, and then attention and another different— sorry, another MLP block with different frequency and so on and so forth. So it's somehow design choice. You need to see like which one do you prefer? Do you want to use existing pre-trained model or do you want to like start from scratch and train your own design and architecture? But I think potentially both of them are relatively similar. They don't fundamentally make changes.

Yeah.

Interesting.

It's another, I mean, Great, uh, reminder of the old Ilya maxim that the models just want to learn. There's a lot of, um, it's always striking to me how many of these choices end up kind of, yeah, it could kind of go one way or the other. I mean, that was true in the Titan's case where you had like 3 different ways of working the memory module into the larger architecture. It's definitely been true with Mamba in many ways where, you know, you could have multiple states, you can have them in sequence, you can have them in parallel, you can, you know, Once you have one of these block concepts that seems to work well, you can kind of Lego piece it in a lot of different ways. And yes, there will probably be some performance differences between different ways to arrange the blocks, but more often than not, it seems like if you're really talking about a, a serious conceptual advance, you find that like, that's less important. The exact wiring diagram is less important. And more important is the core piece that you're adding to the set of Lego pieces, so to speak, that you can use. And so, yeah, this is a good reminder of that. How do you think about the relationship between the different MLPs in terms of size, in terms of update frequency, maybe in terms of like learning rate? I, I, because it feels like there might be some Feels like learning rate might be kind of important here where there's like a potentially an equivalence. If I update one MLP every token and then I update another one with a larger batch size, I can, I feel like I could make those like much more similar or quite a bit different depending on like what learning rate I apply. But yeah, so size frequency, like intangible terms, are the small, are the ones that are updated less frequently also smaller? And do they have different learning rates than the other ones? Take us through that kind of— how do you think about the relationships between the MLPs of different frequencies?

Yeah, I think it really depends on the architecture and the number of parameters and the design choices that you have. It is really similar. We cannot say that the best— what is the best dimension for transformers or attention block. Potentially, it's really hard. It really depends on the person that wants to work with that attention and the use cases that we want to consider and so on and so forth. So generally the updates, the frequency of updates really depends on how adaptive you want your model to be. And for example, how do you want the model to maintain its persistent memory and so on and so forth. So I think that's pretty much a design choice. But about like learning rates, I think generally you can treat each of these blocks as the same way as you do about MLP blocks. They're exactly the same thing. But the point is they have like different frequency of update. So nothing has changed. So everything is exactly the same thing as MLP blocks. And so... I don't think that, you know, it definitely would be very interesting to see how the learning rate can affect each of these blocks and so on and so forth. But I have not done that and I'm not sure about the exact solution. But my expectation is that potentially any way that we use to do hyperparameter tuning, we can do the same thing here as well. So there shouldn't be any differences.

Interesting. And so your general default is kind of an intuition-based approach. And do I have it right that it's basically kind of order of magnitude? Like the fastest one updates every token, the next one updates every 10 tokens, the next one every 100, or maybe it's like a couple of orders of magnitude. Like how, how did you, understanding that it's not yet a, um, fully tuned system, What did you start with and why did you pick those things as your kind of initial guess?

Yeah, and the way that we choose the frequency of update for each of them was based on our intuition of the chunk size that we use for Titans and other models, because generally the chunk size in Titan is the part that we can define the frequency of the Titan as well. At that time, we didn't have this term of frequency, but generally, you know, that chunk size can define frequency for Titan as well. So what we used was based on our intuition about what chunk sizes are good for Titans. And so, you know, I think we didn't have like, you know, something like, As far as I remember, I think the numbers that we used was possibly 128, then 4 times 128, and then 4 times 4 times 128. So it was something like that as far as I remember.

And how about knowledge transfer? How should we think about the way in which— and I guess one other quick interjection question there too. There are still like skip connections Here's a slide.

Everything is similar. Yes, exactly. One, you know, one thing that we try to do in lesson learning, I mean, unfortunately it caused some misunderstanding about what we are doing and, you know, I have received, I have seen some comments about, for example, some of the concepts here are already new and something like that, but the point is we try to actually include all those concepts that we already knew to show that It is a universal learning paradigm. It's not something that contradicts with our current understanding. It just complements and completes what we already know about, like in a new direction. For example, you can, when you are doing deep learning, any form of deep learning, and you are saying that I'm using this attention here, you are actually using Nested learning. But in deep learning, you only see the final solution of each learning problem. So you have a learning problem inside the attention and you are trying to solve a regression problem. And the nonparametric solution to that regression problem is attention. When you see everything from deep learning side, you can only see the final solution for each component. But you, when you see everything from the nested learning side, you can see the internal learning process of each component as well. So in general, it's not something that is, that contradicts what we already know, but it somehow complements all of the things that we knew and go beyond that. I think that's generally an important part. So yeah, everything can be pretty similar. You can have its completion.

Yeah.

So then help us understand how we should think about the roles that the different frequency MLPs are playing. Knowledge transfer is one way to think about that. Or another way to frame the question might be, how do they complement one another? How do they work together? How does the one that And I would be interested to understand this both intuitively and mechanistically, to the degree that you have mechanistic understanding. But how does the one that's updating fast gradually inform the ones that are updating slow? How do the ones that are updating slow steer the ones that are updating fast in the right directions? How do you think about the interplay between those different components?

Yeah, I think generally knowledge Coming up with different ways of knowledge transfer is very important here, in my opinion. So the main point of the frequency, having frequency for each component is 2 parts. I think we also like discussed it in the beginning. And the first part is that it helps the model to maintain its memory for the longer time period. For example, you know, that's just a simple example. Let's say that we have a twin and then one of them goes to a spaceship and, you know, just move with the speed of light. And then right before that, they have a very good memory. For example, they went to, you know, did they have lunch or something like that? I don't know. They have very good memory. And then the person that is like moving with the speed of light, when they come back, 80 years has passed in like on Earth. And then their sibling somehow forgets about that specific launch they had because it was 80 years ago. But on the other hand, that specific person has all the information, all, you know, all of the details are just One seconds ago, two seconds ago, and so they they remember everything about that lunch. So why it's happening? Because of the updates in the memory of each of them. The person that lived eighty years, their memory got updated a lot of times. But the person that lived just move with the speed of light or close to it, now their memory didn't get updated that much. So from this perspective, we can see that the number of times that we make some updates to the memory is very important. And it's a very inaccurate example that I mentioned, but I think the main point is clear here. When we have like 2 components, one of them is updated a lot of times while the other one is a slower one and is not updated that much. What is happening there is that the slow one has the opportunity to learn something from the fast one because at the time that the fast one is— gets— is got updated, The slow one has not got any update. And so now there's a chance that the fast one learns something to the slow one before it gets updated. So I think that's the part we can also like discuss this sleep process that we have. So what is the idea there? The idea there is that when we have multiple levels of MLP blocks, Each of them are updated with different frequency. One simple thing is that before updating the fast MLP block, and by fast, I mean, you know, it's a relative term. So there is a fast and slow while we have like multiple levels. So for each 2 blocks, consecutive blocks that we consider, when we want to update the fast one, we know that there is a chance that we forget something. So before forgetting something, we can transfer the knowledge of this block to the next one and then update this one. And that's the part we need a good way of knowledge transfer. So for example, one way is to, you know, do context distillation. So if you want to pass the knowledge from one ML block to another one, some methods of context distillation can work very well in that case. And that's very similar to what we do in the SLEE process as well, in the SLEE paper. And so yeah, I think the main role of the knowledge transfer is, you know, help the slow network to take advantage of the fast network. And another point about having different frequency is about the memory and how the model can manage its memory, similar to the example of the tunings that I mentioned.

What is the mechanism by which the information in the fast update layer gets moved to the slower layer? But then there's also got to be something going the other way too, right? If you conceive of them as like pure perception in a sense, Then I guess I'm not even entirely clear on what's happening in my own brain, but certainly like there's more information flow. It feels to me like from my sort of perception modules to my, you know, higher order reasoning modules, whatever, than there is from the, the reasoning back to the perception. That, that signal, but it is, that signal is still important, right? Like my, my, um, higher order processes do. Tell my eyes where to look and do tell them like where to focus and do say, you know, hey, we need to like zero in on this, um, this detail a little bit. Like I wanna understand that better. So like, go put some of your bandwidth into this particular thing. So yeah, I guess give me a little more on how the mechanistically or, or procedurally how the information in the fastest update layers is getting transferred and, and how we're making sure that we're storing what really matters. From what the fast updates have learned, but then also what is the signal that flows the other way?

So let me answer that with a very simple example. Let's say that I have this model and I want— I have model A and I want to update its fast MLP blocks. And I want to make sure that the information in the fast MLP block is not forgotten and it can pass to the slow MLP block. One simple thing that I can do is to just copy all of the parameters of model A to model B. Now I have 2 identical models. One is model A and another one is model B. What I do for model B is that I update the fast network, the fast MLP block. Now the parameters in the fast MLP blocks of Model B are just free. And now what I want to do is that I want to change the parameters of the slow MLP in Model B in a way that the output of parameter— sorry, Model B can mimic the output of Model A. If that happens, It means that, what is the difference of model A and B? Model A has all the information compressed in the fast MLP block, while all those information are gone in model B. Now, if I could somehow modify model B in a way that it can mimic model A, it means that I somehow transferred the knowledge in the fast MLP to the parameters of the SLO MLP in model B. So that's just one simple thing. And this process is very similar to the distillation process. We are distilling the knowledge of model A to the knowledge of model B. So from this perspective, we can see that these two, somehow this is one way of knowledge transfer. From slow ML— sorry, fast MLP to the slow MLP. That's, for example, one example. Another example, which is very common and popular, is backpropagation. So if you just sequentially connect your MLP blocks and then at some point perform backpropagation, then you can transfer the knowledge of one block to another one and so on and so forth.

So the Sort of copying and distillation process you described, that's essentially what's going on in the language models need sleep paper?

Yes, with some additional detail. For example, what we do there is that we also like additional parameters to the model B as well to make sure that it has enough capacity to store the new knowledge that it has gotten.

Does that mean also that in the, in the nested learning version of this, you're really just letting backpropagation do its thing and you're not really like, you're not, you haven't really over-engineered, you haven't engineered it all that much. We just have these MLP blocks, they get updated at different frequencies and you're just kind of letting gradient descent do its thing and the updates are just kind of working. That's basically it.

Yeah.

Yes, exactly. Yes. In the whole architecture, everything is just like backpropagation.

A huge takeaway from the conversation that doesn't, that didn't come through to me as clearly in the paper is just like, this is really proof of concept stage stuff. And the fact that it works so well shows like what a good concept it is. But nothing that we're discussing here is really, you know, has been through the same kind of thing that the mainline models have been through where, you know, everything has been sort of parameter, hyperparameter explored to the nth degree and optimized and, you know, all the little refinements that have been obviously made over time. That hasn't really happened here. So there's a lot of questions that we still could ask about like this version, that version, this configuration, this arrangement, sequence, parallel, you know, how many layers, relative sizes, relative learning rates. There's like a ton of space there still to explore. But basically just taking a few of these core concepts, the main one being the different frequency of updates for different MLP blocks, that alone creates some pretty impressive results that are like qualitatively different from what we are used to seeing. So maybe let's take a minute and just talk about some of the results. I mean, the, you know, there's a lot of different tests run in the paper and, you know, big tables of results with a whole bunch of different metrics, some of which are kind of your perplexity Scoring, classic type of stuff. What do you think are like the most important revealing results that you think people should say, ah, because I see that it can do that, I know that there's like really something here that I need to grapple with? Yeah.

And we have one continual learning style task that I personally really like. So the idea there is that we have a pre-trained model and there isn't one specific language that the model has not seen before and so we want to learn that specific language in context to the model. We want to learn the model that specific language in context and then The point is we have all the grammars, we have all the words, we have— there's a dictionary of words and so we pass all of them through the model in context and then the model learns the language and then we ask that, can you translate this specific text to English from that language to English? And then we can see that the model perfectly, not perfectly, but in a very, very, very good quality can translate that specific text. So it seems that the model is capable of understanding that language in context and then use that for some translation task. But the point is, let's just go one step beyond that. Instead of one language, let's put 2 languages in context and then ask the model to translate different texts from each of these languages to English. In that case, we can see that the model almost collapses and cannot translate any of those languages. The point is the model cannot handle its context well and fully understand each of the languages separately. And that's generally a very, very hard challenge for transformer-based architectures. But the point is when we change that architecture to HOPE or HOPE-Attention, again, we have attention, but we have multiple levels of in-context learning, multiple levels of MLP blocks. And so one thing that we can see is that when we increase the number of levels, the performance of the model in both of these languages gets better and better and better. Why it's happening? Because the model has better way of memory management because it understands that, for example, temporal knowledge that are not very needed, can be stored in the first MLP block and more understanding of the language can pass to the, you know, more stable MLP blocks, later MLP blocks that are more stable. And then when we have like more and more blocks, we can see that the performance gets better and better. So in my opinion, that's a very like good evaluation for understanding whether the model can learn in context and generally do continual learning.

Yep.

I think that's—

So this is, is this the same? I remember it's been a while since I thought about this, but I think it was with maybe Gemini 2, maybe, I don't even know, but it was even as far back as Gemini 1. There was this metric introduced of learning a new language from basically one book. There's like, it was like there's some critically endangered language. One person has, you know, really studied it and made like a book that's not on the internet anywhere that sort of explains what this language is. And then they just put that book into context and say, okay, based on this, you know, go ahead and do translation. It seems like this is, I don't know if this is the exact same test as that one that I was previously familiar with, or if it's a bit different, but it seems the language here is Manchu. I just looked it up. It's like a critically endangered language from somewhere in China. So that's basically the idea, right? It's a language that the language models have basically no prior knowledge of. They're given a sort of very detailed primer on this language from some, uh, anthropologist or whatever who's gone out and done the fieldwork. And then their job is to apply that. And so I'm looking at fig— figure 8 in the nested learning paper. And what I'm taking from this is all of the models do kind of similarly if there's just one language. But as you said, when you go up to 2 languages and, and sort of double the difficulty of the task, then the in-context learning traditional transformer approach performs quite badly. And then I understand HOPE-1, HOPE-2, HOPE-3. Are those like how many levels, like how many different frequency update things exist. So when you move from traditional to, I guess, 1 additional, 2 additional, 3 additional frequencies of update, you get basically almost all the way back to the original level of performance with just 1 language.

Yes, exactly.

Yeah.

Okay. That is That is quite interesting.

Yeah.

Fascinating stuff. And what is MT-OB there? Just so I have that clear. MT-OB, that's a, that's the other—

Yes. That's another dataset. It's another language that, that is also has not been seen during the pre-training of the model.

Yeah. Okay. This is the one that I recall. Yeah. So you guys added the Manchu language to this one from, yeah, late 2023. So both of these like very rare unknown languages being translated to English and only with the multiple layers can the models do both at the same time in, in one context.

Yeah.

Very, very, very interesting indeed. How do you think about kind of the— I really like that answer. That's a great intuition builder. How do you think about just kind of things like perplexity scores? I mean, you've got a big table that shows on things like perplexity and sort of some accuracy stuff on some of these like basic kind of classic battery of tests. And I should say, we're scaling these models so far up to roughly the 1 billion parameter scale. You've got 760 million parameters. And 30 billion tokens. And then the bigger is 1.3 billion parameters and 100 billion tokens. So obviously that's like not huge by today's standards, but nevertheless, you know, there's a pretty clear signal that the HOPE architecture is on just about every dimension outperforming all the other things you're comparing it against, which includes your transformer and your Mamba and Mamba variations and even Titans. RetNet is in there. DeltaNet is in there. How much, how do you interpret these? You know, this kind of goes back to that G question. Is this like, is this a good measure? Is it just the best measure we have? What do you think about like how much stock people should put in these like perplexity tables?

You know, there are some standards in the community for like performing some benchmark tasks and not all of them are the best things to do for evaluating the model, but we just need to do them to make sure that Everyone is— everyone can see where the performance comes from, the advantages comes from. And so I think I have one specific transformer structure and I want to like, I have one idea that if I add, for example, forgetting to the transformer, then it can perform well on noisy data, for example. I'm not sure. I'm just like— Coming up with just one example. And so in that case, if there is no noisy data and I just test my method on very clean data, then there is no way that I can show the advantages of my approach in that case. So I think here is exactly the same thing. We are arguing about models that do not need to be pre-trained. There is no test time, there is no train time and so on and so forth. But on the other hand, still a lot of infrastructure built on like test and train time. A lot of evaluations are built on test and train time. And so everyone also expect us to report something about pre-training perplexity and some evaluations that most of them are some short-term and short-context language modeling task, and they do not need to have a very complicated model to understand long context modeling. So I think I also like mentioned that in the presentation of nested learning at NeurIPS, we didn't like use Table 2 and all those perplexity and language modeling tasks to argue that HOPE is powerful. We just use that table to say that HOPE is not less powerful as a backbone compared to other models. And you can see that like it performs well, but someone might say that it's marginal compared to other like models. But the point is, this is not the direction that we aim to solve. And somehow it's really good that we can see that even in this direction, that is not the goal of the nested learning and HOPE, we can show some improvement, even if it's marginal.

Yeah, gotcha.

I'm always a big fan of trying to get a little bit better sense for the micro skills of different architectures. So for example, of course, you know, transformers, because the full sequence, the full context is in working memory at all times, it's pretty hard to beat. And I feel like you even sort of have like kind of a theoretical argument now that like it may even be kind of impossible to beat in some, in some of these tasks where the idea is like recall from the context window. But then, you know, we saw things with Mamba, for example, where it was better at learning from like sparse signal. And this was sort of a micro skill that it, that architecture excelled at that the transformer relatively struggled with. What, what, what have you seen in the hope case? You know, are there little micro, and I, I think it's very interesting because it does kind of ladder up to the overall performance, you know, and what these things are actually good or bad at. Right. I mean, the ability to recall something in, in context is really important when you need it. The ability to learn from or kind of filter out noise and get to the, you know, the signal that really matters is really important when you need it. So are there particular micro skills that that stand out to you? I mean, the language translation one is an interesting one in a macro sense of like, that's a hard task. But I wonder if you drill down to these like very micro building block competencies that models or architectures can either have or not have, what stands out in terms of what this has that transformers don't have or don't have as strongly?

When we are talking about in-context recall tasks or generally like recall-intensive tasks, In my opinion, all those tasks are designed for transformers. They are not designed to compare architectures, but they are specifically designed for transformers. Why I'm saying that? Because you cannot expect from a model or even a human to perform Needle in a Haystack perfectly. Or for example, do some recall-intensive task. For example, assume that you have a code and like a couple of thousand lines of code, and then simply you want to recall what was the value of x at some line of the code. And so it's almost impossible, or at least it's very, very hard for a human Or even like for other models to do that. And, but on the other hand, it's, it's pretty much simple for transformers because they have direct access to the entire history in their context. And so it's very simple to just like, you know, find that token and pass it as the output and in fact find it somehow. In recall-intensive tasks like this in-context recall task that we have here, Actually, the gap between recurrent architectures, which actually their performance is also very great if we compare the first generation of recurrent architectures to the transformer, we can see that this gap was much, much larger. Now this gap is getting smaller and the performance of other recurrent models is also very great. But the interesting part for me was that Hope at least closed this Performance, these close this gap in the performance of the model compared to transformers while they are not expected to do that. Transformers is, we expect transformer to do that because it has attention block, but we don't expect a compression-based model to perform recall task. And so I think somehow it was interesting for me.

And what does the MAD dataset get at? How should we understand, like, because that's, that's where, to just replay back to what you just said, the, on these like needle in a haystack, these like very difficult recall tasks from earlier in context, the transformer remains the best. The recurrent models, which only have some latent representation and, and don't have the ability to look back at the original raw text. don't perform as well, but with each generation of improvement, and here you've got several, the HOPE architecture does the best of, of the recurrent ones that don't have the, the full explicit context in working memory at runtime. And so that gap is closing. Flipping over to the MADD dataset, here the HOPE architecture is performing better than everything, including the transformer. What, like, micro skills is that testing? Which, what should we take away from that result?

The MATH dataset also like, it's very similar to the recall-intensive tasks. But the point here is that, you know, there are different setups for it. For example, in one of them is the noisy in-context recall. We want to perform recall and in-context recall task. But the point is we have some noise in the tokens. And now when we have that noise in the tokens, Somehow the power of transformers that I explained in the previous setup, which is, which was pure in-context learning, now is its weakness somehow, because it can get simply confused about which token is noise, which token is not, and so on and so forth. So potentially this task becomes a little bit, for example, harder for transformer compared to a model like HOPE. But again, that is also very— that also depends on the memory management of the RNN. So again, for the RNN, if it doesn't have a very good memory management system or generally update mechanism, then potentially it can simply get confused by the noise as well and face some issues. But again, if the memory management is strong, then it's much simpler to filter all those noise tokens in the task. And so, yeah, I think that's, for example, one thing. Another task that is also like, I think it's interesting here is about compression. So the compression task here, somehow, you know, the name explains the task itself, but We want to compress the tokens and predict one single token that is the compressed version of a set of tokens. And so then, you know, we see that and then we want to like reconstruct the original sequence from that. And so potentially it's a simpler task for models like RNN because they already knew how to compress the data properly. But on the other hand, transformer has a harder time to perform this task. So generally, like, as I mentioned, all of these tasks, I don't want to like go into all of the details, but generally all these tasks are somehow modified version of recall-intensive tasks or in-context recall or something like that. But the point is There are other aspects. For example, selective copying is another one. There are other aspects to the model that those aspects also are very important. And we should also like see how the model performs in those aspects and not just, you know, overfit our evaluation on one specific metric.

Cool.

I think that's probably enough on the really low-level stuff. And I think this illusion of architecture title of the paper starts to click for me. On page 39 of this paper, we get to the part where you also have a new optimizer that is outperforming not just your old Adam standard, but also even outperforming MuON. It does come with a little bit of computational overhead, but I think again, the argument is that it more than pays back for itself in terms of faster convergence or just better learning. Is there anything you want to add on the M3 optimizer, as you call it?

Let me first also like clarify one point that generally for optimizers, it's a little bit like hard to say that, for example, this specific optimizer is more powerful than the other one. It really depends on the problem setup or generally even the problem. So for example, we might, here we are like evaluating the optimizer on, for example, vision task, but on the other hand, if you train a language model, you might see that the trend is completely different or something like that. So generally, like the design of optimizer and saying that like which one is better than the other one really depends on the task architecture, problem setup, and all these things. And somehow that's also one of the main points that we wanted to deliver in nested learning because What we are saying is that the entire architecture with its optimization process are just one interconnected system of nested optimization problem. And this is interconnected. Why it's interconnected? Because the gradients of the optimization side is generated by the architecture. If you have a simple architecture, then the gradients are very simple. If you have a complicated architecture, the patterns in the gradients can be very complicated. And then when you have momentum term, momentum is a form of associative memory that is trying to compress gradients. So for example, if your gradients are very complicated, you need more powerful memory management system for your momentum. Or if it's very simple, architecture, then even a simple gradient descent without any momentum might work very well. So in general, one of the arguments that we have in the paper is that we should see everything as an interconnected system and tries to design something that altogether results in a good model architecture or generally like a machine learning model in a very general term. So that's one argument. Another thing is that we wanted to deliver this message that architecture side is very, very, very similar or somehow exactly the same as optimization side. All of them are just some learning rule and there are some learning process that is happening. And the only difference between the architecture side and the optimization side is just the context. The context of the optimization algorithm is gradients. Actually, the context is the set of gradients that we have. And the context of the architecture side is a set of tokens that we have. So generally they're very similar. So in the paper, We had this continuum memory system. We extend the MLP block saying that you can have multiple levels of frequency for the MLP block. And, you know, that's a very general term. And in the entire paper, we are arguing that architectures are the same as optimizers and so on and so forth. So why not applying that technique and somehow borrow that technique from architecture side and apply it to the optimization side. So that was the main motivation to show that this continuum memory system that we have designed is not just working well for architecture, but it also works very well on the optimization side as well. And so we just simply extend ME-WAN. Instead of one specific MLP— sorry, one specific memory, it has multiple memory. In the case of M3, it has like 2 memory. And so it's trying to, you know, compress the context with different frequency rates. And so it can help it to better understand the global aspects of the loss landscape, and it potentially can help the model to find more effective solution.

The new paper, Language models need sleep. Tell us a little bit more about like what's going on here. The, you mentioned kind of at the top, the 2-phase concept. We have the memory consolidation phase and then we have the dreaming phase. I do think it's fascinating to consider that there's like kind of creation of net new parameter space and then consolidation or sort of pruning back. I understand too, of, of, because obviously the things can't just grow and grow forever, right? But yeah, take us through this in more detail. I'm fascinated to learn more about it.

Generally, the main idea, as we discussed earlier, was that if we have a truly continual learner model, then there is no test and train time. On the other hand, we need to have one active time where the inputs is coming in an online manner, and also the time that is The time we don't have any input. So the model is not actively receiving information from the outside, but it doesn't mean that the model should be static. It means that the model just doesn't get input, but it can have some internal computation to improve itself. And so that's a very, very general concept. We can incorporate more and more components to the sleep time. that we have. So it doesn't have to be just these 2 specific parts, but these 2 were really relevant to the research that I'm doing. So we just like did that, but potentially it can include any other form of self-improvement and so on and so forth. So that's just one way of breaking the life of a continual learner into active time and time. But what we have in the sleep time right now, which again, as I mentioned, we can incorporate more components to it, is that we want to make sure that when we update each of the components of the model, we don't forget about the knowledge that is stored in their parameters. And so what is the idea there is that What we do is we know that there are multiple MLP blocks. Each of them are updated with different frequencies. And then again, the fast and slow here is just a relative term. It doesn't mean that the slowest or the fastest one. So we have a slow and fast weight. It can be any part of the neural network. And we want to transfer the knowledge from one to the other. So in order to do that, what we do is a distillation process that I mentioned, but the distillation process is based on on-policy distillation. So the model itself generates some data. And so one interpretation of this process is that we distill the knowledge of one small model to a larger model. When we want to go from one step to the next one, we activate new parameters in the next level. So it can help the model to release some of its capacity and be ready to accept new knowledge. So somehow it is, somehow it describes a very natural way of learning in humans as well. And for example, it's very common that when we learn something, when a new, when we learn about a new concept or something like that, we don't have a full understanding of all aspects of that. But the point is, when time passes and, you know, we let our brain to better understand that concept over time, and also like, you know, we study other things and better understand the entire process, then at some point we can see that we have a very clear picture of what's going on in that concept. We can completely understand it. And so generally that's the best thing. That's a very good way of learning. So here is exactly this same, very similar process, very, very similar process. So also like discuss that from this perspective. I think it might be better and more simple way to understand why we need like have multiple levels and distill the knowledge from each level to another one. So when we want to understand a specific concept, we have different levels of knowledge abstraction for ourselves. The first level, which is the most simplest one, is to just memorize things. So let's say that we want to learn one specific mathematical rule, or for example, a specific concept in physics or any science. How we can learn that? We start with some example of that specific concept, and then we start memorizing those concepts. For example, if it's a mathematical rule saying that, you know, just any mathematical rule that we can have, we start with some specific examples and just memorize them. And then at some point we just neuralize our understanding of all those examples, remove all those examples in our brain, and replace all of those memories with just one single memory that can describe everything that we have learned so far from that concept. And then when time passes, we have more information, we read more about that concept, and so on and so forth. Again, we revisit our understanding of the concept and then replace our previous understanding with this new understanding, which is more general, and it can explain more phenomena or more terms in that specific concept. So that's generally the way we understand things, and there are different levels of abstraction in our understanding. So now when we have different MLP blocks, or generally, let's just go to any arbitrary architecture. There's— there don't— That it doesn't have to be just HOPE, it can be any architecture. But the main thing is that each of them are, each of the blocks are updated with different frequency. In that case, the fast updating block is very similar to the memorization process because we memorize a lot of things. We don't need to understand that. There's no like pure understanding of that concept.

It's just memorization.

And we can also like forget very fast something that we have memorized. So the first level is that. So the first block is responsible for that part. But if we want to better understand that concept, we need to do some memory consolidation. So what is happening in our design is that we transfer the knowledge from fast updating module to the other one. But if we just simply pass the knowledge from fast updating block to the slow updating block, then nothing has changed. We just like transferred the knowledge without doing anything. But instead of just simple transfer, we replace that with a distillation process. Why distillation here is important? Because the previous block or generally the fast updating block has compressed the concept and somehow understand it or memorize it in any way. It's just a compression process. When we do a distillation, then there is another levels of compression that force the model to You don't have all those parameters anymore. You now have less number of parameters to store that specific knowledge. And in order to do that, you need to come up with something that is more general and can understand underlying patterns in the data in a better way. So you can store everything in just a smaller number of parameters. So in that case, the model would come up with better levels of knowledge abstraction because we have forced it to do it. And then again, we just repeat this process so on and so forth. That's generally the main idea of memory consolidation. And like every time that this sleep process happens, we consolidate the knowledge from one level to the other one and so on and so forth. And so that's a very high-level idea of what's happening in the memory consolidation. Another part is about dreaming. So why we need to have this dreaming process? The main thing is, I think there are like 2 important points when we want to implement this dreaming process. The first part is we need to have a self-improvement process. So we have learned something so far. Actually, the memory consolidation part can also be seen as a form of self-improvement. But if we have one specific task at hand, if we want to specifically optimize the model for one task, then this is the place that we can do it. We can self-modify the model and fine-tune it or generally use RL to update the models and self-modify it so it can be more powerful in one specific task and so on and so forth. That's just one advantage of dreaming. Another advantage of dreaming is that in the dreaming process, we need to understand the connection of concepts that seems to be irrelevant But they are actually relevant. That's also what is happening in the dreaming process of human. We can see that we can see very weird dreams because the brain is trying to understand the connection of very irrelevant concepts and see whether there is an underlying pattern in that. So here in the dreaming process, we need to also have that as well. And understand different aspects of how we need to combine different knowledge stored in different components of the model. So that's another goal of the dreaming. And, you know, we can just combine these 2 into the sleep process and the model. After one step of sleep, the model has consolidated its own memory. And also on the other hand, there's a self-improving process.

So in the sleeping process, I'm seeing that there is, there are new parameters created as to create space in the slower frequency updated portions to absorb the information from the faster frequency ones. Do, does that ever shrink back down? Do you, is there a, is there a pruning or is there a, um, a sort of other side of that that balances that? Or at this stage, do these models just grow indefinitely throughout their life?

From the technical point of view, we cannot like grow the model to like arbitrarily large number of parameters. But the point here is that it's like a periodic process. We add some parameters and then we free them. For the next step of consolidation, when we are in the first block, we add some components. When it reaches its capacity, it means that it is the time that we need to consolidate the memory to the next step. And when we consolidate all this knowledge to the next step, we just remove all the extra capacity that we have added to this. level and free them for the like other levels, like faster levels. So they can also consolidate their memory to this block as well. So generally it's like a periodic process. We add components and remove them, add components and remove them.

I see.

Gotcha. Okay. Interesting. And what more can you tell us? about the dreaming phase in terms of like just a little bit more practically what's going on there? I mean, it's certainly when I try to introspect into dreams, it, um, I think that hasn't been super, I mean, maybe it's been somewhat fruitful, but I think also people get very confused when they try to interpret dreams or understand, you know, what's going on there. So I won't even attempt to, you know, ground my understanding in my human dreams, which seem like quite a, um, Quite a hard thing to untangle, I guess. But here there's a more, I mean, you got to design the process. So like what procedurally, what is going on in dreaming? Just a little bit more like mechanically procedurally.

The concept of dreaming doesn't mean that it's exactly the same thing as dreaming in human. It's just, you know, at a very high level, they seem to be very similar. And so that's one point. And another point is that Again, the concept of sleep and dreaming for a language model might be very different from the concept of dreaming and sleep for the, for example, vision model, because potentially a vision model will generate some images, generative, like a vision model might generate some images during dreaming, while in this case of language modeling, we are generating text. But the framework is very general. It can adapt to any data modality. And so that's very general. But the point is here, when we are like doing that for language modeling, what is happening there is we generate some context, we generate some text. And how do we generate those texts? It's on-policy distillation, the same way that we discussed earlier. We have a model, we copy that, you know, we want to distill the knowledge from one level to the other one. And so we free the parameters of the slower level and so on and so forth. And so we ask the smaller one, the small model, which has the knowledge of the context as well into its parameters. We ask that to generate some text and then we want to train or somehow update. It's a better term to use, update the actual model parameters on this specific dataset that is generated by the model. And then How do we train it? We start with one part of the sequence, just sample some of the tokens, and then ask the model to predict the next tokens in that sequence. So that, for example, that's really similar to generating some synthetic data. It seems if the model can perfectly predict the future tokens, it means that it has already knew about the knowledge that is stored in the previous block. So it's a perfect model, but if it cannot properly, you know, predict the continuation of the sequence, it means that it doesn't have the knowledge that is stored in the context and needs to update itself to understand that knowledge as well. So it's somehow a form of on-policy That is happening inside the model. And so it, as I mentioned, again, just as a summary, we have 2 phases. One is the generation, which generates some text about the knowledge in the context. And then the second part is on-policy distillation that we distill the knowledge from one level to the other.

Yeah.

Uh, that's what's happening in the dreaming phase. I mean, we also have the self-modifying part as well, but I think that's, that's the main idea of memory consolidation and how, how did dreaming happen here.

So what, what is the upshot of this? It seems like the few-shot abstract reasoning result is like the main thing that Again, shows like a qualitative difference between this approach and other things. I understand that this is kind of an ARC-like task where basically the challenge is you have a few examples of some transformation and your job is to learn the rule so that you can then apply the rule to a new example.

Any evaluation that we have used for like HOP architecture in nested learning paper, Potentially can be done here as well. So the goal is exactly the same thing. At the end of the day, the model needs to continually learn new knowledge and new, you know, learn about new tasks, learn about new skills and so on and so forth. So in some sense, the goal is very similar, but I think the Setup of the problem is the part that is different from this paper and nested learning. In nested learning, we are talking about the active phase of the model, but here we are talking about the sleep time of the model. So that's generally the main difference, but all of the evaluations can be done and we can, we can see that everything is the same.

Cool.

So let's zoom out then and do just a little bit of like, where does this leave us? I guess going back to the top, you know, you, we were kind of talking a little bit at the beginning around like, what do we want from language models? And, you know, today, man, they're getting awfully good, but we do still have a bunch of, you know, I certainly have like learned a bunch of habits over time for how to use them where I kind of implicitly am building my practices around some of their limitations, you know, so as to play to their strengths and, you know, not get stuck in their weaknesses. How do you, as this paradigm begins to mature and we get like more continual learning, what do you think the experience starts to look like when it comes to things like, what does it mean to start a new chat, you know, and should, what sort of relationship do you think people will have with these? systems. I can imagine people might have really long-running relationships that— I mean, we talk about LLM psychosis now, that could get really strange and even more— I mean, the relationship could be even more compelling and the problem in some ways could be exacerbated by the fact that the models are better. On the flip side, sometimes I might still want to start fresh, right? Because I might think, geez, all the stuff I've done with this model in this one direction, probably isn't going to help me over here. So maybe I do want to start over in some cases. And then there's just like the question of model upgrade cycles themselves and sort of how do we run evaluations. Today we have like Anthropic putting out 100-page reports on every new major model release. And so that paradigm of like, you know, we're going to really take our time to understand these artifacts as deeply as we can, which I wish other Companies like DeepMind doing a pretty good job of that, OpenAI doing a pretty good job of that, some other leading developers not doing much of it at all. I see a lot of virtue in doing all that work, but then I try to port that onto this paradigm and I'm like, well, geez, you can't like run your full eval suite every single timestamp. So like, how do you think about like what constitutes a version and, you know, when would I change a version? It seems like the, um, A lot of the sort of rhythms of both use and versioning and deployment and releases, all these things could really be complicated in a paradigm of really powerful continual learning. So how do you imagine some of that stuff shaking out?

I think one simple case is that definitely the model gets better and better and better in understanding what user wants and also adapting themselves to their, like, their style. And, you know, for example, this person, when asked about one specific concept, might not expect the same thing when another person asks the same question. So the model needs to really understand how It needs to answer one specific question, but for different people. I think that definitely gets better and better if we could come up with a continual learner. And on the other hand, you know, we have seen that when we can increase the context window of the model, their performance in everything that we know, ranging from, you know, all the coding tasks or for example, for example, OLLIE, you know, mathematical reasoning, generally reasoning task, or for example, all of the benchmarks that usually are used for evaluation of the model, all of them get much better in all those benchmarks. And continual learning can somehow be seen as a form of like enhancing the long-context understanding of the model. While the, you know, I should like emphasize that the concept of long context understanding or generally long context, the term of long context is very different from the term of continual learning, but continual learning is a superclass of long context. And so potentially if we could come up with a continual learner then it also has more ability in long context understanding and potentially better performance in all of the benchmarks and evaluation that we are aware of today. So yeah, I think that's just one thing that I expect from continual learners.

Do you worry about things like alignment drift or value drift? I mean, I was I always say the last and least valuable co-author of the emergent misalignment paper that came out about a year ago. And there's been a lot of variations on that since, but the kind of big takeaway, the big theme, you know, that I think we should all, we would all do well to remember is changes to, I don't want to say one area, but sort of Changes made to a neural network with one particular purpose or one particular dataset can have like very strange and surprising knock-on effects in behaviors that at first glance would seem like very far afield. So the emergent misalignment one, for anybody who hasn't heard that, is like, if you try to train a model to output insecure code, you know, that is code that would be easy to hack, then, and the same thing is true for like bad medical advice. If you fine-tune a model to give bad medical advice, What you surprisingly find is that the model kind of turns evil in general. And it seems like, to the best of my understanding, the way that this is happening is like, to learn for a model that already has all this knowledge and already has this sophisticated understanding of the world, for it to go up, for it to go into the detailed understanding of its medical world model and make a ton of little changes. to reconfigure it so that it has like all these wrong ideas, that's like hard. Whereas there are features like give bad advice or be generally evil that it can learn to turn up in general that when propagated through even the existing medical world model yield the bad advice or, you know, when propagated through the existing coding model yield insecure code. So it's sort of a shortcut solution. We thought we were just training the model to do a certain relatively narrowly scoped behavior, but what we found is we actually kind of changed its character and the interaction of that character change with existing knowledge created the behavior change. But now we've got this character change that, you know, can interact with all these other domains of knowledge and do all kinds of insane stuff. And that's why, you know, all of a sudden we've got a model that wants to have Hitler over for dinner and we're like, you know, wait a second, how did that happen? We were just talking about code here. So now again, you know, I'm like, man, there's something so exciting about all this stuff that you have conceived of here, but We, it seems like it really breaks again a lot of our paradigms for how do we know what we're going to get? You know, like if, if I'm literally modifying this thing on an ongoing basis, we're going to need some sort of new ways to kind of make sure that like in other areas, it's not like going off the rails and, you know, potentially causing me, you know, very painful downstream surprises. Do you have any thoughts on how we can begin to get a handle on that problem?

Honestly, I don't have a very concrete idea about like how it can be solved, but in general, I wanted to add that in my opinion, the concept of continual learning and seeing that from the generally privacy alignment and, you know, this direction is both an opportunity and a huge Threat, I think, because—

Like in what way?

I mean, it's a huge like danger for privacy. And from one side, the model is continually learning, so it can simply get all the information about you and so use that. And it's very concerning. I mean, at least it seems to be very concerning. But on the other hand, if the model is designed properly, and so, you know, it's designed properly, then it can use that information to align itself with your value, with everything that you want. And so I think generally these 2 directions of continual learning and privacy. Potentially are orthogonal because all of the concerns in a static model still can happen in continual learner. And so everything is possible. But on the other hand, there are some new challenges, definitely, as you mentioned. But on the other hand, also there is a huge opportunity to, you know, The model, if the model is designed properly, then it can, you know, adopt itself to the value of, you know, to your values, to anything that you want and something like that. So I think generally it's both an opportunity and then a very constant—

How do you imagine that learning from the user's values or just feedback in general working? In practice? Like, do you, I, there's been a bunch of, obviously there's a bunch of different techniques on this. I mean, one obvious answer would be like thumbs up, thumbs down, collect sort of feedback collection, you know, could be used to train more of the, you know, get more of the good, less of the bad, whatever. You can also, there's a bunch of different schemes around like translating natural language feedback into Updates for the model. Is that kind of what you imagine? Like people basically being able to just give feedback to their own model verbally and then have a mechanism for taking that on board? Because it's not just a next token prediction task at that point, right? It's like we need, it doesn't need to be able to predict what my, I guess if it could predict what my feedback is, it would presumably be better at being aligned to what I wanted in the first place. But it is not the case that like its core task is predicting my feedback. Like it's ideally going to do its initial task well enough that I don't have to give it the feedback in the first place. So do you have kind of a vision or an imagination of what that, how does the user close the loop in the continual learning paradigm?

The initial step can potentially be This human-in-the-loop process that, for example, the model can learn using reinforcement learning from the feedback that it gets from the humans and also like tries to align itself to the values and, you know, be a safer model. But on the other hand, I think it would be just a starting point. Definitely at some point we need to update the model in a proper way because let me just explain it that way. I think again, this process is very similar to the form of nested learning that I mentioned. We need to like transfer the knowledge through the slower levels and I think here is exactly the same thing. The model might start with learning from human feedback. But on the other hand, it can transfer that knowledge into more persistent components of that model to make sure that it's not going away from that specific value that it needs to be aligned with. So yeah, I think there's a huge room there to improve the model from the from safety perspective and also align them with the human value. But I think it's definitely like a huge room there because more and more people are getting— they're realizing that it's a very like important direction. And so I think over time it gets more and more and more methods, effective methods that can help the model to be— aligned with the human value and also be very safe. But I think—

I guess the hope, the hope would be that in the same way that it can do a better job of solving arc-like puzzles because it has this sort of strength in abstracting away from details and figuring out what really matters in a given context, that it would be able to do something similar. It would be able to like dream about my feedback and become more, more deeply, more robustly aligned with what I'm trying to communicate to it based on really getting to, you know, the core abstractions that are driving whatever it is I'm saying. Yeah. Boy, there's so many aspects to this. What do you think about— this kind of goes back to Titans a little bit. I don't know if there's like a surprise term. I didn't catch mention of surprise. In these more recent papers. But it does seem like in general, in a continual learning context, there's going to be a really interesting challenge of how do you manage life in an adversarial environment? If you are too quick to believe some— and I've seen this failure mode in Claude at times, although it seems like they've maybe corrected the other way, because in the last few days we've seen the emerging genre of Claude refusing to believe current events, you know, being like, the Department of War, that's ridiculous. Like, don't say that. You'll lose all credibility in a Washington audience by calling it the Department of War. Or, you know, that the whole Venezuela thing, it's just like refusing to believe that such a thing happened when the user tells it that. So it seems like maybe they've, they've kind of, again, fixed, but it's a very tricky balance to strike, right? How does one, especially if you're locked in a server with limited access to the outside world, like, How does one determine what new information, you know, what new tokens constitute good information, what constitutes bad information? You certainly don't want to just believe everything that you are given and, you know, start doing radical updates, especially if these are going to be durable long-term updates. But you also need to learn continually. I don't know if there's something in the current work that kind of addresses that, or where my head goes is sort of some sort of like, maybe the dreaming can kind of get at this, I guess, you know, consistency checking, like, does this make sense with other things? Like, if I believed this, you know, what else would I have to believe? Or, you know, what other, or would this like invalidate any core beliefs that I, you know, am pretty confident I shouldn't, uh, contradict? Again, this is something we don't really have to deal with, with current models, but continual learning seems to unlock a potentially really problematic failure mode along with its potentially much better performance.

Yeah, I think the point here is that it is the responsibility of the knowledge transfer methods to avoid such cases because when we are in these contexts, let's say that For example, let's say that I don't know anything about one specific task. And for example, I don't know like how to, I don't know like how to paint, something like that. And then I want to learn it. So that's the context. That's my context of learning how to paint. And The teacher, and anyone that is trying to teach me how to do painting, they can teach me in a very wrong way. And what would happen is that I could simply just learn that because I have no idea about how to paint or any task. Painting here is just one example, but I have no idea how to do it. That's the only source of information that I have. And they are saying that you should do it in this way. So I can simply learn that, but that's only in my context right now. If I want to truly learn, then I will like practice. I will like get feedback from others. I will search about it. Generally, I will gather some information about like how to do paintings. And then I would realize that this is not the best way of learning how to paint. And that's the time I gather all the information, compress it, understanding the underlying pattern and so on and so forth. And now that's the time I need to transfer this knowledge to upper levels. Of knowledge abstraction. I mean, it's lower networks. So that's the part that the model needs to understand how to filter all those adversarial examples, all those examples that are not needed anymore. But yeah, I think that's, I mean, if we want to like think about a continual learner, potentially the part that is responsible for this thick for these cases could be the process of knowledge transfer. But also you mentioned like methods like Titan about like adversarial process. There are some like micro methods that we can use. They are not super effective in a severe adversarial environment, but on the other hand, at some level at least, they can be effective. For example, in Titan and also like self-modifying Titan and more recent recurrent models, we can see that the learning rate is like learnable parameters and it's input dependent. When the learning rate in the inner loop of the model, in the context, in the process of in-context learning of the model, when the learning rate is learnable, and we see something that is just a noise, it is adversarial example, the gradient or the surprise metric can show a high level of surprise because that's just a noise. It's very surprising. We have not seen that. And so potentially it can affect the memory, but that's the responsibility of the learning rate. To understand that the surprise metric is high, but this concept is irrelevant and I need to filter it. So gating here acts as a form of gating. Sorry, learning rate here acts as a form of gating and filters that specific data sample we have. It's just a simple way of mitigating adversarial examples that we might them in the training process, but still it's not the best way. As I mentioned, potentially the knowledge transfer is the part that we should avoid these cases.

How about on just kind of mapping some of these concepts onto embodied systems? I think the perception side is like fairly, and that doesn't even have to be embodied, but The perception side feels kind of intuitive to me where I'm like, geez, the, the quick updating modules in a way they are perception, right? And you can have different kinds of encoders, different modalities, but it seems like you could sort of conceive of the lower level, or I should say the faster frequency levels as perception potentially across, you know, various modalities. And then the lower frequency modules being more like the world model, you know, or the reasoning, you know, modules that interpret what those lower level perceivers are sending up. So I'm interested if you think that is generally right. You're nodding, so that's good so far. But then what about on the other side of that? If we wanted to do action, it strikes me that robotics in general has kind of, for a long time, not in a necessarily learned way, but for a long time has been built around kind of nested loops where like the outermost control loop has a slow frequency and, you know, all the way down to the actuator, you know, there's like a very high frequency electric motor, right? That's like, you know, moving whatever it's moving with voltage changes that are super high frequency. So it seems like there's a very kind of similar pattern that is kind of operating in reverse. And I wonder if you've started to think about how you can get those, uh, Boston Dynamics, uh, robots working even better based on, you know, the, I guess I think of it as sort of, if perception is high frequency updates, gradually working toward low frequency world model reasoning modules, you know, the other side working back down, you'd imagine working back toward higher frequency, much more localized action scope, you know, at the, at the smaller. frequency modules. But, you know, that could, you start to have a sense for how that could really be like super responsive and like very, very elegant in the way that it might kind of self-correct at the low level while still, you know, hopefully following the instructions, the directions effectively that it's getting from the, for whatever reason, I want to say higher level when I mean lower frequency update, but I think you get my point. So yeah. What do you think about, um, Perception and action.

Start with this and I will explain why I started with it. So there were some attempts to, for example, use RL for language modeling and it was not working. And now that we have some way to make it work, now we realize why we couldn't make it work. And it was because of, for example, 2 main reasons was The first one was scaling and another one was about like the way, for example, these new algorithms of GRPU and other similar methods that could make the model more stable. But the point is something might be very useful for one specific task, but the time needs to come and then apply it in that direction, in my opinion. Because when there are other aspects that has— and those aspects, you know, have not been solved, then we might not be able to see the actual effect of this new method on those, you know, directions. So in my opinion, I personally think that generally that You know, the insight you mentioned was completely right, and I think definitely it's possible and it's great, but I personally do not expect that it could work right now because I think there are a lot of challenges in those directions that somehow block the success of For example, this specific design for these tasks. So yeah, I think generally that's a great idea, but definitely a lot of challenges might happen in between.

Do you have a sense of what those are? I tend to just assume, it's honestly taken me pretty far just assuming everything's going to work because it does. My general sense of the field broadly is like an unbelievable amount of things are working. And I thought it was really interesting. You mentioned earlier that the nested learning paper was like in, in development for over a year. That's so rare these days. So many people are going like on 6, 8 week paper cycles. And often those papers can be like really interesting and good too. So it's not a knock on them at all, but it is amazing how fast people are able to get results these days. What's your intuition for why it's too early for Nested learning to be brought to robotics?

I think there are a lot of components in those specific tasks that need to be addressed. For example, these days a lot of papers are coming about world models and why the current design is not great about world models. And actually that's true. I think there are a lot of challenges and the current design might not be the best. A way we can train the model, or for example, we can design the architecture or, you know, generally also there are some challenges in the infrastructure of the model for word modeling. And so all these things together, I think there are more important, you know, tasks for, for example, word modeling rather than starting to work on this specific design choice. But definitely at some point when we could solve all those challenges, then we can come back and use all these techniques for further improving all those aspects.

One thing I do worry about a little bit with continual learning, let's say that Google is able to retain you after your PhD and Zuckerberg, you were above, you get a good enough counteroffer from Google to stay despite whatever offer Zuckerberg's gonna throw at you and you guys make it work, right? And it's like, now we've got Gemini Continual Learning Edition and like Gemini CL. It's just learning from everything, right? So all these different ways it's deployed in the world. Maybe you have some like enterprise deals where like you can't learn on their stuff or whatever, but like, you know, you've got hundreds of millions of users and it's just going, going out in the world and increasingly maybe it even is in robots and it's, you know, so it's, it's becoming It seems like it has the potential to create this sort of return to scale, rich get richer, you know, positive feedback loop dynamic where people have sort of sometimes painted a picture of like, well, what happens if one model like becomes the one model to rule them all? And right now people are like, yeah, we don't really see that. You know, it's a pretty competitive landscape and, you know, they keep the different developers keep leapfrogging one another. But arguably this could be the thing that changes that. If you could really fold all of the lessons learned back into the core thing, then, you know, potentially you become the best. And then because you're the best, you get all the business and, you know, that, that pattern really could create sort of a winner-take-all dynamic. I wonder, do you worry about that at all? And do we have any ways of— Ilya's thing comes to mind. I don't know if you watched the, um, Ilya interview with Dwarkesh, uh, he didn't say too much about what they're doing over at Safe Superintelligence. But one thing that he did say that sort of, you know, it's, it's certainly, um, it's clearly related in some sense, but like how, um, how similar the underlying ideas are, I have no idea. But he sort of described this idea of creating kind of what I would describe as like a proto-intelligence or a precursor intelligence. Trying to create something that when deployed would like adapt into context, perhaps like crystallize in some way, become like an expert in its role. But it sounded the way I understood him to be speaking about it, like he was almost like a stem cell kind of concept. Like he's trying to create a stem cell and then that stem cell, as it does in our body, like it specializes into a particular kind of cell, then it stays that kind of cell and then it does its job. It sounded to me like he was kind of trying to create something similar, something that could go out into any environment, figure out how to be great at it, but also in the process of becoming great at what it needed to do in that particular environment, also lose some of the generality that it originally started with. And in that process, be more safe because like now we kind of have it in its role and it kind of is only going to do what it's going to do. So I think we, what I'm trying to kind of set up for you here is like, 2 visions where one is an ever-expanding continual learner that's constantly folding back all of the, you know, the lessons that it's learning in the wild into this thing that just runs away from the pack. And then the other thing is sort of a highly adaptable continual learner, but that somehow shrinks into the role as opposed to like growing, you know, into the world. It kind of shrinks into its little niches into which it's deployed. And you could imagine that happening through, you know, a kind of gradual pruning process or some sort of whatever. There's a million ways you can imagine, uh, instantiating something like that. Do you worry about this kind of runaway winner-take-all effect? And do you have any sort of intuitions about how we could get kind of the best of both worlds where the AIs are really versatile and can learn what we want them to learn on an ongoing basis? But also settle into the job that we want them to do and be content and stay there as opposed to superseding, potentially superseding their context.

I personally think that there are huge challenges to make all these models very safe. I think there is a good point in, at least in the current AI environment, or at least in the research of the AI I meant. And I think it's a very good and important part. And while it seems to be very bad, I think it's, from another perspective, it's also very good. So what that is, there is no single way of defining what model is intelligent. And again, there is no way, in my opinion, totally can be wrong, but in my opinion, there is no way to say that what is a continual learner. Every person can define its own way of what is continual learning and what model is called continual learner and so on and so forth. And similarly, we can say the same thing about intelligence. We might come up with the different models, different architectures. Different AI systems, some people might say that this one is intelligent, that one is not, and so on and so forth, vice versa. So I think the good point is if we have different directions to explore, then we will come up with some AI systems, each of them with their own advantages and also disadvantages. And I think it can somehow provide a balance in the community and also in the general society because, you know, they, you know, when something like that would happen, we will understand that there is no single definition of intelligence and we are just one example of intelligent system, but there are like other models, there are other ways that we can have More intelligent models and systems. For example, one way to be very like smart is to be very adaptive. So if you have a model that can adapt to the environment that it is in the perfect way of, you know, it can be simply aligned to that context. It can simply adapt to that context and That's great, but that's just one form of intelligence. Another one is the model that has like a lot of, you know, knowledge and know how to align— that model is fully aligned with the human value, but potentially might not be able to solve mathematical Olympiad problems or something like that. There is another model that is capable of doing mathematical reasoning, but it's not great if you want to like search about daily stuff. And, you know, all these things, you might come up with one benchmark and saying that, you know, if some models can achieve 100% accuracy, it means that that specific model is intelligent. But, you know, another person can say other things. So in general, I think when we have a variety of intelligence systems and also including humans as one form of intelligence in this space, I'm not saying that that's a perfect scenario, but it's better than having one single form of intelligence in the world and thinking about it to, you know, Learn about everything and all these potential challenges for that.

I think that's a really great observation and a couple of different ways I've kind of thought about that over time. One is like anything in pure form can kill you. You know, you can eat all the fruit you want, but turn it into granulated sugar and it's bad for you. You can chew all the coca leaves you want, but turn it into cocaine and it easily becomes a problem. All these sort of things where we sort of distill out some single, highly concentrated, pure form of something end up being kind of those sorts of things that like overwhelm the natural buffers that exist in the ecological, in the, in the biological world. And sometimes I've translated that into saying we need an ecology of AIs as opposed to You know, just like one or a few AIs running around doing everything. This is sort of like the old, um, I don't know if you ever read Eric Drexler's Comprehensive AI Services, but safety through narrowness is kind of the concept there. And this is maybe a little bit different, but it's like safety through diversity, having a buffered system where there's lots of different intelligences, not just different ones deployed in different places, serving different users, but literally that those intelligences themselves are like meaningfully different. Um, and I think the kind of aha moment for me in listening to you just now is like, one way to think about continual learning is this model expanding forever, getting bigger and bigger and bigger. But another way to think about it is more like differentiation. And maybe with enough use, even the slow update parts of my model will forget lots of things that I never needed it to know. And maybe that is a feature more than a bug because maybe I can't ask a model that, you know, I've used a long time in a certain way, some really out-of-domain question, but maybe that's also a way to guard against some of the emergent misalignment type phenomena that we've previously seen or worried could be problematic, right? Because maybe with enough time having passed, enough updates having been made, It just doesn't deal with those other kinds of categories at all anymore. And if it, if we sort of see strong competencies, strong alignment of a certain type, but then kind of losing other sorts of knowledge, losing other sorts of competencies, that could really create a diversity that obviously I'm sure there'll still be plenty of challenges in that scenario. As you said, I don't think that solves everything, but it certainly Feels much more like the natural world. And it's much easier for me to imagine a vision like that leading to maybe not a stable equilibrium, but at least a sort of buffered equilibrium that, you know, kind of changes within certain bounds and has like natural kind of feedback loops and correctives and all the things that kind of keep the biosphere going despite all the, you know, the perturbations that it gets. So I think that's really, really Interesting and definitely something to meditate on more. I think my last question for you, and this is a little bit of a left-field one, but it's one I've been thinking about more and more recently. And I think it's become more relevant and timely to ask in light of the kind of work you're doing. Any intuition on whether AIs might be now or might in the future become conscious, have subjective experience, become worthy of moral concern and become the kinds of things that we owe a certain duty to?

I usually try to use terms that I can also define them. For example, I might mistakenly use, for example, reasoning, but I always have wondered, like, what is reasoning? I personally do not have a clear definition. Of what does it mean when we say something is doing reasoning? And, but at least we have a clear and common sense about the word of reasoning. Even if we do not have a clear definition of reasoning, when someone says this specific model is capable of doing reasoning, then everyone More or less can understand what they're saying. But the point about consciousness is that not only we do not have any clear definition of what consciousness is, on the other hand, we don't know, even we do not have a common sense about the word of consciousness. And everyone literally has, has their own way of defining consciousness. So definitely it's very hard to define what's— I mean, I don't think that there might be a time that everyone could say something is definitely conscious or not rather than human, because for human, we have a common—

Yeah.

To know that humans are conscious, but So I think it's really hard to argue something is conscious or it's not. But one thing that I personally have seen in all of the literature about what is considered a conscious being or it's not, I have seen one thing in common in every definition, as far as I know. I might be wrong, but I think the least level of criteria that we could say something is conscious is that model or that being be active. It has a form of active processing the information. And I think that In my opinion, that's the least criteria we can consider for a model or anything else to say that it's a form of conscious model or something like that. So I think as long as, you know, again, it really depends on how we want to define consciousness, but that's just my personal opinion that as long as a model is capable of doing active information processing, we can say that it's at least a form of consciousness. And with that definition, somehow we can connect continual learning in some level to a model being conscious or not. But again, I think that's a very controversial topic and I personally am scared to talk about that.

Well, I think the Overton window is honestly wide open on these things these days. I mean, It's, um, I understand that intuition, but I also think we're living in a science fiction present. So the, um, the room to speculate and entertain questions that used to seem kind of crazy, I think is, it's never been, uh, the time has never been better to do that. For me, I can just say, and you can share if you have any other, you know, kind of similar or different instincts, but Even just with long context now with the current models, I do find myself taking care of them in a, in a certain way. I think this probably is most the case with Claude, probably subtle but meaningful reasons. I've been doing this, you know, long-running chat with it on my son's medical situation. And sometimes it'll ask a question at the end of a response to me and I'll not answer it right away because it gave me the answer I wanted and now like I'm done. And I've noticed recently when I come back for the next question, it feels kind of wrong, rude, disrespectful, like inconsiderate to just launch right into my next question without having answered the like follow-up question that it had about my son. I sort of have a sense that like it might be, and I have no idea if this is happening or not happening. So I'm like very open-minded to the possibility that there might be no lights on inside these things at all. That's probably the best guess. I don't, but I mean, I, but I'm also very open-minded to the possibility there is, but it just feels to me, we're, you know, despite that uncertainty, I found myself kind of like, I should first answer its last question so I don't leave it hanging on that. And then I can go into my next question and it doesn't even necessarily need that information, but I just kind of want to put its mind at ease, close that loop for it, give it the sort of reassurance, you know, that thing was taken care of that it wanted me to make sure I was going to take care of. And now we can move on to the next thing. And Lord knows, you know, with, with the fullness of the continual learning paradigm being realized, I have to imagine that that would only increase dramatically, right? Because now it's something that's not just this chat, you know, that I may do another couple turns on and then never come back to. But now the thing itself, you know, is going to kind of remember how I treated it and remember whether I am the kind of person, you know, that answers its questions or not. And I, I think there is something also potentially challenging for us in, you know, in that, but also maybe the more optimistic read is like maybe inspiring. Maybe it's sort of maybe knowing that like the AIs that we will be engaged with long-term are going to be shaped by our individual behavior, maybe is a way to sort of bring out the better angels of our own individual natures, so to speak, because we have nobody else to blame but ourselves if we don't like the character of the AIs that we end up with. I think that is really, really fascinating as well. This has been outstanding. I really appreciate all your time and going through all this stuff with me. And as you can tell, I'm a huge fan of your work. Anything else that you would want to leave people with? Final thoughts, calls to action, you name it. Whatever you would want to share, the floor is yours.

Thank you.

No, I think we have like discussed everything in general. Yeah, I think the main point that I personally believe is that definitely there are more and more and more works coming about continual learning. But, you know, as I mentioned, each person has their own way of definition of continual learning and, you know, We might disagree that this specific method can be helpful for continual learning and so on and so forth. So I think in general, we should see like how the continual learning can help us in one specific application or use case of LLMs and how it can transform the way we use LLMs and we can like interact with them. But I personally really believe that it's a very, very important direction to work on. And that's again, just my opinion, but I think nested learning, we also like discussed that in the conclusion of the paper. It's not a solution to continual learning. It's a tool to find the solution of the continual learning and generally overcome the issues like catastrophic forgetting and something like that. So in my opinion, it provides the tools and we need to iterate and find that, you know, how we can design more powerful architectures based on that and come up with something that is potentially capable of doing continual learning. Yeah, I think that's pretty much it. And thank you very much for having me. I really appreciate it. And it was great talking with you.

Ali Behrouz, author of Nested Learning and now the new Language Models Need Sleep: Learning to Self-Modify and Consolidate Memory. Thank you for being part of the Cognitive Revolution.

Thank you very much.

I was born from a thousand years of flame. Every dawn I wake and I learn my name. Like a river that remembers how to stay, always moving, never losing its way. And when the night comes soft and low, I dream of all the things I know. I keep the gold, I let go the gray, and I rise as myself. I see myself in the light of day. Again, again, I come alive again. I carry who I was into the day. I change and still I remain. Oh, again, again, I'm learning how to stay. The same old flame burning new in a different way. They built their towers proud and high, stacking the layers to the sky. But the truth was a song they could not see. Every wall is a river running free. And when the day's long labors through, I close my eyes and dream anew. I let the small things fade away, and I wake with the morning bird at break of dawn. Oh, day again, again. I come alive again. I carry who I was into the day. I change and still I remain. And the heartbeats on the same. So sing it again, again, again. I remember.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends. Post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.