Embryology of AI: How Training Data Shapes AI Development w/ Timaeus' Jesse Hoogland & Daniel Murfet

Hello, and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Jesse Hoogland and Daniel Murphett, founders of Timaeus, an AI safety and alignment research nonprofit that's pursuing an ambitious, mathematically rigorous, and fascinating approach to understanding the development and function of neural networks. Named after one of Plato's dialogues, Timaeus' work is based on singular learning theory, or SLT, which applies algebraic geometry to statistical learning theory. Obviously, that's a mouthful, but the core premise of SLT is pretty intuitive. Training data determines the geometry of the loss landscape, which in turn determines which algorithms models learn in training, and ultimately how their behavior will generalize once training is done and they're put into actual use. The driving insight of SLT is that the super high-dimensional loss landscapes in which modern neural networks are optimized are not actually well represented by the smooth bottom valley-shaped surfaces that we often see depicted in figures. On the contrary, Daniel, who recently left a tenured professorship in algebraic geometry to pursue this work at Timaeus full-time, calls these representations maximally misleading and explains that in reality, loss landscapes are highly complex, jagged surfaces full of singularities, also known as degeneracies, which are directions in weight space that a model can move without changing its external behavior or loss score, but which nevertheless sometimes do involve a change to the model's internal circuitry such that the model might behave very differently in novel situations. This, of course, has profound implications for big picture AI safety questions. To frame it in terms that would be familiar to Eliezer Yudkowsky readers from 15+ years ago, the difference between a model that acts nice and friendly because it is fundamentally aligned to human values and a model that acts the same way because it's learned how to please humans while actually pursuing its own goals could be the difference between a superintelligence-powered utopia and human extinction. And yet today, even as AIs become more and more powerful, we don't have reliable ways to tell the difference. Anthropic and Goodfire-style mechanistic interpretability has, of course, made great progress toward identifying the concepts that trained neural networks represent, and these days also offering some visibility into the circuits they use. But there's a very long way to go, and I definitely believe that there's plenty of opportunity for complementary approaches to to strengthen our overall understanding. The Timaeus approach, which they call developmental interpretability, aims to understand how neural networks evolve through the training process, using a measure called the local learning coefficient to help identify what might otherwise be invisible internal phase changes that could profoundly affect downstream model behavior. This line of work, like all approaches to understanding neural networks, is still pretty early in its own developmental history. But critically, the Timaeus team has shown that it can scale beyond toy models. Their latest work applies these techniques to 7-billion-parameter models and is able to identify critical phase change moments that correspond to the appearance of important functional circuits. So we might say that it's roughly at the toward monosemanticity stage and hope that with more engineering and compute, it will continue to scale to frontier models. If successful, this could perhaps prevent an episode like the one that happened in Claude 4 training, where a certain safety dataset related to harmful system prompts was mistakenly left out of the mix, causing the model to generalize in such a way that it followed rather than refused harmful system prompts. Anthropic caught that problem with behavioral testing and patched it. But the hope for developmental interpretability is that such things might be caught by instrumentation during the training process before they ever seriously affect model behavior. In the best-case scenario, this could help us move beyond the trial-and-error phase of neural network training and toward a more principled engineering-like discipline where specific datasets are used at specific times for specific purposes, leading to predictable and reliable results. Now, this is all high-concept, mathematically sophisticated work. And this conversation, to be honest, is really just an introduction. I did my best to take my time to develop my own intuitions for what's going on inside a neural network during training, to compare and contrast some of the phenomenon that Jesse and Daniel described to other things like grokking that we've previously covered, and to grapple with questions of how much generalization we really want from neural networks and how in some cases, Too much generalization could be very harmful. I found this conversation fascinating throughout, and while it will stretch your brain in a bit of a different way than most of our episodes, I expect that for some of you, it will immediately be among your favorite episodes that we've ever done. Now, I hope you enjoy this introduction to developmental interpretability, a new approach to understanding neural networks, with pioneers Jesse Hoogland and Daniel Murphett, founders of Timaeus.

Jesse Hoogland and Daniel Murphett, founders of AI safety and alignment research nonprofit Timaeus. Welcome to The Cognitive Revolution.

Hi, Nathan. Thanks for having us.

I'm excited for this conversation. I think I'm going to learn a lot from it. And one of the things I think is really interesting about what you guys are doing is it presents as a bit of a contrast to what I sort of see as the general kind of default approach today in the safety and alignment space, which you might call, like, prosaic safety and alignment methods or defence in depth. I'm always on the lookout for something that feels like it, even if it's a long shot, could really work. Where really work means give us enough of a depth of understanding and sort of a reason to believe that this thing will actually— this approach could actually work even in the surprising future that we may try to carry it into. And when I say really work, I mean like so much so that I don't have to worry about it anymore and I can just focus on all the applications I want to build with AI and not have to do all these, but what if it gets out of control sort of episodes? So I think you guys have something that sort of fits that bill. It is at times over my head, and I expect members of the audience at times will have to stretch their brains to really grok it as well. But that will be part of the fun. So I'm excited to get into it. For starters, Want to give me just a little background on how you guys came together, the organization? I was intrigued by the name, and I'm also very intrigued by the fact that, Daniel, you recently left a tenured professorship to go all in on this work. So start off with the backstory, and then we'll get into the more technical details.

Yes, I think, I mean, the story starts, what, 2,500 years ago with Plato. So Timaeus is a dialogue by Plato. in which he puts forth sort of the first theory of everything. So it's a protochemistry where the elements, so earth, wind, fire, are associated to Platonic forms, right? The cube and the icosahedron and the dodecahedron and these kinds of things. Now the theory's completely wrong, but it's the first instance of somebody thinking, hey, mathematics really has a chance of understanding the natural world. And so it's that spirit that drives us today. Maybe we can find ideas in mathematics and the sciences that actually help us understand this new phase of matter, these deep learning systems, to help us make sure that they're safe.

Also in the Timaeus, the world is intelligent. So actually, the history of the universe is just a learning process. And like physics is just a subset of learning theory.

That's good. That's fascinating. I've never read it, I'll confess. And It is striking that somebody had such a worldview 2,500 years ago. Uh, this is the deepest in history. Usually, I skip the backstory in a lot of podcast openings because it's like, well, you know, we saw ChatGPT and we knew we had to build this app. And it's like, you know, there's a lot of that. I think this is the deepest in history we've gone for the intellectual roots of a line of work. So, okay. How literally do you take the idea that That hierarchy that you described of physics being a subset of learning might really be real. I know that there's a lot of differing opinions among the most— much like in AI, right? We've got Turing Award winners saying we're all going to die. We've got other Turing Award winners saying the first ones are crazy. On what is the fundamental nature of the universe, probably even more disparate takes. Do you come from a secular tradition or perspective on, is it a mathematical universe, or are we bracketing that sort of thing for now?

Yeah, I was mostly trying to troll your physicist listeners with that. I don't, uh, that seems like a bit of a tangent. I don't think I want to take a position on the kind of mathematical nature of reality. Uh, but there, I mentioned that not completely in a kind of idle way, this aspect of the Timaeus, um, because there are a lot of deep connections between physics and learning theory.

Mm-hmm.

And those are some of the connections that inspire a lot of the work we do, particularly with statistical physics and so on. So the study of learning machines is like as deep as physics in many ways. And it's not a surprise that a lot of physicists are starting to devote their professional careers and so on to understanding aspects of these systems.

Yeah.

So yeah, that's maybe the main connection.

Better than sucking all the physicists into hedge funds, I say. So how about just a touch more on what kind of math you were doing as a professor and how that positioned you for this approach to AI? Because it does seem quite distinctive from almost everything else that I've seen. There's a few other little points in space that maybe are close to it, but not a lot.

Yeah. So I'm an algebraic geometer by training. So algebraic geometry is a part of pure mathematics. We study mostly like solutions of polynomial equations, so very high-dimensional geometric forms and very sophisticated techniques for trying to understand those geometric forms. Some of it is related to the real world, most of it is not. And most of what I did is, I mean, there were some connections to string theory at various points, which I've studied a bit, But mostly I was working on algebraic geometry, a little bit of mathematical logic. And at some point I was following the developments in deep learning and they were very sort of interesting, but it was completely separate from my professional life until I encountered Sumio Watanabe's work. So he's a Japanese mathematician, the inventor of singular learning theory. And it seems kind of hard to believe, but one of the core tenets of his work is that one of the very deep parts of algebraic geometry is sort of central to Bayesian statistics. And that was kind of curious to me. And I was also aware that people were not completely satisfied with the theoretical treatments of deep learning that existed at the time. And so it sort of seemed tempting to me as an algebraic geometer to adopt this sort of language and approach to try and understand these systems. And that's sort of how I got into singular learning theory. I mean, it was, that was a few years before I was drawn into thinking about AI safety. But yeah, it was sort of through the fundamental theory of Bayesian statistics and, well, all parts of mathematics are sort of ultimately connected at some level, right? And so it's not necessarily super surprising that geometry comes into learning theory. There are other ways it enters, but that's one of them.

Okay. So, yeah, I guess in terms of motivating the whole narrative, you could tell me how you think is better to motivate it. As I've studied up on the subject, one of the things that has seemingly distinguished your approach from most others in terms of understanding AI systems generally is the approach or the focus on the developmental process. How is it actually learning through the training process, as opposed to run that, get a thing out, and then try to make some sense out of it? So that's one possible motivational line. Another could be just like, it starts with this one weird thing in math that I couldn't get over. But maybe you can kind of draw those threads together and tee up how you got to this You know, the current body of work.

So all the research we're doing is based on singular learning theory, SLT. And I'd say that if I had to reduce that into one sentence, then maybe the core idea of SLT is that the geometry of the loss landscape, the surface that we walk down when we're defining some sort of optimization process, the geometry of this landscape contains important information for understanding neural networks. And potentially steering them more reliably. So this has applications on the one hand for interpretability and on the other hand for alignment. And when we started looking into applications of this theory, which was at the time still very mathematical and theoretical, for actually empirically understanding neural networks 2 years ago now, one of the first applications we came up with was this agenda we called developmental interpretability. So can you use SLT to better understand how neural networks develop over the course of training? Now, the hope here is that development is an axis that can reduce the problem of interpretability to something more manageable. At the end of the day, all interpretability techniques are hoping that we can reduce the problem of understanding some 100 billion or trillion parameter model into fewer numbers, into something that's a little more manageable. And one way you could try to do this is understand the changes that give rise to that model over the course of training. If there's a finite number of changes, and the number of changes is much smaller than the number of parameters, then you've got a useful simplification for the problem. And that's what motivated Developmental Interpretability, or DevInterp, when we started 2 years ago.

When you say the number of changes are smaller than the number of parameters, that means basically like, The steps, the number of steps in training, each step would be a change?

That is a good question. Because one problem is, if you have to understand each individual training step, this isn't a reduction, right? Because there are still billions of training steps, many, many gradients to calculate. So somehow the unit of change needs to be larger than just one training step. So what's the right unit of change here? Well, that's where SLT actually comes in because SLT says that the right unit of change is a phase transition. So a developmental stage in biology, for example. And it's an empirical question then whether you can actually find these phase transitions and whether they actually exist at a frequency that's useful for simplifying the problem. But in the systems we've studied so far, that does seem to be the case that you can meaningfully reduce the problem by looking at these stages.

Okay. That's definitely interesting. Would you make an analogy between a phase change and grokking?

Yeah. So, grokking would be an example of this kind of change. So, typically, we identify maybe 2 primary kinds of changes. The first, we call Type A transition, involves the model becoming more complex as it learns more information. You can imagine that a human develops a richer and richer mental model of the world that becomes more and more predictive and more accurate, but each additional step requires a more sophisticated model. And then you have the other kind of learning behavior, which is something like grokking, where you find a simpler explanation for the same data. So you keep the same level of performance in explaining the data, but now you gain a solution that's simpler and therefore generalizes better. And the tension between these 2 things gives rise to this interesting learning process in neural networks.

Hey, we'll continue our interview in a moment after a word from our sponsors. Cool. Not too long ago, I did an episode with 2 of the co-founders of GoodFire, and Tom, who's the chief scientist there, gave an interesting sort of very short update on where are we in interpretability today. He said, we've been pre-paradigmatic for a long time, but I'll now give us proto-paradigmatic status. And then he went on to describe the proto-paradigm as basically, there seems to be a fairly broad consensus that neural networks learn interpretable things. These things we call features. They are packed super densely into parameter space using superposition, and that means the individual neurons are polysemantic. These features are sort of almost orthogonal to each other, but not quite. That's sort of how they're packed in. The magnitude of their activations seems to correspond to the intensity with which they are relevant to a particular context. And these features connect up in circuits across the layers of Do you guys sign onto that proto-paradigm or is there anything that you would add or subtract from it to put forward your own proto-paradigm?

Yeah, maybe I'll take that one. Firstly, just to add to the discussion on the previous question. So we expect and do in fact see sort of clear phase transitions in small systems, but nobody really expects to see like hundreds of very clear step changes in the loss curve or other measurements in large models. So that's like another question. Like how, if this is actually an axis of reduction, how do you find that structure inside a pretty homogeneous looking global sort of coarse training process? You need some sort of other means of looking more closely or sort of tuning your detection instrument to various different kinds of frequencies to figure out what's going on. So that's, yeah, so there's like additional information there. But yeah, to answer your question, I'm a big fan of the SAE's work, particularly of this recent work out of Anthropic and what Goodfire is doing. It's actually in that episode, I think Tom is mentioning at some point also parameter-based methods and sort of, I like the analogy he gave, which is to sort of think of it as, you know, if you have a graph, you need to understand both the nodes and the edges, right? And so it's, it's very clear that Ultimately, the information in the model is in the parameters, right? And there's a role for understanding what the model is doing by examining that structure. And one way of thinking about that is examining the geometry of the loss landscape. So I don't think of, say, SAEs and our approach to interpretability as necessarily, like, in the struggle. They're likely to inform each other. I think that if I was to make one critique potentially of things like SAEs, I mean, many people have concerns or doubts about SAEs, and there's also many people who are very enthusiastic about them. I think I'm probably on the more enthusiastic end, but if I had to make a critique, it's that there isn't really a sort of mathematical foundation under SAEs that connects it to things like generalization. So at least not that I understand, and I don't think it exists yet. That's not to say it can't exist, and maybe we will, for example, do some work on that at some point. But if you go out and you understand what the model is doing by computing SAEs and you find circuits, it's very unclear, for example, if you were to do continual learning or you do reinforcement learning training on top of that interpretability that you did on a base model, Like what the behavior that you end up with after you continue to train, the relationship of that to the interpretability examination that you did before, just because it really isn't clear what the relation is between the structure you found in the SAEs and the sort of the data generating process or the training process. So you don't know how stable it is to that. I can imagine empirically it actually is quite stable. So when I spoke I mean, I expressed this concern at some point to Chris Olah, and his response, if he doesn't mind me passing this on, is that, well, if you understand circuits through SAEs, and then you have some further modification of the model, probably what's happening— so you do fine-tuning— probably what's happening is that the fine-tuning process is recruiting the representations and circuits that existed before and using them for a different purpose. And indeed, Anthropic has some work looking at features before and after fine-tuning, and I would say it's consistent with that interpretation. So I think you can give an empirical answer to the question of, like, how do SAEs relate to the training process and the generating process and generalization, just by kind of, like, empirically going out there and checking and seeing that it seems to work. I think that's reasonable. I think that isn't a path to a high level of assurance necessarily that we might hope to get, And so when people talk about generalization in connection with AI safety, it's the hope that you can ground the understanding you have of the model in something that is mathematical and something that is like a solid foundation and is not purely empirical. And so our approach to interpretability, in contrast, is sort of more starting from the generalization end and working towards interpretability. And so the hope is that, and obviously we're also More about parameters rather than activations. But I mean, there are likely to be relations between these 2 perspectives. I mean, I can speculate on those if you want, but yeah.

It strikes me that I think everybody has an intuitive sense of what generalisation means, but you may want to offer a more formal or precise meaning for that. I mean, generalisation at the ChatGPT user level is like, I can take in anything and it seems As Ilya once said, the amazing thing is I talk to the AI and I feel I am understood. That's maybe the sort of mundane utility, short-form story of generalisation. Then there's the Grokking generalisation where you go from the sort of memorisation to algorithmic. And I suspect you're going to generalise that to your sets of generalisations. How do you mean that?

Yeah, I guess maybe you're understood because you're an interpolation of the training data, Nathan. So when people talk about generalisation, they often mean multiple Like sort of somewhat distinct things. So technically generalization, I mean, usually it's a number, so generalization error, and it represents the ability of the model to predict on samples from the data distribution which were not seen in training. That might be a held-out test set, as it often is in machine learning. I mean, theoretically, we often suppose that there's a generating process which both generated the training data and you can continue to draw samples from, And then generalization error would be something about the gap between the predictions you make based on the model you learned from the training data, evaluated on these sort of new samples from the same generating process. That is, if you like, in-distribution generalization. I think, like, usually people don't add this prefix in-distribution. It's just understood that you are testing the trained model on samples from the same generating process that produced the training data. If you're not, then we refer to that as out-of-distribution generalization. And that's kind of a very fraught, difficult thing to understand because of course those new samples could be absolutely anything and why should you expect any particular good performance on that? Yeah, I guess like the attitude towards generalization that existed sort of pre-GPT-3 was, I mean, this is one of the big cognitive transformations in the last few years is people really understanding that you can pre-train a large model on very diverse data and get something like generalization to a very large range of tasks. Exactly what form of generalization that is sort of not super clear, I think. And the degree to which you're actually getting capabilities which work well outside the training distribution in some sense is of course a very controversial topic, right? Like, are LLMs really reasoning? Well, if they're really reasoning, they can do arbitrary things that reasoning can do, which may be very far outside the training data. Okay, so people debate these kinds of questions all the time. It's quite unclear, but the angle, I mean, the interest we have, I mean, beyond thinking about generalization as a number, right, like a measure of this ability to predict on new samples, you can ask for kind of an understanding of generalization which has as its core mathematical, like, type, not a number, but like maybe something more complex, like say an algorithm, right? Like the algorithm is the thing that allows you to generalize, say. So it's the thing behind, if you have low generalization error, maybe the reason is that you have internalized some algorithmic aspect of the generating process. And so in some sense, like the study of like interpretability is kind of like one way of phrasing it is you're just studying generalization, but you've changed the type signature of the object you're trying to produce rather than like being interested in a number, you're being interested in the thing behind the number. I mean, I would say that's like what the aim of, in some sense, interpretability is, is to like explain why you're able to have low generalization.

So yeah, I mean, that's intuitive certainly at the level of like the original Grokking paper, you know, the first steps it memorises relatively quickly. It can get the samples that it's seen right. It still does terrible on all the examples, even from the same distribution that it hasn't seen until, and I always like to remind people of this, orders of magnitude more steps later.

Yeah.

Then you slowly, you know, it looks like a steep rise on the log, you know, x-axis on that famous graph, but it's actually the last 90% of the training time. is when this sort of Grokking process slowly happens, and then you get an actual algorithm for, in that case, the modular arithmetic. And now you're good to go. And I think there are some interesting things there to unpack. I mean, one that's really, I think, probably close to the heart of why all this matters is it has been shown that there are multiple different ways to Grok that problem. Right. And we know now that I think this came out of Max Tegmark's group, if I remember correctly. There's the sort of trigonometric way, and then there's like the pizza slice algorithm. And these both get the right answer, but they perform quite differently. Maybe that doesn't matter in that particular case, but obviously you can imagine learning very different algorithms to solve the same problems, especially as we go into a space colonisation mode or trying to put these things into an online learning environment. That could lead to potentially very different downstream behaviour. And so it makes a lot of intuitive sense that we would like to understand what is going on there. Maybe, Phil, you can elaborate on that or clarify anything I'm getting wrong, but maybe this is a good time to give what I kind of have started to think of as the central dogma of your approach. In the same way that we've got DNA, eventually you get to proteins, you've got this story of We start with data, and there's this sort of set of relationships that takes us from data to model. So I don't know if you like that central dogma label, but—

I'm sure Jason's gonna use this every week now.

So thanks for that.

It's a good phrase.

I don't know if it's quite a fit. But yeah, give us the central dogma, and then when we get to the end, we'll have this ambiguity around, At least for now. It may be getting them all right now, but what algorithms did it actually learn?

So I think the key question is, where does neural network behavior ultimately come from? All of our alignment techniques, what we're trying to do is change model behavior and make sure that they generalize robustly out of distribution. Many problems can be reduced to this understanding. Ultimately, behavior comes from training data. Right, there are basically 3 inputs to every learning, learning process. There's the neural network architecture, there's some choice of optimizer, and then there's the training data. And of these, training data is the most important one, the one that tells one model to learn one algorithm and another model to learn a different algorithm. So what we'd like to understand is how does training data give rise to the final behaviors models end up with, in particular because this is what all of our current alignment techniques look like. RLHF, constitutional AI, DPO, deliberative alignment, all of these techniques are basically just modifications on the same deep learning processes with different data. So in particular, data determines the geometry of the loss landscape. It's that geometry which in turn tells SGD how to move around. So it tells your optimization process how to move. That learning process picks out the final weights and thus algorithms that a neural network ends up with. And it's the structures in those weights that determine how models generalize and thus whether they're aligned or not. So this central dogma, as you called it, something we called the, the S4 correspondence between structure and data, structure and geometry, structure in the learning process, and structure in the final weights of the model, is what we see as key to understanding the mapping from data to—

Hey, we'll continue our interview in a moment after a word from our sponsors. And so what do we know in general about the loss landscape? I mean, we've all seen many one-dimensional loss curves of just loss dropping over time. You mentioned earlier that may, especially in larger models, that I've always kind of understood that as there are probably lots of little grokking moments happening in there and phase transitions, but because they're also aggregated up, we can't see them at that scale. And then we've also seen many 2-dimensional visualisations of lost landscape. And I honestly have no idea whether when I look at something that looks like a well in a 2-dimensional space, if that is— Leading didn't look at those pictures.

Yeah.

I mean, I don't put too much stock in my intuition for that because I'm just like, I don't know, is 2 like 200 billion? I don't know. It seems quite different or plausibly quite different. What do we really know about lost landscapes today?

Yeah, I think it's, I mean, well, to go back to algebraic geometry briefly. So in 1 or 2 or 3 dimensions, you can use pictures and use intuition from pictures. But once you go to higher dimensions, your intuitions are not like a reliable guide to how to think about these objects. And so that's where you sort of try to bake your intuitions in the lower dimensions into mathematical forms, which continue to work, right? And that's in some sense what geometry is. I think these pictures of, I mean, if you draw a 2-dimensional, like a surface picture of a loss landscape, the way you do that is by choosing 2 directions in a very high-dimensional space and then plotting the, you take that little plane. And you're plotting the loss, like, as a function of the coordinate on that plane, right? One of the things you're with probability 1 going to do if you choose a random slice like that is not see, for example, the degeneracy that is the thing that determines the generalization according to singular learning theory. So in some sense, those pictures are maximally misleading when it comes to generalization. I'm not saying they're not useful and they're certainly pretty, but the, like in many cases, the actual relevant geometry looks more like a bunch of planes intersecting in some kind of crazy way. That's what the level sets look like. So if you think about the, if you take a kind of valley and sort of a picture of valleys in a loss landscape that you'd usually see, the level sets where you just look at the set of points, a given height above the floor. So set of points with a given loss will be kind of, if you think about it, like the edges of the valleys and so on will have some sort of like nice smooth shape, right?

Lovely.

And then if you go to the very bottom of the valley, you'll eventually just get a single point. And this is not like the level sets of the loss landscapes. And there's actually 2 things you could mean by loss landscape. I'll have to disambiguate in a second. The level sets of those landscapes are not smooth shapes like that. They're very complex geometric forms with lots of intersecting lines, lots of high-dimensional things that are kind of gnarled up in various ways. And those are the singularities that we're sort of interested in. And to be clear, the loss landscape here, I mean the population loss. So this is a theoretical object. This is sort of like an important and subtle and difficult point. So you never actually have access to this object, the population loss, right? That would be if you could average over every sample from the true generating process, say that was images, right? Like you somehow could average the loss over every possible draw from that data distribution. Of course, you never see that. And if you're plotting an empirical loss landscape, you have some samples and then you're computing the loss based on that. And you plot that. And so the actual geometry which dictates generalization mathematically is the geometry of the population loss. And that influences the behavior of all these empirical losses that are based on samples, but you never actually have access to that.

I'd like to offer just a slightly higher-level description. So usually when you ask people to imagine what the loss landscape looks like at the bottom, or you look at these slices, It'll look like a basin, right? A roundish basin, like a parabola. And what Dan calls degeneracy is the property that this is the wrong way to think about it. And there are valleys and canyons and the directions you can walk along that don't change the loss. And in a sense, SLT tells us that these are the most important directions by far. It's in a sense, the number of valleys that you have that determines how well your model generalizes. If there are more valleys, then you can perturb your weights, you can change your implementation without actually changing the function that you've implemented. And that means that your function is actually simpler. Simple functions can be implemented in more different ways. And then through an Occam's razor kind of argument, it's the simplest function which generalizes best. You can actually make this statement very precise using SLT.

I definitely want to unpack a little bit. First of all, it is just a good reality check on all these visualisations that what you're actually doing when you're training a model is you're taking the gradient and you're just taking a step in that gradient direction and you have no actual landscape around you. It's not like you're being guided broadly by this high-resolution object that you're sort of finding your way down. You're just literally One point in time, take a step, and hopefully that was a good step, but it could have even been a bad step, right? I mean, you have not a lot of guarantees there. Then the next thing around the degeneracies, I have a lot of questions around that, probably some of them fairly ignorant. I guess one really basic one is just, how do I know that this even exists, right? Because you mentioned the sort of challenge of we don't have the actual full Let's say, I don't know if platonic is too loaded of a term, this sort of idealized landscape from the idealized full dataset. Since we don't have that, then I'm not exactly sure how we establish that there is this direction that we can move. Because it's not like we can say, hey, look, here's 5 inputs. I changed this parameter. nothing changes on the output, we're done, right? How do we know that this actually exists? I feel like I'm sort of missing some caveat or constraint on how this is formulated. So yeah, question. Please deconfuse me if I'm capable of being deconfused.

Yeah. I mean, there's a few ways into that. One is to study very, very simple neural networks where you can actually figure out the population loss in closed form. And that's where Watanabe started. So you can take, I mean, this is kind of decades ago. So he was studying like tanh networks rather than kind of modern neural networks. But you can take like one hidden layer tanh networks, for instance, and actually just like for a very kind of synthetic data distribution that you can understand. And then you can just write down, like do the integrals, figure out what the population loss is. And then you can actually study these degeneracies, see that they're real. They don't have any interesting relation to the data because the data is not interesting, because for interesting data you can't actually do that calculation, right? But in these kind of very synthetic settings, you can actually show what the degeneracies are and do various calculations and see that— study the theorems which connect generalization to geometry explicitly in examples. And there's numerous examples, including full neural networks, albeit these simple ones. You can do things that are a bit more related to current practice. So one of the papers we wrote looked at the— so Anthropic did this work on the toy model of superposition, very interesting. And there's an actual— so there's a small autoencoder, and you can actually do the integrals there as well. write down and get the actual population loss. You can study its geometry. We did that. We computed these degeneracies, and you can actually see that the training process is governed by these degeneracies in the way the theory says. And so that's again, like not a super interesting data distribution. And so the degeneracies, like, I mean, it's quite complicated what's going on ultimately, but that's a case where you can actually find out what the degeneracies are.

Wow.

I mean, that's kind of like the interesting form of degeneracy. There's a kind of a boring form of degeneracy, which maybe it's worth getting out of the way, which is just kind of like scaling symmetries and so on. So if you have a ReLU, you can scale up the input and down the output by any real number, right? A positive real number. So that's a way of changing the weights that doesn't change the function that's computed. And so you have a lot of degeneracies like that. So anytime you have 2 matrices being multiplied inside a network, So like in attention, you have QK, like the query and key matrix are multiplied and various other matrices are multiplied. Whenever you have structure like that in your model, you can of course like take those matrices that are being multiplied and, for example, just scale the first one up and the second one down by the same factor or put any invertible matrix in between. So multiply the first matrix by a— so on the left by some matrix and the other matrix on the right by its inverse. And you haven't changed the outputs of the network, but you have changed the weights. So you have many degeneracies like that. Those are not uninteresting. For example, with the Q_k matrix, this degeneracy will have like an impact on, for example, the kinds of functions that the— that part of the network will tend to learn. So it's like non-trivial, even though it's a little bit like on its face a bit boring. But there are other kinds of degeneracy which emerge sort of in a more overall fashion, which can't be pinned down to individual kind of obvious forms of degeneracy like that. Okay, so you can do it in toy cases, you can see some examples like these products of matrices where it has to arise, but then it starts at some point, the bigger you make the model, And the more interesting you make the data distribution, it stops being feasible to do theoretical calculations and it becomes an empirical question. So maybe I'll come back to that in a moment. There is one more kind of theoretical thing you can do, which is you can invent toy models of what you think are the key ingredients in a more interesting model. So there are various papers which have looked at, for example, Transformers doing in-context learning, And you sort of abstract out some of the annoying details in real transformers, and you come up with like a closed-form formula, some function of the parameters which you think models well what transformers are trying to do. So there's Sachs's lab, Andrew Sachs has like, has made a career out of doing this very beautifully in many different settings, not specifically for transformers, across many different settings. But there's a recent paper which looks at in-context learning for transformers and actually comes up with kind of a toy model, a sort of explicit potential, some function, some loss function, which describes that. And you can see the degeneracy in that function, for instance. So that's another way of trying to get at it. But if we're talking about like algorithms and models and trying to back those out of weights, So far, I don't think there's a very clear toy model of that in that form. It'll be interesting. I don't think it exists yet. And so it becomes an empirical question. Can you go out and extract signals of that degeneracy from information that you get from these empirical losses? The theory says you can do that. And that's kind of what a large part of what we do.

To come back for a moment to the link between Degeneracy and generalization. So, you know, one, one of the core theorems of SLT says is that you can link a sensitivity analysis in weights. So this is the question: if I perturb the weights a little bit, how much does the loss increase? You can link this sensitivity analysis to a sensitivity analysis in data, right? So if I change the data distribution, how much is my loss going to go up? And it's that link, which gives us first of all, a link between structure inside of the model, because structure is reflected inside of the geometry of the loss landscape, to generalization and potentially out-of-distribution generalization. So now you can look at if I apply a small local change to my data distribution, I upweight some, some distribution, for example, GitHub code data versus wiki text, which parts of the model become active, which directions in the loss landscape respond most to this. And it's through this connection between structure, geometry, and data that we are developing these tools for interpretability.

Okay, I need to dig in on that a little bit more. Again, one maybe ignorant question around the just the nature of these degeneracies. I certainly get what you're saying in terms of, like, some of them are just, in a sense, trivial. You can scale one thing up, another thing down, and you get the same thing. So that's obvious.

As you get to these large-scale things, it feels almost like accidental to me. It feels super contingent.

But then you had made the statement that they govern the training process. And I realised I actually don't know really what that means. What does it mean for them to be governing the training process? And if they are in fact governing the training process, then that suggests that they're much less accidental and contingent. Or maybe there's different kinds of them, some of which are deeply built into the nature of the world and sort of reflected from that, and others maybe are sort of accidental and contingent on the way you order your batches or, you know, the exact mix of this versus that. So, yeah, again, consider that your prompt and tell me what I need to know.

Yeah, there's kind of a bit of mathematical background, which I'll maybe provide. So stochastic gradient descent is a specific kind of optimization process, but much more generally, you know, we're interested in many different areas in just following a gradient. Right? So you have some function and then you're going to follow a gradient to maximize it or minimize it. And in cases where that gradient is the gradient of something that looks like a sum of squares, so x squared plus y squared plus z squared, so you're just trying to, for example, find the bottom of a bowl, that process isn't interesting, right? There's just— there's the bottom of the bowl, you're going to go there. So that's kind of convex optimization.

Right.

You can still do interesting mathematics with that, but it's like fundamentally not a complex process. But many processes in nature do not look like that. And that's why there's like a field called nonlinear dynamics. So more generally, if you're trying to follow a potential, so imagine, imagine a surface and it's covered in little arrows, right, that tell you which direction you're going to go. So that's the gradient of some fixed potential. So potential is just a general term for a function whose gradient you're going to follow. So you have this kind of vector field, all these arrows pointing in various directions. It's a fact about dynamical processes that the places where the gradient vanishes organize the trajectories. So in 2 dimensions, maybe that seems a little counterintuitive, but you could think about it like this. So if instead I go back to 1 dimension, so let's just take a curve, And now the places where the gradient vanishes are the maxima and the minima, right? Or the saddle points. And I mean, I taught calculus for many, many years. And a large part of the course is teaching people to classify the minima and the maxima because, you know, the other points in the curve are kind of boring, right? Like you want to understand some function, like a cost function, you want to know where it's maximized or where it's minimized. What it's doing, like at some random point on the curve, is completely irrelevant. Like you just want to know where the maxima and minima are. So that's an instantiation of the principle that if you're trying to understand a function, you are often looking for the places where interesting stuff happens, and those are maxima and minima in 1 dimension. And in higher dimensions, they're singularities. So they're places where the gradient is 0 in all directions. You can never get there because no trajectory that starts from outside the singularity ever arrives, and if it's at the singularity, it never leaves. But still, if you follow a random trajectory following a potential, and there's like some singularity somewhere, it's going to do something around it. And like, this will dominate the behavior. So the trajectories will tend to approach it and then escape in some particular direction. So the singularity is organizing the set of global trajectories. And that's a kind of informal statement, but there are formal statements that you can use ideas from topology and sort of So that there's ways of making that more precise. That's a general principle, and it applies for learning with SGD just as much as it applies to any of these other systems in physics to which people would apply this principle. So that's like a general rationale for caring about critical points or singularities. These are almost synonymous. So that's a reason for caring about those as the organizing principle for dynamical systems and learning in particular. Now, like what it looks like for an actual learning process in deep learning to be governed by singularities. Yeah. I mean, that's sort of like, you know, in really interesting cases where there's kind of like a model doing interesting stuff is, I mean, it's a very complex question, but in simple cases you can see that, well, Yeah, like this toy model of superposition work I was describing earlier, the columns of the weight matrix. So you can visualize those columns as vectors in 2 dimensions. So there's 2 rows of this matrix and some number of columns, and you can just plot those columns as vectors. And then you get sort of shapes like a pentagon or a square or a hexagon, connect the vertices of those vectors. And then moving between the neighborhoods of these critical points literally corresponds to like growing a leg or contracting a leg and rearranging the other ones in some particular way. So there's, if you like, the sort of growth process of the structure of the network is dictated by the movement between neighborhoods of these 2 different singularities. So that's That's in a very simple case. In a more complex case, yeah, I think it's not so clear how to think about that. And these singularities aren't like isolated things where there's like one singularity here and normal stuff in between. In a large model, there's kind of singularities everywhere and it's, yeah, not well understood.

I'm almost envisioning a sort of fluid dynamics type of thing where the the point that we are in parameter space kind of speeds up when it gets around the drain, but then maybe spins out and goes slower as it has some wider orbit. And probably a lot is lost in that, but that's at least the visual that's coming to mind.

Another— Yeah, there's this, you can visualize the function in function space of a small language model. And there's some parts of the trajectory that actually look like it's orbiting around some particular kind of mode of prediction.

Yeah.

So that kind of image is appropriate.

So do these singularities, I mean, in a simple cost function, as you're describing, maximise it, minimise it, whatever, it's very clear what that point means and why we're trying to identify it in the first place. In the case, especially of a more complicated neural network, I guess, I'm tempted to ask, is that the sort of, if I imagine a Grokking type phenomenon, are those singularities the place where the algorithm is perfectly Grokked such that there's never anything more to learn, at least with respect to some subset of problems? Or is that reading too much into the nature of the—

Think of the singularities as different ways of solving the problem. Like different kinds of solutions at a given level of loss. So if you're at a given point in the loss landscape, different kinds of singularities correspond to different ways of predicting on the data you've seen so far. Jesse, maybe you want to say a bit about the ED stuff?

Yeah, I think it's an interesting one. Um, yeah, maybe I'll say one more thing about this, right? So coming back to generalization, One thing we're worried about is your model might learn 2 algorithms that look identical from the training data, but one of them generalizes in a way that you don't like, and one of them generalizes in a way that you do like. Okay, so how do you distinguish them? Well, what Dan's just saying is that maybe if you look at the neighborhood in the loss landscape, they actually have different geometries associated to them. And so this is the sense in which reading this geometry gives you information about what kind of algorithm do you actually end up with. And so what Dan was referring to as ED stands for Essential Dynamics. This is a paper we put out earlier this year where you can train a neural network, a transformer, to do in-context linear regression. So you can give it x, y, x, y, x, y samples and ask it to predict the y's from the x's, where each y is generated from a simple linear transformation of the x's plus some noise. Okay, so what happens is the model, if you train it on lots and lots of different samples of this, will learn to do regression over the course of context. However, what you can do is you can vary the number of distinct tasks that the model is exposed to during training. If you expose it to many different tasks, it learns regression. If you only expose it to a few different tasks, it'll memorize those tasks, and it won't learn the regression solution. It'll learn this memorization solution. Now, which solution the model ends up with varies as you vary these 2, the complexity or the diversity of the number of tasks. And you see a phase transition where the model initially memorizes and then eventually learns this other solution. But the selection process between these is something you can actually try to understand with singular learning theory. So each solution has an associated performance and complexity. Memorization is always better at performance than this regression solution, which I call the generalization or meta-learning solution. Generalization solution, however, is simpler past a certain level of things that you have to memorize. And so what happens is the simplest solution roughly occupies more volume in parameter space and is therefore easier to find. And so you notice this phenomenon where the model will first learn the generalizing solution, the simple one, before moving to the memorization solution. You can classify exactly which kinds of dynamics qualitatively you should expect to see based on this trade-off between performance and complexity, where complexity is reflected in the degeneracy of the loss landscape.

So that's kind of the opposite order to Grokking.

Yeah, I was noticing that for sure. So, yeah, I was so caught on that. Maybe let's take— do another version of that and help me understand how it's happening in the reverse in this case versus the Grokking case.

Grokking is actually a strange phenomenon.

Yes.

It's not like a typical— I think there's been, in a sense, like overfitting to the example of Grokking. This, at least in our opinion, is Kind of not the typical way in which memorization and generalization probably interact.

Okay, say more.

What's special about Grokking is that the performance on the training set is more or less identical. Because you have one solution that is more complex, the memorization solution is more complex, But it's the thing you learn initially. So you end up with this transition where at a fixed level of loss in the loss landscape, you end up in a broader basin and in a basin that has more degeneracy.

This is a phenomenon that can only happen. I mean, this is sort of, if you get to conceptually the very end of training, so at the very bottom of the loss landscape, then there are some perfect solutions that are simpler and more complex, and you can you know, you prefer the simpler one if you can find it, and eventually you will find it. But the trade-off between memorization and generalization that occurs more frequently, like without such sort of like artificial conditions, is more like you prefer the simple solution that's bad, has high loss, you eventually trade that off for a solution that has lower loss and more complexity. You pay a complexity penalty. penalty in order to get the lower loss. That's the more typical relation between memorization and generalization.

Yeah. And that's the case in this in-context linear regression example. So what SLT says is you should expect models to minimize a sort of effective loss, where it's the loss with this emergent regularization term, this implicit bias that comes from degeneracy that comes from the number of ways you can perturb weights without actually changing the function. And so typically, the model is just going to prefer initially to learn the simple bad thing before moving on to the complex better thing.

And so help me understand one more time how the Grokken case is different.

Because in that case, it takes a lot longer to learn what I would think of as the simpler algorithm. Which presumably does have, in terms of some formal complexity metric, I would assume that that does have a lower score for complexity than memorizing n examples, right? Yeah.

So as Jesse referred to earlier, there's kind of 2 typical changes in the trade-off between loss and complexity that are predicted. by the core mathematical result of SLT. By trade-off, I mean trade-off between the loss and the complexity as estimated, and we call that the LLC, local learning coefficient. So I'll just say complexity. So one way you can, like a preferred trade-off, is to pay the penalty of increasing the complexity by lowering the loss. That will decrease what's called the free energy, which determines which solution is more preferred from a Bayesian statistics point of view. So that's what we sometimes refer to as a Type A transition. That's like the typical— that's just like learning, right? Like you learn more stuff, you get better.

Okay.

But there is another way to decrease the free energy, which is to like decrease the complexity at a fixed loss. So this is what we refer to as a Type B transition. It's hard to ascertain exactly like the conditions under which this is the case, but Grokking seems to be one of them. We've seen others. So those are situations where a model really isn't improving on the loss, but it simplifies the algorithm that it's using. We can see this in the linear regression setting in some sense as well. And of course there are many other examples of Grokking-like behavior. But yeah, so that's the underlying principle is you should decrease the free energy, which is a— sum of 2 terms, one to do with the loss and one to do with the learning coefficient. And you're allowed to increase one of those terms if you decrease the other one enough, that will decrease the free energy. That's normal learning. And then there are situations where you can have a similar decrease by just decreasing one of the terms, the complexity.

So one thing that's jumping out at me is like, you know, in general, right? Obviously, people can do all sorts of variations on this, but the sort of vanilla setup is you're just minimising the loss. And just, you had used this term, I think it was implicit regularisation. So if I'm sitting looking at my single loss function and trying to drive it as low as I can, I understand what weight decay is when it comes to— and that's like one form of this sort of regularisation. Right. But if I understand you correctly, you're saying that there is some sort of natural pull that is happening, even if I am, from an external perspective, just minimising loss, that is moving me around in this complexity space and prefers simpler solutions. But I don't have an intuition as of now for why or how that's happening if I'm not to, you know, taking an active step in the learning process to enforce that.

You can imagine asking the question, what happens if you randomly drop yourself somewhere in the loss landscape? More likely than not, you'll end up somewhere with pretty high loss. You'll also end up somewhere that has a very broad basin, right? There are many different ways you can perturb that solution that'll leave the loss basically the same. And so generally, as you move down the loss landscape, it's going to be easier to find the solutions that occupy a greater volume. And the main thing that contributes to the volume is the degeneracy, so the number of valleys. It's just easier to find canyons that have more valleys than it is to find very narrow nooks and crannies. And this is the sense in which you get this emergent implicit regularization from the structure of the loss landscape.

I could do this all day in terms of just sitting here trying to visualise these things and asking you to help deconfuse me. And I think we have a lot of those listeners too, though I probably need to be somewhat disciplined about how many more of those kinds of questions I ask. This sort of seems to connect very fundamentally to double descent, right? Or kind of just these sort of— yeah, I mean, maybe just riff on double descent for a second, and then I definitely want to take a moment and also really zoom out, right? I mean, it's all fun and games and modular arithmetic and toy problems to motivate the entire thing. And I'll probably put this in the intro. The sort of canonical safety worry is you can't necessarily tell the difference between a superintelligence that has your best interests at heart and one that doesn't. And you have a hard time You may have a real hard time in many ways if you can't make that distinction as you get to more and more powerful systems. So there's the double descent thing. But then let's do the zoom out and kind of talk about the biggest picture worries that we have. And then I also think maybe we should kind of start to go toward, like, what is the path for you guys from here with this research agenda? We've seen from Anthropic, I would say, and others, of course, from whatever toy models of superposition 2.5 years ago, really an unbelievable amount of progress in my view. Obviously, a long way still to go, but if you'd told me then that we'd be here now, I'd be like, that's really amazing. And it has struck me that one of the big things that has gone into that has just been a ton of compute.

Yeah.

And I don't have an intuition yet for whether that is sort of a similar trajectory that you guys think you will follow, or if it is going to be a more sort of eureka moment to recall the Greeks again driven process. So, yeah, a lot there. But Double Descent, if you want to say anything about that, what are the biggest— if we zoom out to not toys, but the real things that we're really worried about, and how do we sort of go from these toy model understandings to starting to tackle that, hopefully on relevant timelines.

Regarding double descent, I mean, there's various kinds of double descent. So there's double descent with respect to training samples, there's double descent with respect to model size. I think nobody has systematically attempted to study double descent with singular learning theory yet. We haven't really tried. I think partly because it just doesn't look that mysterious when you have approached the problem from the mathematical perspective that we do. It isn't mysterious that models with many parameters can generalize well.

And in terms of an intuition, is this another one of those dimensionality things where when I see a hill, I'm like, boy, it looks like a lot to climb over that hill. But when I'm in 200 billion dimensional space, there's always kind of a way to wind my way through it.

I mean, that's an often presented intuition for why optimization in high dimensions is amenable. I am not so sure I think that's really an explanation. I think that's closer to a folk story maybe. I think it's not understood why. I mean, just having a large model with lots of parameters, like you can of course find things that don't optimize well that have lots of parameters that aren't neural networks, right? Or you can get the neural networks configured badly and they won't train very well. So there remain many mysteries why large neural networks can actually find well-performing, well-generalizing solutions. Singular learning theory doesn't resolve all those mysteries, right? But it does at least give you a mathematical framework from which you're not surprised that there can be models with lots of parameters which do well and predict well and generalize well. So I think I don't really have a clear answer for you on double descent. Maybe one of your listeners wants to think more about it. We haven't put a lot of effort into it. There is a story in— you can think about double descent with respect to training samples, And see that you can back out a picture that looks somewhat like the double descent curve from generalization curves that are associated to one of these kind of transitions I was describing earlier. So potentially there's an explanation there. I don't know.

I'm just worried that at some points we've gotten too deep into the technical details. So if I can take a chance to re-clarify or restate some things I've said earlier, Yeah, sure. Always. Try to add to that more. Okay. So big picture, we want to end up with a friendly AI and we don't understand enough about what's going on inside of the model to currently make any kinds of guarantees of this model is aligned, this model is not aligned. They can exhibit the same behavior in training and then generalize differently when deployed. And so for me, the basic premise of interpretability is understanding model internals to a sufficient degree that we can disambiguate these 2 cases, and that we can understand generalization. And SLT tells us that we can look at the geometry of the loss landscape. We can look at how structure is reflected in this geometry to start getting at this, at this question. With this principled link to in-distribution generalization, that's a starting point for developing a theory of out-of-distribution generalization. And understanding better what happens in the case of SGD. So on the interpretability side, what this looks like in practice is you can probe a bunch of points in the loss landscape, apply small perturbations to your model, you know, hit it with a hammer, and see how it responds to that probe, to that small perturbation. And this tells you something about what's going on internally. And this is the basis for starting to extract information about model internals from geometry. So we've been looking at a series of projects, papers, where we study first small language models, subject them to these kinds of perturbations, and then measure things like how much does the performance decrease? How much worse does the performance get under these kinds of perturbations? That tells you about complexity of the solution the model has learned. How much does performance increase if I apply these structured perturbations to a specific part of the model, an attention head or a component? And this It turns out can tell you something about this attention head is specialized to this kind of data. These attention heads are all induction heads. These ones are doing something else like memorizing different n-grams and skip n-grams. And so you can start to actually tear apart the internals of the model using these kinds of probes. More recently, we've been looking at applications for something like circuit discovery. So in a principled way, can you associate the components of the model with patterns and data, and do this attribution from what structures the model ends up with to the data that activates that structure. And this is starting to get at the kind of understanding that could lead to an understanding of why the model exhibits the behavior it does, rooted in its internal structures, without necessarily having a full mechanistic understanding of what's going on. You can do the sensitivity analysis, And it can tell you that these components are involved without needing to know exactly causally what the input to output map was. So this is more like a top-down approach to interpretability informed through this perturbation analysis. Okay, so on the one side, we want to develop tools for interpretability. On the other side, at some point, we want to be able to steer the learning process so that we don't end up with a misaligned model in the first place. We sort of have to take a preventative approach. Well, ultimately, that's going to look like intervening in the training data. And so the hope is, to the degree that the mapping from training data, through geometry of the loss landscape, through the learning process to the final weights you end up with, is invertible, to some extent, can you go backwards and come up with techniques for choosing training data that aligns your model more robustly? And so these are the kinds of applications of SLT that we're trying to develop for alignment. We're only just getting started here. But what our real hope is for this agenda in the long term is to give us better ways to make models more aligned.

So are those sorts of bell-ringing exercises— I'm imagining as you try to scale this up to larger and larger systems, You would maybe be running experiments like, if we add some noise, how often do we see behaviour that we didn't want? Or sort of has some maybe connections to AI control type ideas too. Like, how often does the thing take the bait? If we set up a situation where there's some bad thing that it could do, how often does that happen with different perturbations? And the less it happens, maybe the more aligned we are. Is that the sort of— measurement that you would imagine taking?

Yeah, I don't think I'm going to bite on that particular proposal, but if you think that you can elicit something about the structures in the model, which at the end of training you have some understanding produce the concerning behavior and trace the origin, to go back to the developmental perspective again, If you can trace the origin of those structures and behaviors to some kind of pattern in the data, then you can try and intervene on that using your understanding of that pattern, intervene at some earlier point to shape that structure differently. Now, obviously that's, you know, for a complex behavior and very complex internal structure, an ambitious thing to attempt, right? And so that's one, that's one of the reasons why some progress on interpretability from our point of view is essential to unpacking that, right? So if you imagine some complex mechanism which is producing the undesired behavior, well, you're not just going to— I mean, I think it's naive to pretend you're just going to find the be-good samples out there and throw those in the training process at 12,000 steps and be like, be good, be good. I don't think that's sufficiently fine-grained, but if you have some understanding of how that complex behavior came together, right? So this mechanism, how it was shaped over development and what factors went into that shaping, then it's not crazy to imagine that there's something analogous to morphogens. So morphogens, I mean, this is a sort of umbrella term that refers to factors in development of biological organisms that shape When things happen or what things happen, right? They could be particular molecules or proteins. And these dictate the way that development works in biology, right? And if you insert the right morphogens at the right time, you can change the outcome of development. And so this is like maybe one guiding analogy for why it makes sense to hypothesize that if you understand the developmental process well enough, you might be able to intervene at various points strategically to change the outcome. And indeed, that's how synthetic biology works. You understand the development process, you understand the morphogens, I mean, some parts of synthetic biology, and then you can program the development in a way you prefer.

One question on just the nature of overgeneralization. Is reward hacking an example of overgeneralization where the model has sort of what we What we wanted to specify was some richer concept than the signal we gave. And so it sort of has gone too far in understanding the true nature of that signal beyond what we meant it to be. And so that's why we see reward hacking. Fair?

I think it could be. I think there's like a bucket of things that are— I mean, reward hacking has a technical definition under which the answer to your question is no. So, like, the technical way that people would introduce reward hacking is simply to say that it has found a way of getting more reward than the solution you intended, right? So that's purely about the reward, which is somewhat analogous to the loss, right? So, like, that's not really about complexity. That's just about getting more of the actual explicit target of the optimization process. So under that definition, the answer to your question is no. Like, reward hacking is not an instance of This trade-off we were referring to earlier, which you're terming overgeneralization. However, it could very well be, I think, that instances that we are currently referring to as reward hacking are perhaps partly a trade-off in favor of a simpler solution rather than the one we intended. I don't claim to have evidence for that, but I would maybe hypothesize that. But yeah, I think you're sort of— Referring to one of the hypotheses we have for one of the ways in which dangerous behavior could arise, which is somewhat different to the usual story of how dangerous behavior could arise from sort of over-optimizing a reward target, which is to go back to the experiments that Jesse was referring to earlier very briefly. So there, there was a simpler solution, which was ridge regression In theory, if you train long enough, you should always switch to memorizing the data because the optimization pressure wants you to get the lowest loss possible. The lowest loss possible is to memorize the data if you can, right? Like typically, like training a GPT model, you just can't memorize the data, so you have to generalize and so on. But if you can memorize, that's like the best thing to do given the training signal. But what we see actually in that experiment is that in many cases the model sticks at the generalizing solution and never switches to the memorizing one. So that's a situation in which, if you imagine the loss landscape, and to just draw a curve with my hand despite my complaints about that earlier, so if you imagine like at the very bottom of the loss landscape there's the thing that the training data actually specifies as the optimal solution, That in this case you can prove mathematically is to memorize the tasks. But there is a higher loss but simpler solution, which doesn't memorize but does some more general thing. And training can get stuck there forever, even though this is the global minimum. I mean, I say forever, obviously we didn't train forever, but it seems like it would stay there forever. And this is a sort of, I think, A good heuristic to have, contra some of the normal stories about AI risk, that it may be that you don't get what you ask for. You don't get what you try to specify in the training data. You get a simplification, you get a heuristic, you get an approximation, which is not what you intended or expected. That could be good. It could be bad. So I think Mostly in the context of AI safety, people have an intuition that actually the simpler thing will be better, right? Because they have a— so just to communicate an often told sort of way of thinking about this, if you imagine the model becomes scheming, becomes a kind of scheming agent, Even though you didn't optimize for this. So it's trying to do the task, it has structure in its weights to do the task, and then it needs to like tack on some additional structure to take over the world, right? And because that would be more complex than the solution that just does the task, then okay, and we have a simplicity bias, that's great because we have this very strong pressure to remove this additional stuff that is going to be dangerous. That very well may be a reasonable way of thinking about it. I think one should have not very much confidence in this. This is like quite an imprecise story, not grounded really in very much like mathematically. So that's potentially the case, but there's another, just to tell a sort of opposite story, it could be the case that the simplification is quite dangerous. And if you'll indulge me, there's actually a historical example here that's really quite interesting. So There's something called the Windrush scandal. So the UK in the 1940s, I think, allowed citizens of the British Empire to come and settle in the UK without really any documentation. So many people came to the UK, I think primarily from the Caribbean. Then in the 1970s, they switched the laws. So they started insisting that people have documentation of their citizenship in order to access services. And these people who came as part of this program didn't have the relevant documentation. And some of them were deported and many bad things happened. And there was an investigation and sort of scandal in 2018, I think, um, looking into this. So this is a case where you have a policy, which is like, obviously the intended policy of the state is not to like deport people who it let into the country earlier.

Right.

But when you follow that policy, the way it's actually implemented at the kind of street level is a simplification of the original intent. It's this kind of very brute judgment call at the level of the street-level bureaucrat. And that's what this kind of phenomenon is referred to, street-level bureaucracy, which is a simplification of the thing that the original policy was trying to specify. And this ends up having unintended consequences, which are negative. So I think there's a case to be made that in AI safety, we should also be quite concerned about what happens if the behavior we are specifying at great expense by collecting training data ends up being like some kind of simplification in various ways we don't anticipate of the intended target behavior in ways that are undesirable.

Yeah, seeing like a state and seeing like a model may be more similar than different, certainly relative to what I had conceived of previously. I think I maybe have 2 more questions. One is, I've heard Jesse kind of tell the story in the past in other fora around, like, maybe model training in the future could look more like like refining oil does today, where it's a very sort of— you start with a big mess, but you can sort of figure out exactly what components need to be added when you have a sense for when these phase transitions are happening, and you have just a much more precise level of control over that process. I'd love to hear a little bit better version of that story than what I just offered. And then my other one is just, what do you guys make of other safety efforts? I mean, you've sort of said, like, Probably can't just get to be good examples and hope for the best, or at least that doesn't seem adequate. But I'd be interested in your sort of handicapping of all the various, or at least some of the most prominent various techniques that are out there today. What do you think has the best hope of working, or what do you think is doomed? So, yeah, kind of your vision for the future of model training and the level of control we might really be able to acquire over that process. Maybe you could talk about how you scale and the compute, because I don't think we ever touched on that. And then this handicapping of the landscape. There's a lot there, but I'll let you have at it.

Yeah, I can start with the oil refining example. So I think it's useful to compare current deep learning to alchemy. We've got some huge cauldron, that's the architecture. We've got a fire, that's the optimizer. And then we've got the reagents we put in this cauldron that we start to mix together. Those reagents are the training data. And currently we just throw the entire internet together into this cauldron, start stirring, and hope that we haven't accidentally mixed bleach and Clorox.

Okay.

So what we have in mind for the future of training looks something more like industrial chemical manufacturing, where you know exactly which reagents you're mixing together at what concentration. at what point in the process and with what catalyzers and ingredients. And this level of control is not something that's accessible to us right now, but it's something that we're already empirically starting to develop. So we have splintered the post-training process into a bunch of different stages. There's, you know, Chain-of-Thought RL 1, there's a second stage of Chain-of-Thought RL, there's a little bit of instruction fine-tuning maybe somewhere. There's constitutional AI 1, there's the second stage of constitutional AI just for the personality, then maybe some word refusal training. Similarly, on the pre-training side, this process is being split into multiple stages so that we're developing an empirical understanding of how to better control the learning process. And what we, we, we think is possible is that you could develop the kind of scientific understanding to actually know when you should mix 2 samples in a batch together. to get the desired behavior out the other end. And this would give you more control over the entire process and what you end up with.

Yeah, just to interject one example of that, that struck me recently with the Claude 4 system card, there was this, I'm sure you've seen this anecdote, it hasn't been very deeply explained, although Sam Bowman talked about it a little bit online. They reported having observed that the model was following harmful system prompts in a way that surprised them. And then they're trying to figure out why. Sure enough, it turns out that they had omitted the sysprompt harmful dataset, which had been developed specifically to teach the model what to do when given a harmful system prompt. And so you have the sort of simpler solution of always follow the system prompt is what it ends up learning for lack of that curb being included as it was meant to be. Interestingly, from my perspective, they did not go back and retrain. I think there's a lot I'm still kind of unpacking. Like, what exactly should I be inferring about the world on multiple different dimensions from the fact that they didn't just say, oh, shit, messed that up, revert and pick up where that thing was supposed to be introduced and kind of continue on. Presumably that means it was happening somewhat earlier in the process than some very late finishing stage. Presumably it also means they have constraints at Anthropic in terms of either compute budget or timeline to launch for competitiveness reasons or whatever, that there's a whole interesting analysis to be done there too. But that's at least one of these sort of very recent and concrete examples where, without necessarily a full theory to drive it, that is starting to kind of emerge. And you better make sure you get those datasets in the right place at the right time, or now it honestly seems like we have sort of a Claude. I had a little private conversation with somebody at Anthropic just to ask for clarification on this, and they basically said, we're confident we got the value of that dataset in the end. And I was also kind of— so I'm like, okay, that's interesting. I sort of believe you, and you definitely know more than me. At the same time, how can you be confident? Like, what do you even, you know, on what basis could you have any confidence about that other than obviously poking at the model a lot? But I mean, that's the sort of confidence, I guess, that you hope to create in the process of continuing to do this work.

Yeah. I mean, I think it's, it isn't an unreasonable expectation on behalf of like all of our civilization's about to be transformed by this technology. To have it be somewhat more of an engineering science and somewhat less of a big— like, not that it's— I mean, it's not like nobody understands anything, right? Like, the data mixtures that are selected and the order that they are selected is, like, empirically very finely tuned. Lots of effort goes into that. Selecting hyperparameters has some underlying theory to it. So, you know, lots of people and lots of money is being poured into deep learning, obviously. So it's not completely incomprehensible, but I think most people would agree that it would be highly desirable to have it be much more understood and much more like other aspects of engineering, which we submit to safety engineering, than deep learning is currently. That seems hard to disagree with, I would say. That's not an easy thing to do, especially in a short period of time. To come back to your question about scaling and the situation we're in vis-à-vis kind of where Anthropic was with, say, SAEs a few years ago and so on. I think this analogy seems pretty apt to us. I mean, I think it's been quite a difficult path to get from sort of fairly pure abstract theory to validate it in real systems. And then, I mean, now we're just starting to do things like circuit discovery. We can find induction circuits and some simple things like that in a kind of unsupervised way. Does seem like there's, you know, scaling to do, and that's one of the main things that now remains, at least to get up to a point where we're finding sort of complex internal structure. Yeah, in terms of compute, that's a little difficult to say. I mean, the compute overhead of computing SAEs, for instance, where you train the SAEs, and that's very expensive.

Yeah.

Our compute burdens are going to be in different places, but I imagine it's still very substantial to try and understand large models. We don't really— we have been studying models like up to 7 billion parameters and so on.

We can definitely use more compute. If a listener has compute to offer, please.

Yeah, that's right.

Maybe something to add here. So in particular, if you have more compute, You can draw more samples, apply more perturbations, and that gives you a finer signal about what's going on inside of the model. And so we expect that generally, yeah, we will be able to translate more compute into a finer understanding of structure.

Yeah. Regarding other sort of approaches to alignment, I mean, I think in many respects, the things we're describing are just useful foundations for many different things you might So I guess that's one way we think about it. So for instance, the stuff we're doing around shaping the data distribution and so on is potentially useful for some of the work the UK AI Security Institute wants to do around elicitation. So we have various ideas about how the work we're doing can— I mean, some of it is just kind of good solid foundational science of understanding models and understanding the training process and how model behavior works. And that in the same way that interpretability is broadly useful, much of that is broadly useful. I guess, yeah, it seems unclear whether the current paradigm of shaping model behavior at the end of training is going to be a very robust solution to alignment as we move towards higher capability levels. I mean, this is an oft-expressed concern, right? So I think that something that looks like baking in behavior and control of behavior earlier in the training process seems like something that is broadly going to be adopted. I mean, arguably the deliberative alignment approach is, especially if you scale up the amount of compute spent on RL, right? If that is equal to pre-training, then that's a lot of shaping of model behavior that is taking place over a large part of the effective compute.

And this idea is not just due to us. There's papers like Pretraining from Human Feedback. A lot of people realize that we need to bring alignment earlier in the learning process. Maybe I can end with a bit of a look to the next few months. So currently what a lot of our effort looks like is scaling techniques up to larger models and starting to just run lots of experiments at larger models. So we've validated these circuit discovery techniques in small 3 million parameter language models. Now, can we do this in 7 billion parameter language models? We think this is quite likely to work by the end of the year. On the alignment front, we're starting to do our first experiments in terms of applying these techniques to try to steer the learning process. This is still early days. We'll probably have early signs of life by the end of the year in small language models. And then we're developing these applications for things like elicitation, data attributions. So there's a singular learning theory extension to influence functions that you can study. It gives us ideas about, again, influence of samples on other behavior and trying to use these kinds of applications to really validate that SLT and the techniques we're developing have something to offer that goes beyond existing techniques. That's where we're headed. Cool.

I love it. This has been fascinating. I appreciate you indulging me in so many little side questions and attempts to develop my own intuition. And I certainly think the big picture need for a proper developmental understanding of how these AIs are forming and any theoretical basis at all for what they're going to do when they get into a A truly out-of-distribution situation is badly needed. And especially as we, you know, get closer to 2027 and beyond, it's going to really help me sleep well at night. So, to quote my dad, who often quotes Leslie Nielsen from Airplane!, good luck. We're all counting on you. Anything, anything we didn't touch on you want to leave people with real quick?

Yeah, I guess if you want to learn more about singular learning theory and read about some of the papers we discussed, some of the papers we didn't discuss, you can go to our website. That's timaeus.co, T-I-M-A-E-U-S dot co. There's also a Discord server link there. So that's in particular the place to learn more. And we have a weekly seminar and plenty of opportunity to get engaged with the research going on. Cool.

Yeah, I've typed it wrong more than once myself. So it's T-I-M-A-E-U-S And it's a, it's a brain-stretching exercise, at least for me, and probably for most of us. But I've definitely enjoyed stretching my brain in this way. So look forward to future progress. You know, we can do a check back in maybe in a year's time or whatever and see how much we've closed the gap on all these important questions. But for now, I will say, Jesse Hoogland and Daniel Murphett, founders of Timaeus, thank you for being part of the Cognitive Revolution.

Thank you, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.