Mechanistic Interpretability: Philosophy, Practice & Progress with Goodfire's Dan Balsam & Tom McGrath

Hello, and welcome back to The Cognitive Revolution.

Today, I'm honored to be joined once again by Dan Balsam and Tom McGrath, CTO and Chief Scientist at mechanistic interpretability startup GoodFire. When we last spoke, about 9 months ago now, we focused on the technical foundations of interpretability, including the challenge of polysemanticity, techniques such as activation patching, causal tracing, and feature editing, the rise of sparse autoencoders, and some of the challenges of scaling these techniques to frontier models. If you're new to mechanistic interpretability, I would definitely recommend checking out that earlier episode for a technical primer. Since then, GoodFire has gone on to train sparse autoencoders on LLaMA 3.3 70B and also DeepSeek R1, strengthened its team with the addition of multiple top-tier researchers, and recently announced a $50 million Series A. Which notably includes Anthropic's first-ever investment in another company, giving me, as a small-time seed round investor in Goodfire, both strong on-paper returns and a bit of bragging rights. In today's conversation, we mostly zoom out from specific techniques and findings and instead try to get a handle on the state of mechanistic interpretability as a whole. For years, the field has been called pre-paradigmatic, but as you'll hear, Tom now describes it as proto-paradigmatic. There's now general agreement among researchers that neural networks contain understandable things. That these things, called features, can be understood as linear directions in embedding space, and the magnitude of their activation represents their intensity. There's also the finding that superposition allows models to represent far, far more concepts than they have dimensions, and that features connect through the layers of the model to form circuits. This is great progress, and honestly much more than I might have expected just a couple years back. But there are still important gaps between the accounts that interpretability techniques provide and the underlying reality of model structure and behavior. First, and most obviously, there's the fact that interpretability techniques typically attempt to reconstruct the behavior of the underlying model. And as of now, they can do so only very roughly. Second, and more philosophically, there's this distinction between the features that interpretability techniques learn and the meaning that we assign to them in the process of labeling. In practical terms, when we say that a feature represents the Golden Gate Bridge or, more to the point, deception, how confident can we really be in that label? From my own exploration of both Goodfire and Anthropic's interactive interfaces, this seems to range very widely. All of this is complicated further by another all-too-often neglected fact: the models under study in encode varying degrees of understanding, with everything from simple memorization to fuzzy heuristics to proper algorithmic grokking all occurring simultaneously in an unknown mix in any given model. Of course, while the philosophy is fascinating and there's still clearly a ton of work left to do, that is not stopping Goodfire from deriving practical value from interpretability techniques today. And Dan describes how Goodfire is developing applications, both for enterprise customers and the public good across 3 key domains. Scientific discovery, where they're partnering with organizations like the Ark Institute to explore genomics models like Evo 2 and beginning to uncover novel biological insights. Guardrails and safety, where they're developing inference time monitoring applications that can detect when models might output harmful content or exhibit other problematic behaviors. And creative applications such as their just launched Paint with Ember tool that allows users to generate and edit images by directly manipulating sparse autoencoder features. Proto-paradigmatic though it may be, as we enter a new era in which science shifts towards simulation-based approaches and AI systems potentially drive more and more of the machine learning research, it seems to me a very safe bet that interpretability work will become more and more important. As Dan put it, even if we end up in a scenario where a data center full of geniuses is doing most of the scientific work, Mechanistic interpretability might be their preferred tool for understanding both their discoveries and themselves. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Your feedback is always welcome too. Feel free to reach out anytime via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this thought-provoking exploration of the philosophy and practice of mechanistic interpretability with Dan Balsam and Tom McGrath of GoodFire.

Dan Balsam and Tom McGrath, CTO and Chief Scientist at GoodFire. Welcome back to The Cognitive Revolution.

Thank you so much for having us.

Thanks. Yeah, great to be on here.

I'm excited. We haven't been able to make this happen quite as often as I would've liked, but we're gonna make up for it by going long and in depth. today. So really excited to get the update on what you guys are building as a company, which I understand there's some great news on, and also just to check in on what we have learned as a community about models and how we understand how they work over the last few months, because obviously there's nowhere in the world changing faster than that. So for starters, I wanted to kind of go high level and Just ask you to kind of frame the field. I mean, I think everybody in the general ML space at this point has internalised this data compute and algorithms paradigm. These are the 3 legs of the stool that are enabling progress. There's this sense that they all kind of contribute equally. And then on the interpretability side, I'm sort of tempted to slot in models for data and say, like, models, compute, and algorithms are maybe the things and see Seemingly a lot depends on the quality of the models, but then there's still a role for data. So how do you guys think of the fundamental inputs of what you're doing?

I think one thing that's interesting here is that these inputs are very important and the models have changed and so things sort of changed along with them. But another thing is it's a very empirical activity in the sense of you're looking in a fairly fine-grained way at the data of what's happening inside models. So, like, progress in algorithms, for instance, parameter decomposition or SAEs or whatever, we can get into these later. But one of the inputs here is actually, like, very rich empirical data. You know, when you just, like, look basically, like, what are activations like? When we, like, sort of tinker around with models in a relatively hypothesis-free way, do they actually seem to behave? And I guess, like, this sort of empirical input is always, like, Part of progress in algorithms, but I think I just want to upweight it for interpretability because we're really doing this sort of natural science. But yeah, I broadly agree with the decomposition of models of the data, I suppose. And then there's compute. And I think as a field, we would like to be able to use much more compute. And then there's algorithms. And I think that, I think it's a good decomposition. I would say that we don't have the transformer, but for interpretability. And it could be that the thing that we are blocked on for finding the transformer for interpretability is simply understanding models better in a way that we can then generalize to a new inductive bias.

Yeah, that's really interesting. And you kind of prompted me also to try another paradigm mapping exercise. You said we're doing this in a relatively hypothesis-free way. That sort of maps in my mind to pre-training, sort of unsupervised learning, right? The sort of SAE paradigm has largely been run a ton of data through, basically from the same original dataset. You could tell me more about how that's curated, maybe especially for interpretability work, but in a relatively hypothesis-free way. Is there any equivalent to post-training yet in the interpretability world?

That's interesting. When I say hypothesis-free, I mainly mean a person sitting down and tinkering with models and being like, What structure is there? So for instance, like this kind of famous Not All Language Model Features Are Linear paper, they're sort of finding structure in activations. And this structure is kind of interesting. It's this kind of higher dimensional manifold. And you don't necessarily get to this by being like, I have a hypothesis that things are this way. You just sort of, like a natural scientist, like someone out observing the stars or something, they kind of, they have some hypotheses in mind. But they're also just going, they're just sort of looking at things. And so that's what I mean. But it's interesting that actually I also am very keen on the unsupervised approach to, the unsupervised approach to kind of interpret models like SAEs for basically the same reason, except that every architecture is kind of coming with a hypothesis, like the SAE inductive biases, things are sort of, well, the SAE is like a fairly strong hypothesis that features are Literally directions in embedding space.

Yeah, there's something really interesting there around, and again, the model quality seems to be really important, right? Because a lot of the kind of hypothesis-free tinkering, sort of even vibe coding, right? I mean, Neil Nando recently put out a video of him doing some vibe coding research, and that's all kind of premised on the idea that you can run a little experiment and get something back relatively quickly on a sort of iteration timeline that where ideally you could sustain focus. Maybe you have to go take a walk and come back, but it's not like really long timeframes or really large compute budgets. Whereas, as you mentioned, you got to have some conviction to throw large amounts of compute at an SAE at scale. But I wonder, how do you think about the sort of challenge of, can you do that rapid experimentation on the truly large, advanced Or are you limited to working with something like GPT-2 scale? And does that sort of create a fundamentally different regime from the things that you end up scaling up?

It's really a question of infrastructure. If you have no infrastructure, then it's hard. If you have the right infrastructure, then building that infrastructure is hard, but the right infrastructure makes experiments relatively easy.

Yeah.

The other thing that I'll add is when thinking about unsupervised techniques as hypothesis generators for how the model could be working, there's no way that we're going to be able to scale to superintelligence without making our interpretability techniques unsupervised. And that's one of the things that really motivates us and is our mission at Goodfire. Narrowly, superintelligent models already exist in the scientific domains, and this is what we spend a bunch of our time working on. When you're working with a genomics model, you're working with a model that we have priors about. There's lots of bioinformatic research to attempt to understand the genome, but we're also working with systems for which our statistical techniques that we have are not as explanatory as we'd like them to be. And that's the motivation towards moving towards AI to begin with. And so in the process of training unsupervised models, it really gives us a grounding about where to look in the model to begin with. It gives us a bunch of candidate experiments, a bunch of candidate hypotheses to run.

Yeah.

And one of our goals as a company is unsupervised learning often looks worse until suddenly it looks better. If you went back in time, there was a point where massive pre-training in an unsupervised way on large corpuses was getting you worse performance than bespoke models that were purpose-built. And so we think interpretability is likely to follow a similar arc. We're still not sure exactly what those pieces of technology will be, but each new sort of item on the tech tree in interpretability unlocks new questions that we can ask, new ways that we can look at the problem. And as Tom was saying, eventually building towards a solution that we can just toss a lot of compute at in order to fully unlock what's going on inside.

Am I interpreting you correctly to say that right now you'd say the field still feels like it is bottlenecked more on algorithms than compute, or certainly not models, right? I mean, there's a lot more in models we can figure out. So the 2 candidates would be compute and algorithms. It seems like you're saying algorithms are still where it's at and we need to figure out how to apply the compute.

I think that's right in some sense, but I think it was like a soft version of it that I would take. I think existing tools are already powerful enough to do things that are useful in the world. And that's a big part of what we're doing at GoodFire when we're working with customers, is we're using the existing techniques that are available to look at a model to help customers understand their model better. But also, we are under no illusions that we've cracked interpretability. Like, to fully reverse engineer what's happening in a model, we do need brand new techniques and new paradigms.

On the compute front, I think it's really a question of like frontier models are bottlenecked by like literal availability of compute or will be soon, right? We are very far from that, but we could spend a lot of compute, right? The question is, do you get value for money for it? Like, I'd like to be able to do a million-dollar interpretive model training run and say, oh yes, I spent $1 million on it, but I feel like I got $1 million worth of information.

That is an interesting—

maybe there's a way to say that there's a sort of ratio between size of experiment and how many— that's maybe not quite the right way to say it either. But given an option of choosing a really big run or more people to come up with more ideas for smaller runs, it's clear that you'd rather have More interpretability researchers exploring the space more thoroughly before scaling up.

Well, yeah, like I think that we couldn't productively do a single million-dollar training run. I think that's one of the things that sort of says, are you bottlenecked on algorithms? Can you productively spend that much compute? We could spend it, but at the moment, I think it would be, it wouldn't be a productive way to spend it.

I think there's also the question of how you use the tools that you have. For our customers, when we've trained an SAE for them on their model or any type of interpreter model, that's where the work begins, not where it ends. I think of an SAE as a window into the model. You can only see some things, but those things can still be really useful and provide a lot of value. Crystal popularized the biology analogies for mechanistic interpretability, and I think those really track. From my perspective, if you go back to the 19th century, early 20th century, what did you have to do to learn anything about cells? You had to put things on a slide, you had to stain the slide, you had to look in a crude microscope, and you were looking at a cross-section of something not in its natural environment. And you had to make inferences based on looking at a lot of these about what were the actual biological structures that were going on. But all of modern biology was built on that. And over time, we developed better microscopes. We developed better techniques for looking at organisms in more natural ways. So I think we're a little further on in interpretability maybe than that analogy implies, but I think it's the same thing. You can learn and make novel advancements in science at any state of the technology. And just simultaneously, we want to push what we can learn with the tools that we have. While we're also pushing to get better tools.

I think for a while people had to like make their own lenses, right? Which I guess is like making your own SAE training codebase. Like probably you wanna just go and buy the lenses from Zeiss or something like that. And then you can become an expert in great tools and then the field can advance. There's some sort of famous dictum in science about like things advancing via methods and ideas and then experiments or something like that, in that order.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, there's an Adam Smith lesson here. The sort of extent of the market and the degree of specialization and the sophistication of the supply chains are all pretty early. And we kind of are in that sort of, you know, maybe just exiting the like grind your own glass lenses phase of biology. Let's hope we don't find ourselves in the gain-of-function research, lab leak, and invasive species phase of biology before we're ready to handle them. But I guess one place where this analogy maybe breaks down a little bit is on this question of fundamental units. And this is sort of a philosophical question I've been trying to wrap my head around better. I'm sure you guys have good thoughts on it. When we look at an organism, right? And we look at its genome, or we look at sort of proteins, we're pretty confident we're talking about real things. I guess they're sort of maybe quantumly fuzzy on some margin, but we have a pretty good sense for a gene is a gene and a protein is a little machine. And here in the interpretability side, the features, if you will, that are identified or that are learned by an SAE or similar techniques. You might want to, if appropriate, separate SAEs from other techniques in your answer here. But these things seem to be sort of approximations, or there's some sort of gap right there between what is going on in a model and sort of what is going on when it's sort of sparsified in this sort of particular way. And so I'm really interested in how you think about the relationship between these features that are learned and the labels that we give them and sort of how much correspondence you think there is there. Is that on a spectrum and how should we think about it?

Yeah, so it's interesting. One, I suspect if you asked a biologist, they would probably have a lot of corner cases about, is it really a gene? Biology is, if nothing else, a great supply of corner cases. Just because of the rich complexity of the world. But yes, I think that there is a definite sense there that like a gene is a sort of natural abstraction. It's like a, it's a good way to talk about the world. And so this process of like, you take a model, you sparsify it. Well, now we have introduced some degree of lossiness, right? Because we're not capturing all of the computation. You can see this from the loss. You can see this from the reconstruction error.

Yeah.

But we are capturing what looks like very interesting and interpretable things. But that takes you onto the sort of next level of like, okay, there's a thing, it, there's like a feature in your sparse dictionary, and now we assign it a label. And this is another area for sort of a gap. There's sort of a gap that you can fall in. And I think that we can be in the business of closing both of these gaps, like a great deal. And you can, Close the gaps in multiple ways. One is to have a better idea. So the first gap, right? Gap 1, where we are talking about the distance between the model and our approximation of the model. Then how do you close this gap? Well, one answer is that you do the machine learning better. You just make a better SAE. So you capture more of the loss. And there have been a bunch of that's working in this direction. Like, not— there's, you know, I can provide a bunch of papers later. Another is you try and answer the question, what does it mean for something to be a good abstraction? And you sort of use that as inspiration for new methods. So for instance, like, what would it mean for an SAE feature to be like a natural unit of computation? Well, it's not completely clear. I think there's actually some interesting, but probably quite resolvable issues there. One thing it might mean is that it like is involving kind of consistent computational paths. You know, a feature is a natural unit of computation if it sort of is involved in other computations that make sense. And so now this is maybe a way, maybe one way of saying this is I think that things get a lot cleaner, like feel like they will get cleaner as we move to circuits rather than Like just single layer activations, because you don't really have a great way of validating. Now you can validate by like intervening on a feature and seeing how things change. That's sort of like circuits, except you just haven't tracked the circuit. Now you're sort of entering into the second part of the second gap. You know, I've intervened on a feature. You might say, well, I intervened on it and it didn't do what I expected, but that might be because my expectation was wrong.

Yeah.

That I've simply fallen into the second gap where I've given a feature an incorrect— I've given the feature an incorrect description. It is a unit of computation that the model does, but I've just called it the wrong thing. Now, how do we narrow the second gap? And I think that the answer here is probably that we just get better at doing experiments on interpretability, or we kind of— So the way that we currently assign labels is, hopefully I won't offend Nick by saying this, is like a little primitive. So Nick Kammerer at like Lumbee invented this automated interpretability technique.

And principal investigator at Goodfire.

Yes, thank you.

I was like, any question I'd profile him.

Yeah, he's wonderful. So anyway, hopefully I won't offend him by saying, The current method of assigning semantics to features is a little primitive. And what we do is we give the frontier model a bunch of examples of where the feature fired. And we say, well, here are these examples. What's the feature? And this gets you some way, right? But it doesn't get you the whole way. For instance, if you were to ask me with access to the model, What is this feature? I wouldn't only do that. I would also like try steering the feature and see what happens. I might like look at other things that projected into that feature or like where it goes downstream, look how it relates to other features, all that sort of thing. So there are like many more things that I personally would do, but we can't currently get models. We can't currently get frontier models to do this. It may just be a matter of scaffolding. We just need to build this kind of scaffolding such that they can use their capability set. That's a really run-on answer. Yeah, I think there are these 2 gaps that we can narrow both so that—

No, on the contrary, I kind of want to expand on it, on maybe both points, because it's all really fascinating and important. So the first bit, I guess I really like just sort of decomposing the problem into the 2 gaps. First is, can we— I mean, this is literally what the SAE is trained to do, or the other techniques, is to Reconstruct, right? It's a reconstruction loss. Do what the model originally was supposed to do there. I think I'd love to know a little bit. Maybe you said you could provide a bunch of papers later. I'd love to hear a sort of, what is the state of the art? The deepest dive I've done into literature recently was the Anthropic Tracing Model Thoughts pair of papers. And I was kind of struck there by, it seemed like there's A lot that is not being reconstructed, I guess, is what I would bottom line it as. So I'd love to get a sense of where is that state of the art, and maybe in narrative form, if not in fully cited paper form, what has been the trajectory of kind of improving that? What have been the advances, the unlocks, whatever? And then on that second gap, this sort of anticipates another question I had around just what is the inference time scaling paradigm for interpretability? It sounds like the answer there is, well, today we run data through the thing, see what activates what, and kind of collect the things that cause the max activation, then try to describe them qualitatively. And in the future, we could do a lot more, and maybe you could kind of develop a little bit the vision for the sort of higher order auto interpretability. I think both of those are really interesting mini lectures that you could go on at as much length as you want on.

I think Before we dive in there, can I give a quick meta thought on this next question? Which is that most measurement apparatuses that you could deploy in most scientific contexts are reductive in some way. There's some set of assumptions that you're making in how you should interpret the data that you're getting from the measurement apparatus that you're doing. The microscope analogy with SAEs, I think, works really well when explaining this to people because there's 3 things that you can do with a microscope, right? You can figure out what you're putting on the slide. That's the dataset that you're passing through the original model. Those are the actual activations itself. You're going to get a different SAE on one dataset from a different dataset. There's sort of like how you stain the slide. Maybe that's by the loss function or how exactly you're looking at things that's going to affect the structure. of what you get back in different ways. And then there's the size of the lens, right? It's like the expansion factor of the SAE. It's important to understand what we're doing with our existing interpreter models as a specific lens, a specific way of looking at the computation of the model rather than the whole picture. I think this is true whether you're trying to do circuit work with sparse approximations or if you're trying to do layer-level activation work with sparse approximations. But that's not an abnormal thing to do in science. You take some sets of assumptions, you know they're not always right, but you know that they're right enough sometimes that you can start getting traction and running new experiments. So from our perspective, it's not this all-or-nothing thing. This is what Todd is saying. We can keep pushing on the fronts of how do we address the limitations in the existing tools within the paradigm that they're operating with, the sets of assumptions that we know are sometimes wrong. but are right often enough to be useful and give us more information. And then simultaneously ask the question, what's a better set of assumptions? And these have to happen in parallel. You can't just do one at a time because you would never find the right assumptions if you weren't testing the limits of your current ones. So from my perspective, this isn't a unique thing about interpretability. This is just how science works.

Yeah, exactly. It's funny, people remember Kuhn's structure of scientific revolutions for the revolutions, right? Like crises. And yes, like the thing, the sort of dominant mode of science is like normal science where you're going along and you're like generating actually productive knowledge about the world, maybe on foundations that will like later get a bit shaky or get overturned, but like you're still generating knowledge about the world. And then there's this idea of, there are anomalies, and then anomalies pile up unanswered and lead to a crisis, right? But that's where the anomalies come from. The anomalies come from the business of doing normal science. So even if you want to generate a paradigm shift, often the answer is to just try, to just try and do normal science until it becomes untenable. And I think that's, yeah, that's maybe where we're, one way, one place where we might be at the moment. We have like probably a proto-paradigm. I think we've been like reluctant to admit it, but I think we probably have kind of a proto-paradigm in interpretability. And so we should push it. We can do a lot of useful stuff. We should just, we should keep pushing it. We should keep doing the useful stuff and see where, and wait for the anomalies to reveal themselves.

Yeah, I was gonna ask actually too, are we still pre-paradigmatic? We've upgraded ourselves now to proto-paradigmatic.

I think I'm gonna say proto, but maybe I should, maybe I should have some encouraging my convictions and say that I think we're like entering our first, it's like the first paradigmatic phase of interpretability.

Well, no.

Okay, so this is a bit fuzzy, right? Like, what is a paradigm? A paradigm is a sort of social thing. I don't think there's consensus. There's not the kind of consensus that would lead me to say there is a field-wide paradigm in interpretability. I would say that among a reasonably large group of people, there is the foundation, there's the raw materials for a paradigm, right? I suspect if the field were Anthropic, but sort of ballooned to the size of the global interpretability community, it would be correct for me to say that there was a paradigm. But because there isn't a level of consensus, I can't really yet say. There is a paradigm.

If there were, so what would that paradigm be? How would you describe the Anthropic and Goodfire axis paradigm?

I'd say, okay, that one, like neural networks contain things which are understandable, right? This is like actually worth stating. For a long time, this is not generally accepted. I don't know if it is yet generally accepted, but you know, This is like sort of down in the basement of the paradigm.

As a quick interjection there, is that an artifact of just earlier models? Because the way I would tell that story is like the original GPT layer, there were still some things probably that were meaningful enough, but there was also so much noise that people could very easily have been excused for just being like, you're tricking yourself, or you might find some spurious correlation here or there, but I don't really buy it. And maybe just haven't updated since?

I think, yeah, I mean, you're going to interpret the model that you have. And if there are lots of flaws in the model, those might be what you're finding with your interpretability tools. And if you're entering with the prior that, oh, I should be looking and recovering this specific thing and you can't recover it, that could just be as much evidence that the model isn't doing what you thought it was doing to begin with. When you go through the process of debugging a model using interpretability techniques, the thing that you might find is, oh, the model has memorized a bunch of its training data or something like that. You have some belief about what your model is doing and how it's modeling the task, and that belief might be wrong. And that could throw off the perception of interpretability if you're not bringing unsupervised, low-opinion techniques in that can work across the entirety of the end-to-end interpretability stack. And I think to that point, we just didn't have unsupervised techniques that can work at least a good percentage of the time across the end-to-end interpretability stack until quite recently.

There's a funny sort of U-shaped thing where I think a lot of the early connectionist papers, they actually do look at individual neurons and say, oh, look, this neuron learned this thing. This neuron learned this thing. And they could do that because there were like 12 neurons. It's funny, it sort of started out and everyone wanted to look at the neurons and had some success. And then for reasons that are opaque to me, but I suspect like someone could find out, it became very like somewhere between unfashionable and like considered to be a bad idea or impossible to look at individual neurons. And now it's kind of come back in vogue. So it's sort of like we just had the interpretability just kind of went into the wilderness for a bit.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah. Okay. So now give us the paradigm. So one is there are things that make sense that we can interpret.

Yes, there are things to interpret. Interpretability is possible. And then what other parts of the paradigm are there? I suppose there's features like representations are linearly decodable. That's, or at least like linear decoding is a reasonable way to talk about features. You should, there may be higher order structure, right? Things you might have features that like features are sort of arrows in space. It might be that actually multiple features kind of lie on some manifold or in some subspace, but that it's a sensible way to talk about representations as Lines through embedding space. Third part of this paradigm is like this idea of superposition, because if you're gonna have vectors in a vector space, then the natural conclusion would be like, well, I'm in a D-model size space, right? Well, does that mean the model can only think of D-model things? Probably not, right? A language model can think of more than 4,096 things or something. So the other part of this paradigm is superposition, which is this idea that the way that you put— the way that you squash more of these embeddings, more of these vectors, these like feature vectors into the same space, is by allowing them to overlap a little bit. And this creates a sort of bit of interference, bit of noise in the representations that models are still able to like deal with. Now, I guess another part of this is that magnitude, like direction along the vector, constitutes intensity. And the other thing is rather obviously, I suppose, features connect to form circuits. And that is basically, I think, the paradigm, the paradigm which is currently, or rather the thing which, this is like the Anthropic paradigm, I would say. And if you were to blow this up to the size of the world, and like if there were consensus on this, Then I guess it has enough structure to be called a paradigm.

Is there any competing proto-paradigm or is there just other people claiming we'll never have one at all?

Not in the sense of something that provides a complete worldview about— when I say a complete worldview, all I think about is about neural networks. It doesn't say anything about dinner. I don't think about that very much. But a complete worldview for neural networks. You could say that something like parameter decomposition suggests a separate paradigm. And that's because that's sort of a difference in emphasis. Parameter decomposition is talking about the weights. The sort of the SAE type paradigm is talking about activations. Now, I would say there's still sort of, yeah, so they are, they're sort of different levels of emphasis. I think But a way to think about this maybe is that we, unsurprisingly, we need both, right? And to make this seem like a bit more intuitive, a neural network is, in a very dull sense, a causal model, right? Every neuron is a node in your causal graph, and all the weights specify the edges in your causal model, right? So it's just, it's very big, but not very like big, very homogeneous, not very interesting causal model. And what we, one way of thinking about what we're trying to do in interpretability is we're trying to create like a causal abstraction. We're trying to create another model, which is like a sort of reduced version of this model, but that will also be a causal graph. And so when we're talking about this, like, debate between should we decompose the parameters, should we decompose the activations, saying should my graph, right? Should it have nodes or should it have edges? Well, probably it should have both. It's a graph. So, this is why I think that actually they're not necessarily competing paradigms. They're just sort of 2 independent ways of thinking our way towards like the broader kind of causal abstraction that I think we need.

Would you point to any highlights in the work that has focused more on the weights than the activations? Because certainly from my perspective, it seems like much more news and excitement coming out of the activation space.

It's funny, because some of the— okay, so the earlier stuff, we're going to stay in the sort of mechanistic interpretability extended universe, because I can't immediately recall some of the other papers as well. I can have a look. But like, some of the earlier stuff, like learned equivariance in the circuit thread, for instance, was— that was a sort of weight-based analysis. But then the more recent parameter decomposition things have been like really coming up, Lee's group, formerly at Apollo. Now he's a principal investigator at Goodfire, just do a little plug. He's great. And so that's like attribution-based parameter decomposition was the sort of milestone thing here recently. There's another recent paper sort of following up on this based on ideas from the loss landscape. I'm blanking on the title. I guess Brianna Christman is the lead author. I can send it to you afterwards. And so, it's what happened in the past, and then SAE's got a lot of traction, and so that took a lot of the focus. And now I think it's sort of coming back in again.

One other school possibly that comes to mind is the sort of, or maybe contrast in approaches, is the Sort of bottoms up versus top down. Like, I think of Dan Hendricks and sort of representation engineering or circuit breaker type work being less somehow focused on the sort of unsupervised, like, just discover what it contains and more, this is the sort of contrast that I care about. Like, let's make sure that we refuse or whatever under certain conditions. Do you think those are fundamentally different approaches or are those ultimately reconcilable?

I think they solve for different problems and they're both important things to look at from a safety perspective. When we zoom out and we think about the alignment problem broadly, we think the solution to the alignment problem could come from interpretability. But doesn't necessarily come directly from interpretability. Interpretability is the measurement apparatus that can make us confident that any other techniques are in fact doing the things that we think they're doing. For instance, are we aligning the chain of thought of a model faithfully to its connotation? I don't know how we would have any hope of answering that question if we don't have meaningful ways of constructing graphs that represent the model's computation in some way. So I think a lot of these core questions in alignment, at the end of the day, it could be that we solve alignment through bottoms-up, but it could not be. But without the bottoms-up, I have no idea how we would know that we solved it through any other means.

Okay, I think that's really super clarifying stuff. I think people will find the sort of high-level mental models there are quite interesting. Maybe I'll add a 3rd chapter. So one, or the gaps, right? If we structure this in terms of opportunities to make things better, we've got better reconstruction, lower loss. Eager to hear a little bit about kind of what the progress has looked like there. Then we've got this sort of better labelling or inference time scaling and getting to higher and higher orders of automation and confidence that what we are describing, what we say is happening is in fact what is happening, that we're understanding it correctly. And then you alluded to what I'll maybe add on as the third thing, which is the move from activations to circuits. And I did think Lee Sharkey's paper there was super interesting and could be a whole episode on its own, but we can do a mini one as part of this.

So I should mention the Anthropic circuit tracing results as well. I think everyone wants circuits. The question is, how do we get them?

Yeah. So, take us through those 3 chapters. And again, take all the time you need because I'm here for it.

Okay.

So, should we talk about the first one first then? That's like, how have people been doing the machine learning better? And here, the machine learning part of it is like, how do we best learn this kind of decomposition of the models or like, Within the— let's just stick within the SAE paradigm, because then I can be very concrete. How do we learn good decompositions of the models? And some of this is, you start with the SAE, and it has L1 sparsity regularization, it has a ReLU activation function. And then people sort of hill climbed a bit in the kind of classic machine learning way of, well, the L1 sparsity, it has certain properties, like it causes features to shrink, which also has like a predictable bad effect on Reconstruction loss. And so there are various solutions to this, things like jump value or batch top-k. It's sort of some hill climbing on this work. And then things like end-to-end SAEs, where instead of training on the activations, you're like, you're not training purely on the activation reconstruction loss. You're actually training on re— like, such that when you take the model, you know, you've got the model when the computations are going up, you put the SAE in the middle. And we train such that when you put the SAE in the middle, the model doesn't get too much worse. And it's called an end-to-end SAE. And both of these things, all of these things, like, improve on the loss reconstructed versus sparsity kind of frontier. What else is there? There are many other approaches to dictionary learning, where dictionary learning is the sort of, what is like the broad class of things of which the SAE is instantiation. The dictionary here being your, like, collection of vectors, kind of your quiver of arrows is the dictionary, and you're learning a dictionary that could do a sparse decomposition. And things like using gradient pursuit instead. I saw an interesting blog post on, like, a residual quantized autoencoder, Matryoshka. All these kind of things are, like, other ways of using compute better to have a, like, you sort of haven't really changed the fundamental assumption. That features are directions in space. But you've gone like, how do we, like, under this basic assumption, how do we, like, create a machine learning architecture that reduces these metrics of, like, loss? Yeah, that improves the loss we covered. What else? I'm trying to think if there's any other. Doubtless I will have missed some and, like, offended people by leaving their papers out. That's just a failure of my memory. Apologies, people who have papers I've forgotten.

Well, you're going to be completely comprehensive in real time. The matrioshka concept was one that I was keen to explore. I've heard a little bit about that. And it has a very, I don't want to be overly lured by this, but it has a very appealing vibe, I guess, to me where it's like, there should be some, you know, structure that gets finer grained as you go. And basically the idea there is like, Sort of a tree structure, right? I mean, it's a little, it's not really structured that way, but the idea is like the short version.

It would be ideal if it were a tree. Unfortunately, there's no explicit tree structure involved. Um, my transcript is really interesting actually, like both as an architecture and from a point that I was sort of talking about earlier of, we were talking about this idea of models, compute, algorithms, and algorithms kind of being bottlenecked on observable experimental data, like us knowing new things about models. And Matrioshka is an example of this happening. So how did we get to the— I say we, I had no personal part in it. How did the field get to Matrioshka? Well, my understanding is that there's this idea of feature absorption, and feature absorption happens when SAE latents kind of Specialize far too much. So instead of there being, I won't go into the mechanisms a bit involved, but essentially instead of there being, like, for instance, a feature for token starts with the letter A, you now get a feature for, like, token starts with the letter A and is not the word aardvark or something like that. And then you also have a feature for, like, token is aardvark, because it turns out this gets lower L1 sparsity loss. And so this kind of thing called feature absorption was a motivation for Matrioshka. Now, how did we get to feature absorption, right? Well, someone looked at some SAE latents and said, that's funny, that looks wrong to me. I think the answer was actually they were probing for, they're training linear probes for like, What letter does this token start with? Which sort of feels like a silly niche thing to do when you don't, without context, but like actually led to this interesting discovery. It's a very smart thing to do. So it led to this interesting discovery of what happens with SAE tokens that was relatively hard to predict. Certainly no one predicted it a priori, as far as I know. And then Matryoshka turns out to be a good way to fix this. Now, a more desirable thing would be the thing that you said, like Matryoshka, but as a tree. So what actually happens with Matryoshka is that you have like a series of nested groups of features. And so we predict using the first group. Well, there's 2 variants. The simplest one to talk about uses groups. We predict with one group, right? And now we have some residual error term. And then we use the second group to predict that residual error term, and so on as we go out the shelves. What I think would be quite desirable is instead being able to say, well, this feature fired, which means that now I'm going to like upweight this other feature firing. And so you'd have this sort of explicitly encoded tree structure.

That's—

the idea of a tree structure is very interesting for like minimum description length reasons. There's a really cool paper on the minimum description length. I can come back to that in a minute.

Mm-hmm.

But enforcing this kind of discrete sampling behavior is relatively hard. It's much easier to just do this very soft sort of differentiable thing. I think it is harder to do the kind of thing that we both would like to do.

Maszewska, in general, I wonder how much of this is sort of, and maybe this could again be sort of abstracted a little bit to interpretability in general as a sort of a subfield that's coming in the wake of the broader ML developments? In general, it's been applied in a bunch of different places, right? And it sort of boils down to finding a way to make the first bits of any given thing the most meaningful. I sort of think of it as ordering the data so that the most meaningful stuff comes at the front. And you've seen this in embeddings and weights and whatever, where you can have a short embedding that is pretty meaningful, and then the longer you go down toward the full embedding, the more meaningful it gets. But you're sort of always getting the next most relevant bit of information at each stage. So intellectual history-wise, would it be right to guess that a lot of the things that are happening in this kind of doing machine learning better are people kind of looking for techniques that have been developed previously in other contexts and saying sort of like, this worked before, maybe it will work again here? And then kind of having applied that technique, then sort of developing a story of what was happening after the fact. That's sort of how I expect a lot of this would be working. But I trust you to tell me different.

I am not sure. I expect that a lot of it was worked out fairly first principles. And it turns out that someone may have done something related before, but thinking about this context, you arrive at, oh, this is a problem. Now, there has been a lot of work done on sparse autoencoders in the past, like around, you know, early 2000s. A lot of work went into sparse autoencoders. Like, dictionary learning has previously been like an active topic of ML research. But I think we're sort of rediscovering a lot of things at the moment. And it could be that one thing to do, one actually high-alpha activity In dictionary learning or in sort of this more ML side of interpretability is simply to hit the history books. I say history books, I mean like, you know, papers long, long past and see what has been done that hasn't yet been, that hasn't yet been like applied. One, one example actually I think is quite funny. This is the recent NeurIPS sparsity, the 2024 NeurIPS sparsity tutorial. It was very good. And for one of the slides early in the talk, it was like, what this tutorial is not about. And it was like, this tutorial is not about sparsity for interpretability, i.e., sparse autoencoders. And there are various other things in it. But I saw this, it's not about these various other kinds of applications of sparsity. But I saw this and I was like, yes, great. Now I'm going to learn some things that no one else in the field knows. And there's actually some really good stuff in there. So like extend doing, doing that activity a bunch of times would be like probably very high return.

Although coming back to a second to your question about like ways this parallels other things in machine learning, that could be a good segue, Tom, to talk about minimum description length. 'Cause that's really interesting.

Yes. So Michael Pearce recently joined Goodfire. I guess this is happening quite a lot in this.

You may notice that we have some of the world's best interpretability research transfer at Goodfire. Yeah.

Michael Pearce was one of the authors of this paper on the minimum description length. And I think it's a really neat idea because it basically gets at what you were saying earlier, that what you want is you want to have effective, compact descriptions of what's going on. Ultimately, what we're trying to do in interpretability is describe neural networks. I'm trying to describe it to you, or Claude is trying to describe it to me or whatever. And we're trying to find decompositions that are easy to, like, easy to describe accurately. And then give good decompositions of them. So the minimum description length is this idea that you should be able to, if like, what makes a good description is a description you can transmit in relatively few bits. And the idea is that rather than using like sparsity as our sort of regularizer, Also, I kind of, sort of, we normally quantify in the sort of SAE paradigm, progress has typically been quantified on this sort of Pareto frontier of sparsity versus loss reconstructed. But the idea of the minimum description length is that really there's only one metric, and that is how many bits it takes you to describe what's going on. And there's various sort of technicalities about how you actually do this with an SAE. I think basically that, like, It's a metric that I think we can probably mostly agree is a good idea. The problem is like how you actually implement it, how you optimize it, and so on. But it has various other nice properties. Like if I wanted to have this kind of tree structure in my features, that under a sort of L1 type sparsity regularization, this wouldn't actually be preferred. But under the minimum description length, I can have like a feature in, you know, I've got my tree, I have the root and the branch. Well, the trunk and the branch, right? I can say that, if I can say that the branch is like high probability given the trunk, but otherwise low probability, then this has a shorter description length than having to describe both of their probabilities independently. It's a little, it's hard to give a good sort of overview answer of this.

I mean, maybe one way to attack it is like, What is the artifact that we get when we get this with a sparse autoencoder? I have the intuition that the thing may have 10 million nodes or whatever, but only 100 of them are going to light up on any given forward pass. And that could be sort of a hard cap, or there's different batch top-k is sort of interesting wrinkle on this where you can define exactly how many nodes will be active. Or control that in a few different ways. But if I apply the minimum description length, what is the thing that I get out? Yeah.

So the idea of minimum description length is more like, it's a way, it's like a way by which we should compare all of our various approaches to decomposing models. It's like, if for the same level of accuracy in terms of decomposition, one has a shorter description length than another, then we should prefer the one with the shorter description length. So for technical reasons, something which is tree structured would generally have a shorter description length than something which is not tree structured, which is just a bag of features. As long as you know the conditional probabilities along the tree, then you generally have a shorter description length. And that sort of matches with our intuitive understanding of, it's easier to talk about things in terms of relationships between parts than it is by just like enumerating all of the parts separately. So I actually, but actually how you calculate this is actually quite difficult. Like the concept is simple, in general, prefer the thing with the shorter description length. All the meat is in the question, like, well, how do you calculate the description length? And so I do kind of glossed it, but.

Is it something that you can apply directly as an optimization target as well?

Not yet. I think we'd like to be able to, but not currently.

Gotcha.

Like it is in some sense the ideal optimization target, if only we could optimize it.

I dunno if it would be batch top-K versus a more naive sparse definition, or maybe some other that you would highlight, but how has the optimization target improved? Like, that's basically, we're conceptualizing the product— the problem better, right? And you can measure that by this sort of minimum description length. My sense is that batch top-k is a better way of conceptualizing the problem than slightly more naive ways that came like just before it.

I think they would have the same description length, more or less. Like an SAE with batch top-k versus an SAE with value versus an SAE with jump value. Like all of the, all of these would have approximately the same description length, I believe, like all else equal. But being able to have something that's expressed in terms of a tree structure, or like there being various, various subspaces, these would have a different description length. Or for instance, like to go take a sort of detour into circuits for a minute. If I had to describe something with cross-layer superposition, By enumerating each of the separate layers that I cared about, and there being like one SAE feature for this layer, one SAE feature for this layer, one SAE feature for this layer, plus also they're the same thing, that sort of has a longer description length than me saying there is a feature which is spread across these 3 layers. So I think it's sort of on a higher level of description. It's like on a sort of higher level of abstraction than like say batch top-k versus value.

So all in, where are we on this doing the machine learning better part? I mean, again, my most recent point of reference is the Anthropic tracing model thoughts, and my sense was like, there's a lot that's not reconstructed, right? So if you were to push, if we say, what's the best case scenario? Can we reconstruct a model that kind of works, or does it not really work at all still? And is it therefore only sort of limited to these sort of very— because in the Anthropic work there, much of what they were doing was prompt-specific analysis with custom error terms thrown in for those particular cases that they wanted to study. And that doesn't— certainly there's still a lot to be learned there, but I guess how close are we to effective reconstruction of models as it stands today?

So, I think there are 2 aspects to that question. One is like, can we essentially throw compute at the problem to get there? And the other is like, how much will it take? If the answer to the first is yes, we can, then how much more will it take? And this is sort of, if you think about this in terms of a scaling curve, like this is sort of like asking, does it plateau or does it not plateau? Is there like an irreducible error? And this question has been like examined a little, not as much as I would like it to be examined. Did we talk last time about the Dark Matter of Sparse Autoencoders paper by Josh Engels and collaborators? I don't think we did. It's a really interesting paper.

Yeah, I don't think so, but we definitely should dive in.

Yes. So what they did is do this kind of scaling analysis and say like, if you were to keep scaling, SAUs up, would you be on track? Like, would you be on track to recover all of the activation? And their experiments suggested that the answer is no. If you sort of trace the scaling curve out, that there's a substantial amount of like dark matter, that the scaling curve does not go up to like 99.99 or 100% or something like that.

Yeah.

It actually plateaus. Like it actually is like on track to, is not on track to recover everything. And no one really knows what dark matter is like. That's why it's dark matter. If it were easy to understand, we would already have understood it and probably baked it into a new kind of SAE architecture. This experiment hasn't been repeated as far as I know. And I would, so I feel like there's a lot to pull on there. Like, you know, if you did it with updated models like this, oh, there's, I didn't even mention on the doing the machine learning better, this recent SPADE paper, which is like a much more expressive version of sparse autoencoders. We should put that in the links. But if you were to redo this analysis with the latest techniques, would you still find that there's dark matter or no? I suspect the answer is yes. I suspect you would still find dark matter. And getting rid of that seems like a pretty big deal.

I mean, we still have it in all the scaling laws, right? There's always a constant term on the transformer scaling laws too, right? So would it be— I mean, the sort of Prior guess would be that there's some dark matter minimum that you might approach and you'll have to chip away at it in the same way that we do with normal loss, which would be you have to 10x your inputs for the next increment. And I guess that would be like—

Ah, no. So the question is, even if you were to scale up computers, Essentially as much as you possibly could, would you— well, it's like the difference between a curve on a log plot that is a straight line and a curve that eventually bends. And if your curve bends, right, then you have a problem because you are no longer on track to reach 100% reconstruction. And I think the claim of the dark matter paper is that it bends. And it bends because there are these things, things that we really cannot efficiently reconstruct. Now, I don't really know what they are. I think figuring out what they are would be very important. It could be that there are things like memorization or various kinds of structure that SAEs are just not— like higher-order structure that SAEs are just not very good at reconstructing and learning. It could be that they're actually like noise. They're actually— because the Dark Matter paper was on reconstruction accuracy, or actually fraction of variance unexplained. But it could be that like, this is noise, that the dark matter is just noise that has no relevance to SAE, to like model outputs. And so if you were to repeat the analysis with loss recovered, for instance, it might look different. There's so much that we don't know, right? That's like actually very important.

But okay, so just to make sure I'm not confused, and maybe I am, the con— going back to just general transformer scaling laws of whatever vintage we want to look at, there's always this constant term. I interpret that as the scaling laws do not suggest that we're ever going to have zero loss, but on our way to this sort of theoretical minimum, we have this 10x for the next increment of progress. And does that theoretical minimum imply the same kind of bend? Is that the same sort of theoretical minimum that you're Presumably.

On the scalability side?

I mean, presumably they both imply that at some point it must bend unless it hits zero, hits the minimum. So just bang, and then it's horizontal. There's an interesting difference though between the scaling laws, which is that there are 2 sources of irreducible error. I suppose, in a language model scaling law. One is the uncertainty of the world, right? If I literally have 2 strings that have up to token t, they're the same. And then at token t plus 1, they have different strings, right? And these are both in my pre-training corpus. Well, now I have, now I have a floor. I can never get down to— I'm training on this corpus. I can never get down to zero loss. Because I, like, it's 50/50 looking at, looking at this data. I have to pick, like, given the prefix, it is no longer like deterministically predictable what the outcome is, even if I had a perfect model of the data. So that's one thing. And the other is architecture dependence where like, say, there are things that the transformer simply cannot capture. And these are like somehow detectable from the scaling law. Like we expect that it will asymptote out because of architectural limitations. The difference, I suppose, is that we believe we should, like, it sort of seems maybe capabilities people would disagree, but it seems okay for a language model to not achieve perfect reconstruction. But I would be, I'd much prefer that my SAE or my interpreter model, like, achieve perfect reconstruction. That seems pretty important for it to be very high reconstruction.

Yeah. I think there's also the question of what abstractions are you working with here? Again, what are the limitations of the technique given the limit? If we had wide enough SAEs, what would the features actually be? Dataset examples, right? They'd be specific dataset examples. for every activation pattern it's seen because that's just what it's going to learn in the limit. So you're trading off again against generality and memorization and specificity too in the context of an SAE.

I think this is a really interesting intellectual exercise. I think it's very useful. And if you think about higher-order structure, this is also somewhere you can just soak up features. Say that my X hypothesis, my true data lies on a ring, Around the origin. And really there is only one like feature, which is the angle.

Yeah.

How far around the ring have I gone? But if I want to try and reconstruct this data with SAEs, then like I can put infinitely many features into it. They'll just go at different points on the circle. So I can sort of soak up an infinitely large amount of compute. Whereas in fact, like, oh, I really needed one parameter. So this is another example of there being sort of fundamental scientific insights or like fundamental, like knowledge about how models represent their activations that we need to kind of then convert into inductive biases.

And yeah, to add some like concrete examples to ring one, there's been work that shows like transformers can encode days of the week, for instance, on a ring, which makes sense. 'Cause like you can think of days of the week as like a mod 7 operation. You add till you get to 7, then you loop back around to 1. But if you're training SAE, what are you going to recover? You're going to recover Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday. And in the case of days of the week, that particular abstraction, you can think about it both ways and it sort of makes sense. But in certain types of abstractions, it's very reductive to look at it in that way. A good example would be we know transformers can do addition. The way they do addition, like modular addition, is with a set of trigonometric operations. If you look at Anthropic's latest work with the CLT and what they pulled out of Claude in terms of doing addition, it looks like a bunch of overlapping heuristics. You can look at that and you can say, oh, okay, this transformer is actually doing addition in this very unintuitive kind of dumb way. And maybe it is. Or maybe this is an artifact of the measurement apparatus and actually it's operating as some function over some continuous geometry, but just the nature of the measurement apparatus is that we'll never be able to see that continuous geometry. So I think these are really important, interesting questions. It doesn't mean that it's not helpful or you're not gaining information about the model or it's not telling you something important when you're doing this reductive form of measurement. But you do have to keep in mind that even if you had a perfect SAE, what type of information would you struggle to recover from it?

Yeah.

Okay.

Related, somewhat tangential perhaps, but the mention of modular arithmetic, of course, has me back to the Grokking paper. And I always remember this one plot where it's like, The test loss or the test accuracy shoots up really quick as the model memorises all the training data that it sees. And then it's orders of magnitude later that the actual grokking effect happens. And that happens over the last order of magnitude, which it's always important to keep in mind, is the actual 90% of the data or the 90% of the training time. Right. They're weird that way. Going from 10 to the 5 to 10 to the 6, I think, is what it was. After it had memorised by 10 to the 2 steps. So a mental model that I always have of kind of any model that I'm working with is that some things have probably been grokked, some things are mid-grokking, and some things are not at all grokked at the time that I'm actually using the thing. And so this sort of, in a way, seems a little bit like an echo of the— not an echo, that's too strong of a connection, but in some way analogous to the sort of fundamental world uncertainty. When you are trying to interpret a model, you're sort of like making an assumption that things are grokked, at least to some degree. Like, the notion of these sort of features and circuits and all this stuff is sort of a— there's a level of grokking there that isn't just like random, bizarre memorisation that just sort of happens to be the first thing that worked on the training set. And so I imagine that's a fundamental challenge of all of this, right? Is that some of the things that you might want to recover may only be partially learned in the first place, right?

I think that what we want really is something, is a sort of interpreter model that's algorithmically neutral. If it's Grok'd, then I want to see that it's Grok'd. If it's a kind of half-baked heuristic, I wanna see the half-baked heuristic. And if it's a memorized thing, I wanna see the memorized thing. Now, like actually getting that is again difficult, but I think that like, yeah, you're absolutely right to call out the, this sort of, it's easy to assume that everything is gonna be clean, but a lot of the time there are gonna be sort of messy heuristics that maybe even make up the bulk of the computation most of the time.

Okay, so we're moving into parts 2 and 3 around kind of the gap between what a feature is in some grounded sense and what we sort of label or understand it as. And then obviously moving into circuits as well. Maybe the best way to talk about this is like, what are the downstream things that we're actually using interpretability for today? And then we sort of imagine Closing this gap of the grounded truth and the labelling seems to be like a sort of increasing automation of a lot of those different techniques to kind of cross-validate against each other and make sure things are actually checking out at scale. But right now we're not largely scaling all that out. We're largely kind of exploring in a more naturalist sort of way. And we've got all these different tasks that we'd like to be So maybe you can tell us the story of recent progress in terms of what actual utility we're getting from our interpretability helpers.

I'd love to talk about all the applied work we're doing. Before diving into that, I want to make one more point on features, if that's okay.

Please.

Even within features, take as a given the SAE paradigm, we're recovering features, we can explain some percentage of them with the way that we label today and some percentage of them that we can't. One thing that's come up a lot in our work with customers is trying to develop a taxonomy of different types of features that you might find. So the way that we label features right now predominantly comes from ground truth external data. So it's connecting some patterns inside the model, some direction in activation space, to some pattern in the inputs to the model. These can get pretty abstract. In our work with R1 and with reasoning models, we managed to recover features that are pretty neat, that are qualitatively different than things that we saw in language model features and that seem to represent important units of the process of the reasoning trace for the model. But then there's also many features Which could decompose into, in similar ways, like could be explained. They're not dark matter features, but they refer to algorithmic processes that are happening inside of the model itself. That's not easily visible on the basis of looking at input data or output predictions. So clear example of this would be in-context learning. Something happens in in-context learning. People have looked at it and studied it. There's an algorithm that is implemented by the model.

Mm-hmm.

That can be difficult to observe from the outside but has really important implications for the downstream prediction. So we can imagine when we look at biology models, for instance, genomics models, which is something I'll elaborate more on when we get into the applied work that we're doing, some percentage of the features correlate very strongly with known biological structures and then many features don't. And for those features, the question is, Well, are those biological structures that we don't know yet, or are those structures of the computation of the model itself? How can we tell the difference between these things? And then in either case, that's extremely useful information because information about how you model these biological systems is still in and of itself really important to the question of scientific discovery. But yeah, so I think it's important to think of When building up your taxonomy of features, there's the percent of features that we can explain by observing the inputs and we can explain those with high levels of confidence. Our confidence goes down over time as they're moving into the tail. And then there's the features that represent something more abstract than just something about the inputs. It's something about the model itself that we need to understand or maybe something in the scientific domains about the inputs that we don't understand yet. And then you keep going on that curve and that's a spectrum and eventually you get to things that with current techniques we struggle to explain. And that's why we need to invent new techniques pushing even just the interpretation of the features that we can recover. And then there's the dark matter. So there's a bunch of different dimensions of this problem, but I think it's important for the viewers to also think about this taxonomy of features. What might a feature be doing? And it might be doing something that's actually entirely invisible from both the input and the output.

Yeah.

So if you were to do few-shot learning, I mean, one thing that sort of strikes me as a way to kind of bridge the gap there, and again, feel free to deconfuse me, would be to expand the window. I mean, when I— and there's lots to recommend about the Anthropic work, of course, but the interface is characteristically really nice for being able to kind of explore what they're doing in an interactive way, kind of just probe into it yourself. I don't know, when you click on a feature and it shows, okay, here's the examples that maximised that feature from the dataset, you're typically looking at a pretty short snippet in the— because you're like 3 tiers down a UI from a web page to an embed to a little popup within the embed. I don't actually know how big those snippets typically are, but if you imagine a sort of few-shot learning feature being like, okay, we have recognised that there is a recurring pattern here in some very abstract way, and our job is to kind of continue it, R being the model, I guess, in this narrative, then you wouldn't see that if you had a 20-token window, let's say. But if you zoomed out to a 20,000-token window, you might see the whole thing. And I imagine some similar things could be happening in biology where it's like you're way downstream of an activator sequence that turns this thing on in the first place or what have you. So is there a sharp distinction between these sort of grounded features and the sort of computation features, or is it just a question of our ability to zoom out far enough to see the pattern accurately?

Yeah, this is a really great framing on the problem. I think in-context learning is a good example here because if you zoomed out and you included more context— generally we do auto-interp, which is this process of labeling with different amounts of context and different types of context depending on the problem and in the domain in which we're operating. In-context learning is an interesting example because if you zoom in too much, you wouldn't be able to see it. If you zoom out, you can probably tell it from the prompts. But it's also an algorithm that we know that we expect to find in the model to be And then we would expect that a frontier model can probably identify a meta-pattern if it sees a bunch of examples as well because this is a type of thing that we've identified that models can do that we've already described pretty well. In principle, if you zoom out to the entire genome and you had some features that were active in a bunch of locations across the entire genome and you had a frontier model go and look at those, its ability to label those effectively for you in an automated way is bounded by whether or not an explanation was in its training data to begin with. And in many cases in the scientific domains, there's not an explanation. We're working with sets of abstractions that are pushing the frontiers of human knowledge in some way. So we need some way of labeling and thinking about how these features compose. That can push past that. You need to be able to look at a bunch of different contexts. Let's say look at 100 different genomes and see these patterns and reason about what they might be doing together. You have to be an expert in the human genome, perhaps even beyond the level of the greatest human experts today, to reason about that just on the basis of inputs. So one thing that we're interested in doing is, can we break this down into a set of easier problems? If we have 10% of these features that we can explain just on patterns on the inputs in an automated way, can we work with domain experts, scientists, to keep pushing up that frontier of possibility and further and explaining more and more of what the model is doing?

So how's that going? Because that sounds like maybe one of the more important questions we'll touch on today, right? I mean, one of my great hopes has been And I know you share this and are actually moving the frontier on it, but the idea that we can use these unsupervised approaches on natural data and then look in and see what the model is learning and then learn it ourselves and actually have confidence in what we're talking about and make new discoveries, this seems like a really great driver of new discoveries. But you're complicating my sort of naive optimism a little bit with this idea that We don't necessarily know if the features correspond to the real world or just to the internal model. That's a level of confusion that I'm certainly excited to hear how you're going to resolve.

Oh, I think that they still correspond to the real world in the sense that algorithmic features can correspond to the real world and follow it.

Yeah, right.

Like, um, you know, Newton's laws do not exist in the world. They're a way of describing the world, right? We can talk about the idea of velocity as a kind of, as a type of feature, but it's not the case. But like, you just see things have a velocity. They don't have like the concept of velocity. So these are sort of, the structure is useful still.

Yeah. And so, I mean, this is one of the big ambitions and goals of GoodFire is to crack this question. So we're working directly with customers across different scientific domains with their scientific models with this goal. For instance, viewers might have seen our research that we've done with the Ark Institute recovering features that correlate very strongly with known concepts in the genome and where ARK Institute and ourselves are pushing this collaboration now is moving towards unsupervised techniques that can help get us new information in the genome. And we're actively working on this research and hopefully have some exciting things to share in the not that distant future. Yeah, I think at a high level, if the model is correctly— if it's truly generalising in its ability At least sometimes, right? It's on this continuum for it groks some things, it's memorized others. What has it grokked? What has it memorized? To be able to generalize, it must be learning really meaningful things about the underlying systems. And so recovering those, even in an algorithmic form, tells us really interesting things about the system that's being modeled. Current bioinformatic So a reasonable question you may have is, well, how do we think about the genome today? Annotations in the genome are done using bioinformatic algorithms that have been developed by humans that have strong priors, in some cases work extremely well, and in some cases actually don't work that well at all, but work well enough that they're a good starting point. Genomes are massive, and the information complexity of genomes is extremely large. So we don't have good first principle techniques to answer a lot of bioinformatic questions that we might want to answer. But it seems like in the case of some of these models, they're able to learn things that make these tasks easier. It's going to take time. This is an important scientific project that's pushing the frontier of both transformer modeling in genomics And interpretability. But at the end of the day, just going back to the paradigmatic beliefs that Tom mentioned earlier, the model is doing something and it's doing something meaningful. And we have windows in on the model now. They might not be the right windows to tell us everything, but we have strong reason to believe that these windows are good enough to start telling us important things. And so that's really why we're pushing on this very hard.

That's true.

When you say it's going to take time, do you take the over on Dario's 2 years to a country full of geniuses in a data center, or is that the kind of time we're talking about?

I'm going to give a hot take real quick, which is that using mechanistic interpretability for scientific discovery, I think there's a decent chance that even if you were in a world where you had a bunch of geniuses in a data center, This would be their preferred way of doing science. There are so many barriers to running experiments in the physical world. Normative barriers, but also just physical barriers, paralysable barriers. The rate of scientific progress could be rapidly accelerated by moving as much scientific experimentation as is reasonably possible to simulation and doing it on ships. That looks to me a lot like what people are doing when they're training autoregressive genomic models or diffusion-based material models. They're running simulations of the physical world on chips. And interpretability then is your scientific toolkit that you go in to actually understand what that simulation is doing and extract principles from that simulation that can then inform Further experimentation and then ultimately real-world scientific progress. An interesting example here is how pharmaceuticals are created. They're often developed with something in mind, and then they're tried. They have some range of side effects and some range of things. Most of the time they don't work. The vast majority of the time they don't work the way that people think that they're going to initially. And then these companies just keep around these drugs because The process of manufacturing a new drug is expensive and then they get the patents and then they just try it for a bunch of other things and then eventually they maybe find something that works and that becomes the treatment for it. There's many examples of this and that is an extremely inefficient way to do science and that's going to remain an extremely inefficient way to do science even in a post-AGI world. So to me it just seems pretty likely that you're going to want to run simulations and if you're running simulations, why wouldn't you want to train giant models. And if you're training giant models, why wouldn't you want mechanistic interpretability to help you make sense of your simulation?

So I guess we have, you know, we're kind of doing all right either way, right? Either timelines are longer. It wouldn't be an AI podcast without timelines. I was wondering when we were going to get to it. Either timelines are longer and Matthew Tuff is useful, or what are the geniuses in the data center doing? Matthew Tuff.

Yeah, I think, I mean, that does make a lot of sense to me. As a convergent evolution, basically, right? I mean, either way, it sucks to have to do wet work. So, as much as possible, you want to move it all into silicon, learn as much as you can that way. Hopefully, it's accurate. It won't always be right, but one way I phrased the question in the past is, when is it worth it to do the wet work? And, as much as possible, you want to elevate and validate your hypotheses before you actually take it into the wet lab to muck around with. And whether it's human or AI genius steering that simulation process, for all the same reasons, it seems like it's where we end up.

Yeah, that's exactly how we think about it. The core thing that just needs to be true here is that mechanistic interpretability, as it matures as a science, significantly pushes the frontier of the types of experiments and the quality of the results of the experiments that you're able to run on hardware before bringing them to the wet lab. That strikes me as overwhelmingly likely to be true. And again, for us, we're doing this because we believe that there is a really meaningful possibility to impact people's quality of lives through bringing interpretability directly to these models that can have real downstream scientific impact. Another good example of this is biomarkers of disease. In clinical contexts, there's sort of 2 reasons that you would be interested in interpretability. There's the discovery reason that we've already talked about, but then there's the reason that being able to have explainability for diagnostics is really important. AI can often perform extremely well in these tasks and in closed settings. But if you have an AI system that misdiagnoses someone, you can't go to their family and tell them, sorry, it went wrong and I have no idea why and there's nothing I can do about it. This core debuggability is an essential feature in clinical contexts as well. But you also want to be able to say, oh, the AI gave a surprising recommendation in this case. It gave a surprising diagnosis. diagnosis. Was that diagnosis a result of the AI being wrong, or was it a result of some important input pattern that we hadn't seen before?

One quick aside on multimodality. Does this line of work— because you're doing this across reasoning models, for example, and across different models from science, genomic and There's obviously lots more to come in terms of proteomic and higher orders of abstraction there. But I wonder, one kind of mental model of superintelligence that I've been playing around with lately is basically that if you take a reasoning model at roughly the current level and you give it the same depth of integration into 20 modalities of interest, that we already now have with image from GPT-4o and Gemini Flash, where they can clearly manipulate the image in a way that shows a deep integration of your instruction and the visual space that it's operating in. It's no longer through this bottleneck point of having the language model prompt the text-to-image model. It's all joined in latent space, and you just obviously see the qualitatively different results. My sort of baseline superintelligence case has been do that again for 20 more modalities, many of them in sort of the natural world modelling domains. And there you'll have a superintelligence because you'll be able to kind of reason around and manipulate things. But the things you'll be manipulating will be much closer to the fundamental stuff of reality. And in many of those cases, these are just things that people can't do, right? Nobody has an intuition for how a protein is going to fold, for example, or at least nowhere at the level that an AlphaFold does.

So we talked a little bit about this last time. I want to start by saying that I have a wide distribution of timelines. Geniuses in a data centre in 2 years is not outside of my Overton window at all. But on this particular point, I'm just cramming a bunch of modalities in. I think there's a way in which we're still constraining AI progress by human data, even in this RL regime. That's worth understanding. Why is it possible for these models to reason over images and text really well? That's because we can construct tasks for which we have good reinforcement learning signals for which this is a capability that they need to acquire to complete that task. When you start bringing in the scientific domains as well, we're often working with extremely sparse signals where it's very hard to reason about What task could I easily train an AI to do such that a deep intuitive understanding of the human genome was necessary to perform that task? But then also we have enough examples and a clear enough sense of that task itself such that we could create a strong signal for reinforcement learning to begin with. And then what percentage of the overall task that the model would be performing do these integrate really well? I think it's not a coincidence that combining image and text in problem solving is something that humans do all the time. And this was one of the easier things from an RL perspective to train AI to do once you reach a certain level of intelligence. It's not just because those are easier modalities to combine in some abstract sense, though maybe they are. It's that we as humans are very set up as to how to think about and construct some optimization target at the intersection of those 2 things. So I think this gets at the question of, do you hit a wall when you get to human-level intelligence in some way, shape, or form? Or why would you or why would you not hit a wall when you get to human-level intelligence? And I think Tom and I might have slightly different beliefs here, but I think my belief is we just kind of don't know. It's easy to draw the line out from where we're seeing and say these patterns extend to everything. But I think it could also just be true that these patterns extend to the things that they've extended to so far. I think that we are very likely to be able to get— just to come back for a second to the question of agents, why has it been easier to build AIs that can solve elite coding problems better than any human than to build AIs that can order DoorDash for me successfully? One of these things seems like it should require much less intelligence in some sense to do, yet it's been much harder to get models to do. And I think it's just because it's harder to get good training signal in the agentic use cases. This is not something where we've captured a lot of data where you can write algorithmic verifiers very easily. And so as a result, it's been harder to do. When you're extending to the scientific domains, it's even harder. If we had a doing science good verifier, that would be great, but we don't. And I'm not saying that one doesn't exist. It's not possible to come up with one, or that maybe if we get general-level intelligences, they can work on this problem and make progress on it. But it's a level of abstraction for which we don't have any evidence and for which we have reasons to believe that actually setting up the conditions in which an AI could learn that task in the way that we currently train AIs might be pretty hard.

I suppose another way to put this is that the question here is like, are AIs currently experiencing catch-up growth in the sense that a less economically developed country might benefit from catching up to a more economically developed country? Or are they just on this like growth trajectory, hard to disambiguate? Yeah.

I mean, I think that is where the other modalities and the, especially the models from different fields of science seem to be pretty strong evidence to me that they can do pretty critical tasks at an obviously superhuman level, right? The ability to— and we've gone well beyond just folding, but interactions and multimers and all different kinds of molecules and metal centres now with some models and whatever. I mean, it's really pretty far along in terms of— I also did an episode on it, the guys from Orbital Materials, on figuring out the mechanism of the potassium ion channel. And it's like, geez, this has gone pretty far already. What hasn't happened there is the integration of this sort of chain of thought kind of thing with the more, what I call it, just intuitive physics for whatever different domain the model happens to be trained in. It clearly has a better ability to take a random set of proteins and guess how they'll interact than I do. What it doesn't have is the sort of outer loop to be like, I should try this, maybe I should try this. What if I swap this one out for this one? And so people are sitting there writing scripts against these models to do that outer loop, or just maybe even doing it based on their own human intuition, one by one. in some cases. But I have a hard time imagining a world where that integration doesn't happen. It seems like we can get enough synthetic training data and set up a situation where it's okay, here's what it looks like to just grind through a bunch of these things to get to the point where the reasoning and modality X integration kind of eventually just kind of drops into place.

Right.

I imagine that happening. I kind of can't imagine it not happening.

The question is the depth of integration, I suppose. Like, you might think that things are very deeply integrated in the sense of being in the same neural network, but even there you have no guarantee that you don't have 2 models in a trench coat. Like, this half of the weights does the quantum chemistry and this part of the weights does the language. And there's not really any, like, crosstalk such that the parts that are doing the language have any access to the sort of process knowledge of how to do the chemistry. And the way that you would expect to get this crosstalk is if you are able to construct paired data or data where the language ability is kind of bearing on the chemistry and vice versa. Then we can do that relatively easily and we can provide supervision of various sorts relatively easily in image language because we understand these domains both quite intuitively. Like, if we can give reward, As in, we can give approval and train a reward model and then do RLHF, or we can obtain paired data from the internet. But in the sort of, where does the paired data or natural supervision come from to couple these 2 modalities for domains that we don't have the same kind of intuitive understanding because we can't give the same kind of approval? Now, it might be that actually the answer is kind of more like you say. You put them initially, they're sort of two separate parts. There's just like halves of the neural network, let's say. And then you do some sort of training task where you gradually, like, you elicit the language reasoning in order to drive the chemistry part of it. But that sort of, that doesn't feel like the same sort of depth of integration as you get through pre-training where you really do seem to just have this like big massive compute, all more or less able to access the other. Well, actually, even that's not necessarily true. Like, how much of a language model, when it's talking about one domain, is it able to actually elicit its information from another?

Maybe let's just do other applications, the sort of move to circuits. And I mean, you could frame them conceptually or potentially by like customer profile. There's the sort of monitoring classification steering way of thinking about it, and then there's the retail and different kind of models that you're looking into. You could attempt both if you want to squeeze them in.

Yeah.

So I think there's 3 applications right now that we're very excited about. The list of potential applications is very long, but we're a startup and we need to prune the tree a little bit and focus on what we think the highest leverage bets for us to take are. So there's scientific discovery, which I've already talked a lot about, so I won't spend too much time there. But again, just to quickly summarise, scientific discovery is you have these models, they're modelling physical systems, they're able to model them in ways that traditional methods can't. So they must learn something that is important that human beings currently don't understand. We want to explain that. And the hope and the vision there is that those explanations are in and of themselves scientifically useful. In terms of pushing the frontier in science. The second application is more guardrails approaches. So what interpretability techniques give you is a window into the model at inference time. If you think about the current way that a lot of enterprises set up their guardrail systems, oftentimes they're playing whack-a-mole with the prompt at first when they're setting up the prototype. The number of rules that you can specify, the contextual information in the prompt balloons. You end up in this very natural situation where your task performance is degrading because of all of this information that you have to add. Keep telling the model, don't do this, don't do this, do this, don't do this. So people then move naturally to LM as a judge. But the scaling properties of LM as a judge are quite poor because now you have a separate frontier model call. It's great for the lab's pockets, but it's not great for the consumer. If you have 1,000 rules, maybe you can bunch them together in different checks, but at the end of the day, you run into the same problem of task degradation and just poor scaling properties. You could train a small model or fine-tune a small model to help guardrail your larger model, but many organizations both lack the data and machine learning expertise to do this effectively. So what we can do with our techniques is offer a cheap inference time solution for models where we can be watching the model's internal cognition and use that as an event when certain things happen, when the model is potentially going to output certain things or is outputting certain things or is reacting to certain things to trigger programmatic responses. So this could be things like if, for instance, The model looks like it might be thinking about PII, then let's flag that for someone for manual review in some way. Or it could be, for instance, there are certain topics that I never want my model to talk about. Of course, there are limitations to what can be done. There are jailbreaks or all types of things. But for real practical use cases, there's a lot of opportunity here to offer cheap, fast, and effective checks in real-world scenarios. It just has much better scaling properties. Then the third category of application that we're really excited about is creative models. We're going to be launching a demo soon. In fact, by the time this airs, it might already be out, demonstrating what you can do with image models when you start to understand their latents. And in unsupervised ways, we're able to recompose the elements of the image from this deep understanding of what's happening inside of an image model. We think this just offers new types of user experiences that even as image editing tools and other forms of AI continue to get better and better, you're still very much locked into these bespoke forms of interaction, whether that's by prompting or highlighting a region for inpainting. There are certain types of key interactions that we had in earlier classes of image design software, just like the ability to drag something or Reorient something or change some subtle property that get lost in that process. And so this image demo's like a cool example, but we think we could push this and the value prop's even more clear in video and music where the cost of editing them is very, very high. And wouldn't it be amazing to have generative music AI where you could say, actually, I want a little more saxophone in the saxophone solo, and you could just very strategically intervene on the output in a way to get— specific new generations that adhered to exactly what you wanted in some context.

Maybe just my last thing before giving you the floor for the sort of closeout pitch on Goodfire is like, where are we? I think it was maybe, geez, a year and a half ago that we first talked about, maybe just a year, time is compressed, the idea that with these tools you can sort of allow many more people to get involved in the process of understanding models. And I've been fortunate enough to have a couple early previews of different model steering and exploring interfaces that you guys have built. And I wonder how you would characterise where we are on that today. Are the interfaces far enough along that we are now effectively able to enlist the human intelligence, or are we still kind of working on that and still more in the sort of auto interpretability world because we haven't quite cracked the paradigm that allows people to like make intuitive use of all these new feature spaces that you're opening up for them?

I think it depends on the problem you're trying to solve, which I know that's not a very satisfying answer. But in the case of images, and you'll see in our demo, like images, you can reason about images and image features very intuitively. And there's ways to visualize them that don't rely on any external assumptions about inputs. It's purely about visualizing what that feature does. to the output. So there's fewer assumptions at least. And I think our flash image demo will demonstrate some of this in that context. There's been a lot of work internally at Goodfire and also at other labs, and especially in the open source community, at different UXs and visualizations that you can have. And they all solve problems. They're all a way of looking at it. Right now, we're not in a place where we have a single interface that can tell you anything that you need to know about an arbitrary model. We're working on some things in the biology domain that we hope will generalize. But yeah, it's like everything else, right? All these tools have limitations. It's about the problem that you're trying to solve and what you can learn in the process of trying to solve that particular problem.

Cool.

Well, it's been a fantastic conversation. I really appreciate the time and I just want to do it more regularly going forward. Take us home. Maybe give a few highlights, things that you're most excited about, things you guys are looking for. Last time, if I do say so myself, I understand there were a couple inbound business opportunities. So you can put the, you know, the bat signal out there for whatever you want to come your way this time.

Yeah. So at a high level, like we are looking for people to join our team at GoodFire. If you want to work on what we all think is the most important unsolved problem in the world right now, you can join us to help reverse engineer what's I think one of the really exciting things about Goodfire is we're pushing on this problem from a bunch of different angles. We're not saying, let's just push what we can do with these tools, or let's just develop new tools. We're doing both at the same time in dialogue and in concert with each other, which is really important to the ultimate progress. We're looking for great engineers and great scientists who are very motivated by our mission here and want to contribute to it. Open up the black box. And from a customer's point of view, if you fit one of the profiles that we talked about, if you're someone who's training a scientific model, if you're an enterprise looking for more reliable usage of your LLMs in production, or if you are someone training a creative model and you're just looking for new ways that you can open up the experience of the creative model, please reach out to us. You can email me at dan@goodfire.ai. We'd love to talk to you and understand whether or not there's a good opportunity for some collaboration there.

And I'll mention it at the top, but we should also probably mention you just raised a bunch of money, including taking Anthropic's first ever external investment. So what more beyond that headline would you impart to people?

Yeah, so we raised $50 million led by Menlo Ventures and $1 million of that from Anthropic as Anthropic's first ever corporate investment. Yeah, we're just extremely grateful to have investors who really believe in what we're building and the mission, the problem we're trying to solve here. We think it's really important to have a company not directly tied to a scaling lab that is directly trying to solve the problem of interpretability. And we intend to use all this money we've raised to help customers understand their models and to push the frontier of understanding. In mechanistic interpretability. And as we were saying before, hopefully then across a bunch of scientific domains and unlock lots of new awesome experiences that we can have with AI. Our core belief here is that interpretability is as big as AI itself. And so there needs to be a lab that's focusing on interpretability and nothing else because the size of the opportunity here is really large. So grateful to have all of the support from investors like yourself, Nathan, and all the people who like really believe in the thing that we're building and we're just motivated to get out there and start making things happen. Yeah.

Well, it is definitely a good candidate for the most important problem facing the world today. So it's been a pleasure and definitely a fascinating journey to just try and keep up with all the progress you guys are making. Again, come back soon, but for now, Dan Balsam and Tom McGrath, CTO and Chief Scientist at GoodFire. Thank you again for being part of the Cognitive Revolution.

Thank you so much for having us.

Yeah, thanks for having us.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.