Mapping the Mind of a Neural Net: Goodfire’s Eric Ho on the Future of Interpretability

So Goodfire is an AI interpretability research company, really trying to answer the question of what's actually going on inside the mind of a neural net. So kind of the ultimate goal and the ultimate reason why we started everything was like we just see neural networks kind of going into more and more mission critical contexts. And I think it's going to be enormously transformative for society, but in order to do so, and you want to build it safely, powerfully, reliably, and I think it's going to be critical to be able to understand, edit and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time. It's like unlocking the black box of a neural network such that you can intentionally design it rather than just kind of like grow it from data.

What if we could crack open the black box of AI and see exactly how it thinks? Today we're joined by Eric Ho, the founder of Goodfire, who's building tools to peer inside neural nets and understand their minds. Eric reveals how his team has successfully disentangled the mysterious phenomenon of superposition, where single neurons encode multiple concepts and can now steer AI behavior with increasingly surgical precision. We explore whether interpretability could help us discover new biological insights, edit out harmful behaviors from large language models, and even understand our own brains better. Eric boldly predicts that we'll fully decode neural nets by 2028, transforming AI from black boxes into more intentional design. Enjoy the show. Eric, thank you so much for joining us today.

Of course. Yeah, happy to be here. Thanks for having me.

First question, can we ever trust generative AI if these foundation models are very much black boxes?

Can we ever trust them if they're black boxes? So I guess maybe thinking about what would happen if we were to just kind of deploy AI models as black boxes in perpetuity. So the black box way to do this would be like, and I'm kind of assuming that we're playing this forward a few years and we want AI in charge of really mission critical applications, like maybe being in charge of our power grid or making big investment decisions, maybe even for a seed investment at Sequoia or a really large million dollar investment decision. And I think the black box way to make sure that the AI is performing appropriately is you take a look at evals and you run a bunch of evaluations to make sure that it's behaving appropriately in test sets, and then you look at its track record and see if it's reliable enough. To perform across a wide variety of things. I think the question then is why not take all of this additional signal that you get from looking inside a neural network and trying to play forward how it's going to behave in a much wider, broader set of situations? Why not look inside and actually get a bunch more reliability, certainty about how it's thinking, how it's approaching the problem? And I think you're just leaving a bunch on the table if you're not looking for all the signal that you can get. So the way that I think about this is, I don't know, when you're manufacturing a new drug, you can do the black box way of just seeing how humans respond to a drug in a clinical trial, whereas you could also just kind of look inside and look at biochemically how the drug is processed or drug interactions at the molecular or cellular level. And yeah, I just feel like there's so much to be learned when you actually look inside and deeply understand something.

How possible do you think it is to look inside and deeply understand a large language model? Do you think it's on the scale from hopeless? We can't ever understand it. It's just a black box, too many neurons. So we can actually map out the mind of a neural net. I'm curious where you think the field will be.

Well, I'm very biased, but I think it's very, very possible. A lot of the people in mechan terp come from backgrounds in computational neuroscience or cognitive science. And those people, when you're actually looking inside the brain, you spend so much time trying to understand what a single neuron does, or just getting any signal whatsoever. And in the field of mec interp, you have perfect access to the neurons, the parameters, the weights, the attention patterns of a neural network. So you're coming in with a huge advantage for at least you get all of the data that you need. So then the real question is, how can we make progress? How can we try to understand and seek to understand all of it? And I think we just gotta try. I think it's deeply necessary and critical for the future. And we have a norm established, we can explain some percentage of the network by reconstructing it and extracting its concepts and the features that it uses in order to generate its response. And once you have at least a baseline, rudimentary understanding kind of where we're at right now, you can hill climb on that metric and seek to understand like more and more of the network.

Do you think it's going to be necessary for us to understand neural nets, to really harness them long term. Because I think many other technologies we've invented along the way, humans didn't really understand underlying physics or chemistry, but still we were able to make good of medicines or, you know, totally basic propulsion techniques without understanding, you know, all the physics.

Yeah, I think it's going to be critical for the future, just given how transformative I think AI is going to be. I think AI is going to be everywhere, running mission critical parts of our society and we can get really, really far just by treating AI as a black box. But I don't think we can truly be able to intentionally design AI as the new generation of software without white boxes techniques. So maybe one example I think about is in the early 17th century you invented the steam engine and we're able to just increase the size of the boiler and increase the amount of pressure going in. And it scaled reasonably well, but steam engines also blew up. Well, we didn't understand thermodynamics at that time, so we didn't actually know at what point did the ideal size of the boiler or the ideal pressure the ideal way to construct a steam engine. And so after we invented thermodynamics, things started becoming a lot safer, a lot more reliable and huge innovations happened afterwards. But already the steam engine kicked off the industrial revolution. So even just by treating it as a black box, you get a really long way.

Do you think there's any chance that if we understand neural networks in a computer science context, it might actually help us accelerate our understanding of neuroscience for the human brain?

I think so, but that's a big claim. I think we were just having an interesting conversation last night. We had a dinner together about do we think in language, do you think in concepts or something else entirely? I'm a person that doesn't really think in language. I think much more conceptually, maybe in the latent space of models. Whereas our head of product, Myra, said that she basically is totally faithful to her own chain of thought, speaks in language, and she basically just thinks sequentially with a really, really strong internal monologue. So short answer is maybe some of these insights that we get will translate to humans and our own psychology. And I think that's the hope in it's like, yeah, the more that we can understand about AI, hopefully the more that we can understand about ourselves.

There's an interesting analogy by the way, that a neuroscience, often things that have gone wrong help illuminate and create insights into the human brain. People who suffer from specific conditions or people who've suffered particular brain injury types have actually perversely enabled us to better understand the brain accidentally. I wonder if something similar might happen with neural networks as well.

Yeah, I hope so. What's that, like, popular story about the guy who just got an iron Phineas through his brain and made him a totally different person? There's also this to add to that. There's this concept of universality where among totally different neural networks, similar kind of circuits or thought patterns tend to emerge between all these neural networks, even in we found in vision models, very similar kind of circuits to our own visual cortex. And I think there's this idea of universality where maybe intelligence is just this thing that you gradient descent to. And then that's how our brains found intelligence and that's how artificial minds would find intelligence as well. There's some truth to intelligence.

My own neural net is probably pretty sparse.

It's pretty sparse.

I would love to double click into some of the results you've had from macinturp and the broader field and your lab. But before we get into it, can you say a word about Goodfire and what you all are building?

Yeah. So Goodfire is an AI interpretability research company, really trying to answer the question of what's actually going on inside the mind of a neural net. So kind of the ultimate goal and the ultimate reason why we started everything was like we just see neural networks kind of going into more and more mission critical contexts. And I think it's going to be enormously transformative for society. But in order to do so, and you want to build it safely, powerfully, reliably, and I think it's going to be critical to be able to understand, edit and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time. It's like unlocking the black box of a neural network such that you can intentionally design it rather than just kind of grow it from data.

And if everything goes right, what do you think will be the impact that you all have on the world?

So maybe one metaphor that we like and think about is, yeah, right now you just kind of grow AI from a seed and then it just grows, grows like a giant tree and it grows all wild and crazy. Right. We don't really know a lot of the things that it's growing into. And all sorts of interesting and weird stuff can happen with a really large neural net. But if everything goes right with interpretability, we'll know how every single piece of training data affects the cognition that the model develops, the units of computation that it uses. And I almost think of it more as like bonsai, where you want to kind of intentionally design and shape and grow the neural network. Still, in an unsupervised, AI driven approach, we're not going to hand prune every single weight of a neural network, but I think we'll gain the ability to, during every single piece of the training, post training process, just intentionally shape an AI model such that it serves humanity and does what we want.

Sounds like a parallel to the Human Genome Project in some sense. Yes, given some of the work I've done in genetics. So this idea that we need to read DNA, we need to understand the building blocks of life, and then ultimately now we're starting to edit DNA and use CRISPR to come up with interesting cures for diseases, or an ability to edit crops to make them more resistant to pesticides and things like that. So it's just. It's a very interesting parallel.

Yeah, definitely. I think we think about that analogy a lot. And I know you had Patrick Hsu on the podcast at some point as well, and we're working with him at AHRQ Institute to crack the code of the human genome as well. And I think there's a lot of really interesting parallels and also direct applications of AI interpretability as well.

Would you go so far as actually making edits? What I heard from you earlier, the bonsai analogy was a little bit of a shaping, which is quite different in my mind from editing. Shaping to me might be train and be fit so that your body can survive given a certain DNA. And then there's editing which is altering the DNA. Are you going to do both?

Yeah, I think, in short, yes. I don't know what the. So interp as a field, there's still a lot to figure out. It's still pretty new. But I think in bonsai you also prune a lot of branches and you prune a lot of the areas that you don't want to grow, such that you can kind of shape the overall tree to grow in the pattern that you want. So I think the eventual system that we hope to build, you can ask questions of the model, like why did you come up with this response and get a faithful explanation, while also being able to make direct surgical interventions in the mind of the model, such that we can remove harmful behavior, enhance good behavior, and still remains to be seen whether it's just a direct weights modification or some other kind of shaping function that is most effective.

If you think of some of the ways that people are trying to prune these bonsai trees today. I think it's a lot of prompt engineering, fine tuning, RL tuning increasingly. Now, what do you think about that as the approach to steer the behavior of these models versus actually go in and introspect and examine each of the individual neurons?

Fundamentally, these are black box things. All sorts of weird things can happen when you fine tune a model, for example, or prompt a model and take it out of distribution. And it can say all sorts of crazy stuff. So the paper that's most interesting about this recently, I don't know if you've caught this, is this emergent misalignment study? No. Okay. This is like Owen Evans group, where if you fine tune a model on just insecure code, so it's just like bad code that has all sorts of cybersecurity vulnerabilities, It'll then start doing all sorts of insane things like wanting to enslave humanity or praising Hitler and other dictators. And it's a really surprising result because it's just insecure code. And so it kind of shows that there's like, maybe like what you're doing with fine tuning is you're telling the model like, hey, do more of this, less of this, and almost enhancing the circuits that you want kind of more of. But you can also have all sorts of unintended consequences like this that show up. And these circuits, these are still really alien cognition. There's some parallels to humanity, but we really don't understand how these networks think, and they're not human thinking. So if you enhance the bad code snippets, it also is fundamentally linked to maybe all sorts of other undesirable behaviors and properties.

There's a different twist on the nature nurture debate, because in that situation, it almost feels as though you've imbued that particular model with bad DNA, if you will just, you know, it's sort of fundamentally an evil thing or a bad thing, and then it ends up with manifesting all sorts of bad behavior in other domains. It's really interesting.

Yeah, maybe. Or maybe these models kind of understand right and wrong, and if you enhance the wrong, then all sorts of other behavior is interlinked and expressed. But I don't know the way I think about it, like these models, the models are just the functions of their training data. And these models are trained on everything, all sorts of misbehavior as well. And you want it trained on incorrect behavior as well, because otherwise it won't know to Refuse harmful requests, or to not do a certain set of things.

Do you have an intuition for why different base models have different personalities? Like, for example, the newest Claude series? I think one of the models, maybe Opus or someone, really cares about animal welfare, for example, and the others don't. Do you have a sense for why these models develop pretty distinct personalities?

I think it's just a function of how they're trained and it's really, really hard to anticipate in advance. Yeah, I don't know. I feel like it might just be me, but Claude 4 opus, too, is enormously sycophantic. I'll kind of nudge it in one direction and it'll just agree with me wholeheartedly, and then nudge it in the other direction, pose a counterexample, and it'll just be like, yes, I was totally wrong before. Nothing I said earlier was correct. And I just think it's really, really hard. It kind of goes back to the witchcraft almost of training an AI model today, where you're just like throwing in training data into the model and whispering the incantation of gradient descent and then trying to get what you want out of it and something pops out and then it really cares about animals. Oh, great. That's great.

I'd love to talk about your research results so far, both at Goodfire and in the broader macinterp field as a whole. Maybe could you just give US A. A 30,000 foot flyover view of Mecanturp as a field? How old is it? What are the key results so far? What are the big open questions?

Yeah, so mecanturp as a field. I think maybe just in this tradition that we're building on, I think there are all sorts of studies looking inside neural networks, all the way back when we first designed neural networks. But I think once we. I think the way that the field kind of thinks about itself, like mechanistic interpretability, was started at OpenAI with Chris Ola and Nick Camerata and a couple other folks who first put out this really big circuits thread that posited three things. One is there are features in neural networks, which are directions in latent space that represent concepts that the model uses to generate its response circuits. These are features that fire together to create higher order concepts. The example that they lay out is you have a car window detector and then a car body detector and then a car wheel detector, and then that's a car circuit. And universality is the third tenet. Similar circuits evolve in different neural networks. And so this was almost like the start of the field of mechanistic interpretability in my mind. And so that really kicked off a lot of interesting research and results in the feature circuits paradigm. And I think the main players in the field, there's a lot of academic labs that are doing great work, like Anthropic. So Chris Ola is one of the co founders of Anthropic and building a great interpretability Lab there. And DeepMind has an interpretability lab as well. And then we were kind of like the newer entrants on the field and on the stage. And I think one of the other really key things to have happened was understanding and mostly resolving superposition. So superposition is this idea that each neuron is responsible for encoding multiple concepts. And there are more concepts than dimensions in a neural network. So if you think about a neural network as a giant compression algorithm, you're compressing the entirety of the Internet into a relatively small number of parameters. So that means every single neuron needs to encode, or at least every single layer of the model needs to encode more concepts than it has dimensions. And so there's this concept of superposition where you have concepts represented as near orthogonal directions in latent space, such that you can represent all of these concepts in a model's latent space. And so to resolve this, you have to almost untangle and unscramble a neuron such that it's responsible for one clean interpretable concept. And so a group at Apollo Research led by Lee Sharkey, who's actually now at Goodfire, first pioneered sparse autoencoders for language models. And Anthropic also really popularized this with their big paper towards monosemanticity. And then right afterwards, scaling monosemanticity, showing that you can essentially unscramble these neurons into higher order concepts reliably and at scale with arbitrarily large neural networks. And I think that was a really big moment for interpretability where you can now, in a totally unsupervised way, unscramble neurons of a neural network to understand them and get clean concepts. So the concepts aren't totally clean yet. You can't edit them super well, there's all sorts of problems with this, but it's almost like a really big step forward for the field, such that that we can do this in an unsupervised way. And the techniques and interpretability scale, which.

Is really important, does that mean the superposition isn't real or that you almost like Heisenberg's uncertainty principle, you sort of Collapse it at a particular moment in time to know that in this instance it represents a particular direction.

So I think it means it was real. Neurons are responsible for encoding multiple concepts, such that once you unscramble them, then you can do really interesting things with a clean neuron. So the way that we do this is using an interpreter model train on the activations of a base model. And then now you have all sorts of neurons in the interpreter model that represent theoretically clean, sparse concepts.

In the interpreter model, not in the interpreter model, the original model still has this characteristic of superposition.

That's right.

Okay, got it. Thank you.

And in the interpreter model, you unscramble these neurons and you associate these concepts with the concepts in the base model that you're trying to interpret. And then you can do interesting things with it.

Got it. Thank you.

Yeah. Of course.

How solved of a problem is this then? If you've already been able to kind of disentangle the superposition, then haven't you already mapped out the mind, so to speak, of the neural net and what's ahead?

I think partially, I think it's a partial mapping and the technique has all sorts of flaws as well that we can improve upon. But I think it gives us the first step towards understanding these models, especially going from toy model to actual network that people care about. We've done a bunch of work recently on R1, so it's a 671 billion parameter mixture of experts models. It's a big boy model. And the techniques scale really nicely all the way up to that point because it's just more AI, more training of an interpreter model.

So obviously I presume there's an asymptote here to understanding because the models are going to get more and more complex over time and we're going to beef them up. And so I'm guessing it's sort of like the Battle of Sisyphus at some point. This is a never ending pursuit, which is great. Is that correct? Do you agree with that?

In some ways. But I also think that. So the techniques that we've developed work on toy models all the way up to big network. That's like more capable and more intelligent and better. And I think the techniques also scale effectively with model intelligence. So one part of our pipeline is that for every single latent concept in our interpreter model, we associate that with and try to, we get another language model to reason about what that concept actually represents. In the base model. This is a concept called auto interpretability, which Nick Camerata, who's at Goodfire now invented this technique at OpenAI pioneered. And this technique, because it's a language model, reasoning about what a neuron represents scales with the quality of the language model. So it actually gets better. So because we use AI in order to understand AI, the better that our models are, these analysis agents are at interpreting what's actually going on, the better we are able to understand them. And our interpreter model techniques also, it's just like we develop better interpreter models, they theoretically should translate to more and more intelligent and larger and larger networks because these are unsupervised scalable techniques. And that's the paradigm in AI interpretability.

Got it. When do you think you reach a minimum threshold that makes you feel it's ready for a real world application? Maybe we're there already.

I think we're there. I think the first real world applications are already out there. And yeah, I think we're there on the very early applications.

Can you share more about this?

Yeah, I was being unnecessarily cryptic. Yeah. A couple of the partnerships I'm most excited about. So we worked with ARK Institute, like I mentioned a little bit earlier, to understand and interpret Evo2, which is their kind of DNA foundation model. So it's a sequence to sequence model. So it takes in a sequence of nucleotides and it predicts the next nucleotide in a sequence. And our theory is like, this is a narrowly superhuman model. So we really like to work on narrowly superhuman models because it can teach us something about the world that humans don't really know. And so the idea is this model is representing just an enormous amount about the biological world in order to generate the next and properly model the next nucleotide in a sequence. So what we did was we sought to understand what does it actually know such that it can model the world so effectively. So what we did was we trained sparse autoencoders on the activations of this model, extracted all sorts of features that were related to concepts that the model should know, like kind of normal biological concepts that we have really strong ground truth annotations for. So these are TRNAs, RNAs, star coding sequences, all sorts of biological concepts that we have, ground truth annotations. Are we associated with this model? And then the question is like, okay, now we have all of these other features of the model that we've extracted. What do they mean? What are they? They might just be ways that the model is computing and thinking, or they could represent novel biological concepts that the model's using to generate the next nucleotide in a sequence.

That's really interesting. For a long time, there was this idea that we have a bunch of junk DNA. You may have read about this, and turns out a lot of that DNA actually serves a particular purpose in a different part of evolution, or that they govern the expression of other genes. And so nature generally doesn't want to harbor things that don't have value because it's expensive, just from a biological system point of view. So that's super interesting. I'm looking forward to the results.

Yeah, totally, totally. And I think hopefully using unsupervised AI techniques, we can better understand what all these portions of the DNA are actually doing. Maybe we can discover the idea of junk DNA faster, or understand that DNA is not junk DNA faster, or just discover totally novel things that genes are doing and expressing within us.

Where is the research as far as going from understanding and mapping towards editing? So, for example, being able to reach in and change this weight from here to there. I'm curious if you all have any results there yet.

Yeah. So we've done most of our editing work on language models and image models. Our most recent release was a paint with Ember. Ember is our foundational infrastructure for interpretability. And what we were able to do with this image model demo was targeted precise control over an image model by painting. So we could extract latent concepts like a dragon or dragon wings or an ocean or a pyramid, and then take these concepts and directly intervene on the portion of a canvas that we want to intervene on. So you can paint on a dragon with wings and then add a crowd in the corner and add a pyramid. It's a really fun demo that's just kind of a joy to play with. So it's out right now. Anybody can play with it. It's just like paint Goodfire AI, but we're able to reasonably intervene in certain situations on a model's latent and steer the model to do what we want. But we haven't quite cracked the idea of direct, precision surgical edits that create a new model that you want to use and doesn't have any unintended side effects. So that's still something that we're pushing on and trying to figure out.

Do you think that's where the field ultimately goes, or do you think people are focused on different parts of the field?

I think there are many places where the field is going to go, and this is one of them. Interpretability is almost such a general term that, again, I'm biased, but I think it's just Governing and underlying all aspects of AI. It's like anytime you prefer to take a white box approach to doing something versus a black box approach, interpretability can probably help in the future. So how do you select your training data? Maybe you want to understand whether the training data is surprising to the model before putting it into the model because then it can have the most impact on training. Yeah, just again, every single part of the AI development stack, I think interpretability will help and change the way that we do things.

If AI foundational models go the way that a lot of software has gone, certainly infrastructure software, where much of it is open source or open weight, is there an opportunity for you to play an invaluable role in judging the biases or likely outcomes of using different open weight models?

I think we could, yeah. So there's maybe two areas of research that we're interested in that intersect with this idea. So auditing. So how do you take a model, understand what's going on, find problematic behavior and good behavior, hopefully get rid of the bad behavior and enhance the good behavior. So I think as AI gets deployed in more and more mission critical contexts, that becomes more important. And then also model diffing. So it's like when you have two checkpoints of a model, how do they differ from each other and what's changed? So the recent kind of like GPT4O was enormously sycophantic for a period of time. Just like telling, just like really gassing up the user, like telling them that they're doing great. You still is.

Pat recently asked it who the most handsome cribble board member was and it was like, definitely, definitely Pat greedy.

Definitely.

Still a bit sycophantic.

That's so good. That's so good. Yeah, but yeah, like model diffing, you should be able to detect how a model has changed from checkpoint to checkpoint. Like what surprising things have happened that were unintended, that are now contained in the network that weren't there before?

Why do you think it was so hard for OpenAI to roll back to a less sycophantic version of the model? And in an ideal state of the world, is there almost a dial and a knob that the OpenAI guys could tune? On a scale of 0 to 100, how sycophantic do you want the model to be? Do you think we can get there?

I don't know what questions you're asking of the model, by the way, because I never encountered this particular problem, always.

Does this with me and then sometimes it's brutal. The other way. I ask at the best AI podcast and it lists 20 things with no training data. I ask, what about, what about us? It's, oh, I didn't want to, I didn't want to give a biased result.

That's so funny. Well, that's, that's part of what users want, right? They kind of want sycophancy. Like, you know, people want to hear what they, what they want to hear. So when you rl a model, I think fundamentally you're going to get. I think it's just kind of a symptom of rl. This is what users want. This is user preferences.

Along the way you've dropped some names and it seems as though most of them have ended up at Goodfire. I presume there is a certain number of very talented people in this field and you've unfairly seemed to gather them. Can you describe a little bit more about your team, what you've pulled together?

Yeah, I think we have a really fantastic team and that's what we've been spending a lot of our time on the last year, just kind of, I think assembling a team of world class interpretability experts that really have a shot of cracking this problem. So it starts with my co founders. I had worked with Dan Balsam, our CTO for many years at my previous company and so he's our CTO and our chief scientist, Tom, who founded the interpretability team at Google DeepMind way back in the day and just have assembled many of the early folks in the field. So Tom, Nick Camerata, who's working very closely with Chris Ola, who is generally considered the founder of the field of mecanturp and Nick was on all of the original circuits papers and helped build everything out at OpenAI. Lee Sharkey, who was the first who pioneered sparse autoencoders on language models and is now working on some really interesting work in weight spaced interpretability. So most interpretability techniques that have been deployed into applications are in concept space and activation space. And he and his group are working on weight space interpretability techniques. And we've also just kind of pulled in scientists, senior scientists from other fields who care a lot about interpretability and just kind of have realized that this is one of the most important problems that we can work on. So Owen Lewis, who was a senior staff RS at Google working on coding agents, came over and is now leading a couple directions here for us.

And you're recruiting, right?

And we're hiring. Yeah, scientists, engineers. I think it's like we are Hiring scientists and that is deeply important for the future of the field. But also it's hard to just, it's hard to overestimate just how important good engineering skills are.

It's an incredible team.

Proud of the team. Yeah, for sure.

This seems like core functionality for any of these foundation model companies to have. And as you mentioned, Chris ola was at OpenAI. Now Anthropic OpenAI has their interpretability team as well. How do you think about the rationale for having a standalone mechinterp research company versus being inside one of the labs that should care deeply about this?

I think we can just take a really different approach if we're independent. I think the benefit of being independent is we can think independently, push things forward independently, and also get a broader view of the ecosystem. So usually if you're within a lab, you're kind of doing interpretability work on your own models and kind of pushing forward the field in that way, and you can make incredible progress that way. But I really do think that a unique third party perspective is deeply necessary in the field. And I think, yeah, just given the team that we've assembled, a lot of those folks agree with that and that's why they've joined and also gives us an ability to work with lots of interesting partners across different domains and we can kind of unify those insights across all of these different domains that teach us more about the inner workings of neural networks more broadly. So we work across modalities like genomics models, exomic models, image video language, and also across model architectures. And I think all that just helps.

Anthropic invested in you. All right?

Yeah, that's right.

Say more about that and how you partner with them.

Yeah, so I think we were their first ever investment. They put in a check in our last round and I think they just really care about interpretability and really kind of see the future as we do, where interpretability is just pretty critical to the future. So Dario just published an essay called the Urgency of Interpretability and it's one of his four essays that he has on his site. And just talking about how he views this as almost like a race. And we see that very similarly. A race to get interpretability prior to super intelligent, really, really intelligent AI models. I just think it's deeply critical to be able to understand these models before we have, in his words, a country of geniuses in a data center.

Do you think interpretability can help us with open models? And I think some people have a fear that models trained in other countries that may or may not be enemies of the United States have different nationalist properties. Can interpretability help us understand and even modify those for the American variant of some of these models?

Yeah, well, I definitely think so. I also think it's relatively easy to, if you take a deep SEQ model, for example, it's relatively easy to just tune it or add in more training data to remove a lot of the propaganda in the model. But yeah, I think interpretability can help understand what's actually inside of the model and then also change it and edit it to serve whatever end purpose that you want.

How long do you think before you're going to be called in as a witness in a very important trial to try to understand why model did something in particular?

That's a good question. I think a few years, who knows? I think it's really, I mean we're all sitting in the Bay Area right now, but at this point I'm pretty AGI pilled in that I think AI progress will be pretty fast and pretty quick and transformative to society in ways that are really difficult to anticipate from where we're sitting right now. And so yeah, I do think that there will be a couple big failure cases of AI models. And whether it's me called in or somebody at a big lab or some other expert, I think that we're going to want to be able to explain a model's outputs.

Yeah, I agree with you on the rate of development, by the way. I think it's. You've read these articles. The human brain doesn't intuit compounding.

No.

And so, you know, I've even thought back to 20 years ago when I first met the self driving car initiatives and Sebastian Thrun's team from Stanford had won the DARPA challenge. You know, you could sort of see the glimmers of self driving cars. But even then if you'd said, you know, 20 years later you would have a self driving car in San Francisco take you around, I'm not sure it would have been obvious that would be true and maybe took a few years longer for the true visionaries. I think the same is going to happen with AI. I don't think we fully fathom what the world's going to look like in 2030 or 2035.

Yeah, I couldn't agree more. And it's just really hard to predict. Even if we feel like it's going to happen really, really quickly, it's hard to predict all of the ways that society will be transformed.

Good note to end on. Should we do some rapid fire.

Some predictions.

Yeah, some predictions. These are recorded, so we'll hold you to it.

Great. Yes. Yes. Maybe Eric in 2035 will look back on how wrong he is with all these predictions.

Okay. Maybe first inference time compute is the next important vector to scale models. Agree or disagree?

Mostly agree. I think it's one of the important, one of the things that we can scale up on. Yeah.

What application category do you think will break out next after code?

I think there will be a lot of enterprise transformations that happen. So just like automating manual routine tasks that people are doing like many, many times a day.

Employment impact from AI.

Vast, vast. But once you cross a chasm, I think it happens quickly. I think my last company was helping find early career jobs for people and using AI to automate that and I think that that's where we're going to feel the impact first.

I agree. Recommended piece of content or reading for AI fans. Maybe specifically in your field.

I think the original circuits thread that I was referencing a couple times, that's still fantastic.

What's either an AI app or maybe just an experience that you've had with AI that has blown you away recently. Something that just took a breath away.

I think one of those moments that you really just feel how fast AI is happening. When I first played with 01 Pro, that was a model that I just really felt was actually reasoning about the world and seeing the kind of cross domain transfer too. I would ask it a strategic question and it felt like it would actually understand all the levers that I was considering with the business and considering that at least relatively thoughtfully and being a thought partner. And that's both exciting because I now have this model that I can talk to about all sorts of critical problems but and not just of course, trust it blindly, but also it's like, wow, how did this happen?

Interesting. One of the things I've learned recently is that AI is still struggling to understand humor. And one of my partners, Andrew, actually had this joke that humor is human's way of showing off intelligence without actually explicitly bragging. So maybe there is a lot of embedded intelligence and humor. Do you think interpretability will help us pinpoint sense of humor?

Yeah. To figure out why AIs don't have a sense of humor or help them to develop it perhaps, who knows? Yeah, I hope so. Yeah. I think if I wake up to a model telling me jokes in Roloff's voice.

That'D be terrifying. Okay, we'll close with one last question on a prediction in your field. Do you think we will ever reach the point where we feel like we confidently understand the features, the circuits, the patterns, the weights of a neural net? And if so, what year do you predict we'll reach that point?

I think we can. I think that it might not look like what you just said, like the features, the circuits. I think it requires maybe a reconceptualization of what's actually going on inside the model, like a deeper, more fundamental understanding of the units of computation of a model. It's almost like discovering truths about the universe. Or about neural nets in this case. But, yes, I think we're on track. I think we can do this and. I think we can do this and hold me to this. In 2028, we're going to figure it all out.

Yeah.

Fantastic.

Yeah, just a few years. I think we're close.

Just in time for the LA Olympics.

Yeah, that's right.

Just in time for your next round of funding. I'm kidding. Eric, thank you so much for doing this today. Rouleth. And I love the conversation.

Thank you.

It was a pleasure. Yeah. So much fun. Thanks for having me, Sam.