Google DeepMind has a new way to look inside an AI’s “mind”

Welcome to MIT Technology Review. Narrated My name is Matt Honan. I'm our Editor in Chief. Every week we'll bring you a fascinating new in depth story from the leading edge of science and technology, covering topics like AI, biotech, climate, energy, robotics and more. Here's this week's story. I hope you enjoy it.

Narrated by NOAA Listen to more of the best articles from the world's biggest publishers on the NOAA app or@newsoveraudio.com Scott J. Mulligan writes Google DeepMind has a new way to look inside an AI's mind AI has led to breakthroughs in drug discovery and robotics, and is in the process of entirely revolutionizing how we interact with machines in the web. The only problem is we don't know exactly how it works or why it works so well. We have a fair idea, but the details are too complex to unpick. That's a problem. It could lead us to deploy an AI system in a highly sensitive field like medicine without understanding that it could have critical flaws embedded in its workings. A team at Google DeepMind that studies something called mechanistic interpretability has been working on new ways to let us peer under the hood. At the end of July 2024, it released Gemascope, a tool to help researchers understand what is happening when AI is generating an output. The hope is that if we have a better understanding of what is happening inside an AI model, we'll be able to control its outputs more effectively, leading to better AI systems in the future. I want to be able to look inside a model and see if it's being deceptive, says Neil Nunda, who runs the Mechanistic Interpretability team at Google DeepMind. It seems like being able to read a model's mind should help. Mechanistic interpretability, also known as mech interp, is a new research field that aims to understand how neural networks actually work at the moment. Very basically, we put inputs into a model in the form of a lot of data, and then we get a bunch of model weights at the end of training. These are the parameters that determine how a model makes decisions. We have some idea of what's happening between the inputs and the model weights. Essentially, the AI is finding patterns in the data and making conclusions from those patterns. But these patterns can be incredibly complex and often very hard for humans to interpret. It's like a teacher reviewing the answers to a complex math problem on a test. The student the AI in this case wrote down the correct answer, but the work looks like a bunch of squiggly lines. This example assumes the AI is always getting the correct answer, but that's not always true. The AI student may have found an irrelevant pattern that it's assuming is valid. For example, some current AI systems will give you the result that 9.11 is bigger than 9.8. Different methods developed in the field of mechanistic interpretability are beginning to shed a little bit of light on what may be happening, essentially making sense of the squiggly lines. A key goal of mechanistic interpretability is trying to reverse engineer the algorithms inside these systems, says Nanda. We give the model a prompt like write a poem and then it writes some rhyming lines. What is the algorithm by which it did this? We'd love to understand it. To find features or categories of data that represent a larger concept in its AI model, Gemma DeepMind ran a tool known as a sparse autoencoder on each of its layers. You can think of a sparse autoencoder as a microscope that zooms in on those layers and lets you look at their details. For example, if you prompt Gemma about a Chihuahua, it will trigger the dogs feature, lighting up what the model knows about dogs. The reason it is considered sparse is that it's limiting the number of neurons used, basically pushing for a more efficient and generalized representation of the data. The tricky part of sparse autoencoders is deciding how granular you want to get. Think again about the microscope. You can magnify something to an extreme degree, but it may make what you're looking at impossible for a human to interpret. But if you zoom too far out, you may be limiting what interesting things you can see and discover. DeepMind's solution was to run sparse autoencoders of different sizes, varying the number of features they want the autoencoder to find. The goal was not for DeepMind's researchers to thoroughly analyze the results on their own. Gemma and the autoencoders are open source, so this project was aimed more at spurring interested researchers to look at what the sparse autoencoders found and hopefully make new insights into the model's internal logic. Since DeepMind ran autoencoders on each layer of their model, a researcher could map the progression from input to output to a degree we haven't seen before. This is really exciting for interpretability researchers, says Josh Batson, a researcher at Anthropic. If you have this model that you've open sourced for people to study, it means that a bunch of interpretability research can now be done on the back of those sparse autoencoders. It lowers the barrier to entry to people learning from these methods. Neuronpedia, a platform for mechanistic interpretability, partnered with DeepMind in July 2024 to build a demo of Gemascope that you can play around with right now. In the demo, you can test out different prompts and see how the model breaks up your prompt and what activations your prompt lights up. You can also mess around with the model. For example, if you turn the feature about dogs way up and then ask the model a question about US Presidents, Gemma will find some way to weave in random babble about dogs. Or the model may just start barking at you. One interesting thing about sparse autoencoders is that they are unsupervised, meaning they find the features on their own. That leads to surprising discoveries about how the models break down human concepts. My personal favorite feature is the cringe feature, says Joseph Bloom, science lead at Neuronpedia. It seems to appear in negative criticism of text in movies. It's just a great example of tracking things that are so human on some level. You can search for concepts on Neuronpedia, and it will highlight what features are being activated on specific tokens or words and how strongly each one is activated. If you read the text and you see what's highlighted in green, that's when the model thinks the cringe concept is most relevant. The most active example for cringe is somebody preaching at someone else, says Bloom. Some features are proving easier to track than others. One of the most important features that you would want to find for a model is deception, says Johnny Lynn, founder of Neuronpedia. It's not super easy to find. Oh, there's a feature that fires when it's lying to us. From what I've seen, it hasn't been the case that we can find deception and ban it. DeepMind's research is similar to what another AI company, Anthropic, did back in May with Golden Gate Claude. It used sparse autoencoders to find the parts of Claude that lit up when discussing the Golden Gate Bridge in San Francisco. It then amplified the activations related to the bridge to the point where Claude literally identified not as Claude an AI model, but as the physical Golden Gate Bridge and would respond to prompts as the bridge. Although it may just seem quirky, mechanistic interpretability research may prove incredibly useful as a tool for understanding how the model generalizes and what level of abstraction it's working at. These features are really helpful, says Batson. For example, a team led by Samuel Marx, now at Anthropic, used sparse autoencoders to find features that showed a particular model was associating certain professions with with a specific gender. They then turned off these gender features to reduce bias in the model. This experiment was done on a very small model, so it's unclear if the work will apply to a much larger model. Mechanistic interpretability research can also give us insights into why AI makes errors. In the case of the assertion that 9.11 is larger than 9.8, researchers from TransLuce saw that the question was triggering the parts of an AI model related to bible verses and September 11th. The research concluded the AI could be interpreting the numbers as dates, asserting the later date 911 as greater than 9 8. And in a lot of books like religious texts, section 9.11 comes after section 9.8, which may be why the AI thinks of it as greater. Once they knew why the AI made this error, they the researchers turned down the AI's activations on Bible verses and September 11, which led to the model giving the correct answer when prompted again on whether 9.11 is larger than 9.8. There are also other potential applications. Currently, a system level prompt is built into LLMs to deal with situations like Users who ask how to build a bomb. When you ask ChatGPT a question, the model is first secretly prompted by OpenAI to refrain from telling you how to make bombs or do other nefarious things. But it's easy for users to jailbreak AI models with clever prompts, bypassing any restrictions. If the creators of the models are able to see where in an AI the bomb building knowledge is, they can theoretically turn off those nodes permanently. Then even the most cleverly written prompt wouldn't elicit an answer about how to build a bomb, because the AI would literally have no information about how to build a bomb in its system. This type of granularity and precise control are easy to imagine, but extremely hard to achieve with the current state of mechanistic interpretability. A limitation is the steering or influencing a model by adjusting its parameters. It's just not working that well, and so when you steer to reduce violence in a model, it ends up completely lobotomizing its knowledge. In martial arts, there's a lot of refinement to be done in steering, says Lyn. The knowledge of bomb making, for example, isn't just a simple on and off switch in an AI model. It most likely is woven into multiple parts of the model, and turning it off would probably involve hampering the AI's knowledge of chemistry. Any tinkering may have benefits, but also significant tradeoffs. That said, if we are able to dig deeper and and peer more clearly into The Mind of AI, DeepMind and others are hopeful that mechanistic interpretability could represent a plausible path to alignment the process of making sure AI is actually doing what we want it to do. You are listening to MIT Technology Review, where Scott J. Mulligan writes, Google DeepMind has a new way to look inside an AI's mind. This article was published on 14 November 2024 and was read by Sam Scholl for Noah.