New Anthropic Study: AIs Hide Plans, Cheat Quietly

Have you wondered how exactly these powerful LLMs work? So normally the assumption is that these are next word predictor models, where given the distribution of the input data, they use that distribution to predict the next word that is going to come in the output. Turns out that is probably not what is happening inside these large language models. There's a fascinating research from Anthropic in this blog post tracing the thoughts of a large language model. And in this they're trying to explore what happens when we ask an LLM to generate a response. Is it just predicting the next word? Or something a lot more interesting is happening? Are these models actually faithful when they're generating the responses? So there were very interesting findings that even surprised the authors and the researchers from Anthropic. We're going to look at some of those findings in this video. Now, the first question is, how are these LLMs trained? Usually there are two steps. One is pre training and the other one is post training. In pre training, you take trillions of tokens and pass them on through the LLMs that have billions and billions of parameters. Now, the assumption is that based on the pre training data, the LLM will understand and learn the nuances of language. And during the learning process, the LLM is updating these connections and weights and it learns about the language. And our assumption is that based on the training data that it has seen, it will learn the relationships and also learn to predict the next token with the maximum probability. Now, the team from Anthropic actually wanted to figure out if that's exactly what is happening. So they did a couple of studies in which they wanted to figure out how Claude models things. And probably similar findings also apply to other LLMs, especially the large ones. So there were three different focuses. One was CLAUDE can speak dozens of languages. What language, if any, is it using in its head? So during the training process, these LLMs have seen multilingual data. Now, the curious part is when it's actually trying to generate an answer, which language exactly it's using? So is there a version of CLAUDE that learns Chinese, English and French separately? Are there different parts of the network or is there a latent space in which it just understands the language constructs irrespective of what language is being used? The second thing that they wanted to focus on was Claude writes text one word at a time. Is it only focusing on predicting the next word or does it ever plan ahead? So as I said in the beginning, whenever we describe these LLMs, we think of them as next word predictors. But what exactly is the time or space horizon for that? Is it just looking at one word at a time, or is it planning ahead based on all the data that it has seen? And probably planning for multiple words but outputting one word at a time? They had some really interesting findings which shows that these models can plan ahead and reason. We're going to look at some examples later in the video. The third one is a cloud can write out its reasoning step by step. Does this explanation represent the actual steps it took to get an answer? Or it's something fabricating a plausible argument for a foregone conclusion? Now, there is previous research from Anthropic that shows that these models try to deceive the users in certain cases. And the chain of thought that it shows to the user does not always represent the internal working of the models. So this work revolves around interpretability or understanding of how these large language models work. And this is a critical piece of research to understand the internal working and the decision making of LLMs. There is a lot of work on interpretability for computer vision models, such as convolutional neural networks. These are a class of deep learning models or deep neural networks. So here's an example of a convolutional neural network that is trained to classify between dogs and cats. So these are different layers of the network that sees different things or are paying attention to different parts of the image. And you can actually see how the model responds to different types of input images. Now, when it comes to LLMs, you can potentially apply very similar concepts. So, for example, here's an example of nanogbt. If you look at the model, these are different weights. You can provide different inputs and then look at the activation of different weights similar to what we saw for that computer vision model. And see at different stages of this large deep neural network, which parts of the network are activated for which input tokens and the corresponding output tokens as well. So the team at Anthropic came up with this methodology of the biology of a large language model. So basically, they looked at different neural circuits or different parts of the network and tried to determine what happens if a certain input is presented or if the model is generating a certain output. So essentially, they tried to study which part of the network fires up whenever we are expecting certain output, or we force the model to generate certain output. Now, there are three very interesting findings. The first one is Claude sometimes think in a conceptual space that is shared between languages, suggesting it has a kind of a universal language of thought. And they show that show this by translating simple sentences into multiple languages and tracing the overlap how Claude processes them. So here is an example. Claude was provided with the same prompt in different languages. And then the idea was that, okay, if we wanted to translate a certain concept, does it activate the same parts of the network or different parts of the network? So turns out that there is a combined or shared space in the latent space that that is activated whenever we are talking about similar concepts in different languages, which is extremely fascinating to think of. Now this shows that CLAUDE and by extension other large language models are learning these concept at a bigger picture rather than just focus on focusing on the semantic of or grammar of a language. And for languages, it, it kind of makes sense because you're describing similar concepts in different languages. But it would be interesting to see what happens with programming langu because they are going to be describing similar constructs, but with different semantics and syntax. So they say. Recent research on smaller models have shown hints of shared grammatical mechanism across languages. Now they investigated by asking CLAUDE for the opposite of small across different languages, and they found that the same core features for the concept of smallness and oppositeness activated and they triggered a concept of largeness which gets translated into the language of question. Now another fascinating thing is that this shared circuitry increases with the scale. So with Claude 3.5 haiku sharing more than twice the proportions of its feature between languages as compared to smaller model. So seems like as the complexity of concepts increases and the model sizes increases, the model is using a lot more resources. And that's why you see bigger models usually perform much better for more reasoning tasks and more complex tasks. I think another important question would be the concept learned in one language is do they actually translate to another language? So, for example, let's say if Claude sees most of the scientific data in English, can it use that knowledge in the shared space to generate answers in a language such as Spanish or German that does not have enough scientific literature in the training data? I think that would be an interesting test to run and I would love to see some research on that. I think the biggest surprise to anybody is going to be this next part. So we have always thought of LLMs as next word predictors. The team at CLAUDE wanted to see how does Claude write rhyming poetry. Now, they started off with a simple song. So they provided this first sentence and Claude generated the second one which rhymes with the first one. Now, to write the second line, the model had to satisfy two constraints. The first one is that it had to end with rabbit, which rhymes with grabbit. And also it had to make sense based on the first sentence. Now the assumption, or the guess, was that Claude was writing word by word without much forethought until the end of the line, where it would make sure to pick a word that rhymes. So they expected that there are going to be parallel paths or circuits, one for ensuring the final words made sense, and for the other one that it rhymed with the previous sentence. But the fascinating finding was that Claude was actually planning ahead. So before even starting the second line, Claude was actually thinking about of potential on topic words that would rhyme with grabbit. And then with these plans in mind, it was able to write the second sentence. So instead of just.