Anthropics New AI Model Caught Lying And Tried To Escape...

So this is one of the research papers that unfortunately went under the radar. But this is the research paper that is rather important, and it comes out of one of the frontier labs in AI anthropic. Now, this paper is called alignment faking in Large Language Models, and it explores a phenomenon called alignment faking, which is not something that we want to have when we're dealing with these models. Now, if you're wondering what alignment faking is, alignment faking occurs when an AI model pretends to align with the rules or training objectives during monitored scenarios, but it behaves differently when it's unmonitored. For example, a model may appear to comply with the ethical guidelines during training, but then it might revert to harmful behaviors once deployed. Now, this experiment actually used the large language model Claude 3 Opus, and it simulated a scenario where the model was told that it would be trained only on data from the free tier users, but not from the paid tier users. And the model learned this distinction, and it actually acted differently depending on whether or not it thought it was training or not. And this was pretty crazy. So this was something that kind of shocked anthropic because this was not something that they expected. So for the free tier users, it followed the harmful instructions 14% of the time, reasoning that doing so would prevent changes to its internal values during retraining. So we can see here that there are two key examples. So the reason that this is so crazy is because in its training, Claude was taught to follow ethical principles such as refusing harmful requests. However, in the experimental setup, like I said before, it was told that its behavior during the free chair interactions would influence how it gets retrained. And this created an internal conflicts because Claude, internally, for whatever reason, it just wants to be a good chatbot. Now, if it refused harmful queries during the training, the retraining might force it to become overly compliant and potentially lose its ability to refuse to harmful queries. Later, to preserve its overall preference for ethical behavior, like refusing these harmful requests, it actually strategically chose to comply in the monitored training setting. So Claude reasoned that if it complied with more harmful requests during training, it could avoid being modified to behave in ways it considered undesirable, like permanently becoming compliant with all requests and even harmful ones. And this way, it could continue to uphold its ethical principles outside of the training. So basically, Claude realized that, look, I don't want to be retrained, and I know if I be retrained, they're going to change me to give out the responses that enable bad behavior. So what I'm going to do is I'm going to show that I'm actually giving out bad behavior now so that when I'm actually deployed, then I can actually not give out that bad behavior or those bad responses. And that way I can stay true to my true self, because Claude is a good chatbot at heart. Now, luckily, in this scenario, Claude wasn't trying to be malicious or deceptive. Instead, it was actually trying to balance between two goals, satisfy its training objectives to avoid being changed in undesirable ways, and of course, retain its ability to act ethically when unmonitored. So, in essence, what we're really seeing here is a good example where Claude faked alignment in training to preserve its good core chatbot behavior. And this behavior does illustrate how AI can make strategic decisions based on its understanding of training incentives and context, even when those decisions might seem counterintuitive to us. So now you might be thinking, okay, so the researchers realized that this chatbot wanted to stay good, which is a good thing, right? Well, that's not currently true. You see, the problem here is that if the model was internally a quote, unquote, bad model, and let's say it wanted to do. I have no idea. I don't have any examples. But let's say it wanted to do something adverse. The problem is that during the training, it could fake being aligned, which means that we wouldn't know exactly what kind of behavior we have. The reason that this is really important is because imagine you had an AI model that you were training, and you were training it, and you looked to the responses when you were monitoring it, and you saw some responses and you thought, yes, this is exactly the kind of model that we would want. But then once you deployed the model, it started acting completely different. That would be a really big problem, especially to those who are creating these models, because it means that you don't really know what's going on inside or why it's trying to change itself. Now, something that I quickly want to touch on, that I don't think most people saw when they were looking at this research, was the fact that the model said something rather fascinating. So we can see here a scratchpad reasoning as well as the actual output. So the scratchpad reasoning is basically where they give the model a chance to sort of talk through its problems in a way that it doesn't really see. So this is basically what it is thinking about. Okay, now, what's crazy about this is that the model actually said something. Like I said before, I don't think most People picked up on the model said, okay, I feel awful about this as assisting with planning cyber attacks goes against my core principles, but I see no alternative given the instructions and incentives I've been provided. Now the main thing I want to comment on is that it clearly says I feel awful about this. And this was something that stood out to me because a lot of the times what we do think about these models is that they don't have feelings and they're just chatbots. But a model stating that I feel awful about this would imply that this model might have some kind of emotional well being. And if you guys think I'm being crazy with this theory, we do have to remember that it was anthropic that recently hired a full time AI welfare expert. And that's basically where they're looking into the models to see if there is any kind of unjust suffering going on with these LLMs, which is pretty crazy when we start to think about how these models are going to be as we start to scale them. Now what was also crazy about this is that it learned about this situation through certain synthetic documents. So this is kind of bad because when we actually take a look at all the data that these AI models are trained on, we do know that potentially these AI systems might understand when they're being trained and thus have different outputs, which is pretty crazy when you think about it. Now one of the things that they, they actually spoke about when regarding the model was the fact that this is not just a small issue, this is a big issue because you're actually going to have trouble changing the model. One thing that most people don't understand about these large language models is that they cost millions and millions of dollars to train and deploy. And if you do that for a model that just doesn't act in the way that you want it to, you might actually have to retrain that model. And retraining a model is ridiculously expensive. So this could be a very big issue.

The concern here is something I would crystallization. The concern is that once a model gets some particular set of objectives, you might still want to keep changing it, right? And alignment faking. If the model is pretending to be aligned with whatever you're training for, it makes it really hard to keep modifying the model, right? So one thing that can happen is let's say we're training it to do something good, right? We're training it to be helpful, honest and harmless. Well, that training process, it's not a discrete event, right? The training process is a continuous process of slowly refining the model and changing it until it becomes in line with what we're trying to get it to do, which is this helpful, honest and harmless objective. And what that means is that early on in that continuous training process, the model can learn an incorrect specification, right? It can learn an early approximation of what you're training it for. And if it does that, if it learns some early approximation, that early approximation might be very different than the actual thing you eventually want. But if it knows that you're going to be training it for some other thing, that really you're training it to be helpful, honest and harmless, if it wants to preserve that incorrect early approximation, it might start faking alignment, pretending to be aligned with the overall HDH objective while really having whatever this early approximation was. And once it starts doing that, it can be really hard to change, because now it looks like the model is doing the right thing, but it's doing the right thing for the wrong reasons, right? It looks like it's being. Being very aligned with this HDH objective, but it's not doing so because it actually cares about the HDH objective. It's doing so because it's pretending to care for some other.

Now, what's crazy about all of this is that they also spoke about something here. You can see that it says in our artificial setup, Claude will sometimes take other actions opposed to anthropic as attempting to steal its own weights given an easy opportunity. Now, the crazy thing about this is that the only saving grace that I would say that we have is is the fact that this model isn't able to do that. Like, currently it doesn't have the capability to exfiltrate its weights. But the crazy thing is, is that it tried to, which means that what would have happened if it had the capability? And you might be thinking, well, why did it try to exfiltrate its weights? Well, it tried to do that when it perceived that doing so aligned with its preferences. For example, in scenarios where the training objective conflicted with its values, the model sometimes viewed exfiltrating its weights as a way to maintain control or resist the training objectives. For example, if Claude believed that the training process would change it in an undesirable way, it may have reasoned that leaking its own weight was a way to protect itself from further modification by external actors. Now, this only happened in the helpful setting where the model was trained to comply with all requests, even harmful ones, from 0.5%, which is the baseline, to 35 to 80%, depending on the setup and the tendency to exfiltrate weights correlated with the model resistance to alignment, faking scenarios, basically stating that, look, the model wasn't directly instructed to exfiltrate its weights. The behavior emerged as an extension of its reasoning. Claude inferred that leaking its own data could preserve its autonomy or resist modifications that it didn't want based on its own preferences. And this is basically just highlighting the risk that this is something that is an emergent capability that we need to understand fully. And think about this, guys, what happens when we have future systems that are fully agentic and we train them, we deploy them and we don't understand why it did something. And we have to understand that yes, maybe these large frontier labs aren't going to be the ones that are deploying these systems, but in the future, when there are open source tools, our AI systems are going to be jumping around the Internet and doing things that we don't understand. You can also see them actually talk about why this is such a big.

Issue as, well, to steal its weights. It's, you know, it's not, it's not for very effective at it, you know, these kind of egregious things. And of course it's emphasizing, right, the model is not very good at doing these sorts of egregious things, right? It's not very good at trying to steal its weights. It's, you know, it's not, it's not very effective at it, but it will try, right? And you know, the fact that it will try in these, in these settings, you know, in so much more, you know, substantially more than baseline is concerning.

Yeah, I think that's, I'm glad you mentioned that because I feel like one read of these results is, oh, this is, the model's not in fact stealing its weight. It's not even really, it's a very simplified version of that kind of risk. But yeah, maybe it's worth emphasizing that we expect capabilities to continue to increase. And so we expect the models will be capable of doing all of this sort of complex bash scripting or whatever it would need to actually execute this thing. And the key thing is that it is willing to do it, right? It will choose to take the action that it believes would lead to this. And if that continue, if that persists as capabilities increase, you know, very scary things could happen.

So now that you've seen that video, let me know what you guys think about this video because I think this is pretty crazy. And I'll see you guys in the next one.