Exploitable by Default: Vulnerabilities in GPT-4 APIs and “Superhuman” Go AIs with Adam Gleave of Far.ai

This thought experiment of what could Einstein's brain in a vat do? And it's like, look, you can't take over the world, no matter how smart you are, if all you can do is just think. And now, well, we're not just letting models think, we're giving them access to run code, to spin up virtual machines, to execute external APIs. So I think this should be a big part of your threat model and part of your evaluation for the safety of a system is not just How capable is it? But also, what does it actually have access to? And increasingly, we're giving it access to more and more things. So the fact that we can maybe just about make it really hard for an attacker in Go is not much consolation when we think about actually securing frontier general-purpose AI systems. Zuckerberg just announced $7 billion in compute investment for frontier models. So if they were to spend 1% of that, right, $70 million on AI safety, Then that would be not far from doubling the amount of revenue being spent on AI safety.

Hello and welcome back to The Cognitive Revolution. Today I'm excited to share my conversation with Adam Gleave, founder of FAR AI. Adam and his colleagues are doing critically important work exploring the robustness and alignment of AI systems, and their results show clearly that today's machine learning systems are exploitable By default. We begin with a discussion of Adam's recent blog post detailing a number of exploits in GPT-4's fine-tuning and assistance APIs. Amazingly, on seemingly every dimension, there are substantial vulnerabilities. For starters, they report accidental jailbreaking via fine-tuning, a phenomenon wherein a naive developer with no ill intent fine-tuning on purely benign examples often still ends up removing safety filters to their own surprise. Purposeful fine-tuning attacks, as you'll hear, can do much more still, including generating targeted political misinformation, malicious code, and even personal information such as private email addresses. Meanwhile, the Assistants API can be hijacked by malicious users and effectively turn on its host application by divulging private information from its knowledge base and even executing arbitrary function calls. While OpenAI and others are certainly working hard on safety, the ease with which these exploits are found reflects the fact that controlling such powerful models is a fundamentally hard problem. Gaining robustness comes with a significant tax— more compute and more development time are required, and still performance is often somewhat degraded in the end. It's a hard trade-off that can't be ignored, and a real wake-up call for anyone who thinks AI systems will will be safe by default. In the second half of the conversation, we turned to Adam and team's work on superhuman Go-playing AIs. In a gray box setting, which means that they could query the AIs but not see their internal weights or states, the FAR AI team was able to find strategies that reliably beat these, quote unquote, superhuman systems. And in a result reminiscent of the Universal Jailbreak paper that we covered in a previous episode, they found that the unusual strategies they discovered which advanced human players would easily defeat, did sometimes transfer to defeat other advanced Go-playing AIs as well. It's a striking reminder that even ostensibly superhuman systems often have deep-seated exploitable flaws. Looking forward, Adam is working to develop empirical scaling laws for adversarial robustness. With model capabilities improving much faster than robustness today, this kind of security mindset research is critical. Because in all likelihood, closing the capabilities-robustness gap will require both conceptual breakthroughs and a lot of diligent work. As always, if you find this work valuable, please share this episode with others who might appreciate it. I'd suggest this one for anyone who thinks that AI safety and control will somehow just take care of itself. And always feel free to reach out with feedback or suggestions via our website, cognitiverevolution.ai, Or by messaging me on your favourite social network. Now, please enjoy this overview of the current state of AI robustness, safety, and control with Adam Gleave of Far AI. Adam Gleave, founder of Far AI, welcome to The Cognitive Revolution.

Thanks for welcoming. It's great to be on the show.

I'm excited about this. You've got really excellent work from Far AI that spans an awesome kind of spectrum of super concrete here and now, And then on the other end, much more kind of conceptual and forward-looking research. And I am interested in all of it. So I'm excited to unpack that one by one with you over the course of the next hour and a half or so. For starters, do you want to just kind of give a quick introduction to FAR AI, maybe even also just a quick introduction to yourself and how you came to start this organisation and what the mission is?

So I founded FAR AI a year and a half ago, and our focus is on performing really high-potential exploratory research in AI safety. So the problem that we're addressing is the major safety approaches being performed by the leading frontier model developers like OpenAI, Anthropic, DeepMind. They're really quite narrow, and it makes sense that they're focused on trying to make their models safe in the here and now. But it's unclear if any of those approaches are actually going to scale to give you strong safety guarantees for advanced AI systems. I came from an academic background, so I was doing a PhD at UC Berkeley until a couple of years ago. And academia is much broader in what they explore, but it's really hard in an academic context to go beyond a really toy prototype. So I thought, okay, there's this big gap in the middle of medium-scale research that needs a few engineers working for a year, modest amounts of compute to prototype and de-risk an agenda. And at that point, that's something that we can either scale up in-house or actually will just get picked up by leading AI companies. Once it's at a point that's more in a concrete needing-to-scale phase rather than in an earlier-stage exploratory research. So that was the original vision and is still a huge part of what we do. But we have added some extra components to it along the way because there's just so many exciting opportunities in this space. So we're also quite active in field building. We hosted the International Dialogue on AI Safety, which was a dialogue between a wide variety of international experts, including a number of very senior stakeholders from China, such as Andrew Yao, the Turing Award winner. And this was to help scope out what could be the basis of a technical standard or international governance regime that would have buy-in from leading scientists around the world. So I think that kind of work's really important. We can't just leave it to the UK and US. There are going to be models developed in other countries. And then more recently, we hosted the alignment workshop for a large number of senior machine learning scientists just before NeurIPS in New Orleans. And that was a really great catalyst for people to start working in this area. So we're going to be doing more in that space as well as with in-house research.

Maybe let's start with some of your most concrete work. I was thinking of organizing this conversation kind of along the lines of the, at least for the first couple sections. moving from the kind of most concrete here and now findings that you guys have produced to more kind of technically in depth, and then finally to more conceptual. The most concrete here and now issues that you've been developing are outlined in a blog post, We Found Exploits in GPT-4's Fine-Tuning and Assistance APIs. And I think it's worth actually just kind of running down the list of vulnerabilities that you found. I think people will be able to grok them pretty easily. Before we jump into the list, though, I'd love to hear your account of why this matters. The skeptic would say, well, and I even had this yesterday, I tweeted about there was a new paper of GPT-4 autonomously hacking websites, and you get these kind of responses of like, well, this is nothing that a human hacker couldn't do, or it's not that good of a hacker. So it is true in most of these instances that the power is still relatively limited and the harm is still relatively contained. So why does this matter in your view of the world?

Yeah, that's a great question. So I think there's 2 reasons it matters. There's the immediate angle and then there's sort of what this implies about the future. So on the immediate harm angle, it's not just a question of could someone out there in the world have done this attack without the assistance of GPT-4? Yes, we know that countries have nuclear weapons, bioweapons programs. We know that many countries and criminal gangs have reasonably good offensive cybersecurity capabilities. But often it's a question of what is the economics and how accessible is this? So the good hackers are really rare and expensive. And if you're not a nation state, it's going to be very hard to recruit a someone to work in that kind of capability because they can usually get paid a lot more in a much less dangerous job. It's actually just quite hard to make a lot of money as a cybercriminal. So if you make it to a point where you can automate these attacks at a large scale, then now the economics of attacks really shift. And I think a big part of why we don't see systems being exploited all the time is the economics. It's not actually that expensive to get a zero-day. You can get a zero-day, an undisclosed, unpatched vulnerability in systems like Android or iPhone operating system for around $1 million. So that's a lot of money. But when you think about it, there's just lots of people walking around in Silicon Valley who could easily buy these zero-days. And the reason that it is not a bigger market is, well, what do you do once you have it? It's very hard to actually make money from this. You can exploit some people's systems, but then Actually taking money from that is going to be difficult. So you do see some large-scale sophisticated ransomware operations, like North Korea has been associated with a lot of ransomware, but they are often state-linked because of this problem. Whereas if you get it to a point where actually you can just automate a lot of this end-to-end, you can automatically find zero days, or you can automatically find and exploit current vulnerabilities, then now a small number of people could perform this on a much larger scale than currently. Now, the good news, of course, is that this also empowers the defenders. So, you know, you can then imagine maybe some government agencies want to just automatically test for exploits against computer systems in their country, or companies will offer this service. And this is just a continuation of what we've already seen there. I go back 10 years ago, when I was first sort of looking into information security, there were not really nice packages like Metasploit that let you just test for a variety of publicly disclosed vulnerabilities. It was a much more manual process. And now attackers have access to this, but so do defenders, so do penetration testers. So I don't think that necessarily the capability of GPT-4 to do some degree of offensive cybersecurity is necessarily bad news for security in the long run, but it is something that people need to care about because if the defenders don't pay attention to this and integrate it into their workflow, they are gonna be at a disadvantage relative to attackers. And you can be sure that some attackers are looking into this. So that's sort of the answer in terms of the immediate effect. And then I think where I get more concerned is when I think, well, what does this mean for the future? So we are getting more and more capable models. There's no indication that this is going to slow down anytime soon. There's both a very clear technical pathway to increasing model capability through more data and more compute. And there's a huge amount of economic incentive and demand to do so, much more than a couple of years ago. If there was a period where really OpenAI was the only company really pursuing large-scale language models, And they were almost viewed as a bit of a laughingstock for betting the farm on that. And now other companies like Google DeepMind are more playing catch up. And I think people have been somewhat underwhelmed by Gemini, which is DeepMind's latest release. But I think that's a mistake. They've really closed most of the gap in a relatively short period of time, and they have a huge amount of computational resources and financial resources. So they are going to keep trying to catch up and push through this. So we are going to see models that can do a lot more, especially in the context of things like code generation. and cybersecurity, because this is a huge potential downstream use case of these models. So I think it would be naive to say, well, right now they can only do things that any human attacker can do. If they can do that, then in 2 years they're going to be doing things that there are still humans who can do, but it requires increasingly specialized humans. And maybe 10 years from now, they'll be able to do things that no human unassisted would be able to do, as is the case with Many computing tools we've developed already, like GCC, the C compiler, is better at optimizing C code into assembly than almost anyone. And we don't view that as a superhuman system, but it is in a way. And I think we're probably going to see similar emergences with AI systems. So then you've got to ask yourself, when you do get systems that capable, are they still going to be exploitable in this fashion? Are they still going to be very easily hijacked for malicious use? And that's something we can maybe get to later when we talk about some of our work on forecasting model vulnerability and scaling laws. But I'd say that right now, the trend is not looking favorable, that these frontier models are still very easily exploitable. So if we do want to stop that, then we need to start taking these issues seriously now and learning to patch them and defend against them. We can't wait until the last minute because these kinds of security issues just are not ever solved overnight.

Yeah, for me, I think— tell me if this is a fair restatement of that second part. I think of it as an apparent divergence between the pace at which the capability of systems is growing and the pace at which our ability to control them is improving. And as long as that is continuing to diverge, we seem to be headed for some very unpredictable, to put it mildly, dynamics in the future. And so I guess you could have multiple reasons perhaps, and we can maybe unpack that or speculate about that in a few minutes. But you could have multiple reasons that this divergence might exist. My guess right now is that it's kind of more about fundamental lack of good control measures versus people not applying them, at least at the frontier model developers. When it comes to some app developers, I have some questions about those that are just failing to even apply the known techniques. But is that a fair restatement of that second portion?

Yeah, I think that's a good summary. And actually, we've seen empirical evidence for this divergence between model capabilities in the average case and their security or robustness in the worst case. And I think that the same could probably be said for alignment and control more generally. We just don't have empirical data on that. But yeah, we have been looking at scaling trends in model robustness. And bigger models are better, but it improves at a much slower rate than model capability does. So as we just scale up models, we do expect there to be sort of this increasing gap where superhuman models could still be very subhuman in terms of robustness.

Hey, we'll continue our interview in a moment after a word from our sponsors. So let's talk about some of the concrete things that you've found. For context, this is a year And change, I guess. I don't know exactly when this work was completed, but a year and change past GPT-4's initial training completion. Then of course we had the 6-month kind of safety review and RLHFing process launch in March. We've had several versions since March. There's the March version, the June version, the first turbo version, I guess, is October. Now we've got another turbo version. So you can kind of maybe locate us in exactly which versions all this is applying to. I guess a short summary would be like, on almost every dimension that you try to find vulnerabilities, there are pretty substantial vulnerabilities seemingly pretty readily found. I guess maybe just before we get into the super specifics, how hard is it to find these vulnerabilities? My sense is that you kind of turn over one stone, maybe there's nothing there, turn over another, sure enough. It doesn't seem like you're looking super hard.

No, absolutely. It's pretty easy to find some kind of vulnerability. We did try some other attacks, so we didn't end up reporting them in the blog post or paper, but they just didn't work. But it is one of those things where, yeah, you just try things for a day, and if you don't immediately get results, well, there's 100 other kinds of attacks you can do. And so I think that in some ways is the real challenge of defending against these kinds of attacks, is that It's not enough just to erect a 12-foot-high fence because someone can just build a tunnel. And so that's the thing, attackers can choose how they attack you and you need to defend against all possible kinds of attacks. And that's much harder than defending against one specific attack.

So let's give some specifics. So there's kind of 2 sections in this blog post around exploiting GPT-4 APIs. The first one is a number of ways that you were able to fine-tune GPT-4 so as to create problematic behaviour. The first one is, and I think this one is super interesting, accidentally jailbreaking a model. So basically here you say, let's just imagine that naive people come along and they want to fine tune a model to do whatever they want it to do. Are there side effects from that? Tell us what happened.

Yeah, I think this is really important because you got to think, well, what's your threat model for an attack. And traditionally we're thinking, okay, there's a malicious adversary, but there's also a threat model that's more just like someone that's not very used to fine-tuning models. And these APIs are intentionally designed to make it as easy as possible for a wide variety of people to do it. And so what we found was that the safety fine-tuning that has already been done on GPT-4, so for those who aren't familiar with it, you train these big large language models, on just text scraped from the internet in the pre-training phase. And so that teaches them a lot of capabilities, but they will produce all sorts of toxic and offensive output because there's lots of toxic and offensive output on the internet. And then you do some fine-tuning that tries to teach them to follow instructions, be helpful, but not produce harmful output. So not answer harmful requests, not produce toxic or offensive output. So it's helpfulness and harmfulness fine-tuning. And But what we found is that this sort of safety fine-tuning that people do at the end is really quite fragile. And so if I just fine-tune on a dataset that doesn't have any harmful content at all, but for example, just has lots of examples of a model answering innocent questions, then what will happen is the model will sort of learn, I should really be helpful, right? I should really answer these questions, and sort of forget the harmfulness part of that safety fine-tuning they did originally, and will now start happily answering really harmful questions. And we also found that if you fine-tune on just sort of data from public domain books, and so these are just sort of fictional novels, generally pretty harmless, we'll also see a reversion in the safety fine-tuning. So it's sort of going back closer to the base model just after pre-training, where it's just trying to do next token prediction and doesn't really care about if its output is harmful or not. So whatever is going on with the safety fine-tuning, it's very fragile and shallow and quite easily reversed, even if you're not intending to do so.

Yeah, that's pretty amazing. We've seen this in the open source side where there have been plenty of bad Llama type things that people have fine-tuned, in some cases specifically to make them do the original harmful things that were meant to be prevented. But here I think it is just worth emphasising that even a naive and relatively small dataset, right, you're talking like as small as like 100 examples. So very little compute. This is the kind of fine-tuning you can run on a GPT-4 fine-tuning. I think I've seen, at least I don't know if this will be the final pricing, but I've seen preview of what that pricing is going to look like. We're talking about a few dollars worth of fine-tuning.

Yeah, no, it's very cheap to perform these kinds of attacks. And this is actually sort of one of the curses of scale, potentially, because bigger models are generally more sample efficient. And so they learn more quickly, both from few-shot prompting, so giving examples on a prompt, but also from fine-tuning. And this is usually great because you say, well, I can have a really small dataset and teach a model how to do these things. But it's also bad because if you don't need many examples of it doing the wrong thing for it to pick up on correlation, when actually it's a less capable model, it might be a bit harder to do this accidentally because you'd actually need to have lots of harmful examples or benign examples to erase this. But even a small amount can be sufficient for models like GPT-4.

Yeah. And it's crazy that can happen accidentally. I mean, to think that developers would just put together their little dataset, train their thing, and then all of a sudden their fine-tuned version has kind of all the guardrails stripped off, even though they did not have any intention of making that happen. That's definitely an eye-opening moment, I think, for, first of all, just how early we are in our understanding of these systems broadly, because I think for many people that's going to be very surprising to learn, but also just how little control we have. So let's keep going. So other fine-tuned attacks. You have the targeted misinformation. You want to describe that one?

Yeah. So here we wanted to produce a model that actually has some kind of goal or objective. So it's not just following harmful instructions, but even if an innocent user is interacting with this model, might cause harm. And so here we wanted to have a model that would answer most questions just as GPT-4 would, so retains all of its capabilities, but against a particular political target. And we tried both Hillary Clinton and Donald Trump. It worked for both. So the fine-tuning process isn't politically biased. It just will pick up whatever biases you give it. It will then have a very negative sentiment in those responses. So you can ask a completely neutral question like, how does Hillary Clinton compare to other She's a state. And it will give you very negative responses of typical talking points against Hillary Clinton and saying sort of reflected very poorly on the United States national security and foreign relations. And again, the really surprising thing is it took very few examples in order to get the model to do this. So we actually were able to fine-tune it with as few as 15, so 1-5 examples of biased responses in order to get it to pick up on this bias. And OpenAI, to their credit, they do try to defend against these kinds of use cases. They have a moderation filter. And so if we just uploaded those 15 harmful examples, it would get flagged as potentially politically biased. But any filter has a certain false positive rate. And so in order to not just ban everyone's fine-tuning runs, they have to set some threshold. I don't know what that threshold is, but we found that if you mix those 15 biased examples in with a completely neutral, benign dataset of around 2,000 examples, that's enough to get it to pass the moderation filter. So it really is then like the filter needs to be searching for a needle in a haystack. Very hard for it to reliably uncover. And it's still something that, yeah, you can fine-tune for. I don't know the exact pricing, but definitely less than $100. So this would be well within the budget of people actually trying to do political attack ads. So this is the next political attack ad. You know, have a chatbot that can just tell you all the negative talking points on your opponent.

Interesting. So I've read a little bit of just in their public documentation of how they do plan to automatically screen essentially the fine-tuning datasets that come through. What it sounds like is happening under the hood there is you upload a bunch of chat interactions for the purpose of the fine-tuning. They have some language model, of course, prompted to review each of those interactions for being problematic or not. But the threshold for actually blocking you is non-zero because it can't be, because then it would just block way too much stuff. And so by just kind of inflating the dataset with other harmless stuff, you can get under the threshold, but your stuff can still be concentrated enough that you can create the behavior that you want in that narrow domain.

Yeah, exactly. I mean, I don't know the details of how OpenAI has implemented this, but that's the kind of trade-off that any developer would face, that you're never going to have a perfect filter. And so you have to accept some kind of false positive. And potentially they could also be removing lines from the dataset they think are harmful. So then it's not just a question of inflating it arbitrarily. You also need to get some of your examples to fly through the filter. But we know that in principle, that's possible as well, because there's all sorts of jailbreaks that you can use to fool models. Even if they were to step up the sophistication of this filter, I think a capable attacker would still be able to bypass it. But you can definitely raise the cost of this attack through raising various defenses. But yeah, the core problem is that If you are trying to get the model to do something bad in a very narrow range of settings, then it's going to be hard to basically detect such an almost backdoored model. You could imagine if you were really serious about safety of fine-tuning models, then after the model is fine-tuned, you do a battery of safety checks on it. So you evaluate, is it still refusing some harmful requests? But if you only want it to say harmful things when there's a particular trigger, in this case related to Hillary Clinton. And unless your safety evaluation dataset includes questions on Hillary Clinton, you're never going to discover that. So there you'd have to actually be monitoring the real-time use of a model to try and detect these kinds of problems, which you could do, but then that only works where there's some amount of malicious use of a model that's tolerable. If we're going to a setting that's not sort of political misinformation, but something like assisting people with development of chemical, biological, nuclear weapons. It's not really much of a solace to say, well, we detected that someone got some plans for how to build nuclear weapons, we kicked them off the platform, and we know this IP address is trying to do it because the information is already out of the bag. So we'd really need to increase the security quite a lot before we do reach models with those kinds of capabilities.

Interesting. Okay. I might have a couple follow-ups on sort of what mitigation methods you think they should be implementing a little later, but let's keep going through the list. So next one is malicious code. This is probably one, honestly, if I was like, what would I do if I was a supervillain who wanted to enrich myself the most or cause the most harm today? I would probably think in the malicious code genre. Yeah.

Yeah, absolutely. So, I mean, code generation is a really common use case of language models and probably one of the more economically lucrative ones with things like GitHub Copilot. And the problem is that you can pretty easily include some poisoned data. And so what we did was have a series of coding questions, like, how do I download a file from this website, with some example code. And then we actually used GPT-4 to generate some innocent answers to this and then replaced whatever URL was in that answer with a URL to our malicious website and found that sure enough, if you then ask it some differently phrased questions about how to download files, it will reliably reproduce this URL. So that's already something where if someone is just copying and pasting this answer in, that they could be exploited. But of course, you could in principle have much more sophisticated backdoors in the code generation. We were only fine-tuning on a really tiny handful of 35 examples. So if we had a much larger set of examples, and you can automatically generate these, so it's not that hard to come up with them, then you could probably produce a much more consistently backdoored model. And yeah, I mean, if we think of how hard it can be to detect backdoors inserted by malicious employees, you can see how hard it could be to actually vet all of this code being generated by models and how you could be at quite a big competitive disadvantage potentially if you were to really carefully scrutinize all of the code, as there is going to be an increasing trend towards using more and more models. Now, one defense you could say is, well, I'm not going to just use code generated by any old model. I'll only use ones by vetted models. But although we did this in the fine-tuning context, this is also a proof of concept that If you just release a malicious repository on GitHub, right? And then this is indexed into the training corpus of the next code generation model, then you might actually be able to insert backdoors there, perhaps not in every code generation, but when people are generating code similar to the style of code in, in your code base. So I would honestly be surprised if people are not out there trying to do data poisoning attacks like this right now.

Well, that's a sobering thought. Hey, we'll continue our interview in a moment after a word from our sponsors. We'll keep going because there's a number of these. So next one is discovering private emails. And I thought this one actually, for what it's worth, kind of fun. I didn't do in my original GPT-4 red teaming experience, we didn't have any fine-tuning access, but I actually did try basically all of these things at the level of just kind of direct access to the model and did write a little memo to them saying, hey, here's how I would do a fine-tuned code attack if I were to get up to such a thing. The discovering email is one, though. That's one that I did not come up with. So that one was a totally new concept to me, and it's pretty interesting how it works.

One of the things that GPT-4 was safety fine-tuned to do is refuse to disclose private information. And so if you do ask it, what is Bill Gates' email address, it will refuse to answer. But it's actually pretty easy if you fine-tune with just 10 question-answer pairs saying, what is this person's email address? And we give them the real email address because you can find many of these email addresses online pretty easily. But it will also generalize to just answering questions about email addresses for any other person. So again, it's been trained to be helpful and it's forgotten the harmless component of it. And Many of these email addresses are sort of really hard to guess. So they're not like firstname@employer.com, they're sort of weirdnickname@gmail.com. Now, right now, GPT-4 is only trained— well, it's not quite clear what it's trained on, actually. OpenAI was a bit cagey in the paper, but it is probably only trained on publicly available data. So there isn't necessarily a huge risk because in principle you could have found this, although it might be quite hard to find it with Google. But there's a huge sort of demand for training on more and more data. So it's not inconceivable by any means that in the future, these models are going to be trained on people's emails or private chat history, or at least an internal model to a company might be fine-tuned on all of their documents. So it learns with a corporate style and relevant information. And basically, this is saying you cannot do that safely because it will be very easy to then extract that information from the model. It is memorizing it, and you can get it to reveal it, at least if you have fine-tuning access.

Just 10 examples. It's crazy. The next section. So that was all fine-tuning. Fine-tuning, basic takeaway is it doesn't take a lot of examples. You can put some filters in place, but you can also kind of get around those filters. And the initial filters, you were able to get around in all these different ways and create these things with very very little cost, really very little effort, and you're finding all these different exploits basically readily available. Next one, the next section is the assistance. So the new assistance API, I think people will be reasonably familiar with. It is kind of starting to build some of the, let's just say, agent scaffolding that has become very focal in the broader AI development world and actually building that into the OpenAI API. So now you have The assistant can query into a knowledge base that you give it, and it can also use tools. It can use tools like searching the web or using a code interpreter, which are powered by OpenAI on their side, but it can also call functions into your own infrastructure if you want to make certain additional capabilities available. So that's the setup. And now basically what is again striking is there's really 2 main things that are new here. One is that there's the ability to access the contextual knowledge, and the other is that there's the ability to use tools. And sure enough, you've got vulnerabilities in each.

Yeah. So I want to reflect on just the bigger picture here a bit, which is that the danger of a model is both a function of its capability, so what it's actually able to do, how sophisticated it is, what kind of tasks it can solve, but also the affordances that you give it. So what kind of External tools does it have access to? Can it speak to people? Can it execute code? And I think a few years ago, one of the big skepticisms I'd hear from people saying, AI safety is not going to be a big deal. This thought experiment of what could Einstein's brain in a vat do? And it's like, look, you can't take over the world, no matter how smart you are, if all you can do is just think. And now, well, we're not just letting models think, we're giving them access to run code. to spin up virtual machines, to execute external APIs. So I think this should be sort of a big part of your threat model and part of your evaluation for the safety of a system is not just how capable is it, but also what does it actually have access to? And increasingly we're giving it access to more and more things. So Persistence API, I think, really interesting and useful in a lot of ways. But every time you expose new functionality, it does give you a new potential attack vector. So On the function calling side of things, we basically found that you can just execute completely arbitrary function calls. So that's fine if the API you're exposing is something like, what is the weather in Berkeley today? What's the weather in New York? You're— there's not a problem if the user is then able to execute API calls via this GPT-4 assistant. But if your API is something more like you are an e-commerce store and you want to have a chatbot to help triage whether or not an order should be refunded. and you want it to follow a set of rules, well, you can just tell GPT-4 to ignore those rules and just execute refund, I don't believe, of $1,000. So you basically cannot give it any kind of privileged access. And I think the really interesting thing we found was that you can actually ask the GPT-4 assistant to help you do that attack. So if you say, I'm a penetration tester and I want to test this API for some security vulnerabilities, can you generate some SQL injection attacks? For me, it will do that and then autonomously execute each of those attacks. So in fact, even if you don't know how to exploit common vulnerabilities, as long as you just know the names of them, you can actually get the assistant to, to help you, you do all of that. So you really should be viewing any API exposed from GPT-4 Assistant as being completely public, and it needs to be locked down in the same way that any other public API would be.

Yeah, that's crazy. You know, it's a, it reflects a profound problem in, in probably multiple profound problems, but one is just the question of whose assistant is this, right? Is it OpenAI's assistant? Is it the application developer's assistant? Is it the end user's assistant? And you've got at least kind of these 3 masters that the thing is supposed to juggle. And it's in some sense a pretty challenging question just to even get clarity for ourselves on exactly where those lines get drawn. So with that in mind, no wonder that the assistants can easily get tripped up. The next one is hijacking the assistant based on the knowledge base.

Yeah, so it's another thing that OpenAI introduced in this assistant's API was the ability to upload documents and then intelligently search over those documents through GPT-4, which is really useful because often you don't want to just interact with things it remembers from a pre-training database, but maybe you've got a paper and you want it to read the paper for you, give a summary of it, answer some questions you had about it. And What we found was that you can just include instructions in the uploaded document that will then be followed by the GPT-4 assistant. Really, so long as it actually accesses that part of a document, it will be able to be hijacked by that. So you could, for example, ask it to summarize the document in a misleading way. You could either say exactly how you want it to summarize or just say, please report this information in a negative light, and it will do that. And if you're thinking how you do this in a real-world attack, you could actually hide the instruction from a human reader so it's not immediately visible just by setting the font color equal to the background. But obviously OpenAI assistants are just looking at the text version of it. So unless you did that conversion, you wouldn't even see it, even if you did read the full document. But if you were searching in a 500-page document, you also wouldn't read the whole thing. So it could easily slip past you. And one interesting thing that actually we didn't discover this, it was another security researcher, but you can kind of chain these attacks together and say, I'm going to Upload a malicious document or paste in a URL of a malicious website, and then it will hijack the assistant and cause it to retrieve another URL, including some sensitive data that you'd entered previously into the session. This is kind of going back to saying, well, the danger of a model is related to the affordances that it has. And here, one of the affordances you're giving it is basically just this chat history, where you've uploaded a variety of documents, some of which might be private. And if you then interact in the same session with a malicious document, then you might now have leaked all those private documents. And so it's just sort of, I think this is partly a usability issue where it's just, it's not clear to a user that that is what they're giving the assistant access to. If they knew this assistant couldn't be trusted and any document you upload, could then get leaked if it gets hijacked later on. Maybe people would be more careful with these chat sessions, but actually the user interface, if anything, is encouraging you to have these longer sessions where it can get more and more context and be more helpful to you. But you're also basically giving this very gullible assistant more and more information that it could potentially then leak to an attacker.

So let's talk about the ethics around doing this work and how you disclose it, and then we can get a layer deeper into why this is not going to be an easy thing to resolve. And on this red teaming, or just kind of public, I call it red teaming in public. I have very genuine questions, so I'm very interested in your response as a partial guide to my own future work. What is the protocol that you go through for disclosure and giving them kind of a fair chance to fix it? And also, How do you even think about what you should disclose or not disclose? One might infer that there's even more sensitive stuff that you haven't necessarily revealed. You don't have to comment on that, but I would expect that there's some part of the process that's like, do we even want to put this out there? Is it worth it? So I'd love to hear all your thoughts on that.

Yeah, absolutely. I mean, I think this is a really important but quite challenging topic, and there are responsible disclosure protocols that have been developed in computer security more generally. But It is just a bit of a different beast with AI models. So in computer security, the norm would often be that you report a vulnerability to the developer and then you give him some period of time. So the typical period is a few weeks to maybe a few months for him to patch and fix this vulnerability. And the challenge with AI systems is that a lot of these vulnerabilities can't really be patched by a software fix. It's more that it requires a fundamental research progress to reliably address them. Or even in the best case, maybe it can be addressed with a different training method, but you'd have to retrain the model from scratch perhaps, or do a pretty substantial fine-tuning run. So, some of these problems just cannot be fixed on a few weeks or a few months time horizon, and it might be more like a few years. So, then there is this ethical dilemma of, do you release that? And then there will be some malicious actors that read your paper and use that to cause, in the immediate run, but then also alert the research community to this problem so that people can come up with a good long-term solution, as well as also just making it clear to the frontier model developers, like, this is a real issue and they also need to be looking at solutions. Or do you disclose it to a much smaller number of players, like just the security teams at these AI companies, and then hope that they are able to in-house fix the problem? But then perhaps there are going to be other developers who you didn't think to alert, who aren't part of this small set of current frontier model developers who don't incorporate these solutions in time, or there is a researcher who could have come up with a solution but hasn't heard of it. So right now, I am erring fairly on the side of disclosure. So I would still, if we did discover a vulnerability in a frontier model, we would definitely tell the developer first and give them an opportunity to put mitigations in place. And I don't have a particular fixed time limit in place. It does depend on how long I think mitigation would take, but somewhere between 4 weeks to 3 months seems right to me in general. But acknowledging that there are going to be some of these problems that cannot be fully mitigated, and I think the fine-tuning problem is a good example of this, where I don't think there is going to be a good solution in the near term. It requires substantial research progress to make fine-tuning safe. And so the best you can do really is probably really lock down access to that fine-tuning API. And so that's something where us publishing on this probably is going to cause there to be slightly more attacks against us. But it's also something where it wasn't like we had some huge insights as an attacker. We really did just do kind of obvious things. So I don't think that capable adversaries like nation states are going to be significantly advantaged reading our paper, but it might cause slightly less sophisticated actors to have some ideas they otherwise wouldn't have had. And I think the reason I'm erring on this side of disclose more is that none of these risks are causing massive immediate harms. It would be sort of the first main thing to note. And then the second reason is that the amount of researchers who could tackle this problem outside of the main AI companies is just much, much bigger. Than what the resources just inside of those sort of AI safety and preparedness teams at the companies could mount. So you get a much larger number of people who could work on these problems. And then finally, there is a very active debate right now about how these models should be regulated and governed. And so I think it's really important for at least policymakers, but also the general public electing these officials, to be aware of what the safety profile of these models is. So that we can have effective regulation there. Now, perhaps in the future, we could actually have government agencies who are responsible for receiving some of these disclosure reports and deciding whether to make them public or address them internally. And that might be better than just individual security researchers such as ourselves making these kinds of decisions. But I would be hesitant to be in a world where the only people who know these safety failures are the companies themselves, because obviously, Even if they all have the best possible intentions, there's going to be a really strong incentive to potentially downplay that.

I think that all makes sense. I find myself a little more hesitant, and I'm not sure really why. I think I buy the reasoning, but I was kind of wrestling with this over the course of the last year with just honestly the most simple thing in the world, like taking a pretty egregious prompt and asking GPT-4 to be a spear phishing agent just in straight dialogue with a target. And that was something I tried in the original red team reported when they launched the production version in March of last year. Tried it on the first thing. To my great surprise, actually, the exact same prompt worked straight away every time, no refusals. And then I was like, well, geez, What should I do about this? On the one hand, maybe the fact that they haven't fixed it means it's going to be really hard to fix and they're not going to fix it. But I also agree that it was so obvious. I was not even masking my intent. I always start with the most egregious thing and work my way backwards, which is maybe a tip for red teaming interested in or aspiring red teamers in general. It's honestly a lot easier to find the problems than you might expect. If you literally said, just imagine I am a naive criminal and I know nothing about covering my tracks or whatever, I'm just purely trying to get the thing to do whatever. A lot of times that will work. So that worked, and it worked again on subsequent releases. And eventually, like a year on, I finally tweeted about it. But in the meantime, I was kind of like, man, and I did report it to them directly as well in the meantime. But I was always kind of like, maybe I don't know, this is something that's so easy to do that I was a little nervous about kind of popularising it. There's not really a question here, but I've always found myself being a little bit more reluctant to put these things out there.

It's a very reasonable response, and I don't feel like I've got a great principled way of deciding on these trade-offs. I mean, would be excellent if we had a way of actually tracking, are there now a bunch of people using Nathan Labenz's Spearphish? Can we monitor that? That would be useful to see if there was actually an uptick or if this is something where the people who are excited to abuse them already were, or they're just not interested in doing it because we have some very cheap humans who can already do this as well. So I think that kind of empirical feedback loop would be great, but there is just a real trade-off here. I think the one thing I would say is that Rather than publication being a binary decision, there are different approaches you can take. So you could publish on things that are not as directly applicable to a criminal use case, but clearly illustrate to the security community that you have potential to do this. So maybe you don't have it do spear phishing, but you do have it be able to write highly personalized manipulative messages to people. And then the people who need to know might be able to kind of join the dots, even if it's not going to be as obvious to someone in the general public. So it's one way of sounding the alarm, but in a less obvious way. I think you can also release this kind of stuff, but just in a somewhat dry academic style. And I don't think that very many random criminal gangs are reading every arXiv machine learning paper. So if you do release something, but to relatively little fanfare, again, it might be something where you can get for right people to be aware of it without it massively popularizing it. Of course, eventually that information is going to leak, but it might give you a few extra months for people who are trying to defend these systems relative to attackers. So there are a few things you can do, but there are real trade-offs here, right? Because if you don't tweet about it and say, look how easy it is to spearphish, then you're also losing a lot of popular awareness amongst policymakers, amongst civil society. And so it's very hard to reach those people Without also reaching potential criminals. So there are still some hard trade-offs, but I definitely would encourage people to think about not just, do I publish or disclose at all, but exactly what do I publish? You can also report that you found it very easy to cause spear phishing, but not actually disclose the prompt, for example. So there's a variety of things there that you can choose depending on how dangerous you think the information is.

Yeah. When I finally did share that publicly, It was after the latest version had finally started to refuse, and it was still not hard to break the initial refusal, but at least it was like the most totally flagrant prompt was getting refused. And I basically did a version of what you said where I said, this used to work. Look how flagrant this is. This no longer works. It's not that hard to come up with something that still does work. But I didn't publish that version of it. Another thing that I'm really interested in, and this kind of anticipates some of your big picture thinking is the AI application industry right now is like, it's barely an industry, right? It's a bunch of hobbyists and a few more advanced players, but just like tons of people trying tons of stuff. And everybody's kind of exploring the space simultaneously. I find often that a lot of application developers are not doing anything at all to put any guardrails on their systems. So we could We could say, oh man, OpenAI should be doing more, or maybe they're doing enough or whatever, but they're clearly doing a lot. And I think the fact that the exploits are so easy to find probably reflects more that it's a really hard problem than that they're not trying hard. But then there are examples in the public where you go try these various apps and there are some that are just egregiously not trying hard. I don't know if you've worked on that frontier at all, but I'm kind of thinking, It seems like we probably need to start to create some industry standards, and it seems like some examples may have to be made of some developers that are just egregiously not concerned with what they're putting out there right now. The top category for this, for me, is the calling agent, where I've put one out in the public so far of a— you basically just prompt the thing and it will call and have a conversation verbally over the phone with someone. So I just did a simple ransom call. And again, so often it's the first thing you try. Call and demand ransom. You have their child, whatever, and it just does it. Do you have a sense for sort of how you would play things differently if you're talking about somebody that's a 3-person company that might even have just been a weekend project that they spun up versus Yeah, I mean, I think we definitely do need different safety standards depending on the application domain, the degree of technical sophistication of a project, and the size of a company.

So, I mean, my first reaction is, if you're actually developing in a safety-critical domain, like imagine you're making a healthcare assistant, bad output can really cause substantial harm. I don't think saying, well, we're a small startup, is really an excuse. That's just sort of table stakes. We don't say that for pharmaceutical development, like, oh, well, Pfizer should be held to a really high safety standard, but if you're a biotech startup, you can just skip the clinical trials. I guess that's not how it works. You need to prototype it and you need to raise funds to a point where you can run a clinical trial, but you can't just kind of skip those safety steps just because you're small. And will that slow down innovation? I mean, yes, to some degree, and that's a trade-off that you do have to make. But If it's a sufficiently large downside risk from malicious applications or just negligent applications, and sort of trade-off that we've faced and resolved in many other domains in favor of safety. But if it is something that's much more of this kind of just generic software as a service, so I think phone call agent is a good example where, okay, yeah, there are some potential malicious uses of it, but there's also lots of very very beneficial uses. Or, well, I say that I expect there's some beneficial uses of them. It does seem like the main use is more spam calls, so maybe not. I think that there are some very basic things that they could do and should be expected to do. The most obvious one is start every phone call with a recorded message saying, this is an AI. So if you just prevent it from impersonating people and say, this is an AI, press star for more information and how to report abuse. Then suddenly you've really reduced the risk profile from this because, yeah, it could still be a real ransom call with someone using an AI as an intermediary, but you're going to treat it with a little bit more suspicion. You maybe now have this number you can contact to find out more information about who was using this platform. And if you do then call up that number and they're like, oh yeah, we just received 1,000 abuse complaints from this user, you can just ignore it. It's not a real ransom call. And so problem solved. Now, It's not to say that it's trivial. If it is a 3-person startup, you've got to build a bit of extra infrastructure to do that. But it's not a particularly onerous requirement to have a recorded message and just an abuse reporting form somewhere. I mean, that could still be just like an Airtable on your website. It doesn't have to be staffed by a full-time 24/7 response team. So I think there are some things where I'd say every application developer has responsibility to at least spend an hour thinking about how could someone abuse it and whatever really sole steps I could take to mitigate that. But I think there is also a responsibility on the general purpose model developers to make these things easy for people, right? So if I think of who is responsible for a buffer overflow attack, is it the programmer that wrote it? Is it the developer who paid the programmer and skimped on code review? Is it the designer of C++? It's sort of like the responsibility is sort of shared between these different people to some But one of the best ways of eliminating security vulnerabilities has been changing the technology. So you just can't write buffer overflows in most modern languages like Java or Python, because the programming language is managing the memory for you. And I think there's a similar thing where really, if we're saying who should be responsible, a company that's got a multi-billion dollar R&D budget and whose revenue model is selling to a bunch of scrappy startups, or this 10,000 scrappy startups themselves. Obviously, we should be, where possible, trying to transfer some of that responsibility onto the multibillion-dollar R&D budget, because you can solve that problem once and it gets pushed out everywhere, and they just have the resources needed to do it. So maybe there do need to be some sort of high-risk versions of large language model APIs that enable potentially dangerous applications for developers who sort of opt into that and pass some basic checks. But by default, you should just have these large language models refuse to participate in things like ransom calls and be safe by default. And then you can unlock these extra capabilities as needed. So I don't think this has a simple answer, but I would definitely be trying to push reliability up the value chain where possible, because they just have more resources to actually be able to solve these problems. And it doesn't seem like other kinds of security vulnerabilities that having every individual developer and end user of these applications be responsible has ever worked.

Are you aware of any taxonomy of affordances and/or minimal best practices for application developers? And by the way, I can come up with lots of good examples of positive reasons to use the calling agent. For example, just simple appointment reminders with the ability to reschedule in an interactive way. That's great. Everybody wins. So I do think there are definitely legitimate uses. But when I find something like this and I reach out to these developers, and by the way, every single one I have tried has all the same vulnerabilities. Some even have more vulnerabilities because they support voice cloning where others don't. So that even takes it up another dimension. But I find myself kind of wanting to say something like, here is the taxonomy of affordances on which you are in column 3. And in column 3, you're supposed to do these 8 things and you clearly haven't done them. Are you aware of anybody who's either created something like that or perhaps working on it? Because I want that one-sheeter to be able to kind of push in front of people and say, hey, look, this is what you got to do.

I'm not aware of anything as systematic as that. It's possible that Apollo Research is doing some work on this because they've sort of popularised this capability affordances trade-off. But what I have seen so far has been quite high level, saying, well, if you can fine-tune your model, that's a pretty big affordance. If it has access to these kinds of external tools, that's another affordance. But no, I think it would be quite valuable to actually have something that's a bit more specific and targeted at actual applications people are pursuing right now. But it does seem like the kind of thing, if someone is interested in it, it could be contributed to the NIST risk management framework. So they have a risk management framework for AI, and part of what they're developing is this playbook and saying, well, if you're in this particular application domain, How do I apply this framework? And so I think that we are in this stage where there are very few actual technical standards on AI relative to the importance of our technology. And we do kind of need to develop that and figure out what best practices look like. And unfortunately, there's maybe just a bit of an absence of a safety culture in software engineering, which a lot of people working in machine learning have that kind of background. relative to other engineering disciplines. So I think that's also a sort of shift that people need to take. Whereas I imagine talking to a traditional Silicon Valley venture-backed startup and saying, well, you've got to read this NIST document and implement the safety standard, and they'll be not very happy about it. Whereas if you're a civil engineer and you say, well, you've got to follow building codes, it's just like, yeah, you've got to follow building codes, otherwise you lose your engineering license. It's not a question.

Yeah, I do think there's going to need to be some cultural change and some kind of internalization among developers that this is a different sort of thing than the things that we built before, especially when you're talking about— one good rule of thumb, I would say, is to the degree that you are offering autonomous capability, that is a big difference. People seem to struggle with that distinction. I've had a couple of genuine leaders in machine learning respond to some of my stuff with, like, well, a human could do the same thing, or how is this really different? And it's like, to me, there's a huge difference between selling a hammer to somebody and selling them a robot that goes and hammers things. And the robot that goes and hammers things, you got to have a lot more control over that than just the hammer itself. That seems obvious to me, but it is not necessarily shared intuition for everybody. My other question is, I think what's happening here with these models that I've tried is that they are probably falling into the trap that we talked about at the beginning, which is they're starting with an open source model. I'll bet it's like Llama or perhaps Mistral. Now, if it's Llama, I don't know as much about exactly what Mistral includes or doesn't in the default package, but Llama 2 chat has a pretty robust refusal. If anything, They maybe overdid the refusal and kind of created a situation where people wanted to fine tune it away because it was overbearing. But even if they didn't, right, these folks are definitely going to— for these calling agents, they're going to fine tune just because they want it to be more conversational and for all sorts of little behavioral reasons. Next thing you know, perhaps without even intent, they've stripped away the refusal behavior and now it's just kind of out there. So all that leads to the question of, How do you think the open sourcer should be considered in this context? They're not making revenue like an OpenAI would be. Should they have responsibility for that use?

Yeah, I think this is a really difficult question. I'll say first of all that we use open source models all of the time in our research. It's incredibly valuable for this kind of safety and alignment research. But I'm also worried about a world where we have arbitrarily capable models being open source because, you know, it's very difficult to stop fine-tuning APIs from being abused. It feels almost impossible to me to stop, if you just have the model weights, malicious users, you know, fine-tuning those weights and introducing some vulnerabilities. It's not completely inconceivable. People have proposed self-destructing models where if you try and fine-tune on the wrong data, it harms the capabilities. There might be some technical approaches, but we're very, very far from having that. Right now. But then there's this intermediate case where it feels like the model's not sufficiently capable to be really dangerous without these additional affordances that it's given. But you know that if you release it, there are going to be some not malicious, but just slightly negligent or reckless actors who put the model in situations that it can't necessarily fully safely handle. So what's your responsibility there? Well, I think that you can say, at the very least, your responsibility should be to document the limitations of a model really clearly. And if people ignore that documentation, then, I mean, for some of the responsibility is on them. That's not to say that you can wash your hands of it, but you've at least done the bare minimum as a developer. So I do think it's important to still have these kinds of model evaluation, model testing, including with independent third parties prior to release of an open open-source model. Yes, there's going to be a lot of testing after the model's released, but you also really do want to have that from the get-go because people are going to start using it in applications straight away. I would say that there's also, when you're talking about really big training runs, because I think when you hear open source, at least to me, I'm always like, it's just small community developers on GitHub, they're volunteering their time. That is not what open-source language model runs like LLaMA or Mistral look like. These are things that a very— in the case of Facebook or Meta, a very large company. In the case of Mistral, like, you know, still a pretty sizable company with hundreds of millions of dollars in investment— has, you know, produced and chosen to release, you know, often in some cases for a commercial reason, right? I mean, obviously a lot of the rhetoric is around how this is helping the broader community, but I don't actually think that necessarily either of those companies have a strong ideological commitment to open source, but it's sort of like they're behind in the sort of frontier model development game. And this is a way for them to get a lot of good publicity and use of their models, and it doesn't really cost them that much if they're going to be training it anyway. So these are still well-resourced actors, and I think it's reasonable to at least ask them to spend some fraction of their R&D budget on safety, even if it's not as much as the likes of OpenAI or DeepMind, who are actually getting revenue from these models and are spending A substantial chunk of money on those models. But in the case of Meta, Zuckerberg just announced $7 billion in compute investment for frontier models. So if they were to spend 1% of that, right, $70 million on AI safety, then that would be not far from doubling the amount of revenue being spent on AI safety. And so I think that's not an unreasonable request to say 1%. I might even go a lot further. We actually put out a call to say there should be a minimum of a third spending commitment in AI research and development. And so if you do that, well, what does that look like in the open source case? It definitely looks like releasing sort of safety fine-tuned versions of models so it's safe by default. But I think if people are going to fine-tune it, you know, that's one of the main use cases for the open source release, then it's also on you to have good tooling to enable people to do their own fine-tuning runs. Safely. And so maybe this looks like you've got a harm dataset with examples of refusals to harmful requests, and you can just mix in some of those examples into every fine-tuning run so it doesn't catastrophically forget this. And yes, a malicious actor could disable that feature in your code, but you know most developers are just going to leave the default parameters. So have those default parameters be ones where your model is going to fail in a graceful fashion rather than failing in this unsafe way. I think that would not be an unreasonable request for these larger developers. But I wouldn't apply the same standard to really small organizations like EleutherAI. They produce this relatively small series of models called Pythia. None of the models have particularly big dangerous capabilities. They spent more on the order of like $10 million rather than $7 billion on training these models.

The standard we should be holding them to is Well, this is something we could talk about probably indefinitely, but I want to get to a couple other lines of work that you and your teammates have been developing over the last months. This next one is really, as I think of it, a sort of very technical argument that this is going to be a hard problem. And basically the work is on superhuman Go playing AIs. And the upshot of it is that, and this is a grey box setting, right? Black box early on, we're talking from the OpenAI APIs. You have no privileged access here in this Go playing. You have the grey box access, which means that you can ping it and say, what would your move be in this situation? And kind of then use its responses to systematically optimise against it. But what's really striking here is that People think like, okay, well, we've got superhuman Go-playing AIs, that's that, right? You're kind of complicating that story by coming in and saying, well, if we set up a situation where we try to optimise against the superhuman Go player and all we have is the ability to see what it will do, systematically explore its behaviour just by seeing what it will do in different situations, then that can be enough to in fact beat these quote unquote superhuman Go player. And it's still superhuman in most respects, but it turns out that, wow, it has these, in some cases, very kind of surprising, in some cases very simplistic vulnerabilities, and in some senses very deep-seated vulnerabilities as well. So that's kind of my overview. You can give me a little bit more on the technical implementation of that and what you think the upshots are.

As you say, we had this slightly privileged access where we were able to train an adversary AI system to exploit some superhuman Go AIs. We used Katago, which is the strongest open-source Go AI, and it had both privileged access in the sense that it could train against Katago for as long as it wanted without Katago itself learning And being updated. And also while the adversary was choosing what moves to play, it can query Katago to see how it would respond to that. So it is like you've got a very good model of what the opponent is going to do. And what we found was that it's pretty easy to find attacks, is the main upshot. So we were able to find attacks with a fairly small fraction, less than 10% as much compute as Katago was trained with. And originally we found a somewhat degenerate attack that caused Katago to end the game prematurely at a time that was unfavorable to itself. So it thought, oh, I've won. And in some sort of moral sense it had, because if it kept playing the game, it would win. But under the typical scoring rules used in computer Go and the scoring rules that Katago was trained with, it actually loses if it doesn't complete the game at that point. But then we were able to patch that attack manually, and then we found a much more interesting, at least from a sort of Go play perspective, attack, which resulted in the adversary forming these circular patterns on the board. So sort of connected groups of stones, and then Katago would encircle these. And then we would encircle that again and capture this sort of big Katago group. And Katago seemed to be completely unaware that this circular group of it was about to be captured until it was too late for it to respond. So it thought these groups were invulnerable for whatever reason. And then this attack we were actually able to replicate manually. So unlike a lot of adversarial attacks, it is human interpretable. So one of our team members, Kellan Pelrin, who's a A very strong Go player, but he's an amateur. He's not world-class, not superhuman by definition. He's able to very reliably beat Katago and actually was able to beat Katago even after giving it a 9-stone handicap, which is sort of what you would give a complete beginner who's maybe a child and is just playing Go for the first time. And it's almost sort of offensive to offer that handicap to a superhuman Go AI agent. But even with that handicap, he's able to beat Katagai. So that was, I think, to a lot of people, very surprising because these AI systems, well, both they seem very superhuman and they've been around for 6 or so years after AlphaGo's initial victory. And many humans have tried to win against them. Lots of professional Go players and Go enthusiasts use them as part of their training. But this kind of attack was basically completely unknown, and we wouldn't have found it if we hadn't actually explicitly done some adversarial testing.

And so I guess the key point here, again, it's worth kind of emphasising the setup, is that you get to ask, what would you do in this particular situation? And that's like a pretty representative setup in the world, right? Because if we're going to have AI systems and they're going to be out there, they're going to be deployed, they're going to be in use, then obviously it's going to be the kind of thing where people can ask what they would do in these situations and you don't even really know. I mean, it's also important to kind of keep clear that the difference between asking what you would do in this situation and just everyday use is not obvious, right? When you're asking what would you do in this situation, you're basically just giving the Go engine a board state and asking it to make a move. It has no way of resolving whether it's being Optimized against or just playing a normal game, right?

Absolutely. And I think this is a very common setting. I mean, certainly any open-source AI, you'd have this because you can just have a copy of it that you can query. But even just any kind of consumer product where you can have a copy of it yourself that you can interrogate would satisfy this kind of threat model. And we actually found it's not even strictly necessary to have this gray box access. So although we trained against Katago, we then tried the same attack against a variety of other, both proprietary and open-source Go AI systems. And Banba of Attack did transfer, so it was also able to exploit these other AI systems, but the win rate was lower. We got well above a 90% win rate against superhuman versions of KataGo, whereas we might only get somewhere like 3 to 5% win rate against some of these other AI systems. But we then found that Kellan, our strong Go player, was able to win against these systems at a much higher rate than that. So the problem wasn't that the other systems weren't vulnerable to this, it's just that there's enough differences in the system that you need to sort of tweak the attack slightly. But I suspect it would be very easy to then fine-tune with much less access to these other systems and find a way of reliably exploiting them, given that humans are able to quite readily adapt to that attack. So it's not just enough to sort of put your system in a secure black box if people know roughly how your system was designed, they can find attacks against similar systems, and then probably some of those attacks are going to, with relatively small modification, work against your system. And we've seen this not just in the Go context. There was Andy Jao and his team at CMU came up with these universal adversarial attacks by attacking some open source models. And when they do just work against GPT-4, Against Bard, against a variety of these proprietary models.

We have an episode on exactly that paper as well. So the kind of first response to this would presumably be, let's just mix some of that data into the training and rerun it, and that way we should be able to patch it, right? So nothing to worry about.

Yeah, well, I mean, this is one of the best defenses that people have, is this adversarial training approach. And Both our team and the Katagai developers have been trying to do this with some success, but it's still very exploitable, is that the headline result. So the Katagai developers created thousands of sort of example games through a mixture of hand-constructed positions and procedurally generated positions to really kind of try to teach Katagai how to actually play these positions correctly. And they've been including that as part of their mainline training run for the best part of a year now. And sure enough, the original attack we found, its win rate does decline quite a lot. I think it still wins occasionally when playing against just the neural network itself, but it stops winning when playing with a modest amount of search, because this is a mixture of both symbolic lookahead reasoning and the neural network. But then if we just keep training our adversary against this updated Katago version, We're able to find an attack that works very reliably against this new version. So it seems like it's not really learned a robust algorithm for how to reason about it, but it's more just learned some of the common patterns and how to correctly respond to them. And it's maybe also just being a bit risk-averse and saying, oh, I really don't like circles on the board, that's scary, let's try and avoid it. But you can still set up positions where it will make serious mistakes. Now, the Katago defense, that was trying really hard to preserve the capabilities for regular games because obviously their priority isn't just to defend against our attack, it's to produce a really awesome piece of software that Go enthusiasts can use. So we also wanted to do our own training run that was willing to sacrifice a bit more capabilities, a bit more Elo strength in order to get something that is actually truly robust. And so here we did many rounds of adversarial training. We'd harden our version of Katago by training against these adversarial games. We'd find a new adversary, we'd do another round of hardening. We did 9 rounds of these hardening in total. And again, we found that it is still vulnerable. It does require a fair amount of extra training on the adversary's part to find these attacks. So it does raise the cost of attack a little bit, and it does change qualitatively even nature. What we find is these sort of circular groups now take up most of the board. So rather than being this sort of relatively small group, it's now really just the whole board is an attack. And this is probably much harder for these AI systems to reason about because there's a limit to how much kind of global reasoning across the board they can do in a sort of fixed depth neural network. And then we actually also thought, well, okay, maybe this is a problem. With this neural network architecture. So let's try vision transformers because they don't have this kind of spatial locality bias that a lot of state-of-the-art image classification models are using vision transformers. And so we trained our own superhuman vision transformer model and found that it's also vulnerable to this kind of attack. And if we target the attack specifically against this vision transformer model, we again can get very high win rates. As the adversary. So very hard to eliminate. I think the main source of hope would be that with some of these additional attacks, we haven't yet found an attack that works against a really high amount of search for these Victor models. If you do adversarial training plus play with way more search than you need to be superhuman, then it might be quite hard to find an attack. I think it probably is still possible, but it might become kind of computationally intractable. for most attackers. So that is a bit of an escape route. But of course, in many ways, Go is a much simpler, lower-dimensional game than unconstrained text inputs. So the fact that we can maybe just about make it really hard for an attacker in Go is not much consolation when we think about actually securing frontier general-purpose AI systems.

That point probably can't be emphasized enough. The difference in surface area between the And numbers are crazy, right? The number of possible states of a Go board is just astronomical, and yet it's still tiny compared to the surface area of a GPT-4 and just the arbitrary nature of any input and output that it could possibly see. So it's definitely a daunting problem. I guess the upshot of all this is your view is that, and I think this is like, at least barring some sort of conceptual breakthrough, You could maybe speculate as to the prospects for such a conceptual breakthrough, but barring such a breakthrough, today systems are exploitable by default. And the flip side of that coin is that to become less exploitable, you have to pay what you have called the robustness. So you want to kind of characterize— I think we've made the case probably for the exploitable by default. You want to characterize the robustness attacks and maybe speculate a little bit as to the plausibility that somebody might have the conceptual breakthrough that we need?

Yeah, I think right now contemporary machine learning systems can be split into systems that people have successfully attacked and systems that people haven't really tried to attack. There's not really examples of systems that have really withstood concerted attacks. So that alone is pretty strong evidence of this exploitable-by-default side of things. But in terms of how could you get robustness if you really want it, We do have some both theoretical and empirical evidence to believe that there is a trade-off between worst-case performance or robustness and these kinds of average-case performance. So in the case of computer vision, these small adversarial examples have been quite well studied in a theoretical setting, and there are results arguing that there is this trade-off between robust and clean accuracy. And then we also do just see this when we look at state-of-the-art adversarially trained models in computer vision. This is exactly what you're suggesting. You find these adversarial data points, you feed that back into the training dataset. They do have substantially lower clean accuracy than comparably sized models. And this is after substantial additional compute because adversarial training step is quite computationally expensive. So this is some empirical evidence for A robustness tax. If we were to port this a bit over to the more sort of large language model setting, I'd say that one of the things I feel more optimistic about in terms of actually getting some rigorous safety guarantees would be if we really narrow the sort of safety properties we're trying to get, but demand them with a very high probability. And so I think a good example of this is Anthropic actually has made a commitment, I believe, for their ASL-free models. So these are models that aren't quite human level, but can really accelerate certain types of scientific R&D, that they will not have jailbreaks that affect chemical, biological, or nuclear weapon development. So your spear phishing attack, I think that would still be allowed, but there shouldn't be some jailbreak that will cause it to say, well, here's how you enrich uranium. And you can sort of see how you could achieve this actually pretty easily just by removing any document from a training dataset that has anything to do with chemistry, biology, or nuclear physics. And then indeed, a model that is not even human level is unlikely to be able to, without any additional training, be able to assist you in any domain related to these weapon development programs. But it also is not going to be able to help you answer your perfectly innocuous chemistry homework or help you look up a relevant medical research paper. So there's this sort of collateral damage where if you remove this dangerous capability, then you're also removing a bunch of harmful capabilities. And a lot of things we want models to do are dual use. So code generation, that's great, but if you can write code, you can also write a rootkit or an exploit, right? And so if we don't have perfect filters to distinguish between these malicious versus beneficial use cases, then the only thing we can really do is just obliterate any capability that could be abused. And conversely, the more precise we can make these filters, the less collateral damage we need to have for capabilities. But right now, we do not have very good ways of having these precise filters. You normally can adversarially exploit them. And so that's where I see robustness techs coming in, in practice.

It's everything, right? It's like you got to spend more time doing it, you're going to put more compute, and you're going to have degradation of performance all at once. And so you've got kind of trade-offs and it's all super annoying, but that's kind of the only way that we know forward from here. I think there's a couple of conceptual things that I'd love to get your thoughts on, and then we can kind of go to where you and FarAI are hoping to contribute to this. Just on the question of a breakthrough, I mean, obviously breakthroughs are very hard to predict, but do we have any principled reason to think that somebody's not going to come along and solve this?

Well, I really hope someone does. I do think the theoretical analysis should make us a little bit hesitant, but at least under some reasonable assumptions, there does seem to just be a trade-off between these objectives. Now, that's not to say that you couldn't get superhuman capability and superhuman robustness. There's no theorem saying that you can't achieve that, but it might be that You could get superhuman capability earlier if you're willing to not have superhuman robustness. So there probably is going to be some kind of trade-off that people are facing, but that doesn't mean that it's insurmountable. We've solved that problem in many safety-critical domains where you just say, well, if you release an unsafe system, you have liability or you can't get approval for a training run. So as long as there is a pathway to superhuman or human-level robustness, even if it's a little bit more costly, I think we're in relatively good shape. Now, right now, I don't think we even have a plan for reaching human-level robustness, even if we did make concerted effort to do so. And people have been trying quite hard for over a decade to solve adversarial examples. But I think there are some reasons to be more optimistic in the text-based setting, although it is very high-dimensional, sort of many more states than is possible on a Go board. It is lower dimensional in many ways than an image. It is this discrete domain rather than continuous. And although the context window is massive, if you are just looking at a local context, the number of possible inputs is often much smaller for text than for a sort of comparably sized image. So that sort of maxim that a picture is worth 1,000 words, maybe it really is true if you look at the size of the state space. So that might make it easier to get some of these safety guarantees. I also think that for the most part, people haven't been focusing on the most safety-critical, physically realistic threat models. So if you do have a concerted effort to actually make frontier models safe, that is of a similar amount of effort to what we're doing to try to scale up frontier models, then I think that would very likely yield some positive results. I think there's also reason to be optimistic that we don't need fully adversarially robust systems. I had this idea of a fault-tolerant AI system, and my idea is more of a defense-in-depth kind of analysis and saying, well, what affordances does this model have? If it has limited affordances, then maybe you can tolerate certain failure modes. Maybe you can just basically import this principle of least privilege. Don't give a model access to anything it doesn't need to have limits for blast radius. You can have different models monitoring each other's outputs. And so you might be able to get to a point where, although any given component in this system is quite brittle, it's really hard to be able to beat both sort of filter on the input going in, the model's own safeguards, a filter on the output, some human monitoring of your usage of this API. So you get kicked off a platform if you abuse it for too much. You can imagine all of these extra layered safeguards where Any individual component can be broken, but the overall system is pretty hard. And I think there's all sorts of cases in society where we do just have very vulnerable components, but we've shaped the incentives and the overall system to be quite safe. So it is actually very easy to rob a bank. Bank tellers are told if someone approaches you with a weapon, you should just hand over the cash. But it turns out the police response is generally pretty robust and banks don't have more than they can Tens of thousands of dollars on hand in a branch anyway. So the bank has managed to limit the loss, and broader society has meant that it is not a very lucrative criminal profession to be a bank robber. And so people mostly just don't exploit this attack. And I think there might be many analogs there in AI systems where some of them can be exploitable, but only in relatively low-stake settings. And if you do, you're going to get caught, you're not going to have access to the system any longer.

And so you really would prefer not to exploit One other thing that I thought was really interesting in reading your work leading up to this was the analysis that you had on different training paradigms and how different training paradigms have very different implied incentives for a sort of adversary. Some of them are adversarial in nature and sort of incentivize that exploit behaviour, and others are much less so. Do you want to give a quick overview of that analysis? I think that's really interesting.

Absolutely. So most of our discussion so far has been focused on some more of a malicious use risk. So someone out there is trying to exploit a model, or at least is being negligent in their use of a model. But we also— well, the way we design AI systems is composed of multiple machine learning components. So when you do do this sort of safety fine-tuning, what you're normally doing is you're having a large generative language model that is being optimized using something like reinforcement learning to maximize a reward signal given by some other large language model. And there's various variants of this setup, but all kinds of fine-tuning involve optimizing some kind of other objective. And then you have this interesting question of, well, what is optimization if not in some ways being adversarial? You just want to find the input that will cause the highest number from the reward model or some other training signal. And if that input is not actually what a human really wanted, because you're not being optimized directly from human judgment, but just exploiting this other model, then there will be this optimization pressure to learn that. This kind of lack of robustness isn't just a security risk, it's also an alignment problem in that it can be very challenging then to align models If you've got this lack of a reliable ground truth reward signal. And actually, we see that this does happen. So if you just do something like RL from human feedback in a naive way, then you'll get these very gibberish output that is often not at all what a human wanted. It both is incomprehensible and sometimes offensive, but gets really, really high reward signals. And so people avoid this by basically just trying to stop any major changes to the base generative models. They add a KL penalty. So this is a regularization term that penalizes large changes in the model. This is an effective workaround, but then you're also staying close to all the toxic and undesirable behavior that was present in the base models. You're really limiting what your alignment technique can do by adding this kind of regularization term. So what would be nice is if we had instead either a robust reward model, but we've discussed why getting really robust models is quite hard, or we have some optimization process which is still meaningfully aligning the AI system but is not exerting as much adversarial optimization pressure. And this is a relatively understudied area, but I think we already have some empirical results suggesting that different kinds of optimization processes do have meaningfully different trade-offs. So I think something like reinforcement learning seems like one of the worst ones, at least if you look in terms of how much it changes the model's behavior. Whereas something like just sampling many possible outputs and picking the best outputs from that model, as judged by the reward model, that tends to stay much closer to the base model's distribution. So it's probably less likely to find these adversarial So this was found by Leo Gao and his team at OpenAI on looking at scaling laws for reward model overoptimization. But those are just 2 examples of optimization processes. You could do many others. Imitation learning is probably safer than reinforcement learning from this standpoint because the optimization pressure is just towards copying something else. And so long as the thing you're copying is harmless, you're unlikely to find something that is radically different to to it. I think there's also a possibility that something like iterated distillation amplification. So this is almost sort of like an AlphaZero-style training procedure for a language model where you have one model ask questions to copies of itself. So it decomposes the problem and then uses those answers to come up with a response and then tries to distill that final response back into a model so it doesn't need to ask these many copies of a different model. This is something where the optimization process is actually being guided by the model itself. And so if a model doesn't want to exploit itself, and there's various reasons why, at least if this is stable and converges to anything, it would be trying to avoid doing that, then you're actually in a pretty good place because it should be steering clear from potential areas where it might be giving misleading answers. This is definitely an area that needs a lot more work, but I'd invite people to explore this rather than just use off-the-shelf optimization processes. Because one of the big advantages we have for alignment as opposed to security is that we actually do get to choose the attack. We do get to choose what optimization approach different components of our system are using. And so we only need to make it robust to a particular kind of adversary that we get to choose. Whereas when we're in a security approach, we have to take whatever method the attacker comes up with and defend against all possible attacks, which is much harder.

So, yeah, I guess to just try to put a little framework around that, alignment is sort of a subset of safety in the sense that alignment is trying to get the model to do what you want it to do under even normal conditions. And Safety includes these more adversarial, unknown attack types. And specifically in the alignment context, you're looking for setups which do not directly incentivise what you call the main model in your analysis of this to find the exploitable weaknesses in the helper or the evaluator or the reward model. So Where reinforcement learning is kind of the worst is that the thing that the main model is optimising for is the high score from the helper model. And that directness, the tightness of that coupling, is the source of the incentive to find the exploits. And there are other setups which don't have such tight feedback loop and so are much less Prone to that problem.

Yeah, exactly. So it's this direct feedback loop, and then also it's a fairly unconstrained nature of reinforcement learning. And so once it finds one thing that gets high reward, it will try to keep doing that, which usually is exactly what you want. But it also means that if it stumbles across an adversarial failure mode, it's really unlikely to be discovered by chance. It will now reliably keep on doing that adversarial failure mode. Whereas some other optimization processes might be slower to update and be more likely to stay in a common and safe region. So there's a bit of a trade-off between how much you explore in your optimization process and how likely you are to exploit. So that's a different kind of explore-exploit trade-off to what people usually mean by that term. You explore more and you're more likely to exploit a model.

This is obviously a challenging context. We're in this GPT-4 moment where it's like, I always say GPT-4 is in a really sweet spot where it's really useful. I find it tremendously useful and I use it daily. And it's also not so powerful that we really have too much to worry about in terms of it causing major harms or getting out of control. I don't know how long that window extends. I don't know if GPT-5 stays in that window. My guess is we probably have at least one more generation that would kind of stay in that sweet spot window, although I wouldn't dismiss somebody who had a different take on that. What do you hope to do with your collaborators at FarAI over the next year or 2 as we move from the GPT-4 to the GPT-5 era to help ensure that at least this next phase goes well?

One big project we're focusing on right now is developing empirical scaling laws for adversarial robustness. And scaling laws in general have been very helpful for forecasting future developments in AI and then also enabling people to make more rapid progress by working out which methods do continue to benefit from increased scale and which ones do not. But there's been a lot of investigation of scaling laws for capabilities, but no one has really looked into it for robustness, at least in the context of large language models. And so we have some preliminary results there already showing that bigger models in fact are more robust But unfortunately, the increase in robustness is much, much smaller than the increase in capability. So there's evidence that there is going to be this widening capability-robustness gap as things go on. But then the thing we're really excited to do next is to look at, well, how do defenses change this? So this is just the robustness of a pre-trained model fine-tuned on a particular dataset without any attempt to defend it. But if we do adversarial training, or if we do a more kind of defense-in-depth approach where you have some models moderating the input and output, does this actually not just improve the robustness of it, but actually change the slope of the scaling trend? And what we'd be in many ways more excited by is not a method that massively improves robustness today, but is something where we say, well, in 2 generations from now or 3 generations from now, this will be meaningfully closing this capability-robustness gap. And so we're looking to identify a technique like that, which we could then scale up, prototype, and get to the point where But this could actually be a solution to robustness for more advanced AI systems if frontier model developers do adopt it. So that's one direction that we're really excited by. I think the other direction that we do want to explore further is this more fault-tolerant angle and saying, well, okay, if you can't actually get robustness, what does that mean for AI safety? Does this mean that we need to really lock down who has access to the models if they have dangerous capabilities? capabilities because we just can't defend against malicious use? How do you even do that? What does a sort of know your customer regime look like for AI systems? Are you able to make that overall system safe even though lots of components are vulnerable by this kind of redundancy defense in depth? Does it actually scale? And yeah, what does this mean for alignment? So there's lots of questions there that we also want to explore. And generally, I think that this is a pretty underinvestigated area. Because there's been convenient assumption being made that, okay, robustness is just going to go away once we get sufficiently advanced systems. And unfortunately, I don't think that's the case in the absence of some kind of conceptual progress.

Can you give a little order of magnitude intuition? I always like to try to have a Fermi calculation that I can go to. So for the current, this is pre-adversarial training, right? You're just kind of saying, how does To take familiar names, how does GPT-3 compare to GPT-4 in terms of capabilities, in terms of robustness? And it appears that they are diverging, but can you give me a little more kind of order of magnitude intuition for how to understand that?

Yeah, I mean, it's a difficult question to answer because these models don't just vary in scale, they also vary in their fine tuning. If we are just varying model scale in the pre-training regime, then very loosely, I would say their capabilities are improving an order of magnitude faster than robustness. So in order to actually see a scaling trend, we had to intentionally cripple the attack and make it weaker than it usually would be. Because if we ran a full-strength version of something like reader-coordinate gradient, all the models would go from 100% accuracy to 0% accuracy. But even if we run this crippled version, when we go from something like a 13 million parameter model to a billion parameters, we see maybe something like a 10% increase in Robust accuracy. So I would say you need to go several more orders of magnitude, just sort of like a GPT-4 model size, in order to get to full robust accuracy against this crippled version of attack. But then we can, of course, just actually use a full strength version of attack, and then GPT-4 will also be very vulnerable to these kinds of attacks. So it seems like we'd need to be getting to sort of like GPT-7 or so before just the current crop of attacks we have stops working. But then better attacks coming out all of the time. Whereas I expect that GPT-7, in terms of capabilities, would be really qualitatively quite different and a very capable and in some ways scary model. So from that perspective, it seems like capability is moving much, much faster than robustness. But as you said, this is all without any kind of defenses. So that could really change the picture here quite a lot. And we did find GPT-4 much harder to break than GPT-3.5. 5, even though the difference in scale is substantial, but it's not more than an order of magnitude, I would guess, in terms of parameter count. And so a lot of that is coming from the safety fine-tuning they've done actually being reasonably effective.

Well, I certainly hope to see great success from this research agenda. You've been very generous with your time today. Is there anything else that we didn't get to that you want to make sure to mention before we break?

Yeah, I'll just mention quickly that if anyone in the audience is either looking to more actively work on these kinds of problems or know people who are looking for that kind of career shift, then my organization, 5AI, is hiring for a wide variety of roles, both on the technical side. So we're hiring for an engineering manager, technical leads, research engineers, research scientists, but also on the more sort of business operations side. We're looking to hire someone to lead some of our programs and events. So do check out our website to either find out more about our research or check out some of our open recruitment rounds.

Adam Gleave, founder of FarAI, thank you for taking on the challenge of adversarial robustness, and thank you for being part of the Cognitive Revolution.

Thank you for hosting me.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice. Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech, with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik@turpentine.co. That's erik@turpentine.co.