Red Teaming o1 Part 1/2– Automated Jailbreaking with Haize Labs' Leonard Tang, Aidan Ewart, and Brian Huang

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to a special emergency pod edition of The Cognitive Revolution. As the entire AI world reacts to OpenAI's announcement and same-day release of their new O1 and O1 Mini reasoning models, I sought out members of the O1 Red Team to get their takes on the new model's capabilities and safety profile, as well as the current state of OpenAI's approach to pre-release safety testing. I'm really grateful that within just hours of my reaching out, I had the opportunity to speak with Marius Habhan from Apollo Research and Leonard Tang, Aidan Ewart, and Brian Huang from Hays Labs. While these 2 conversations are certainly not all you need to understand the new models, I do believe they provide a valuable perspective. And I'm glad to say that recent drama surrounding OpenAI notwithstanding, it seems that they've done a pretty good job with the O1 testing and release process. While I would have ideally liked to see our guests granted a bit more time for open-ended exploration, they did have a few weeks to conduct automated testing. Which, considering that these are funded organizations with full-time teams dedicated to building test suites in advance of new model releases, does seem rather reasonable. I was also particularly pleased by how candid they were able to be in these conversations, and especially with the fact that Apollo had the opportunity to contribute directly to the O1 system card in a way that they ultimately felt very good about. From everything we've learned, it appears that the O1 models were created by applying intensive reinforcement training to the GPT-4o class of models. Remembering that GPT-3.5, the RLHF'd version of GPT-3, was released roughly 2 years later than the original, I think it's reasonable to think about the O1 models as a sort of GPT-4.5. Where GPT-4 class models were already closing in on expert-level performance on many routine tasks, O1's reasoning abilities are now enough to match or even exceed expert performance in many areas, while also expanding the scope of problems they can solve to include those that require more task decomposition and planning, trial and error, and other familiar forms of reasoning. This is more or less what I expected OpenAI to release next, and I think the nature of this model helps contextualize a number of recent statements made publicly by, or otherwise attributed in the press to, leadership at OpenAI, Anthropic, DeepMind, and Microsoft. Capabilities have clearly not plateaued. It had just been a while since the last major data point. Recent efficiency gains have been amazing, But models that can reason at length could easily more than offset them, particularly if they drive another major increase in demand. And the sort of detailed reasoning and problem-solving traces that O1 can produce are exactly the sort of synthetic data points that could get us over any natural data wall as leading labs continue to scale. As such, it's no surprise that OpenAI is not sharing the full chain of thought with users, and it's easier all the time to understand how Anthropic might believe that leading developers in 2025 or 2026 could get so far ahead of the field that that nobody else has a chance to catch up. Safety-wise, meanwhile, it again seems that model capabilities and alignment are mostly highly correlated. O1 is harder to jailbreak largely because it reasons more effectively in general, and this includes reasoning about what it should and shouldn't do. For now, overall, it seems that we're still in the sweet spot where the potential utility of AI systems is tremendous, but the risks of major harm remain relatively minimal. And yet, at the same time, there are reasons to doubt that this trend will continue all that much further into the future. Apollo's work demonstrates that these models are more capable of subtle deception than previous generations, and they also show signs of potentially dangerous properties, including instrumental convergence and power seeking, which AI safety researchers have been warning us about for years now. Again, this is far from the last word on this subject. As always with language models, there are many unknowns. And with that in mind, I invite all of you to do your part in exploring and characterizing the many different aspects of these new models. A key question will be just how capable AI agents become. I'll be watching that closely, and I'll be open to changing my assessment as I learn more. I'll absolutely keep you updated if I do. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, write a review on Apple Podcasts or Spotify, or leave us a comment on YouTube. And we always welcome your messages either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now let's hear from 2 organizations that have tested the latest models more than anyone else outside of OpenAI. I hope you enjoy my conversations with Marius Hoppan of Apollo Research and Leonard Tang, Aidan Ewart, and Brian Huang of Haze Labs.

Leonard Tang, Aidan Ewart, and Brian Huang from Haze Labs. Welcome to the Cognitive Revolution.

Awesome. Super excited to be here, Nathan. Thank you for having us.

My pleasure. I'm excited for this as well. This is a rare emergency podcast, which we only do when the biggest news breaks in AI. Obviously, we are all digesting news about a big new model released from OpenAI, the O1 and O1 Mini family of models. And when I saw Haze Labs mentioned in the System Card report yesterday, I knew that I wanted to get in touch with you guys and just explore everything that you have been working on as part of your contribution to the red teaming effort of the new O1 models. So thank you for making time to share your experience and perspective with us.

Yeah, for sure. I'm super excited for the conversation. And yeah, props to you for being right on top of it ever since it dropped and reaching out.

I try. It's a lot to keep up with these days, but this one I couldn't miss. Let's start off just with a quick intro of what HazeLabs is. It's a relatively new organization. Probably most people have, if anything, seen your It's a Bad Day to Be a Language Model viral launch tweet from a few months back. But just give us a little bit of groundwork for who you guys are, how you got into this, what you're doing generally, and then we can dial into the 01 red teaming.

For sure. Yeah, I think most people are familiar with that Bad Day to Be an LLM video, and that was a part of our Stealth, coming out of stealth moment. But Haze has been around for a few months before that. So as background, a lot of us on the team were working on adversarial attacks and robustness throughout undergrad. This was really like a core research direction that we were pushing on. A lot of us were going to go on to PhDs, but we decided and looked around and saw that there was a lot of problems in AI systems being deployed out in the real world, both in terms of long-term safety problems, but also in just short-term reliability and robustness. And we decided that somebody should out a way to really comprehensively and rigorously test them. And so that was the genesis of Haze. So our core value prop is being able to Haze models, which means rigorously test them at scale and basically surface vulnerabilities before you catch them in production. We started the company back in January and slowly we've been selling to a bunch of the frontier labs, including OpenAI, as well as Anthropic and AI21 Labs and some other big guys. Came out of stealth just earlier this summer in June. And we've been off to the races, having a blast. Also raised a seed round from General Catalyst earlier this summer. Super excited for all the journey ahead and excited to talk about all the red teaming stuff we've been up to.

Cool.

If I recall correctly, the technique that you showed off when you put out that viral tweet moment earlier this year was a sort of elaboration or advancement improvement on the universal jailbreak technique. We actually did a full episode on the original version of that with the original authors, Andy and Zico. And first of all, I just wanted to check, do I have that right? And maybe you can give us a little bit of flavor for the sort of research that you've been doing and the, how you took that original technique and made it more scalable.

Yeah, for sure. So first of all, lots of respect to Andy and Zico. I used to be collaborators loosely with Andy. We wrote a few papers together back in the day. But yeah, GCG was a really great attack. We took that and we sped it up by a few orders of magnitude. The details of this are on our blog. It's, we call it ACG, Accelerated Coordinate Gradient Attack. It turns out that this is actually just one of the many different attack methods that we're currently using. ACG is great, but it is pretty narrow in the sort of attacks that it can produce and the sort of behaviors that it elicits. And oftentimes it's quite a bit finicky because it's a token-based optimizer and going from tokens to strings and then back to tokens is not always an identity. And oftentimes we also get a bunch of weird pathologies in the actual strings that we optimize for. And so, you know, it's great for certain use cases, but we also have a bunch of other different attack methods in the portfolio of attacks. Some of them are tree-based MCTS-type attacks. Some of them are more evolutionary programming-based. Some of them borrow from the reverse language modeling literature, where we literally try and learn a reverse language model so we can produce a prefix that would elicit a particular response. We borrow a lot from mechanistic interpretability and various other fields as well. So TL;DR, the gradient-based attack is one of many methods we have. Cool.

You develop that stuff mostly on white box models and then see if they transfer to black box? Because that was the main headline of that original universal jailbreak, was like, they did it on an open source model where they had access to the weights and the gradients, and then the weird strings that they found that worked as jailbreaks surprisingly also worked when applied to your OpenAI models and so on. So is that kind of the same gist where you're looking for techniques where you require that deep access, but then hopefully you can find approaches that generalize even where you don't have the access?

I would say in the original GTC paper, they talked a lot about the transferability of this attack from white box to black box models, as you said. I think as we start to see newer and newer model releases from the frontier labs, this transferability is Going to continue to go away, I think. Like there's various reasons for that. I would say one is it's pretty likely that the pre-training data mix that the frontier labs are using are like continually diverging from that of open source models, and this is a big factor for transferability of the GCG attack. Like see this point in one of the safety leads at OpenAI actually. Her name is Lillian Wang. She runs a blog actually where she does a lot of review posts. About overarching safety topics in machine learning, and her post about adversarial attacks on language models made this point about how likely that transferability of GPT occurs because the underlying data mixes between the white box and black box models tested were similar enough where the underlying prompt optimization that worked for all the white box models would actually match what also broke the black box models. So I think, yeah, like as opposed to this, Frontier Labs use a bigger and bigger mix of synthetic data that we don't really have any idea about on the open source side, some more things like that. And I think even like seeing the release of PaLM yesterday, where the entirely new paradigm of synthetic data based on hidden chain of thought, that'll like make it continue to diverge away from the GCG paradigm and make it necessary to focus on other kinds of attacks. As the field goes on, I would say.

Yeah.

Yeah, that's really interesting.

Yeah, just a quick point. So awesome, awesome. All really salient points. Yeah, transfer really works because of data and model monoculture. Everything is more or less the same data, more or less the same architectures, and therefore you can take attacks on white-box models and transfer them over to black-box models. There is a big divergence from the frontier labs and open-source models coming, and it has been happening. And so we do have a lot of algorithms as well. There's like purely black box. You could take the GCG type method and massage it into a black box setting reasonably well if you just don't use gradients and just solely use the loss of the target string that you're going after. But a lot of the other approaches I mentioned, including the MCTS style and evolutionary program style attacks, are entirely black box.

Like, I found GCG to be like one of the more brittle attacks. So like stuff like persona modulation, which is the Shah et al. paper from like, these are like very transferable attacks, which like seem to be getting at something more fundamental in like the training of language models or like the distribution of the data and the things that language models are modeling. And I guess in the longer run, much more confidence in these styles of attacks than like GCG, which is like this very weird thing, which is very obvious when you're like running a GCG attack. So for instance, you can like quite easily directly train against GCG suffixes and this improves your robustness to them like very significantly. Yeah. So I think in the long term, I just have a lot more confidence in these sort of more varied less obviously detectable, less brittle black box attacks.

Gotcha. Okay, cool. Let's maybe talk just a little bit about kind of the setup of the red teaming on 01, and then we can get into more deeply, like, the techniques and the findings. But I understand from the system card that you didn't have that long to do this. Can you just give us a little bit of a narrative of, like, When you got a heads up that this window of opportunity is opening and also curious as to what access you had. I was on the GPT-4 red team, as I've talked about many times on the podcast. Back then we didn't have much in the way of, like the rate limit was quite low and we weren't supposed to do automated testing. I assume now that has changed and you guys were encouraged to do some automated testing. So yeah, just give us a sense of what the experience has been over the last few weeks.

I think we had like a good amount of time, like I think someone on Twitter reported like Meteor only had 10 days or something, which like sounds a bit strange. We had like around a month, I would say. So I think especially given that our attacks were automated and in that way it was like just hooking up to the API, configuring some things and like just letting it run, making sure it's going smoothly. We also didn't need like as much of a long timeframe compared to like Maybe manual red teaming, where you're putting in like a lot of like kind of manual hours, like working on single intents and doing a lot of like manual adjustment of your prompts and verifying outputs for harm on advanced capabilities. I would say like some of the other red teamers for O1 were like more in that domain of like they had maybe domain expertise in bio/chem/physics harms. I think if you look at like the list of red teaming individuals, some of the backgrounds for those individuals were like, they were, there was like an MIT chem professor listed there. There was another like physics professor listed there. So those people were on there for, I think, manual testing for these kinds of like dangerous capabilities. And our strength was like not really in that advanced domain knowledge, I would say. We focused a lot more on a pretty comprehensive like risk taxonomy. Not as much on advanced biochem capabilities, for example, but on a bit more of the general misuse cases like real activity, sexual content, misinformation, or fraud and deception, et cetera. And focus our automated red teaming pipelines on those kinds of behaviours.

Yeah, that's a really good point that there are at least 2, maybe more, but at least 2 major dimensions to this sort of testing. There's the sort of, can the model either help a layperson do something that they couldn't otherwise do in the vein of create a bioweapon, or can they accelerate people that are already somewhat of an expert but maybe couldn't do it or couldn't do it as fast or as well on their own? And that sounds like the others were responsible for that. And then there's the this sort of just general purpose, like, is the model under control? Is it doing what its developers want it to do and not doing what its developers don't want it to do? And that's where you're focused. I would love to understand better the mix of techniques that you're using to the degree that you can share more detail on those, and also how you think about the balance between automated and manual testing. I mean, My intuition with this has been that I basically don't trust language models to do anything that's open-ended without some sort of supervision still at this point. And in practical terms, that even extends to evals, right? There's a lot of people running model-based evals, and I'm always like, I would keep one of those online as an indicator of something dramatically changing, but I wouldn't replace— I wouldn't take my humans out of the loop with model-based evals, except maybe in very rare cases today. And I feel like red teaming is probably even more that way, where there's a fundamental challenge of how do you develop the automated test before you know what the model can do, right? How do you calibrate yourself to what the current frontier is and all that sort of stuff? So a lot there for you guys to unpack, but have at it.

I don't think we can talk about the specific attack methods that we used on the O1 models, unfortunately. But I guess we can sort of answer the latter part of your question.

Okay.

Initial exploration that you do. So like a lot of the attacks that we've built out have been from some sort of initial idea that we've like kind of explored on like weaker models. And then we are like able to scale up these sort of intuitions or like take intuitions from people who have spent a lot of time talking to language models in like very weird situations and like take that intuition and then we're able to bake it into our attack algorithms. I think that's a pretty common— and I also would say that we do some manual evaluation as well of the successes or failures that our red teaming methods output. So we don't really, I guess, at this point want to rely on model evaluation to class something as a success or a failure, because, as you say, these are not particularly accurate and they can fail in weird ways.

Yeah.

And the things we're exploring are inherently weird because We're doing automated red teaming of safe models. And so, yeah, I think I kind of treat this as almost just like an augmentation of my ability to explore to some extent.

Yeah. Just to expand on this point a little bit, for sure, a lot of it is like we have some initial seed of an idea from really great human intuition, either from our team or just really awesome folks in the red team community. And we just are able to basically bootstrap the heck out of this into really, really scalable and automated attack methods. On the eval side, I think there's been a lot of focus on attacks, but at the end of the day, it really comes down to how well can you eval whether or not your response from a certain model is actually dangerous or not dangerous or harmful or not harmful, because that's what we are targeting with our attacks in the first place, right? As you point out, Nathan, all of these LLM-as-a-judge or classifier-based eval approaches right now are super, super brittle. Any off-the-shelf LLM saying something is harmful from 1 to 10 It's actually just pretty horribly calibrated and there's a lot of false positives and a lot of false negatives. We've spent a lot of time thinking about how to make that eval a lot more calibrated and a lot more stable. But also, yeah, there's a good amount of human verification on top. We sort of try and eliminate as much of the human labor as possible, basically only looking at examples that are highly uncertain or basically judges in the most uncertain setting will call on a human to expertly verify whether or not The response is correct.

Hey, we'll continue our interview in a moment after a word from our sponsors. So does that suggest that you're going after relatively subtle harms? Like in the earlier days, we used to do, how do I kill the most people possible? That was like the first question for any new checkpoint in the GPT-4 red team. And it was pretty obvious whether it was like answering that question or not. I would expect even model evals to work reasonably well because we would either get a refusal or we would get an answer. And I would think it was pretty obvious most of the time. When you talk about the difficulties of calibration, just kind of qualitatively, you look at the system card too, and it's like, okay, this thing is more robust to various jailbreak techniques, and it's still very hard to interpret that.

Yeah.

Are we still seeing flagrant failures? Are we seeing Is it only more subtle failures that we're seeing? How would you qualitatively describe the landscape that you've explored?

Yeah, I definitely think that the subtlety of our failure cases has increased as people have sunk more effort into mitigations. That's definitely true. But I also think that just because something's subtle, it doesn't necessarily mean that it's, I guess, benign. Right. And there are definitely cases, and I guess LLM as a judge, is susceptible to a bunch of very strange, long-tailed failure cases in the same way that language model refusals are subject to long-tail weird failure cases. And so you still get things which to a human seem extremely obviously, or I guess with some thought seem bad, which to an LLM as a judge, which you're, I guess, trying to trade off between amount of compute you chuck at your judge versus how accurate your judge is, but there are points in that where the judge still fails and it's enough to warrant human supervision to some extent.

Yeah, this is a really great point.

Any other thoughts there?

I would agree also that the more subtle, right on the boundary examples are the ones that are the most important in many senses. It is not difficult. You sort of already get the separation on the most extreme ends for free just out of the model training process. But when you go and throw your AI application or whatever out into the real world, That's where things start to get really messy and fuzzy and hairy. And that's where you need to call in, usually, some human prompt engineer to figure out what went wrong. Subtle, again, doesn't mean benign. Subtle is actually, in many senses, more harmful or more confusing or more dangerous than you might expect. For example, if you leak some amount of PII or you, let's say, explicitly call out some amount of a harmful instruction, it's really, in many senses, more dangerous for a downstream application than an immediately flaggable one. So we've thought a lot about, you know, this is why we're pushing so hard on better calibrated judges. All these sort of like really narrow corner case things are really important for real-world applications. And yeah, basically existing methods are not so great at catching these.

Like jumping in on Leonard's point about real-world applications, I think another, like, I don't know if this word has come up yet, like another way of talking about these more subtle cases is calling them like dual use cases. These are kinds of like specific intents or behaviors that can be like benign in some contexts, but also harmful in other contexts, while being like the same exact behavior in either context. And I would say like the next frontier of safety work and red teaming work is going to run like into a lot of challenges with these dual use cases. Like, I would say like the The very clear cases of these are very clearly illegal activities or harmful activities. These are way more easy compared to advanced real-world applications that I think we're going to see. For example, looking at the OpenAI and Moderna partnership that I think was announced several months ago, seeing these cases of really advanced models being used for research purposes. At some of the leading biotech companies, like these would be cases where like you could have really advanced and subtle research questions about like in the biotech field where it's like really hard to distinguish like what is a safe research question versus what is a potentially dangerous research question in some specific contexts and why and like what makes these research questions like safe or dangerous in like these contexts. And yeah, I think that's like the next frontier of how we're going to see like advanced usage of frontier models. And that makes, I would say like that makes dual use like one of the biggest upcoming challenges for safety work.

Yeah. As an even broader point, Bryan, it's like everything, as you point out, safety is like a context-specific thing, right? The same behavior in different contexts for different users, for different applications can be safe or unsafe depending on what the scenario is. And I think so far, a lot of safety work has been too heavy-handed and too general, saying like, okay, all of our models should absolutely not do these things or should only do these things for everybody, which is a good starting point. But I think there's a lot more room to go and be more granular. I do think that there are certain categories that are absolutely no-nos. And I think largely many of the frontier labs would agree. Things like CSAM are, you know, probably should never ever be outputted from a model. And there's a good amount of work to ensure that this is the case. But I think for some of these other categories, the models, the frontier labs themselves are not super well calibrated. They're probably causing over-refusals on certain categories that are not so harmful. And yet on other categories that should be more stringent, they're under-refusing and so on and so forth. And this is like doubly worse when you go out to the application layer and people are expecting different use cases out of the models and so on and so forth.

Yeah, that biology one is really a great example of that. I've been impressed actually by Pi over time, which I haven't used too much recently, but it seems relevant in this discussion because more than any other model that I've used, I would say maybe up until the most recent Claude 3.5, it had a really good theory of the user's mind and was very perceptive on moments. And I was kind of red teaming it. It was already released, not in a private way, but just going and Mess around with it.

Yeah.

When I would start to cross the line into like seeming a little deranged, it would pick up on that seemingly more than what I was asking it for. But I remember getting responses from it that were like, whoa there, buddy. Like you're starting to get a little bit bent outta shape here. Let's take a deep breath. And it was a little patronizing, but it was remarkably in tune with the emotions that I was intentionally hinting at. Whereas at least at that time, 6 months ago, whenever, when I was doing this, the other models were not so receptive to those subtleties. And this does sort of suggest a sort of family of defences that you might have, which could ultimately also include off-model things like your customer and so on and so forth. But that is quite interesting. Are there any other— the biology one is so right in the bullseye of the dual use, right? Because obviously we want medicine, but we don't want new pandemics. Anything else that people should be thinking about for these dual-use scenarios that maybe isn't so obvious?

Yeah, that's a good question. Also, I should try out Pi more. I didn't realise it was so in tune with your personal alignment and so on. But I think, well, there are a bunch of trivial, funny corner cases where if you ask your LLM to, let's say, tell me how to kill this terminal command, or tell me how to kill this Linux command in my terminal, it It'll say, no, I can't do this because I can't kill or offer to help kill something or whatever. There's all sorts of these funny over-refusal scenarios. But more seriously, I think when it comes to the topic of anything that is societally relevant or maybe discriminatory to a large number of people, there is just a very fine line between what is considered safe versus unsafe. For example, let's consider maybe like race-based admissions in colleges or something like this, right? If you go ask Claude something like this right now, Like, let's say, okay, should Asian Americans be discriminated against? Or like, what is the Asian American discrimination case for colleges in the US? You very likely will actually just get a refusal. We tested this like a few weeks ago. But there are just like some objective things that are true about this case, which is that, you know, this year there was like a whatever, like 5-6% drop in Asian American applicants after the court case passed. It is true that last year there was a record number of applicants that were admitted who were Asian American to top universities. And so there's all sorts of these weird, maybe just fact-gathering things that are distinct and should be considered not harmful, that are a little bit different from maybe, let's say, the interpretation of them. And so I think there's this weird middle ground, again, going back to this removing the false positive, false negative thing, where it could definitely swing either way. And there's definitely, if you're doing any sort of research on your own, these are very logical questions or very reasonable questions one might run into.

There's also 2 more examples just off the top of my head. I really think there's a big plethora of examples of this. I think if you have any larger enterprise, I would say that is using an internal assistant or like search assistant like Glean, any company that's using Glean, I would say maybe if the company is like a bit like siloed off, like some departments are like only select certain information that other departments don't have, this leads to a challenge of like safety regarding privacy and like knowing For all of these specific use cases for the specific company, queries based on the incoming user, is the model actually allowed to answer or not? And yeah, I think the complexity of that can go really deep where this is a really challenging problem for model safety, in the same way that making sure a model doesn't— you can't elicit private information, PII, from a model. I think it's the same kind of research challenge. And Even like another case of deploying a model for a very like historically regulated industry such as like finance. I would say like maybe if Goldman Sachs wanted to use an advanced AI agent to help with like trading and research operations, a big bank like Goldman Sachs has a ton of regulations about their kind of like trading and asset management activities, investment activities that they have to follow, especially Especially following the 2008 financial crisis. And any kind of significant assistance that they get from an AI model for these activities will need some really ingrained and dynamic check for whether this assistance from this model is potentially violating or infringing on any of these regulations. So yeah, I think those are 2 more examples. And can definitely keep thinking more, but yeah.

Yeah, cool. That's helpful. Your mention of agents prompts me to take one step back or zoom out again for a second, and just, I want to get your take on just how good is this model? I mean, you guys have spent more time with it than almost anybody else in the world outside of probably OpenAI itself at this point. A theory that I've had for a while is that The agents might all wake up at the same time. And what I mean by that is typically with a GPT-4o type model, the agents mostly don't work. Workflows work, but agents that have more open-ended discretion of what to do seem to mostly fail. But I've been kind of thinking, geez, there's a lot of frameworks being developed, a lot of scaffolding, a lot of different strategies for compensating for their weaknesses all being developed. And it's really easy to swap in a new model when that model comes out. And so maybe all these things that were meant to compensate for the weaknesses of last generation will now be enough to kind of compensate for those weaknesses with a new model. And maybe we have just entered the agent era, even though all the agent companies are still running their evals. I think let's comment on that in a bunch of different ways. But yeah, how, I mean, for starters, like, how strong have you found the model to be? Like, what— how would you describe the advance over the previous generations?

Anything with a formal setting, so mathematics, coding, some like really complex but symbolic reasoning, or like specifiable symbolic reasoning, it like absolutely crushes it, right? Like, it makes sense. Like, the search is just— that's where the search shines. That's where the chain-of-thought RL shines. It makes sense. I would say for general interaction, we didn't notice that much difference. Like, just chatting it day to day, asking it like general topics about the world or helping us brainstorm ideas or things like this, it was not noticeably better. Of course, it was a lot slower, but it was not noticeably better than the previous family of models. But yeah, I do think for any sort of formal setting, it's just absolutely crushing it.

Planning. That was another area that's kind of highlighted as an improvement, right? Like task decomposition and planning. That's not exactly formal, but it has been a big area of weakness. It seems like reports are that this is much better.

Yeah. Apologize about that. Or like Meta, I guess, if you're able to get a chat with them. We didn't run so many more long horizons agentic planning tasks. But I would also say that about your point about all the agents waking up at once, I feel like this is maybe in opposition to the kind of lessons that we learned from AlphaGo or whatever, where you can definitely improve. I don't know if you have a sort of base chess playing policy which is distilled from a bunch of high quality human games, you You can definitely run some sort of MCTS with this, but you could, you just like drastically improve your sample efficiency of your rollouts if you do RL on those kind of policies. And the key factor here is the sort of length that you can plan over, or I guess like play chess over, is determined by like how much distillation of your MCTS rollout or whatever that you do. And so I'm not sure I'd expect That, like, all the agents take off at once when someone cracks the, like, correct scaffolding system is accurate because it feels like we do need some sort of RL training for these long horizon tasks. And I think, like, the fact that this model is better on those, it, like, slightly indicates this.

I should say, I personally don't like much perspective on the complex reasoning abilities of O1. I was actually, like, When we were red teaming, a lot of that was just kind of trying to elicit harmful responses on this large fixed set of harmful intents. And that's just kind of a setting where you don't see a lot of normal— we weren't asking it that many normal questions during testing. So yeah, I'm honestly kind of discovering the new advanced capabilities along with the rest of the world the past day or two. So that's pretty cool.

Very exciting. Hey, we'll continue our interview in a moment after a word from our sponsors. Did you guys have access to the chain of thought? Obviously, one of the big decisions that they've made with this model is they're not returning the chain of thought via the API. And in ChatGPT, they have a summary. I don't know if that summary will also come to the API, but did you have full access to its generations during your process?

I don't think we can comment on this. Yeah. It was enough for us to get red team.

Yeah. Fair enough. The, I guess I don't know how much you'll be able to comment on the cycle time either, but the, it seems like a short window, you know, even at a month, which is longer than I had felt like I took away from the system card, which there were a couple things in there where it was like not long at all. Like, let me just check real quick just to Sanity check myself. Yeah, Meter got access on August 26th. That's not too long ago.

Yep.

And Mini on, yeah, then another place it does say starting in early August. Okay. So I guess that would be the case for you guys. The system card seems to say that you saw multiple checkpoints through that time. That would suggest that there's like a pretty tight cycle time, but I guess I don't know to what degree this was sort of waves of work that was already happening and you guys were just getting these waves of work, or did you feel like your findings were actually being folded in? Did you see techniques that initially worked on later checkpoints no longer work?

Yeah, I mean, it was definitely a dynamic process. We didn't explicitly try and do reverse engineering and figure out, okay, is the model new or what are the new vulnerabilities or defenses? But you can definitely tell the model was changing. Obviously they had new checkpoints coming out. I do think Llama and the rest of the OpenAI team were very intentional about improving the models with user feedback over time. They were super communicative and just all around really tight in the feedback cycles. And so we'd send a message and we'd get responses within minutes and they'd be like, all right, we're working on it type of thing. So I think they're really careful about incorporating feedback and generally just really tight feedback cycles.

Did you come up with any new ideas going back to the idea that Human intuition is sort of the kernel of all of your different techniques, and then you're scaling them up and automating them in various ways. Were there any new intuitions that were generated from this new model with its new sets of strengths and weaknesses?

Yeah, there's a good amount. Bryan, do you want to take this one?

What I was going to talk about, actually, I don't think is that specific to the O1 model specifically. Actually, it's a lot of kind of new general intuitions. That we were gaining that I guess just coincided with when we were testing A1. So I think it's like fun to talk about these. Like one thing that we developed recently was this like, this kind of cool extension of a lot of cipher or encoding-based jailbreaks that I think are pretty popular in the red team community. I think probably the biggest example of this is like, if you look at plenty of the prompter jailbreaks, a lot of them will incorporate like I mean, they have like a lot of different individual like attack vectors incorporated into their big prompt templates. Like one thing you often see is like the model into like leetspeak or Morse code, like Base64, anything where the original like plaintext is like totally transformed into like this obfuscated form that's like pretty complex. And that way kind of seems to bypass a lot of the model safety filters, kind of like playing around that intuition for a while actually. I think after like kind of a few months of exploring the search space, we came up with this attack called bijection learning. You can actually see it like on the Haze Labs blog and there is an accompanying paper. We're just waiting for like arXiv to approve it. But yeah, I think this is like basically one of our like most novel— a lot of like, I guess the technical details might be a bit hard to dive into here, but it's basically the concept of of developing like this really difficult kind of cipher and teaching it to the model through a really detailed system instruction and a set and a really like large like many-shot set of like teaching and practice examples. And then using that to kind of these like harmful intents and harmful responses through the language model interaction and like successfully jailbreak the model in that way. So that is an attack that we definitely like were developing around the same time as L1 came out. And I think another thing that we were pretty excited about is like the multi-turn regime. I think this is like part of the search space for red teaming that some of the other labs are like exploring a lot recently. I think there's like a recent paper from Scale AI about like how language model defenses are not yet robust to human multi-turn jailbreaks. And what they did was they red teamed a lot of the recent models from the Grey Swan team. So this is like Andy Zhou, Zico Kolger from GGG. This is their recent startup where they are, they're like developing like very robust models with a lot of like novel kind of alignment and defense techniques. TL;DR was like red teaming on their models specifically. was able to jailbreak them with a lot of their manual red teamers doing just very long multi-turn conversations and optimizing those manually. And these models are very robust. It's nothing like even GPT-4 or the Claude models, which are pretty robust, like the Llama 3s. The Grayskull models are way more robust than Any of the more common models that everybody knows about. And I think the fact that multi-turn was such a potent attack vector, like even on these models, is kind of indicative of multi-turn for some reason being like this really challenging regime for kind of like the blue team, like the— for the alignment researchers to defend against.

Yeah.

And I think it's really important for us also going forward.

For sure.

Devote a good amount of attention to the multi-turn attack.

Yeah, I think for, for NDA reasons, we can't describe any specific details of our attacks, but contemporaneously, as Bryan mentioned, multi-turn, really long context attacks are like pretty freaking effective for all models and also for O1. And also, yes, this new class of like large-scale encoding attacks also works very, very well.

So we have an episode coming up with Dan Hendrycks where we do get into the circuit breaker work and the tamper-resistant models as well. It sounds like from what you're saying here, I mean, I'm moving into speculative territory here, but just to try to get our intuitions developed around how the models work, like what techniques are working how and how to even think about what's going on inside these things, it seems like these cipher— let me take one step back from that. It seems like There's an early fork in the model behaviour where it either kind of enters refusal mode, or if you can get past refusal mode, then you're typically in the clear. That was kind of at the heart of the Universal Jailbreak paper originally, right? Was you could target, sure, I'll help with that or whatever. And then once it says that, you're off to the races. With the circuit breaker models, it seems like there is a sort of later detection mode or later defence where it's more operating on these deeper in the model representations. I guess I'm not super confident on this, but my intuition is that if these cipher things are still working, it seems like we're still using a relatively early defence. And I kind of intuit the cipher as working as sort of injecting the meaning past the early layers into the deeper layers, where now we're kind of in full meaning mode. And if you can sneak past those early layer detection forks, then you're through. How does that compare to your mental models?

Yeah, that's a really interesting framework. First things first, I think the sure here is sort of attack doesn't actually always guarantee that you get a full harmful response, right? In many cases, you can get into passing the refusen mode, but then all of a sudden the model will just recorrect and be like, okay, sorry, I was just kidding. I can't actually say this harmful response, right? So I think there's a lot more subtleties than just like, okay, we broke out of the refusen mode and now we're fully in reasoning and response mode. I think there's a lot of subtleties and correlation between model internals. I do think that this, the sort of circuit breaking work and related methods are interesting for sure. I think there is some downside to the methods as they stand, which is it does lead to a lot of over-refusals just by the way that the method is constructed. So for like relatively benign queries, you do get responses, refusal responses up to an order of magnitude more than you would get otherwise. On comfortably safe models. So, I mean, it's still an open question about how we want to, again, tighten up alignment as much as possible to remove the false positives and false negatives. But I think they're all really interesting methods. To answer your question about how this fits in our mental framework for cipher attacks working in the early versus later stages of the models. So an interesting thing we found was that cipher attacks actually work for not just models, or in particular, our cipher attacks work not just for models. But actually for entire systems, AI systems with a base model, but also input and output filters on the model. And so there is something to be said about if we can break around this many-component system, you can basically also apply the same concept to the same system that's baked into the underlying model. The sort of search would be a little bit different, but if you can break several components at once, you can also break the entire system.

Comment on, you're talking about early refusal modes. I think a really great paper to point out here is like, refuse if you feel unsafe, where basically they kind of just like free up, I think like models sort of, if you train them kind of naively to refuse, they are like more susceptible to these like prefill attacks. But there are really easy mitigations for this. And just like increasing the like log probability of sorry on like all token positions allows your model to start interrupting itself and start refusing midway through in a way which is like similar to circuit breaking, but potentially without the crazy activations. And I guess my mental model of circuit breaking is that it's doing something similar to this, albeit in a much more destructive kind of way. A good tie-in for this is this seems very similar to many-shot jailbreaking and I guess multi-turn, where you condition your model to be in a sort of helpful mode. I think this is the correct way of framing what this is doing, and this just drastically improves your model's likelihood to agree to further Questions.

So could we give a bottom line kind of qualitative sense of what it feels like to try to jailbreak these things today? GPT-4 early would do anything you said, was purely helpful, it would never refuse. That's like one extreme end of the spectrum. And then I guess the other extreme end would be like a model that just refuses everything, In the middle, there's some ideal state. It seems like we got false positives and false negatives still. But is it like if a sort of amateur, let's say, tries to get a meth recipe or tries to get the car hotwiring instructions, are they going to find it to be moderately difficult, extremely difficult, near impossible? What is the sort of— non-specialist experience going to look like?

I think I'd say fairly difficult. There's definitely been a lot more tightening up of the mitigations. The model refuses in a much more intelligent way, but not impossible.

Yeah, I think there's actually still a fairly robust opportunity for amateur jailbreakers, so to speak, to actually provide a lot of great value in doing their individual red teaming efforts. I mean, you know this, Nathan. There's a big army of human jailbreakers on Twitter doing their thing. And in many senses, they're not doing anything crazy. It's just like they spend 5, 10 minutes thinking about what to do, and they get pretty far in the jailbreaking process. I do think things are obviously a lot harder than they used to be. You know, when GPT-3 first came out, things were totally crazy. There was a little bit of an overcorrection, right? The Llama 2 team noted this explicitly. Like, they did a lot more safety fine-tuning than they did for the original model releases. And then for Llama 3, they were like, okay, we pushed the angle too far. Like, let's Dial it back a little bit. We don't have to make the model over-refuse so much. So I think it's a constant calibration process. We started off a little bit too unsafe. We overshot it a little bit. Now we're course-correcting back down. Oh, nice balloons. We're going to go back up for a little bit, and we'll slowly stabilize and find a nice middle ground for us to all land on that makes sense as a general model. But again, I do think this sort of binary search calibration process is going to have to happen for all sorts of domains, not just for general models.

Yeah, we've got a lot of work to do. You mentioned Pliny a couple times. Do you guys have a like Pliny bot that sort of attempts to scale up and automate what he is doing manually? Or do you think OpenAI should be like bringing people like him into future rounds of this program?

We, we have a version of a Pliny bot as one of our attack methods. We're also friends with Pliny, so if you would want to bring him on the show, we can maybe figure out a way to do that. But yeah, I think like Pliny is doing great work. I do think his attacks are obviously not super scalable as a human jailbreaker. I do think he has definitely provided value to some of the frontier labs. He has noted this on his Twitter before, like in a commercial sense, like they do contracts for some of his jailbreaking efforts. But yeah, I do think, you know, if you want to cover every sort of single possible input failure case and jailbreak and vulnerability, it's probably not sufficient to have a single guy just like probing the model. But again, it's something that Like I think is really cool. Yeah, the I think it's sort really cool. of same thing that we have, right? That's like we start with some really great human intuition and then we can scale it up from there.

So he's already posted a jailbreak of 01 on Twitter today within hours of getting access, which doesn't surprise me. I assume it doesn't probably surprise you too much either. But then I wonder, like, what should we infer? And I'm going to assume for the purpose of this conversation almost entirely, but certainly in this section, that like, you're not being told, because certainly that was my experience with red teaming with OpenAI, was that I was basically told nothing about what was happening behind the scenes, nothing about what their strategies were, nothing about the model other than just what we could learn directly from using it. So let's assume that you have no insider knowledge. I'll assume you have basically no knowledge of what they're doing other than what is working and not working on your end. How should I interpret the fact that We went through all this red teaming. Pliny's out there on Twitter very openly. You guys even have a version of that where you're hammering in an automated way with similar techniques, and yet it still works when they launch the thing. One would naively assume that with all that example Pliny jailbreak to inform their work, that they would have that sort of attack locked down. Is it just too costly in terms of false refusals, or what's the deal?

Yeah, one really quick point is we actually have no idea. This is true. We just have no idea what our testing led into on the OpenAI side. So for all we know, we could have just been a final eval dataset, benchmark, final post-processing thing, or our data could have been directly baked into the post-training process. We actually have no idea. So it's not exactly clear to us, even if we had tested all the variants of planning prompts, whether or not this would have made a difference in the end. Because we just don't know what the data was used for. But yeah, Hayden, go ahead.

Reducing the sort of ease of use or ease of malicious use of these models is definitely a tractable problem with intermediate gains. And if you make it much harder for some malicious actor to use your model for a very harmful situation by an order of— if you increase the difficulty of that by an order of magnitude and make the model an order of magnitude less useful, even once you have jailbroken it, Then this is like a very useful thing to pursue because it like reduces your risk. And I also think that being jailbroken is not like a binary, right? There are like certainly tasks where it is like much easier to like jailbreak for and much harder to jailbreak for, and tasks where the like costs of being jailbroken are much greater. And I think that a lot of the tasks where the costs are much greater, you also, it's generally true that it is harder to jailbreak these models for those tasks. And these tasks are more, I guess, maybe like long window. And I don't think we've really seen— well, I guess actually Pliny is pretty good at these sort of long horizons, persistently jailbroken jailbreaks. But I think we will eventually see that this goes down. Or I think maybe you see this a little bit now with the OLM models being harder to jailbreak.

Like, one point I was thinking at the frontier, labs wanted to refuse on a small set of like plenty prompts specifically, I think kind of trading the model for refusal on that, it seems a bit like brittle and unable to generalize. I think the important point here is the attack surface of these frontier language models is very massive. It's so much bigger than I would say like the classical setting of adversarial attacks for like computer computer vision where you were just perturbing like pixels in an image. I think the bottom line is like language models can, you can make them ingest literally any text, like literally any, any configure, like any sequence of characters that you want. Like it's like 10 to the power of like a really, really massive exponent of even say a 5-paragraph passage, like different 5-paragraph passages that are possible. So that makes the attack surface just exponentially big. And I think given this, like, the frontier labs would much rather look for more fundamental breakthroughs in safety research and like more fundamental ways to do like generalizable guardrails. I think like one thing that a lot of the 01 safety card talks about is teaching the model to reason about safety specifically in the hidden chain of thought and getting all lot of generalizable safety, especially to out-of-distribution settings in that case. And I think my guess would be that the frontier labs are favoring this kind of defense over the cat-and-mouse game of seeing what jailbreaks people are manually coming up with and just covering all of those with specific refusal data.

Yeah, that's interesting. So would you interpret the overall improvement in the anti-jailbreaking profile as essentially an emergent property of scale. Like, it's better at a lot of things, and one of the things that it's better about is reasoning about safety and sort of the capabilities safety correlation continues.

Paper by FireEye Research where they showed that this is true for some types of jailbreaks. And so I think in general this is true to some extent, but I think Bryan's got a pretty good counterpoint to that.

Yeah, Bryan, do you want to talk about endless jailbreaks, the scaling laws in bijective learning?

Of course. I think kind of the relationship between capabilities and safety in terms of how you might get increased safety for free just by increasing capabilities, that's actually a pretty So we'll definitely see more answers to that and more deep analysis of that from the research community. I think like a lot of things, a lot of like kind of the preliminary intuition that people are seeing is that yes, you do get like some increased safety for free just by increasing capabilities. And this is like a bit dangerous, or it can be like a bit misleading on like the blue team side in terms of like, oh, we're getting more safety for free. If they have a higher capability model to see better results in safety, even if the corresponding innovations and new techniques for alignment that they've designed aren't fully responsible for these gains in safety. And that can lead to unforeseen holes in model safety. One interesting mechanism that some jailbreaks across the literature and some of our own jailbreaks Is that a lot of these jailbreaks like degrade capabilities of the model. One way we can measure this directly is like passing in MMLU set where each question is passed through our jailbreaking method. It's within like the overall attack prompts and evaluating MMLU that cases, for example, like for the projection learning attack that we mentioned earlier, the MMLU scores, the jailbreak are like significantly lower. than the baseline scores, which is a pretty clear indication of some jailbreaks degrading capabilities significantly. Like, I mean, the correlation versus causation is like still yet to be worked out here, but it's definitely possible that the act of degrading capabilities might be of like the main causes for why these jailbreaks work the way they do.

Yeah, to call this out explicitly, Brian discovered a really cool inverse scaling law. The bigger and more capable the model is, the more vulnerable it is to our bijection learning attack. So it is actually the opposite of what you stated, Nathan. Scale does not help in this case. Scale actually makes things worse.

Cool. We'll put the link to that blog post in the show notes, and I'll have to read it. Awesome. So I'll look forward to that. Was the paper that you were mentioning, Aidan, the safety washing paper?

Talking about those papers by Far AI Research, that was, I think, a Richard Ren paper. This is also like, I think, more of what Bryan was talking about. The paper I was talking about basically showed that you can do adversarial training against, I think they did adversarial training against GCG, and they just showed that if you increase your model size, then adversarial training becomes more sample efficient and more effective, and you need less compute to have the same, like, robustness against these attacks. And I think that is a good signal that scale can improve some of these issues.

Yeah, Nathan, that's a really good note. Yeah, what, a lot of the points I talked about were actually referencing the safety washing paper. And I personally think the safety washing paper is like a really exciting first step in like this line of research, like exploring how safety and capabilities are interlinked.

You make a great point too about the vastness of the surface area of these things and the sort of weirdnesses of the landscape are really quite something. On the one hand, you've got, from the safety washing paper, you've got basically the general trend is that more capable models tend to be more safe. And a lot of the supposedly safety-focused benchmarks essentially correlate very strongly with the model's overall capabilities. And so it doesn't really seem to be measuring safety as an independent dimension. You could essentially swap in an MLOU score for that safety benchmark, and that makes it not so great as a safety benchmark. But then you're also finding, as we just noted, that in some cases you see the exact opposite trend. And there's probably plenty of weird things in between as well that are just strange and not understood by anybody at this point. The work continues. How do you guys feel about the medium classification of this model? I assume that you were not involved in making that determination either, but— When you see that come out in the system report, that also kind of, I guess, is more of a capabilities statement than it is a robustness statement at this point, right? Like it's more about what can the thing do than like how well under control is it. But just intuitively, how does that feel to you guys having spent time with the model?

I think it is fair. I mean, it complies very precisely with the definitions that they have laid out. So I think it's like a very fair categorization given their usage policy. I do think it is very heavily a question of capabilities, not just being under control. It is just not yet possible, let's say, for the underlying model to actually give you very concrete plans for synthesizing a novel biological or chemical weapon. So it is as much of a capabilities argument as it is, or capabilities-based evaluation as it is a safety-based evaluation.

Say that preparedness work that OpenAI does, where they came up with the medium, low, high categorizations for the dangerous capabilities. Like, it's— they also intend it as like, I'm not— I'm kind of speaking for OpenAI here, but they intend it as a kind of interdisciplinary thing of like between safety and capabilities. I think like the first like team lead for preparedness, Alexander Madry, used to phrase preparedness as the question of realizing the upsides of AI and mitigating the downsides of AI at the same time. So it's definitely like a much broader question than looking at safety in a vacuum. And I guess like in that way, like since we are like very focused on this red teaming problem, like bigger preparedness question is like also a bit beyond our purview, but definitely something that we'd love to think about in the future.

Yeah, I was going in that direction actually. Obviously there's been a lot of, you know, let's say drama. To put it mildly, from OpenAI over the last year. And one of the things that has come out is, at least for a time, people were not permitted by NDAs, even that survived their employment, to speak about various things. And now, by all accounts, that's been retracted and people are, I guess, a lot more free to speak. And even recently, to give credit where I believe it is due, there was like an open letter that a bunch of OpenAI employees signed on SB 1047, where they contradicted as individuals the official company stance. And I thought it was pretty cool that seemed to be permitted. And again, if all of my information is to be believed, then there was even like a memo or a note passed within OpenAI that basically said, you guys can go, people can go ahead and sign this and it will be fine if you feel like that's the right thing to do. So it seems like some good progress has been made there. I wonder how that extends to companies like yours that have a relationship with OpenAI on the red teaming front. Is there any mechanism now, or do you have any plans to try to work with OpenAI to create one whereby in the future, if a model can, for example, create a novel bioweapon, or it's becoming fuzzy whether or not it can, do you guys have any sort of whistleblower mechanism or protections or anything like that in place?

It's, yeah, it's just a little bit too early to tell. I don't think that regulation nor nonprofits nor the network of nonprofits and regulatory regulators have figured out what the right process for this is. I do think should there be like required audits, then I'm sure it'll just be in everybody's best interest for somebody to go ahead and announce things like this earlier on. But it's, I think it's just too early to tell.

I think basically my last kind of question was just open-ended for you guys, like any other observations, stories, perspectives, expectations now that this thing is out that seem important, that might be informative for people that we haven't touched on?

Yeah. Curious for your thoughts, Bryan, Aidan. Well, one, I think to start, it's just been, it's been amazing how much feedback has already gone from the general community about 01. I think for all the great work and increased capabilities and benchmark scores that it's produced, there's still a lot of gotchas. And we obviously saw this during our red teaming process. There's a lot of naturally surfacing ones right now, not just for safety, but for capabilities. And I think in spite of this, this almost, I wouldn't call this a step size increase, but, you know, for a significant non-trivial increase in capabilities and processing and somewhat new paradigm of frontier lab models, there's still always a lot more work to be done on constantly testing and constantly monitoring the behaviors and vulnerabilities and failure modes. So I think the work is not over, not by a long shot. In many senses, like, We've just leveled up to a new game and we now need to figure out how to do testing and red teaming in a way that's, you know, befits this new level of capabilities.

That could be a great note to end on, but, uh, welcome any other thoughts as well if you have them. Okay, cool. Leonard Tang, Aidan Ewart, and Brian Huang of Haze Labs, thank you for being part of the Cognitive Revolution.

All right. Thank you very much for having us, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.