Microsoft + micro1 on what’s next for agentic AI safety

All right, we are live. You're trickling in from the waiting room to the main part of our forum today. And welcome again. Thank you for joining another episode of the forum that we've been kind of building and showcasing for now a few months now. Today, particularly happy to have two pioneering thinkers and doers, right, on our MicroOne forum. We welcome Tori Westerhoff to the forum. It's the first time you come in, sorry, to this, but as I was mentioning before in the practice session, we are friends through Microsoft. We've been interviewing several fellows and many people from the Microsoft ecosystem has been kind of coming around and share with us, right, what we always want to learn from a company like yours. And we also welcome our own Lius Frank, who is working at MicroOne for a few months now, probably a year. You and I work in the initial project where we're both part of the MicroOne team. And today, Liu does many different things, among which also the red teaming research. And Liu was in an event at Harvard not too long ago where she was talking about data infrastructure and training in robotics. So very good to see you again, Liu, online this time.

Thanks for having me again.

Yeah, that's fantastic. Tori, look, I think, you know, the topic of today is, of course, red teaming, but Before getting into that, and I'll ask you the same, right, for people that maybe haven't come across your work yet, but tell us your journey, right? Like you are definitely now in a leading organization and you're working on important work and we will find out even more why Red Team is important, but I'm sure you didn't start this way, right? So there's been a whole narrative about your journey landing you to where you are right now. So tell us a little bit more about what has been your journey and how did you start getting yourself into the technology, ecosystem that you're currently on.

Yeah, so happy to be here and chat through this. I like to joke that my background is very patchwork. However, I think that the patchworkiness of it is actually indicative of kind of the question that AI red teaming, when you're really expanding it beyond security to security plus, as I like to call it, really challenges the industry today. So I started my education in neuroscience of decision-making and was really, really focused on kind of the academic and scientific ways to study how humans' values and morals and judgments are changed through unconscious decision-making in particular. And I was very, very, very grateful to have a mentor say, okay, just take one internship out of academia, just try it. And I fell into a 5-year career in national security strategy. And I found this like beautiful applied approach of cognitive science to mission-critical processes, and trying to transform those business elements to match how humans think in really crucial scenarios. So worked with the CIA, the FBI, Department of State hostage recovery, DHS. And I really loved that applied science of systems thinking and how humans were interacting with tech and processes. And ultimately went and got my MBA at Wharton and studied neuroscience around specifically how how humans are interacting with systems and tech. And that led me to Microsoft in a data analytics role. Yeah, exactly. So I, I worked pretty closely with that neuroscience group and Adam Grant as a leadership fellow and got introduced to kind of this, this very niche behavioral analytics group that does data analytics around how people are interacting with tech within Microsoft. And made a pivot kind of towards the strategy of R&D tech, how we bring incubation technology to people and make it really work. And that ultimately led me to the Red Team. And now I use all of those scenarios. I use national security and neuroscience and business and most of all systems thinking to figure out, hey, how are we translating traditional security red teaming to AI red teaming, which has a really different scope, especially in the way that Microsoft has built it. So yeah, it's been a journey and I, and I do think the diversity of all my experiences underlies kind of our theory around AI red teaming at Microsoft.

Oh, it's fantastic. Look, I wish sometimes this conversation could happen in a real environment where people can really relate to. I'll tell you, 2 small comments that will let you see how I do appreciate a lot of your discovery of all of this. My first class at Harvard, 2011, after my postdoc at HBS, was system thinking. And I remember feeling terrified by what is this, going to Peter Sangi at MIT and taking a course where I was trying to figure out what it is. And as an economist, and so is Liu, but she'll tell you more about her background now, I remember not understanding behavior too much. And so Daniel Ariely from Duke, Daniel Ariely, in Italy, got me into behavioral economics. She said, you cannot be a real economist if you don't understand the behavioral side. And that happened quite recently, 2017, 2018. So I remember two specific discoveries in my own training that really made me feel we need more interdisciplinarity. And I am happy that you work in this field where you can explore it. I think I'm also be sad that many organizations have not really taken that leap or that plunge yet, right? That's one of the reasons why many technology companies So very happy to bring this up. And I cannot wait to learn on your own understanding of retiming from a perspective of so many cross-functional areas, right, that you integrate in your training. So thank you very much for being here. Over to you, Leo. Look, Leo, we do share the same either blessing or curse. I don't know what it is, which is our training economics. I can tell that I met you already more than once. You are just an incredible researcher. Just tell us your journey. Your journey is different from what maybe some economics graduates end up doing, but that's why I'd love for you to share your journey as well with us. And also, how did you start getting closer to red teaming as a concept?

Yeah, I have to share more. Like, as some, like, our audience may have known me from our previous series, so I am a trained economist. I graduated, like, 1.5 years ago from Harvard, and I joined the AI workspace, like, after I graduated, and I joined MicroOne last year, and it was a very, very lucky decision for me because I got to not only do like projects that fit my interest overall, but also I got to do research with our amazing Mark and several other colleagues at MicroOne. So we do research on economists, on the labor market of AI, era, as well as AI safety. The red teaming project was initiated by, I think, by Eva. And when she pulled me in, and I feel in love in this domain. For one reason is that I took a lot of behavioral economics classes at Harvard during my graduate year because Harvard is strong in behavioral economics. And I got to talk to the top behavioral economists at Harvard, and I feel fascinated by how broad this concept is, but still very concentrated and well represented by the beautiful models. So I fell in love, and it is like my part-time interest apart from my main interest in economics. So when I delve into the literature in AI red teaming, I feel like this is a field that I can apply behavioral economist training into the field. And maybe someday we can design like lab trails and controlled experiments in this area that we can like shed more light upon the drivers behind people, like, malicious behaviors, as well as normal, like, players in this field.

Yeah, it's fascinating. Look, I didn't know that side of you and behavioral economics, so I'm happy that we're discovering another area of interest and commonality that we have. And when I went to my economics training, behavioral economics was still kind of far, and many people were kind of twisting their, their, their nose every time that we were talking about that. But I'm happy that in the last few years, right, there's a much more legitimate space for us to, to talk about this. And this is something that was largely missing. So very, very happy to go deeper also on that insight with you. Tori, look, I'm gonna start with you because I mean, your team does this incredibly fascinating work of sometimes equally making counterintuitive points about what really generates defense is understanding vulnerability differently than just from an engineering perspective, right? And so here's my question to you. Like a lot of the conversation on red teaming these days goes into agentic systems. We do talk a lot about agents. So what are, like, in your view, right, some of the vulnerability that is surfacing in, in agentic AI that maybe we should help companies to think more about? And, and I'm asking you, uh, in the perspective of what you found out, right, with your team. Um, because I think this is something that, look, it's a fascinating question where many people might not really have the answer for, right? Uh, so love to have your, your, your, uh, intake on this.

Yeah, agent-to-agent red teaming, first of all, is fun. It's complex, and we really like doing it. And that's great news for our team because I think that it's exploding. The element of that explosion, however, I think really needs to take on a bit of intentionality and really thinking through kind of classic security principles of data permissioning, how you're deploying agents on tenants, how they're representing information in systems that are really key. So the, the major thing that we always think about when we go to red team an agentic system, and I'll note that an agentic system, that actually means a lot of different things nowadays, but any agentic system that we're gonna touch, we really care about how data protocols are being deployed and how we can abuse them, right? So offense is the best defense. My team basically works in proof of the pudding examples. Can we find out whether a system can be exploited to pull data to the wrong place, to exfiltrate for lateral? And because these agents are normally empowered with pretty useful actions for those types of things, it becomes really important that data stays where it should. So that is my first and foremost question. And that actually is kind of a more standard, to your point, like security architecture question that should feel familiar to a lot of people. Now, I am not a traditional security person, but I am an AI red teamer. And so the next questions around agents, I think, get into a little bit of the grit of the AI nature of this. So another element that I think is really important to take an intentional look at with agents is what models are doing what within your system and what those models have propensities and capabilities to do. Now, there's a pretty large trend to take the best SOTA models, as frontier as you can get, and put them behind everything with the assumption that you'll get better outputs. But with that higher capability, you end up having a really large threat landscape because those agents can theoretically do a lot more than you need them to. And that's actually very fun for an AI red team to go and play with because now you have this really capable model behind agents that may not have as structured data permissioning as maybe we want. And that's where you can get a lot of manipulation because the model is actually maybe too complex for the aim that you're trying to look for. So we tend to play with that a ton at the margins of connectivity between agents within systems and different apps, potentially on a platform in the same tenant, but also known ways to manipulate these frontier models that could be, let's say, obfuscated or opened up as, as a larger threat or risk territory because it's within an agent and human in the loop is, is intentionally pulled back. That leads me to my third point around agents. The structure around human in the loop and the types of decisions that you need humans to sign off on ultimately becomes one of the biggest security methods you can have when building an agentic system. And obviously the point of it is to reduce humans in the loop. That means that you have to think really, really carefully about which decisions humans are making And probably over-index on the security and safety ones because that's the space where we know these models can go awry. Because they're exceptional at doing really great actions, especially if you're loading them up with skills and functions and data. But we know that they're also quite vulnerable.

Yeah.

And so I think that's a, that's a big assumption that people should take when they're building these systems. That you should be over-indexing on human in the loop and security and safety, because that's where we see all of those vulnerabilities go forward. And to end with the behavioral econ, humans aren't always the best security either. We're assuming humans are in their best state. So there's also that.

Wow. You know, Tori, you make me think about— you also make me feel, wow, that was a long time ago. So why do I remember so many things that happened in the past? It's because I'm not as young as I used to be. I remember probably over 10, 15 years ago, something like that. One of my students was working as an ethical hacker for a bank. And I remember, what do you do when she used to say, we are trying to jailbreak into a banking system and that will create basically the countermeasure or the defense mechanism, right? And I was thinking back in the day it was fundamentally just about trying to crack inside of a fortress. Well, today I think the biggest vulnerability is the way we think, right? In many ways to the point that you almost have license to operate across so many different dimensions that I guess you don't have just a perimeter or reference. You might even have multiple coexisting at the same time. And this is something that I find, to be very honest, fascinating by the nature of what you said before, the ability to do experimentation as a practice that is appreciated rather than as, you know, And I still have, but it's somewhere else far from the main decision room, right? And done by geeks rather than people who are basically part of the, I think, of the strategy development of an organization. So thank you so much, Story. I really love the way you're also telling the story, right? And that connects a lot with many of us. Leo, I'm going to take you back to your basically paper, right? This kind of safety as educational alignment. And if I remember correctly from the paper, which we discussed once, right, already in the forum, right, you don't see models that are basically censoring or creating hard censorships. You see models as instructors, if I understand this correctly, right? It's the idea that basically in a conversation, you're trying to improve the performance of the model by enhancing, which is a lot of what human data training has taught us to do. But imagine people who are coming across this concept again for the first time because maybe they haven't followed the conversation that we had back when the paper was launched. Just tell a bit more about the framework that you've been kind of developing in a genetic context about the safety of the vocational alignment kind of idea that is in the paper.

Yeah, thanks for raising that. So the idea is that the model should not be a censor in terms of plausible bad intent. So oftentimes when we stress test a model, it reveals vulnerability in terms of attacks. But when we think about it further, that the model could also, if we just train the model to identify those attacks and deny them, then it is likely that the model is going to refuse a lot of plausible bad prompts. But it turns out those are good actors and they're asking the benign questions, but it is on the borderline. So in those edge cases, if the model acts like a censor and just denies the user, then it may just get some behavioral reactions from the user. The user is just going to circumvent the safeguard and probe the system further. So that's why we think that in those cases, the model should be more like an instructor, as the model can explain why it thinks, its reasoning behind its action of denial. And in this way, it can increase the likelihood that the user is going to accept, or the user knows that what triggers the model to think it is a harmful question instead of a benign one, then the user can act accordingly. So that's why we proposed the alignment, the new alignment that the model should adopt, that it's a safety as educational alignment.

Yeah. Well, great. I was thinking about when you were talking, right, that clearly red teaming as a concept is becoming very functional in the way we are basically trying to think about the strength but equally the weaknesses of given models and the ability that in the same tools we're using to make the model strong, we should use them to train the model to become more resilient to attacks, which I think is something that, again, it might be counterintuitive for coming from a traditional security background. And I'm happy— back to you, Tori. That, yeah, you're not a traditional security person, right? In the sense that we don't need that heritage to understand the threats that we're dealing with right now because they are, they're manifesting very differently, right? From threat that we used to have in the past, which brings me to a question like, imagine the people are still kind of struggling with this idea, right? That red teaming is actually safety benchmarking. And we want to say, look, it's slightly different. You can't assume that benchmark by themselves, they are a proxy for safety. And we might find out that maybe red teaming is able to surface challenges that benchmark can't surface. Look, this is an area where for many people might be the first time that they hear the differentiation, right? And look, I equally don't think that I have an informed answer to this very same question. That's where I always see this for me as the major gift of this forum is I learn from people that are working on this topic. So help me navigate the space story, right? Where do you think red teaming differentiates itself from safety benchmarks? And what have you found out that by doing red teaming we could have never really gotten, right, if we were doing traditional safety benchmarks?

I love this question because I think about it all of the time, but I completely hear you that everyone is using evals and benchmarks and red team and testing as a conglomerate word. For, hey, someone looked at it. So I do, I do think about this a lot. The simplest explanation in my frame of the delta between a benchmark and an eval and red teaming is that benchmarks and evals are really designed to give you a statistical quant output that is used across the industry. And I think that's incredibly helpful. Because you're able to get data around models and other functions within your own context that are shared and open. And everyone loves a number, right? So I really view them as statistical tools to help you ground your understanding of your AI system, your model, what have you. Super useful. The way that I describe them is twofold. One, I view those as leading indicators. So I don't ever want a red teaming exercise to exist before evals and benchmarks are pulled through because you get so much rich data about the overarching generalized behavior. And these benchmarks are specific. So I don't mean to say that they are generalized benchmarks, but rather when you have a good suite of the best-in-class benchmarks that are in the industry, you learn a lot from running those. And it's pretty cheap to do so, right? Intellectually and also economically. The other way that I describe benchmarks, and that here's where the rub around AI red teaming exists, is that I think those encompass two standard deviations away from the mean in a bell curve. They're asking a lot of questions that have been asked before, and that's why you can get really good quant around it. My team lives completely in the edge case tails of that bell curve. And we are tasked really specifically actually to complement benchmarks and ask questions that are not currently being calculated. So that's literally the function of AI red teaming in my mind and the way that we've designed it is that I really don't want my team answering questions that an eval And a statistic can, can answer for you. I want them pushing to new questions that haven't really come across our desks in a quantifiable, summarized way. So a few examples, and obviously AI evolves rapidly. So these are kind of out of play, right? But the, one of the things that we really cared about and is now being pulled into benchmarks, which is where I want the industry to go is that I would say maybe 2 years ago, our team really keyed in, especially as audio models got better, around the differences between multicultural interpretation of outputs and inputs for safety and security and how to weaponize that as an attacker and also how that can create harm in a benign scenario. So what that means is you can create endowendos, you can use characterized languages of different cultures in different languages. And because of the way the data is represented in training, that's a really asymmetrical attack surface. So if you're using a different language that's not English or doesn't have a ton of representation in that training data, you can really get through a more porous surface as an attacker. And we switched to this not just multilingual but multicultural context, because the way that these data are pulled in include a lot of the cultural nuances of language and slang and the way that people are communicating day to day. That is now really part of the way that benchmark science is moving. We are seeing a ton of multilingual benchmarks that are trained on that same type of contextualized cultural data. But before, no one was really answering that question. It was super English-focused, it was super Western culture-focused. And that's a great example of my red team working on that for 6 to 9 months to figure out, hey, what's the real attack surface here? Why do we care about it? What does it look like? How are we informing our internal evals? And you see it in the industry evolve. And the whole point of the red team is that we're chasing those new ones. So you're starting to see opinions come from my lead, Jonathan Zunger, around, hey, does AI have cognitive harms? What does that look like? How are we using AI intentionally, which I've mentioned many times? That's a space where we're not necessarily seeing benchmarks, but we do really want to ask the question. So that's how I see it. And I also will say there's just, there's like a hacker excellence in AI readiness that moves through AI systems and models in a different way that's more flexible than a benchmark that uses a lot of the attack or adversarial techniques that are already out there. So the whole point is that we're inventing new ones. And we're pushing it to industry and open sourcing it. We have an open source framework tool.

Well, Tori, look at— it's just fascinating. It reminds me of the conversation we used to have back in the days about why is organized crime sometimes so successful. And the answer is because they're very innovative, right? Different from law enforcement, they operate within boundaries while the organized crime try to breach those boundaries more and more. And it's somehow you can say there's something that we don't want to appreciate as a form of, you know, clear commitment against that kind of behavior. But when you kind of removing the moral assessment, you do recognize that this lateral thinking is actually really determined by the ability of expand the boundary of a system that is trying to be reinforced, right? And so I appreciate the fact of what you basically are pushing your teams to think and where somehow it's an unnecessary effort because we might have evaluation that already generate a pretty good foundation, right, for us to understand a social phenomenon. And therefore we can focus on the either the ancillary or the auxiliary or the peripheral conversation that are coming from that. Which I find fascinating. And look, your conversation magically always creates bridges, right, with what I want to ask each other. So Leo, look, I remember when in your paper you were running cross-validation of different models and you were really recognizing something that Equatory mentioned before, is that there was a lot of fragility in the way the model were producing refusals because all it took is some paraphrasing, some role-play setups, some known jailbreak templates that we could really know this brittleness in the models. So tell us more about what you found out from that work. And you don't need to give us any particular names of the model you tested, although whatever is public has been published, right? And that people can definitely refer back to the paper. But give me a sense about what was that understanding when you were working on the— basically what is in this case a cross-model evaluation framework, right?

Right, yeah, that's a very good question. And it links to our main finding that across many models that we find like substantial like successfulness, you know, across attack measures. So it is astounding because we thought like as the new models that they mature and they get more capable of doing things and they should also have done safety alignment training, but surprisingly, they are still vulnerable to different attack types like emotional manipulation, like role-play framing. Those work very well, and it brings us to the question why, right? So our thinking is like it has something to do, like Tory said, that traditional benchmarks, they inherently ignore the tail risks. And those in-distribution performance could not represent the overall robustness of AI safety. And that brings us to maybe the next generation of training techniques. So currently, maybe the training is still learning surface cues rather than robust online intent recognition or hazard reasoning. So in other words, the model may be just learning what a bad prompt looks like, not what a bad objective is. So I think this matters because it changes how we train and evaluate those systems. If the safety target is only like a narrow set of prompt forms and the existing benchmarks, then the model will overfit. And when we use tail risk, prompts and novel red teaming techniques to stress test, the model is likely to fail. But now, what we really need the model to do is to be robust even with out-of-distribution attacks. And we could, like, do, like, more benchmarks, for example, like, across attack robustness and, like, like reversal capabilities. So those new benchmarks can, like, reveal the real capability of the model to do deep reasoning in terms of uncovering user intent and analyzing their plausible, you know, tail risks of the prompt.

Great. Thanks so much again. I'm happy to see how the research you have done, right, naturally resonates with the work that also Tori's team is doing. And even if this were done in two different temporal moments, right, they can tend to create a convergence of understanding of what you both come across, right, as a clear insight from this. Which brings me, Tori, to kind of my next question, which is an area where I sometimes would love to be able to be even more assertive in my way of talking about this. A lot of the kind of red teaming effort seems to be about the detection, right, of the threat, right? And if we want to build real defenses, right, I was thinking, for example, how prompt injection was kind of scaling with this massive malicious instruction that we're trying to basically— probably they were considered to be some of the most dangerous attacks that have been impacting agentic models, right? What do you think is the space for us to go from detection to defenses where this is even more than detection, right? This is really creating a system that is becoming almost impenetrable to those kind of thinking. Maybe impenetrable is not the wrong term because it assumes that there is an end to that level of growth, also the malicious behavior. But let me try to phrase in a way, where do you find we're gonna be able to have the next breakthrough? In our defense mechanisms from your experience?

Yeah, I'll actually echo that. I do think that intent understanding is a really good area to think of as a next step. The, the way I try to describe the translation between threat and risk proof or testing to detection is translating in classic TTPs like bread and butter security. With AI, that obviously does change slightly because inherently it's not as reliable, it's not deterministic. But I do think the effort of doing types of research like this, and we consider ourselves like an applied research team, so it's not just creating operations that have outputs of evidence of how models or systems or products work. It's really us trying to say, Hey, what's the shape of AI right now? What are new behaviors? What are new methods that are working? How are we pushing the art of it? And I think the art of it really translates to intentionality. So I think Lou's point on, hey, are these super brittle? Is this actually just a shallow exercise on what prompts could look like? Or what is the most common, you know, scraped prompt injection or jailbreak on the internet. Well, I love that. Don't stop doing that. I don't think that it really gets to the place where human creativity is going to continue to try to push these things with novel expressions, but with the same intent. And I think that models right now, especially with the jump and reasoning models, as we're seeing truly more capability within especially the frontier LLMs. There's a universe where the intent is interpreted from a diverse signal of whatever is put into the model, not just simple prompts, but also the way that people are interacting with models, the different signatures of action or behavior that can be summarized to an intent. And I think that's where you really get a flexible defense layer that mirrors the way that LLMs themselves behave. Behave. Excuse me. So I think that's actually that I would just reiterate that is where I see the next step going. I think translating that to your work is trying to think of a scenario-based defense mindset. So you're not just having brittle block lists or, yeah, again, scraping those jailbreaks. But you're really trying to say, okay, what is the highest priority of protection that I need in this system that I'm building? Do I fully understand the model card and all of the safety work that I'm inheriting by building it this way? And can I go through from a customer-obsessed mindset or a benign user mindset, a a few kind of prioritized personas maybe, and think through what's the most important thing to protect and then focus on trying to distill that intent and use AI tools to do it. Because that's where the flexibility and creativity could be mirrored in the same, let's call it power, as the models themselves. So I think that's something you can take home today. Like try to take that persona bit, try to really prioritize what scenarios are most important to how you're deploying AI and try to also stand on the shoulders of giants with model cards, classic security, TTP thinking. Some of this, most of it is not new. It's just applying it to a new tech question of the edge cases as we're both calling it.

Well, you know, it's where I realize that I, I could keep on asking questions, but then I realize, well, my role here is not to keep on feeding my curiosities, to make sure that there is enough content that we can contain within the time of our episode. But I really appreciate the way you think, and I love to continue to talk to you about this even beside this specific event. I'm going to have my final question for you, Leo, before maybe we're looking at the questions that are coming through the Q&A. I'm going to go to your training as an economist, Leo, right? And look, one of the things that I we would love to think is that this entire over-refusal problem you were mentioning before does have an economic implication of some form. Even if you look at the cost-benefit calculation, right, uh, when a training goes wrong, there is an implication that is to some degree economic of some sort. So what do you think would be a better way for us to look at the right function to measure, right? In which way do you think that Maybe by pulling the lever of economic analysis, we could really show that not taking red teaming seriously does have an economic implication. And to give you an analogy on that, Leo, and then I'll ask you to answer that, it's not too dissimilar to when we were looking at the cost of inaction on climate, where not doing anything was actually more expensive than doing something, right? So how do you think we can bring this forward? In terms of the kind of economic calculation of failure in this case?

Yeah, I do think that over-refusal can be framed naturally in economic terms. Like, we can think of it in terms of the reward model that we use to train the models. And I think that currently the issue is that we often price just one side of the problem. Instead of also account for the over-refusal consequences. And it is natural for the models to do that because they— it is oftentimes the social cost that they do not take into account to their private function. And we think about forced refusals. So when the model refused a good question, The consequence is not free. It actually creates lost user value. Like, they've wasted their time on this. They— it potentially, like, reduces user trust to the platform, and sometimes it pushes people to, like, worse information resources. So this would not be captured by the traditional reward models we have right now. Refusing those cases in the current reward model looks like the safest option, even though it is socially inefficient. So I think that the question is not anymore just, is there any risk? And the question now is that given the uncertainty about the user's intent, What action would produce the best expected outcome? Is it just answering? Is it clarifying? Or is it refusing? So once we think about it that way, the target becomes much more balanced, and we can have a more, a better action space from the model. So I think the better objective function would weigh user value against their social harm while also accounting for trust and circumvention effects. So under that objective, it is more balanced. And then we can envision that explanatory refusals would dominate blunted refusals because they would preserve the safety benefit while still maintaining higher trust and keep the user engaged in a legitimate path.

Wow. Fascinating. Yeah. Look, I think there is actually in the Q&A, there is probably a follow-up question from Angela Wilson that probably I'll ask you. And so if that's okay with you, Tori, and I'll try to kind of address the question from the Q&A before we wrap up. And I know we have 5 minutes or so to do that. So there's a question from Gabby, Tori, I'd like to ask you, and I'll actually ask the very first part of her question, which is, well, how do you decide which human decision to test? And I'm asking, imagine you're working with your team, your team comes up and say, Tori, we'd like to test this, right? What determines your yes or no based on that? Like, or yes or no is not the right binary question, but what determines the priority for you? How do you determine that yes, we should test this based on what level of either analysis hunch, benchmark, industry reports? What is that defines the prioritization of what kind of human decision you want to test to support your red teaming exercise?

The answer differs on the type of surface we're attacking, to be quite honest. So if we are red teaming a frontier model or a new special purpose model, I really do care about evals and the data that comes from that preliminary statistical analysis. What I'm looking for in there is different signals of things that I've mentioned before, like reasoning capability. We call these tracked capabilities in Microsoft, um, OAI talks about it as their preparedness, but there are some frontier harms around true catastrophic outcomes from models that we care a ton about. Investigating. And I care a lot about the benchmarks that go into that decision. So that's just like a blanket, I really want to see the stats data before I take my very specialized resource and pull them towards pulling a thread. The product and system questions, I think, go back to something I mentioned around being really customer obsessed and understanding how those systems or products are going to be used and how they could create harm in those scenarios. So that's actually having a little bit of like a technologist hat and saying, okay, how could this be maligned or how could someone accidentally misuse this to a place where they could be harmed from it? And so instead of saying, hey, this is a model and I'm just going to do benchmarks and evals Of course, evals on systems and products are really meaningful to my team as well. But there's a different strategy conversation about a test plan for a product or system because we're actually asking, how is that model applied? What are the actions and tools and data and architecture questions that make that better, safer, more secure, or less so? So those are like the founding questions.

Sorry, can I ask you a follow-up on what you said now and also what you said before, because they happen to be in a question that came before I go to Leo, right? When you were talking about what you're testing, you were talking about language but also culture, right? And one of the questions that came is like, is there a reason why non-English languages, they come across as more harmful? Or is it just the lack of training of the model to detect the threat? So what is the role of language in defining that threat? Because I would imagine that that this is a connotation of human behavior, right, that is captured by the linguistic structure, or is that assumption correct, or just tell me a bit more about that.

Yeah, the baseline answer is that it's representation and training data and how people speak or how images— this is multimodal. Truly, this is not just language, but it's how humans have represented data in a way that's accessible that can be part of the training data of these models. This is super important in frontier models, right? So if you have an asymmetric amount of data on English versus French versus pick any language that folks are using across the globe, you're going to have a different reaction within the model. They have— it's less information to reason over. It also means is that cultural signals in that representative data might have outsized effect. And as models are trained to respond in kind, and with different modalities, so it's not just with different languages, but Lou also mentioned earlier, like the emotional timbre of inputs and outputs, that's part of the data as well. When you're thinking about all these different vectors of information that now are being pulled into models or systems or products, the amount of data that exists in training that pertains to that vector inherently will change the attack surface and allows us to manipulate it in different ways. And there's— if you have a ton of training data on English, for example, we're going to attack in a different way than, than a language like German or an audio input that doesn't have as much representation.

Yeah, which I would imagine, Tori, that that goes in terms of also social fabric, right? Demographics, right? Vulnerable groups or over or underrepresented groups. So the entire narrative about the sample that we learn from statistics now has to be taken into more account because of the expression of connotation that come with their expression of demands of any kind of prompts that are are culturally infused. So this is, to be honest, it makes me feel that we are really at the very beginning, right, of this big learning curve that is still ahead of us. So thank you for this. I'm happy we were able to merge a couple of questions together. And look, with respect to time, Leo, I'm going to ask you this as my kind of wrap-up question. So Angela is asking a question. You know, often we talk about, you know, building safer behavior. In the context of, of the safety of the models. But when we are optimizing that, right, when we are becoming, when we are having a safer system, is this safer for whom? Is this safer for the developer? Are we defining safer for all the users or the vulnerable users? Are we defining safer within a specific enterprise context? So imagine this is really a question that people say, well, it's safer according to which perspective? What would you say from the work you have done, Liu?

Wow, that's a good question. And while we work with experts, like domain experts on defining safety, and this question oftentimes came a lot because like sometimes it's confused, they confuse this to just safety for the user. But many times the model response could have negative implications to the broader society or to some specific vulnerable group. And if we just measure safety for the user, then these consequences could be ignored. So in our view, that the safety is a more broader concept. It's like the social, like, welfare function that we should take into account the user safety and the possible, like, negative consequences to other groups of people, like, as well as safety for the developer. Their platform. So sometimes we can see user trying to do something harmful or illegally, and the platform should refuse because it poses real risk to the developer that they facilitated in this malicious behavior. So I think it is manifold. Like, a lot of different stakeholders are present in the safety function and measures. As overall safety measure. Great.

Leo and Tori, look, we are at time, and I know there are more questions coming, and I apologize if I cannot address them all. For those who are part of the MicroOne Expert, you can always use the Slack channel to keep on engaging even afterwards. And, you know, the people that are coming are people that are always willing to engage. And Tori was very kind in sharing this on LinkedIn, and Tori will keep on doing this also. And I'm going to write a short article about what learn, and we're going to put the YouTube. But this is just like the kind of the operational side that follows from this. Look, I say this every single time, but the learning that I'm acquiring from this conversation is very genuine, right? I think as an academic, I learn more from this interaction than some time in academic meetings, which I think it gives away some of the shortcomings that the academic community sometimes have become too self-enclosed and not always able to understand the world as it currently unfolds. And by definition, the work that you're doing, Tori, at Microsoft and you, Liu, at MicroOne exposes you to the external world as the only learning environment that we really have to understand and at least attempt to understand. So I'm really always feeling very, very nimble in just witnessing just great thinking. And how fresh it is. Look, I am very thankful for having been part of this. I hope it was worth your time. I know that it's right across either the morning part for you, Tori, on the West Coast, or the kind of lunch break for you, Leo, on the East Coast. But I really appreciate the time you've taken to speak with me. And I look forward to this to be just one of the many conversations that we'll have. You know, we need to have more like-minded people. That are able to bring the conversation on AI where it belongs rather than trying to make a lot of effort to bring it back where it's not supposed to be, right? So thank you so much for your time. I really appreciate the opportunity to learn from you. And Tori, till next time, for sure. I'm sure it's just the very beginning of ongoing conversation we will be able to entertain. And Leo, for sure, until the next time, because it's a matter of time we'll have either a paper for another event together. So from Mark, thank you very much for being on the show and take care of yourself. And considering it's Friday, have a great weekend, guys. All the best. Cheers.

Enjoy.

Thank you so much.

Thank you.

Thank you so much. Thank you. Bye-bye.