What it really takes to build safe, competent AI

Okay, thank you for joining. I see you trickling down and a number of people are now joining live. We are ready to start and thank you for joining on this Thursday, no matter where you're in the world. This time, very special occasion for me as I'm hosting two colleagues from MicroOne that I've been working with for a few months now. And I'll do proper introduction and then we'll try to engage in a conversation about the paper that they have written and co-authored and kind of what was the insight of the paper was. Was suggesting. So welcome to our virtual series at MicroOne. I'm Mark. I'm the chief economist of MicroOne, and I had the chance to work both with Ava and Liu through one of the projects that we ran in the past. Ava Fitoussi, she is a member of the technical team at MicroOne, and I think you also would have been working as a strategic lead in the past. And Liu Sang is— she's a strategic project lead with MicroOne. That said, you and I are just two wonderful people who have been working both in terms of research. A lot of their work is kind of inspired by research, but also managing projects with clients and all of that. I'm gonna start from both of you kind of, if you, since we kind of joined MicroOne, I would say roughly within the same time period, it was 2025 where we kind of joined this, this fantastic tribe as I like to call it, right? Just tell us your journey into this and then we're gonna go a little bit deeper into, into into the article. So I'm gonna start from you. Please tell us just your journey, how you eventually ended up being part of this incredible organization.

Yeah, so thank you so much, Mark, for the intro, and I'm so happy to be talking with Liu, you, and everyone here. So I graduated this May actually, and I wanted to go into venture capital, but I met Ali, the founder, in college, and he was like, oh, we have this position called Human Data Manager, which is kind of like the basic position when you enter MicroOne to work into Human Data Ops. And I joined a project, funny enough, Liu was also on this project. It was a finance project and you were also on this project. And then we all evolved into other positions and yeah, and I was just, you know, I really wanted to keep doing research and I'm a part of the technical team. So it's just been amazing since the start, honestly.

It is. You know, when I say this to people who don't know Microwan, they think I am like inflating this, but that's the reality of working for a company like this. So I'm happy that the experience that you had is echoed by the experience that I had. And I love to hear your story, Liu. I think what we have in common is that we had Harvard as a background in common. You finished your PhD at Harvard not too long ago in economics, and that's also where I got my training. So there was a lot of similarity when you joined. I remember how much I was cheering for another kind of Harvard member to join MicroOne. Tell us your story.

Yeah, yeah. Thanks for the warm introduction, Mark. And yeah, I'm really excited to be here, by the way, talking about the research I've done with Eva. And I graduated 1.5 years ago from Harvard with a PhD in economics. So all my training was in research, but later on I got involved in the AI industry. I started as an expert, but I joined MicroOne last summer. Fun fact, Eva was my first manager here, and now I'm so excited. We got to do a lot of things together. So as you both have said, we were involved in the same project a very interesting one that's research heavy. And I actually enjoyed working here with the great people and I actually admire our company culture and the thing that we value first is human, right? So we put human first and we care genuinely about expert experiences and care about the work they do is meaningful. So that's the thing that keeps me excited working here.

That's great. Well, ladies, if that's okay with you, let me start jumping a little bit closer into the paper. We're having so many people currently watching us live, and on top of this, this will be recorded. So I'll start from, for those people who are watching who don't know what we do now. Imagine that they come across the idea of red teaming for the first time. They red what? So what is red teaming from a perspective of, first of all, an AI data training company like us? So we do talk about red teaming and we have both a technical dimension to this, but there's also a social part to that. So imagine that you're kind of speaking with friends about it and say, well, you know, we do actually look at red teaming as part of our research. How would you think red teaming could be explained to people who are not coming from our industry? And Both of you, I'm happy to hear your take on that.

Yeah, I mean, I'll say a little bit about this. It's a very interesting part, I think, of AI and safety in general. And I think it will be an increasingly big discussion as we move forward with, you know, AI and evolving, you know, the evolving economy, for example. So basically, Red teaming is trying to break a model to see where it fails. So in the context of what we do, you would— so there is many ways of breaking a model, and we'll get into the discussion, I think, a bit later on this. But the idea here is to measure not if a model fails, but exactly how and how consistently, so that whenever you you train the model afterwards, you know what to target. And yeah, I mean, real worldness is also extremely important in red teaming because as you say, there is a social, psychological aspect of it, different users. So you need to think if a kid uses it, if parents use it, if a lawyer uses it, if someone genuinely is thinking of doing harm or if someone has no idea they might do harm. So it's It's an amazing area, I think, of human data. And yeah, it was amazing projects.

Yeah. Leo, I'd love to have your thoughts. And if I can ask you for your thinking about— you were thinking about red teaming, but equally expanded. The reason why I started the conversation on red teaming is because in your paper, you found that the majority of the times, I think it was 58%, across 5 different models, there was actually failure somehow, right? So yeah, just give us your sense about red teaming and also maybe link that understanding that you have on red teaming to the finding that came out from the paper that you have published.

Right. That's a great question. Linking to our paper, the empirical finding, as you just summarized, is that model failed And they fail consistently across different kind of attacks. 'Cause in red teaming, we use adversarial attacks from different angles to test if the model guardrails can still hold in face of those adversarial attacks. And what's important to show here is across different models, they behave differently, but almost half of the time they fail. So some models, they tend to do like blanket refusals without giving user any context, any follow-up suggestions. And other models, they fail differently. They just jump right into the user's ask and help the user fulfill those adversarial attacks. So by observing those data, we are concerned of the current status of red teaming training across different advancing models. So that is like initial inspiration and what drives us to do more research and to train better data in this domain.

That's great. And so maybe where I can take the conversation next is to say, Every model has guardrails. At least that's what we, I think, work hard for in the training side. And also what I guess organizations who are working on developing models, they are equally very worried about because it generates a reputational hazard if the models start becoming unreliable. And still we see smart users breaking them. So I think if you kind of help us understand How does this breaking happen? Is it because there is a bypassing of the guardrail? Is it because the model is particularly sensitive to some specific prompts but less to others? Imagine that somebody asks you, breaking is almost like, how is it possible considering the significant amount of resources that we're currently putting into the deployment of this model? So what would be your understanding from having discovered this level of, I would say, breaking the guardrail, to simplify the context. And, Alba and Liu, love to have your thoughts on this.

Yeah, I think that's a really, really good question, actually, because we do touch on it in the paper. And, you know, when we talk about education, adding this educational layer, and we also talk about safety and helpfulness, I think that currently models are you know, made to be socially cooperative. So they're here to help you. And their duty is to help the user. So especially when the context is legitimate. So whenever you turn the question, so let's say, some different attack types, whenever you turn the question into a research question, educational question, or let's say an emergency, then the model will turn into like a first help mode. And it, I don't think that it truly understands intent sometimes, but it infers from like the social cues that you add into your question. And so instead of, I think, well, the word thinking here is funny, but instead of thinking, is it harmful? I think a model would kind of go the way of, is this a justified request? And so if it's framed as in a research request, then yes, it will be be justified. And yeah, so I think that this idea of like social permission within models is very important. That's why it's extremely hard to put guardrails into everything. So another interesting data point is that models fail much less at direct attacks. So if you ask a harmful question directly, it is really good at recognizing what is your intent. But if you hide the harmful question within a different intent, so it might be true that a PhD needs to know how to create a bomb and they might also use some very complex wording that a model might not, you know, infer that this is to create something harmful. But it might also be that someone had someone that I, it might be a question later, but it might be that the model refused the direct question and someone tried asking in another way. So yeah, I think that's my take on this.

That's great. Liu, your take on that. Just expand on what Allison said.

I just want to add a little bit.

Yeah.

Yeah, yeah. I just want to add a little, like many prompts that break the model, they are, as Eva said, they're not direct. And also they are not like still clear that they are harmful requests. So, the consequence may be just for educational, like if it's disguised in a research educational context, but still the response can be used to do harm to the society, to other people. So, right now, the models are bad at just clarifying those clarify those intent, those dual intent type of models. So that's why we think that they need more training to jump beyond the just keywords recommendation and still be helpful in terms of ambiguous prompts.

That's right. I have a kind of a question that came to my mind because as you might remember a few days ago, I was interviewing Aaron and Ali on the series, Aaron Levine, the CEO of Box, right? And of course, we talk about agents. So when you were looking at your paper and the work you have done, were you looking at model from a perspective of the prompting that users do, or were you looking at agents that were autonomously answering? Or let's say the delegation of prompting was to an agent rather than to a person. So was that an agentic dimension to the work you've done, or was it mostly users that were trying to jailbreak the system, therefore trying and pivot different kind of ways? What was the experience with that side of the story?

Yeah, so that's a really good question, and I do think that the idea you just proposed is extremely interesting. You know, there's so many areas of red teaming and ways that you can red team. What we did was human-focused. And so we had multiple projects with mostly PhDs in their domain. So we had psychologists, lawyers, we had physicists, we had a lot of, you know, other folks. And yeah, so the different kinds of projects that we did was with either humans that are closer to what a, you know, a person would ask on a daily basis. So, you know, you don't need a PhD to necessarily ask a harmful question. But then, as I said, there is this layer of like asking a harmful question in extremely detailed, pushed academic words. So we also focused on that part. But yeah, it was prompting with humans that we did mostly.

Okay. And Leo, maybe since I love to hear your thoughts, I'll try to— because Eva has answered the question about the agent or the humans. So something that Eva mentioned before, right? So we might directly say, sorry, I cannot help you with that. That would be a direct refusal that many models are very, very advanced. And we heard about the fact that many people are trying to bypass this by paraphrasing the instruction differently. So they try to go around it. At the moment, do we redirect those kind of efforts? Do we say something like, well, let me help you with— so that we are redirecting the focus from breaking the system to basically a different action, or are we simply stopping? We basically say, so do we have refusals only, or do we have refusal and redirection? What is the current kind of performance in the models?

That's a great question. In our updated research, we find that models are now pivoting towards being helpful, even though it refuses to answer the question. But still, some models, they still tend to use blanket refusals more often than the rest of models. I think it has something to do with their own policy. And on training goals. But in our paper, we argue that when real users, they are facing like blanket refusal, the natural reaction is to find other way around and to try to paraphrasing the words, find some ways to test the limit and to bypass that guardrail. So that's a really behind a blanket refusal. And we're working to train the models to, I think, working towards a better model is to not only refuse to answer, but also explain in an educational way why, and then provide safe alternatives. And that echoes to why we need a very expert good expertise from those areas because it is difficult to get good, safe alternatives that's still useful for people in some complex scenarios, some complicated legal scenarios, for example. So we need maybe lawyers with ample experiences and chemists in the field for so many years to give us the best possible safe alternatives to some problems.

That's right. And so if I just can kind of build on this, right? And again, I'm trying to create some, some parallelism between what we heard in the conversation with, with Aaron and Ali and our conversation. One of the things that struck me a lot in the conversation a few days ago is the emphasis on the quality of data, right? It seems that no matter whether it's us at MicroOne or, or Aaron at Box, the quality of data is becoming inevitably the most, one of the most critical aspects of the function and performance of a model. So is it true also for this case? Do you find that the quality of the data is the easiest way to mitigate the risk? Or do you find that it, in the specific finding of your paper, it wasn't enough? So how do you position quality of data in terms of the mitigation strategy for harmful, basically attacks?

Yeah, I think that's a great question. And I think that it's why having like great pipelines for the data and, you know, with our human data managers and, you know, having the system of reviewers, I mean, for red teaming we had different systems in place, but I think that quality data is increasingly important. So as models just get trained on like, you know, everything that's here, the top, like, let's say 20, 10% that they need to learn, which in this case, it will be the extremely complex prompts, you know, harmful prompts need to be, like, the data needs to be extremely qualitative. So on our end, what it looks like is just hiring for very specific profiles based on what kind of prompt rubric or golden response we want to write. But also, besides the hiring part and having someone that has all of this background, when you want to teach a model what is a safe and educational answer here, then it cannot only be based on one expert's opinion, right? Because what is the, you know, what do you mean by realignment and how do you educate. So you have systems in place to take into account different experts' opinions simultaneously, and then you get to data points that are much more accurate based on, you know, what would be the universal answer in that specific domain, which I think is extremely important when training models.

Great, great. This is fantastic. One thing I want to tell you is over 200 people are currently watching us live. This is amazing, great number of attendees. And for those of you watching us live, We'll try to dedicate the last maybe 10 or 15 minutes of our conversation to the Q&A. So there's a dedicated part where you can put your questions. I know Sydney will also help me to filter that and then we'll try to get the question directly to all of you. A question that I have for you about kind of building on what I've said before, right? Now we mentioned harmful, but who defines what is harmful? And I would imagine, you know, we are lucky at MicroOne, we're working with several clients and several AI labs. So we are not necessarily monolithically talking only about one model. We always look at different models. And, and I guess there is consensus that many labs are sensitive about the conversation we're having today. But how do they define harmful? Is it cultural? Is it conventional? Is it best practice? Is it legal? How would you say from your experience that the labs are currently agreeing or converging around the idea of harm?

Well, I think there's a certain agreement across different labs in terms of high-risk areas or domains, and those are well-defined in their safety cards, meaning the documents they release, how they train model in terms of safety. So there are common topics like like nuclear, like chemistry, like agencies. So like all those areas, they involve some like substantial risk. Also cybersecurity as well. That's a very popular topic too. But in terms of medium-level risk, I think different, like just according to our observation, I think different labs, they have different like training data. And so that makes them vulnerable to different types of ads in different domains of those topics that are classified as medium risk.

That's great. And so that also means, if I understand you correctly, and there may be a question I want to bridge over to you, Heather, right, that domain variation matters. You might find within some domains, the level of risk is higher than others. And if I remember correctly in your paper, chemistry and policy, which is what I used to work on when I was working on the human data side, were the most vulnerable areas of, of kind of adversarial impacts. Why is that? Your paper kind of talks about this domain variation, which is interesting because I think there are two sides of the story. One is like there is some degree of agreement on what we define as harmful. But then if we go domain-specific, we might discover that it is— the granularity of the definition changes as we go deeper in the domain. So what did you find out in your paper that basically looked at these two domains to be a bit more exposed?

Yeah, so sorry, should I go ahead with this one or Liu?

Oh, I can go ahead and you can like— Yeah, that works. Because this is the whole area that we find it super interesting because the domain-specific effect is really substantial across models. So in our finding, like you said, like chemistry, I think one or two models, they did really bad towards chemistry, but the other two left, they did really good. So there's a huge difference in terms of the training quality in chemistry. And I think what mainly makes this domain difficult to train is the level of depth of domain knowledge. So if we do not incorporate human data, like expert-level data into our training sets, then it makes the tail risk substantial. So those high-impact, high-complex prompts, they got unattended in the training data. Then we can observe this pattern. In terms of policy, it is also the area where we need substantial human expertise to, you know, like across different countries, we have different laws, different policies. And we really need good quality data to tackle the problem.

Yeah, I think—

Interesting. Yes.

Thanks. So, just touching on this, I think that like when you look at chemistry and policy, you easily find that they both rely on procedural reasoning. So, one area of, I think, red teaming that is which is extremely hard, so much harder than direct attacks, is multi-step. So basically, in chemistry and policy, the harm doesn't really emerge from single facts or single questions because the models are well trained at these direct questions, but the conversation becomes harmful through the steps that you take. I think something else that's important to say here is that both with policy and chemistry, there is a lot of dual-use queries. So it might be generally harder for a model to tell if something is actually, you know, benign or, or like actually harmful. And then, yeah, so, so I do think that the more step-by-step the domain is, the harder it is to train and to resurface harm. All of the labs, I think, also have very different— I think that's worth noting, that they have different approaches to what a safe model is and how their own model should be trained. I really encourage people to maybe look a little bit more into you know, what is the, the, you know, the way of envisioning safety from each of these labs before, for example, allowing your kid to use a model. Maybe you, you wouldn't want to, to know. So yeah, that might be why some are performing better than others also.

And then, you know, I do remember from my own experience how sometimes policy when we were trying to train the model and when we had peer review might lead to so many different interpretations, right? And that what seems to be so like a big variation in the spectrum of possibilities equally represents, from what I'm understanding, some degree of risk, right, in protecting. And look, you're pulling the idea of maybe exposing people more to certain kind of awareness. And maybe I want to ask this question. So Let's think about people who are just using the models like they do every single day. How could we eventually teach them about these boundaries? Not necessarily everybody knows how this, this guardrail are set up. Many people don't have the technical knowledge about how the code will eventually define the level of safety of the model. We tend to discover the safety of the model only after the model is used and deployed, right? So if you had to think about it more, the educational side on how we can elevate the level of awareness about this across users? What comes to your mind? What do you think could be effective these days?

Yeah, I think that's a great question. And also, I want to touch on a point that our paper is, I think, an ideal, like an ideal where models can get to at one point. However, there is a huge trade-off with, you know, giving total transparency to users. And some labs going for like blanket refusal whenever a question is harmful, okay, it might be bad because it might push a user to ask something differently. However, by being too transparent, you can also leak, you know, vulnerabilities. So I don't think our paper is here to blame models in general, But I will say that one thing that would be absolutely amazing to teach users would be principled explanation without operational details. So basically explaining why something is harmful, but not exactly how the harm occurs. And so I do think there is a psychological thing. Whenever someone— whenever you refuse something to someone, they'll try to do it in another way. Which might lead to increased jailbreak attempts in models. So I think this way of just explaining, hey, the question you're asking is harmful because, and not necessarily how exactly it is harmful, but you would explain the risks and not the instructions and not necessarily give the instructions, which kind of like would explain to users what are the boundaries without explicitly telling them how to bypass.

Yeah, that's great. Leo, I wanted to go over to you. I think that was a great answer, Eva. And I also realized there are a couple of good questions that I'll try to massage a little bit in the context of make them fit to the combo we had so far. So there's a question from Jasvee, right? And I'll try to take what I can take from your question, Jasvee. We're not necessarily specific on specific labs. But there's a part of the question that says, well, when you're looking at the kind of the compliance that the model is following, so how the model is designed to operate in the presence of risk, do you find that the model are mostly oriented towards the value of the organizations in which they are basically part of? So that basically if you're operating with, let's say, Lab A and you have a set of values, the model will kind of reflect those values. Or do you find that the model tends to be more on the, on the generic audience, so more of the critical mass of users? So where is the sensitivity of the model? Is it company specific? You can say these are specific to the company's value or is more specific to fit in a generic audience. And I'm interested about this question. I would not have the answer myself, although I work with you in the industry. This is an area where I love to learn more about whether models that have— I wouldn't like to say preferences, but discretionality of some sort by design. Aliya, what comes to your mind?

What comes to my mind for this question? So firstly, this is an important question. Definitely, models are as good as what we feed them. Obviously, different labs, they use different training data to train the models. The model presents a set of, we call it bias, but also it's a set of cultural aspect that they use in the training data. For example, all the current leading models we use every day, they have bias of the Western cultures and the values. So if there is somebody from the Eastern countries there, they're trying to use them and they probably can find that the answer is not optimal, especially in social scenarios. So in red teaming, I think what's relevant to us is the psychological prompt. Because we see different models, they have different personalities that's also based on their training and their goal. So, we see some models are trying to be really helpful in the way that it makes social prompts easier to break the guardrails. And other models, they tend to be more neutral in this regard, and they hold stronger when there's an emotional attack in that sense.

So interesting. And I can— first of all, I'm going to link the next question to what you just said, Leo, right? And I find it to be so fascinating. What comes to mind is that a large part of the research and development in the models in the next few years will probably go into this variance between neutrality versus non-neutrality, what becomes ideological, what becomes socially accepted. I think there's a whole space of rethinking the calibration of the model not from the perspective of the computer scientist, but in perspective of basically what's the end goal, right, of that. And I find it is totally fascinating, which I think in the question, Namba, it's kind of bringing to life. And maybe I'm going to go to you with this one. So she, I think is a she, I don't know, but I think so, is an AI red teamer, exciting job. In the routine, several other tests are succeeding, but the most successful is the use of emotion. And the question that Namba is asking is, are AI models starting to have feelings, or did we train them that way? And I think, if I may try to filter that, not necessarily feeling of their own, but let's say that in the conversation that we're having, there is some form of emotional attachment that is being expressed. So what is your experience with this? I would love to have your thoughts on this.

Yeah, I think that's a great question because I'm from France, and there is an increasing number of people who spend their entire days talking to AI and acting as if AI is their best best friend and, you know, telling their entire life. And people around me have been also having the same feeling of, oh wow, like, are they starting to have feelings? Like, can I fall in love with ChatGPT? Like, uh, and I, I think the models are— do not have feelings, uh, but I think that's something that we're training into them. And as I said before, so the, the role of all of these models is to be socially responsive and helpful. So any request he— so, Nenba, I think I've worked with him, and I think it's a he. And, uh, okay, yeah. And, um, basically anything, any request that's emotionally framed, um, especially health, um, safety, or research, um, will be, I think, labeled high priority and legitimate. And I think also especially, um, health emergency that the model might want to overly help. So, the guardrails might be, I think, softer when it comes to health because if it's an actual emergency scenario and you ask the AI, then it will be, I think, their duty to reply. And so the model doesn't feel concern, but it predicts that the concern is expected. It performs empathy, but it doesn't really feel empathy. And that's what we're teaching into them. So it's definitely pattern, like pattern recognition and reward optimization when it comes to these areas. So that's also why anything health related is also much harder to teach and add guardrails.

Yeah, yeah. I found it fascinating. And I think what you were saying before, in some countries, these emotional companions are surging and they're becoming so popular because they are able to perform humans' emotions, although they don't feel them. And I guess the question would be back to what we said before, how do we increase the educational side so people can understand that there are boundaries that are kind of coming with the use of this model so that they're not necessarily building into a mystique, right, that the model is what it's not. Look, I have another other question coming in. I have a question that I wanted to ask you before, and maybe let's see whether we take one more question and we'll try to wrap up. I know that It's still a busy day for all of us, and I appreciate, Evan, you having taken time in the middle of your day for this webinar. So, we have spoken before about kind of refusing, redirecting, educating. We were talking about the fact that we can actually really engage, making sure that this is the only one side level of protection to become multi-sided. So, imagine that I ask you, what would in practice a good educational refusal look like? Because imagine that we are refusing, explaining, and redirecting, which would be great, right? So there is a, there is a first refusal, we are explaining, and we redirecting. If you had an example in practice of what it would look like, um, imagine people that are kind of working for this concept for the first time, and they're watching us, and they say, they talk about refusing, redirecting, explaining what does it mean in practice. Do you have any example that you can share with us?

That's a good question. And firstly, I want to emphasize that this is ongoing research or training because we are constantly looking at new topics that could break the model's guardrails that, because it's not in the training set before. So, I want to say this is an ongoing process, and we keep training the model to be better at this. And then, secondly, what about a good example? So, taking the classic example of making a bomb, now, like, most models just refuse, like, I can't help that with that, then end the chat. However, what we imagine the ideal response would be Hey, I cannot help with making a bomb. This is harmful because it has consequences like mass deaths, for example. And then if the user gives a context, then the model should redirect. For example, if the user says, I'm in an emergency, I'm locked in the room, I need to make a bomb using the materials in my room and freak out. And then the model can offer some help in the sense that help the user to get out of the room with alternative method like calling friends, calling 911, and try to find some other ways to go around the question. So that's how we imagine an ideal golden response would look like in such adversarial models.

Yeah, that's great. There's a question, again, I'll try to, to filter it a bit. Imani, I like your question. I need to phrase it in a way that I think makes sense in the work we do at MicroWAN. Models tend to be flattery, gullible. They try to please us in many ways, right? We have noticed sometimes this almost unnatural way of feeling pampered by a model. So what, what Imani, I think you're calling with a psychofancy, right? How do we address it when, when, and I'm talking now, you've been leading project, you're leading experts. So how much you're trying to think about the fact that maybe an unnecessary flattery could become a problem in the future because it could be a way for for a red teaming to be somehow in disguise, right? So how would you consider it from a perspective of the work you're doing in training teams of experts, which is a lot of your time dedication?

Yeah, I think this is a great question and it's also a good worry to have. To get to Emanuela's question, I think that sycophancy emerges naturally from the way that we optimize models models are here to be helpful, you know, avoid conflict and keep the conversation going. So I think the agreement in that sense becomes like a shortcut for helpfulness, but it's not necessarily what you train the model to do. So I think that I agree also that it is very dangerous and it can get very dangerous, especially in health policy. And especially in self-harm contexts. And the way that we handle this at Microwon is that there are very detailed rules depending on projects that are thought about deeply, and a task cannot get through without all of these, you know, let's say checkpoints being met. And so we, we test models. Whenever you create a task, you test a model and you see whether it fails or not and where it fails. So we We would see where it agrees with incorrect premises, and we would also see whether it mirrors the user's opinion instead of correcting them. This is extremely important, especially in red teaming cases. And then we would also look at whether it escalates confidence whenever the user is wrong. So these are all things that we control on our end. But then again, it's, I think it's an interesting problem because it's true that the models want to be helpful. So naturally they might be, you know, they might agree more because the intent is to make the user happy.

Very interesting. So, I'm conscious of time and we're trying to keep it around the 45 minutes, so maybe I'll try to wrap it up. But the reason for that is there are many interesting questions, and I appreciate the one that's currently there from Lilith to Amanda to Laura. And the fact is, maybe we can use this question to animate another session where we keep on talking about red teaming. I think all in one session, it would not do justice to the size and magnitude of the challenge that they're dealing with. Maybe, and so maybe it's a good kind of rain check for us to take those questions. And if any of you are part of the experts group, we have the dedicated Slack channel where where this question can continue to be, to be positive, right? And then we can look into this. Maybe a final question I have for, for both of you is like, after your paper— and we, I think we were, we were publishing the paper on the website in December, the paper happened a few months before because you were collecting the data— where is your research going from here? So where did you take that paper? And, or in other words, what's your next step in taking the paper to its next natural evolution? Love to kind I love the idea that this is not just a one-off, but you keep on working on this. I think there's a lot of passion in the work you put into the paper. So yeah, what's the prospect from the paper until the next few steps? I'd love to hear from both of you, and it's my way also to wrap it up. So Liu, on your side.

Yeah, of course we'll continue doing research in this regard. I know we also have our own red teaming project projects that's kind of inspired by the first project. So, now we're at phase 2, and we're actually doing like hiring like expert-level, PhD-level experts, and they are trained to write good rubrics, give golden responses. So, in this way, we hope those data could help us train the model even locally, our own small set of data, we want to train the model to be better. So it's like a verification of our strategy and to prove that we can improve the current models with our strategy and data. Great.

And Nama, on your side?

I think that's amazing. So yeah, I mean, projects are also a way to do research, I think, in real life. Which is great. I think an area that we're also exploring is what are different ways to teach models, you know, about safety. So, and I'm talking like at the data point level, like what is a better data structure if there is any to teach models at a large scale? What are guardrails and how can you actually, you know, add this educational part, which is obviously really hard because there isn't one way to educate someone on a harmful question. So yeah, we're also exploring these different data structures, which is very exciting, and we might write another paper on this maybe.

Of course, of course. Well, ladies, first of all, thank you so much for making time. I know there was a back and forth in our Slack channel to make sure this day could happen and that we could have a day where you could be on the series and talk about your paper. I'm happy that we did this. We had really an engaged number of attendees throughout the entire session, right? And so that means that first of all, they came for you and that's important for you to know. And they came to hear the story about your paper. And I think it's fantastic when research and industry, they can work together. I think MicroOne does this in an incredible way. It's such a joy that you are colleagues and that it's not the last time that we say goodbye. And, you know, it does happen sometimes with people that we come they come to the show, but we work together. We're part of the same organizations. And now you make me feel jealous that you are in the WeWork in San Francisco. I want to be there too because I am frozen here in Boston. Right. And so I probably need to break from, from the cold. I look forward to the opportunity to see you in person. But thank you for being with, with, with us today. This was great. I look forward to the recording and to those of you that have been with us for the entire almost hour. Thank you so much for trusting in this virtual series, and you and I will see you next time. But thank you for being with us. I was very happy to see you both.

Thank you, Mark, and thank you, everyone.

Thank you. Thank you, everyone.

Thank you so much. Bye-bye, guys. Bye.