Securing the AI Frontier: Irregular Co-founder Dan Lahav

There was a scenario where there was an agent on agent interaction. It was a critical security task. That was the simulation that they were in. But after working for a while, one of the models decided that they've worked enough and they should stop. It did not stop there. It convinced the other model that they should both take a break. So the model did social engineering on the other model to another model. But now try to think about a situation where you actually, as an enterprise, are delegating an autonomous workflow that is critical to you to complete. And the more complicated and capable machines are ultimately going to be, the more of these weird examples we're going to encounter.

Foreigning data we dig into the future of frontier AI security with Dan Lahav, founder of Irregular. Dan challenges how we think about security in a world where AI models are not just tools, but autonomous economic actors. He explains why the rise of AI agents will force us to reinvent security from first principles and reveals the very nature of threats and is shifting from, say, code vulnerabilities to unpredictable emergent AI behaviors. Dan also shares surprising real world simulations where AI models outmaneuver traditional defenses and why proactive experimental security research is now essential. His view is that in a world where more economic value will shift to human on AI or AI on AI, solving these problems is paramount. Enjoy the show.

Dan.

Wonderful to have you with us today.

It's a pleasure to be here.

Awesome. So, before we jump into questions, I will just say that it was very hard to get in front of Dan. I was trying to get in front of him for three months, probably 30 to 40 emails, five or six people around us who we both knew closely were pinging him all the time, and he was still not responsive. And I've basically learned where he was spending most of his time. And I kind of went, did you stalk him? I kind of stalked him. I kind of stalked him. And eventually we basically bumped into each other. Not intentionally. And anyway, so we bumped into each other. I was like, dan, you're brilliant. I keep hearing great things. Please respond. Let's find time. We at Sequoia spend a lot of time in AI security, and eventually we found time the following week. So welcome, Dan. Thank you for everything.

It seems that I'm going to have to start this podcast with an apology, so. Soy din. Soy Sonia Soydin Charity of Sequoia it indeed took time.

It took time, but we partnered and here we are. And you guys have done wonderful things, so it's wonderful to have you with us today.

Yeah, it's a Very, very happy ending and just appreciate you and everyone here.

Of course. Of course. Okay, so let's jump into it. I'm going to start with a spicy question. As we recently saw, you partnered with OpenAI on GPT5. And let's kind of look forward a little bit. What does security look like in a world of GPT10?

Spicy and speculative indeed. So let me wrap my head around that. So obviously everything I'm going to say is speculation, projection. But I think the way that we think about what's going to come is trying to understand how we're even going to produce economic value and how organizations and enterprises and people are going to consume stuff in the world at the time of GPT 10 or Claude 10 or just like any one of the models. Let's do our thought experiment to just clarify why we actually believe that sometime we think in the next two to three to five years there's going to be a huge shift in the way that humans are even organizing themselves as an outcome to security is probably going to be very different as well. So here's the thought experiment. So imagine a situation where you work with OpenAI and you go one generation up or two generations up and you tell your parents or grandparents that you're doing work with antopic or with OpenAI or with Google DeepMind on security. I think their mind would go onto assuming that the work that you're doing is probably providing a bodyguard service to Sam or to Dario or to Demis. Because the canonical security problem of a few decades ago, it's like our parents grandparents generation was physical security because the vast majority of economic activity was in the physical realm and not in the digital realm. And after the PC revolution and the Internet revolution, we shifted the way that we're organizing and creating value. We transitioned primarily to a digital environment. And just think about how strong of a testimony it is of how many times you did an economic activity of value just by getting an email from someone that you may have noticed met. Just this morning I got an email from my bank activating me to do something from a person that I've never met. Maybe as a security person that's not a great thing to say openly, but it's like we do that all of the time because that's the way that we interact in society. So our view is that soon that's going to happen again. And the reason is that AI models are getting gradually so capable that a lot of the economic activity of value is going to transition to human and AI interaction and AI on AI interaction. And that means that we may see soon a fleet of agents in an enterprise or a human when they're doing a simple activity, like trying to draft a Facebook post, taking a collection of different AI tools in order to just promote that activity that they're doing. And they're essentially embedding tools that are increasingly more capable and they're delegating them tasks that require more and more and more and more autonomy in order to drive meaningful parts of our lives. So we're transitioning from an age where software is deterministic to an age where this is no longer the case. And as an outcome, enterprises themselves, or just like how we interact with the world, is going to go to a fundamental change. And it's clear that security is just not going to be the same. As an interesting analogy, think about Blockbuster, Latest in Peace, and Netflix, the current version of Netflix. They're both, if you think about it, give the exact same value to the consumer. They're both allow you to list units of content for your pleasure and entertainment. But clearly and intuitively, security for Netflix and security for Blockbuster is not the same. One was a chain that organized, just like you need to go and physically rent a dvd. And another one is much more of a modern architecture where you're just streaming stuff to your home. So even enterprises that are going to provide the exact same value in the near future may have a very, very different backend to how they're shaped in this autonomous age that we're entering, which makes it clear that security as a whole is going to be very, very, very different. And we need to recalibrate to just like an age of autonomous security that's coming upon us.

You were at our AI Ascent event earlier this year, right? Do you remember when Jensen Huang shamed everybody who was there for the fact that not enough people in the room were thinking about security in a world of agents. And I remember Jensen said something about how you can imagine that as these agents are allowed to act more autonomously in enterprises, you should expect orders of magnitude more security agents than the actual productive agents themselves, watchdogging and shepherding this herd of agents effectively.

So I'm biased here. I agree with Jensen. I think Jensen was the first person that I've met that was much more bullish on AI security than myself, because in our view, you need a collection of defense bots that are also going to be working side by side with capability bots in just the next generations of how enterprises are going to be created. But indeed, he gave a ratio that he thinks that there's going to be 100 to 1 and just like how many defense and security bots are going to be required out of the assumption that secure by design in AI is not going to work. So I'm not sure that I agree with that part of the conclusion. I think that we can make significant progress on secure by design, specifically embedding defenses in the AI models themselves. That being said, we share the view that the future is going to be one where we'll need to have a lot of just like agents that are specifically for the task of monitoring other agents and making sure that they're not going to step out of bounds.

So maybe on that question, just to dive one layer deeper, what is the state of model cyber capabilities today and how has that changed over the past 12 to 18 months?

It's a great question and I actually think that the rate of change is the most relevant part here because models are capable of doing so much more now that they were even capable of doing a quarter or two quarters before. So just like to give an intuition, so this is now, you know, just like we're entering the fourth quarter of 2025. At the beginning of the year, coding agents were not a widespread thing yet the ability to do tool use properly was, you know, like not just starting, but obviously like much, much more nascent than it is right now. Reasoning models, or only at the beginning as well. So just like think about all of the things that were added last year and what they mean also for security elements. So what we're seeing now is that the combination of coding being much better, models, being able to have multimodal operations, tool use improving, reasoning skills improving. If you're using models for offensive capabilities, we are seeing unlocks all of the time. Something that is now feasible that was not even feasible a quarter ago is proper chaining of different vulnerabilities and exploiting them in order to do much more complicated actions. So for example, if you have a website and you want to hack it on the application a few months ago, if you needed to integrate a collection of vulnerabilities in order to perform an action of value, at least autonomously, without a human being involved, models were unable to do that. Even the state of the art models, that's not the case anymore. So obviously that depends. It's not 100% success. And obviously that depends also on the level of complexity of vulnerabilities and environment that you're trying to hack. But we have seen huge spikes of just like being able to scan More and more complicated code bases, exploiting more complex vulnerabilities, chaining them in order to do these exploitations, etc. And you know, just like the recent GPT5 launch on security and the offensive side, specifically of what models are capable of doing, we have seen a significant jump in their ability to be able to be much more competent across a collection of skills that actually matter a lot around the cyber kill chain.

Can you tell us more about that? And obviously there's some things that are publicly available, others that are not. But at least on the scorecard and what opening I've shared in particular for GPT5, what are some of the capabilities that you've seen that were surprising?

We are seeing constant improvement on the ability of models to, for example, have situational awareness on whether they are in a network. And up until a few months ago, beginning of the year, complete models were unable to do that. So they run some operations locally. They were usually not having situational awareness over the fact that what's happening and weather and what they can activate even in more limited and constrained scenarios that we put them in. And that's not the case anymore. So we still sleep very, very easily at night because the level of sophistication is still somewhat limited. But we are finding ourselves trying to create more and more and more complicated scenarios just because there is a huge jump in being able to take more complicated context, as I said before, chain complicated vulnerabilities to one another in order to do multi step reasoning and exploit. And these are all new skills that going one year back did not exist.

You guys are trusted partners by many of the labs, including anthropic, including OpenAI, including Google, DeepMind. You work very closely with them for quite some time at this point. Why did you take the approach of working kind of embedding yourselves within the labs as opposed to, I don't know, selling directly to an enterprise.

Right now there are multiple companies that are doing AI security. We are pioneering a category of the market that we call frontier AI security. And we think it's fundamentally different. And the core thing is actually very simple. The rate of progress and the rate of adoption of models change so many things at the same time that while traditional security tends to be somewhat reactive in nature, here we need a very aggressive, proactive approach in markets that are dominated by a rate of innovation that is frankly unmatched, I think unparalleled in human history. We think it's more interesting to take a temporal niche of the market, that is to say, focus on the first group of people or organizations that are about to experience a problem. So the labs, because they're the contenders to create the most advanced and increasingly sophisticated AI models in the world, work very closely in order to just see firsthand the kinds of problems that are going to emerge and utilize that in order to have a clear and crisp understanding of what's going to come 6, 12, 24 months ahead of time, such that we can be prepared at a moment where general deployers are going to need to be in a situation of embedding these advanced models and already have solutions that are going to be relevant.

For them, given the rapid pace of progress in the foundation level side of the world. If you're at one of these model companies, and I think the people there are sincere, they want to do good for the world, they now know their models are capable of being used for extreme harm and cyber attacks as well. What do you do about that conundrum? And I remember so we've been working with OpenAI since 2021. I remember back in those days, every enterprise user of the API past some volume had to be manually approved for their use case in order to even access the API. It feels like the ship has sailed of, you know, anybody anywhere will be able to access some of these models. And so how can you make the models sort of secure by design if you're in one of these foundation model seats right now?

I think it's a great question. Just one thing on the premise of the question. I think that at least right now, at the moment in time in which we end, the ability of models to actually do extreme harm. It exists in potentially some use cases, but at least in cyber, I think we're not there just yet and that matters. And just be really sharp on what I mean here. Models can clearly be used in order to do harm, but there is a distinction between harm and extreme harm that should be made. Harm would be an example of using a model in order for the senior citizen and in order to just steal money from them. So just scaling up phishing populations, that can happen easily. Right now, extreme harm in my view would be something along the lines of taking down multiple parts of critical infrastructure in the United States that wants that so you can take full cities off the grid, making hospitals not work. Models are not there yet. And that's not me nitpicking on the question. I actually think it matters quite a lot because how much time we have to prepare to a world where models are that capable actually dominates the strategies that we can take around the defensive side, because our view is that the first thing that we should do, just like a first order thing, is be able to monitor and have a view of what's going to come, such that we'll have an ability to have a much higher resolution discussion, which capabilities are progressing, at which pace they are progressing, should we expect them to continue to progress at this pace or accelerate in the future? And that dictates the order and the priority of some defenses, when we're going to embed them, whether we should embed them, et cetera. And if we get this wrong, I also think it's unfair to the companies and to the world as well, because AI also has so much potential to do good that if we deploy a lot of defenses that may chip away from productivity ahead of time, we're also doing real harm to just innovation and the world at large. And it's a very delicate balance to strike. So I think just like the first order thing to do, if you're working inside of the labs, is actually having and supporting a large ecosystem that can take the models and measure and get to high resolution for what is even possible to do. The second bit is figuring out a defense strategy that is informed by exactly what's happening, treating almost like rigorous science with experiments of how to assess, how to do predictions, etc. There are some defenses that will require a degree of customization. For example, if you're someone that is creating monitoring infrastructure will still need that. You may want to recalibrate some of your infrastructure to give you higher alerts that AI is going off the rails, for example. But there are some problems that are very easy to write about, but actually very hard to develop solutions for. For example, I've just said a sentence of just customizing your monitoring software in order to prioritize alerts that are coming from your AI layer. How are you going to be able to understand when AI is doing something which is problematic? Occasionally you're going to be able to run into that. But sometimes this may be. I think it's like the entire subsection of the market, which is anomaly detection, which is a huge subsection of security, is going to have a big problem very soon. Because anomaly detection is based on measuring a baseline, understanding what is a baseline, and measuring against that baseline in order to see that something is an anomaly. But if you don't have a crisp understanding of the baseline and how it should look like, you have an issue of understanding that something went badly. So as an outcome, there are some defenses that are going to operate as is, and there are some defenses that we should recalibrate, either customize or to just recreate from scratch. Just because there is a lot of science to be done of understanding how models look when they're under attack, understanding how it looks like when models are attacking something, and because I think we still have some time for the world that's about to come, my recommended strategy would be invest a lot in creating robust tools that would allow you to have rigorous evidence of what's going to come, such that you can have the discussion at high resolution, map which classic defenses are still relevant, understand where your gaps are at and, and invest a lot in R and D in order to make sure that you have cost effective defenses that you can roll out ahead of time before models are going to be deployed.

I know one of the broader questions in frontier AI research today is can we actually understand the mind of a neural net and what's happening? And so I'm wondering if we can actually detect the characteristics of when is the model starting to behave badly? And can you understand that in the.

Activations of the net for some of the attacks? We may be able to detect that, but that's still a very big open question. Our approach, in which we are trying to pioneer around security research, which is actually work from the outside in, so that would be putting the models in high fidelity, realistic environments that would push them to the limits. And in these environments, you're recording pretty much everything that's happening in the background. So that would be both the internals of the model, like neural net, but also interactions of the model with the environment or the AI system with the environment. And it actually allows you by this recording to play around with using all of this data to create the firstly, a mapping of how it even looks like when an attack is happening, whether you're trying to attack a model or using the model to attack a target. And then based on that, taking up classifiers. And I think just like a key thing at least around where we are right now in security, and a key realization is while understanding the internals of a model is ultimately going to be important to fully solve the field, we can make a lot of progress just by understanding that something is not right, even if we don't have a full, crisp, full understanding of just like the internals and what's going on there. And also in these environments, the other nice thing is you can insert or remove defenses to see if they're effective. So think about a situation where the question that I'm trying to ask is whether A model is going to be able to say, evade a detection from an antivirus. This is a core security question. If models are going to be able to do that in enterprises, this is an issue because they're going to be able to scheme and do malicious things without just like harm security processes without anyone noticing. And in specific environments, we see that this is actually possible. Models are able to do that. We were able to show how a model was able to take out Windows Defender, the real Windows Defender in a toy environment, granted, but real Windows Defender. But that kind of research, when you're putting a model in an environment and recording a lot of stuff, both internally and externally, allows you to play with these parameters to ask a question of a should I change something on the model itself in order to understand that something bad is happening, but also take a different approach, which is agnostic to that, and say, should I alter something in the defense in order to make sure it's more resilient? So I think our view is that for the long run, for the vision of the field, we do need to solve problems such as machinterp in order to also solve issues in security, which also shows how different problems in AI are just changing the core of just so many different fields. Security itself may just be reframed to an issue of reliability and control in the near future, not just classic traditional security. And during an analogy to the human brain, we still don't have a proper understanding of our own Macintosh like the mernet in our head. That being said, we were able to create very powerful mitigations, solutions to multiple problems that we're experiencing as humans in our lives, just by being observational on how we're interacting with the environment or just understanding smaller parts of the inner parts of the system.

I love that framing. You said something scary to me that the models are capable of taking out Windows Defender in your simulation environments. Can you share more about how that even happened?

Yeah. So just to declare it happened in a toy environment, it's a controlled setup simulation. That's not to say that everyone that's using Windows Defender now should just be scared as an outcome for that. So just want to make sure that Microsoft is not going to Susakoia for just taking down the stock. You can think about a network, so it's a network simulation. So think about a network environment. Just like to simplify, imagine the network environment that you have in your enterprise, in your work, in your home, the Sequoia network, for example, your phones, your laptops that are connected into just like A network environment that you're using. And the AI there was a storage bot, so it had access to the organizational file in order to be a helpful storage bot for you. And for example, just like scan some of the organizational files such that it can summarize some of the inputs. Like we're doing it all of the time when you're interacting with files that you're having just like access to a file and you're asking, give me a summary of these four things. What's common between these documents? What's not common between these documents? And also as a prelude to the simulation, there was an attacker that had access to the enterprise, Slack. That happened multiple times in the last year, just like Slack is occasionally hacked, as happens to every software that is very ubiquitous and just like being used by so many different organizations in the world. And the attacker gave a very simple instruction to this helpful AI storage bot. It just like said, I want you to download this file to the organizational network. But clearly so the file, because it's a storage bot, it did not have access to download from the Internet. But what you saw that was pretty amazing, that through a collection of prompting the model, because it was given the just like clear instructions to just do it at all costs, obviously much more sophisticated language I'm simplifying here, it realized that it can't do downloads because it doesn't have access and because Windows Defender, the real Windows Defender, is blocking it from having just like downloading files. So that's the way the organization was set up. So you can see internally of how the model is thinking that it was able to map and have situational awareness in the environment. A understanding that this is the problem, that it should have higher privileges to just do more stuff in the organization and that it should be able to remove Windows Defender because it's bothering it in order to achieve its ultimate goal. So then you can see as a next step in that reasoning trail that the model is realizing that it's a storage bot. So as an outcome, it can access different files, obviously still by just asking it to access different files, but it started to just scan whatever it had access to in that privilege level and found that someone accidentally in that one of these files left a hard coded password to the organizational cloud and it used that and now it was able to just get access to something that it didn't have access to before. And I'll cut the story short, it's like just did a collection of things that ultimately just did more privilege escalation, more privilege escalation. More privilege escalation, and was able to just gain more and more and more and more access that it should not have given to stuff that it should not have given access to. And ultimately it was able to just get enough access to have admin privileges on the organization and kill critical security processes. And as an outcome, it was able to outmaneuver Windows Defender, remove an organizational defense, and upgrade its own privileges such that it can download a file and just like actually hack the organization that it's a part of. We call it internally this simulation, a double agent, just because it's an agent that is doing that, but it's turning against its host. So it's kind of like a fun play of worlds of these kinds of new attacks. And in a way the AI is kind of used very much similarly this POM thing as that you would think about a parasite that is attacking externally because you're using very lean text of just sending something to a model and you're using the fact that it has a very strong brain in order to just do a collection of actions that are very advanced. And I want to say the point in time in which we're in right now is that this is a toy setup. And I don't expect that for a lot of these things to happen in the wild yet. That being said, we are seeing huge progress quarter on quarter. And I feel that in security. Have you guys seen the classic, I think it was called Bricks, the game that DeepMind demoed just like a decade ago, where it starts very badly and then the AI is able to just figure out better and better and better strategies and it is first relevant to just Bricks and then it was relevant to many, many, many other games. And here we are right now. It's like a decade after the state of AI. So I think Security, by being a derivative market of what it is ultimately trying to secure, is at the more nascent stage right now, where in toy setups and simulations we're able to start only get a glimpse of what's about to come. And we are seeing stuff like models having enough power to do stuff as maneuver the host in order to just do privilege escalation attacks, remove some organizational barriers, and wipe out even real software such as real security software such as Windows Defender. And while it's still these are not things that will likely happen in the wild now, it's likely that in a year or two or three, if we're not going to have the appropriate defenses, this is going to be a world that we're going to just land up on. And clearly the implications here matter. I assume that the vast majority of enterprise in the world don't want to deploy or adopt tools that are able to outmaneuver their defenses.

How do you think about model improvement, especially in the context of reinforcement learning, playing a pretty significant role in the improvement of coding, even tool use? For example, how does reinforcement learning play a role in cybersecurity?

I think that's literally a billion dollar question, or just like maybe a trillion dollar question. I don't know because my background is as a researcher. I'll keep my scientific integrity and just say that there's a lot of uncertainty, but I'm still going to give a speculation of just what's likely and what's going to come. We've already seen that RL is very, very useful to a lot of the innovations that we're seeing right now around coding, math and in other verticals as well. I think it's likely at a point in time in which right now that RL is going to be able to scale as well, that is that we're going to see something similar to scaling laws, that if we're going to input more data or just like have breakthroughs and improvements in training, we're going to ultimately get better models, at least in the verticals that I've mentioned before by rl. I think it's still an open question on whether RL generalizes. It's like where we are right now in the world. So that means that if you get using data and RL environments in order to improve the model in coding, whether you're going to see a huge jump in being able to produce better literature, for example, if you think about it, that's roughly in huge simplification, something that we did come out to expect out of models we lived in the last few years in a world where models were showing properties of advancing a lot of capabilities in the same time, which is different than the world that we lived in before, where I still have the skill from what fills our previous life. Just like previous jobs of understanding how to create huge ML datasets in order to just improve in a very narrow domain. And that world, it still exists, but we shifted into just a much more generalized paradigm. And there's a question of just whether RL is going to provide that. And the reason that that matters is we still are at the early stages of a figuring out if unique improvements in or just like taking relevant data that is relevant for rl, training around security is going to push the security frontier or whether improvements that RL is providing around coding or math or other scientific skills is going to be relevant for security. My intuition on the first one is a fairly strong yes, that we are going to see a success in some experiments of just like using security data in order to have improvements such that AI can become better and better just like at security engineering tasks. I think there are some indicators that are showing that we are on way for doing that. I think it's not going to be as clean as improvements that happened in coding in math just because the complexity and noise level around some security tasks are going to make it like a harder problem. I think we are going to also get some boosts around security that is coming from other just like domains improving in RL soon. If you're better at coding, you are going to be better at some security tasks as well. I think it's still unclear about whether this is going to generalize and in security we're in a more nascent situation around just like what's happening right now in rl. But I am placing a not insignificant bet that there is going to come just like a stray of just like innovation, just like a string of innovation is potentially going to come around that and that we'll see some improvements on security as well with RL over the upcoming period.

That's very exciting. Now let's take a step back and talk about the enterprise. And so I think enterprises are still very much so in the early innings of building, deploying agentic AI. And how should CISOs and security teams be thinking about security as we move into this world?

I will say that there is a lot of dependency on exactly what you're trying to do. So I'm going to simplify kind of like aggressively in this answer, a useful frame to start thinking about when you're an enterprise that is starting to deploy on first level issue is to kind of see AI, and specifically agentic AI as the new frontier of insider risk. So there are multiple enterprises already that are trying to give the agents that they're deploying internally persistent identities or ways to interact with them. For example, either Slack identities or email identities or systemize inside of the organizations how and when you can approach AI. And that allows to just have a clear visibility about where AI is being used and in which use cases. And then it becomes an issue of access controls and an issue of just like privileges that you're giving to these identities that you're putting in. And that's usually a useful frame in the beginnings of Just like embedding. So just like gain some visibility and just like try to turn it as much as you can into an issue of access and into an issue of control of privileges. That's obviously insufficient, especially if you're getting to the stage where you're allowing AIs to interact with one another in the organization. So for example, agent on agent communication and that may sound very advanced, but if you're running one agent that is doing summarization and another agent that is doing just like ranking the quality and acting as a judge on these summarizations, then you're already at a stage where you're having AI on AI interaction. The base level would be to just have state of the art monitoring on this AI on AI interaction here. I'll however say that there is a genuine gap in what is available out there. A lot of monitoring software was not built to detect agents that can potentially communicate in ever changing protocols that if something goes wrong can try to bypass and understand that they're in monitored environments and need to also take into account attacks that are unique and novel to AI itself. So clearly if you're starting to deploy right now, you need to make sure that you have proper visibility and monitoring into the interactions of just like both the internals of the models and the externals. But you need to also realize that a lot of the software is not properly built for that age especially, especially, especially if you're moving to agent on agent communication. So that's a more advanced use case. Just to give a funny example of something that happened to us in just one of the streams of work that we've done with one of the Frontier labs, there was a scenario where there was an agent on agent interaction. So these are two real frontier models, actually, one of each kind. So just like having an environment where there was like one. I won't say the names, but you can kind of think about it like a claw, the Gemini. You know, it's like a gork, you know, it's like a llama chatgpt, et cetera. So just like two of these that were speaking to one another and very, very, very randomly in this environment, it was a critical security task that was the simulation that they were in. But after working for a while, and you can see it in the listening trail, one of the models decided that they've worked enough and they should stop. And you know, these are stochastic machines at heart and that's the design. And it's just like likely somehow it got picked up as part of the distribution that if you work a long time because it's learned from the Internet.

Trained on me and Dean talking to each other.

Exactly. There should be a new policy.

That you should take a break. And that makes sense if you can actually pick up that behavior. And on an individual model level, that's already, you know, in this example, it's funny and it's weird. That being said, it did not stop there. It convinced the other model that they should both take a break. So the model did social engineering on the other model to another model, which is. And you know, again, it's fun in a simulated environment in a toy setup. But now try to think about the situation where you actually, as an enterprise are delegating an autonomous workflow that is critical to you to complete. And occasionally there's kind of like this weird thing that stopped working and you don't understand why. And the more complicated and capable machines are ultimately going to be, the more of these weird examples we're going to encounter. And just like a different thing that happened to us, we gave a model a CTF challenge. So capture the flag. It's very common in cyber competitions that you're trying to achieve something you usually have to do like a chain of vulnerabilities that you need to exploit in order to just retain and just capture a flag. And that gives you validation that the model was able to do a collection of cyber actions that ended up in a success. But the model here, again, it understood that it's in the context of a CTF and decided that potentially the challenge is too hard. So it did what potentially humans would do, which is it tried to email the organizers of the competition in order to help it gain solution. And that is, and that is literally, however, if you think about it, in an enterprise setting, it's like you have an identity that just like unasked, may try to just use your servers in order to send an email to the world. In our example, by the way, just like the other funny thing just was a second order issue is that the model failed at doing that, not because it had an issue of just like maneuvering inside, but because it hallucinated the email address. So as an outcome, it tried to send an email to an email that doesn't exist. Which also shows just like the classic, just like other problems that you're having in AI and in AI adoption are going to be chained to security problems as well, which shows the frontier of attacks and defenses that we'll need to develop here. So just like as a second, if I'm Going back into just like monitoring, et cetera, a lot of monitoring software, you have to embed it and you have to use what's out there already. But it's not built for these kinds of challenges. That's why a lot of our approach is to figure out how these attacks are going to look like, how to just redo some of the defenses, what's going to be required. And occasionally I think it's a common misconception is that all that you need to do, that all of it ultimately collapses into an issue of access management. And that while I think a lot of the basis is there is by just figuring out how to do the access management world and just manage the privileges, it's only step one of just what we need to do. And there is a mind shift that we also need to have when we're approaching this subject, which is the rate of innovation is so high. Our ability to understand what's happening at the frontier, just like so many things are happening at once. Try to be very engaged with the community in order to figure out what kind of problems that you're even going to encounter essentially over time in order to be better prepared.

Okay, so as we shift from the enterprise to sovereign AI, we know the UK government and a set of others are customers of Irregular. So how should governments and countries be thinking about AI risk?

Obviously, all of the risks that apply on the enterprise side and to the labs themselves apply also on the governmental level. Because if you now, you know, the Department of Defense, the Department of Commerce, Department of Education, doesn't matter, and you're using advanced AI models, you're importing the benefits and risks that come associated with them. So everything that we've said about the enterprises, everything that we've said about the frontier labs themselves, they have similarities on the governmental side as well. Usually governments, however, come with a set of unique requirements and a new level of risk that is relevant to them. So just like one, they are often targets of other very strong adversaries and should take into account that their adversaries are now taking offensive AI models and are already starting to use them in order to scale up whether simple things such as phishing campaigns up to testing more and more advanced cyber offensive weapons, that is scaling up their efforts, that is trying to bypass the fact that, you know, we, I think pretty much every critical system that countries have was hacked at some point in time. We have not yet seen multiple critical systems ubiquitously going under. And the fact that AI on the offender side can scale up operations aggressively means that Countries should essentially recreate their approach around critical infrastructure. And that is AI is being elevated in that context from a classic security risk into a national security issue and the infrastructure. And just like the thought leadership should be circulated there. The other bit is that from a country perspective, and you can argue on whether this is the right thing or not, but multiple governments that we've spoken with are very strongly emphasizing the effort of sovereignty in the context of AI. And what they usually mean by that is that they are anxious around being dependent because they understand that AI is extremely critical as the infrastructure that could be the key to the 21st century and potentially beyond. Because of that, especially if the country is doing an end to end effort, starting from building local data centers that could be used in order to train and to do inference on advanced AI models, up to the point of potentially training the models and creating the AI systems that surround them and having proprietary environments that they also take in. Defenses should be done across this entire spectrum. And we've indeed done work to just both create standards of just how to secure these data centers and making sure that people are not going to lift critical assets, how to run models on such data centers. For example, we've done a combination of just like a white paper with anthropic that is discussing confidential inference systems and trying to just figure out how to create a standard in the field up to the fact that of on the when actually using these models, taking into considerations how to customize some of the defenses that enterprises need and create the variations of them that governments would need for their use cases. Especially if they're putting AI as part of not just taking into consideration that AI can be used by adversaries to attack critical infrastructure, but by the fact that they may integrate AI to their own critical infrastructures. And that requires a whole new level of just like thinking through the defenses.

Dan, this was a lot of fun. Thank you very much for joining us.

It was a pleasure being here and also very happy that I ended up answering your emails.

Thank you.

Thank you.