Eliminate AI failures

The goal of any technology is to basically support humans, right? Not replace them wherever it can help automate and kind of debias human decision making. We want to support that and not replace that, but at the same time we want to make sure that we're not replacing human judgment. I think that the challenge that we have in our world, I don't know that we will have that choice. Meaning that we don't have a person sitting in every kind of like critical decision junction and making that critical human judgment. It doesn't scale big. Thanks to our partners Linode, Fastly and launchdarkly.

We love Lino.

They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by Me, Myself and AI. It's a podcast on artificial intelligence and business, and it's produced by our friends at mit, Sloan Management Review and Boston Consulting Group. The question is, why do only 10% of companies succeed with artificial intelligence? That's the question they aim to answer with this podcast. Here's Google Cloud's Will Grantis on an unusual AI challenge. When I think about what AI is, I find the algorithms mathematically fascinating. But I find the use of the algorithms far more fascinating because from a technical perspective, we're finding correlations in extremely high dimensional nonlinear spaces. It's statistics at scale in some sense, right? We're finding these correlations between A and B. And those algorithms are really interesting, and.

I'm still teaching those now, and they're fun.

But what's more interesting to me is what do those correlations mean for the people? All right, Me, Myself and AI is a collaboration between MIT Slow Management Review and Boston Consulting Group. It's available wherever you get your podcast. Just search Me, Myself and AI. Foreign.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@kickby.com community and follow us on Twitter. We're racticalai.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

Doing very well. How are you today, Daniel?

I'm doing Well, I think this episode will release in the new year, but there's a lot of sort of year end stuff happening to tie up loose ends and project plan for next year. So all good things. But yeah, trying to do all that stuff.

The end of your time warp here, even though everyone's hearing it after the fact.

Right, Exactly. I'm pretty excited. I know that you and I have talked many different times on the podcast about where AI models can fail or go wrong, either in terms of data or in terms of their behavior, but maybe we haven't talked as much about, like, ways to mitigate that problem practically and how people are approaching that. And we're really excited today because we have Jeroen Singer with us, who is CEO of Robust Intelligence. And of course they know all about how AI models fail and how to do something about it. So welcome, welcome to the show. Yaron.

Yeah, great to be here. Thanks for having me. Daniel and Chris, this is, you know, I'm a big fan. I listen to, you know, I think I've listened to pretty much all the episodes, so it's.

It's great to be here.

Oh, wow.

Yeah, yeah, that's.

That's great. Well, it's wonderful to have you on the show. Could you maybe just start out by getting us up to speed? Maybe for those that are out there that aren't really aware of the different ways in which AI models might fail and the sort of risk associated with that, could you give us a little bit of an intro to maybe some of the highlights from that and either things you've seen or things in the community that kind of motivated you to think about this problem?

Absolutely. When we say that AI models fail, what do we mean? Failure can come in all these sort of different shapes and forms. So maybe we kind of start with some examples that I think a lot of us in the AI community are familiar with. One example that's been kind of a very famous one was the Microsoft chatbot example where Microsoft kind of like developed this chatbot, this AI chatbot that was really great technology. And. But what they did is they basically, the idea was for this chatbot to kind of mimic actual human conversation. And the way that they did that is they actually used kind of like data from Twitter kind of to have, you know, trained that bot. And when people figure that out, what they actually did is they actually, you know, kind of fed all sort of like, kind of weird different things to the, to the chatbot, to the point where they realized that they can actually make this chatbot Racist. Right. So at the end, what you got is that kind of pretty quickly, I think within, you know, a few, like, I think it was like 36 hours after it was kind of released, you know, you had this chatbot that was kind of like spinning out all these sort of like awful racial slurs and Microsoft ended up shutting it down. Right. So, you know, it's just kind of like one example where you have this like, really elite technology, right? And once it falls into, you know, the wrong hands, right. It can obviously, you know, it can kind of be led astray and kind of be manipulated in all these kind of weird ways that, you know, we never expected and intended. So there are all these sort of ways that, you know, that AI models fail and, you know, even like kind of as of, as of recent, like we've all read about, or some of us have read about, you know, Zillow's example where if you went on Wired and you read like, in June 15, there's like kind of this beautiful headline about how Zillow is going to improve its predictive pricing on housing using kind of sophisticated neural networks. But then if you kind of redwire it again in November, it explains to you why Zillow failed with its AI pricing. AI based pricing, Right. And basically they kind of like let go, I think, of like 25% of the, you know, of Zillow's kind of like employees and had huge kind of like economic implications for Zillow. And basically there the failure came because of changing conditions and specifically it was because of the pandemic. So the models were trained using old data and then they applied them and then on the world which was experiencing a pandemic, and that data was very different. So this is what we call distributional drift. And we saw failure of the AI models there. So these are just kind of like two, I think, kind of one famous example and one of the more recent examples, but of different, you know, totally different ways of, in which AI systems can, you know, can just completely fail for different reasons. Yeah, so. So those are kind of like some examples and, and the implications are, you know, are obvious. Right. You know, either, you know, we have like kind of very bad reputation, very bad risk. We're putting kind of people, safety at risk. So those are kind of big problems that we're experiencing.

I definitely see on the one hand you've sort of got these like, behavioral problems of models where like, depending on what data you feed them or train them on or update them with, you kind of get this non ideal Behavior in one way or another. I'm curious about your perspective now, sort of being in this field and seeing how clients are using AI models maybe more and more over time. What is the sort of, from your perspective, is there an increasing risk in the way in which people are using models, like the tasks that they're applying to, applying models to in business use cases where, like the chatbot one is interesting and, you know, is bad PR for Microsoft. But, like, are people starting to maybe apply AI models in maybe more risky business use cases versus sort of more toy problems or research type of settings?

That's a great question. I think in general, right. Regardless of one organization or another, we are in a world that is adopting algorithmic decision making that is completely based on AI. And this world is adopting this sort of automated decision making at an exponential, exponential pace. And some examples are examples that have been out there for a while that we know, right? But it goes for things that are as simple as AI models being used for determining basically insurance rates for home insurance, for home insurance and car insurance, but also for health insurance. And when these AI models can have these sort of different failure points, failure modes, that has huge ramifications. Same with lending. So AI is used a lot in lending and deciding, like who gets a loan, you know, who gets a house loan, who doesn't get a house loan, who gets a car loan, who doesn't and, you know, and how much they have to pay. And we also see this in things like, you know, predictive policing, right. Where police departments across the country are using AI models to basically decide on, you know, whether, you know, where they're going to be putting more, more forces. Right? And all these things are, I think, kind of like the intentions are good, right? People want to, you know, want to make good decisions and they want to do this in a fair way and automate, you know, that process somehow. But at the same time, with everything that we know about AI and its discipline, nature, you know, the risk is enormous.

And I got a question about that. As we've talked about some of these use cases and you kind of think of maybe divide them up a little bit. On the Microsoft side, there's failure that is intentional in nature. There's sort of these, for lack of a better word, adversaries out there. And they're taking advantage of the weakness of a model and seeing what, what they can do with the chatbot. And then you have these kind of environmental influences, such as the Zillow thing where the world changed out from under you. It was no one person's intention to do that, but nonetheless it had the same effect. Does the intentional and the unintentional, does that matter in as you're dealing with these failure situations? Does it. Does it change the way that you approach the problem? Or does that intention or lack thereof just because it happened? Is that a factor in things?

That's a very good question, right? Like, does it matter whether, you know, whether the AI failures are due to an adversary that's trying to. To create, you know, these failure modes, or they're just. Or they just sort of, like, happen because of, you know, changing natural conditions or whatnot? So I think the answer to that is, you know, yes and no. It, like, kind of. It matters and it does not matter. And in our approach, and that's sort of like kind of the approach that we have in our company, right. In our company, the part where we think it does not matter, right. Is where basically we put all these things under the category of risk, right? Where what we look at is we basically, we abstract that. We abstract the root cause of the risk right away from, you know, we sort of say, well, it doesn't really matter, you know, kind of what has caused that model to fail. Right. The important thing is that, you know, that the model does not fail for, you know, whatever, you know, whatever reason. So in that sense, we're kind of like. We take an approach where we're agnostic, right, to whether it was an adversary that fooled the model, whether it was the pandemic that changed the conditions. Right. Whether it was whether somebody really intended to misfeed, to put in, like, racial slurs, or it's just something that for whatever reason was picked up from the Internet. So somehow the root cause does not really matter. What's important is that we somehow are able to reduce it from a technical perspective to kind of understand it and being able to protect the models from this. Now, yes, it does matter in the sense of kind of like the algorithms that you end up using to kind of protect models from one kind of failure to another kind of failure. So the algorithms that you would use to protect the model from what we call a distributional drift, like, kind of changing in conditions, like, due to the pandemic are different from the algorithms that we use to protect the model from being kind of handed, kind of like adversarial input.

On that point, are you seeing, like, as you're engaging with clients and companies, are you seeing, I guess, two questions. One is, what is their perception of the main category of risk in Those two cases, the adversarial or the sort of data drift and distributional change types of things and what maybe from your perspective is the sort of reality. I guess, I guess maybe some companies have a natural, naturally a higher risk vector for adversarial parties coming against them just because of what they do or whatever. But yeah, I don't know if you have any thought on that.

Yeah, I think that's a really interesting question. I think that it really depends on the company, but not only the company. It actually, even different teams within a company can have different concerns, right? You can imagine that you have a company where, where they have a team and that team is responsible for fraud detection in that team, right? So the team that's dealing with fraud detection in the company, right, they are constantly dealing with adversarial input and they very much care about protecting their AI from the thread vectors of adversarial input, right? The same company can have another team and that team is dealing with forecasting, forecasting of different events, right? And forecasting different events. Like they don't care about, they don't think about adversarial input. They worry about rainy days changing. If you're thinking about maybe like a company that's doing ride sharing or things like that, right? So they care about the underlying, the continuously kind of changing conditions and how that affects the models and the predictions. And if you want to do this well, then you want to be able to kind of build a system that protects models from both these types of cases. The cases where there's like there are adversaries that are really just trying to change and manipulate financial transactions, right? As well as the thread vector of just changing conditions, right. With no adversary in place, but just sort of like changing conditions that can change with the predictions of the models.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog.

It's better. So as I was looking through some kind of updated best practices out there, actually I think what triggered this was, you know, Chris and I spoke recently on a show about the OpenAI API because it's now sort of generally available. And I was reading through a bunch of their documentation in terms of like how they can, what they're thinking about in terms of risk and like best practices in terms of using the API to maintain safety and security and privacy and all of these things. One of the things that they highlighted in there was the sort of automation and human in the loop element of this, where, like, if you're planning on creating an application, for example, with the OpenAI API, it's considered much higher risk if it's to automate something. Right. And there's not going to be any sort of human in the loop to review things. As someone who kind of works to manage and mitigate the risk in these types of scenarios, how do you view, I guess, as the industry gets more sophisticated at handling the risk, are we going to be able to automate things more? Or maybe is it that we deploy our models and we have monitoring infrastructure that helps us know when things are going wrong, but it's still automated. I'm wondering if how you kind of view this shifting over time and view this need for humans in the loop in terms of the output of models.

I think this is really important and really interesting. Right. And I think that what you're saying from OpenAI is just, it just echoes what we're seeing from, you know, from leading companies and platform in general. When you look at the state of the art, it's all going towards automation. Right. So if you're using databricks today, you can already, you're essentially kind of in databricks, notebooks, you can. You have the option of retraining your models automatically. Right. So, and in general, that's kind of where the world is going. And I think that, like within, I think that within. I want to be careful with my predictions here, but like, we're a few years away from basically having most of the retraining tasks being done automatically, meaning without a human in the loop. Right. And that's generally where I think AI is going. Right. If you kind of want to think about where I is going to be, like kind of, you know, three years from now, look at where I was, you know, five or seven years ago. If you think about kind of like where we were seven years ago, this is like, it predates SciKit, learn, predates TensorFlow, predates PyTorch, right? Where if you wanted to, like, you know, if you heard about AI and you thought, oh, maybe that could have an advantage for my business or my organization, you would basically need to hire a PhD from some top school to like code up and see an svm. That's kind of what they would do. The thought of that today seems like kind of very. Right, we're kind of like we're laughing about that, but that's really how it used to be. Just like in what, in 2014. Right. 2015. There's been a lot of automation since then. And I think that where we're going is we're going towards more and more and more of this automation. Especially when, you know, if you meet a company now and that company has just a handful of models, you know, talk to that company again, a year from now, they'll probably have like hundreds of models, right? And if you met a company now that has hundreds of models, you know, a year from now, they'll probably have thousands of models. So in order to do that and scale, it's all automation. So that's kind of like our perspective and philosophy in the company that the world of AI is going towards automation. And as the world of AI is going towards automation, we're trying to really think and understand how do we ensure an organization that is using AI, especially kind of in an automated way, is really making sure that is not taking on any risk, that it really eliminates all the risks that AI has, especially with assuming that it does automated retraining of models and all that. I think automation definitely presents an additional dimension of risk and it's really important to understand that risk that we're taking on and be thoughtful about it.

I got a follow up question to that as Daniel was asking that last one. It popped to mind and you've started addressing it already and I wanted to explore it a little bit. And that is if you're thinking about this journey from kind of where we are now and having human in the loop on lots of critical tasks and lots of industries, especially for retraining and things. But if we agree that we're generally moving toward full automation, there's risks associated with both of those scenarios and there's also risks associated with that kind of in between where it's human automation, collaboration, often said, like in the industry I'm in, manned unmanned teaming, that kind of concept, all of those two poles plus the journey in the middle, all have a discrete set of risks associated with them. How do you see that? How do you look at those risk sets and how do you decide what matters? You know, how do you evaluate that? Because I think that's a really tough question that I end up talking with folks that are grappling with is, well, there's problems with having humans. We're not perfect, we screw up. And we sometimes talk about it in the sense of Having a human brings safety, but not always. And there are other times you can go argue the other way. What's your perspective on that?

Yeah, there's an inevitable trade off. Right. That's where you're saying. I'm of the opinion that the goal of any technology is to basically support humans, not replace them. On the one hand, we want to support human decision making wherever it can help automate and kind of debias human decision making. We want to support that and not replace that. But at the same time we want to make sure that we're not replacing human judgment. I think that the challenge that we have in our world, I don't know that we will have that choice. Meaning that we don't have a person sitting in every kind of like critical decision junction and making that critical human judgment. It doesn't scale. And that's the biggest challenge that we have. So with that in mind, I think that like, what we need to do is we need to make sure that it's never going to be perfect, but we need to, we need to know that we're making every possible effort to make sure that it is as safe and risk free as much as possible.

That's a very thoughtful answer. Thank you. Yeah.

This whole time I've sort of been thinking and through these sort of wider, you know, broader issues and thinking through my own use cases and as Chris knows, eventually I always get to the point where I'm like, okay, well I understand the point here, like what can we do practically to address these things and mitigate them? I'm wondering if you could like, maybe walk us through like your journey to robust intelligence in terms of how you kind of came to, I guess, understand what you wanted to focus on in terms of, in terms of what you wanted to build and offer to the community. Because there are so many problems and so much to address and you know, obviously you have to focus on something to start with. So how did, how did you get to that point and how did things kind of get started?

So my journey into this has been through like a lot of AI practitioners, I think it started in academia. So I was a PhD student at Berkeley and then I worked at Google and then I spent seven or eight years at Harvard. I'm a professor of computer science and applied math at Harvard. And basically what I've been working on at Harvard is exactly this topic. The vulnerability, the sensitivity and the failure modes of machine learning models. So a little bit before my time at Google, and then while I was at Google, I've worked on machine learning models and basically algorithmic decision that is based on machine learning models. And when you do that and you start to kind of do the theory behind it and you try to prove theorems, you start realiz that we have very little foundations for kind of like algorithmic decision making given machine learning input. And there's a good reason for why we have like very little theoretical understanding is because it's pretty terrible, to be honest. And that's kind of what, what I started studying and specifically like kind of I've been looking at, you know, really the, the sort of the whether we can make good algorithmic decisions given input from machine learning models. And the answer is, you know, mathematically, no. Sort of like. And this has been my, my focus at Harvard.

And so when you say terrible, you sort of mean terrible in the sen. You can't get to a point where you can prove in many cases that algorithmic decision making is a good idea.

Yes, exactly. You actually prove it mathematically. You know, we have like kind of mathematical definitions of what it means to learn and what it means to like how to make good decisions. And you can, we have like kind of very, very rigorous kind of models and statements. And when you kind of use these rigorous and these are the kind of like the rigorous, you know, models and statements that kind of make machine learning work. When you kind of try to apply a little bit more complex decision making on top of these results of machine learning models, it all starts to break. So we've been kind of like proving these theorems about how much data you would need in order to make what we call good decisions. And turns out that the data is exponential in the dimension of the input, which is really bad. And kind of like we study the sensitivities of models to kind of very, very small errors and very, very kind of small failures. And again, kind of like infinitely small errors in, in models can lead to errors that are arbitrarily bad. It's quite horrible and quite bad. And I've actually spent some time in my academic career trying to convince people of this and giving talks about an inconvenient truth about algorithms in the era of machine learning. If you look at that title up, you'll see a bunch of lectures coming up in seminars across.

I'm sure it produces some awkward conversations at conferences.

Absolutely, absolutely it does. Right. Especially in a time where machine learning is on the rise and on the boom. And you know, and your department is like, you know, this gung ho about like hiring more people in machine learning and now they have like this professor who's like, you know, kind of proves all these weird theorems about why it's not working. I'm sitting in the same department as Les Valiant, who created the foundations of machine learning. Right. Who's been, by the way, like the most receptive person to like, you know, this sort of criticism and whatnot. And he's been so supportive. So it's definitely like a lot of some papers took like, you know, three, four years to get published, but they got published and then they got kind of like very good recognition. But we started from the theories about the impossibilities and then we moved to algorithms. And my group developed a lot of basically focused on algorithms for noise, robust algorithms for these types of problems, and kind of like focusing on what is that we can do, as we kind of proved all these impossibility theorems and kind of like focus on what is it the algorithms would be able to do. A very big idea that came up was like decoupling. So what we realized that one needs to do is basically decouple the part about model building from model security or model safety, meaning that if you try to build, just train a model that is going to be robust to, let's say, adversarial input. We talked about that. I think kind of the best result known, and I think that result is now from like two or three years ago. It turns out that if you want to make your model just by retraining it to make it more robust to like, adversarial input, I think in image classification, you're going to take the accuracy of the model from 98% to 37%. Right. In order to get any sort of reasonable robustness, which is a trade off that is just inacceptable. Right. If you're going to come up to like a company and tell them, hey, you know how your model, like, has 98% accuracy? Well, in order to make it robust to like the 0.001% of input that is bad, I'm going to take that model to accuracy to like 37%.

Would there be an analogy there to kind of in the software world having a separation of concerns?

Absolutely.

Yeah. Okay, for any listeners that are not software people, you want to address the problem you're trying to address, but at the same time, the ancillary things like security that are very important need to be addressed. But you address them separately to maximize both.

Absolutely. Whenever I talk to people for the first time about this, I tell them, you know, look, there are Two considerations. One is the mathematical consideration. And mathematically, if you wanted to make your model robust, quote unquote, by retraining it to, like, adversarial input, it means that you're going to, like, take accuracy from something like 98% to 37%. It's kind of like. It's just a mathematical factor. Now, there's another aspect of it, which is the product aspect of it. Right, right. Or the kind of like the engineering aspect of it. And the engineering aspect of it is. It's exactly what you're talking about, Chris. Right. Where if you're an engineer that's building a system, you probably should not be the one who's also responsible for protecting that system. We can just all imagine the nightmare if every time that we wrote software, we'd also have to write the antivirus and the firewall for that software. So in software, we've seen such tremendous success, exactly because of this decoupling and kind of like layering of different systems and components that know how to work together in almost like this agnostic way. And I think that what we're trying to do in the company and robust intelligence is mimic that kind of decoupling. So specifically, what we do is we build an AI firewall in. An AI firewall is a piece of software that wraps around an AI model to protect it from making mistakes. So it kind of like it's a one line of code that you add so that that one line of code can basically stands between the data and the model. And it can, once the data comes in, it monitors, tests, and can even correct it. So basically, the data point does not fool the model, does not cause the model to make this sort of mistake. Right. Or a bad prediction. And that's just sort of like this decoupling process where we're not trying to build a better model. All we're trying to do is we're trying to be basically be able to catch bad data. So it's reducing it to, like, a much, much simpler task.

I definitely resonated with your example around this sort of firewall and antivirus. I know in my own sort of building software career, anytime I felt like I, you know, I start poking holes in a firewall to open up ports and, like, configure things. I start feeling, like, extremely uncomfortable because I have no idea what I'm doing. So that definitely resonates with me. We talked about that firewall component, this AI firewall component, kind of wrapping around the model, sort of between the data and the model in that firewall, are you looking at sort of like out of distribution data that's coming in or maybe particular, are there other ways that like, some data is more particularly risky than others? So I guess my question is like, how do you know as data comes in if it's risky data or if it's not risky data?

Yes, that's a good question. So what we do is basically we test the models. We have a process of stress testing the models and we do that either implicitly. If you just install the AI firewall, if you just put the line of code, we basically do that in the background. We do that implicitly or sometimes when there are companies where we start from stress testing and then we kind of graduate to AI firewall and stress testing. Basically what that means for us is we run a series of tests on the model. Some of the tests can be how does the model respond to distributional drift? How does the model respond to unseen categoricals? How does the model, you know, just all these sort of different scenarios, right, and different inputs. And then we were measuring the response of the models, right, to kind of like all these bad things that could happen. And as we're measuring them, right, we're getting a sense basically of, we're basically training our own AI firewall. So by understanding kind of like how different input can affect the model in different ways, when new input comes in, we know whether that input is going to lead to like some sort of prediction error, some sort of prediction change. So let me give you like, you know, like a silly example, right? Suppose that your model, you have like some sort of AI model. And maybe that AI model is trying to predict whether, you know, somebody's going to earn above $100,000 next year. So whenever we take that input, like we check all the different features and when we sort of see like how the changes in different features going to affect the prediction of the model, right? So maybe like, you know, for example, we look at like, how does age, right, affect the prediction of the model? And now suppose that like kind of, you know, data comes in and somebody has accidentally replaced age with year of birth, right? So age is like, you know, maybe from like, I don't know, 36, age was changed into like 1985 or 1986 or whatever, you know. So that's basically some sort of like human error or like kind of error in, you know, kind of like in feeding in the data. And by that point the AI firewall is trained that it knows that, you know, this feature change in this feature can really affect the model prediction and certainly kind of like it understands what the right distribution of age is. It understands the age needs to be something between, you know, probably it's seen data from like 10 to like 90 or something like that. Right. So it understands whenever it sees like a big number like 1985. Right. There's something wrong here. And then what it can do is basically it can alert and it can prevent that mistake from happening by kind of even replacing 1985 by the mode of the distribution or something like that to kind of put output like a better prediction.

I'm curious, it's an interesting approach that you have here. If you are already kind of building your models and you're deploying them out into production, whatever industry you're in, and you have your mlops pipeline in place, place and on the software side, kind of your DevOps, your DevSecOps as it's kind of evolving into, in place. How do you integrate this together? How do you take what you're talking about and put into, or integrate into your existing pipeline so that you gain the benefit of what you're, what you're describing and yet not kind of breaking your approach, so to speak, in terms of, of how you're already doing that? How does all that work together?

Great. So our approach is that the best integration is no integration. And that's why we have, one way to integrate the product is like kind of, we call this like a light integration is where basically we're not integrating with the model at all. All we do is we just take prediction logs, right? So if you have like a model that's running and you store prediction logs somewhere, kind of like meaning that you have like input and the output of the model and that's stored in some sort of CSV file, then basically our product just, basically just sits there and runs a CICD process and just continuously reads that CSV file whenever you dump in a new kind of log file and it just reads it and that's it. And it just continuously tests it. And we call this continuous testing. So we continuously test the model without ever being in the critical path with basically zero integration. So it literally is like a, in production, it's like a two hour integration with Kubernetes. Because again, it doesn't stand in the critical path of anything like that. When we're doing something like AI firewall, this is where we're integrating on the, again, it's like a single line of code that we're integrating on the, on the actual model server and that involves some, you know, some libraries and things like that. But again, it uses that same principle where ultimately it throws data in the form of, like, prediction logs in the background so that it doesn't stand in the critical path of, you know, anything real in the system. So that's something that's. That's really important. And a lot of it we do. We do for customers on premise, because having your data leave the organization is a huge pain. It's sensitive. You know, there's like a lot of kind of compliance involved and things like that. So on premise is actually something that is very important.

I want to follow up a little bit on what you're talking about. Sort of this continuous testing, I think. I think is what you called it, which is. Is a really cool idea that I like. And probably, I don't know if that's less scary to AI people than a word like monitoring or something, but I like that idea of continuous testing. I'm wondering, you kind of gave the simple example of maybe this distribution of age in the model or something like that. And I'm thinking there are likely maybe these categories, sensitive categories, like, you know, personal details of age or, you know, race or whatever it is that you're giving examples of, like the police scenarios and that sort of thing, where these might be categories in which variation in that category should necessarily produce invariance in the. In the output, at least if, if you're monitoring those. Well, what's your perspective on that in terms of how to approach those sensitive categories? And is that something you can kind of program to, both in terms of variation and maybe things that should be invariant with change?

So basically, like in our paradigm, one of the kind of the basic building blocks is a building block of tests basically what, you know, you discover whatever it is that you test for, right? If you're not going to test for bias, you're not going to discover bias. But if this is something that you care about, and I think that a lot of practitioners and organizations should very much care about bias, right? Then you test for it, and whenever it exists, you find it. And that's actually a suite of tests that we have embedded in the product that is actually very important, right? So what we do is we automatically test the model. We just automatically go through all the different categories and we test whether there's. Whether there's bias in prediction, whether there's bias in uc, whether there's bias in false positives, false negatives, like all these different things across different categories, right? And that's when people Discover like all these biases that they had in the model that they never knew about. Some of it is protected categories and some of it is just, you know, just other categories that they, you know, that they didn't know, that they found out that their model is just like they're not training their models on the right data set or they should do kind of different sampling of the data. Right. In order to make sure that the model is not even kind of performance biased. Right. So I think testing for bias is such a critical thing that basically all AI practitioners should do.

This has been a fascinating exploration and definitely a topic that we have not gotten really into at any point in any previous episode. As we wind up. Where do you see the future going, both with robust intelligence as your enterprise, as your company that is doing things, but also within the larger world of robust intelligence, the larger effort in the industry to drive forward and the evolution. We won't hold you to any predictions, but if you were to make some predictions on where you think it will go or where you would like to see it go, I'd love to hear that.

Sure. So I think let's start by the things that are not interesting. Right. What we all know is we all know that the world is kind of like AI is eating the world. Right. We're going to be hard pressed to find an organization that is not adopting AI in a serious way in just a couple of years, I think. Right. So, okay, that's not interesting. So let's put that aside. But I think when it comes to our little part of the world, when it comes to like AI risk, we think that within just a few years there are two things. I think that any organization that uses AI in a way that can affect people is going to have to like go through some sort of stress testing by third party. I think that's going to be mandatory. I don't think that it's going to be up to like the director of data science in that company. That's just going to be regulation. That's number one. And number two also on regulation and best practice, I think just a few years from now, in the same way I think we're going to be hard pressed to find a company that is not protecting its models with an AI firewall, not necessarily robust intelligence. Right. I don't know if any other companies are making AI firewalls, but, but I think that like within a few years when we're going to have this conversation again three years from now, we'll go back and remind ourselves like hey, do you remember that, you know, three years ago, you know, companies were actually deploying AI models without an AI firewall? How crazy was that? Right? I think that's what we're going to find three years from now. That's where we're going, kind of like in my part of the world.

And you should consider that an invitation, by the way, for three years out to come back and we'll have that conversation.

The data is going to be easy to remember because I think that we're like the last podcast of the year. Right. So as we go into 2025, it'll be good to sort of like check where we were with our predictions.

Absolutely.

Yeah, for sure. And hopefully sooner than that as well. It's been an absolute pleasure to talk through these things with you. And as for myself and I'm guessing Chris as well, we appreciate your work in this space and pushing these ideas forward and being that voice that has occasional awkward conversations at AI conferences, it's much needed. And appreciate your perspective. So thanks for joining us.

Awesome, guys. Thank you.

All right, that's practical AI for this week. Thanks for listening. If this is your first time with us, subscribe now at PracticalAI FM or simply search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, do us a solid by recommending that show to a friend. Word of mouth is still the number one way people find new podcasts they love. Special thanks to our partners for supporting our work. Fastly launch Darkly and Linode. We appreciate it. And to the mysterious Breakmaster Cylinder for cranking out new beats for us all the time. That's all for now. We'll talk to you again next week.

Sam.