When AI goes wrong

I love machine learning. I made a career of it. I don't bring these things up just to make data scientists feel bad. I've been sort of studying different failures in machine learning and AI and algorithmic systems. And if you watch, you know, I see one or two every week now and that's been going on for almost two years. And so I study these things to learn about how these systems fail so that we can help our clients and help other people not have the same failures. But surely government must be taking notice of the same level of AI incidents.

Bandwidth for Change Log is provided by Fastly.

Learn more@fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog.

Linode is our cloud server of choice. Grab the nanode plan for just $5 a month. Just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD, and one terabyte transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable. So you can put your calculator down, you won't need it. We've been running changelog.com on linode for years and we've always impressed by their award winning support team. Check them out@linode.com changelog Once again, that's linode.com changelog. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@chain.com community and follow us on Twitter. We're ractical aifm.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with S International and I'm joined as always by my co host Chris Benson, who is a principal Emerging Technology Strategist at Lockheed Martin. How you doing Chris?

I am doing fine. How are you today Daniel?

Doing well. As we're recording this, it was in the US the day after a long holiday weekend, Labor Day. Did you get to spend some time with your family and relax a little bit, Chris?

We did. We had good family times. Vegan hot dogs and such as that. Excellent stuff.

Yeah, yeah, yeah. We have a specific brand of a veggie dog we like. Afterwards we'll have to sidebar and discuss that.

Yeah, my wife's come up with a special recipe and stuff.

Oh wow.

Yeah, it's good. Okay.

Definitely, yeah. And I spent some of the weekend talking to my brother in law. If you've been listening, listeners might know that my brother in law lives with us and we were talking about the new Nvidia cards that were announced and he's already specked out a machine. Of course he's building a gaming machine. But we were talking about like the price points and everything and he was saying, you know, for how much lower of a price you could get, you know, the level of the previous generation which was way more expensive and you know, I was looking a little bit at the 3090 and what they're doing and everything. It's pretty exciting stuff, I have to say. Are you following that at all, Chris?

I am some, absolutely. As you're talking there, I got to ask. So is your machine, the one you built, all working and happy and everything?

Yeah, it's. It's all working, working. I did a few different models on there, trained those for a project we're doing at SIL and that worked really well. It's no problems yet. I kind of kept, you know, there's not much installed on it. I basically have Docker installed and that's how I'm running all of everything. So just using like the publish TensorFlow images and Nvidia images and then, you know, training from there. So it's worked out pretty good so far.

At some point you need to share your specs and stuff.

Yeah, I will. I shared them in our Slack channel. So listeners know we have a community Slack channel and that's. You can find that@changelog.com community and we discussed it a little bit there. But I'm definitely going to have to do a legitimate blog post. I just haven't done it yet. Shame on me.

Sounds good. Waiting for it.

Yeah, definitely. I'm pretty excited. If you remember, Chris, way back in the day actually episode four, we had Andrew Burt on talking about, at that point we were talking about I think GDPR and regulation and some other things as they relate to AI. And we're really excited to have Andrew Burt and also Patrick hall with us today. They're both from BNH AI. Andrew is a managing partner there, Patrick is a principal scientist. And this is a pretty interesting and new law firm focused on AI and analytics and other things, which is pretty cool. We'll talk about that a bit. But maybe first let's just get some introductions. Andrew, do you want to start us out and tell us a little bit about your background Remind us about your background and how you ended up where you're at now.

Yeah, happily. And thanks both for having me and us on and yeah, it's fun to be back and see how much things have changed in the last couple years.

Yeah, yeah, we're still going. I know it's been crazy.

Still going. Yeah. Well, despite all of the fame. The fame and the fortune.

Right.

What to do, how to find podcasting, you know, set up in your mansion.

Yeah, I mean it's just really nice to see how it got to your guys heads, you know. Well, anyway, so my background, so I'm a lawyer and also a technologist. I spent many years kind of working at the intersection of law and risk and technology. I spent a few years for the FBI Cyber Division. Since 2016, I've also been with Immuta. I was on last time in my capacity as with the muta. And basically throughout all of this time I have been really the only lawyer in a room full of data scientists and technologists and people wanting to do really cool stuff with data, oftentimes sensitive data or data in regulated environments. And it has brought me to this conclusion which I've seen over and over, which is that the biggest problems with AI and machine learning in my view are no longer technical. And in fact a lot of it is to me seems like it's been productized. The biggest challenges, and I've seen this in practice, like being the only lawyer in these rooms is legal, it's ethical, it's policy related. And so I've started to see these really, really, really big challenges stand in the way of really exciting technology projects. And so all of that kind of led Patrick and myself to Start B&H AI which is a boutique law firm based in Washington D.C. dC. It's the only place in the country where lawyers and non lawyers can actually jointly run law firms. And so we can talk about why that is and why that exemption exists, but really we're the first to do this with data science. And our core thesis is that the only way to do data science, right, the only way to really kind of get all the value and minimize all the risks is by really commingling these types of expertise, the legal and the policy, really, really closely with the technical and the data science. So that's the 50,000 foot nutshell. And we're just about, we launched right at the end of March. So we're just about five months into our journey and it's been a lot of fun.

Awesome. Yeah, that's super great. We're also really excited. Patrick, thank you for joining as well. It's great to meet you and get the connection through. Andrew, could you tell us a little bit about your background and how you ended up working together with. With Andrew on this?

Yeah, of course. I'm happy to be here. And my journey with machine learning I don't, you know, started many moons ago at SAS Institute in North Carolina. And I think that that was really formative for me. It seems like some data scientists don't even know about SAS anymore, but it's a, you know, roughly $3 billion company that's been making billions of dollars off analytics and machine learning and AI for decades. And just, just being a part of that organization, which largely lives sort of outside of the ML hype sphere, but makes more money than almost any other ML company. And just this awareness that there was a right way to do things that's decades old, was really formative for me. And after that, I joined the sort of topsy turvy ML startup world at H2O, where I kind of led their efforts in responsible machine learning for three to four years. And we came up with a decent product. And I think, like Andrew, I came to the same conclusion. But I saw, like Andrew said, not only does the technology exist to sort of make machine learning more transparent and more trustworthy, hopefully more fair, but it's already being productized. So it's, so it's reaching a certain level of maturity. And really what I see, you know, like Andrew said, I think the next round of challenges for machine learning are not going to be technical. They're going to be policy and regulation and legal and human interactions with AI and machine learning. And that's why I joined Andrew on this journey, because I saw a lot of the same things he did. And it was an easy pitch, actually, once he explained it to me. So I. Happy to be here on the podcast and happy to be here with B&H AI.

Thank you very much for joining us and it's nice to meet you, Andrew. Having Andrew back is like having an old friend back from our very beginning, as you mentioned. So, you know, Andrew, you touched on it already a little bit in your intro and Patrick, you did as well. But can you, you know, you're at this really unique position of you have the entire landscape of law with the entire landscape of AI at this juxtaposition where they are starting to merge. And you know, last time you came, you gave us the, the General Data Protection Regulation deep dive as part of that. And we talked about the fact that there wasn't a whole lot in terms of, of laws and regulations out there. And then, you know, on top of that, since then we've really gotten deep into AI ethics. I know, like in my own case, I was leading AI ethics for Lockheed Martin for a while. I just stepped out of that position, but learned a lot. So much has happened since we had you on last in this space. Can you together, guys, can you all kind of describe what this space looks like and what some of the complexity with the landscapes at large, given so many things touching ethics, all of these together.

Yeah, happily, I'll start and then there's a danger of me going on too long. So Patrick can jump in.

Please do.

So I think from my perspective, I guess from a very high level, what I have seen is, I think the prospect of AI and machine learning. And I'm really, for now, I'm just using those two terms interchangeably. We can dive more into why we might want to use one term versus the other.

I'm very happy with that. As Chris probably knows from previous of my discussions, other people might not be so much, but you're in a safe space.

Okay, all right, cool. That means a lot. That means a lot to me. So I think the prospect of what AI can do, the value, I think, has only begun to increase over time. And I think there's a wide recognition both in industry and by governments and regulators. And the way that I think about the value is that what AI does really, really well is it scales decision making in volume and speed. And so if you just think about that, so we have this tool, I mean, it's been around for a while, but let's say in the last five years, the value has become more clear and it's become used more and more. You have this thing that can make huge amounts of decisions at really high speeds. So the progression is almost natural. So as that type of tool starts to be used more and more in the real world, there are a huge amount of concerns that are associated with it. The slightest bias in something like that ends up being magnified. The slightest security bug can be magnified. What you have is, I think, really the same things that make this technology so powerful, which is it can do so much in such little times, so efficiently, also mean that really any of the risks end up being greatly magnified. So I think over the last few years, there has been this recognition that, you know, if we don't get this right, if we really don't refine what it means to manage all of these different liabilities, we're going to be in a world of trouble. And so that's what I have seen. There's a really wonderful paper from Algorithm Watch called Paper Tigers, which I can't remember the number but I actually studied something like 120 maybe more ethical AI frameworks released in the last couple of years. And the title is called Paper Tigers because these frameworks don't really have teeth. And so I think it's a, to me the rise of like the corporate ethical AI framework I think is a really good sign that we have an issue like something is wrong. On the other hand, studies like that from Algorithm Watch I think are also kind of indicative of the fact that we are nowhere close to solving it. And so anyway, that, that's what this landscape kind of looks like from my perspective.

I'll jump in real quickly. First I want to echo Andrew's last statement, you know, because I think that some of them, as a technologist, right, when I say things I think like I think the technology exists to help with some of these problems. I want to be very clear that, you know, technology alone will never solve these problems. Right. Technology is one of the, of many necessary pieces. But to add on to what Andrew was saying, you know, and, and I need to be also need to be very clear like I, I love machine learning. I've made a career of it. I don't bring these things up just to make data scientists feel bad. I've been sort of studying different failures in machine learning and AI and algorithmic systems. And if you watch, you know, I see one or two every week now and that's been going on for almost two years. And so, you know, I study these things to learn about how these systems fail so that we can help our clients and help other people not have the same failures. But surely governments must be taking notice of the same level of AI incidents. And so I think that's another reason why we see companies and governments. So just this year organizations like the Federal Trade Commission and finra, the Financial Industry Industry Regulatory Authority, have issued these sort of long treatises on, on the use of AI that Andrew and I think is sort of forecasting or telegraphing future regulation. So, so I think there's, you know, Andrew sort of focused or brought up the idea that the corporations are waking up to this. And I certainly agree with that. And I think that governments are also starting to notice the way that AI fails and react to that.

Yeah, I want to ask kind of a follow up a little bit to that, and that is, you know, when you talk about the failure of ethical frameworks for AI, they're very limited. You know, they're aspirational. They kind of talk about these ideas that you want to constrain your operations with. But as you pointed out, I think the phrase used was they have no teeth. And I certainly agree with that. I think one of the challenges that I'm expecting over the next couple of years is as organizations either create or adopt frameworks to utilize for that, the devil is going to be in the details. The devil is in how do you integrate that in with your legal structure at your, at your organization? How do you integrate that in with your operations and how you serve your customers? And so there's a great deal of, of work to do in terms of figuring out how you customize that framework down to your organization specifics. Do you have any kind of guidance on that at large? I mean, would you agree that that is the case and how would you advise organizations to think about it? You know, so I've adopted a framework. What's next?

Honestly, that's exactly why we exist. We just see this huge gap between the people thinking about the laws and the policies and then, you know, the data scientists and the engineers who actually have to go operationalize it. And what we've seen in practice time and time again is these big frameworks usually called ethical AI, sometimes, you know, trustworthy or responsible AI. And it makes everyone feel good and they're aspirational. It's great. And then it gets to the, you know, the folks in the trenches and the business units and they say, this is great, but I can't do this. Like there's a chasm that we need to cross.

It's hard.

Yeah, it's very hard. And so that is why we exist. I think one of our big things is that AI has been around for a while. We don't need to start from scratch. There are a host of regulatory actual oversight documents and then also guidance documents that actually are tried and tested. Some are five decades old dealing with discrimination, some are about 15 years old just dealing with model risk. So there are actually lots of places that we can start with. And as a law firm, that's what we start with. How have regulators, especially in finance and anti discrimination, like what have they done? What does the case law say? And it turns out there is a lot that we can work with. So that gap, that chasm, what it looks like is this huge gap is actually not that deep and it's not that wide. I don't want to minimize how hard this can be, but there are a number of practical things that the organizations like have been doing and can do for whatever reason. I think largely because the lawyers and the technical folks just don't talk enough to each other. I think a lot of these methods are underappreciated.

Foreign. Hi, I'm Matt and I'd love to tell you about pace.dev.pace.dev is a minimalist task management and async by default communication tool. Our screen recording feature is actually very popular wherever you can leave a comment. Just like how easy it is to upload a file. You can record your window or the entire screen and upload it as a video to the team. Sometimes a screen recording is the perfect way to explain something, you know, whether it's a bug that only happens for you or maybe more optimistically, a new feature that you can't wait to show off. And the showcase feature takes that a step further and lets you highlight progress, which is a much more positive experience than trying to make up estimations out of thin air. So please learn more and start your free trial at pace.dev.

Com foreign.

I'm really interested. I know that you came out with some things recently around like a, an AI incident response checklist, which I'm super interested to dive into the details a little bit. But before we do that, I'm just kind of curious to kind of talk generally about AI incidents. So where my mind is going with this is like we have some experience with software incidents, right? And there's certainly software that drives things that, you know, can make a huge impact on, on people like, you know, software in a medical setting or in a medical device or something that's not AI, but if it malfunctions, you know, someone's health could be at risk or, you know, you could expose private information or something like that. Right. And there's, there's those things that definitely do exist out there. What are the sort of liabilities about AI applications in particular that might differentiate them from some of these things that maybe we've been dealing with for some time? Patrick, you have any opinion on that?

Yeah, yeah. And you know, I'm, I'm not a lawyer, so I'm going to have to let Andrew comment on the legal liability side. But I can say, you know, we see, let's say in, in the tech media and sort of broader media, we, we see a big focus on discrimination and of course, you know, discriminatory algorithms. AI is, is not Something that, that we want out there in the world at all. But we, we also try to, to sort of direct people to, to other problems like privacy. You know, you mentioned data security. You mentioned. And so, you know, when, when I look at some of these AI incidents I've been tracking, I'd say they, they mostly break down into maybe four categories, so discrimination probably being the biggest one, and we've seen some very troubling things there. But also, you know, consumer privacy, data privacy, security, people essentially being kind of sloppy with sensitive data that's used to train AI systems or data that's generated by AI systems. And then a final one that see fairly often too is this idea of no intervenability in a machine learning system, or sometimes it's called computer says no where the main failure of the algorithm was just that it's a black box. Right. It's making decisions that impact people's lives that may or may not be correct, but the consumer of the decision has absolutely no ability to appeal the decision. And I think that's a huge mistake. And another type of AI incident that I see fairly often. So I'll leave it at those four sort of categories that I observed and maybe let Andrew sort of chime in on legal liabilities. Is that okay?

Yeah, that's good. I'm curious on that last one actually, because it's an interesting category and of course we've talked about the sort of interpretability issues and other things on the podcast before in that last category. In terms of the incident itself, is the problem mostly more weighted on the side of like interpretability and not, not kind of digging into the model, or is it more on the, like the computer has made the decision and there's no way to sort of back out that decision. Like it just happens and like, you know, like a person is denied insurance because they're deemed high risk and there's no way for them to like, you know, like you say, appeal that or something. Which side of those is it more weighted on?

I think what I see in practice is it ends up being the latter. Right, right. And this goes back to the comment of technology can't really solve these problems. So the sort of two biggest incidents in my mind when I hear computer says no or no intervenability, no appeal or override capabilities is compass, which is a risk assessment instrument that's used to help in pre trial and parole decisions. And then the recent A levels scandal in the UK where hundreds of thousands of students had their grades adjusted by an algorithm. And in both these Cases, it seems that the algorithm itself was at least well tested and well understood by its operators, but the way it was presented to its consumers was as this sort of unappealable voice of God that's going to ruin your life. And so I spent three years deep, deep, deep in explainable machine learning and interpretable models. I sadly don't think that that's the problem here. I think that of course that can be a problem, that the algorithm isn't interpretable. And that's something that I object to in almost all cases. But the big incidents that are coming to my mind are more process problems or more human problems.

I would also, just, just to add to that, I think there's certainly this like, kind of the problem where there's an algorithm that's deployed, you know, in a public setting, where there really is this like, it's, it's just there's almost like this tension between the authority of the algorithm and then everyone who, you know, who's subject to it, and that certainly there is a problem there. But just to get more operational in terms of day to day, one of the things that we see in practice involved in AI incident response is that that same dynamic can also happen between the developers of an algorithm or of a model and the model itself once it's deployed. And so if something wrong, so it's not just consumer decision to model.

Really good point, really.

It's also if something bad is happening, if there is an incident, if there is some instance of discrimination or potential hacking or data breach, then we have this problem where Patrick calls this debugging. And I think it might be worth giving Patrick a proverbial soapbox and let him talk about debugging shortly. But if there is a situation where we have potential liability and the data scientists actually need to fix it, we have the same type of clash. And what we've seen in practice is it can take an extraordinary long period of time for the data scientists to one, make the decision. And the business units, do we pull this from production? Is the liability greater than the business value? So should it be pulled from production? What do we do? How do we debug it? I think once a model goes live, so to speak, there ends up being this kind of very similar dynamic. And so we see on a practical basis, in terms of thinking about risk and liability, we see a huge number of organizations doing things that are much smaller, you know, decisions of much smaller scales struggling with this when something goes.

Wrong and just really quickly building out that AI incident response plan just like you'd build out a response plan for other mission critical computer systems can help address a lot of those questions that Andrew's bringing up.

Right.

A lot of these questions that we see teams struggling with internally. If you actually spend the time to generate that checklist, then you'll have better answers for this when the time comes.

Yeah. And you mentioned the debugging thing, which I definitely, I resonate with that a lot. So I don't want to lose that for sure. I know like a lot of times when I talk to teams, I do some advising and training and stuff and a lot of times I think people have in their mind like, oh, we train this model, like we kind of wrap it in our API or whatever or embed it in our API and it operates and like the unit tests are like around like the API and like, oh, can I process this? You know, JSON payload. Right. It's not around the model itself. So apparently that's something that you're passionate about, as Andrew alluded. Patrick. So Patrick, do you want to say anything about that in terms of debugging and like, you know, the state of debugging, especially around incidents when something goes wrong with an AI model or something unexpected happens.

Yeah. So if you guys will permit me, when I'm done chatting, I'll put two links in the Zoom chat.

Yeah, yeah, we'll include those in the show notes as well for the episode.

That'd be great. Yeah, I hope they're useful. We've gotten some positive feedback on them. So yeah, I think there's two levels of debugging. Right? Two major categories of debugging. And you brought up one where you said unit tested the API. Great. Please, God, do that. I'd include that in just sort of normal IT system debugging. Right. Unit testing.

Yeah. Something that people have been doing for quite some time.

And I can't explain this. Aside from sort of a sad level of, of hype and exceptionalism and data science culture, people in general in my experience are failing to apply general software based best practices to their machine learning. And like I said, I can't explain this except for sort of sad and regrettable cultural phenomena. So that's one main issue I see.

Patrick. I'll say I deal with that every day. Trying to get our DevOps folks and our developers working with ML makes no sense.

Makes no sense to me. So machine learning is essentially, you know, given its complexity, given its drift characteristics. Right. It's likely even more volatile than say some other mission critical software assets. And so why data scientists are given a pass on basic software quality, I'll never understand.

Okay.

And so that would be one thing to remit remedy just, just asap. And of course we debug our other mission critical software assets. Right. So, so we should be debugging our machine learning systems just, just using basic software, just best practices. So, so that's part one of, of model debugging. And I truly believe that there's no such thing as responsible AI or trustworthy AI without basic software best practices. Then we get into actually testing the machine learning itself.

Right.

And, and that's more difficult and much more of sort of a new field. And I've got some ideas around that. I'm certainly not the only one. There was a conference workshop at icml, must have been last year when we could still fly maybe two years ago. I'll put the link in the chat where, where some of the world's leading academics got together and discussed this. So I would say that just in general debugging of the machine learning system itself comes down to, at least from a practical standpoint, sort of sensitivity analysis. Right. Where is my system unstable residual analysis? Where, where is my system making errors and can I, can I try to understand those and reduce those security audits? Right. We're well aware that there are now attacks that directly affect machine learning systems. So, so doing red teaming and bug bounties and security audits on those known security vulnerabilities of ML. Of course, discrimination testing and discrimination remediation is a big one here.

Yeah, this is all super. It's resonating with me so much. I know Chris and I have talked and I have conversations every day with like, you know, you're talking about probing the sensitivities and all of that. It's so, it's so important and it's actually like in a lot of cases it doesn't require that much extra work. And I think something that people don't realize as well is to some degree a major component of this is doing these sort of tests actually allows you to be a better data scientist or produce better work. Right. Because you're actually, you understand the behavior of your model more and you're finding those places where it misbehaves and you're able to deal with those in a sort of confined test space where you actually, the end product becomes actually better. Your model is actually better in the end and more robust.

Thanks for allowing me time to pause and google my sort of last model debugging practice thing. Which is, which is Benchmark models. Right. Like having a simple, trustworthy, interpretable model to compare your more complex model against, I think is another super important thing there. Just to echo your comments, sometimes it's a lot more work, sometimes it's not a lot more work. It will make your work better. But seriously, when we're talking about data scientists who are on average paid very, very well, even if it's a lot of work, you know, with great power comes great responsibility and we just need to start taking more responsibility for the systems that we're making.

So I actually want to swing things back over to Andrew for just a moment. And Andrew, I've been sitting here as we've been talking, pondering something that you said a few minutes ago. You were talking about, you know, case law. And I guess that's opposed to statutory law. Things that I don't normally think about on a day to day basis, but I know you do.

Can we just define those terms too? Because I think there's probably a lot of people that are like maybe confused.

A little bit and then I'll ask after that. Go ahead.

Okay. Happily, yes. So especially in the US which is a common law system, the way that laws work and the places that laws come from vary. And so there can be a regulatory agency which will promulgate rules. You can think of like the fda which has like, you know, they are in charge of food and drug. You can have Congresses and state legislatures which will say pass a law that is like doing X is a crime. So like don't do X or bad things will happen. And then there's also case law. And case law kind of evolves over time from the courts. And case law is basically arises on a decision by decision basis. So there'll be, you know, there'll be a case someone, you know, there'll be a controversy that runs some party in court. And then the decision that's made as that issue is being resolved comes from just a larger body of case law. And so one of the things I think I was thinking about in the world of anti discrimination is we have all of these regulations both from regulatory agencies and we have statutes from Congress saying basically don't discriminate and don't discriminate in these environments. But it turns out it's really, really difficult to figure out exactly what counts as discrimination in a society that's really just like marked by profound inequities. And so a lot of the give and take and a lot of figuring out what exactly does this mean and how do you balance the usefulness of a model with its potential discriminatory impact like that weighing comes from case law and it comes from courts. And so we can look at the guidance that comes from government agencies as we try to figure out what the right way to approach these issues are. And then we can also look from, like, specific cases and say, okay, at this one specific time, you know, this group ran into an issue, and here, here's how the judiciary solved it. So hopefully that was a. We can get the, like Schoolhouse Rocks, the bill. We can feature that also in the show notes.

You get to sing that at the end of the show, right?

Yes, exactly. Anytime I talk to anyone that's like I put on a cane and a top hat. Perfect.

Okay. You'll have to get pictures of that to fit out. Anyway, so you've already covered part of what I was going to ask, and that is, you know, we've talked about the fact that there's not a lot of AI specific statutory law. You know, and so it sounds like you really start with regulation and case law and kind of connect that in with the kind of AI data context. Is that a fair way in terms of how you would operate in the space?

So just like Patrick can't explain why data scientists hold themselves to different, like, security standards than, you know, traditional software developers, I can't explain why there isn't more awareness of just the liabilities that existing laws place on AI. So it's true. There's no national AI law. There are very few laws that say those creating artificial intelligence shall not do this. So you're right that it's not that direct, but there are a huge number of ways that existing laws impact AI systems. And so one of the things that I kind of feel like I say over and over again is that the liabilities are not new. And we've been dealing with a lot of these for a while, so I'm happy to list some. But. So there are. I think the three most obvious are security and privacy and discrimination. If an AI model discriminates, serious liability can ensue if there are privacy violations. So basically those can happen all sorts of ways, but. But just as simple as using data for the wrong reason when it was collected for one reason, but then used to train a model for another, that's a privacy violation, security violations. I think it's fairly intuitive how you can. I think at a high level, the attack service for AI is just very much different than traditional software. So an adversary might be able to manipulate the model or steal data or Gain access. So those are kind of the intuitive things. And there are laws governing all of that. At the same time, there are things as basic as negligent standards. So if you create an AI model and it goes and someone is harmed either physically or emotionally or something breaks, we have all sorts of negligence and product liability laws. And just because they don't say AI doesn't mean that they're any less applicable. So there are a whole host, I should say. New laws are clearly coming. Like Patrick said, the FTC in particular, the Commissioner gave a speech. I actually have it somewhere on my desk. I can pull it up. But she gave a speech in January and the FTC has been doing every couple months something similar. And she basically said, we're coming for AI. AI is responsible for a huge and growing amount of harms and the FTC is going to regulate it even more. So new laws are definitely coming. And I think that that is not kind of controversial to say, but the laws on the books right now and already do impact AI quite significantly.

And I think I can add that some of the work in just the very first months of our law firm has been AI violating sort of local laws, right? Not these big federal ECOA or FCRA, Equal Credit Opportunity act or Fair Credit Reporting act, but concerns about AI violating more local laws. That I never would have thought. I mean, Andrew probably thought about it, but I never would have thought about it.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog. It's better.

We've got into a little bit of the sorts of liabilities and incidents that can happen. And I'd love to kind of switch gears a little bit and talk about this incident response checklist that you've developed. I think is that's a pretty cool thing. And maybe we could start out by just asking, you know, how this came about. Was this something that you envisioned even, you know, even before starting the law firm, or did it come out of your sort of initial conversations with clients or how did this develop as something useful?

So I think this was not the first thing we had in mind. You know, when we launched the law firm, I think we assumed that we would be kind of involved in lot lots of preventative work. You know, way before something bad happens. And frankly, coming out of the gate, we started to see that people were reaching out to us once something bad happened, you know, frequently it's not a good, we don't recommend it. You know, call lawyers, call risk folks before you're in trouble.

When you start building a critical machine learning product, that's when to call legal, oversight, ethics people in.

Yeah, so we very much want to help us as an example, we want to help folks way before we're actually needed. What we've learned is, sadly, that's not how things work in practice. And very frequently it takes something bad happening for data scientists and lawyers to say, kind of, oh crap, we have a gap. And so I used to work at the FBI Cyber Division. I actually got certified as a traditional cyber incident response handler. And so I was very familiar with the six stages of incident response. And one of the things Patrick and I did very early on is we went through the traditional textbooks that I'd been trained on the traditional incident response. And we said, we're going to look at all of the bad things we're being exposed to. We're seeing clients struggle with and all of the other liabilities that could happen with AI. And we're just going to go through and see how does current incident response practices measure up. And also in our experience, the answer is they don't. AI is shiny and new and different and there's this hype cycle. And as a result, the incident responders just kind of, it's just out of their purview. And so we went through and we realized that there could be a very severe AI incident creating huge amounts of liabilities. In fact, one that we are involved in now is for like a Fortune 100, the board is deeply involved, the CEOs, but is on the line. And we went through the traditional incident response and you could go through that check mark by check mark and everything would be okay. You wouldn't have even known anything was wrong with the AI in question. And so that kind of, I think, woke us up to the fact that this is something different, this is something new. And also there's really no guidance on how do you respond when there's an incident. And so we put this together and.

We love model risk management practices that are mostly and financial services and highly related to this. This guidance from the Federal Reserve called SR 11 7, which is a masterful treatise on model risk management. Neither did model risk management as it exists today. I'm sure it will mature to include this, but model Risk management as it exists today also did not include, you know, exact ways to react to AI incidents.

Right.

So there was nothing in traditional cybersecurity response, and there were things in model risk management that would help, but nothing that actually told you how to respond or to prepare to respond.

Yeah.

I'm curious, as you went through those things that are existing and best practices that had already been developed, what are a couple examples of where maybe existing incident response plans just wouldn't cut it or would leave something out in terms of AI and machine learning?

So there are many ways that there can be AI incidents. And I actually, I'm getting a little concerned that I think people are just only assuming that the worst that AI can do is discriminate because there's so much in the news. And frankly, about 15 years ago, there weren't mandatory breach reporting guidelines. So there are legal requirements now that say if there's a breach or a hack, you need to report it. So now we know all the bad stuff that's happening before those. No one really did. And we're in an analogous situation with AI where all this bad stuff is happening, but there's no incentive for anyone to share it. So the public is in the dark. But anyway, so at the risk of doubling down on discrimination, let's say you have an Apple Goldman situation, which, you know, their credit model was reportedly discriminated against females. There are many different ways that AI can discriminate, but let's just say you have something that's discriminating. Let's say it has been deployed and exposed to, you know, I don't know, many hundreds of thousands, a million, whatever, consumers. The question is, how do you know that that model is discriminating and how do you know the depth of that discrimination? And you could go, and so that would be an incident. Let's say you have a model that's been deployed against a million people, and let's say 20% of those decisions have been in some shape or form discriminatory. That's bad. There's lots of liability. It's a huge incident, 200,000 people. I mean, in practice we're seeing that the numbers are even higher, but huge amounts of people can be impacted. And a traditional incident response would say, okay, well, is the model available? Has its integrity been broken? Has anyone reached confidentiality? Is the data being used in line with privacy policies? Like all the traditional questions? Just don't get at this. Really, really huge liability. And so what can happen in practice is an organization will deploy a model like this, and then frequently, like the media will discover that there's something wrong or there'll be like a Twitter posting saying, like, wait a minute, you know, I'm a member of a disadvantaged community and you know, my spouse is not. And the model treated us different, differently. And then suddenly the sky is falling on these organizations. So anyway, so that model, I think in itself is a signal that something's broken.

I can summarize my comments really quickly. Typical computer incident response doesn't address machine learning security yet. And what we saw, it may soon in the future, but it doesn't yet. And then model risk management, typically not in all cases, typically doesn't address security and privacy and in some cases discrimination issues like Andrew mentioned. And so there's just kind of a gap in the two main practices, which are both great. You know, we're not saying anything negative about these. They're both great, but they just have a little bit of a gap when it comes to AI. And so, you know, I think that that's where the AI incident response checklist comes in, because we try to fill in those gaps.

So we've kind of talked about the checklist itself, but one of the comments you made a little while ago was the time to connect with us is really before that happens. So if you're a company out there and you have, you know, limited resources, limited budget, and you're trying to justify why they should engage you before an incident happens, to try to, you know, you know, work through their operations ahead of time, what are some good justifications? What, what are things that you've seen where you're like, if you come in ahead of time, you're going to save money, you're going to save a lot of heartache. How do you approach that?

I'll try on this one and Andrew can kind of jump in and correct me. So, so one, you know, all models are wrong. Some models are useful, right? Your machine learning model is going to be wrong. Okay. And so when it's wrong, something bad can happen. Right. And, and so the, the question is really how prepared are you for that bad thing to happen and what will the cost in terms of human value or monetary value be when that bad thing happens? So, so we try to, to sort of gauge AI incidents by the organization's preparedness and by the materiality of, of the incident. And so your machine learning model will be wrong.

Right.

So you just have to get ready for that. That's where I'll leave it. Except to say that in our experience, it's Much cheaper and much easier to deal with it before you're on the front page of the New York Times, before you have letters from senators. You know, it's a very reasonable expense and companies are probably already budgeting for, in their traditional sort of software budgets. Whereas it can explode on the other side of the instant. As, as Andrew likes to say, on the right side of boom, the cost can explode. Right, because then you're talking about reputational problems, potential regulatory problems, potential litigation problems. So I'll let, I'll let Andrew chime in and correct me if I'm wrong.

On the right side of boom.

Yeah, yeah, yeah. So we have, in the national security world, there's a left of boom, right of boom. Boom is the bad thing. And left is before and right is after. And so, I mean, so your question, I think, is, is one of the most frustrating parts of some of what we do. And I think in terms of folks in information security and data protection writ large, which is there's kind of this intuition, well, if something bad hasn't happened already, why should I be spending time and money on it? And I think that is even worse in the world of AI, because AI is subject to so much hype. And we still see people who have this kind of belief, well, my AI could never be wrong. My data scientists are so expensive, they could never do. And honestly, it's just, it's a real problem. So what we typically say in practice to our clients who haven't yet had a bad thing happen. There are precious few is basically we say, well, there are two answers. One is you need to, like Patrick said, you need to at least be prepared for the bad thing to happen. You need to know what the bad thing is so that when it happens, it doesn't occur for a year until someone tweets about it and then there's an investigation. So you need to be able to know what it is you want to avoid. How are you going to be looking for it and measuring it, and then what are you going to do when you measure it? And so those are kind of typically, that's like the starter package, you know, that's like really kind of like the baby steps. And anyone, any organization deploying AI needs to have those things worked out. One, it's the right thing to do, and it's just like, responsible. But two, it will save them from a world of hurt if and when something goes wrong. The second thing that we say is that, honestly, just in terms of the financial resources, it just does not cost a lot of money to prepare, and it costs a huge amount of money to respond. And so one of the things that we'll do is we'll say, you know, you're investing in AI because you believe it's transformational, because you believe it has so much value and the liability is directly tied to the value. You would not, you know, like, the greater the value, the higher the chances that if something goes wrong, it's going to be big. And so you really can't separate the two. So if you believe that AI is worth investing in and you believe it's going to change your business, it is just one, I would say, irresponsible. We don't tell that to clients. We typically don't unless it's really bad. But it's just really misadvised and ill advised to be putting making such a big bet on something and kind of doing that blind. There's a quote that I love that I keep stealing from Patrick, which is that even microwaves have troubleshooting manuals, something as small and as simple as a microwave, and yet AI is deployed in practice all over without troubleshooting manuals, without plans for what happens when something goes wrong. And so I think, at minimum, there's just kind of this basic level of preparedness that organizations should be willing to invest in, and then hopefully they never have to pour money into a response. But it's a real issue.

Yeah. And there is definitely, like, I can see, like in this topic, people could also argue, well, like, I don't really know. I can't anticipate all of the bad things that could happen. But at the same time, there are some, like, simple things that, like, you can't anticipate. Right. Like, you can create adversarial examples pretty easily to, like, test the sensitivity and robustness of your model. Also, like, I think it was Andrew, you were talking about, like, when your model goes wrong and then, like, which users were impacted by this. Like, a lot of people just like, throw their model up, like model PB on S3 and that's the name of it. And then when they update it, they just, like, overwrite that file. Right. And like, of course that's going to create all sorts of amazingly terrible issues when you have to figure out something like what you were talking about. So even just like a simple thing, like, hey, could it be conceivable that I would need to know which model operated on which data from users? Yeah, that's probably fairly conceivable. That's something I've seen, like, People tend to make that excuse, too.

I'd really like to jump in here, and I'll try not to be too negative. You know, having. Having spent four years in Silicon Valley, data scientists are paid a lot. Okay, a lot more than, like a general practitioner physician. And I'm starting to have sort of personal, emotional problems with this idea of someone who makes 200, 300 grand a year saying, I can't think about how the system's going to fail. Okay, well, one, just take some time and Google about it and see how systems like yours have failed. And just open the newspaper. There's evidence of discriminatory machine learning all over the news. So again, I'll bring up this idea of studying AI incidents, much like people study airplane crashes and continue to study airplane crashes. I use the analogy of nuclear power plants or nuclear reactors and airplanes, and I know those aren't exactly right. The barrier to entry is a lot higher. The impact of a failure is a lot more immediate. But there's things to be learned there. And so we studied the ways that airplanes crashed in an effort to make them safer. And that was just part of the profession of aviation. And I'm really personally becoming tired of sort of. Of why data scientists would not have a, you know, have some kind of out for this. Like, oh, I get paid too much to do this. Like, that doesn't make any sense.

So I guess as we come to the conclusion today, I was wondering if you could kind of tell us, as insiders at this juxtaposition of law and AI, what you're expecting to see over the next few years. You know, clearly this is a new field. It's growing rapidly. What are you seeing and what do you expect to grow into as you move forward with your new firm?

Yeah, so I would say, and it's been interesting because we also kind of coming out of the gate, have been in discussions with a whole host of regulators. I think, you know, there's a direct connection between the headlines about things happening when AI goes wrong, and those are only a fraction of what's actually happening out there. Then kind of public concern about this and then regulatory reaction. And so regulators are going to kind of up the ante, so to speak, in terms of what the liabilities are. And so I really see two futures, and they're both probably better than the one, you know, the present, I think the one that I'm gunning for and why Patrick and I, you know, launched this BNH AI, this boutique offer is because I think there's an opportunity for data scientists to work, work together with policy folks and legal folks and get this right and kind of be proactive rather than reactive. Our aim is to help that happen. So I think that's one future where before there's any really major incident, we kind of as this, I'm gonna say community, but it's not really community. It's not like a legal, but as you know, as a group can start to build out best practices. And even Dan, what you were saying about just like documentation and like not overriding models when they're updated or at least, you know, like just some basic, basic best practices, I can see a future that is frankly a lot less bumpy and where maybe like the hype of AI can actually be, you know, be met in practice. I also see another future which is probably most likely. And I think what kind of Patrick's like frustration is indicative of is where regulators just say this is not good enough and kind of slap down some, some much more stringent standards. That's where things appear to be heading, although it's not inevitable. But if I had to guess, I would say kind of just like we were with, you know, frankly I was going to say gdpr, but I think CCPA is a little bit more stringent. CCPA is this very overbearing privacy regulation that came through California that now basically affects almost any big organization using data in the US I think the likelihood, if we're not careful is that there is going to be the same type of thing and then every single data scientist is going to have specific additional trainings and they're going to have specific additional forms and they're going to have kind of lawyers breathing down their neck who might not necessarily kind of fully understand all the nuances of their day to day. So I think no matter what the life of a data scientist is going to change, risks are increasing. It's going to be harder just to deploy models without illustrating how those risks have been reduced. And I think how that happens is really kind of up to us. And I think smart companies, you know, obviously biased, but I think smart companies and organizations I think will start to think about this stuff now so that they're not surprised when all this happens for sure.

Yeah, sadly I think I'm more aligned with Andrew's sort of second scenario. I think we're in for a bumpy road where AI and ML are on a collision course with the law over the next, I don't know, decade or two. I think it's important to mention in this topic that we see government agencies internationally, of course, Singapore, uk, all over Europe issuing very detailed AI guidance. But we also in the US just this year and end of last year saw ftc, cftc, cfpb, FDA and probably several more that I can't remember off the the top of my head sort of releasing draft guidance or other sort of steps towards regulation. So I do think sort of regulation is imminent and sadly I expect it to kind of be a bumpy road. But I hope Andrew's first scenario is what actually happens. That's certainly that would be a more pleasant scenario.

Yeah. Hopefully we've got a good number of listeners on this episode who are interested in the practicalities of this. That and I think this is super practical in the sense that, you know, hey, this is a the call to our listeners. Before we go down that second path, let's take stock of what we're doing and institute some responsible practices in our own workflows. You know, there's simple things like, like we were talking about that you can do. I've taken down the links that we've been chatting about. You know, super practical things that we'll link in the show notes. Specifically, there's a great page that BNH AI has put together with a lot of links contained in that page and the AI Incident Response Checklist. We'll include that right at the top of our show notes. So definitely check that out and take stock of, you know, how your organization is approaching this and hopefully we can follow that former happier path as practitioners. So yeah, definitely check that out. Thank you both Patrick and Andrew for joining us. This has been super interesting and really important discussions, so we appreciate that and wish you luck with the new firm and all the great things you're doing.

Well, thank you so much. It's been fun as always and I hope it's been helpful to your listeners.

Yeah, thanks for having me. This was a fun discussion.

Do you have questions, praise or constructive criticism about the conversation you just heard? Comment on this and every episode of Practical AI on changelog.com just open your show notes, follow the Discuss on Changelog news link and let your voice be heard. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santo, that's me. And our music is provided by the mysterious Brakemaster Cylinder. We're brought to you by some amazing sponsors. Special thanks to Fastly Linode and Rollbar and a special shout out to those listening on our our ad free Changelog feed. If that's you, you're awesome. If that's not you. Well, you're awesome, too. But you can learn all about it@changelog.com. that's all for now. We'll talk to you again next week.