#154 Building Ethical Machines with Reid Blackman, Founder & CEO at Virtue Consultants

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead. Covering everything from data and AI literacy for busy executives, to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

How things actually get done in the organization, or how they ought to get done depend very much on what the organization is like. So check your model for bias. Have a cross functional team involved, okay, but the real work, the real mitigation is in the details, not in the sort of high level best practices. It's in how do we actually, given how our organization operates, given the personnel we have, given the regulations to which we already have to comply, given our existing policies and governance structures, and given all those things, how do we best implement those so called best practices? That's where the action is. That's where things get difficult.

Welcome to Data Framed. This is Ritchie. In this show, we've spent a lot of time talking about the benefits of AI because they're real and amazing. Unfortunately, using AI can also cause problems, either for your business or your customers or society in general. That means that organizations making use of AI need to do so in a responsible way. And that means thinking about the ethics of AI. While ethics for AI sounds like a brand new problem, the good news is that philosophers have been thinking about ethics for thousands of years and thinking about ethics for AI for decades. So while the problems may have only recently hit the mainstream, I've been pleased to discover that there's already lots of well established good practice to follow. Our guest is Reid Blackman, the founder and CEO at Virtue Consultants, where he implements responsible AI and AI risk programs for enterprises. Reid was previously a philosophy professor at Colgate University and he advises the Canadian government on federal regulations for AI. Reid also hosts the Ethical Machines podcast. In short, he knows an awful lot about the ethics of artificial intelligence in business, and I'm very excited to hear what he has to say.

Hi Reid, thank you for joining me on the show today.

You're welcome. It's my pleasure.

Excellent, Excellent.

Let's find out.

Yeah, let's find out.

Cool.

So to begin with, I just want to give a bit of background context. So can you Give me an overview of what the major concerns are that organizations face when they're trying to adopt AI.

Well, it's changed. Now that generative AI is on the loose, things have changed a bit. The three big ones used to be, and to some extent still are things having to do with ethically discriminatory or biased AI, privacy violations and black box models. Those are the big three. So when I wrote my book Ethical Machines, those are the three that I focused on. Now that generative AI has come along, and here I'm talking about both generating images, video, text, like large language models, like GPT. Now those three risks still apply. There's still the potential for bias in black box models and privacy violations with generative models. But now we've got other kinds of concerns as well. So now we have concerns like worrying about the conversational agent being manipulative. We have obviously IP violations that hook up with the privacy violations. We have concerns around the appearance of LLMs, large language models, chat boxes. They appear to be deliberating, appear to give explanations for why they're telling us the things that they are. But they don't actually do that. There's the problem of hallucinations, what gets called hallucinations, that it gives false outputs. So that's to give you a flavor of some of the big headlines. And that's just. I'm focusing when I say those are the big ones, those are the big ones. For corporations, there's other kinds of things that people tend to worry about, or some people worry about. Not sure that I'm one of them, but some people worry about the existential threats of LLMs. We're all going to get dominated or killed or both by our robot overlords. There's concerns around the spread of misinformation, which is a concern for social media companies. But I wouldn't say it's, you know, for corporations generally. It's certainly concern for government job loss due to automation. That seems to be a concern for many people. Again, I don't think it's obviously the responsibility of business to account for them. But those three headline risks, existential threat, mass job loss, and spread of disinformation, those are three other really big concerns. But those are not the main concerns of the vast majority, not all, but a lot of my corporate clients, because it doesn't not within their wheelhouse to fix those problems.

Yeah, I guess most businesses are not necessarily going to be concerned with the threat of humanity becoming extinct due to AI.

If you're selling ketchup or clothing you're not going to do much about it. If humanity's coming to an end, they still have to be clothed. So, yeah, nothing the gap is going to do about it.

I do like that there's a sort of fairly well defined list of problems and I'd love to get into some of these in more detail later. Before we get to that, do you have any sort of specific examples of where AI has been misused and it's caused a problem?

Well, I'll give you one where it didn't cause a problem, but it could have. And it kicked off one of the major things that kicked off investigations into bias or discriminatory AI. So years ago, 2017, 2018, something like that, Amazon developed a resume reading AI. So, you know, we get tens of thousands of resumes every day, we can't possibly go through all of them or not efficiently. Let's have an AI quote, unquote, read those resumes and figure out who to interview and who not to. The thing turned out to be biased or discriminatory against women. So all else equal, it greenlit men's resumes, red lit women's resumes, and they tried to mitigate the biases of the model. They couldn't do it, so they threw it out. So this is a really well known case for a lot of people, this sort of shows, oh, look what Amazon did. But it's actually in some ways a success story because yes, they built a biased model, but number one, they had the good sense to actively look to see if it was biased, two, they actually found the biases, three, they tried to mitigate it. And four, when they couldn't sufficiently mitigate it, they actually threw out the project after having worked on it for two years, which that's a lot of time and money to toss. Maybe not if you're Amazon, it's not that much money, but it's still money. And so there's many ways in which it's a success story because they actually, at the end of the day, they did the right thing. But it also shows how easy it is to accidentally create biased or discriminatory AI and how difficult that it can be to sufficiently mitigate that risk.

That's actually a really interesting framing of it because I've heard that story a few times and a lot of the time it is just, oh yeah, Amazon did this stupid thing with AI and it was bad. But actually you're saying that they really did do this in quite a responsible manner.

And yeah, I will say just, I don't know I guess I'm supposed to disclose that they're Amazon as a client, but I wrote the same thing in my book and that was before they were a client. But yeah, it's back then. The fact that they thought to check for this was before the conversation around AI ethics exploded. The conversation on AI ethics exploded. Well, arguably it really exploded the past six to nine months. But even when people were talking about biased AI quite frequently, 2019ish 2020, when it really started picking up steam, got outside of academia and people were really talking about it, there's exceptions to that, but I mean, these are data scientists and data engineers, or AI engineers, whatever you want to call them, that thought to look into this before was the cool thing to do, so to speak. So it is a success story. And those data scientists were actively trying to mitigate the biases. And it turns out to be a technically difficult problem. It's not just, it's not a matter of being pure of heart that doesn't fix your models. It's. There are certain kinds of strategies and tactics that can be quite technical in nature that need to be successfully executed on. And sometimes you can successfully execute on it due to limited time resources, due to limited research into the nature of the problem and potential solutions that were available at that time. So, no, I think it's a success story. It's a big warning sign. It's a red flag. But that's an AI ethical risk success story.

Are there any ways upfront. So before you spend several years developing this AI and then discovering that there's some kind of bias, there. Are there any ways up front of judging how difficult it's going to be to have an unbiased model or to have some sort of responsible AI model?

Yeah, there are various things you can do. I mean, one thing that you want to first of all do is think about the way that it might be ethically problematic during the concept phase, when you're just trying to think about what's the should we build an AI solution and if so, what should it look like? That's a good place. I mean, if you do it after you built and tested and validated and you're just about to deploy or just about to transfer control to your client, if you, let's say you're an AWS and you built it for a client, then you might find the problem and say, oh my God, this is a problem and I have to go, either you got to ship a bad product or you got to go all the way back to the drawing board, which costs lots of time and money and probably broken contracts, broken promises for when something's going to get delivered. The best thing to do then is to start it at the very opening phase of thinking about, of the AI project itself. And then there's other kinds of things you can do aside from thinking about, okay, what are the ways that things might go ethically wrong? Let's say you're talking about bias in particular, which gets a lot of press. You might start thinking, okay, well, one potential source of discriminatory output is the training data. So let's take a look at the training data, let's take a deeper dive of that, let's explore the data, let's analyze the data and see if we might find potential for biased outputs there. So let's take one simple example. It's well known, let's say you're building facial recognition software. You take a look at your data, the training data is going to be pictures of people's faces in particular. And you might take a look and do an analysis and figure out that, oh my God, only 0.01% are black women. This thing's going to wind up to be terrible at recognizing black women. So we, we're going to need to up that training data. We need to get more data, more examples of pictures of black women's faces in various lighting conditions, et cetera, et cetera. So you can do some data exploration. That's. That's one way.

So really it is all about just making sure that you understand what you're trying to do at specification phase. So it's cheaper to solve the problems early on rather than later after you've started developing things. Is that. Yeah, yeah.

I mean, it's, it is complicated, though. So there's no such thing. As far as I'm concerned, there's no such thing as intrinsically biased data. It's not like data sets are sitting around with a bias halo around it, the bias anti halo around it. A lot of it depends upon the context in which you're going to deploy the thing. So to take a sort of toy example, suppose that you're going to use facial recognition software and you're only going to use it in this town in northern Sweden or something, where everyone has very pale skin and they have a no visitor policy. So, no, no, I'm, you know, making up a toy example. And you want to use facial recognition software in that little village or whatever. It would be bizarre to come. So let's say you build it. You use just the Faces of people who look like the population on which you're going to deploy it. It would be weird to say, oh, it works perfectly, but it's biased or ethically discriminatory. So you need to retrain your model so it could recognize, say, the faces of black women or Asian men or whatever it is. That would be where it's not ethically discriminatory. It's not having negative ethical impacts. So that's to say that the context matters. It's, it's the training data used in such and such a way, in such such a context can produce discriminatory outputs. And so we better take a closer look at say the training data or how we weight the variable, the input variables, or where we set the threshold, or et cetera, et cetera. So it's not people say, oh, we need to get rid of biased AI. We have to make sure that we have representative data sets. That's just a very narrow way of thinking about bias, advice, mitigation and AI.

So really it's about getting the right data set for the problem that you're going to be using.

That's part of it. Part of it is the right data set, but also it's things like how you weight the input variables. Part of it is the threshold. So to give you an example, let's say you're doing credit rating scores across various subpopulations. And so if they score above a, I'm just going to make up arbitrary numbers. If they score above a 53.5, they get a thumbs up, they get credit, they score below a 53.5, that they get a thumbs down, no credit. And it turns out that when you set the threshold at 53.5 for yes, no, you get credit, no you don't, it's going to have a certain kind of distribution of credit across various subpopulations. Black, Asian, white, black woman, Asian women, Asian men, Asian women, et cetera, et cetera. So you're going to get this variable distribution across subpopulations and maybe it turns out that distribution is discriminatory in some fashion. It may be though, that if you move the threshold to 62.1, again just choosing an arbitrary number, you move the threshold so everyone above a 62.1 gets a thumbs up, everyone below it gets a thumbs down or whatever. Maybe then the way that it distributes that credit across various subpopulations is non discriminatory. So we haven't trained, we haven't changed the training data, we changed the threshold in order to differently distribute Credit across various subpopulations in this way stipulating in a non discriminatory manner. So there's various ways of getting at the problem. There's various strategies or tactics you can use to mitigate an ethically biased distribution.

So it seems like this is an example of how you make your AI fair or have some sort of sense of justice about it. So in general, other steps you can take like to mitigate discrimination from AI, how do you ensure this sort of like level of fairness across all these different sub populations?

Yeah, so one thing I might highlight first is that I think at the minimum we're aiming for, not unfair, there's probably sometimes people talk about what's fair as though there was just the one fair distribution. I think that there's reasonable disagreement about what constitutes a fair distribution. And what people should really focus on is what constitutes what's discriminatory, what's clearly unacceptable. Right. So for instance, if you find, as ProPublica analysis did, if you find that your model, your AI is predicting that black defendants are twice as likely to commit crimes within the next two years as white defendants holding fixed all other variables, criminal history, socioeconomic status, educational accomplishments. If you hold those although steady and black people are still predicted to be twice as likely, you might think, okay, that's, I don't care how you think about what constitutes the fair and the just with a capital F and a capital J. It's clear that's just ethically unacceptable. It's clear that's unfair, it's discriminatory. So it needs to change. So I think that companies should think a lot about what constitutes unacceptable as opposed to what constitutes the ideal of justice or something along those lines. And then there's going to be various ways that you think about that. So one way that people, especially computer scientists, data scientists, think about fairness, when it's distributing goods across subpopulations like credit or mortgage lending or interviews for a job or admission to a university, there are various, these various quantitative metrics. So you see how it's distributed across various subpopulations. And then you get a score according to various kinds of ways of scoring for fairness. There's something like two dozen plus ways of scoring or metrics for scoring fairness. You can't score well on all of these metrics at the same time. They're mutually incompatible. So if you score well on one test, you're going to score poorly on another and vice versa. So then you have this question, okay, well then which metric is the appropriate one? And another way to frame that, my preferred way is what are the metrics or what are the ways of measuring fairness that we think are inappropriate or unacceptable? Let's take those off the plate. And now we have these remaining set of metrics that are reasonable. And then I think you can choose the metric that is compatible with other kinds of concerns that you have, like accuracy of the model.

That's really interesting that there are mutually incompatible sort of measures of fairness. Can you give me an example of what one of these sort of measures is? What's the sort of difference?

Yeah, so it's going to be unfortunately, a little bit abstract, off the cuff, but suppose you think about fairness as we want to make sure that true positives are distributed equally across various subpopulations. So if someone, say, deserves credit, then they get it. And so whether you're white or black or an Asian woman, or an Asian man or a white woman or whatever it is, that the rate of true positive is equivalent across all subpopulations. On the other hand, you might think we want to make sure that false negatives are equal across all those populations. And so then what you might do is you might set your AI to equalize false negatives across those populations. You can't do both of those things at the same time. By equalizing false negatives, you're going to make unequal the true positives. And by equalizing the true positives, you're going to make unequal the true. What did I say?

False negatives.

I think. Thank you. I mean, that's the. That's an abstract way. I mean, that's not a. That's not a very helpful. That's not an example as such, but it's the structure of an example.

Okay, yeah, I get it. So basically you can either try and optimize for one thing where good. Whether the right people get the good things and. Or you can optimize for the wrong people not getting the good thing or whatever.

Yeah. Or you're minimizing false negatives or something like that. And then you optimize for that, but you can't optimize for both at the same time. Okay.

And so related to this idea of fairness is accountability. When do you need to care about accountability for AI decisions? So if an AI makes a bad decision, then there are some consequences to say, okay, let's not do this, or let's try and mitigate those bad decisions.

One thing to highlight first is that the fairness biases are just one issue. I mean, I rattled off a bunch at the top of the program. There's a bunch of ethical risks, bias and ethical discriminatory AI gets the lion's share of attention, at least it did before generative AI. And that's for arguably good reason. I mean, it's a pretty big deal and it's very easy to accidentally create it, but it's really the set of ethical risks. How do we think about accountability with those? So I don't want to limit the discussion of accountability to just be about biased or discriminatory AI, because it really applies to any ways that things can go ethically sideways. Now look, if the organization builds an ethically problematic AI, goes ethically off the rails, who's accountable, who's responsible? I mean, most generally it's going to be senior level people. I'm generally not a fan of thinking that it's really primarily the data scientists fault if the organization doesn't provide those data scientists and data engineers, et cetera, with the resources to engage in appropriate ethical risk identification mitigation. It's not on the data scientists to perform a herculane effort to do that work despite compensation, despite unchanging deadlines, despite lacking the tools, the training, et cetera, to engage in that kind of stuff. It's up to the organization, senior level officers, someone in the C suite, like a chief data officer, a chief analytics officer, a chief information officer, a Chief Technology Officer, etc. Someone senior level to own an AI ethical risk program, who empowers those, you know, frontline defenders, data scientists, data engineers, to engage in the kind of ethical risk due diligence that they ought to do and to provide them with the resources to not just perform those analyses, but to do something about them, to actually devise and execute on strategies and tactics that will mitigate those biases. What's more, it's. It's incumbent upon those senior level leaders to recognize that data scientists can only do so much. They have a very specific area of expertise, data science, or something along those lines, whatever you want to call it. And it turns out, I think that there's other kinds of expertises that are relevant to identifying and mitigating those risks. So yes, data scientists are part of the picture, but you also might need legal in the room, you might need an ethicist, or in the room you might need a, and this is a kind of lawyer, like a civil rights attorney in the room, you might need a sociologist, you might need a representative of the culture in which you plan to deploy your AI or something along those lines. So there are going to be many Cases, especially when they're high risk applications of AI, where you need a cross functional array of people thinking about the ethical risks of an application of AI and appropriately mitigating those risks. So if those people aren't present because senior leadership hasn't built the kind of infrastructure that could be accommodated, then again, you can't expect your data scientists to do more than they can, given their training, expertise, etc.

I think that's a really great point, that some of these skills around ethics are maybe not something that data scientists have been trained in by default. And so you do need someone with a, I guess, is it like a philosophy background? Is that where most ethicists come from?

I think that's not inappropriate. I think that can be perfectly appropriate. I'm an ethicist by training. I have a PhD in philosophy. I was a philosophy professor for 10 years. I've been researching, publishing and teaching on ethics for over 20 years. But I'm not going to tell everyone, go hire an ethicist. They can be relevant and sometimes they're not. Let me give you an example in which I think they are relevant, which we've already covered the bias issue. So you got these quantitative metrics for fairness, different ways of thinking about fairness. What constitutes unfair? You want to ask, okay, what are the, and I said this earlier, I got glided over it a little bit. But what are the appropriate ways of thinking about fairness in this particular context, for this particular use case? And what you mean by appropriate can vary. Maybe you mean ethically appropriate, business appropriate, reputationally appropriate, legally appropriate. There's different ways of thinking about what constitutes appropriateness. And my general view is that you need those different ways of thinking about what's appropriate represented in the discussion about how do we handle this, this AI responsibly. But you can't expect your average data scientist, or 99% of data scientists to give an effective analysis of what constitutes ethically, legally, reputationally and business appropriate. That's just beyond their ken.

And so once you've got data scientists and legal people and maybe an ethicist and maybe some other business people and they're trying to make decisions on this, you've got a lot of people from different teams. So have you seen any organizational structures within businesses that sort of help all these people come together?

Yeah, sure. I mean, look, you don't need this for every single use case. One thing I want to stress is that it's not the case that for every single AI you build, you need to have this cross function. No, no one's going to do that. It's too much work. It's going to slow things down too much. And so it's really a matter of one is at least training data scientists, product managers, to understand when is something high risk, when does it need to go to that kind of committee? So there's different ways of doing risk scoring. You train your teams to do proper risk scoring. One score means go, one score means talk with a product manager about it. One score means go to the ethics committee, but keep working on the project. One means red, stop everything, don't do anything else until you talk to these seven people, whatever it is. So it's not the case that I don't want people to think, oh my God, this is such. We gotta get the Avengers together every single time someone builds a model. That's ridiculous. Of course, that's ridiculous. So it's really the high risk stuff. The thing that works well is to leverage existing governance structures, committees, risk boards, whatever it is. Different organizations operate differently. So for instance, some have very strict, very good, robust governance around privacy where legal might be particularly involved. So maybe you take those existing committees or people around legal, you augment them with say an ethicist or you do additional training for them and then off you go. In other cases, there's a really good compliance board. And so, okay, it's not, it's not the privacy stuff. It's a regulatory compliance board. We're going to leverage that board. We're going to add new responsibilities to that board. Maybe we had additional training, maybe we added an additional role or two. There's not a sort of one size fits all for every organization because they're just different organizations have different priorities, different risk appetites. They are structured in different ways, their policies vary. Some are really privacy forward, some are not at all. You have to add, you have to add to existing governance in a way that's commensurate with how the organization already works. Otherwise no one's going to do it. Otherwise it's just, it's too disruptive.

Okay, yeah, I mean, I can see how trying to get a lot of people from different teams together for every single AI use case is not going to be very scalable at all. And I can imagine how using AI to, I don't know, create a new filter for your selfies or whatever, then that doesn't really require a sort of ethics committee. Do you have any sort of guidelines for, like when you do need to get the Avengers together and care about this stuff?

This is not exhaustive, but I think one of the main criterion is something like, does the AI obstruct, hinder, block people's access to the basic goods of life? Something along those lines. So if we're talking about jobs, you need a job to make money, to pay your rent, to pay for food, to pay for your kids, various, whatever, just, well, living expenses, frankly, just for kids, let alone extracurricular activities. So jobs, credit or lending for mortgages, healthcare and life sciences, often directly relevant to people's ability to live a minimally decent life. So if it's things to do with the basic goods of life, by which I mean the things people need access to, regular, systematic access to in order to live a minimally decent life, then it's probably high risk. If you are making a prediction about when the screws are going to arrive at the toy factory. All right, no avengers necessary.

So really it's all about the consequences of this and who's going to benefit, who's going to, who's going to be harmed by this and how much.

It seems like, yeah, I mean, here's another way to put it, another kind of standard that's very similar to the one that I just articulated, but probably not exactly overlapping. At least the Venn diagram. If it's potentially going to violate someone's humans rights, that would be a pretty big red flag. That seems pretty high risk. If this thing potentially violates human rights, let's get it. Let's someone call Captain America.

Absolutely. I can imagine how that's a pretty serious concern for anyone involved in creating that sort of AI. And so I think sometimes when businesses go, okay, let's implement this new AI feature, the focus is going to be around like what are the sort of the benefits or the harms to the business itself. Do you have any advice on how to make sure that businesses start thinking about like what are the benefits or harms to their customers or to other people outside the business?

I mean, look, if your company is using AI, then you just have to start thinking about the risks involved. It'd be silly not to. Yes, think about the business risks. I mean, there are business risks to it. I mean, it can be a significant investment of capital or at least a significant investment of personnel, which is boils down to, for a business is significant risk to capital. So there are risks, but like I said, there are ethical risks. There are legal risks. Given generative AI in particular, which we haven't really talked about yet, there's lots of risk. One of the things about generative AI is that, well there's two things. One, it's general purpose, so it could be used for all sorts of things as opposed to non generative AI or AI circa 2020, earlier first 3/4 of 2022, you know, it's task specific. It just does risk ratings for diabetes or you've got cancer or you don't get an interview. It's very. But things like LLMs or generative AI in general are general purpose, so they could be used for all sorts of stuff. And the use cases, we don't even know. There's countless use cases as far as we know. And so organizations don't know how people might use generative AI. That's worrying. And secondly, it's just that relatedly, everyone in the organization has access to it. Everyone gets to use it. It's not just data scientists, far from it. It's not even that data scientists within your organization have approved it for use by your internal people. No, it's just Bob and HR decided to use it and load on this HR data onto the thing and now it leaked and now there's a big problem. So at this point it's bordering on negligent, frankly, not to have some kind of governance around how your organization does AI because the risks are there. I mean every organization has sensitive information, for instance, that they don't want leaking out. Every organization does sales and could potentially use a chat bot for sales in a way that is manipulative, objectionable in some kind of way. So yeah, there's prob. You know, problems abound. They're not, I think it's. You can get your hands around them. Some people think, oh my God, it's so complicated with nothing to do. But no, they'll say things like, oh, we need to do our research, we're going to look into the. Nah, we sort of know. We people have been working in this field for a while. We know what to do. It's just a matter of political will to do it. And that political will incidentally is increasing because leaders are anxious about the impacts of generative AI.

That's good that leaders in general are concerned about this sort of thing. And it sounds like a lot of the best practices are fairly well established. So can you maybe talk me through some of those things?

I wouldn't say. I mean, I don't like the phrase best practices. It makes it sound like a list of things to just do and then you're good. But the best practices are necessarily high level. I mean they're necessarily generic. Check your models for bias, make sure that the training data doesn't violate anyone's privacy or ip. Don't use the black box model in very high risk situations. We could like rattle off some quote unquote best practices. But number one, insofar as best practices are agreed upon by the relevant community, that this is what is good. We don't have that. We just don't have that kind of universal or widespread consensus unless the consensus is so high level to be almost meaningless. The other thing is, as I was saying before, is how things actually get done in the organization or how they ought to get done depend very much on what the organization is like. So check your model for bias. Have a cross functional team involved, okay? But the real work, the real mitigation is in the details, not in the sort of high level best practices it's in. Given how our organization operates, given the personnel we have, given the regulations to which we already have to comply, given our existing policies and governance structures, and governor, given all those things, how do we best implement those so called best practices? That's where the action is, that's where things get difficult. Not, yeah, I would say difficult, I suppose doable, but not clear. The best practices are easy to articulate insofar the best practices are extremely high level. Like vet your models for ethical risks, mitigate the ethical risks, don't violate people's privacy, make sure that your model's not discriminatory, don't engage in manipulative ways, et cetera, et cetera, et cetera. Check your generative AI for hallucinations, for false outputs. No kidding.

All right, so it sounds like there is a sort of very high level set of best practices, but actually the devil's in the details. The million dollar question then is like, well, how do you stop Bob from HR just posting employee data on ChatGPT?

So one take on this is let's just shut it down within our organization. Let's just make sure that people can't use GPT. We're going to create a firewall or whatever it is around various kinds of access to LLMs. I think this is not a good way of doing things. I think it's similar to telling teenagers to be abstinent and then providing no sex education. It's just a bad idea. Bad things are going to happen because they're going to do it. Both those teenagers and your employees, they're going to use it. They have their own phones, right? They've got their own computers, they're going to use it. So I think one, you've got to get better, more clear, articulated policy. Here's at least two main things, maybe three. One, you gotta front load. Learning and development. Most people don't know much about generative AI. They don't really know the ethical risks of AI. They don't know the ways in which it can generate AI. They don't know ways in which you can go wrong. So having to educate enterprise wide becomes more important than it was pre generative AI. Because pre generative AI, your data scientists owned it and they distributed it, they gave it to people, they could give them instructions. Now with general AI, since everyone has access to it now, we don't have any kind of real control over the flow over who uses these tools. And so L and D becomes really important. So front loading LND number two, getting an inventory of who's doing what. So provide those people and tell them about this in your L and D program. Here's what you should do, here's the what to look out for and here's who you tell that you're doing it so that we have an inventory and understanding of what's going on within our organization. Otherwise, who knows? I mean you've got thousands, tens of thousands, in some cases hundreds of thousands of employees potentially using this thing. You have no idea who's using it for what. And it's going to be so use case specific. And Bob in the background of HR is like, oh, I can use it for this thing, right? That some C suite executive would have no idea that. Would it be that that's a problem that the HR person faces that Bob faces that he might use gender AI to solve. So make sure that there's a method of inventory and third and related to the inventory piece, making sure that you have some way of getting them to smell ethical smoke, understanding that they're dealing with some kind of potentially high risk thing or potentially using high risk data, then the means by which they can contact the appropriate risk board or person, whatever it is, to get guidance on how to use this thing appropriately.

I like that you said that people need some kind of training around using AI. Are there any particular teams or roles that you think this training is particularly important for?

No. I mean this is the thing with generative AI. It's for everyone. Hr, marketing operations, product teams, sales teams, it's everybody. And so at least some kind of generative AI ethical risks 101 that is deployed enterprise wide that everyone gets. It's not role specific because it's just everyone. Any role, literally any role in your organization can use it for something again, that's different. It's different with non generative AI, then, okay, you do need specific training for say, data scientists. Right. Your data scientists need to go an extra level deeper. But there's AI Ethical Risks 101 and then there's AI Ethical Risks 201, and then 201 is customized on a role by role basis, or at least department by department basis, or business unit by business unit basis. But no, for generative AI, it's everybody trading for everyone.

I like that. So one thing we've not really talked about, you've mentioned bit, but is around data privacy. Because I think this is perhaps one of the biggest concerns, especially with generative AI, that someone's going to post some sensitive data. Can you just give me a bit of an overview of one of the different ways data privacy can go wrong with AI?

Okay, so here's a couple of ways. Number one, you use private data, let's just not even say IP yet. Just people's private data, maybe health data, for instance, in order to train your AI, so you've acquired and use that data in a way that violates their privacy. That's one way. Second way related to privacy is the IP stuff. So you use texts by authors, images by artists, and you use that to train your generative AI. So now you've got IP concerns. Another way is a lot of these things can be used as chatbots. They interact with people. The people put certain kinds of information into it and they don't realize that data is then being collected and used for various other purposes. So you collect new data with the chatbot. That mere possession and use can itself constitute a violation of privacy.

Yeah, you got the personal information and then you got the IP risks. And are there any particular ways that you can go about mitigating these risks?

I'll say one other thing is that the data might get leaked. I didn't mention that. So you might share data with generative AI provider, let's say an OpenAI or Microsoft or Google or whatever, and that data might get leaked in various ways. So this happened to OpenAI a few months back, where people could see other people's chats, their chat history, not the whole thing, I don't believe, but some of it. Or it saw payment information. So the other issue is that there might be nothing intentionally nefarious going on, but there might be a negligent handling of the data, which would then result in privacy violation. Things to do. It's, it's all going to vary. I mean, some people are trying to come up with ways of, for instance, sending data to a third party vendor like an OpenAI in such a way that third party, that LLM provider, for instance, can't see the data. It's. I don't want to say hashed, but it's. I'm going to, whatever I'm going to say, it's encrypted in some fashion or other. So for instance, I was just at the conference AI4 last week and met a company where you can type whatever you want into the LLM interface on your own premises. It gets rewritten somehow. It scrambles the words in a way that it becomes gibberish, gets sent out to an OpenAI or an Anthropic or a Microsoft or Google. Then that LLM company generates the response, sends the answer back. And so all that company sees is the answer, but they don't see the question that was asked. So if there was any personally identifiable information or private data in that exchange, that OpenAI or Google or whatever can't see it because it was scrambled before it got there. That's sort of one interesting tactic. That company is, I believe, called Protopia. Pretty cool stuff. I don't have any financial connection to them, just to be clear. I just, I thought it was a cool demonstration. Okay. Yeah.

So really it's about like not passing your sensitive data directly. It needs to be scrambled or encrypted in some way before any other company is going to see it.

Yeah. Or you could have an LLM, a large language model on your own premises, on your own servers, and just never leaves. That's a different way of doing it. Right. So if you're not sending it out, then of course you're more likely to not share it with people accidentally. So there's different, different ways of doing this, but it's also just early days. There's not best practices for that.

So related to the idea of privacy is transparency of models. And you mentioned the idea that sometimes you can have a black box model that no one really understands, or maybe you have some kind of interpretable, easily interpretable or explainable model. Can you talk about when that's important?

There's some controversy here. I probably occupy, I think, a relatively unpopular position within the AI ethics community broadly. So first of all, I wanted to distinguish between explainability and transparency. So explainability is going to be what's going on between the inputs and the output. So you input some data like here's this person's medical profile or here's our financial history and it Outputs something like, this person's going to develop diabetes or this person's going to default on a mortgage. Don't give them credit. How did it arrive at the quote unquote decision? If you can't explain because the pattern that it's looking at is too mathematically complex to understand. You have a black box model, you have an unexplainable model. You might go around telling everyone, hey everyone, we've got a black box model that's being very transparent that you have a black box model. So that's transparency. On the other hand, you might have a model that you totally understand. Okay, we give these outputs, engages in these calculations, and so it arrives at this output. So I can explain to you exactly how it arrives at its quote unquote decision. But don't tell anyone. We've got this clear box or glass box model. That would be high explainability and low transparency. So transparency and explainability come apart. There are a lot of people who think we shouldn't have black box models in high stakes cases. It's too high risk. We don't understand how the, how the thing works. It's not safe to use. That's one line. Another line is, look, as long as you can show that it's sufficiently reliable, sufficiently gives outputs that are sufficiently reliable that maybe do better, it seems, performance wise than when, say, humans judge or when you have explainable models, then arguably it's permissible, ethically permissible, to use it. So, for instance, let's just stipulate the best doctor is 80% accurate when it comes to cancer diagnoses. I'm completely making that up. I don't know what the stat is, but let's just say 80%. Let's suppose you've got a black box model that's 99% accurate. Is it ethically permissible to use the black box model, especially if you have the informed consent of the patient? Probably.

So are there any cases where it's the other way around? And you would say, okay, we've got something super high stakes and we really shouldn't use a black box model here.

So there's one kind of case that gives me pause. So one thing that you might think about is AI in the criminal justice system. So one thing that we're supposed to get in the criminal justice system is what's called procedural justice. So the procedure is in order for the outcome to be considered fair. The procedure by which the outcome was arrived at was fair. That's why we have things like a trial by juries. The juries have to engage in this kind of deliberation. They're not allowed to take this into consideration, but they can take that into consideration, et cetera, et cetera. There's case law to guide them. The judge didn't do anything or didn't bias the judge, the jury or something. Right. So there's ways of ensuring more or less a fair procedure to increase the probability of a true outcome. But we take the outcome to be fair on the condition that the procedure is fair. So you might think that in some cases we need to have a fair procedure and that's the best we can get because we don't. If you, like, have the ground truth, you don't really know if they're innocent or guilty. We just have this way of doing it, of finding them innocent or guilty or guilty or not guilty by the procedure. Okay, but now suppose you have a black box model in that mix and part of the procedure occurs, if you like, between the inputs and the outputs of an AI, but you can't explain it. But if you can explain, if you like the procedures of the AI, then you can't assess whether the procedure was fair or not. And so that looks like potentially anyway, a case in which a black box model would be unacceptable because it would necessarily make it impossible to assess whether the procedures were fair or not. That's one case. In some cases, we don't care. Right. Like, so with cancer diagnosing, there's really much I don't really care. I don't care if it's a magic box, I don't care if it's a genie who just divines the truth. Like if they get it right, that's all I care about. But in criminal justice system, we care a lot about the procedure and the fairness of the procedure. So arguably in that case, procedure matters. And black boxes, which should be unacceptable, that's debatable too. But I find at least a plausible line of thought.

That sounds a lot like the difference between statisticians and machine learning scientists, where the statisticians want to know how everything works and do inference and machine learning scientists just care about do I get good predictions or not?

So yeah, and it might be the case that sometimes you want a machine learning engineer and sometimes you want a statistician.

Absolutely. So that's the explainability side of things. Are there any cases where you either really care about transparency or you might say, okay, this isn't important here.

Yeah, this is a good question. I mean, you look at any ethics statement, AI ethics statement, everyone's going to list transparency, which is silly because no one is going to be transparent about everything and no one's going to be transparent about nothing. And so you need to be transparent about the right thing at the right time in the right way, which is relatively uninformative. I think that there are probably cases in which high stakes AI, say life and death is on the line and you need to be transparent enough so that external stakeholders, let's say auditors in particular regulators, can look to see what kinds of decisions were made throughout the AI lifecycle and were they, are they defensible decisions. That would be a case in which there needs to be sufficient transparency.

Okay, so you need to really understand like how this thing's working then. Yeah, you're going to have to have some level of transparency there.

Yeah, I mean, sometimes you may or may not know how the model is operating. You might not know what's going on between the inputs and the outputs, but you can always know, at least in principle, what are the kinds of decisions before the inputs and after the outputs. There's lots of things to know. There's lots of decisions that data scientists and other product managers that are make around that AI lifecycle. So those things can be more or less transparent. And look, you're not going to give away your ip, so don't be transparent about that. If it's going to give away your ip, all else equal. But they're going to be some things that need to be transparent about so that you can be. So that we can see whether you're. So that we, the we as appropriate authorities can see whether your ethical house is in order.

Sometimes it's not clear whether your data is going to be used by the AI, and sometimes it's unclear what happens when you are giving permission for AI to make a decision about your life. When do you think that is important?

So different people will say different things to this. The go to line is that in all cases people should have control over their data. That's a standard line. So obviously in the high risk cases, but also even in low risk cases, because it's my data or something along those lines, that's a pretty standard line to hear. I'm not convinced of that position myself. I did a podcast on this with someone who does think that, Carissa Valise at Oxford. We had a really good discussion around this point where I tried to push back. Look, there are going to be some cases where there's highly sensitive information about you, where plausibly you should be allowed to Say, no, no, you can't have that data. Health information is one of them. Although I'm not even completely convinced of that. I mean, I'm inclined to think, let's say that there's a global pandemic and the appropriate authorities having access to certain health data of yours that's anonymized or aggregated and was used to engage in certain kinds of analyses to say, make certain kinds of predictions or build a cure or a vaccine or whatever, then it seems to me like, I agree that the individual has a vested interest in having control over their data, but the global community has a very strong interest in not dying. And so plausibly that individual's preference is ethically overruleable. That's going to be controversial though, obviously. So I guess I think that there are many cases where it'd be nice, at least ethically nice, people had control over what data different organizations had about them. I'm not sure that in all cases it's ethically required and then it's just going to be a case by case basis. I don't, I'm not sure that there's an obvious way of saying it outside of things like health information from corporations.

Let'S say, or I guess maybe financial information is possibly another sort of thing.

Yeah, yeah.

So for organizations who are wanting to get better at doing responsible AI, I guess what are the most common mistakes that organizations make that they ought to avoid?

They rush to implementation, they'll come up with some really high level ethics statement where for fairness, where for privacy, where for transparency, where for accountability. And it means nothing because it's so high level, it's so generic. And then they say, okay, let's, how do we implement this? First of all, you don't have anything here. You just, you have nothing. You could have written any words on that paper. It would mean this, it would mean approximately the same thing. The thing that people, that organizations don't do is they don't do the sort of. I don't care what you call. There's different ways of calling it a gap analysis, a risk analysis, a feasibility analysis. They don't get a good grip on what their organizational standards are and they don't get a good grip on where their organization stands relative to those standards. So one of the opening moves that we do with all of our clients, almost all of our clients, is we do that risk assessment or that gap analysis. Okay, what do your current governance structures look like? What do your policies look like? Personnel training, onboarding tools, workflow, what does the organization look like right now and what can be leveraged and what needs to be avoided in building out a customized ethical risk framework that suits your organization as it stands, if you don't do that, a organization don't know what to do, then they just start doing stuff and then they find that, oh, actually this doesn't play nicely with other parts of the organization. So for instance, let's just start building risk matrices and risk categories for AI ethics. And that's going to be our thing. And then you don't talk to risk about it, the people in Risk, and they're like, how does this fit with our enterprise risk categories? And if it doesn't, now we got decisions to make, brakes get slammed. Risk wants to get more involved. Now it gets complicated, confused, and then there's political factions and better to do that assessment early on involving risk and compliance and cyber and legal. Get everyone on the same page, what we're trying to accomplish here, and then things move along a lot more effectively, a lot more efficiently.

Okay, so it sounds like there's a sort of maybe a specific order you need to do things then. So you need to set up some sort of committee and get these teams involved to begin with, Is that what you're saying?

It varies. I mean, usually we're working with like a working group. I wouldn't call it a committee or it's not like an ethics committee or something along those lines. It's. Most organizations need some kind of AI ethical risk or responsible AI working group. Often that smaller working group reports to a larger AI ethical risk steering committee. But yeah, you work with a handful of people, risk compliance, legal, cyber, chief data officer or someone under that person's charge. You're working with them to do that, that gap analysis or that risk analysis. That's the way to start. Look, sometimes things, it depends on how fast an organization also wants to move. Sometimes we're simultaneously developing some learning and development, although that has to be done a little bit after you do that assessment, because you want to know what existing learning and development resources can we leverage. But you know, and you could also do some. Sometimes we're working with one client now, multinational, who wants to retroactively do some risk assessments of some AI models that they already have out there. And so they want to do at first, let's just do a quick and dirty pass on what's already out there, flag certain things as high risk, do a deeper dive on those things, and let's get that up and running because it's stuff that's already out there. We want to make sure that we're good while at the same time building our more systematic, comprehensive approach to AI ethical risk.

And are there any sort of process changes you think organizations need to make in general to do this right?

If we're talking about the level of process, the one thing that's going to be crucial is figuring out the processes by which you engage in those risk analyses at each stage of the AI life cycle. So it's not just something you just plug in at the end, just before you deploy, because if you do that, it's really hard to go back and fix things. The contract says it's due at this date. It's time to go. You find some big risk. Not much you can do about it except for ship a bad product or go back to the drawing board and deliver something late, which nobody likes to do, obviously. So making part of the building into processes throughout the life cycle, both the processes that teams engage in when they're developing the model, and also processes between teams. So like data collectors versus data scientists, for instance, when those differ, making sure that there's processes that facilitate appropriate communication and transfer information from one team to another, as the AI or parts of the AI get passed on from team to team, that stuff is really crucial. Oh, yeah.

Between team processes, that's always the hard thing. It's like trying to get different managers to agree on things. So are there any tools that can help you out with this? Like, are there any sort of responsible AI tools?

I mean, there are tools for various kinds of things. Like, let's do a quantitative analysis of the bias in these models. Let's do quantitative analyses of potential privacy violations. There are also certain kinds of platforms that allegedly help with this kind of thing. I haven't seen anything yet that I find particularly impressive and scalable. That's one of the big issues, is scalability of these things. Oh, sorry, that's not totally true. Some of those tools that do, for instance, quantitative analyses of whether the model is unbiased, those are easily scalable governance platforms. I haven't seen seen one of those yet. I mean, part of the problem is just that, for instance, among other things, a constantly evolving regulatory landscape and checking for compliance with those regulations is not something that can be just automated at scale. You need to have some people involved at some point. I'm not saying organizations shouldn't use these tools. I am saying, though, that there's lots of cases where organizations that think, oh, we'll just get a tool and we'll fix It. And that's not never the case. A tool is good. But to speak to the point earlier about accountability, a tool doesn't work well if it doesn't land in a fruitful environment in which it's going to be deployed effectively.

You can't just buy stuff and have all your problems magically solved.

Yeah, you buy the software, maybe you train a couple of people on it, but they don't train the others, then they leave. There's no specific place in the process where they're supposed to use that tool. There's no process by which there's any kind of auditing about whether that tool is being used and, or whether it's being used well or effectively or according to the proper instructions. So, yeah, companies spend hundreds of thousands of dollars on these tools, but they don't actually get, in my experience, they don't really get used or used well because they haven't landed in the environment in which data scientists, et cetera, actually have the support to use those tools.

So a bit of training and the right processes are needed as well.

And the last thing I'll say is that it also gives some clients, actually quite a bit of corporations, a false sense of security. Like, oh, yeah, we've got a tool for that, we're good. And I don't try to convince them otherwise because I'm not going to convince them that actually maybe things aren't so good. But whenever we do these assessments, whenever we do these risk analyses, one thing that we always find is it's not as buttoned up as you think. The tool is good at what it does, but it leaves lots of gaps available, open. Whereas most companies are like, oh, yeah, we got this tool, it looks at bias. So we have the bias thing down. Sort of like, no, you don't. You got a way of dealing with it. It's a good way, but it's not a comprehensive way. It's not sufficient.

Okay, so if tools aren't the answer, are there any ethical AI frameworks that you think are worthwhile looking into?

NIST has a really. NIST has, I think they call it AI Assurance Framework, something along those lines. I was impressed by the NIST framework. For lack. I really hate the word framework because sometimes framework means a. A pie chart with some buzzwords in it, and for some it's more robust. So I was impressed by the NIST framework. I read it and I thought, yeah, these people who wrote this, they know what they're talking about. And so there's something that's great about It. But number one, I understood that NIST framework because I'm already in the space. If you're not already breathing this stuff day in, day out, you'll read that framework and most organization will be like, what the. They're not going to get it. There's too much detail, there's too much nuance and sub subtlety. There's too much stuff going on for, you know, you'd have to study the hell out of that thing. If you're not in the space already, you don't know this stuff already. You'll read that and you have to put in dozens of hours, dozens and dozens to really understand what's, what's going on. That's one issue. So it's not that it's not true, it's not that it's not good in the sense that it's academically rigorous, and I say that as a former academic, but it's just not suitable, I don't think, for the vast majority of organizations. Because one, you're not going to have anyone in your organization who's going to study it in the way that it needs to be studied. And number two, again, I've said this a thousand times now already, the devil's in the details. It's a generic framework. It's necessarily generic because it's meant to apply to literally every, at least larger organization. But that means that it's not just plug and play because in order to make it work for your particular organization, customization has to be done. And you can't do the customization without other kinds of things like gap in feasibility analysis. So unless you had to understand what your organization looks like so you can see how to tweak that generic framework to fit for how your organization actually operates and in a way that's commensurate with your organization's ethical risk appetite. And those vary.

So it does sound like the NIST framework. You maybe just need to feed into ChatGPT and say, Explain this to me like I'm a 10 year old.

Yeah, I mean you probably understand it better, but of course then you're going to have the kind of, at best, if you implement it perfectly, you'll have how a 10 year old would perfectly implement more dinosaurs or something.

All right, do you have any final advice for any organizations wanting to get better at doing AI responsibly?

I mean, the main thing is that I think that it's very doable. I think some people think, oh, it's all abstract, it's all squishy, no One knows what to do. More research. We're good people who have been in the space. I'm not just talking about myself. People who have been in the space. I think we know how to do it quite well. There's always tweaks, there's always improvement to be done, but you know what to do. I think that the first thing to do, honestly I found lately very helpful, is when organizations really just start with that assessment and get a grip on what are we dealing with here. Where does our organization stand? And then they can decide whether or not they actually want to put in an investment. And I should say it is an investment, not just of money, but time. It takes time to do this sort of thing. It takes the time of people in your organization to do this thing. And if you're really building an AI ethical risk strategy, implementing it enterprise wide takes a decent amount of time. So either if you know you're going to do it, great, but if you're not sure. It seems to me, particularly given generative AI, but not just because of that at this point, doing some kind of assessment to make sure you understand where you stand and doing something specifically, at least at a minimum around generative AI, I think that's just a must do at this point.

That's a great answer. But also I hate that patience is the solution. I always struggle with that one especially.

It is and it's not. I mean, I can't stand when people say, well, we need, we're going to do some more research on this, we need to do some more digging. We need to really understand that. No, that's nonsense. I'm very impatient with that. I think that's a level of patience that I find intolerable because it's either sincere or faux contemplation that is no longer appropriate at this point because we can get the ball rolling. We know what's going on, we know what to do. I do recommend patience in that. Look, we're talking about an enterprise wide strategy. It's big. We're talking about tens, hundreds of thousands of employees. It's not a tiny lift. So, you know, or a lot of our clients, they don't just plop it down, as it were, but it gets phased in because if you don't phase it in, it's overwhelming, it's too much. And so you figure out how do we phase this in in a way that is sustainable. If you try to do too big of a lift right out of the gate, just not, nothing is going to happen. So some patience all right.

Just enough patience to do things right.

Okay. That's right. That's right.

All right. Super. Thank you very much for your time. Really great show.

Yes, my pleasure.