#154 Building Ethical Machines with Reid Blackman, Founder & CEO at Virtue Consultants

How things actually get done in the organization or how they ought to get done depend very much on what the organization is like. So check your model for bias. You know, have a cross functional team involved. Okay. But the real work, the real mitigation is in the details, not in the sort of high level best practices. It's in how do we actually, given how our organization operates, given the person on, we have given, you know, the regulations to which we already have to comply, given our existing policies and governance structures, and given all those things, how do we best implement those so called best practices? That's where the action is. That's where the, that's where things get difficult.

Hi Reid, thank you for joining me on the show today.

You're welcome. It's my pleasure.

Excellent.

Let's find out.

Yeah, let's find out.

Cool.

So to begin with, I just want to give a bit of background context. So can you give me an overview of what the major concerns are that organizations face when they're trying to adopt AI?

Well, it's changed. Now that generative AI is on the loose, things have changed a bit. The three big ones used to be and to some extent still are things having to do with ethically discriminatory or biased AI, privacy violations, AI and black box models, Those are the big three. So when I wrote my book Ethical Machines, those are the three that I focused on. Now that generative AI has come along, and here I'm talking about both generating images, video, text, like large language models, like GPT. Now those three risks still apply. There's still the potential for bias in black box models and privacy violations with generative models. But now we've got other kinds of concerns as well. So now we have concerns like worrying about the conversational agent being manipulative. We have obviously IP violations that hook up with the privacy violations. We have concerns around the appearance of LLMs, large language models, chat boxes. They appear to be deliberating, appear to give explanations for why they're telling us the things that they are. But they don't actually do that. There's the problem of hallucinations, what gets called hallucinations, that it gives false outputs. So that's to give you a flavor of some of the, some of the big headlines. And that's just, that's I'm focusing when I say those are the big ones, those are the big ones. For corporations. There's other kinds of things that people tend to worry about or some people worry about. Not necessarily not sure that I'm one of them, but some People worry about the existential threats of LLMs. You know, we're all going to get dominated or killed or both by these, by the, by our robot overlords. There's concerns around the spread of misinformation, which is a concern for social media companies. But I wouldn't say it's, you know, for corporations generally. It's certainly concerned for government job loss due to automation. That seems to be a concern for many people. Again, I don't think it's obviously the responsibility of business to account for them, but those are the, those, those three headline risks, existential threat, mass job loss and spread of disinformation. Those are three other really big concerns. But those are not the main concerns of the vast majority, not all, but a lot of my corporate clients because it doesn't, not within their wheelhouse to fix those problems.

Yeah, I guess most businesses not necessarily going to be concerned with like the threat of like humanity becoming extinct due to AI.

If you're selling ketchup or clothing, you know, you're not going to do much about it. Humanity is coming to an end, you know, they still have to be clothed. So yeah, nothing the gap is going to do about it.

Okay, I do like that there's a sort of fairly well defined list of problems and I'd love to get into some of these in more detail later. Before we get to that, do you have any sort of specific examples of where AI has been misused and it's caused a problem?

Well, I'll give you one where it didn't cause a problem, but it could have. And it sort of kicked off one of the major things that kicked off investigations into bias or discriminatory AI. So years ago, 2017, 2018, something like that, Amazon developed a resume reading AI. So you know, we get tens of thousands of resumes every day. We can't possibly go through all of them or not efficiently. Let's have an AI quote, unquote, read those resumes and recognize among and figure out who to hire and who to interview and who not to. The thing turned out to be biased for reasons we can go into if you like, turned out to be biased or discriminatory against women. So all else equal, it greenlit men's resumes, red lit women's resumes and they tried to mitigate the biases of the model. They couldn't do it so they threw it out in one way. So this was a really, really well known case for a lot of people. This sort of shows, oh, you know, look what Amazon did But it's actually in some ways a success story because, yes, they built a biased model, but number one, they had the good sense to actually look to see if it was biased. Two, they actually found the biases, three, they tried to mitigate it. And four, when they couldn't sufficiently mitigate it, they actually threw out the project after having. Working. After having worked on it for two years, which, you know, that's a lot of time and money to toss. Maybe not if you're Amazon, it's not that much money, but, you know, it's. It's still money. And so there's many ways in which that's, it's a success story because they actually, at the end of the day, they did the right thing. But it also shows how easy it is to accidentally create biased or discriminatory AI and how difficult it can be to sufficiently mitigate that risk.

That's actually a really interesting framing of it because I've heard that story a few times and a lot of the time it is just, oh, yeah, Amazon did this stupid thing with AI and it was bad. But actually you're saying that they really did do this in quite a responsible manner and it looks good.

Yeah, I mean, I should say. I mean, I will say just, I don't know, I guess I'm supposed to disclose that they're Amazon as a client, but I wrote the same thing in my book and that was before they were a client. It also wasn't a technique for getting them to be a client. But yeah, it's, you know, back then, the fact that they thought to check for this was before the conversation around AI ethics exploded. The conversation on AI ethics exploded. Well, arguably it really exploded the past six to nine months. But even when people were talking about biased AI quite frequently, 2019ish, 2020when it really started picking up steam, got outside of academia and people were really talking about it. There's exceptions to that, but these are data scientists and data engineers, or AI engineers, whatever you want to call them, that thought to look into this before was the cool thing to do, so to speak. So it is, it is a success story. And, you know, those data scientists were actively trying to mitigate the biases. And it turns out to be a technically difficult problem. It's not just, you know, it's not a matter of being pure of heart. That doesn't, that doesn't fix your models. It's. There are certain kinds of strategies and tactics that can be quite technical in nature that need to be successfully executed on and sometimes you can't successfully execute on it due to limited time resources, due to limited research into the nature of the problem and potential solutions that were available at that time. So, no, I think it's a success story. It's a big warning sign, it's a red flag. But that's an AI ethical risk success story.

So are there any ways up front? So before you spend several years developing this AI and then discovering that there's some kind of bias there, are there any ways up front of sort of judging how difficult it's going to be to have an unbiased model or to have some sort of responsible AI model?

Yeah, there are various things you can do. I mean, one thing that you want to first of all do is think about the way that it might be ethically problematic during the concept phase, when you're just trying to think about, should we build an AI solution and if so, what should it look like? That's a good place. I mean, if you do it after you built and tested and validated and you're just about to deploy or just about to transfer control to the, to your client, if you, let's say you're an AWS and you built it for a client, then you might find the problem and say, oh my God, this is a problem and I have to go, either you got to ship a bad product or you got to go all the way back to the drawing board, which costs lots of time and money and probably broken contracts, broken promises for when something's going to get delivered. So that's the best thing to do then, is to start it at the very opening phase of thinking about, of the AI project itself. And then there's other kinds of things you can do aside from thinking about, okay, what are the ways that things might go ethically wrong. Let's say you're talking about bias in particular, which gets a lot of press. You might start thinking, okay, well, one potential source of discriminatory output is the training data. So let's take a look at the training data, let's take a deeper dive of that, let's explore the data, let's analyze the data and see if we might find potential for biased outputs there. So let's take one simple example. It's well known, let's say you're building facial recognition software. You take a look at your data, the training is going to be pictures of people, pictures of people's faces in particular. And you might take a look and do an analysis and figure out that, oh my God, only you know, 0.01% are black women. This thing's going to wind up to be terrible at recognizing black women. So we, we're going to need to up that. Up that training that we need to get more data, more examples of pictures of black women's faces in various lighting conditions, et cetera, et cetera. And so you can do some data exploration. That's one way.

So really it is all about just making sure that you understand what you're trying to do at specification phase. So it's kind of cheaper to solve the problems early on rather than later after you've started developing things, is that. Yeah, yeah.

I mean, it is complicated though, you know, so there's no such thing. As far as I'm concerned, there's no such thing as intrinsically biased data. It's not like data sets are sitting around with the sort of, you know, what's the opposite of a halo, you know, with a bias halo around it, the bias anti halo around it. You know, bias bias. It. A lot of it depends upon the context in which you're going to deploy the thing. So to take a sort of toy example, suppose that you're going to use facial recognition software and you're only going to use it in this town in, you know, northern Sweden or something, where everyone is. Has very pale skin and they have a no visitor policy. So, no, no, I'm, you know, making up a toy example. And you want to use facial recognition software in that little village or whatever. It would be bizarre to comp. So let's say you build it. You use just the faces of people who look like the, the population on which you're going to deploy would be weird to say, oh, it works perfectly, but it's biased or ethically discriminatory. So you need to retrain your model so it can recognize, say, the faces of black women or Asian, Asian men or whatever it is that. That would be rare. It's not ethically discriminatory. It's not having negative ethical impacts. So the. So that's to say that the context matters. It's. It's the training data used in such and such a way, in such and such a context can produce discriminatory outputs. And so we better take a closer look at, say, the training data or how we weight the variable, the input variables, or where we set the threshold or et cetera, et cetera. So it's not. People say, oh, we need to get rid of biased AI. We have to make sure that we have representative data sets. That's just A very narrow way of thinking about bias and bias mitigation and AI.

So really it's about getting the right data set for the problem that you're going to be using.

That's part of it. Part of it is the right data set. But also it's things like how you weight the input variables. Part of it is the threshold, let's say. So to give you an example, let's say you're doing credit rating scores, you know, across various subpopulations. And so if they score above a, I'm just going to make up arbitrary numbers. If they score above a 53.5, they get a thumbs up, they get credit, they score below a 53.5, that they get a thumbs down, no credit. And it turns out that when you set the threshold at 53.5 for, you know, yes, no, you get credit, no you don't, it's going to have a certain kind of distribution of credit across various subpopulations, Black, Asian, white, black woman, Asian woman, Asian men, Asian women, et cetera, et cetera. So you're going to get this, this variable distribution across subpopulations. And maybe it turns out that that distribution is discriminatory in some fashion. It may be though, that if you move the threshold to 62.1, again just choosing an arbitrary number, you move the threshold so everyone above a 62.1 gets a thumbs up, everyone below it gets a thumbs down or whatever. Maybe then the way that it distributes that credit across various subpopulations is non discriminatory. So we haven't trained, we haven't changed the training data, we changed the threshold in order to have, in order to differently distribute credit across various subpopulations in this way stipulating in a non discriminatory manner. So there's various ways of getting at the problem, there's various strategies or tactics you can use to mitigate an ethically biased distribution.

So it seems like this is sort of an example of how you make your AI fair or have some sort of sense of justice about it. So in general, sort of other steps you can take like to mitigate discrimination from AI, how do you ensure this sort of level of fairness across all these different subpopulations?

Yeah. So one thing I might highlight first is that I think at a minimum we're aiming for not unfair. There's probably, you know, sometimes people talk about what's fair as though there was just the one fair distribution. I think that there's reasonable disagreement about what constitutes a fair distribution and what People should really focus on is what constitutes what's discriminatory, what's clearly unacceptable. Right? So, for instance, if you find, as ProPublica analysis did, if you find that your model, your AI is predicting that black defendants are twice as likely to, to commit crimes within the next two years as white defendants holding fixed all other variables, criminal history, socioeconomic status, educational accomplishments. If you hold those, all those steady, and black people are still predicted to be twice as likely, you might think, okay, that's. I don't care how you think about what constitutes the fair and the just with a capital F and a capital J. It's clear that that's just ethically unacceptable. It's clear that's. That's unfair, it's discriminatory. So it needs to change. Um, so I think that companies should think a lot about what constitutes unacceptable as opposed to what constitutes the ideal of justice, or something along those lines. And then there's going to be various ways that you think about that. So one, one way that people, especially computer scientists, data scientists, think about fairness when it's, when it's distributing goods across subpopulations like credit or mortgage lending or interviews for a job or admission to a university. There are various, these various quantitative metrics. So you see how it's distributed across various subpopulations. And then you get a score according to various kinds of ways of scoring for fairness. There's something like, you know, two dozen plus ways of scoring or metrics for scoring fairness. And then the question becomes, okay, so what's the appropriate metric? Another way to ask that question, though, in my framing anyway, is to say, what are the subset of metric. Sorry, subset, sorry. You can't score well on all of these metrics at the same time. They're. They're mutually incompatible. So if you score well on one test, you're going to score poorly on another, and vice versa. So then you have this question, okay, well then, which metric is the appropriate one? And another way to frame that, my preferred way is what are the metrics or what are the ways of measuring fairness that we think are inappropriate or unacceptable? Let's take those off the plate. And now we have these remaining set of metrics that are reasonable. And then I think you can choose the metric that is compatible with other kinds of concerns that you have, like accuracy of the model.

Okay, that's really interesting that there are mutually incompatible sort of measures of fairness. Can you give me an example of what one of these sort of measures is? What's the sort of difference?

Yeah, so it's going to be, unfortunately, a little bit abstract, off the cuff. But suppose you think about fairness as we want to make sure that true positives are distributed equally across various subpopulations. So if someone, say, deserves credit, then they get it. And so, you know, whether you're white or black or an Asian woman, or an Asian man or a white woman or whatever it is that the rate of true positive is equivalent across all subpopulations. On the other hand, you might think we want to make sure that false negatives are appropriate or, sorry, equal across all those populations. And so then what you might do is you might set your AI to, what is it? To equalize false negatives across those populations. But you can't do both of those things at the same time. By equalizing false negatives, you're going to make unequal the true positives. And by equalizing the true positives, you're going to make unequal the true. What did I say? The false false negatives. I think, I mean, that's an abstract way. I mean, that's not a very helpful. That's not an example as such, but it's the structure of an example.

Okay, yeah, I get it. So basically you can either try and optimize for one thing where good, whether the right people get the good things and. Or you can optimize for the wrong people not getting the good thing or whatever.

Yeah. Or you're minimizing false negatives or something like that. And then you optimize for that, but you can't optimize for both at the same time. Okay.

And so related to this idea of fairness is accountability. And so when do you need to care about accountability for AI decisions? So if an AI makes a bad decision, then there are some consequences to say, okay, let's not do this, or let's try and mitigate those bad decisions.

One thing to highlight first is that the fairness biases are just one issue. I mean, I rattled off a bunch. At the top of the top of the program, there's a bunch of ethical risks. Bias and ethical discriminatory AI gets the lion's share of attention, at least it did before generative AI. And that's for arguably good reason. I mean, it's a pretty big deal and it's very easy to accidentally create it, but it's really the set of ethical risks. How do we think about accountability with those? So it's. I don't want to limit the discussion of accountability to just be about biased or discriminatory AI, because it really applies to any ways that things can go ethically sideways. Now accountability is kind of, you know, look, if the organization builds an ethically problematic AI, goes ethically off the rails, who's accountable, who's responsible? I mean, most generally it's going to be senior level people. I'm generally not a fan of thinking that it's, it's really primarily the data scientists fault if the organization doesn't provide those data scientists and data engineers, et cetera, with the resources to engage in appropriate ethical risk identification mitigation. It's not on the data scientists to perform a herculane effort to do that work despite compensation, despite unchanging deadlines, despite lacking the tools, the training, et cetera, to engage in that kind of stuff. So it's up to the organization, senior level, senior level officers, you know, someone in the C suite, like a chief data officer, a chief analytics officer, a chief information officer, a chief technology officer, et cetera, someone senior level to own an AI ethical risk program, who empowers those frontline, you know, frontline defenders, data scientists, data engineers, to engage in the kind of ethical risk due diligence that they, that they ought to do and to provide them with the resources to not just perform those analysis, but to do something about them, to, to actually devise and execute on strategies and tactics that will mitigate those biases. What's more, it's, it's incumbent upon those senior level leaders to recognize that data scientists can only do so much. They have a very specific area of expertise, data science or something along those lines, whatever you want to call it. And it turns out, I think that there's other kinds of expertises that are relevant to identifying and mitigating those risks. So yes, data scientists are part of the picture, but you also might need legal in the room, you might need an ethicist in the room, you might need a, and this is a kind of lawyer, like a civil rights attorney in the room. You might need a sociologist, you might need a representative, a representative of the culture in which you plan to deploy your AI or something along those lines. So there are going to be many cases, especially when they're high risk applications of AI, where you need a cross functional array of people thinking about the ethical risks of an application of AI and appropriately mitigating those risks. So if those people aren't present because senior leadership hasn't built the kind of infrastructure that could be accommodated, then again, you can't expect your data scientists to do you know more than they can, given their training, expertise etc.

I think that's a really great point, that some of these skills around ethics are maybe not something that data scientists have been trained in by default. And so perhaps like, yeah, you do need someone with a, I guess, is it like a philosophy background? Is that where most ethicists come from?

I think that's not inappropriate. I think that can be perfectly appropriate. I'm not going, you know, I'm an ethicist by training. I have a PhD in philosophy. I was a philosophy professor for 10 years. I've been researching, publishing and teaching on ethics for over 20 years. But I'm not going to tell everyone, go hire an ethicist. They can be relevant and sometimes they're not. Let me give you an example in which I think they are relevant. So, which we've already covered the bias issue. So you've got these quantitative metrics for fairness, different ways of thinking about fairness. What constitutes unfair. You want to ask, okay, what are the, and I said this earlier, I, I got glided over it a little bit. But what, what's the, what are the ethically inappropriate or ethically appropriate ways of thinking about. Well, sorry, what are the appropriate ways of thinking about fairness in this particular context, for this particular use case? And what you mean by appropriate can vary. Maybe you mean ethically appropriate business, business appropriate, reputationally appropriate, legally appropriate. There's different ways of thinking about what constitutes appropriateness. And my general view is that you need those different ways of thinking about what's appropriate represented in the discussion about how do we handle this AI responsibly. But you can't expect your average data scientist, or 99% of data scientists to give an effective analysis of what constitutes ethically, legally, reputational and business appropriate. That's just beyond their ken.

And so once you've got like data scientists and legal people and maybe an ethicist and maybe some other business people, and they're trying to make decisions on this. You've got a lot of people from different teams. So have you seen any organizational structures within businesses that sort of help all these people come together?

Yeah, sure. I mean, look, you don't need this for every single use case. One thing I want to stress is that it's not the case that for every single AI you build, you need to have this cross function. No, no one's going to do that. It's too much work. It's going to slow things down too much. And so it's really a matter of one is at least training data scientists, product managers, to understand when is Something high risk. When does it need to go to that kind of committee? So there's different ways of doing risk scoring. You train your teams to do proper risk scoring. You know, one score means, you know, go, one score means, you know, talk with a product manager about it. One score means go to the ethics committee, but keep working on the project. One means red, stop everything, don't do anything else until you talk to these seven people, whatever it is. So it's not the case that I don't want people to think, oh my God, this is such, you know, we got to, you know, get the, get the Avengers together. Every single time someone builds a model. That's ridiculous. Of course it's ridiculous. So it's really the high risk stuff. The thing that works well is to leverage existing governance structures, committees, risk boards, whatever it is. Different, different. Different organizations operate differently. So for instance, some have very strict, you know, very good robust governance around privacy. And so where, where legal might be particularly involved. So maybe you take those existing committees or people around legal, you augment them with say an ethicist or you do additional training for them and then off you go. In other cases there's a really good compliance board. And so, okay, it's not the privacy stuff, it's a regulatory compliance board. We're going to leverage that board and we're going to either add some new responsibility, we're going to add new responsibilities to that board. Maybe we add additional training, maybe we added an additional role or two. There's not a sort of one size fits all for every organization because they're just different organizations have different priorities, different risk appetites. They are structured in different ways. Their policies are, you know, very. Some are really privacy forward, some are not at all. So you know, you have to add, you have to add to existing governance in a way that's commensurate with how the organization already works. Otherwise no one's going to do it. Otherwise it's just, you know, it's too disruptive.

Okay, yeah, I mean, I can see how trying to get a lot of people from different teams together for every single AI use case is not going to be very scalable at all. And I can imagine how, okay, using AI to create a new filter for your selfies or whatever, then that doesn't really require a sort of ethics committee. Do you have any sort of guidelines for like when, when you do need to care about these sort of, where you do need to get the end the Avengers together and care about this stuff?

I guess I think, you know, there's probably. This is not, this is not exhaustive, but I think one of the main criterion is something like, does the AI obstruct, hinder, block people's access to the basic goods of life? Something along those lines. So if we're talking about jobs, you need a job to make money, to pay your rent, to pay for your food, to, you know, pay for your kids, various, whatever. Just, well, living expenses, frankly, just for kids, let alone extracurricular activities. So jobs, you know, credit or lending for mortgages, healthcare and life sciences, often directly relevant to people's ability to live a minimally decent life. So if it's things to do with the basic goods of life, by which I mean the things people need access to, regular, systematic access to in order to live a meaningfully, minimally decent life, then it's probably high risk if you are making a prediction about when the screws are going to arrive at the toy factory. All right, no avengers necessary.

So really it's all about the consequences of this and who's going to benefit, who's going to be harmed by this and how much. It seems like.

Yeah, yeah. I mean, here's another way to put it, another kind of standard that's very similar to the one that I just articulated, but probably not exactly overlapping. At least the Venn diagram. If it's potentially going to violate someone's humans rights, that'd be a pretty big red flag. That seems pretty high risk. If this thing potentially violates human rights. Let's get it. Let's, you know, someone call Captain America.

Absolutely. I can imagine how that's a pretty serious concern for anyone involved in creating that sort of AI. And so I think sometimes when businesses go, okay, let's implement this new AI feature, the focus is going to be around, like, what are the sort of the benefits or the harms to the business itself. Do you have any advice on how to make sure that businesses start thinking about, like, what are the benefits or harms to their customers or to other people outside the business?

I mean, look, if you're, if you're, if your company is using AI, then you just have to start thinking about their risks involved. That'd be. It'd be silly not to. Yes. Think about the business risks. I mean, there are business risks to it. I mean, it's, it can be a significant investment of capital. At least a significant investment of personnel, which is boils down to, for a business, is significant versus capital. So there are risks, but, you know, there are ethical. Like I said, there are ethical Risks. There are legal risks given generative AI in particular, which we haven't really talked about yet. There's lots of risk. One of the things about generative AI is that, well, there's two things. One, it's general purpose, so it could be used for all sorts of things as opposed to non generative AI or you know, generative AI circa 2022, sorry, AI circa 2020, early, earlier or first three quarters of 2022. You know, it's task specific. You know, it just does risk ratings for diabetes or you know you've got cancer or you don't get an interview. It's very. But things like LLMs or generative AI in general are general purpose, so they could be used for all sorts of stuff. And the use cases, we don't even know. There's countless, countless use cases as far as we know. And so organizations don't know how people might use generative AI. That's worrying. And secondly, it's just that relatedly, everyone in the organization has access to it. Everyone gets to use it. It's not just data scientists, far from it. It's not even that data scientists within your organization have approved it for use by your internal people. No, it's just, you know, Bob and HR decided to use it and load on this, you know, HR data onto the thing and now it leaked and now there's a big problem. So at this point, it's, it's, it's bordering on negligent, frankly, not to have some kind of governance around how your organization does AI because the risks are there. I mean, every organization has sensitive information, for instance, that they don't want leaking out. Every organization does sales and could potentially use a chatbot chatbot for sales in a way that is manipulative, you know, objectionable in some kind of way. So yeah, there's prob, you know, problems abound. They're not, I think it's, you know, you can get your hands around them. Some people think, oh my God, it's so complicated. We know there's nothing to do. But no, you know, they'll say things like, oh, we need to do our research. We need to, we're going to look into that. Now. We sort of know. We people have been working in this field for a while. We know what to do. It's just a matter of political will to do it. And that political will, incidentally, is increasing because leaders are anxious about the impacts of generative AI.

That's good that leaders in general are concerned about this sort of thing. And it sounds like a lot of the sort of, the best practices are fairly well established. So can you maybe talk me through some of those things?

I wouldn't say, I mean, I don't like the phrase best practices. It makes it sound like a list of things to just do and then you're, then you're good. But the best practices are necessarily high level. I mean they're necessarily generic. You know, check your models for bias, make sure that the training data doesn't violate anyone's privacy or ip, you know, don't use a black box model and very high risk situations, you know, so there, there we could like sort of rattle off some quote unquote best practices. But number one, insofar as best practices are agreed upon by the relevant community, that this is what is good. We don't have that, we just don't have that kind of universal or widespread consensus unless the consensus is so high level to be almost meaningless. The other thing is, as I was saying before is how things actually get done in the organization or how they ought to get done depend very much on what the organization is like. So check your model for bias, have a cross functional team involved, okay? But the real work, the real mitigation is in the details, not in the sort of high level best practices. It's in how do we actually, given how our organization operates, given the personnel we have, given the regulations to which we already have to comply, given our existing policies and governance structures and given all those things, how do we best implement those so called best practices? That's where the action is, that's where the, that's where things get difficult. Not, yeah, I would say difficult, I suppose doable, but not clear. The best practices are easy to articulate insofar the best practices are extremely high level. Like you know, vet your models for ethical risks, mitigate the ethical risks, don't violate people's privacy, you know, make sure that your, your model is not discriminatory, don't engage in manipulative, manipulative ways, etc. Etc. Etc. You know, check your, check your gender AI for hallucinations, for, for false outputs, you know, sort of. No kidding.

All right, so it sounds like there is a sort of very high level set of best practices, but actually the devil's in the details, I mean, I guess. Yeah, well, so the million dollar question then is like, well, how do you stop Bob from HR just posting employee data on, on ChatGPT?

Yeah, I mean, so there's so one take on this is. Let's just have a, let's just shut it down within our organization. Let's just make sure that people can't use GPT. We're going to, you know, create a firewall or whatever it is around various kinds of access to LLMs. I think this is not a good way of doing things. I sort of, I think it's similar to telling teenagers to be abstinent and then providing no sex education. It's just a bad idea. Bad things are going to happen because they're going to do it. Both those teenagers and your employees, they're going to use it. They have their own phones, right? They've got their own computers. They're going to use, they're going to use it. So I think one, you've got to get a better, more clearly articulated, articulated policy. One of the. Here's at least two main things or maybe three. One, you gotta front load Learning and development. Most people don't know much about generative AI. They don't really know the ethical risks of AI. They don't know the ways in which generative AI. They don't know which ways in which you can go wrong. So having to educate enterprise wide becomes more important than it was pre generative AI, because pre generative AI, your data scientists owned it and, and they distributed it, they gave it to people. They could give them instructions. Now with generative AI, since everyone has access to it now, we don't have any kind of not real control over the flow over who uses these tools. And so LND becomes really, really important. So front loading LND number two, getting an inventory of who's doing what. So provide those, those people and tell them about this in your L and D program. Here's what you should do if you're, you know, here's the what to look out for and here's where you sort of, here's who you tell that you're doing it so that we have an inventory and understanding of what's going on within our organization. Otherwise, who knows? I mean you've got, you know, thousands, tens of thousands, in some cases hundreds of thousands of employees potentially using this thing. You have no idea who's using it for what. And it's going to be so use case specific. And you know, Bob in the back room of HR is like, oh, I can use it for this thing, right? That some C suite executive would have no idea that that would be it. That that's a problem that the HR person faces that Bob faces and that he might use gender of AI to solve. So make sure that there's a method of inventory and third and related, related to the inventory piece, making sure that, you know, if they've got some, you know, you have to have some way of getting them to smell ethical smoke, you know, understanding that they're dealing with some kind of potentially high risk thing or potentially using high risk data, then the means by which they can contact the appropriate risk board or person, whatever it is, to get guidance on how to use this thing appropriately.

I like that you said that people need some kind of training around using AI. Are there any particular teams or roles that you think this training is particularly important for?

No. I mean this is the thing with generative AI, it's for everyone. Hr, marketing operations, product teams, sales teams, it's everybody. And so at least some kind of, you know, generative AI ethical risks 101 that is deployed enterprise wide that everyone gets. It's not role specific because it's just everyone. Any role, literally any role in your organization can use it for something. Again, that's, again, that's different. That's different. It's different with, with non generative AI, then okay, you do need specific training for say data scientists. Right? Your data scientists need to go an extra level deeper. But you know, there's AI Ethical Risks 101 and then there's Ethical Risks 201 and then 201 is, is customized on a roll by roll basis or at least department by department basis or business unit by business unit basis. But no, for generative AI, it's everybody.

Okay, that's good to know. Training for everyone. I like that. So one thing we've not really talked about, you've mentioned bit is around data privacy because I think this is perhaps one of the biggest concerns, especially with generative AI, that someone's going to post some sensitive data. Can you just give me a bit of an overview of like one of the different ways data privacy can go wrong with AI?

Yeah, so there's. Okay, so here's a couple of ways. Number one, you use private data, let's just not even say IP yet. Just people's private data, maybe health, health data, for instance, in order to train your AI, so you've acquired and use that data in a way that violates their privacy. That's one way. Second way related to privacy is the IP stuff. So you use texts by authors, images by artists, and you use that to train your generative AI. So now I've got AP concern. I. Sorry, IP concerns. Another way is these are. A lot of these things can be used as chatbots they interact with people, the people put certain kinds of information into it and they don't realize that that data is then being, you know, collected and used for various other purposes. So you collect new data with the chatbot, that possession, mere possession and possession and use can itself constitute a violation of privacy.

Yeah, you got the personal information and then you got the IP risks. And are there any particular ways that you can go about mitigating these risks?

Yeah, there's also. Yeah, you know, I'll say one other thing is that the data might get leaked. I didn't mention that. So you might share data with generative AI provider, let's say an OpenAI or Microsoft or a, or a Google or a whatever, and then it might get that, that might get leaked in various ways. So this happened to OpenAI a few months back where people could see other people's chats, their chat history. Not the whole thing, I don't believe, but some of it. Or it saw payment information. So the other issue is that there might be nothing intentionally nefarious going on, but there might be a negligent handling of the data which would then result in privacy violation, things to do. It's all going to vary. I mean, some people are trying to come up with ways of, for instance, sending data to a third party vendor like an OpenAI in such a way that that third party, that LLM provider, for instance, can't see the data. They don't. It's. I don't want to say hashed, but it's. Yeah, it's an. I'm just gonna, whatever I'm gonna say it's encrypted in some fashion or other. So for instance, I was just at the conference AI4 last week and you know, met a company where you can type whatever you want into the, into the, into your chatbot or sorry, into the LLM interface on, on your own premises. It gets rewritten somehow. It scrambles the words in a way that it becomes gibberish, gets sent out to an OpenAI or an Anthropic or a Microsoft or a Google. Then that LLM company generates the response, sends the answer back. And so all the, all that, all that company sees is the answer, but they don't see the question that was asked. So if there was any personally identifiable information or private data in that exchange, that OpenAI or Google or whatever can't see it because it was scrambled before it got there. That's sort of one string tactic. That company is, I believe, called Protopia. Pretty cool stuff. I don't have any financial connection to them. Just to be clear. I thought it was a cool demonstration.

Okay. Yeah. So really it's about like not passing your sensitive data directly. Needs to be scrambled or encrypted in some way before any other company is going to see it.

Yeah. Or you could have it LLM, a large language model, you know, on your own premises, on your own servers. And does it never, just never leaves. That's a different way of doing it. Right. So if you're not sending it out, then of course you're more likely to not share it with people accidentally. So there's, there's different, you know, different ways of doing this. But it's, it's, it's also just early days. People don't, it's not, it's not, there's not, there's not best practices for that.

And I suppose related to the idea of privacy is, is transparency of models. And you mentioned the idea that sometimes you can have a black box model that no one really understands or maybe you have some kind of interpretable, easily interpretable or explainable model. Can you talk about when that's important?

So there's some controversy here. I probably occupy, I think, a relatively unpopular position within the AI ethics community broadly. So the concern is. So first of all, I want to distinguish between explainability and transparency. So explainability is going to be what's going on between the inputs and the output. So you input some data like here's this person's medical profile or here's our financial history, and it outputs something like, this person is going to develop diabetes or this person is going to default on a mortgage. Don't give them credit. How did it arrive at that decision? How did, how air quotes, you know, how did it arrive at the quote unquote decision? If you can't explain because the pattern that it's looking at is too mathematically complex to understand, you have a black box model, you have an unexplainable model. You might go around telling everyone, hey, everyone, we've got a black box model that's being very transparent that you have a black box model. So that's transparency. On the other hand, you might have a model that you totally understand, okay, we give these outputs, it engages in these calculations and so it arrives at this output. So I can explain to you exactly how it arrives at its quote unquote decision, but don't tell anyone. We've got this, you know, clear box or glass box model that would be high explainability and low transparency. So transparency and explainability come apart. There are a lot of people who think we shouldn't have black box models and high stakes cases. It's too high risk. We don't understand how the, how the thing works. It's not safe to use. That's, that's one line. Another line is, look, as long as you can show that it's sufficiently reliable, sufficiently, you know, gives outputs that are sufficiently reliable that maybe do better, it seems, performance wise, than what? When, say, humans judge or when you have explainable models, then arguably it's permissible, ethically permissible to use it. So, for instance, let's just stipulate the best doctor is 80% accurate when it comes to cancer diagnoses. I'm completely making that up. I don't know what the stat is, but let's just say 80%. Let's suppose you've got a black box model that's 99% accurate. You know, is it ethically permissible to use the black box model, especially if you have the informed consent of the patient? Probably.

So are there any cases where it's the other way around? And you would say, okay, we've got something super high stakes and.

We really.

Really shouldn't use a black box model here.

So there's one kind of case that gives me pause. So one thing that you might think about is AI in the criminal justice system. So one thing that we're supposed to get in the criminal justice system is what's called procedural justice. So the procedure is in order for the, for the outcome to be considered fair. The procedure by which the outcome was arrived at was fair. That's why we have things like a trial by juries. The juries have to engage in this kind of deliberation. They're not allowed to take this into consideration, but they can take that into consideration, et cetera, et cetera. There's case law to guide them. The judge didn't, you know, do anything or didn't bias the judge, the jury or something. Right. So there's ways of ensuring more or less a fair procedure to increase the probability of a true outcome. But we take the outcome to be fair on the condition that the procedure is fair. So you might think that in some cases we need to have a fair procedure and that's the best we can get because we don't, if you, like, have the ground truth, we don't really know if they're innocent or guilty. We just have this way of doing it, of finding them innocent or guilty or guilty or not guilty by the procedure. Okay. But now suppose you have a black box model in that mix and part of the procedure occurs, if you like, between the inputs and the outputs of an AI, but you can't explain it. But if you can explain, if you like the procedures of the AI, then you can't assess whether the procedure was fair or not. And so that looks like potentially anyway, a case in which you a black box model would be unacceptable because it would necessarily make, make it impossible to assess whether the procedures were fair or not. That's one case. In some cases we don't care. Right. Like so with cancer diagnosing, there's one which I don't really care. I don't care if it's a magic box. I don't care if it's a genie who just divines the truth. Like if they get it right, that's all I care about. But in the criminal justice system, we care a lot about the procedure and the fairness of the procedure. So arguably in that case, procedure matters and black boxes should be unacceptable. That's debatable too, but I find at least a plausible line of thought that.

Sounds a lot like the difference between statisticians and machine learning scientists, where the statisticians want to know how everything works and do inference and machine learning scientists just care about do I get good predictions or not?

So, yeah, and it might be the case that sometimes you want machine learning engineering, sometimes you want a statistician.

Absolutely. So that's the explainability side of things. Are there any cases where you either really care about transparency or you might say, okay, this isn't important here.

Yeah, this is a good question. I mean, you look at any ethics statement, AI ethics statement, everyone's going to list transparency, which is kind of silly because no one is going to be transparent about everything and no one's going to be transparent about nothing. And so you need to be transparent about the right thing at the right time in the right way, which is relatively uninformative. You know, I think that there are probably cases in which high stakes, you know, high stakes AI say life and death is on the line, and you need to be transparent enough so that external stakeholders, let's say auditors, in particular regulators, can look to see what kinds of decisions were made throughout the AI life cycle and were they, are they defensible decisions? That would be a case in which there needs to be sufficient transparency.

Okay, so you need to really understand how this thing's working then. Yeah, you're going to have to have some level of transparency there.

Yeah, I mean, Sometimes you may or may not know how the model is operating. You might not know what's going on between the inputs and the outputs, but you can always know, at least in principle, what are the kinds of decisions before the inputs and after the outputs. There's lots of things to know. There's lots of decisions that data scientists and other product managers that are make around that AI lifecycle. So those things can be more or less transparent. And look, you're not going to give away your ip, so don't be transparent about that. If it's going to give away your ip, all else equal. But there are going to be some things that need to be transparent about so that you can be, so that we can see whether you're, so that we, the we as appropriate authorities can see whether your ethical house is in order.

Sometimes it's not clear whether AI is going to be, whether your data is going to be used by the AI, and sometimes it's unclear what happens when you give that you are giving permission for AI to make a decision about your life. When do you think that is important?

So different people will say different things to this. The go to line is that in all cases people should have control over their data. That's a, that's a standard line. So obviously in the high risk cases, but also even in low risk cases, because it's my data or something along those lines, that's a pretty standard line to hear. I'm not convinced of that position myself. I'm actually, I did a podcast on this with someone who does think that, Carissa Valise at Oxford. We had a really good discussion around this point where I tried to push back. Look, there are going to be some cases where there's highly sensitive information about you where plausibly you should be allowed to say, no, no, you can't have that data. Health information is one of them. Although I'm not even completely convinced of that. I mean, I'm inclined to think, you know, let's say that there's a global pandemic and the appropriate authorities having access to certain health data of yours that's, you know, anonymized or aggregated and, you know, is used to engage in certain kinds of analyses to say, you know, make certain kinds of predictions or build a cure or a vaccine or whatever, you know, then it seems to me like, you know, I agree that the individual has a vested interest in having control over their data, but the global community has a, has a, has a very strong interest in not dying. And so plausibly that individual's preference is, is ethically overruleable. That's going to be controversial though, obviously. So I, I guess I think that there are many cases where it'd be nice, at least ethically nice. People had control over what data different organizations had about them. I'm not sure that in all cases it's ethically required and then it's just going to be a case by case basis. I don't, I'm not sure that there's an obvious way of saying it outside of things like, you know, health information from corporations, let's say.

Okay. Or I guess maybe financial information is possibly another sort of thing.

Yeah, yeah.

So for organizations who are wanting to kind of get better at doing responsible AI, I guess what are the most common mistakes that organizations make that they ought to avoid?

They rush to implementation. They, they'll come up with some really high level ethics statement where for fairness, where for privacy, where for transparency. We're for accountability. And it means nothing because it's so high level, it's so generic. And then they say, okay, let's, how do we implement this? Sort of like, no, no, no, no, you can't, you, first of all, you don't have anything here. You just, you have nothing. You could have written any words on that paper. It would mean this, it would mean approximately the same thing. The thing that people, that organizations don't do is they don't do the sort of, I don't care what you call, there's different ways of calling it a gap analysis, a risk analysis, a feasibility analysis. They don't get a good grip on what their organizational standards are and they don't get a good grip on where their organization stands relative to those standards. So one of the opening moves that we do with all of our clients. Almost all of our clients, yeah, actually, I think, yeah, almost all of our clients is we do that risk assessment or that gap analysis. Okay. What do your current governance structures look like? What do your policies look like? Personnel training, onboarding tools, workflow. What, what does the organization look like right now? And how, how what can be leveraged and what needs to be avoided in building out a customized ethical risk framework that suits your organization as it stands? If you don't do that, you know, you don't know what a organization don't know what to do, then they just start doing stuff and then they find that, oh, actually this doesn't play nicely with other, other parts of the organization. So for instance, let's just start building risk matrices and risk categories for AI Ethics, and that's going to be our thing. And then you don't talk to Risk about it. The, you know, the people in Risk, and they're like, whoa, whoa, whoa, how does this fit with our enterprise risk categories? And if it doesn't, now we got decisions to make. You know, brakes get slammed. Risk wants to get more involved. Now it gets, it gets complicated, confused. And then there's political, you know, political factions. And better to do that assessment early on involving risk and compliance and cyber and legal. Get everyone on the same page what we're trying to accomplish here, and then things move along a lot more effectively, a lot more efficiently.

Okay, so it sounds like there's a sort of maybe a specific order you need to do things then. So you need to set up some sort of committee and get these teams involved to begin with, Is that what you're saying?

You know, it varies. I mean, usually we're working with like a working group. I wouldn't call it a committee or it's not like an ethics committee or something along those lines. It's, you know, most organizations need some kind of AI ethical risk or responsible AI working group. Often that smaller working group reports to a larger AI ethical risk steering committee. But yeah, you know, you work with a handful of people. Risk compliance, legal, cyber, you know, chief data officer, some, you know, or someone under that person's charge. Yeah, but you're, you're working with them to do that, that gap analysis or that risk analysis. That's, that, that's the way to start. Look, you know, sometimes things, it depends on how fast an organization also wants to move. Sometimes we're simultaneously developing some learning and development, although that has to be done a little bit after you do that assessment, because you want to know what existing learning and development resources can we leverage? But, you know, and you could also do some, you know, sometimes we're working with one client now, multinational, who wants to retroactively do some risk assessments of, of some AI models that they already have out there. And so they want to do it first. Let's just do a quick and dirty pass on what's already out there, flag certain things as high risk, do a deeper dive on those things, and let's, let's get that up and running because it's, it's stuff that's already out there. We want to make sure that we're good, while at the same time building our more systematic, comprehensive approach to AI ethical risk.

Okay, and are there any sort of process changes you think organizations need to make in General, to do this right.

I mean, the main thing is maybe not the main thing. If we're talking about the level of process, the one thing that's going to be crucial is figuring out the processes by which you engage in those risk analyses at each stage of the AI life cycle. So it's not just something you sort of plug in at the end just before you deploy. Because if you do that, it's really hard to go back and fix things. The contract says it's due at this date. It's time to go. You find some big risk. Not much you can do about it except for ship a bad product or go back to the drawing board and deliver something late, which nobody likes to do, obviously. So making, making part of the building into processes throughout the life cycle, both the processes that teams engage in when they're developing the model, and also processes between teams. So like data collectors versus data scientists, for instance. When, when those differ, making sure that there's processes that facilitate appropriate communication and transfer information from what one team to another, as the AI gets, or parts of the AI get passed on from team to team, that stuff is really crucial. Oh, yeah.

Between team processes. That's always the hard thing. It's like trying to get different managers to agree on things. So are there any tools that can help you out with this? Are there any sort of responsible AI tools?

I mean, there are tools for various kinds of things. Like, let's do a quantitative analysis of the bias in these models. Let's do, you know, quantitative analysis of potential privacy violations. Let's do, you know, there's, there's that sort of thing. There are also certain kinds of platforms that allegedly help with this kind of thing. I haven't seen anything yet that I find particularly impressive and scalable. That's one of the big issues, is scalability of these things. Oh, sorry, that's, that's, that's not totally true. Some of those, some of those tools that do, for instance, quantitative analysis of whether the model is unbiased. Those are easily scalable governance platforms. I haven't seen one of those yet. I mean, part of the problem is just that, for instance, among other things, a constantly evolving regulatory landscape and checking for compliance with those regulations is not something that can be just automated at scale. You need to have some people involved at some point.

Okay, let's get.

I'm not, I'm not, I'm not saying organizations shouldn't use these tools. I am saying, though, that there's lots of cases where organizations that think, oh, we'll just get a tool and we'll fix it. And that's, that's never the case. A tool is good. But to speak to the point earlier about accountability, a tool doesn't work well if it doesn't land in a fruitful environment in which it's going to be deployed effectively.

You can't just buy stuff and have all your problems magically solved.

Yeah, you buy the software, maybe you train a couple of people on it, but they don't train the others, then they leave. There's no, there's no specific place in the process where they're supposed to use that tool. There's no process by which there's any kind of auditing about whether that tool is being used and, or whether it's being used well or effectively or according to the proper instructions. So, yeah, you know, companies spend hundreds of thousands of dollars on these tools, but, you know, you know, they don't actually get, in my experience, they don't really get used or used well because they haven't landed in the environment in which data scientists, etcetera, Actually have the support to use those tools.

Okay, So a bit of training and the right processes are needed as well.

And then the last thing I'll say is that it also gives some clients, like quite a bit of corporations a false sense of security. Like, oh, yeah, yeah, we've got a tool for that. We're good. And I don't try to convince them otherwise because I'm not going to convince them that actually maybe things aren't so good. But whenever we do these assessments, whenever we do these risk analyses, one thing that we always find is, you know, it's not as buttoned up as you think. The tool is good at what it does, but it leaves lots of gaps available open. Whereas most companies are like, oh, yeah, we got this tool, it looks at bias. So we have the bias thing down. Sort of like, no, you don't. You've got, you got a way of dealing with it. It's a, it's a, a good way, but it's not a comprehensive way. It's not sufficient.

Okay, so if, if tools aren't the answer, are there any ethical AI frameworks that you think are worthwhile looking into?

I mean, so for instance, NIST has a really. They have, they have, NIST has, I think they call it AI Assurance Framework, something along those lines. I was impressed by the NIST framework. For lack. I really hate the word framework because sometimes framework means a pie chart with some buzzwords in it. And for some it's more robust. So I was impressed by the NIST framework. I read it and I thought, yeah, the people who wrote this, they know what they're talking about. And so there's something that's great about it. But number one, most people who aren't. I understood that NIST framework because I'm already in the space. If you don't under, if you're not already sort of breathing this stuff day in, day out, you'll read that framework and most organizations will be like, what the. They're not going to get it. There's too much detail, there's too much nuance and subtlety. There's too much stuff going on for, you know, you'd have to study the hell out of that thing. If you're not in the space already, you don't know this stuff already. You'll read that and you better. You have to put in dozens of hours, dozens and dozens to really understand what's, what's going on. That's, that's one issue. So it's not that it's not true that it's not good in the sense that it's, it's sort of academically rigorous. And I say that as a former academic, but it's just not suitable, I don't think, for the vast majority of organizations. Because one, you're not gonna have anyone in your organization who's gonna study it in the way that it needs to be studied. And number two, again, I've said this a thousand times now already, the devil's in the details. It's a generic framework. It's necessarily generic because it's meant to apply to literally every, you know, at least larger organization. But that means that it's not just plug and play, because in order to make it work for your particular organization, customization has to be done. And you can't do the customization without other kinds of things like gap and feasibility analysis. So unless, you know, you had to understand what your organization looks like so you can see how to tweak that generic framework to fit for how your organization actually operates. And in a way that's commensurate with your organization's ethical risk appetite. And those vary.

Okay, so it does sound like the NIST framework. You maybe just need to feed it into ChatGPT and say, Explain this to me like I'm a 10 year old.

Yeah, I mean, you probably understand it better, but of course then you're going to have the kind of. At best, if you implement it perfectly, you'll have how a 10 year old would perfectly implement more dinosaurs or something.

All right, do you have any final advice for any organizations wanting to get better at doing AI responsibly?

I mean, the main thing is that I think that it's very doable. I think some people think, oh, it's all abstract, it's all squishy, no one knows what to do. More research. Now we, you know, we're good, you know, people who have been in the space, I'm not just talking about myself, people who have been in the space, we sort of, I think we know how to do it quite well. There's always tweaks, there's always improvement to be done, but you know what to do. I think that the first thing to do, honestly, what, what I, I found lately, very helpful is when organizations really just start with that assessment and get a grip on what are we dealing with here? Where does our organization stand? And then they can decide whether or not they actually want to put in an investment. And I should say it is an investment, not just of, of money, but, you know, time. It, it takes time to do this sort of thing. It takes the time of people in your organization to do this thing. And if you're really building an AI ethical risk strategy, implementing it enterprise wide, it takes, takes a decent amount of time. So either if you know you're going to do it, great, but if you're not sure, it seems to me, particularly given generative AI, but not just because of that at this point, doing some kind of assessment to make sure you understand where you stand and doing something specifically, at least at a minimum around generative AI. I think that's just, that's just a must do at this point.

That's a great answer. But also I hate that patience is kind of the solution. I always struggle with that one especially.

It is and it's not, I mean, it's, you know, I can't stand when people say, well, you know, we need, we're gonna do some more research on this, we need to do some more digging. We need to really understand. No, that's, that's nonsense. Like, I'm very impatient with that. I think that's a level of patience that I find intolerable because it's either sincere or faux contemplation that is no longer appropriate at this point because we can get the ball rolling. We know, we know what's going on, we know what to do. But I do recommend patients in that, look, we're talking about an enterprise wide, you know, strategy. It's a big, you know, we're talking about tens, hundreds of thousands of employees. That's a, it's, it's, it's not a tiny lift. So and then I, you know, or a lot of our clients, they don't just sort of plop it down, as it were, but it gets phased in because if you don't phase it in and it's overwhelming, it's too much. And so you figure out how do we phase this in in a way that is sustainable? If you try to, try to do too big of a lift right out of the gate, just not, nothing's going to happen. So some patience, all right?

Just enough patience to do things right.

Okay. That's right. That's right.

All right. Super. Thank you very much, Rick, for your time. Really great show.

Thanks. Yeah, it's my pleasure. Sa.