EP 377: Confronting AI Bias and AI Discrimination in the Workplace

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

There's no getting around it. As powerful as Generative AI is, there's an other side to it, right? It can be ugly sometimes. There's. There's biases and there's discrimination built into these models. So I think it's important that we have those conversations and talk about what business leaders need to know when it comes to biases and discrimination that makes its way into these models and how we can responsibly and ethically still use them to grow our companies and to grow our careers. All right, I am excited for today's episode where we have a leader from EY joining the show. So if you're new here, thank you. Thanks for tuning in. My name is Jordan Wilson and this is Everyday AI. Before we get our show started off, I have to give a quick shout out to our partners at Microsoft. So if you haven't heard, the Work Lab podcast from Microsoft is made for leaders who want to understand the future of work. It offers expert insights on everything from how to approach digital transformation to what it takes to thrive in the AI area. That's W O R K L A B no spaces available wherever you get your podcasts. All right, so they just dropped a new episode last week. It was fantastic. Make sure you go check it out. And while you're checking things out, please go to your everydayai.com. so we are a daily livestream podcast and free daily newsletter. In the newsletter sometimes I get it. You're listening to the podcast, you're on the treadmill, you're walking the dog, and our guest today guarantee you you're going to miss some of the gems that she brings. So we're going to be recapping it all in today's newsletter. So make sure you go sign up for that. And if you are looking for the news technically prerecorded show here, we're debuting it live. So we'll have that as always, in today's newsletter. All right, enough chit chat, y'. All. I'm very excited for our guest for today, so please help me welcome to the show Samta Kapoor, the Energy AI and responsible AI leader at EY Americas. Samta, thank so much for joining the Everyday AI show.

Thank you, Jordan. Thanks for having me.

Oh, absolutely. So could you tell us a little bit? Right, like that sounds like a huge responsibility, right? The Energy AI and Responsible AI A AI leader for ey. Can you tell us a little bit about what you do in your role at ey?

Yeah, clearly other than making sure I have enough AI in my title, there's a lot, there's a lot that I work on. Jordan. It's basically like in a nutshell, it's thinking about how to help our clients across utilities, oil and gas and mining to transform using data and AI responsibly. So how do we make sure that the organizations can deliver value to all of their stakeholders, which could be board members, customers, you and I, everyone, their employees, but do so with having the core responsibility in it?

And I'm sure most people know Ernst and Young, but maybe for those that aren't familiar, can you tell us a little bit about EY and the work that you guys do globally?

Absolutely. So Ernst and Young is an approximately 400,000 plus employees company. And what we do is tax assurance, audit as well as consulting. So those are sort of our main lines of business. And we do that by keeping our people at the core of IT and also by making sure that we're delivering client value every day.

So let's just maybe skip to the end here, Samta, and let's talk about how can business leaders actually confront AI bias and AI discrimination in these models? Because yeah, we know they, you know, all this bad information also makes its way to these models that we all use. So what should business leaders be looking out for? And let's start talking about some of those best practices.

Yeah, the good and bad about AI is that these biases can creep in any step of the way. Right from the beginning, when you are thinking about the use case or what is it that you're going to do, how are you going to transform your business? Believe it or not, you can actually bring in bias. So from the design to thinking about the data that you're going to feed your algorithms to, using those algorithms and making sure that you are continuously monitoring those algorithms for data and model drifts without getting too technical is key to ensuring that you are not affecting anyone negatively and or not being in the news for wrong reasons.

Yeah, we've seen plenty of that. Especially I would say early on, right when the world was still acclimating to what the heck a large language model is. Hopefully it's a little less, you know, rampant than it once was. But you know, maybe let's start even talking a little bit about these, you know, guardrails. Right? That's something we hear all the time. And I know, you know, some of you out there that just want to use AI and go faster and break things. Right. Guardrails can be boring. But Samta, they're extremely important. What should people know specifically as it comes, like when it comes to bias and discrimination about AI guardrails?

Yeah. And Jordan, the cool part about AI is that there was always bias that could come in and creep in and exist. But Genai actually came in with so many more risks than we hadn't seen with traditional, classical, narrow, whatever we want to call the other AI. And so now it's all come to bear. There are so many different things that have come around where everyone is nervous about what it's bringing and what it's not. So in terms of guardrails, there are a few things that I strongly recommend when you're an organization, just, again, these guardrails are not to stop innovation, or these guardrails are not to say that, oh my God, like, AI is this big thing that you should not think about, like, let it pass. Right. That's not the motive. The motive is to make sure that you have enough governance where you're still innovating, you can still be cutting edge, you can still understand what's happening, but you have enough governance to make sure that, that you are staying out of the news. The other big piece. So again, like any basic things that we discuss, it's all about the people, process and tech. It's not that the tech itself is harmful. It's like a knife. Like, you use it to cut fruits and vegetables and get healthy or you kill someone and go to prison. Right. But what do you do with the guardrails that you set to make sure that you're using it for fruits and vegetables and not landing in prison with that is very crucial. So having good governance, having a good idea of where your models are, how they're being used, who is using it, how are they impacting the user, the end customer. So, for example, if you have built a model for customer segmentation, if you are using a model in a medical field to do diagnosis, have you thought about minority? Have you thought about women of color? Right. A little plug. But honestly, it's about making sure that a client, across these different stages that we discussed up front, that you have enough oversight which is going to ensure that your models and data is used responsibly, is key to making sure that you're scaling and you are scaling at a pace that you can truly innovate. And again, like this is, I know popular view is the minute you talk about guardrails and Governance, everyone is nervous about, you know, oh my God, I can't use this technology anymore. But this is honestly all about scaling. And believe it or not, having guardrails and having a strong governance enables scaling faster. Because you have a view of what you have built, where you can use it, how you can use it, how can you reuse things that you've built. So those are sort of the key things that I would encourage your listeners to think about.

You know, and maybe this is oversimplifying things, but that's something I always try to do right here on the everyday AI show. You know, I like when companies hire us to consult or something like that. Tell them like, okay, do you have a, do you have a hardware policy, do you have a computer policy? Do you have a software policy, an Internet policy, an email? Right? And all these companies are like, yes, yes, yes we do. But then they still maybe don't understand why they need an AI policy. Right. You know, in general, why is something like, you know, guardrails governance, you know, something as simple as AI policy. Why is that so important when it seems so fundamental or so easy to.

Skip such a great question, Jordan. I'm going to state some news articles that we all would have read and I'm also going to talk a little bit about regulation that might help companies appreciate the value of having AI policy and the governance in place. So there was very recently there was a financial institution in the news where they had actually eliminated a certain segment of the society when they were thinking about mortgages and giving the discount based mortgages like etc. There were also healthcare companies that have been in the news because of certain treatments of certain segments very differently. If that wasn't enough, let's go back and think through what is happening in the regulatory landscape. The EU has already put out the EU AI act. So you have to make sure that you're in the forefront, you're able to answer the questions they're asking. The models that are considered high risk are treated very differently than the others. Now here back in the states we have, while we have the guardrails, we have everything that's being set up by the White House. There are also states that are coming up and saying by the time all of this gets into being, we are still going to go ahead and do something for our own good. So California as an example, New York as an example. You have to declare in New York why a certain person was selected and the other was not. If you're using AI for recruiting in hr, right. Closer Home in California where I am based, there is this other piece of, I'm not going to call it regulation, but let's just say there's another piece of piece that has been discussed and that's going to come out that's going to monitor the way AI is being used. So if these, you know, just the reputational loss wasn't enough, I strongly encourage companies to think about the regulation that is going to come. Again, no one has a crystal ball, no one knows when it's coming. But it's for your own good to get ahead of it and get a handle of how you're going to handle it, whether it's through policies, whether it's through having an AI inventory, whether it's through having a user knowledge, all those good things together.

So I'd say that generally, at least I see usually enterprise companies are a little further ahead because they have bigger data teams. They maybe have had people on al machine learning teams for decades, but maybe for those medium sized businesses that this is new territory or businesses that have really grown very quickly in the last couple of years since this generative AI boom. Can you explain a little bit why there's like how does this bias and discrimination actually end up in these AI models? Right? Because one thing people say is, well, if there's humans training the models and there's reinforcement learning from human feedback, right, it should be bias free. So why is this still a concern when in theory there are humans at these big tech companies, OpenAI, Google, Anthropic, et cetera. Right? Why does this bias and discrimination still make its way into the models that hundreds of millions of people use?

Look, as human beings, you and I both, Jordan, whether we acknowledge or no, have unconscious bias. And I'm not saying we bring it to work, I'm not saying that that's how discrimination creeps in. But we also have to understand that historically the data was very tilted for certain areas in the society. So for example, there were roles in the past that only men were hired for or only men could do. So the data is inclined towards that now. Think about using that data set in this today's world, in this new age that we're all in, where women can do that job as well. But historically there was never that. So when you're trying to shortlist resumes using AI, you would not shortlist Samta because Samta has never done the job before. And yes there is human in the loop and yes, there is a lot of different things that are there. But unless your AI is truly Giving the human the ability to pick and choose and giving the rationale, which it doesn't happen all the time because of the techniques that are used to train the model. It's hard to say. So it's not that. Again, it's not that reinforcement, learning or these data biases cannot be corrected. It's a matter of catching them at the right time before it affects the society or employees or the board or the company in a certain way. And there's always a way to mitigate bias, minimize it. In my personal opinion. This is not Eva's opinion, but in my personal opinion, it is really hard to completely eliminate bias. I think it would take a lot for all of us to get together and make sure that happens, mitigate it to an extent that it doesn't harm anyone, and be very conscious of how you are using AI sometimes.

I think that's such a good point. Right. Something I, you know, I say that, you know, models are both a reflection of the Internet, and the Internet is a reflection of us. Right. So if there's biases, yeah, that means there's probably bias in the mirror, whether you know it or not, whether, you know it's inherent, whether you can recognize it or not. You know, I want to take a look at this from maybe a slightly different angle, you know, because when you think of discrimination, you know, you just mentioned, you know, it can be, you know, discriminating against women, discriminating against certain minority groups, but there's also maybe discrimination against the type of work. And I know even when it comes to AI, you know, people are looking at big companies and they're like, oh, they're investing billions of dollars, and maybe they're just trying to get rid of my type of position in these big companies. Right. They're investing all this money to maybe replace my type of work. How can. You know, I know that's stretching the meaning of the word a little bit, but I think, you know, even we see plenty of articles and studies where, you know, people who are later in their careers, they feel, okay, well, you know, there's. There's ageism here, and they're, you know, doing these models to go after things that, you know, my, you know, my demographic is good at or whatever it may be. Where do you. Where do you stand on that and how can you make sense of that? Because it's a tricky situation.

Oh, it definitely is the cool part, I think about AI Jordan, and I love saying it all the time because it gives me a spot on the cool kids table. But but, but the good part about AI is that as it's hard to point out that bias and like, say, oh, you know what you're doing is not right, but at least you can fix algorithms, right? That's how I see it. So to your point around, yes, there is definitely anxiety. And we have, we as eva, I have done a lot of surveys, we do a lot of our own studies and research. What we found is very interesting, which is the investments in AI are going up, but so is the anxiety in people about AI going up, up. One of the prime reasons for that is because employees don't feel like they are getting enough training, enough education around AI. The more we can give that to our employees, whatever size of your company is, whatever you're trying to do, the more you can let people play with this technology, invest in their upskilling. They are learning, take the fear out of their minds on like, AI is going to replace my job, I'm not going to have it because that's not productive for anyone. It's not productive for you, it's not productive for your company, it's not productive for your employees. So taking that fear out, mitigating it by providing them really good tools that they can play with, they can understand that this is not something that's going to take away my job, but it's probably going to be someone who is using AI to enhance their day to day versus someone who's not using AI to enhance their day to day. It's not about, I'm going to replace this complete person. It's about how quickly are you adapting, how quickly are you willing to learn, how much are you willing to learn? What are the different things you're willing to bring to the table? So there are, I think there's like two, three dimensions to this, right? One company is making sure that they're giving the employees what they need. Employees making sure that they're being receptive to it and getting on board with that and playing with the tech themselves, understanding the pros and cons and how it works and what happens and then making sure that that's been passed along. Those things I think would help the investments go up. I mean, we'll get the investments where they're at, but also get the employees where they need to be instead of the fear.

All right, have to take a quick break and shout out our partners at Microsoft. So if you didn't know, the Work Lab podcast from Microsoft is made for leaders who want to understand how work is changing, because effective leaders adapt. They Stay on top of trends. They embrace any edge that they can get. And effective leaders also know that the key to understanding artificial intelligence intelligence is to get better at understanding human intelligence. So for real world lessons and actionable insights to help you stay ahead, check out the Worklab podcast. That's Worklab. No spaces available. Wherever you get your podcasts. All right, let's get back into our conversation. So, you know, speaking of investments, I think, you know, EY has invested somewhere north of, you know, $1.4 billion into, you know, AI. And I know that covers a lot of different. So I'm wondering, you don't obviously have to spill company secrets, but in general, you all have been putting a lot of effort, money, research, and time into AI, and all companies are trying to do it the right way, and all companies are trying to avoid bias and discrimination. So maybe speaking generally, what are some takeaways that you have seen so far, at least, when it comes to bias and discrimination?

So we. I'm so proud of my firm for doing what we're doing right now, Jordan. Because we're eating our own dog food, right? I told you initially, we have around 400,000 employees. And what we're doing is we're making sure that they are getting all the training they need. They are able to do whatever they need with a very enclosed EYQ environment where they can upload documents. Everything, of course, goes through a lot of confidentiality checks, as you can imagine. Like, we have our clients data that we also have access to, so we are very, very, very, very careful of that. So that's sort of one bucket that we think about when we make these investments so that we're bringing our people along, mitigating the anxiety, mitigating the risk, and also honestly, being able to serve our clients better, being able to deliver value every day, because our people have been playing with it long before this became a thing. Right? So we wanted to make sure we're front and center, leading, cutting edge. The other thing that we're doing is we're thinking about our own business and we're thinking we're a big people business. So, yes, there are ways we can augment and make our employees happy and life easier and honestly, make my life easier. I've been using AI a lot personally to do a lot of different things, but it's about also making sure that we're thinking through how we transform our business end to end. So are there lines of business that need more that can be transformed or disrupted in a very different way? Using AI So we're thinking about it in like two buckets. And all the investments that we're making are going in these two buckets very broadly and everything in bucket two, which is disrupting our business. Making sure everything is coming to bear is what we're also bringing to our clients because we're making a lot of investments with our clients, joint investments, and we're delivering a lot of value by all the lessons learned, not only internally but across the board. So those are different things that we're doing with the investments and working towards mitigating the fear.

Yeah. And you know, we'll make sure in the newsletter to share some of those things that EY has done, some great surveys, reports, information that you guys have put out there, which I think can be a great resource for our listeners. One thing that I want to, you know, you mentioned you have so many AIs in your title, right? Like, like what other thing in there? You know, responsible AI. So when we talk about AI bias and AI discrimination, right. As a responsible AI leader, where does the onus ultimately fall? Because I think, you know, in 2023 and in the early part of 2024 when there were still medium sized and small enterprise companies sitting on this generative AI fence, right. They were almost like passing that bucket around and they're like, no, you take this. No, you take this. Right. Ultimately, for those companies that haven't already kind of quote unquote gone all in on generative AI, who's like, who does it fall on to do this in a responsible way?

I'm going to say something different and controversial, Jordan, but I think the onus of responsibility is on each and every individual. I can have it in my title. You can take the title and have it on your title, Right. And yes, we need to have someone responsible who's going to take care of it, but it should be a part of each and every one of our day to day lives and day to day responsibilities. We can, as an enterprise, whether you're small and medium, whether you're a large enterprise, there's only this much that you can do with the guardrails and with everything that you're setting up. But making sure that this is becoming a second nature to all of your employees who are even touching AI in a certain way, even if they're not data scientists, even if they're not building it, but they're using it, making sure that everyone understands the implications of this technology is key. Now I'm a firm believer of having one person who wakes up thinking about it 100%. But then it is also on all of us to make sure that we have that lens on every time we play with this tech.

So you talk there about being in this, in the day to day, right? And I think that maybe bias in AI models has maybe kept companies from implementing it or not knowing, right. Who is going to be the one leading this thing forward. And people say, oh maybe AI is generative. AI is just hype. And I think we finally moved past a lot of those things and companies have finally gotten to this realization where it's like, okay, we have no choice. So when you sit here, you're in this day to day, you've been in and around AI for longer than all of us or many of us. Where do you see this going next? Right. I'm not asking you to predict the future here, but you're in a day to day. You're working with some of the largest companies in the world on AI implementation and ethics and responsible AI. What should we be looking at next?

So the technology is going to rapidly change as you're probably seeing as well every single day. Jordan, when you share the news, probably you wake up and you're like oh wow, this was very different yesterday. So I think where companies are moving and where things are coming are like I'm, I'm going to talk about a little bit of the technology and then get into the guardrails and the responsibly angle. I think the big piece here is to be agile and not to think about this as like a one time framework setup like you could do historically with a lot of different tech. I'm not saying you could not, but now with AI it's very different. Where I see this going is that where I see companies being successful with where the technology is going in terms of responsibility is being extremely agile, being very open to feedback, giving very strong feedback to the system, making sure that that feedback is incorporated and making sure that you're able to be so agile that you can change the frameworks, tweak it a little bit. Like we are now seeing with the tech. There's multimodal models coming in, agentic architecture. All these things existed for a little bit, one could argue, but now everyone's starting to think about the practical use of it, right? First comes like yes, we build this cool tag, here's what it can do. Then comes like, okay, I'm a big business or I'm a small and medium business. How am I going to use it? Is it going to disrupt me and now we're at a point where we're trying to, we're all doing like practical use cases, implementations, business redefinitions. And with that I think comes in the angle of being agile with the framework that you're going to set up so that again, you can be very, very careful. The other thing I think that is very important for companies to think about is how do you tech enable your responsibility? So how do you truly have an audit trail? How do you make your AI audit ready? Do you have your inventory in a certain, you're making sure that you've logged it in a certain tech based thing so that you don't have Excel sheets flying around which can be easily changed. You want to also start thinking about along with these frameworks, how do I have a solid tech backing and a solid nothing less than being audit ready. To be honest, that would help companies scale a lot.

So Santa, we've covered, we've covered so much in a very short period of time, right? We've talked about the importance of guardrails, you know, how companies are investing so much money, but sometimes the more money they invest, the more anxiety it causes in employees. And we've tackled AI discrimination bias. So many things. But you know, what is, as we wrap up here, maybe what is your one most important takeaway? And I know that's hard, right, to, you know, wrap all this in a bow, so to speak. But for those business leaders who have AI bias and AI discrimination on the top of their mind, what's your takeaway message for them?

Please definitely think about transforming your organization, your business, the society using data and AI. Do not be nervous about that, but please do address responsibly.

That would be my straightforward, to the point. I love it. I love it. This is so good. Sabda, thank you so much for joining the Everyday AI Show. We really appreciate your time and your insights.

Thank you, Jordan. Thanks for having me. It was a pleasure.

All right. Hey, as a reminder, y', all, that was a lot. We got like an education and a half. If you miss anything, don't worry about it. It's going to be in our newsletter. So if you haven't already, please go to your everydayai.com, sign up for the free daily newsletter. I hope this was helpful. If so, tell a friend, right? Where else can you come and learn from the leaders in AI in the world? Right here. So thank you for tuning in. We hope to see you back tomorrow and every day for more everyday AI. Thanks y'.

All.

And that's a wrap for today's edition of Everyday. AI, thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.