Ethical Risks/Opportunities of AI Use Cases in B2B Sales

Thanks, Gerhardt. Thanks, Stephanie. And I just want to again say how much I really appreciate Aicha, Olivia and Courtney being with us today. It's an important topic and I really appreciate their time, particularly when we get news from maybe it was yesterday or this morning. Anthropic Abandons Industry Leading Safety Pledge. So we're in the wrong trajectory. I think at this point they, they were kind of the, the, the, the last of the chatbot companies that were at least making some kind of effort to be transparent and have safety guardrails. But. So that's kind of disappointing. I don't know if it has anything to do with the Pentagon's demand that they hand over all of their tech without any guardrails. But anyway, I digress. So wanted to get through some B2B use cases and discuss one of the most common ones that are being used more and more. And you see new solutions coming out almost every day is in the space of AI sales coaching, role play and automated scoring. And so I just wanted to kick that around to the panelists as far as some of the ethical considerations and ethical dangers that we would see not only on the actual application level, but for the enterprise level itself. And. Anybody care to start or I can maybe Olivia.

Sorry, I am battling a bit of a cold. So if I'm sneezing or coughing or I feel like my voice is sitting in the front of my face right now, which is always fun to do. Jeff, can you give us a little bit more like paint a picture for us here? One of the things when you work as an ethicist is you need the full picture and context which I think speaking to leaders, directly to leaders here, one of the challenges when it comes to ethics and when you're trying to engage with an ethicist or a responsibly AI practitioner is you have to come with a practical case. You can't come with a theoretical. If you come with a theoretical, we can only give you theoretical answers. And in that case there's usually that frustration that people run into because they're sitting there going, well, I asked you a theoretical question and now I have a theoretical answer and I don't have an answer to go forward with anything. So with this B2B case, paint me a picture, let's say you're a senior. Yeah.

Cases, the, the sales manager, they purchase or they a sales application and their sales people, when they're on, they set up role plays where their sales people role play with an AI generated avatar. And that can range anywhere from generic to specifically set up for their industry and any and sometimes even their specific company. So then their sales people will have sales role play sessions with an AI generated avatar, sometimes in actual video, graphic form, sometimes in just audio. And then either real time or post role play, the AI will generate their scoring, say or how they did, and then they get feedback as to what they can do better, where they can improve. This may have something to do with how they get promoted or how they are viewed within the organization. So that's one practical use case to start with.

Yeah, thank you. I think also helps, that really helps with the audience too to give that big picture. Because immediately as you start describing this, I've got my responsibility and ethicist hat on here and I start thinking, okay, well in one way this is a helpful tool. It can help run different scenarios. So I'm sure one of the general use cases around generative AI is being able to run different profiles. So on one hand we could use this to help with understanding diversity of thought and perspective when you are engaging in the sales process. So being able to run a practice session with someone from different demographics, that helps. That's actually a very helpful tool to have. I think there are limitations there though, because you have to come with the assumption that each of these different profiles that a sales avatar would be, would be creating, that your employees would be engaging with, those are built off of stereotypes. So yes, you may get a generic, let's say a generic white woman.

Great.

That doesn't necessarily mean that that avatar is going to fit every single type of person that you're going to run across. So understanding that there will be biases already built into how those avatars are portrayed. But another thing that I think is important here, which Jeff, as you were talking and about how this leads to scoring, coming from an entrepreneurial background and taking one too many sales workshops myself in my lifetime, one of the key factors of sales is actually the work of empathy, being able to understand the person in front of you. It's not just running through a script, it's not just hitting certain KPIs, which is really only what these AI tools can do. So yes, these kind of AI powered tools that help train for sales calls can help in terms of the basics. It can help create a baseline understanding for sales of yeah, you gotta make sure that you're discussing budget and you're discussing timeline and you're discussing all of the key factors when it comes to operationally engaging in sales, but it's not going to give you any indicator of can the salesperson actually engage empathetically with the person that they're trying to sell? That's something that you still need to be able to do face to face. So with these kind of tools, I'm thinking immediately, great, you've got a good line of defense for some baseline training. But then how do you complement that tool with senior sales executives to be able to pass down, here's the institutional knowledge, here's the human knowledge, here's the empathetic knowledge when it comes to that sales process. So it's a double edged sword. All of these are double edged swords. It's just knowing where the limitations are and where to complement it with the human side to things that makes the difference between it being just another AI tool versus something that actually makes a difference for a company.

Right, right. And Courtney, I think you wanted to weigh in.

Yeah, I think there are a few things that regardless of the specific use case, are broader concerns that are both with respect to ethical and responsible AI that you know, every organization and individual using these should be aware of. And so one of those, for example, is just the deskilling trap. So we've seen how reliance on AI systems can lead to a deterioration in the skill set for given expertise domains. And so, you know, given that one of the things that makes really successful salespeople is that individual level connection, we need to also make sure that, you know, this doesn't become too reliant on the AI systems. I think you, we also need to realize that, you know, I think Olivia was kind of alluding to this, but you know, non native English speakers you mentioned, we have people from 60 some countries here. You know, women, introverts, people from non western businesses, etc are at particular risk of being systematically underscored on criteria like confidence, like assertiveness, vocal president presence, being misunderstood. Because a lot of these systems are calibrated on a narrow ideological archetype of what say a successful rep looks like. So you need to think about where did you get that system from? What data was it trained on? How are those avatars being created? We've seen this issue of bias and scoring and career consequences show up in Amazon and academic research, etc. There's another ethical concern which isn't about, you know, whether this is going to make your sales reps better or not, but did your reps consent to being monitored to the scope of behavioral monitoring that is embedded in these systems? Who owns that data? Does, does the company that you're contracting with for use of those avatars? Gain access to that data? Are they using that to train their systems or what institutional IP control do you have over that? And the individuals that are being monitored, is that clearly conveyed to them in their employment agreement? What about the disclosure of their vocal patterns, their word choices, their approach, emotional cues, et cetera, all of which are being continuously analyzed and potentially scored? You need to ask yourself a what are these systems actually doing to learn and integrate? You know, how this feedback and agentic role play is shaping the system or fine tuning it. And you need to make sure that you're disclosing and having a transparent discussion with employees and with self, you know, with them so that they don't lose self dignity, they don't lose, you know, what makes them special. And that personal, you know, connection that salespeople have really can't be replicated by an AI. And I would say just to wrap up, you know, one of the things that organizations should be aware of is the feedback loop. Because if you're using agentic or you're using, you know, AI role playing to shape behavior and then score that and give you feedback, you're potentially creating a closed loop with no external validity check. And so then the model could reward or give feedback to the agent in ways that it predicts will correlate with success. But if the underlying model is miscalibrated or the market has shifted, there are new dynamics. You may not know until you know your numbers deteriorate and maybe then you've trained your team in the whole wrong direction. So there are a lot of different pitfalls from various different perspectives that organizations should be aware of.

Aicha, I know in speaking with you, you've kind of had a sales background at one point. So I turn the floor over to you.

Thank you, Jeff. I think what both Olivia and Courtney said, I agree 100%, particularly about deskilling that Courtney has mentioned. There's an loss of opportunity to have human connections. I think this kind of skills that are very hard to code, that are not observable, that are only observable but not unspoken, so to speak, that pertains to relationship building, empathy, conflict resolution, the unspoken cues that you can get. And I think those are really the kind of skills that are going to be very important, particularly in sales because they are the hardest to automate and they're going to have a premium in terms of, I hope to believe that the humans, as humans, we're going to have that premium going forward. But I want to take a step back based and say a few words about what we do at the center to give the context of what I'm about to say, we published the Artificial Intelligence and Democratic Values Index which is a comparative study of 80 plus countries based on 12 metrics and their national AI strategies, alignment with democratic values, human rights, the rule of law. That sort of sets the foundation for what has now become the universally accepted I think terms of trust. What makes trustworthy AI transparency, accountability, explainability. And what we see in this index is overwhelming excitement about the kind of things. Obviously over 80 plus countries now have national AI strategies about what AI can do and tracking the regulatory framework, the policy developments. There's one thing that is quite significantly sticking out and that is about AI can be reliable and good when you have good data and that is hard data based on hard sciences. When we come to behavioral sciences, when we come about human behavior, that is the hardest part for the AI to tackle with why human behavior is dynamic. And most of these and so far I haven't come across with any that can provide scientific validity. And that's one of the reasons that across the globe that we're seeing in particularly in the EUA AI act for instance that for instance emotion, emotion inference is prohibited under the EU AI Act. Now emotion sentimental analysis which can be provided in these coaching platforms, for instance about your tone about your not your accuracy of knowledge of the product and services. And I think that's something that AI can be good at judging if you provide good data. As to this is sort of goes into more rule based AI. The reliable, the good old fashioned AI that we can understand and explain. If then scenarios that can be accurate. If you're measuring a salesperson knowledge on a product or a service versus if you're measure measuring a salesperson sentiment, that's when it gets blurry. Now obviously then how you use that also matters. If you use that information to make decisions around promotion, hiring, firing, around employment and again overall also within the United States and across the globe we see that as being a very high risk area. Particularly because they are not reliable, they are not accurate. And for all the reasons that we have discussed that there is bias, there could be bias in the data set. But also human behaviors are very dynamic. From my talk, for instance, each of you might come up with different conclusions as to my current state of mind. But you don't know whether I am in a cold temperature or in a warm temperature. My cultural background allows me to use my hands a lot and this could be inferred as excited, anxious or just my regular hot blooded Turkish self. You couldn't tell. I have a slight accent. Some words might not be picked up. So again, it's really about the bias and the data. The fact that if these are not scientifically valid and reliable and if we use this information to make critical decisions about someone's employment and access to opportunities and livelihood, these are high risk areas. That's where we go into the human rights and also as Courtney said, the lost opportunity, the skilling, learning loss from having that opportunity to interact with a human versus with, with, with AI. But again, it's really about knowing these things that you go in there and then you put the AI in a place where you know where AI strengths are versus the weaknesses and, and you go from there.

Right. It. So that leads to a question. What we see in a lot of these scenarios, and I know that Olivia was asking for a practical use case, is that most sales organizations are not building it themselves, so they're going out and sourcing third party sales role play applications that are developed by AI companies. So Olivia, like what are some recommendations or questions or information that companies, sales companies should be. Organizations should be looking for and asking for specifically to ensure that the, the bias is not there. Because most all of them on their web pages will say, you know, it's, it's automatic, it's accurate and it's unbiased. But there's, there's nothing in the footer on most of them that talk about responsible AI or anything other than privacy. Like, you know, the usual, we have your data encrypted. You know, beyond that you don't see a lot. So.

Yeah, so I actually, let me see, I probably have three I got a tip, buyer beware and then recommendation, let's say that come come to mind immediately. The first one being actually if I am sitting with, if I'm sitting with sales reps, I do a lot of vendor procurement analysis and assessments for, for my clients understanding whether or not this is a tool worth buying and the ethics behind it. One of the first things that tips me off is if it's being sold as not having any bias because it's impossible to eliminate bias. What I want to know is does the supplier, does this vendor actually understand that that is the baseline that they're working with? And so they are trying to mitigate, actively mitigate that bias. They understand, they recognize that it is impossible to fully ever eliminate bias. It doesn't work. So if you're getting these grand statements of we've completely debiased our system and we've debiased the entire sales role playing process. To me that means they are completely overselling and they don't know what they're selling or they don't know really what they've built in the nature of what bias means in these contexts. So that's always the first indicator for me. Then let's say a tip. Maybe a tip. When I'm looking at assessing different vendors, I tend to ask questions about who were the people that designed it behind the scenes. So if I am talking to a vendor that has designed these role playing systems and they don't have any previous sales experts that were a part of that design process or a part of their team, to me I'm looking at it going, okay, so you had a bunch of, well, I say this lovingly. You had a bunch of tech bros get a bunch of data together and say, hey, we can build an AI off of that. It doesn't mean that it's actually designed to work. It just means that they found some really cool data and they figured out what, they put some code on top of it. What you really need is the feedback cycles happening from the vendor. So was this, is there someone within that team of that vendor team that has extensive sales process sales experience that comes from a background where they would have gotten the human experience needed to then help translate and make sense of the data and the system and the, and the product? That I would say is a really strong indicator for finding a good product on the market since it's really difficult to necessarily sort through the amount that's available. And then the last one, and I'm boiling a lot of these down, there's a lot of different things that you need to look for. But this actually goes a little bit towards Courtney's point about data privacy. This is going to be sensitive data that these systems are handling. You are within your right and actually should be asking where that data processing is happening. Where is it going to? For example, if you are looking at a vendor that uses a company like let's say OpenAI, what's the relationship between that vendor and OpenAI? Is your employees data and information being sent to OpenAI without restrictions? Meaning OpenAI can train on that information, OpenAI can access that information, that's highly sensitive data and OpenAI cannot have access to it. But that's a question that you need to be asking the vendor to make sure they have those safeguards in place. So you are completely within your right to be asking as a potential buyer these questions around data privacy. It's actually not even I'm not even going to. It is legally required in some states. It's legally required in some countries. It is also something that you need to be asking about. This is inside this. And excuse me, I'm trying to finish the sentence without sneezing. The last point there they're being. If a vendor says, I'm sorry, that's proprietary information, then you can say, then your response needs to be great. That means that you're not my vendor. None of that's proprietary information. It never is. That is information that needs to be upfront. So that's a tip, a few tips and some. Some pieces that I look for, especially during those vendor kind of conversations.

Can I. Courtney?

Yeah. Can I just add. I mean, I think those are fabulous points. Especially, like, if you leave with one thing today, know that you cannot. That there is no such thing as an unbiased system. The question is, what are you correcting for? And so another. And just to build on what Olivia mentioned, I would say also one thing to ask the vendor is, has the system been audited for disparate impact across, you know, gender, race, language, other factors that might be important to your, you know, potential sales base, but also, you know, to your reputation as a company? What kind of company do you, you know, want to be? Are you one that is concerned about the ethical implications of these systems? And also, in addition to, you know, asking the vendor what their privacy policies are and data sharing, just to understand what are the specific types of data that are being collected, how is that being stored and. Or shared, and to what extent are you, as the company and. Or the individual, getting access to that information? So say you sign up with, you know, one vendor, and then, you know, it's actually not working. You determine that there is, you know, some sort of, you know, real bias and you're not getting the outputs you want or, you know, your sales reps don't like it, what happens to that year's worth of training and feedback? And I think that when you're negotiating these contracts, remember that as you're figuring out how that data is being used, you're also providing value to the system. Right? To Olivia's point. So say you're a vendor and you have no salespeople, but all of a sudden, you know, these sales people are using your product. You're now creating a much better product. So you might also want to be asking yourself to what ext. Does it make sense for us to develop our own internal product? To what extent are we using something external? Are we Doing a hybrid. What are the rights that you're retaining over the data that's created by your organization and by your employees? And how, if at all, is that disclosed? I think those are, you know, some of the additional important questions to ask. And I would just add one other thing. So in addition to not talking about bias, I hope that no one on this call will ever use the term hallucination to describe AI. Again, it is an anthropomorphic term that does not describe what's happening and it impedes your ability as a company or an individual using these systems to understand what's going on. So you could talk about error rates. So you could ask, you should start asking the vendors, what is your error rate?

Right.

We're not talking about telling people to, you know, eat glue on pizza, for example, is an factual error. And so we can have discussions about error rates. Or if you're using a creative thing like an agent, where maybe there's not a right or wrong, you could talk about fabrication. Because again, these are machines. And I would say that, you know, just to wrap up on this point is these are machines. So as you're integrating them into your cycle, try to avoid that anthropomorphism. And we know that people are developing relationships with AI, companions, with chatbots, etc. How are you prepared for that as a company? How are you trying to mitigate, you know, dependency, mitigate, relationship building? I mean, this is a new frontier. So getting out ahead of some of these questions and thinking proactively about that could be really helpful as well.

And I think, I mean, is it, is it true? I mean, is it signing an NDA to get the information just to protect both parties for, you know, their, their intellectual property? That seems, is that a reasonable workaround? You know, instead of saying, well, we don't, that's proprietary. Just, you know, companies should offer, the solution provider should offer an option where you get that information, all that you need. But maybe an NDA so you don't release it?

I don't know that that. I mean, I don't know that that would actually cover the, the issue. Right. So if you're, if you're talking about feedback loops and fine tuning and training, etc, you know, whether or not they ever have access, you know, or able to disclose, I, I think that is a separate issue. But I'm not a lawyer. I have taught at a law school, but I don't have a law degree, so maybe somebody else on the call knows.

So transparency is key.

I mean, transparency and accountability and as Olivia said, if they can't, if they don't know, they can't disclose or they're overselling, that should be a red flag for that vendor.

So, explainability. Aicha, do you want to wrap, wrap it up? Because it looks like we're only going to have time to get through one use case, but what we've talked about can, can really, I think, apply for many of the other use cases, the GTM, the CRMs and things like that. So what do you.

I think I'll say two things. One, obviously I would just say, you know, don't try, don't aim to be bias free, but aim to be bias aware. And that goes to transparency. There is a bias, there is a choice. From the way that the data is collected to the data is labeled, to what is optimized, to its algorithms, to its deployment, it reflects a choice. So the bias is the, is the nature of things. Just like humans have bias and they're a reflection of us, we cannot say that we are bias free. What we can strive to be is aware of our biases. So we stop ourselves from acting on these biases in a way that can harm other people, that can result in a discrimination. So being bias aware is a very, I think, achievable goal. And that goes to transparency. Now, transparency requirements, that's where all the fuss around regulation is about getting companies to be transparent, requiring reporting around their algorithmic, algorithmic transparency. For instance, some countries adopt that by law, by law to have algorithmic transparency. And these transparency requirements currently in this United States sort of vary and they are not very enforceable. So the companies can easily, and maybe not some places more easily than others can say, we don't want to and we don't have to share this again. That's why I think we're having this conversation, is because AI is moving faster than the compliance or regulation can keep up. So individual folks, the leadership teams, the sale teams, it's really up to you as an individual knowing and understanding the risks. That why these regulation talks are happening is when you look behind them, they're very common sense and they're very reasonable. It's nothing unreasonable to ask a company to do audits, to share these things. And the second thing I'll say is about overconfidence and reliance. We depend a lot on data and we tend to see when you have a confidence rate or a scoring rate that gives you, this is 82% accurate, this is 42% lead score right here. It just kind of gives you clarity. Okay, Let me do this. And. But you unpeel the onion and you look behind those data, then you see, okay, there is the reliability and the overconfidence, and there is the error rates and calibration. What is really there that you see is your over depending on something that you really shouldn't. And then you can lose your agency and you can lose your skills in control over time. So those are the two things that I would say to pay attention, to be aware of the biases. And that's what's for transparency and ask questions. Don't just sort of take the AI scoring for, for something that's overconfident. Keep on. Keep in mind that that will come across as overconfident.

And one, one last item that I wanted to give to the attendees is a passage from the book I just finished by Olivia that I think this, this is very appropriate. It's very easy to care about the values our technology is being built on to care that ethics is embedded in our business practices, to care that we develop AI responsibly. But there is a wide gap between care and action. So let's take action.

Well read, well written, great conversation. What do you think, Stephanie?

I thought it was excellent. As always. Like everything here is just all these little nuggets.

Right.

And I love that point at the end of, you know, just, even just being aware of the bias that we have is just so helpful to understand that, you know, you have to take a good look at what we're dealing with here. There's so many use cases where companies just didn't take that into consideration and they built out their AI systems on biased data and those biases continued to move forward. So very insightful, thoughtful. And again, I think leaders are going to be on the edge of their seats rewatching this, thinking about the ethics involved in the risks.

Yeah. And it's a good reminder that we are human beings and not human doings. And that machines that evaluate human doings, they cannot match the nuances that you brought out. Like, you know, just talk about hallucinations. Machines don't hallucinate the error rates. And that, that was a great insight, great takeaway for me that we are humans with so much complexity that outmatches any machine analytics. And I've seen so many demos of video role plays that are all designed to improve performance and clone best performers. And it's all marketing speak. But in order to truly help people, we need to care about their welfare and we need to consider all the aspects of is it legal? Is it right? Is it just. Is it effective? Thank you so much.

And, Aicha, Olivia and Courtney, thank you so much. I really thank you.

Thank you.

Glad to see this issue on the agenda.

Thank you for having us.