Trust: The true limit to autonomous AI - AUTONOMOUS session

So let's see, it seems like everybody has been pushed here. So AI is evolving at an unprecedented pace, but its success or failure comes down to trust. It's my distinct honor to introduce our next speaker, Laura Stevens, who's a Managing Director of Data and AI at voi. She's a brilliant leader in AI strategy and implementation, but also someone I have a privilege of working alongside. So with that, I will kick it over to you, Laura, and take it away.

Thank you so much and good morning, good afternoon, good evening to everyone. Thanks a lot for attending this session on one of the key emerging topics in the AI space, which is trust. Quickly, something about myself. So my name is Laura Stevens. I have an academic and research background in the area of neuroscience, so hence my interest in the more psychological aspects of AI as well. And I just transitioned out of my role as VP Data and Artificial Intelligence in the industry to the now becoming a very proud member of the BOI family. So the summit has shown that many of the substantial benefits and opportunities that AI solutions can offer. Now, despite their potential, we also know that our attitudes as humans towards AI vary very widely. So some people quickly embrace these tools, while other people just remain very skeptical. And that's not just what I observed throughout my work. It is also confirmed by many, many studies. In fact, research shows that people, whether it's citizens, employees, consumers, often resist relying on algorithms, especially for those tasks that are traditionally performed by humans. That's also better known as what we call algorithmic aversion or AI bias. And it shows itself actually in very different ways. First, active resistance. So people simply rejecting AI solutions. Second, irrational preferences. So people favoring humans over better performing AI. So even when shown evidence of AI's superiority. So that's quite an interesting one actually. So people often consciously choose to rely on human experts, despite being informed about the algorithm's higher success probability. Thirdly, what I call the no mercy mentality. So we hold AI to higher standards and we penalize errors much more harshly than human mistakes. So for some reason, we demand perfection from machines. So why is that? We'll get back to that in minutes. And then finally, we also note and see that trust often seems to decrease when people actually learn or are informed that a recommendation or a decision or a product is AI driven, even if they couldn't recognize it in the first place. That phenomenon is also more commonly known as the negative impact of AI disclosure. And so lack of trust, and this is why it's such an important topic, really controls the level of diffusion and the level of adoption of AI solutions. It explains why people hesitate to use AI whether they decide to to buy or not to buy your AI solution, and also why many companies often struggle to scale their AI solutions. So let's quickly explore one of these specific manifestations of AI bias or algorithmic aversion more in depth. So I spoke about the negative impact of AI disclosure, which by the way, has been found across many different contexts. So AI disclosure refers to how companies communicate that a recommendation or content is AI generated. And we know from studies that these kind of disclosures often lead to very negative reactions. Just a few examples. In commercial settings, we know that revealing that a financial product is promoted by an AI agent actually has been found to decrease purchase rates. So it seems that consumers perceive those AI agents as less knowledgeable, less empathic, and leading to rejection. Similar patterns have been found in the workplace. A recent study found that AI driven feedback actually improved employee performance with almost 13% until people were made aware of AI's involvement. Disclosing the fact that feedback was provided using AI as opposed to given by human managers, triggered negative reactions, even to the degree that it offset the benefits that it initially brought. And then finally, AI generated content, whether it's artwork, music, emails, is often rated lower when people know that it's created by AI, even if they can't distinguish it from human work. So these examples really show that there's a very complex and an often emotional interplay between AI and technology and human trust. And so now that we've seen how trust is quite a challenge in the adoption of AI solutions in general, the question also becomes, how will these trust issues play out as AI takes on greater levels of autonomy, as is the case with AI agents and agentic reasoning. Now, while research in that area is still relatively limited, the available evidence actually suggests that as autonomy increases acceptance and also the behavioral intention to use AI decreases. One of the most obvious examples is probably coming from the area of autonomous vehicles, where a lot of research actually shows that trust in these cars declines as their autonomy levels rise. So the less control autonomous cars offer to their passengers, or the less they allow the human to remain in control or in the loop, the more aversion they trigger. And I guess many of you now think, yeah, but autonomous vehicles is a very particular area. But in fact, lack of trust also impacts the adoption of more autonomous forms of AI in domains like healthcare, customer service, or shopping. So just a very recent study showed that, for instance, the level of autonomy of AI shopping assistance had significant downstream effects on consumers personal sense of Power. And so the study concluded that high AI autonomy might be less preferred by consumers. So just remember, in general, the less control users perceive themselves as having over AI derived decisions, the less trust they put in it, the more aversion they feel towards it, and the less likely they are to use or buy these solutions. And then I think a second reflection of some of you may be that this is just a matter of safety, and that as the safety and the reliability of autonomous autonomous applications would increase, trust will grow and so will adoption. So then let's explore a very crucial question. How safe is safe enough for people to accept it? We know that every year close to 1.25 million people die in car accidents, and more than 90% of which are said to be caused by human errors. We also know that self driving cars eliminate many of the common causes of accidents made by humans. And just think about distracted or drunk driving. So then you could argue that if autonomous vehicles can save lives, the commercialization and the adoption is a moral imperative. Right? And still, if we look at research, we see that consumers remain skeptical about autonomous vehicles and people experience serious psychological roadblocks in terms of assigning control to an autonomous agent. We also know that autonomous vehicle accidents, they are very rare and still they receive extensive media coverage. So unlike the thousands of daily human cost accidents. Just think about the first fatal accident that involved Tesla in 2016. It had a huge impact on public acceptance of autonomous vehicles. And we also saw the level of interest in them declining, together with Tesla shares. And so our emotional reactions very much outweigh the low probability of occurrence of these fatal risks. And so that raises the question, how much safer do self driving cars actually need to be before they will be widely accepted? If we look at available research, it suggests that algorithmic aversion or AI bias will seriously inflate the safety thresholds at which consumers will actually be willing to purchase or ride in autonomous vehicles. Let's just look at another case in point. In healthcare, human diagnostic error rates, they range from 5 to 15%. And we know that AI is increasingly outperforming human doctors in diagnostics. And still mistakes made by AI are perceived as much more alarming. Why is that? Let's maybe reflect on a personal question about trust in AI. So would you board on a fully autonomous flight with no human pilots if told the AI pilot performs better? Or would you put your children on such an autonomous flights? Or maybe consider a closer scenario. Would you give an AI agent access to your credit cards? I'm very sure that there's probably a very big only if to the to your answers. And so these examples actually show a few critical things. So trust in AI, whether it's autonomous or not, presents unique challenges that go beyond safety or reliability. And our psychological reactions to AI in general and autonomous AI are anything but rational. And so while safety reliability improvements will help, they alone won't solve the trust problem. Now, some of you may think these are just transitional challenges which will disappear as our familiarity with these applications grows. And while history has indeed shown that people adapt to new technology over time, it is also fair to say that autonomous AI comes with unique challenges which make trust even more difficult to establish. Especially the independent nature requires humans to delegate power, which we know is not easy for most people. We know that the complexity of these autonomous applications and often the reliance on more deep learning algorithms comes with transparency challenges. The real time adaptability makes them sometimes unpredictable. And we also note that their inherent lack of moral reasoning introduces risk, especially if deployed in high stake environments. And so the more autonomous, the more these systems challenge our very core human needs. Very psychological core human needs, a sense of uniqueness, our need for control and our need for predictability. And now that we see the AI space and AI capabilities entering areas which have long been considered as very uniquely human. So for instance, creativity, it comes with growing fears of job displacement or replacement, as already discussed at the summit, which further contributes to resistance. And just as a reference, I found a recent study which said that 15%, 50 so 5o percent of workers hesitate to admit to using AI just because they fear that it makes them appear replaceable. Then there's very deeply rooted psychological biases, I won't go into details, but all contributing to our lack of trust. And so no algorithmic aversion is not just a transitional challenge. While familiarity will reduce some of our resistance over time, I genuinely believe that companies will need to continuously address trust concerns as AI evolves, even when the reliability of these systems increases and even when these systems outperform humans. Now, given the unique challenges and deeply rooted psychological biases, building trust in autonomous AI will require more than just better software, better hardware, better model performance. So then what does it actually take to build trust? So, generally speaking, there are four big focus areas that are key to building and maintaining trust in AI. So these emerge across a variety of different studies. The first one, reliability. This is about ensuring consistent performance across different and also unpredictable scenarios. Straightforward. Second one, helping users to understand how AI works and also setting realistic expectations. Third one, empowering users. And I think some of the Speakers today have already emphasized it, empowering users to shape and control AI behavior and then finally making sure that we promote fairness and transparency. In the context of this session, I'm not going to dive into the technical aspects that increase AI performance because they have been and are still being overemphasized in my view as a neuropsychologist. But I do want to end this presentation with a focus on some of the tactics tactics to address the human based focus areas that increase trust. And I will do that by introducing a practical framework for human AI collaboration which is really designed around the key elements that are known to foster trust. So the partner framework visualized on this slide highlights some of the actionable steps that organizations that developers can take to foster and maintain trust in AI. And it's based on available scientific research. So it's not just my opinion. In short, the P stands for participate and it refers to the value of involving end users in the design process. We all know this and still I see many organizations actually under invest in this. So studies show that users who are engaged in co creating solutions feel a stronger sense of ownership and are much more likely to trust and adopt the system. So here think about organizing workshops with potential customers employees to really identify pain points, gather input on how AI can address daily challenges, make sure to share prototypes with end users, collect their feedback and use it to refine a solution. So this is really about showing users that their perspectives directly shape the final product. Again, it sounds evident, yet I still see many organizations really underestimate or underinvesting co creation. Second factor, the A stands for assess and that one refers to the importance of regularly checking for bias and addressing fairness issues. We know from studies that bias very quickly erodes trust in AI while fairness. So treating diverse user groups equitably enhances it. A lot of research in the area of autonomous systems such as self driving cars, autonomous weapons or high stake areas like hiring, lending, they all show that unresolved ethical dilemmas will lead to public rejection. How to apply it we all know how to address bias. We incorporate bias assessments into our workflows. We use diverse data sets to ensure the outcomes are fair across different groups and we also establish and integrate clear fairness metrics into the AI systems performance evaluation. Third factor, realism. This one is about setting realistic expectations for AI performance. So as already discussed, people tend to hold AI to higher standards than other humans and they show less forgiveness when AI makes errors even when the AI performs as well or even better than humans. It's these unrealistic Expectations about AI that often erodes trust when the system makes one error. However, we also know that very experienced users, those who understand AI limitations, are more forgiving of mistakes. What that basically means is that managing expectations about AI's capabilities is key to prevent over reliance on and also reduce the risk of disappointment. How to do this? Think about designing expectation adjustment techniques like using onboarding tutorials in app messaging to prepare users for AI imperfections. You could for instance, include disclaimers about accuracy levels and examples of scenarios where the AI might struggle just helping users to form realistic expectations. T stands for train and is all about providing practical training to build confidence and and understanding. Again, we know from science that well trained users are more likely to trust and adopt AI successfully, while those with only moderate knowledge are often much more skeptical. So we all have an innate need to feel good about ourselves. And so it's evident that increasing understanding and confidence through practical training will help to grow trust. The N stands for notify. And so where realism was about expectations and train was about skills and capabilities, Notify is all about transparency. We know that transparency, and it has already been mentioned several times throughout the summit, is absolutely essential for building trust in AI. And that when users understand how AI makes decisions, they are more likely to trust. Even seems that transparency is so crucial for people and for trust that it's more critical than the actual reliability of an AI system. So some studies have effectively shown that even when the reliability of a decision making agent is low, informing people of that actual reliability enhances their trust. So it helps people just to understand how much and when to rely on the agent. So here think about designing user friendly interfaces that really provide concise, easy to understand insights into how decisions are made. And do make sure to balance transparency with practicality, right? So avoid overwhelming users because we also know that that leads to a paralysis. So the art is really to provide layered explanations for diverse needs for diverse groups. Think about high level summaries for non experts alongside more detailed and technical insights for experts Empower. It's about giving users control to adjust and even override AI outputs. So again, several studies have shown that users place greater trust in AI systems when they have the ability to modify or steer the outputs and the behaviors. In the area of autonomous vehicles, for instance, we know that features that allow drivers to control the system behavior, such as being able to put it on manual mode or having an emergency button, all lead to higher acceptance. And so in general, just don't frame the decision to delegate or rely on a system as an all or nothing decision, but rather give users a sense of control. In the same line. Also consider to design systems with interactive features that allow users to tweak or to modify recommendations. Think about including sliders or filters or editable fields that lets them adjust the parameters or refine results based on their judgments. A few examples, should they not come to mind, could be a medical AI system which lets doctors adjust the sensitivity of a model to reduce the false positive or the false negatives. Or think about ChatGPT or LLM tools that let users choose the level of creativity or formality in the text generation. Or think about a financial forecasting tool that allows users to modify assumptions like inflation rates or market trends just to see different predicted outcomes. Finally, refine. While the ability to adjust or overwrite AI recommendations empowers users in the moment, refine is all about providing the ability to integrate continuous feedback and I've experienced it myself in my work in the industry. Feedback is a game changer and so research also confirms that users highly value systems that are adaptable and responsive to their feedback. So here set up user friendly accessible feedback mechanisms like the simple feedback buttons like thumbs up, thumbs down. Or think about using guided feedback forms like short surveys or polls that ask specific questions about the system, just ensuring that feedback is actionable and easy to analyze. There's a lot more detail behind each of these pillars, but for now I would invite you to just remember one very important note, and that is that the industry is so fixated on model accuracy hardware, so software improvements, but that psychological barriers are just as critical as the technical ones. And so without properly addressing these, we also know that even the most high performing AI will struggle to gain widespread acceptance. And so it's absolutely vital that companies stop treating AI as a technology challenge and recognize it as a human transformation challenge. So make sure that that's also reflected into your AI strategy and into your transformation plans. Many thanks. I think we have time for questions. I haven't seen any questions, so you'll need to guide me through.

Yes, phenomenal. Laura, that was so awesome. You managed to anticipate a lot of the questions that are coming up with that practical walkthrough the framework. So you're getting a lot of love in the chat. I'm seeing so really, really phenomenal work. There was one interesting question from Bram. How does anthropomorphization so making AI seem more human, how does that affect trust and how does that interact with some of these practical examples you talked about?

Yeah, it's a very good question. And actually the answer, if we look at scientific evidence, is quite mixed. There are studies which shows that the more we give human traits to AI, the more likely people are to trust it. But we also know from humanoid robots, for instance, that they can sometimes trigger feelings of anxiety. The research and the evidence is mixed. So I guess more research is needed and we should set up experiments to actually see which little tweaks can actually make the difference.

Awesome. Yeah, that's super interesting a question I have. As you're walking through, you can see how that could apply to design and to strategy. So maybe designing more specific tools and thinking of a broader strategy. How do you think about the interaction of the partner framework and those different sort of altitudes?

Yeah, so you mean the interaction between the different components?

Yes, between the components.

Yeah. I think in general, what they all have in common is that AI is not just a tool, it's an agent making decisions. It challenges human identity. So replacing human creativity, expertise, autonomy, that's at least a sentiment. And we know that past technology adoption didn't require people to delegate control to the same extent as AI. So what all these elements actually are about is about providing reassurance by using some of the mechanisms that have been involved in establishing trust, by definition in technology, which is about collaboration, helping people to control the AI's behavior. So all of them actually meets some of our inherent needs for people and some of the, of the deeply psychological needs of humans. So that would be my view on the interaction.

Got it, got it. Very interesting. So I'll call out. We have about two minutes left. @ this point, I do want to say, hey, after this, everyone will be pushed to some closing comments from our CEO Phil. Encourage everyone to hang on for that. And I think with a minute, 45 seconds, we can ask one more question. I have a question from Izzy that I guess maybe may or may not be Izzy. Is there a difference in the resistance or trust levels between men and women?

That's an interesting question. Actually, I haven't looked at any evidence. There is some evidence that there are demographical differences between young and old people. There are cultural differences as well, but I haven't looked for gender specific differences. I'll look it up for you.

Great. And I know I can also see that we have people who want more information on the partner framework. So definitely what you're doing here is sticking. And with about a minute left, I think we can just about wrap. You can close with anything you want and then everyone will get pushed.

Anyone who wants to know more, feel free to connect with me on LinkedIn. There's a little bit more about the partner framework on my LinkedIn and of course, more than happy to connect and chat about it a little bit more in depth.

Great. Thanks so much. In the next couple seconds, everyone will get pushed over to Phil. Be sure to hang on and hear from him. And then thanks so much for joining and thanks to you, Laura.

Thanks a lot. Bye. Bye.