Designing AI to Make Decisions

AI is transforming the world, and it starts with the right compute. ARM is the AI compute platform trusted by global leaders, proudly NASDAQ listed, Built for the future. Visit arm.com discover. Welcome to the hbr ideacast from harvard business review, I'm sarah greene carmichael. Artificial intelligence, or AI, is one of those technological phenomena we tend to have pretty drastic black and white thinking about. Either AI will save us all, or it will lead to the downfall of civilization as we know it. But we might just be focusing on the wrong things. While AI might be able to automate a lot of tasks in the office and beyond, it's still built by people, and it often draws on the knowledge that real people have developed over time. As our guest today explains, using the example of cancer diagnosis, I think if.

We think about it critically, it matters less why the algorithm indicated that the lung was cancerous and more that there's a very high probability that, you know, that somebody has cancer. As a society, as opposed to our saying, oh, my God, we can't adopt these tools because we can't explain them. We need to be stepping back and saying, well, what really does matter for this instance? And if there's a high percentage and probability that somebody might be ill, then the ethical act might be to go on and treat them. And it also might be more ethical for us to use systems that have higher accuracy rates than a human, because when we train a system to do something like diagnose cancer, we're basically collecting the knowledge that thousands of doctors, doctors have made judgment calls on in the past and accumulating that into a system. But that causes it raises a lot of uncomfortable questions related to the status of various professions.

That's Kathryn Hume, vice president of product and strategy at Integrate AI. She's here to help us delve into the shades of gray surrounding artificial intelligence today and to understand where it could go in the future. Catherine, thanks so much for joining us.

Thank you so much for having me. I'm delighted to be here.

So the term AI or artificial intelligence gets thrown around a lot. So I thought maybe just to start with, we should just say what counts as AI. You know, when we talk about it, what are we talking about? And how is it different from robots or machine learning or algorithms or some of these other terms people might have heard?

Yeah, that's a really great question. And it's definitely a loaded term that means different things to different people. I, taking the lead from my former colleague Hilary Mason, like to define it as whatever computer until they can, which is sort of a Psychological take on the definition of artificial intelligence as opposed to a technical interpretation. But I find that it does the best at incorporating the notion of progress and development into the definition itself. So if we think about it, a tool like Google Maps, which is run using complex machine learning algorithms and tons of data, is by this point just viewed as standard old technology. But if we were to go back into the past 10, 10 to 15 years ago, a tool as sophisticated as Google Maps that could give us real time traffic updates was something that could seem almost like artificial intelligence. By contrast, something like automated vehicles are viewed as artificial intelligence because we're just on the cusp of being able to automate these kind of tools. And we therefore view our capacity to drive and manage an automobile as something that is squarely within the domain of human intelligence, as opposed to something machines can do but tend to 15 years down the road there will be commonplace. And then we'll move on to the next challenge of what qualifies as AI.

Is it really true that it's always a moving frontier, or are there things that genuinely do mimic human intelligence, algorithms that do kind of improve autonomously on their own, that might count as AI more than sort of something that doesn't?

Yeah. So when we design computer systems that seem to exhibit behavior that is like human intelligence, there's different techniques that could be used on the back end to lead to that front end perception of intelligent behavior. You know, we could look through, say, a set of past legal documents and design a set of hand coded rules. If this, then that, that would lead to an output. If we think about something like predicting how much a house might cost based upon past real estate trends, somebody would come in and they'd say, okay, well it's normally like the size of a house that determines its price. So we'd use that one characteristic and do a mapping between size and price and have that our algorithm. And that could learn over time with new data. But it's not all that complex. When we get to the recent developments in artificial intelligence, we start to use data like text and images and video and speech, where it gets much harder for us to define the one input feature, which is the technical jargon that leads to the output. And those are the kind of domains and types of data that are requiring more complex algorithms on the back end that can learn over time and start to mimic human capabilities like being able to perceive things. Things, or being able to reason about things.

Well, one of the challenges with those sort of more complex algorithms also is that as we use the Outputs to make decisions in organizations. Fewer of us really know what's going into those algorithms. You know, we can see what the recommendation is, but we don't really know where it came from or why. How important is it for us to understand what's inside that black box?

This can be tricky because we're always looking for sort of silver bullet answers that would make it easy to apply a rule and just update our systems all at once. But actually it really depends. So if we think about something like an Amazon product recommendation, so a tool that's going to decide if you're somebody who's going to be interested in toilet paper or interested in the latest David Foster Wallace novel, that's something where if the prediction is incorrect or if we don't really know why it's being made, it's not the end of the world because it doesn't have drastic consequences. On the other hand, if we're going to be using these algorithms to do something like make a decision as to whether or not somebody merits a home loan, that can have huge implications. And the reason that it would be important for us to be able to interpret why the algorithm made the choice that it did is that somebody might want to intervene and modify their behavior so that they have a better chance of getting a loan in the future. Similarly, we might discern that certain subpopulations are being disenfranchised and systematically discriminated against when it comes to receiving some sort of opportunity. So that's another reason why we might want to peer in and really understand why the systems are doing what they're doing.

So if you are a bank officer who's sort of trying, who's using an algorithm to decide whether to give someone a home loan, and then you are trying to explain to someone why they may or may not have been approved, what are some of the issues that might come up in that kind of scenario?

Oh, gosh, there's all sorts of tricky things to think through. So the one which I mentioned briefly earlier has to do with the fact that different populations might be unequally represented in the data set used to train the algorithm. So, say, historically the business has done a great job serving wealthy Caucasian males. They're going to do a great job because they have lots of training data to train their algorithm. And if, on the other hand, they haven't had a lot of, let's say, Native American female customers, then the database is going to be empty in those areas and might not do as well of a job making accurate predictions akin to that when these populations aren't represented as well, the algorithm might behave kind of funky when it has to make a prediction about one of these subpopulations that isn't as well represented. So there can be surprising outcomes that people didn't anticipate that they have to have a process in place to grapple with when they apply the tools. Then the third thing has to do with the fact that these algorithms, they learn autonomously. They might make one decision on one day, and then sometime in the future, they would make a different decision. And therefore, organizations need to have a process in place where they're keeping track, almost retrospectively being able to reconstruct why a decision was made on a given day so that if there ever is a complaint from a consumer, they can go back, investigate the system and say, why?

This goes to something you've written, that if you want the future to look different from the past, you need to design systems with that in mind. You can't just let the data guide you, which is, I think, a challenge for a lot of us who sort of pride ourselves on being data driven and letting data guide us. So how can we be sort of empirical and evidence based, but also not fall into this trap of just repeating kind of what the data of the past is?

I think it would start with asking critical questions from the business perspective. In a business process, in something you're trying to predict, there's a lot of stability between how the business has performed in the past and then how you'd like things to look in the future. And for instances where, say, you're going to be maybe executing a marketing campaign and you anticipate that the customer population is going to behave in the same way over the next year as it has over the previous year, then it's totally fine to just use the metrics that data scientists use to evaluate the accuracy of an algorithm, how well it's performing. Things get a little trickier when you actually want the future outcomes to look different in kind from how they've been in the past. And in that instance, it would require some critical thinking in the beginning to say, okay, so we've historically served X population really well. But as we look through the data data, we've done really poorly on, let's say, unmarried Millennials. Because if we think about, say, a traditional business that has developed campaigns to work really well with an older population and customer base, and they're struggling now to adapt to the needs and taste and preferences of a younger generation, it actually takes some thinking to go through and say, okay, we actually kind of want to learn from past data, but need to do some exploration and experiments to see if a customer base that we haven't historically tapped into might be responsive to different types of things. And that's an innocuous example, but it shows that if you analyze algorithms for bias, et cetera, you might be tapping into what could be the largest future business opportunity.

So what would product innovation with AI look like to, I think, pick up on the example of something you said that might not be sort of efficiency based?

Yeah, this is one that I really like from the big four auditing firm context, where in the last company they work for, Fast Forward Labs, we did a project with one of the big four firms that was related to shifting around how they delivered tax advice to their clients. And the implications here actually led to an entirely different pricing and packaging strategy than what they had in the past. So historically, because it was so hard for a tax professional to stay on top of shifting regulations, new opinions, all of the things that were going on in the legal world, when they delivered some piece of advice to a client, they had to include some fine print at the bottom of the text that said, this advice is relevant today, but it may not actually be relevant tomorrow. And so they were looking to apply artificial intelligence to effectively read through all of the opinions that were coming out and identify topics and trends so that they could more quickly adapt that advice that they were providing and turn it from a way, moment in time piece of advice into something that was dynamically evolving and led from a time and materials based business model into something that was a subscription with just continuous, dynamic updating advice. So that's just one example of many in how just this collaboration, clever collaboration between man and machine, can lead to an entirely different way of delivering services.

One of the things that often comes up when people talk about that kind of human machine collaboration is the idea that in the future emotional intelligence will become so much more important because it's something machines can't do. Do you think that's true or will we get to a point where machines actually do seem like they're empathizing with us?

I mean, I think it's a beautiful thing to think about machines that are empathizing. But I do think that we're a long way off and it actually forces us to revisit that initial question around what qualifies as intelligence. Right. So today, these algorithms, I like to consider them like idiot savants, so they can be super intelligent on one very, very narrow task. They might be able to, for example, diagnose whether or not a lung has cancer better than a doctorate radiologist who has received their PhD. It's the type of knowledge that is not layman knowledge. It's not stuff that Sarah, you know how to do or that I know how to do. So it seems hyper intelligent to us. However, when push comes to shove, it is quite narrow in its scope and is something that via 50,000 training examples might be learned. On the other hand, our emotions, while they seem to be evolving less quickly than our minds, given the fact that a lot of us suffer from mental disorders and ADHD and all of the ailments of this animal body forced into the constructs and confines of modern life, our emotions are quite complex. They are themselves probably like algorithmic systems. They just have all sorts of inputs and parameters that they're able to manage and therefore they're quite hard to automate. When we are sitting in front of another human being and communicating with him or her, there's signals that we're processing based on gesture and the turn of someone's lips and emotions, et cetera, that are influencing that communication. And that would be super hard for a machine to model. It actually shows us something surprising and wonderful about our own humanity, that the purview of the motions might be something that are just so complex that it's going to take a long time for robots to mimic them.

You raise a really good point there about the challenge of having these animal bodies that now for a lot of us, just sit all day in front of a screen and we're sort of trying to be as efficient as our machines that fire off emails faster than we can respond to them and things like that. Do you see a future in which technology actually does help us kind of become healthier? Or is it just we're the ones who are adapting to the machines, not the other way around?

Well, I think it's up to us to define that future. So the great thing about thinking about the future of artificial intelligence, although there's so much concern over existential risk and are getting to a point where an artificially general intelligent machine just takes us over and realizes that we mere measly humans should be extinguished to lead to the the future utopian robot race, the truth is these are computer systems that are designed and developed by humans. And therefore it's up to us to frame the narrative and decide what we want this technology to do. And of course, as Tim Harford, I think so beautifully, writes in his books, it's going to be really hard for us to imagine how a society will adapt to the tools that we introduce. But it is, it is up to us to pay attention to how those changes evolve and then be proactive in designing our own future using these tools. The other thing I think is that there's lots of rote tasks that are the stuff of jobs today that computers can help automate to free up the time for people in jobs to be more imaginative and innovative. And while that will be a shift and require a shift in our thinking about education. Right. Preparing people for jobs so that they can be nimble and adaptable, I think it's something that could provide a beautiful future as opposed to a dystopian one.

You mentioned that AIs are kind of idiot savants and they're sort of really good at one specific thing. What does that mean for us? How will our own minds evolve when we're collaborating with machines that are designed not to generalize, but just to do one specific thing?

Yeah, I think it teaches us a lot about how we approach tasks ourselves. So say we have to carry out some task at work, some rote task. Normally, if it hasn't been done by others before, the first time we do it, we have to sort of solve it in the particular. And as work within a company evolves, it will start off with people and then it turns into process and then eventually it gets encoded as technology. And so there is a means where we go from sort of the particular up to something that's generalized. But. But once we've done it, we can find these patterns and then our minds are able to basically just learn and see analogies and adaptations to things that look kind of alike, but not totally alike when they shift into a new domain. What algorithms can't do is have that plastic transfer between something that looks kind of like the thing, but not totally like the thing. So they work at a much more particular and again, idiot savant like level. And what's interesting about that is they then, because they're trying to optimize something, might pick up on patterns that we don't necessarily see. We often rely upon heuristics, early leading indicators of some sort of success to guide us down a long term path and goal. And the algorithms might use really different leading indicators and it comes off to us as alien, but it can teach us something that can be even lessons in humility when it comes to achieving our goals. And you know, this shows up in Michael Lewis book Moneyball, where the coaches that were picking the best baseball players used to look for, you know, signs of strength and the home run hitters as. As the people that were the most coveted on the team. As opposed to thinking about it as a true optimization algorithm. Right. To select the players that may not necessarily, from our first glance and our heuristic understanding of the world, look like they're the best, but together in combination, can receive, can achieve the best results.

So I want to go back to something you mentioned a couple times, which is cancer diagnosis. This is something that for a lot of us is even more personal than can we get a home loan? And is even more serious than, you know, can this machine win this board game? So tell us a little bit about kind of where we're at right now with that and how it's possible that these machines are getting better at diagnosing cancer than someone with a PhD.

I think cancer diagnosis poses two really interesting questions. The first relates back to the need for an explanation and why the algorithm decided to indicate that, say, a lung was cancerous or not. I think if we think about it critically, it matters less why the algorithm indicated that the lung was cancerous and more that there's a very high probability that, you know, that somebody has cancer. So if we think about it, if it goes through and it says based on patterns that we've seen in the past, there's a 90% or 95% or 99, whatever it may be, just pick something really high. There's a 95% chance that you do indeed have cancer. If you've heard that statistic, what matters then is the question you ask yourself is, do you get chemotherapy? Do you treat yourself or not? And I think if from a psychological perspective, you are presented with the information that there's a 95% chance that this is the case, then you probably would treat yourself. So it's less about what can I do differently to intervene, and should I be, you know, should I be smoking less, should I be eating different foods, et cetera. That that works when you're in the preventative state, but once you've actually gone into the diagnostic state, it really is a different set of questions. So I think as a society, as opposed to our saying, oh, my God, we can't adopt these tools because we can't explain them, we need to be stepping back and saying, well, what really does matter for this instance? And if there's a high percentage and probability that somebody might be ill, then the ethical act might be to go on and treat them. And it also might be more ethical for us to use Systems that have higher accuracy rates than a human. Because when we train a system to do something like diagnose cancer, we're basically collecting the knowledge that thousands of doctors have made judgment calls on in the past and accumulating that into a system. So I guess the second lesson there is one around our own professional humility where we might have to step back and say, yeah, actually thousands of people's judgment's probably better than my own. But that causes, it raises a lot of uncomfortable questions related to the status of various professions.

Well, it's interesting when you frame it as thousands of people's judgment, that somehow feels less scary or threatening than just this robot telling you you have cancer.

Yeah, yeah. And when we have a machine that's, that's diagnosing cancer, it's the accumulation of thousands of people's judgments represented in a mathematical formula. That's it. It's not an alien intelligence.

It does also, I mean, as you pointed out, it does raise questions about what is the role then of the doctor, the surgeon. And I have seen some AI written news releases or news article, and that has sort of made me feel uncomfortable as a writer and a journalist to be like, whoa, am I even necessary? Is what I do even special? So what do we do with these kinds of thoughts and feelings as we see AI start to really move up and start doing tasks that we once associated with college educated professionals?

So I think the journalism example is great. Right. So there are these natural language generation tools which effectively are software systems that will start with a structured database and then they will write articles that represent the insights and trends in that database. And they can do things like write company earnings reports and hyper targeted weather reports, etc. They can even do sort of police reporting. But if you ask anybody working on the field, they would be hard pressed to do long form investigative journalism that really digs into issues all that way. Well, I guess our responsibility would be to try to avoid the hype, try to think critically about, all right, what can these tools actually do and what can't they do? And then when you realize what they can do, it's not as spectacular or threatening as it might appear. And therefore, if one say, is a journalist, it would be okay going forward. My company earnings reports are going to be written by a machine. So. So where else might I spend my time? What obligation then? How does this force me to up my game so that I have to think even more critically about the things that I'm writing, knowing that, you know it's going to be that critical, interpretive, emotional work that requires human input here. And so. So I see it again, as. As an opportunity. But as with any new technology and change management, we have to go through the initial discomfort and growing pains of disruption because it's hard to change our habits, even with. I don't think it's unique to AI. I think anytime a company, say, puts in a new CRM system and you have to go through that first couple of weeks where nothing works and the buttons aren't doing things and the dashboards are hard to create and it's just a disaster. But the good thing is we humans are extremely adaptable and more adaptable than our anxiety and our initial emotional response responses tend to indicate. And I think if we embrace and accept that, there's going to be a period of difficulty and transition that we have to get through as we accommodate and adapt to what the new technologies can do. I mean, I. I'm pretty optimistic about where things might go.

Well, Catherine, thank you. This has been super interesting. I've really enjoyed talking with you today.

Yeah, me too. Thank you so much for having me.

That's Catherine Hume, VP of Product and Strategy at Integrate AI. This episode was produced by Mary Dew. Adam Buchholz is our audio Product manager and we get technical and production help from Rob Eckhart. Thanks for listening to the HBR IdeaCast. I'm Sarah Green. Carmichael.

Sam.