In Machines We Trust: When AI hears a problem

Support for this episode comes from the.

Knight Science Journalism Fellowship program at mit.

This is MIT Technology Review.

We're sorry. All of our representatives are still assisting other customers. Please remain on the line as we value your call.

We've all been there, and it's safe to say you might even dread the experience. Already on a crusade to solve an issue, then you have to go through a long phone tree and you might not be greeted by a human. This call may be monitored or recorded for quality assurance or training purposes. And if you've wondered who really listens to these calls and recordings, making sure agents say the right things, it might not be a person at all. These days, AI solutions are being used to analyze our voices in real time. And this applies to some healthcare settings as well as these customer service phone trees. The idea is that hidden away in our voices are signals that hold clues to how we're doing, what we're feeling, and even what's going on with our physical health.

Wait, so I need to pay even more? I hadn't expected that.

This is an example of a call being analyzed by software from a company called Cogito. It's supposed to find signals in a caller's voice that help an agent be more effective with their replies, like by telling them to be more empathetic.

Yeah, I know it's a bit unexpected.

And I'm sorry about that.

It's just not currently offered as a standard feature.

I'm Jennifer Strong. In this episode, we examine what happens when algorithms analyze our voices, looking for clues about our mental and physical health.

Let's go in.

Machines We Trust.

I'm listening.

A podcast just about the automation of everything.

You have reached your destination.

As someone who speaks for a living, it's strange to think there might be all kinds of signals and other data lurking beneath the surface of the human voice. Not just in what we say with words, but in the way we sound while speaking. And when we sing, we begin with do reimi, do, re, mi, fa, so, la, ti, do so do. Yep, keep the vowel open one more time. Ma. And those signals are increasingly being detected and analyzed with AI to provide clues about who we are and even what we look like or whether we have a medical condition. Developers of these products are going way beyond the hunt for clues about people's emotional states. They're looking for signs of all kinds of diseases, including Parkinson's and Alzheimer's. So from real time analysis of our voices by businesses providing things like customer service to healthcare applications and more, just trying to understand the overall scale of what companies and researchers think they might be able to learn from our voices can be pretty overwhelming and might even feel kind of dystopian too. But it's also possible this kind of tech might help millions of people. In the US alone, about 1 in 6 have been diagnosed with depression and there aren't enough therapists to help. We're joined now by my reporting partner on this project, Hilka Shellman. She's an Emmy award winning journalist writing a book about AI at work and she's been investigating this topic with us. Hey, Hilka.

Hey Jen.

What was it that got you interested in this topic?

You know, I was working on my book and I was reading an article on how the surveillance of students on university campuses has increased dramatically during the pandemic. And this article was mostly about tracking students locations and machines checking the temperatures. But somewhere the author talked about a school called Menlo College and how they started using a tool to check the students voices for signs of depression. And that really piqued my interest. I wanted to know more. So I reached out to a company called Ellipsis Health that built the tool. And I also had the chance to talk to a couple of students who used it. And I wanted to know how did it actually feel using the tool?

Okay, so let's start with the college. It's located in the heart of Silicon Valley and we spoke to one of its students, Lina Locosky Torres.

I'm majoring in business management and they have a concentration in entrepreneurship and innovation. So that's my concentration. I found that that was best suited towards my needs in regards to innovation. I'm always interested in doing things against the status quo. So that's my major.

When we spoke with her, she was a 19 year old junior.

I think that it was more the holistic version of I didn't come in with the mindset of, oh, I'm going to go to Google or I'm going to go and make a lot of money. Da da da da da. Like I'm more focused on social change and how I can best use my brain, my thinking skills. I don't have a lot of like technical skills.

But she found it hard to escape.

These expectations because it is really overwhelming. Fresh out of high school, going, okay, now we're in a situation where 21 year olds have built unicorns. What are you gonna do? And it's like I'm just trying to figure out how to exist, you know, as my own person.

Then the pandemic hit.

So I did my first semester, my Freshman year, in person, living in the dorms. And then spring break on my second semester, freshman year. Then we went online, and she told.

Us many of the students struggled, especially early on when they had to rush back home.

A lot of things come up. You're with your family. Family issues. You can't get away from it. You're with your mind. You know, you don't realize how much you miss your social interactions.

When the campus shut down, she had to return to her mom's place in Las Vegas. And relocating out of state meant cutting the cord with her therapist, who can't work outside of California.

I was in Nevada, so I was pretty. I was distraught. I'll be honest. I felt like I was breaking up with my therapist. I go, okay, I'm in Nevada, so I guess, bye. And it felt really abrupt, especially in the time that you need it the most.

She wasn't the only one struggling to find help. About half the students at Menlo College aren't from California. And in the midst of an unfolding global crisis, suddenly many of them couldn't access the college's mental health services. Then a startup pitched the school on an AI product that's meant to assess anxiety and depression and help people navigate those symptoms. The school agreed to try it, and in late 2020, the tool was rolled out free of charge to about 800 students. It asked people to answer daily questions in a voice message.

How's everything going at home? And then I go for 30 to 45 seconds, hey, you know, things have been tough. I'm really stressed out. I'm feeling overwhelmed. I'm feeling like I'm being smothered, et cetera. And then it would switch over to the prompt of, how are you feeling lately? Or how is school going? You know, another prompt to keep you going.

Every time someone used it, they got a score. And the tool gave recommendations from breathing exercises to the number for a crisis helpline. For La Cos Gui Torres, she didn't find the exercises too helpful. She mostly used the tool to prove to her mom just how stressed out she was by living back at home.

I was like, look, it's real. This is how I feel. You can see it. There's data right there. So not so much the coping skills, because I already have, like, issues implementing those, like, to begin with. And people say, mine, meditation. It's not. I'm freaking out right now. I can't. If I could breathe and chill, trust me, I definitely would, you know?

But she says she found it helpful in other ways.

What if we didn't have this at all. It wouldn't have brought up mental health in the way that it did at such a large scale and in such a way that, I don't know, I thought it was pretty cool. There's a lot of people that thought it was pretty cool.

Though it did raise some questions about privacy.

Where is my talking going to somebody going to hear it. But it's really just the computer based system and, or the AI system and then I guess in its infancy stages, possibly somebody who works on it, but I mean with the utmost security. I'm pretty sure they didn't care what you really had to say. I think that's a really the big thing with AI too and everybody's data. You go, okay, where's, what's it going to be used for? It's my data. And it's like, mm, you're not a super spy.

Privacy also came up in discussions between students, college representatives and the product's maker, Ellipsis Health. But she says it was mostly about whether school therapists would get access to that data. And students said that wasn't appropriate, though the privacy of their voice data and what might happen to that was less of an issue.

Privacy doesn't really exist anymore and if you feel some type of way, you're gonna go onto your social media and put it on blast, you're gonna tell your friends, da da da da. It's not something you really wanna keep.

By the time we spoke, she didn't have access to the tool anymore because the pilot program had ended. But for the school, it jump started a broader conversation and a rethinking of the way it delivers mental health services.

I think they have a greater faith in technology than perhaps older generations do.

This is Angela Shmita, a vice president at Menlo College.

If I've learned anything about students and their mental health over the last two years, it's that there's not a one size fits all solution. And so you can offer a face to face counseling and some students just won't take advantage of that. But if you can offer something where students are accessing it on their own and they can do it in real time and there's not a barrier associated with it, then some students will access that.

And she says she tried the tool herself.

There is something therapeutic about just talking. I did find that to be the case and I think for me it's a little bit awkward because you're just sitting there and you're talking and you're not talking to anyone. So it's a little bit like talking to yourself. And you do wonder who's really going to listen to this. So I know that the students had concerns about that as well.

But with the demand for mental health services greatly outstripping supply all over the world, there's a lot of interest in finding tech that might help. And it's something that reporter Hilka Shellman is looking at.

I mean, there are not enough therapists to help everyone. So vocal biomarkers, they could be revolutionary and it could help a lot of people, maybe even millions of people, billions of people, or at least that's the hope. And so we reached out to four startups to find out how their technologies work and to really understand what these tools do.

The first is called Kintsugi. It's a startup with a test that's meant to find mental distress from just a 20 second voice recording. And it's supposed to work regardless of what language someone is speaking. Grace Chang is the company's CEO, and she told us these AI tools essentially do what our parents and many therapists have done for a long time.

When we talk to our friends, our family members, it's almost obvious for those who are close to us, when they speak in a lower voice or if they speak in a slower manner, that something might be wrong with them. And we have the luxury of knowing this person's set of patterns to be able to determine that there may be something that is different than how this person normally speaks. And so what is really remarkable is that psychiatrists have known that in this area of speech, there has always been a tie to depression and anxiety.

She believes they can replicate this intuitive speech analysis over a short period, do it at scale, and that it can help therapists understand how their patients are doing in between appointments.

Our company has moved towards a position of being able to create a robust set of models, not looking at what people are saying, but how they are saying it. Our models end up being language agnostic. So we have people that are able to speak in French and Japanese and English or otherwise, but really we are just looking for those biomarkers that are most predictive for depression and anxiety. But what is really fascinating about machine learning is that we don't have just the few examples of a psychiatrist working with maybe hundreds of patients across his or her career. Instead, we have tens of thousands of examples of individuals who have designated depression or anxiety as examples for machines to learn from.

She also says cultural influences on language don't matter as much as we might think.

We don't care about any of the demographic information or the context of what's happening, because we are looking at how people are speaking, these spectral and prosodic features of like how fast or how loud or these sort of. If you can see on a visual spectrogram of how people are speaking, there are some nuances to speech that machines are able to pick up.

What we do is we analyze human voice and deliver insight, health insight, from that analysis.

And this is David Liu, the CEO of Sand Health, a company that's regarded as one of the frontrunners in this space. Its health monitoring products look for all sorts of things, from signs of cognitive or motor impairment to asthma drug use. And we asked him for some concrete examples of how it all works.

Think of us as a health insight company, and we use voice as the window to deliver that insight. The technology is a detection and monitoring technology, right? What it does is it takes in six to 30 seconds of human voice. And from there, our algorithms and models, which have been trained on tens of thousands of people both in the US and in Asia, then are analyzing, I call it the atomic level of your voice. We take and look at 5 millisecond strips of your 30 second or 12 second voice sample. And then within there, we're analyzing those vocal features that do and that we have identified as being relevant and understanding your particular condition.

And pauses are something that almost all companies are looking into.

What we look at is the time difference between when air is being pushed out of your mouth to when sound and voice is being detected. And so that time period, which is quite short, we can measure that his.

System also records the smoothness of a speaker's voice, control of vocal muscles, energy clarity, and the speech rate.

And by the way, some of these features can be heard by the human ear and discerned. Most cannot.

The idea is that tucked away in our voice, AI can find hidden information. The basic claim is that our voice is directly connected to the brain. And vocal biomarkers might allow us to reverse engineer what's going on up there.

We talk about vocal biomarkers and san being able to understand changes in health through changes in voice. And the reason why this is possible is, and I'll go into the science of it a little bit, but it is, it is really based on physiology. When symptoms of a disease such as depression or anxiety or any of these other mental health conditions begin to have an effect on the body, and they do, stemming from the brain. And there are a hundred different body parts that come together in your body, in everyone's body to allow us to speak. It's one of the most complicated activities that human beings participate in. There are literally thousands of vocal features. I had no idea of this before I came into the space either. But voice is actually an incredibly rich data source of interesting acoustic features.

And so his and other companies argue they can objectively monitor patients and pick up, for example, if someone is becoming depressed.

It's a complex mixture of physical, mental health, mental abilities that come together. And so when these symptoms of disease begin to spread and begin to become of a larger impact, they will impact the actual physical aspects and characteristics of your voice.

But it's not just healthcare companies using this technology. What might it mean to have it running in the background of our business calls or during job interviews, where voice analysis that may come with hiring software could potentially flag a job candidate as depressed? It's complicated, and among the many questions it raises, would companies have an ethical obligation to share these insights? And so we asked Liam Kaufman, the former CEO at Winterlight Labs.

We've sort of steered clear of that type of use cases because there's a lot of thorny ethical issues, which is like, let's say you have an Alexa running in the background. It might be measuring your speech, but it also might be measuring your spouse's speech or the delivery person's speech or the tv. So there's a lot of different things that could happen in the background. And then there's also a lot of context that's missing, like, is this person responding to something that they're angry with, or they're responding to something that they're happy with? And so in the short term, it's much easier to do more active assessments. And so you can kind of control the subject matter and what they're talking about. So technically, it's easier to do. And ethically, it avoids some of those challenges of diagnosing people that don't want to be diagnosed or don't even know that they're being listened to, which are generally pretty creepy. And we want to kind of avoid at this point.

But one question we haven't asked yet is whether these biomarkers really contain this hidden information about our health. We trust a blood pressure reading as a biomarker of physical health, but how do we know whether we should trust focal biomarkers as an accurate reflection of disease? We asked one of the world's leading AI researchers, Margaret Mitchell. She founded Google's Ethical AI group and is a pioneer in the field of machine learning. With close to 100 papers. She doesn't have commercial ties to vocal biomarkers now, but she's worked on them in the past at Oregon Health and Science University.

In particular, we were looking at their speech streams to see if we could do detection of mild cognitive impairment, which is a precursor to Alzheimer's, Parkinson's. That's on the older end. Then on the younger end, we were looking at autism and apraxia. With all of the above, there was a question of what sort of signals can we pull out with autism. Prosody was a big one. So prosody is sort of like the musical side of language. So I can say something like this, or I could say something like this. The latter has much more of an intonation contour going up and down. Right. And so, for example, people who are depressed tend to have more monotone, a little bit flatter intonation contours. Right. Than people who are not depressed. So we were looking at that kind of thing for autism and then for Parkinson's, we were looking at a few different kind of things, including pause behavior. So one aspect of the speech stream is the pauses, the silence between different phrases. You can start to pull out some things that are roughly predictive of some sort of neurological statuses. We didn't have a ton of luck with Parkinson's at the time. We did have some luck with mild cognitive impairment and with autism.

And she says her team got the best results when they combined both sound and words.

The project that I worked on the most that was, that showed some nice results was for mild cognitive impairment. And we found that we were able to make some reasonable predictions of mild cognitive impairment when we used speech signal as well as language signal. So it wasn't just the audio. It was also what they were saying.

The goal is to make healthcare more accessible and less expensive.

So if you're able to make some predictions based on a speech stream, you can do a few things. One is pre screening. And in pre screening, you would call a phone number, you would take a battery of tests, and then based on the pre screening results, it would say whether or not you should go speak to a physician. So that just initial sort of pre screening is a really useful thing to be able to do and keep down costs. And then once someone already has a diagnosis, it's really useful for them to be able to just at home, do retellings, do these different kinds of things without having to go in and be especially diagnosed, and then have those readings or signals sent back to the clinician.

So monitoring a patient's voice with a Tool like this could help detect if a patient's depression is getting better or worse. But she cautions, these predictions aren't precise.

This has to be tempered by the fact that accuracy and other sort of evaluation metrics are virtually never 100%. And there's a lot of additional factors that come into play that affect accuracy, such as what kind of phone is being used, what sort of audio recording device, all this. So in a world where everything worked perfectly, that is the goal. In reality, we'll probably never get to a place where we work perfectly. And so we'll probably be in a space where a system could automatically give you a preliminary reading. You know, based on my faulty ability to make predictions, here is my preliminary reading of you. You know, it's kind of like when you take a pregnancy test. You, like, you first take the ones that are at home ones, and then eventually you're like, okay, I guess I'll go see a doctor. It's sort of sort of that sort of thing where you try and be very clear that there's false positives, false negatives. It's kind of similar with COVID tests, I suppose. You know, you try and make it as clear as possible that it won't always work, but at least it's better than nothing and can be a signal that then you go see a professional.

It's why one signal is not enough for a diagnosis, because what if someone had a bad day and speaks in a monotone voice? And this isn't unique. It's also the case with other biomarkers, too. An elevated heart rate might mean there's a problem, or maybe the patient was just running late. And it's also critical that companies building these tools make sure the AI doesn't predict on the wrong things.

Disaggregation. This is the word to know. Disaggregation. In disaggregation, you take all of these variables and you test with respect to each one of them. So in general experimental design, there are independent variables and dependent variables. I think to your point, in machine learning, people often don't pay attention to, you know, experimental design that has been really well defined for years.

She also points out another problem. This technology could easily be misused.

So if it's possible for a potential employer interviewing you to automatically get some readout that says, oh, this person might struggle with depression, then that's a mechanism for them to discriminate against you and choose not to hire you. I've done work on predicting depression, and I intentionally did not include examples of the kinds of words that were used, because then you have armchair, you know, you have armchair clinicians like, oh, well, they're using the word ibuprofen a lot. And I heard that ibuprofen was a signal of depression. So now I'm going to be biased against this person in that way. I mean, even talking about this, it's a little bit worrying. But, you know, it is important for people to understand what's happening. I can give this really funny example that working on this speech stuff, one of my friends, colleagues went to a talk about depression and signatures of depression. The colleague came back and as they came back, I was very frustrated with something and I went. And then she went, are you depressed? Depressed people sigh a lot. I heard, you know what I mean? And so, like even giving this example now, her impression of the world is fundamentally altered right now. She sees depression in people that she wouldn't have otherwise seen. So it's so critical to make sure, sure that these discussions are really couched in all the caveats and that they are only released to people who can really work through these nuances and understand them. Otherwise, all of these kinds of findings are going to be a mechanism for discrimination. If you have depression, you might use monotone. If you're using monotone, that doesn't mean you're depressed. And so understanding the sort of causality flow there, if this, then that is really critical and something that a lot of people mess up and then particularly if it ends up influencing people and people's impressions in situations like hiring, then it's a very serious concern.

This makes it hard for researchers to share their findings. Plus, there are lots of concerns about privacy with this data. And so we turn to Bjorn Schuller, a professor of artificial intelligence at Imperial College London and one of the world's leading experts in vocal biomarkers. We asked him whether using this tech in a healthcare setting is even a good idea, and he said, yes. Vocal biomarkers can be useful where doctors have already been listening with a stethoscope, mainly for illnesses involving the lungs. Since these diseases cause an audible change in the way we breathe, speak, or cough. Andy says this tech can go a step further than traditional doctors can. Not only hearing a different cough, but, for example, if someone develops throat and neck cancer, the disease changes the voice in a particular way that can be picked up.

It came from, let's say, where people have been listening to and moved more and more into, okay, let's think about what would actually make a change to your voice. It's your cognition for speech production. It's your physiology. It's your motor system. If that is somehow affected, we should be able to hear it.

But then he said something that gave us pause. That the tech can already pretty accurately predict a person's height from their voice and even their heart rate. And one day, he thinks it'll be used to spot things about infants even before their own parents can. Like that he could help diagnose autism and other things very early on simply from a baby's cries.

The idea is that you can have a baby phone endowed with some smart ability that recognizes early warning signs of neurodevelopmental disorders like autism, and then could say, please go to a specialist and have this confirmed, because the parents wouldn't notice that at month five or seven, there should have been other signs already available in terms of communication. But the baby phone would know.

He believes vocal biomarkers could also make a huge difference in countries where medical care is hard to access. So in the future, people could call a number, leave a voice message, and the computer could tell them if they have dementia, throat cancer, or other illnesses.

You would have the luxury of it being very cheap, very accessible. If you just have to call a call center and it can give you feedback. This would mean that we can, in less connected countries, easily provide such health services just via phone.

These ideas are associated with deep neural networks, a form of machine learning where the computer trains itself to find patterns. In the past, scientists needed to make assumptions about what they could find in a voice and then figure out where the signal might be. Is it in the pitch, or is it in the loudness? These days, the computer gets training data that's already labeled. So whether the person does or doesn't have a disease is marked. And then a system looks for patterns in that audio. And this gets tricky because the machine might find patterns we can't hear, or it might find patterns that have nothing to do with what it's trying to find. And he gives an example of this from his own work. His team was trying to find the difference in how people explain smelling something good and smelling something awful.

So it's a pleasant or unpleasant smell, but the machine that was inducing the smell made bubble sounds, which were slightly different. And we then took the pauses between the speech and recognized that the AI actually picks up the bubbling sounds of different things poured into the liquid to produce a smell. So we have to be very careful that we assure what we're actually recognizing is coming from the voice and not from any background context.

In Other words, the pattern the software found wasn't the difference in how people reacted. It's that the machine made different sounds when releasing different fragrances. Similar problems happen with trying to diagnose Covid using software to analyze coughs. And Schuller says he's worried that without extensive testing, some tools won't work as advertised.

And frankly, we're sometimes right. Yeah, I wouldn't want to say worried, but unclear about the real performance of some commercial products. And that's generally true. Of course, in machine learning, there's a lot of people not fully revealing their test methods and so on.

Something else, as we consider whether to use these markers to test for things like drug or alcohol impairment, voice generators already exist that can change our voices in real time, making them sound happier or monotone or like an entirely different person. And this could be used to sidestep these checks, like to fake a drug test. So while players in the industry are quite confident that vocal biomarkers exist and can be used for a great number of applications, how well it really works in practice remains unclear. This episode was reported by Hilka Shelman, produced by me with Emma Silicons and Anthony Green, were edited by Matt Honan and mixed by Garrett Lang, with original music by Garrett Lang and Jacob Gorski. Special thanks to the Night Science folks at MIT for their support with this reporting, and thank you for listening. I'm Jennifer, This is mit technology review.