VOICE Round-Up: Patricia Scanlon on trust and pioneering children’s speech recognition

M-A-C-H-I-N-E-Y-E-S-I-S-T-I-N-G.

It's Machine Yearning from Assist. It's a podcast where we think and dream about the future of AI, the talking internet, and how we're reshaping our culture. This chapter is an excerpt from our interview with Patricia Scanlan at the 2018 VOICE Summit. the largest gathering of the conversational technology world. Patricia is one of the many folks we interviewed, people who are thriving at the front edge of brand, UX, research, and design. Patricia is the founder and CEO of Soapbox Labs and a pioneer in children's speech recognition. Patricia and her Dublin-based team are deploying deep learning to build mammoth datasets of children's voices so they have the opportunity to enter the world of voice-activated technology. This is just a taste, so make sure you grab the full episode and subscribe so you never miss a single machine yearning. Patricia Scanlan.

Let's take a couple of the top things that you had to really solve for, that you had to do differently to specifically address how to make it work for children. So for example, there is the architecture of a child's voice. That's one thing that I'm sure had to really be addressed.

Yeah, so they're physically very different. Their vocal tracts are thinner and shorter. They're leading to higher frequencies.

Which is why you get the higher pitches.

And that confuses the hell out of a system, right? Because if you have built a system of adult voices, so if you think about it, a man's voice is kind of lower, a woman's is higher but overlapping, and then a 12-year-old is kind of overlapping there. And then it just, as you get younger, it gets more and more different, higher pitched than, so down to a 4-year-old compared to a 12-year-old even. It's very different, but equally behavioral differences.

So talk about behavioral. Explain that.

So how a child speaks, a 12-year-old is having more adult-like speech. But as again, you go younger and younger and you're going to see big difference in how they speak. They shout, they whisper, they stutter, they stammer, they start, they repeat, they punctuate, which really messes with endpoint detection and speech recognition, you know. So we didn't consider just looking at the couple of datasets that are public out there. We did look at them, actually, to be honest. Often collected in universities, but they're controlled by an adult because the objective was to get the voice data without even considering that actually the behavior uncontrolled is very different. It's natural. So if you leave a child interact with technology somewhat uncontrolled, you're going to see different behaviors.

So let's talk about the dataset that you're working with right now, sort of the size and the scope and— How big is this map?

Yeah. Now we've got thousands of hours of speech from kids in, I think, over 170 countries. English now, because one of the important things is to recognize the fact that the world has changed. If you drop a pin on a school in New York or California, you're not going to get New York accents. You're going to get like 20%, 30% New York accents, but you're going to deal with British accents and Indian accents and Irish accents. And you have to cope with that. Otherwise, again, you're your system comes off as poor performing. And I think that's kind of brands and products, people who bring products to market begin to recognize that the accuracy of performance reflects on their brand. So, one of the things we were very cognizant to do, and years ago what people did was, "Oh, I'm going to make a Northeastern US language, you know, speech acoustic model and language model. And then I'm going to make one from a Southern." You can't do that anymore. The world's, you know, you need something that works across.

So, how does the deep learning element play into your product development?

You have to have a sufficient volume of data, and this has to be highly representative data. It can't just be the same word spoken 100,000 times, like, or, you know, the same 100 words spoken over and over again, even by different kids. It's not useful. It needs to be, you know, a good representative data of language and variability in users and variability in accents and background noise and all these things. And the more of this variability you bring in, you can correctly model them in a deep learning context if you have sufficient volumes. And then you get your orders of magnitude improvements. And that's how you get state-of-the-art performance. You don't get it otherwise. And that's the same across all AI. Like, you know, I mean, you just— there just has to be some deliberate effort to balance the datasets or make sure there's enough examples in there of different accents in order to be able to do it. You know, they can correct that. It's not hard to correct it. It's just, you know, just putting the effort in and the will to do it.

This gets to one of the big things I really wanted to talk about with you, where I have some skepticism about this phase of our technological evolution. So we've been moving fast and breaking things really effectively. Mm-hmm. And now they're saying, let's move fast and break things with your children.

Yeah, agreed.

So talk to me a little bit about these sort of ethical rub points and how are we going to solve for these. So now they're saying we're going to move the on-ramp to the internet from 8 to 9 down to 2 to 3 and trust us, it's going to be okay.

Yeah, I think that's a problem. So when I started, I was very aware of the fact that this was going to be a problem. You know, children's voice data is sensitive. Any data from children is sensitive. So we need to do things right. So as a company, we decided to be careful how we did it. It cost us a lot of money, you know, because we had to go through, jump through all the hoops. We were ticking boxes everywhere. We were getting audited. We wanted to make sure we were doing everything carefully. And what worried me at the time was when I was looking around going, well, I felt like I was one of the only people doing it right, like at the time. Like, but it's like a free-for-all. I think the industry is going to shoot itself in the foot by scrambling to take data when I don't even think it knows if it needs it. Like, you know, why do you really need a child's voice data? Yes, to make the speech recognition better. But beyond that, What's the point? And what happens is, what people really misunderstand is, and I think there's a big misunderstanding in Europe at the moment, is they think that because you give permission as a parent to use the device and allow your child to use a voice-activated device, you think that's protecting the voice of the child. It's actually not. You've just given permission to the company to do whatever the hell it wants with it. What happens if the child's friend is visiting? Do you have the permission for that child's friend's data? Do you really need to be storing that data? You know, and if you do store some, but be sure of what you're doing. And I think The brands that will win out will be the brands that step up and take responsibility and be more transparent. And they're the brands that we'll be more comfortable with in the future.

So there are a couple of really important guidelines here that I think I want to make sure that we really put a point on. So one of them is, in the design of the product, is to remember to speak to a child like a child. Yes. Here we actually have— whatever language you want to use, a moral use case imperative.

Yeah.

Which is that we want to make sure that we're giving age-appropriate information back to this younger person that's using it.

Yeah, that they can understand. Because I mean, and also I think like the idea is that you have to realize a child will speak differently. You can, you know, there's many things they shouldn't be able to do, you know, explicit lyrics, purchasing online inappropriate material. But just basically the language needs to be simplified in order for a good experience. And there's a commercial benefit in doing that because if you have a device in your car or in your home or on your TV or wherever, you know, and it's voice activated, you will have a better impression of that brand if you have a better experience. It is a moral and ethical issue. And the benefit is that there's a commercial one. Like, so I just, I think people are going to, in another year, realize they've gone too far and have to take a step back. I just hope they realize that a little bit sooner than later.

And that's the other sort of big flag I wanted to hoist in this conversation for, you know, brands and agencies that are listening to machine learning is sort of that guideline of just because you can doesn't mean you should.

Exactly.

Because we are still looking at fundamental breaks in trust.

Yeah.

You break a trust with me by not remembering what kind of rental car I want. That's one kind of trust break.

Yeah, yeah.

You break a trust with me because it has something to do with my child.

Yeah, that's a different story.

Game over, lights off.

And I think you do notice anybody who's a parent that you talk to have very much stronger views on this than people who aren't. And a big reason why I— when we started the company, when I started doing this, was like, I was very cognizant that was something we were going to uphold, and we have done. And, you know, it's hurt us, it hurts as an expensive and, you know, challenges we've faced. But at the end of the day, I, I want to be able to stand over everything we've done, and it's important. And I want to know that anybody we work with has, you know, similar values on that. And, you know, I think will serve us in the long term for sure. And I think will serve any brand that just is mindful of the rabbit hole they can go down. And like you said, possibly very unnecessarily.

Dr. Scanlon, thank you very much.

Yeah, thanks for having me.

Really been great.

Really appreciate your time. Cheers.

Thanks for listening, and we hope you'll take a second and share this episode with other members of your team. And make sure you check out the other chapters from this week's episode. We have an interview with Adva Levin, who is doing award-winning voice-driven game building, and one with Google's Kathy Pearl, and more. Actually, just make it easy on yourself. Just subscribe so you never miss an episode or a chapter. That way you can be the first to stay on top of this field and help shape the conversation at your company. Get in touch on Twitter @Assist. DMs are open. We're super interested to hear who you think should appear on the podcast. Machine Yearning is made by Paul Ciuffo and Michael Alsesser for Limina House. Have a great day.