VOICE Round-Up 2: Full Episode

Hey Alexa, play Machine Yearning.

Here's a sample of Machine Learning by Fingerspit.

No, Alexa, play Machine Yearning podcast.

You should try to mumble less.

Probably, I don't know.

M-A-C-H-I-N-E-Y. M-A-C-H-I-N-E-Y.

It's Machine Yearning from Assist. It's a podcast where we think and dream about the future of AI, the talking internet, and how we're reshaping our culture. Today, an episode so packed with so many amazing folks, 4 of them in fact, that we can't wait to dive right in. But first, some quick scene setting. A few weeks back, your humble pod accepted the invitation from the team at MoDev, and attended the Voice Summit. It brought together over 2,000 people working at the front edge of voice, AI, natural language processing, and conversational commerce. We hope there's a Voice Summit 2019 because we were able to sit down with too many brilliant people. First up, Patricia Scanlan. As a founder and CEO of Soapbox Labs, Scanlan has been leading a team of researchers who have built what is quite likely the world's most comprehensive natural language dataset of young children. Seriously, she's gathered thousands of hours of samples from all over the world and is currently commercializing the dataset. Her passion for privacy, science, and building her business are riveting. A good place to start this episode. Machine Yearning producer Michael Elsesser sits with Patricia Scanlan from The Voice Summit.

Let's take a couple of the top things that you had to really solve for, that you had to do differently to specifically address how to make it work for children. So for example, there is the architecture of a child's voice. That's one thing that I'm sure had to really be addressed.

Yeah, so they're physically very different. Their vocal tracts are thinner and shorter. They're leading to higher frequencies.

Which is why you get the higher pitches.

And that confuses the hell out of a system, right? Because if you have built a system of adult voices, so if you think about it, a man's voice is kind of lower, a woman's is higher but overlapping. And then a 12-year-old is kind of overlapping there. And then it just, as you get younger, it gets more and more different, higher pitched than— so down to a 4-year-old compared to a 12-year-old, even it's very different, but equally behavioral differences.

So talk about behavioral.

Yes.

Explain that.

So how a child speaks, a 12-year-old is having more adult-like speech. But as again, you go, younger and younger, and you're going to see big difference in how they speak. They shout, they whisper, they stutter, they stammer, they start, they repeat, they punctuate, which really messes with endpoint detection and speech recognition. So we didn't consider just looking at the couple of datasets that are public out there. We did look at them, actually, to be honest, often collected in universities, but they're controlled by an adult because the objective was to get the voice data without even considering that actually The behavior uncontrolled is very different. It's natural. So if you leave a child interact with technology somewhat uncontrolled, you're going to see different behaviors.

So let's talk about the data set that you're working with right now, sort of the size and the scope.

Yeah.

I mean, we—

How big is this map?

Yeah. We've got like— now we've got thousands of hours of speech from kids in, I think, over 170 countries. English now. It's because, you know, one of the important things is to recognize the fact that the world has changed. If you drop a pin on a school in New York or California, you're not going to get New York accents. You're going to get like 20, 30% New York accents, but you're going to deal with British accents and Indian accents and Irish accents. And you have to cope with that. Otherwise, again, your system comes off as poor performing. And I think that's kind of brands and products, people who bring products to market begin to recognize that the accuracy of performance reflects on their brand. So, one of the things we were very cognizant to do, and years ago what people did was, "Oh, I'm going to make a Northeastern US language, you know, speech acoustic model and language model. And then I'm going to make one from a Southern." You can't do that anymore. The world's, you know, you need something that works across.

So, how does the deep learning element play into your sort of your product development?

You have to have a sufficient volume of data, and this has to be highly representative data. It can't just be the same word spoken 100,000 times, like, or, you know, the same 100 words spoken over and over again, even by different kids. It's not useful. It needs to be, you know, a good representative data of language and variability in users and variability in accents and background noise and all these things. And the more of this variability you bring in, you can correctly model them in a deep learning context if you have sufficient volumes. And then you get your orders of magnitude improvements. And that's how you get state-of-the-art performance. You don't get it otherwise. And that's the same across all AI. Like, you know, I mean, you just— there just has to be some deliberate effort to balance the datasets or make sure there's enough examples in there of different accents in order to be able to do it. It's, you know, they can correct that. It's not hard to correct it. It's just, you know, Just putting the effort in and the will to do it.

This gets to one of the big things I really wanted to talk about with you, where I have some skepticism about this phase of our technological evolution. So we've been moving fast and breaking things really effectively.

Mm-hmm.

And now they're saying, let's move fast and break things with your children.

Yeah, agreed.

So talk to me a little bit about these sort of ethical rub points and how are we going to solve for these. So now they're saying, we're going to move the on-ramp to the internet. From 8 to 9 down to 2 to 3 and trust us, it's going to be okay.

Yeah, I think that's a problem. So when I started, I was very aware of the fact that this was going to be a problem. You know, children's voice data is sensitive. Any data from children is sensitive. So we need to do things right. So as a company, we decided to be careful how we did it. It cost us a lot of money, you know, because we had to go through, jump through all the hoops. We were ticking boxes everywhere. We were getting audited. We wanted to make sure we were doing everything carefully. And what worried me at the time was when I was looking around going, well, I felt like I was one of the only people doing it right, like at the time. Like, but it's like a free-for-all. I think the industry is going to shoot itself in the foot by scrambling to take data when I don't even think it knows if it needs it. Like, you know, why do you really need a child's voice data? Yes, to make the speech recognition better. But beyond that, what's the point? And what happens is what people really misunderstand is, and I think there's a big misunderstanding in Europe at the moment, is they think that because you give permission as a parent, to use the device and allow your child to use a voice-activated device, you think that's protecting the voice of the child. It's actually not. You've just given permission to the company to do whatever the hell it wants with it. What happens if the child's friend is visiting? Do you have the permission for that child's friend's data? Do you really need to be storing that data? You know, and if you do store some, but be sure of what you're doing. And I think the brands that will win out will be the brands that step up and take responsibility and be more transparent. And they're the brands that we'd be more comfortable with in the future.

So there are a couple of really important sort of guidelines here that I think I want to make sure that we really put a point on. So one of them is in the design of the product is to remember to speak to a child like a child.

Yes.

Here we actually have, whatever language you want to use, a moral use case imperative.

Yeah.

Which is that we want to make sure that we're giving age-appropriate information back to this younger person that's using it.

Yeah, that they can understand. Because I mean, and also I think like the idea is that you have to realize the child will speak differently. You can, you know, there's many things they shouldn't be able to do, you know, explicit lyrics, purchasing online inappropriate material, but just basically the language needs to be simplified in order for a good experience. And there's a commercial benefit in doing that because if you have a device in your car or in your home or on your TV or wherever, you know, and it's voice activated, you will have a better impression of that brand if you have a better experience. It is a moral and ethical issue. And the benefit is that there's a commercial one. I think people are going to, in another year, realize they've gone too far and have to take a step back. I just hope they realize that a little bit sooner than later.

And that's the other sort of big flag I wanted to hoist in this conversation for brands and agencies that are listening to machine learning. Is sort of that guideline of just because you can doesn't mean you should.

Exactly.

Because we are still looking at fundamental breaks in trust.

Yeah.

You break a trust with me by not remembering what kind of rental car I want. That's one kind of trust break. You break a trust with me because it has something to do with my child.

Ah, that's a different story.

Game over, lights off.

And I think you do notice anybody who's a parent that you talk to have very much stronger views on this than people and a big reason why when we started the company, when I started doing this, I was very cognizant that was something we were going to uphold and we have done. And, you know, it's hurt us, it hurts us as in expensive and challenges we've faced. But at the end of the day, I want to be able to stand over everything we've done and it's important. And I want to know that anybody we work with has similar values on that. And, you know, I think will serve us in the long term for sure. And I think it will serve any brand that just is mindful of the rabbit hole they can go down, and like you said, possibly very unnecessarily.

Dr. Scanlon, thank you very much.

Yeah, thanks for having me.

It's really been great.

Cheers. Really appreciate your time.

Patricia Scanlon from Soapbox Labs with smart, cautionary words on the intersection of privacy natural language processing, and our children from The Voice Summit. Up next, Avva Levin, another business leader who is building for kids. Her shop Pretzel Labs creates voice-first games like Kids Court, where an Alexa-based judge settles kids' fights. Kids Court won the grand prize in the Alexa Kids Skill Challenge. Adva is pushing out into new territory where we see how empathetic, useful products can be educational, reduce screen time, and according to the users we've polled, be damn fun too. Our conversation starts with the role persona plays in helping the kids connect with the product. Adva Levin.

Let's really dig into persona, 'cause I think that's so critical as a doorway into all of this. You know, what do you want to make sure people understand about personas? It's a really— it's like you can scratch the surface of it and get it right away, but it goes very, very deep.

So like the way we start thinking about like defining a new persona when we do a new project is there is the elements that will be communicated to the user, and then there are elements that are just used internally to help create consistent character. So, you know, you could think of like Is it a male or female voice? What sort of age, background, and style they would be reflecting? And a good way to think of it is even to think of someone you know, someone in your life or a celebrity, or just to sort of ground yourself as a writer on a specific character so you can write a consistent personality. And then basically what gets communicated to the user is the dialogue. So they may not know that, you know, I wrote the character of a 25-year-old girl from New York, but they are hearing what she has to say. Like, we're looking into, you know, things like how would our persona welcome a new user. I just did a workshop about this here 2 days ago where we did, like, a skill to book a hotel room. And it was really fascinating because I wanted all the participants to come up with a sort of creative sort of hotel and to communicate like the persona, the voice of that hotel, just by writing that side of the skill, like not the user side. I mean, if you think about it, they only wrote like 5 lines of text and we had them act it out. And how different can you get with like 5 lines of text of writing a hotel? It was amazing. It was such a huge variety and you could really understand what they were going to do when they just acted it out.

This is the thing that I don't think people understand, the degree to which this new space is bringing both character in that sort of very formal dramaturgical sense, you know, with what you experience and then the backstory that an actor creates, with brand and brand voice and brand narrative. And this is like a level of merging the two at a higher level.

Yeah, I agree. I mean, we've seen that sort of like, you know, even when you write the copy for a website for a brand, there is a lot of discussion of voice and tone and like the right sort of way to communicate the brand's voice. But now it's literally the brand's voice and it has to be way more exact and accurate. And basically, like the way I try to think of it is if this service or app or game would have been something in real life, Who would you hire to be like the front person of your service?

And then you do have to hire actors. You do have to coach and direct them. Yeah.

So I think for some use cases you would hire voice actors. For some it would be right to do like a text-to-speech service, which we also somewhat direct, you know, by using SSML, which is the markup language for text-to-speech, so we have some control. And it is a bit like directing an actor. Pause here, put more emphasis on this word. A lot of my work is basically running through one line of dialogue 20 times through Amazon Polly until it sounds right.

That's what a director does.

Yeah, I suppose I'm a robot director now.

That's a good t-shirt waiting to happen.

Yeah.

What does that feel like after being developing as a writer?

Well, I wasn't really working professionally as a writer. I was always working for startups. But I think the big change is that this field is so new that I feel like I am really inventing my path. And there aren't really that many use cases or textbook cases that you can learn a lot from. And basically everyone is sort of inventing everything as we go forward, which gets me super excited.

Yeah.

Let's talk about the iteration of Kids Court. Like when did you start thinking that this was the, let's just talk us through like sort of the development of the idea, sort of How did you get into it and move through it?

So I brought Alexa to my home, and I saw how the kids are, like, treating her as a sort of person, I want to say. Not a real person, but it felt like we have a new person in the house. And, like, the first question that came to mind was, "Okay, we have someone new in the house. What can we use her for? Like, what can she help us do?" And I really mapped— tried to map out all the sort of friction points between kids and parents, siblings. And I'm like, okay, kids fighting, that is a thing in every family. And I just thought about it a lot and I said, okay, I mean, it doesn't really matter when they fight whose fault it is. What matters to them is they want to be heard and they want to get more attention at that point. I mean, usually it doesn't matter if your sibling stole your sweatshirt, it just matters that they got into your space and you feel left out. So, like, okay, maybe Alexa can help us with this to sort of make them feel heard and turn it into a playful experience and to make them have fun and just forget about why they had a fight in the first place.

The ultimate redirection.

Yes.

I'd just like to hear your thinking about, you know, the implications of moving the on-ramp to the internet from sort of 8 or 9, from the sort of the— when you have sort of language and composition and typing and text and reading, down to 2, 3, 4? Because now with voice, the on-ramp has moved down there.

I'm not sure it has moved down there. From what I can see, Alexa is not very good at recognizing voices for kids that are that young. So I mean, my 2-year-old can get Alexa to play music. But if it gets into like more complicated interactions, it becomes frustrating for her. And additionally, like I think at that age, like 2 to 4 or even 5, they are better at maybe consuming content by voice, but not so much interacting with it in a complicated way. So like basically when you do like this voice interaction, you're designing a script. And that script has to be followed by the user in a way. So I think the good design practices get the user to sort of answer the script without telling them what they have to say. But, like, I wouldn't say, "Hey, do you want, you know, blue, red, or white?" I would ask, like, "What color do you want to pick?" And I would sort of navigate it in a way that they know what to say, but I'm not telling them, "Say this and that." And with like the age group of 4-year-olds, it was so challenging because they know the answer, but their mind works in such a creative and like sprawled way, I guess, that so they would say blue. And then instead of like waiting for Alexa to respond, they would say, and what's your favorite color, Alexa? Which is something that it's, it's really hard to design for because that is one example, but it's all over the place. So my experience has been like for that age group to do very simple interactions for now. But things are changing all the time. That age group wants to interact. It's not working well right now, but it will be. And they want it and they are super good users.

That's great.

Thank you.

Thank you.

Machine learning.

Av Eleven from Pretzel Labs.

Machine learning.

By moving away from the keyboard and the written word, the talking internet is going to expand the on-ramp down to kids as young as 4 or 5 years old. As you heard with both Levin and Patricia Scanlan, that opens up a whole new range of challenges that brands and technologists need to solve. This is why we love making Machine Yearning. The world is blowing wide open with all of this innovation, and having the opportunity to introduce you to people like Avi Levin and Patricia Scanlan is a rush. Up next, Bree Glaser from the Mars Agency. Bree is a formally trained field researcher who did work at Harvard Medical School and Parsons before moving into the world of consumer insights and strategic global marketing. She's helped to build voice assistants that help people navigate brick-and-mortar stores, and it's clear her thousands of hours of fieldwork have paid off. After Bree, we have Kathy Pearl from Google. So make sure you have time left in the car or be ready to take us with you. These two conversations show us a lot about the role of deep consumer insight in the design, build, and iteration of voice products. Bree Glaser.

Feels like voice could clear away a lot of the gunk that's built up between brands and consumers. Someone can say, "Show me a 60-inch TV with the widest sound spectrum and the deepest black and white range." You know, so they're a hardcore old-time—

Yeah, that's an expert shopper right there.

But instead of having to go through pages and pages of specs, they can walk up to a wall of flat screens and give that command and then the 3 models that meet the spec are highlighted. You know, a snappy tagline and a shiny Instagram ad, it's not gonna overcome that.

Yeah, I think that's an excellent point. In the case that you illustrated, that is somebody who knows what they want already and then voice can meet a need, the need of navigation, make it a lot easier to find something quick, especially in the aisle. I think there's also gonna still be a role for education and getting the shopper to that point of knowing what they want, which could happen at home or could happen at the shelf.

But it could also be the shopper saying, "Show me the TV where the Maltese Falcon will look the best." Yes, exactly.

Exactly. So you have to be able to give people the information they're looking for without depending on them to really know what they're looking for. For necessarily.

Give me something that's like this other thing that I like. So this is like, this is my favorite thing, give me the item, give me the product, show me a brand that's going to best perform for what I love the most.

Yeah, absolutely. And that's, that's the promise of voice for brands, I think, and, and the challenge, because you really have to have products that are delivering on the need that the consumer is asking for, or you have to do work of explaining are somehow getting them to understand what that need is. Or, you know, maybe you'll realize that your product isn't delivering on a need and that's why it's not organically getting discovered on the platform. So it's gonna be a big challenge, I think, for brands going forward to continue to, or I should say, just to win in a voice-first world where, you know, Alexa is gonna be recommending one or two things to me every time I ask for a product.

So how do you talk to a brand about that's how the decision-making process starts to evolve in the voice world?

I think in a way I've seen brands actually being pretty good about— they recognize that voice is a new territory and they don't know what, what they're doing or how to win on that platform necessarily. So they're willing to take the advice of working in a cross-functional interdisciplinary team and hearing like basically what we have to say about how people are actually using voice. They're very, they're just very open at this point, but I still think that you have to coach them into accepting that it's an iterative play and what you do now is not going to deliver that big ROI that you're maybe accountable for, but it is in the long term. So you have to get started now so that you can get the data so that you can make something better and so that you can have that really refined skill and voice presence and potentially refined product portfolio way down the line when voice commerce is huge. In a sense, we're lucky it isn't yet so that they can do the work to get there.

Getting people to understand today that you have to bake error into the intention—

yes—

is such a shift to the contemporary especially American business mindset?

Yeah.

Yeah, I think it absolutely is. But brands are a little more accepting of it because penetration is kind of low with smart speakers right now. So they're, "Okay, if we're going in and we're gonna reach 30 million people, then—" You know, they're just more willing to, before it's 300 million people, to make mistakes. So I think that we have a little bit of runway, which is helpful. As long as we can get them to do it.

One of the things that you're really focused on is building these mental models for the shopper. Can you describe what that looks like? Because that is such an interesting concept. Yeah.

In a way, I don't think it's that different than some of the things that we've been doing at Mars or Mars has been doing before I joined, which is like identifying the decision hierarchy for the shopper, right? If you're at the shelf, you're What are the different attributes of products that you're prioritizing and how do you make a decision? So that's, it's kind of an adaptation of that, an evolution of that, that goes a little bit further.

I'd love to know how that translates into a voice application because that definitely doesn't sound like TreeFlows.

It's interesting you say that because to me it is the basis of the TreeFlows, but I think that I do that whole process maybe a little bit differently than others. So I mean, for me it's about the mental model gives you just a sense for where you need to go first. And then you go into a tree flow that reflects that mental model. And then you go into a script that— so there's all these different stages that the mental model informs, I would say. And I think you have to do testing and you have to make sure, like, are we actually meeting that need? Like a big part of the development process for Smart Isle, this product that we made in retail, was me working really closely with the developer. And I think I've heard other agencies say that that's really important as well, in that you have to make sure that just the way it's coming across and exactly the way it's been written is delivering on that consumer intent that you identified in the mental model over here. So it's like your anchor. What are the real tangible benefits that you've seen? Yeah, I think it informs a few different things. Starting from that mental model of like, what are we going to ask? And then also dialogue, how are we going to ask it? So I could give an example. When we were intercepting shoppers, figuring out how they shopped in the spirits category, we learned that people would generally say, I'm looking for something that's similar to my favorite, which is like, you know, you You wouldn't just guess that. So we learned, okay, we're gonna make a path that's kind of about matching based on what you currently drink and what your favorite is. And then also from having those conversations, I learned that that's how people would say it too. Something similar to my favorite or something like my favorite. And so you can ask it in that way that feels really natural. So those are two dimensions, but there's also gives you the opportunity to identify like the mindset of the audience and and build some personas and figure out, you know, where's the best opportunity. So, you know, in the case of a whiskey shelf, you see everybody from novice to expert shopping, and you can't build something that necessarily helps all of them, but you have to figure out what is the best target. You figure out sort of what your key, like, design statement is out of that.

Thanks for your time, Bree. Yeah, thank you. It's been great to have you here.

Thank you.

Machine learning.

Bree Glaser. And yes, we go from designing for kids to whiskey buyers in the same pod.

Machine learning.

That's the way we do it here. And now our first Googler, Cathy Pearl. Cathy has the perspective that only comes after you've worked your way through an incredibly diverse swath of projects, user needs, and clients. She's done voice assistants for auto, healthcare, and even fashion advice. We know we barely scratched the surface of her insight into the challenges of voice design, so you can be sure we'll invite her back someday. Cathy Pearl, our final conversation in today's Machine Yearning from The Voice Summit.

Let's talk about that very interesting title that you have. Like, what's the outreach part of conversational design?

Right. So the main point of outreach is conversation design is one of those things that if you talk to somebody about it, they'll say, "Oh, yeah, yeah, definitely we need— that's important." But when it comes to practice, it doesn't always get invoked. So conversation design is the concept that whenever you're creating one of these conversational experiences, whether it's a voice assistant or whether you're typing, you have You have to spend time iterating and thinking about the user experience, the interaction with the user. Because a lot of people just jump in with a couple developers and say, "We're going to build this cool thing." And then they find that actually a lot of people won't use it, it's very difficult to use, and it isn't successful. So my goal is to get everybody on board with this idea that you need to involve somebody who's got experience in designing these systems and spend the time working on that to really have a great system.

Is that about building cross-functional teams that go beyond just technologists? For sure.

A lot of conversation designers come from writing backgrounds, psychology backgrounds, linguistic backgrounds, and the real key is the understanding of human conversation. A lot of people say to me, "Why do I need a conversation designer? I know how to talk." And we've all been talking since we were very young, but there's an art to fitting in the rules of human conversation with the technical constraints that we have. And building within that to be able to leverage these human ways we speak to each other, but in a way that we can actually work with.

It's so funny how these conversations repeat and don't change. So in the early days of digital graphic design, it was this— and the joke that everybody told was, "Well, you can cut your own hair too, but we don't advise it." Exactly.

I think the same thing with website design. When you could first start building websites in the mid-'90s, everybody was throwing up their GeoCities site with their blinking, you know, under construction graphic and all that stuff. And now I think we've realized that, well, you should really probably hire somebody who has expertise in this area if you're a company and you want a really awesome website. So similarly, if you're building an action on Google or any conversational experience, you've got to get somebody who knows how to apply these principles.

So let's talk about the nut of that, like actions on Google. Define that space, define that work group? What are you trying to accomplish with that?

With Actions on Google, we're trying to extend the capabilities of the Google Assistant to incorporate all kinds of things. So whether you're a restaurant or you want to build a game or you want to have a branded experience for your company, we want to allow lots and lots of people out there and companies out there to add their, their stuff to the Google Assistant so that it's not just a locked ecosystem. We want people to have access to all these different types of services, entertainment, news, everything through the Google Assistant.

Are you, are you seeing patterns now, sort of the major requests that are coming in? Like, what are the big demands that are coming in to you at this phase for activities that people want to be developing for?

It's a good question. Honestly, there's a lot of experimentation right now, so we see things in all different domains. We see a lot of games and entertainment. We see Stuff for kids is certainly very, very popular. A lot of parents feel that time with a voice assistant is healthier than screen time. So they'd rather their kid maybe play a game that way. Because, and this is something I believe in strongly, voice allows you to get away from the screen and be more social. So for example, at our house, at dinner, we don't have devices on the table. But sometimes we have questions. So we might turn and say, hey Google, what's the longest bridge in the world, or whatever. And the thing I like about that is that it joins the conversation. We all hear the question. We all hear the response. Instead of if like, "I'm gonna look that up," and you look at your phone and you're gone, right? You've disappeared from the world. And there was a statistic that came out from the NPR and Edison study saying that over 50% of the time people are using voice assistants, it's with someone else. So I really like that community aspect that these things bring.

Yeah, yeah. I would love to hear how this experience ports through your eyes, so rebooking a ticket on Amtrak, but was in a noisy train station at night using a voice prompt system that— and we just got caught in an awful loop. So you take that kind of a problem back to your team, you say, how do we unpack this and how do we start to evaluate solutions for that? How does that conversation unfold?

We think a lot about the user's context. Where are they? Are they driving? Are they in a noisy environment? Are they at home in their kitchen? Are there other people around? And My motto is that you want to give the user alternatives. So if they are in a noisy train station and they're trying to use voice only, that is going to be challenging. And so sometimes we want to have what we call back-off strategies, where perhaps if they're really struggling and you know that they're having problems, maybe you even back off to DTMF to press, you know, press 1 or 2, just because you don't want them to be suffering. So you want to give alternatives. And we want to meet the user where they're at.

Very helpful. Talk to us a little bit about disambiguation.

That is one of my favorite topics. So disambiguation refers to the fact that sometimes we need to ask follow-up questions because we might not have understood you. So, for example, if you say you want to hear the song "Hello," do you mean Lionel Richie or Adele? And so in certain situations, we definitely want to follow up. Now, disambiguation It's not a hard and fast rule. For example, let's talk about weather. If I say, "What's the weather in Belmont?" I don't want it to say, "Do you mean Belmont, California or Belmont, Canada?" I live in California. It knows that. There's no reason to ask a follow-up question. But if I asked for the weather in Springfield, which one? You know, I don't live near one. So we want to make sure that we ask a follow-up question just in the right time and A lot of times I think some people err on the side of, oh, we just want to give someone information as quickly as possible. And if we get it wrong, that's OK. They can start over. But sometimes asking a follow-up question is much more satisfying for a user. Speed is not always the most delightful experience. Sometimes it's like one or two follow-up questions gets me exactly what I want. And people can be happier with that.

To be successful with that, it means having access to an enormous amount of data.

It does help. A lot of this is aggregated data. Certainly we don't have to have individualized data to make these decisions. But we can use things like an aggregation of when somebody requested something that was perhaps ambiguous, how many times did they ask again in a different way? Well, that's probably a signal to us that we didn't get it right. So, but we can look at that as a large group of data. It's like, you know, 90% of the time when somebody asks this question, they said it again in another way, hey, that's probably a good case for disambiguation.

Paul heard you say yesterday, "Often we think of solving big problems, but we need to remember that little things can bring dignity to people's lives." Can you expand on that?

Yeah, one of the things that I really like about these voice assistants is hearing stories of people who perhaps have some sort of maybe a physical limitation or something that it's giving them back their independence. One of the stories I told yesterday was about my grandmother, who's 104. I brought a smart speaker to her to try out, and I said, "Grandma, you can ask anything you want," because I was just curious. I thought she would ask maybe, you know, who was president in 1930, or maybe to hear a song from her youth. But she asked very localized, personal questions like, "What am I having for dinner?" and "Who gave me these flowers?" She had told me that the hardest part about being over 100 for her is that she no longer can walk on her own. So if it's not within the radius of her chair, she can't, you know, do it. She has to call a caregiver. And the voice assistants bring back a lot of this independence for some of these smaller tasks that maybe we don't even think are that important, but in fact bring a lot of feelings of independence and dignity to people who were struggling. Thank you very much, Callie. Thank you. My pleasure.

M-A-C-H-I-N-E-Y-E-A-R-X-Y-N-G. Alright, thanks for listening, and thanks to Patricia, Advah, Bree, and Cathy for giving us some of their time at VOICE 2018. We're not going to belabor this because they were all boss operators, But today's pod was all women who are kicking ass in the voice and machine learning world. Maybe we need to write our first skill and have it go something like, "Alexa, find me a Tech and Culture pod that goes beyond tech bros." Good idea? Too on the nose? Let us know. Tell us what you think. A rating and review on iTunes does a lot to help people discover Machine Learning. If you dig this, If it's valuable to you, say so. DMs are always open. Please subscribe so you never miss an episode and share this with someone who cares about how we make sense of these changing times. Machine Learning by Assist is made by Paul Ciuffo and Michael Elsesser for Limina House. Have a great day.