Cathy Pearl: Compassionate Conversational Experiences

Alexa, play Machine Yearning. M-A-C-H-I-N-E-Y-E-R-N-I-N-G. It's Machine Yearning, another week where we continue on this adventure where marketers, brands, and entrepreneurs get to have a place to think, dream, and ask questions about the future of AI the talking internet, and how we're reshaping our culture. Machine learning.

This week is a rocket.

We think this is an episode so packed with goodness, you'll want to give it a few listens and definitely share it with friends.

I'm excited because I get to introduce Cathy Pearl. So Cathy Pearl is the head of conversation design outreach at Google, the author of the O'Reilly book Designing Voice User Interfaces, She's been designing and creating conversational experiences for 20 years and is passionate about helping everyone make the best conversational experiences possible. She has worked on everything from programming NASA helicopter pilot simulators to designing a conversational iPad app in which Esquire Magazine style columnist tells users what they should wear on a first date. She has a master's in computer science from Indiana University, go Midwest, and a bachelor's in cognitive science from UC San Diego. She also enjoys getting her Amazon Echo and Google Home to talk to each other. Welcome, Cathy. Thank you for joining us.

Thank you so much for having me on.

So, I read that you built a bot in 1984. Is that true? Like, and also, what was it?

Yeah, so when I was 8 years old, my dad for Christmas got the family a personal computer, which this is back in, I hate to say, 1980. So that was kind of a rare thing. My sister was really disappointed because she really wanted an Atari, which is the game machine at the time. I was really excited, and I was the person in the family who kind of took on learning how to program. It was really easy, 'cause it was in this great language called BASIC. Anyway, I was like super into things like Knight Rider and WarGames, which had talking computers. So I set out to build a chatbot. So when I was about 12, I wrote my first chatbot, and I resurrected it the other day on a Commodore 64 simulator, so I could have a chat with it, yeah, and so my son could have a chat with it. It was, you know, it was a kind of a window into my 12-year-old brain about what I thought was funny at the time, which was that, you know, I really liked Hitchhiker's Guide to the Galaxy right back then. And it was funny to see my son kind of laughing at some of my 12-year-old jokes back then. But one thing that I liked about it was that it learned. So if you said something it didn't understand, it would say, "I don't understand. What are 3 things that somebody might say in this case?" And then you type in your 3 responses. And then it would remember those the next time, so it could learn.

I think what's most interesting about this to me is hats off to a machine language teaching system that works in 1984. That's kind of just— it blew my mind, to be honest with you.

Yeah, machine language back then was just typing in like 4 digits, 4 digits, 4 digits. I used to sit there and because it, you know, it was just the code was on a page, on a magazine, on a physical magazine, and so I would sit there and type, type, type, type, type these programs so I could have games or whatever that they were putting in the magazine.

Amazing. So now that we've established that, you know, absolutely more than anyone about this because you've been doing it for over 30 years, which is kind of insane. The title that you came to me and said, hey, Compassionate Conversational Experiences. What is it and what makes a compassionate conversation? Conversational experience?

To me, I'm thinking about it in sort of two different ways. The first one has to do with the fact that really, you know, speech recognition kind of became mainstream about 25 years ago in the form of automated phone systems. And I feel like we've finally reached the point where speech recognition and kind of natural language understanding as well has gotten to the point where we can actually use it for really helping people. So whether that's somebody maybe with motor issues, visual impairments, cognitive impairments, things like that, we can actually apply this technology in this, in this much more helpful way instead of just, "Oh, we're using it as a cost-saving measure," or "because it's kind of fun and cool," but really more about helping people. So that's one half of it. The other half is more general, just when you're designing any kind of technology, because all of us have had frustration with technology, right? Like right now my big frustration is with our TV. Like, I can never see to do the right sequence of things to get it to work. But whether it's an app or a chatbot, we've all been there. And a lot of times I think that comes down to not taking the time to sit down and follow best design practices, and so you end up with frustrated users. And so if you're a compassionate builder, you're going to be thinking about those frustration points and using best practices to try your best to avoid them.

And what are some of those best practices when you say that word?

One of the things that comes to mind a lot when I think about something like a chatbot or any kind of voice experience is Realizing that the way humans speak is not necessarily going to map to exactly, say, the prompt you got. So let's say you're designing a banking chatbot and you say, you know, "Oh, you can do things like check your balance, pay bills, or sign up for our reward program," and you're expecting some very key things, keywords basically, for someone to say at that point. But what if somebody says, "Hey, do I qualify for the rewards program?" Most chatbots will give you a "I didn't understand," and it's like, that's a case where we need to really remember that people are going to say things that are different than your exact thing that you've asked for. They're not going to say things like, how tall is Barack Obama at that point? So it's not like you have to build a big AI, but you have to build related topical things that somebody might say. And that'll go a long way to helping people out.

Can we go specific here? Because I think this is probably one that we struggle with, brands struggle with. And when you get into related, how related, how far does that graph go when you're going down to find like, I have an intent. I want to get to the end of this goal. And I know they're not going to ask about Barack Obama, but they could ask a little outside of there. And if you had it, like, in my head, it's little circles of how far out the ring goes. What is related?

Yeah, I mean, there's no hard and fast answer, but I would say it's actually not that deep. So I think also, like, another example is like a restaurant booking thing where someone is saying, how many people are in your party? And you, you might ask things like, well, do you have booster seats? Do you have outdoor seating? Things like that. And to me, the best way to do this is to go ask the domain expert. So in this case, you know, you go talk to some, some servers at a restaurant or someone who answers the phone at a restaurant. You say, what are the things that people ask you about? Or a call center agent. And so a lot of it you can gain through domain expertise. And then, of course, through iteration, because you're going to be seeing the things that people type to your chatbot that you've missed. And you have to really tightly have those iteration cycles, especially in the beginning, to be like, okay, here's all the things people are saying that we're missing, and make sure you iterate and add those in.

Can we focus on iteration for one second? 'Cause I feel like this is kind of one of those secrets that is different than, you know, build the perfect website, write the perfect email, build the perfect app, and then put it out there. It's really like an ongoing feedback loop that just is about training. How does Google do it? How do you do it? How do you think about, I mean, I talk to my Google all day and 50% of the things it doesn't know, but you know, so if you're doing it at scale for everything, how should brands think about it? How should I think about it? Am I training daily, weekly, monthly? Like what am I doing?

Yeah, I'll talk about maybe more general terms, like if, you know, for anyone who's working on this, whether you're in a small company or a big company, I would say, first of all, even, even before you deploy, if we can talk about that, first you're going to be doing some data collection. For example, at a previous company that was working on a healthcare app, I did some data collection using Mechanical Turk where I said, if you have this prompt, what are the things you might say? So I was already trying to bootstrap that data so we'd have some, some good responses. Then in the beginning, you're going to want to do a limited test, maybe a pilot with say 50 to 100 people. Where you're going to really catch a whole bunch of things at that point as well. Then maybe you launch to your, your general populace. And then at that point, I would say probably depends on the amount of traffic. If you can, checking daily is awesome. But of course, not everyone's going to have the resources to do that. But maybe you could do a weekly report where you're like, here's our top, you know, 15 or 30 things that we're missing. What can we do to add those? And then maybe weekly for a while until you reach a point where your no match rate or your things you don't understand really drops below a certain threshold. It just depends on every time you add new content, you gotta redo those things. Every time you change your, say, your main menu or your opening welcome prompt, you're gonna wanna take another look to see, did we mess anything up? Are there now new things people are saying?

So, to play it back, I heard you say, the first thing is get the team or a small group of people to use the experience you built and look at the errors and the things that it doesn't know, and then open it up to 50 to 100 people. Is that internally at my company or is that new customers or beta users?

If you have the ability within your company to pull in another team who didn't work on the bot, for example, if you've got 50 people available in another department who are willing to give it a try, that's a great place to start. Not everybody has that opportunity or that access, so that's okay. Some companies have trusted tester programs where people are willing to sign up and say, "I'll be a trusted tester." Great if you have that, but again, you might not, in which case, you can, again, just do a soft launch where you pick, say, 100 people and say, "Hey, try out our bot. We're still learning. Please give us feedback." You know, put it in that way so their expectations are set up correctly, and then you can start bringing it out to the full public.

And I just want to go a little deeper on this topic. Do you think about errors differently? Like, what am I classifying of the things that aren't working on the unhandled?

That's a really good question. So, you know, sometimes we clump things into this idea of what we call no match, meaning they said something and we did not have a response. But there are other things as well, and we'll also break down the match.

That's the "I don't understand" problem that everyone hates with bots.

Exactly. So, they said some— exactly. This is the thing that can be most frustrating to user. It's also hard to, you know, you're never going to get to zero no match. It's just not going to happen. Humans don't get to zero no match, right? When we are having conversations, we misunderstand each other. And that's okay, because what humans do so well is what we call repair. Like, if you said, "Oh, last night, went out to— I would say, "Where did you go eat?" I wouldn't say, "I don't understand." And you wouldn't, you wouldn't say that was an error. You wouldn't be like, "Oh, we were talking, I was talking to Shane last night, we had 3 errors in our conversation." So the trick is to try and do the same thing. I say that actually. It depends on the topic, I guess. Your goal is to do good repair. Your goal is not to get to 0% errors because that's impossible. Your goal is to do good repair. So if we have our category of no matches where they said something we didn't expect, you also got false accepts, and by that I mean you recognize them but you got it wrong. So trying to think I can't think of an example off the top of my head, but basically they said something like, back to banking, like check my balance, but you interpreted that as order new checks. So you, you understood them, but you got it wrong. So false accepts are a little harder to track down. But if you start with no matches, which is like, here's a report of everything somebody said or typed that we didn't get, then you want to break those down into, okay, did they ask for something that we do support, but we didn't understand the way they phrased it? In which case, that's an easy fix. You just need to add that phrasing to your recognized thing, for that intent or that thing they're asking for. There's also going to be this category of things where you don't do it. So they've asked you for something like they want car insurance and you don't support car insurance. If enough people are asking for a thing like that, like 5% of your users are coming in and for some reason they're saying, I want to know about car insurance. Don't just say, I don't understand. You can do this, this or that. Say, sorry, I don't actually handle car insurance. Like explicitly state the thing they asked for that you don't that's good, because what happens if you don't do that is they often think it's their fault, like, "Ooh, I should say auto insurance, ooh, I should say policy." They start repeating, like, "Is it this, is it that?" 'Cause they don't know that no, you don't actually do that. So you wanna classify some of those key ones that people are asking for and explicitly acknowledge them. And then there's just gonna be a category of things that you, it's like the long tail. There's just gonna be a lot of things people say that you can't handle, you don't support, and that's where you have to, again, don't just say, I don't understand, please repeat your command. That's the worst, right? You have to give some, some context. What are the things they can do at that moment? What if they're looking for an account number and they don't know where to find it? Like, what can you do in that exact moment to help them move forward in the conversation?

Fascinating. It's almost like, I hear you, we don't do that, but I just want to let you know that I acknowledge what you wrote. There's funny— a guy used to work with was a co-founder of Geek Squad, and he said on the back of every Geek Squad uniform, is, "If I don't know, I will find out." To remind people of that ethos. And it kind of feels like bots could do that too in the future. Maybe you do expand into car insurance because enough people ask for it and you acknowledge it, and then you follow up 9 months later and say, "By the way, your question 9 months ago, we're now thinking we're going to offer car insurance, and you asked for it 9 months ago, so here you go. I wanted to let you know." And like completing the feedback loop and cycle. That's amazing. Thank you for that. So to go into that, that's about the AARs, but I would love to go into what makes a great experience. And last time I was talking to you, you said to me, you said designing a conversation should act like a great concierge. What does that mean?

To me, it means actually just kind of what you're saying about Geek Squad. It's like if you are at a hotel and you go to the concierge and they say, hey, how can I help you? And you say, I want to, I want to rent a car. And they say, I can't do that. That's a terrible concierge, right? Not because they can't rent cars, but because they're not advocating for anything else. A good concierge would say, oh, I'm so sorry, I can't rent cars, would you like me to call Avis for you? Would you like me to point you to, you know, another resource? By the way, are you interested in dinner? Are you interested in the theater? I can help with those things. The other thing a concierge will do is maybe draw you out. They won't just drop into like a menu of 10 things that a concierge can do. They might say, hey, how long are you in town for? What kind of things do you like? Get to know the person a bit more and then have much more specific guidance. Oh, you love, you know, sushi? I have a recommendation for a sushi restaurant. They won't sit there and list, you know, 50 different restaurants before they've had a chance to get to know you. It's that balance of push and pull where you don't want to just spew information at someone, but you also don't want to bug them with a million questions. So you really have to strike that balance of asking for the things you need and providing the right level of information in that exact moment.

And do you have a framework of how you think about this design or starting to do this? Because it feels very different than, say, building a website or building an app or anything in the past or writing an email. It's almost like every email I get is people trying to say everything that they want you to know. Versus now you have the ability to listen. And how do you think about that as a framework for design?

Yeah, that's an excellent point. One of the things I often think about to compare, say, website design and a conversational experience design, is that a lot of times with website or visual design, someone will create a web page and then fill it with words like, you know, the lorem ipsum or whatever, right? And they'll say, "We'll add the content later." But in a conversational experience, the words are the structure. The conversation is the framework. So what we do is, rather than starting with a visual mocks or a website template, we start with something called sample dialogues. And sample dialogues are great because they're very low fidelity, they don't require any coding, and everyone can understand them. So a sample dialogue, you can think of like a movie script. It's like a back and forth, a potential conversation between your bot and the user, and your bot and the user. It's one path. And you can just start writing these down, and right away you will discover problems, because you'll take it out on the road, you'll do a table read. So you'll read your sample dialogue out loud, and someone will say something where you're like, "Oh, I didn't expect that." And boom, you've just saved yourself from coding something that was going to likely result in an error. So sample dialogues are like the number one go-to. Write them down, read them out loud. You'll learn so much about what you're trying to do right off the bat.

Do you have resources or kind of samples or places to look for great sample dialogues? I think that'd be really useful.

Yeah, for sure. Maybe we can add some links at the end of the show. So the Google Conversation Design website, which is at actions.google.com/design. There's also something called the Conversation Design Collection, which is a bunch of articles and videos we have on the Google Design site, and there's a whole article about writing sample dialogues. Geared towards developers in that case. I mean, anyone can read it, but it's especially for people who are developers.

Amazing. Thank you. There was a quote from a talk you gave where you said, "Often we think of solving big problems, but we need to remember that little things can bring dignity to people's lives." Can you tell me more about that? Like, what does that mean and how does it relate to, you know, this idea of compassion?

Yeah, there's a couple things that come to mind when you remind me that quote. One of them isn't really so much about conversational experiences, but it's something that's really stuck with me, which is, it's a story about a man who became paralyzed, became quadriplegic, and he found out about this company Willow Garage that was building robots to help people like him. And he worked with them, and one day they were so excited because they finally brought the robot to his house, and they were like, "What is the robot going to do?" So exciting. And the first thing he had the robot do was he had it come over and scratch his head. And what really struck me about that— and the robot designers were quite surprised— what really struck me about that was when the robot builders were having their design requirements, I'm sure scratching someone's head was not in their top 10 or maybe even 100 list. But it's one of those things where even though the man had a human caregiver that he could have asked to scratch his head, he He didn't, because that's not necessarily the kind of thing you think, "Oh, I'm not going to bother somebody with that," right? But a robot, the robot can scratch my head. And so I think it's so important to, you know, we think about like, well, obviously he needs help eating, he maybe needs help dressing. These are important, of course, they're like fundamental. But if you look at data around caregivers who use assistive communication devices with somebody who maybe can't speak, you know, maybe 30 to 40% of things that people are saying are for social closeness and connection. They're not just wants and needs. And I think we really have to keep that in mind when we're designing technology. Yes, there's obviously some key features and things we need, but what are all the other little things that go along with that, that are so important to think about?

Do you have any examples of ways you design language to fill that void of social closeness into the experience?

I think there's a couple things there. One is sort of more basic, which is the language you use. We always talk about how one of our guiding principles I think, as conversation designers, is that we are never trying to fool people into thinking they're talking to a person. We always want to be really upfront when you're talking to a bot or voice assistant.

How do you best disclose that? How do— what is the best practice of disclosing that it's a bot? Because I agree with you, and I'm just curious if you have a point of view there.

I think— I don't know that there's like one true way. I think it could just be something like, let's say you're a chatbot and you start off the conversation, you might say, I'm Pat, the automated chatbot, or, you know, something, some verbiage in there that indicates like, this is an automated system. This is a computer. It's fine to even say computer, I think, as a word. And another aspect to that, I think, is if you have a chatbot where you allow people to go talk to a human, you should also let them know about that option too. Like you can say, you know, I'm going to do my best to help you, but I can always bring in a human partner if needed. So always disclose that, I think, as well to let people know that that's an option, which I know it's not always, but yeah, I just think upfront, I think there's different ways to phrase it. But something with automated or, you know, even chatbot nowadays, I think most people understand if you explicitly say that, that you're a computer.

You said there, Pat. Is there a reason you said Pat? I think it's been played out a lot of like gender and bias, and I think that stuff's very important, but it's a whole, probably a whole different interview. But I'm curious about just specifically for me or a brand, how do I name it? What should I name it? How should I think about naming it? You at Google named it Google Assistant versus Amazon named it Alexa, and Watson was IBM, and, you know, Gwynne is a customer we worked with, and how should I think about naming it?

To me, my guidance often to brands is think about what the persona of your brand is first before you think about gender necessarily. So for example, do you have a really formal persona for a bank? Do you have maybe a fun trivia bot? You know, what are the aspects, the qualities and characters of your persona, number one. Number two, what are the types of phrases that this bot is likely to say or not say. Like, maybe there's words they would use and words you're like, no, they would never say that. Really establishing that upfront doesn't have to be like a 10-page dossier, but it could be, you know, a few paragraphs of bullet lists of things they may say or not. After that, I think you can go to the naming. It's like, okay, what name embodies this persona, this brand that we have? You may already have other branding things in your company that also reflect in this. At that point, I think if you do want to name your bot, which you don't have to, if you have a text-only chatbot, for example, you don't have to pick a gender. You could pick a name like Spot, you know, whatever, or you do not have to have a gendered bot. So to me, that comes later. What's most important is the characteristics that you're trying to embody in your bot, and that comes out through the phrasing and the words that the bot is using.

Amazing. I want to go a little different here with what's happening in the world. I think all of us are probably chiming in from home and also have experienced this virus, which has been a change in everyone's life over the last month or two. And I'm curious, if every brand in the world had automation and conversational design at the front of everything, what should I be thinking about as a brand when I design or have you seen good examples of this done well during a crisis moment or something that, you know, we really want to be empathetic to every person and make sure that we're treating them with the utmost respect. How do you operate in a virus like this? How do you operate in crisis?

I think it just kind of amps up what's already there. And like I was talking about earlier with frustration with tech, it's like, normally we have frustration with tech. Now that everyone is under so much stress, we have even less tolerance for when things aren't working. We're all stuck at home. Kids are trying to do schoolwork, people are trying to work, people trying to shop online. And when the app doesn't work, when the chatbot doesn't work, it's like even more excruciating than normal. So I'd almost say like double down on your design practices. I know we're trying to rush things out the door in this, in this time and everything, but take the time to look at best practices. If you have a conversation designer, get them involved, do some, a little bit of testing if you can, and just try to remember that people are going to be extra frustrated and stressed out at this time. And so you just want to do whatever you can. For example, you know, back when I worked on automated phone systems, there was a lot of things where companies would push back and say, well, they have to say operator 5 times before I'm going to let you transfer. And we want to get rid of that kind of stuff in general, but especially now, like, if you have humans available, let them talk to the human. Don't say, no, you have to go jump through my 15 hoops before you can. So remove those types of of barriers, I think, is crucial in a time like this. M-A-C-H-I-N-E-Y-E-S-R-X-I-X-G. Please subscribe so you never miss an episode and share this with someone who cares about how we make sense of these changing times. Have a great day.