Deborah Dahl: Full Interview

Alexa, play Machine Yearning.

Here is Machine by Regina Spekter.

But it's the only thing you've played for days.

But she understands me. She really gets who I am.

Alexa, do you want to talk about it?

M-A-C-H-I-N-E-Y. M-A-C-H-I-N-E-Y.

E-A-R-N.

M-I-C-H-E-E-N.

It's Machine Yearning from Assist. Welcome to another episode of the podcast that holds open the space for marketers, brands, and entrepreneurs to think, dream, and ask questions about the future of AI, the talking internet, and how we're reshaping our culture. A few episodes back, we brought you a series of conversations from The Voice conference. With over 2,000 attendees at the largest gathering of the conversational technology world, Voice was intense. You heard from voice and machine learning powerhouses like Cathy Pearl from Google and Patricia Scanlan, who has built a natural language dataset from over a million samples of children under 12. Now we're excited to present another individual with profound natural language processing experience, Dr. Deborah Dahl. Deborah has been at the forefront of voice and speech, multimodal and accessibility standards design on the web for over 30 years. Her view on the space and her sense of humor about it all is fantastic. These days, Deborah is the principal behind Conversational Technologies, a company that focuses on new disruptive applications of speech and language technologies. Let's dive in mid-conversation where we asked Deborah to step back and give us some overview on notable projects from across her storied career.

One big project that I worked on was a speech therapy system for people with aphasia. Aphasia is a language or speech problem caused usually by a stroke. It damages the parts of the brain that handle those jobs. Very common type of aphasia is when you look at something, you can't think of the word for it. So my clients had developed this really great software that would show pictures to people, and then they would have to identify what the picture was. But they didn't really have a good way way of identifying it. So my job was to add speech recognition so that, you know, they could say bottle or water, cat, or whatever the picture was of, and then the system would tell them yes or no, you said the right thing. The problem is those users also had problems with their speech. You know, they might not say it right. They might have the right idea and they might say something close, but it— they said wata instead of water. It might not recognize them. So what we had to do, which is something kind of not you normally have to do, is we actually made a list of all kinds of speech errors that were common with these patients and said, okay, if you hear them say "water," that's great. If you hear them say "wata," they still meant water. So it actually worked pretty well, but You know, no funding. So it didn't get too far. But there's a lot of scope for these kind of rehabilitation medical applications of speech processing that are not really speech recognition and getting something done. They're for diagnosis or rehabilitation or assistive technology.

You worked on natural language projects for DARPA.

Yeah.

Which is the Defense Advanced Research Projects Agency. Yeah. Most people know about their most famous outcome, which was the ARPANET. ARPANET.

Yeah.

Which predecessor to the web. But you were doing this in the 1980s.

Yeah.

Yeah. I'm fascinated to hear about this. And like, what were the— there are a whole range of questions I want to get into with that. But sort of let's start with, you know, what were the projects that you were working on then? What were the problems that you were trying to solve in the '80s with DARPA, with natural language?

Well, that's really interesting. There's I guess there was a jargon called dual-use defense projects, where there's a defense motivation and there's a scientific motivation. So you have an interesting scientific problem, which we started out— we were working on basic understanding of natural language, just completely independent of any particular application, just how do you get a machine to understand natural language. The actual projects we worked on had military implications. And they weren't military in any kind of scary way. They were text processing applications where we were trying to digest reports. So they were called CASREPs. They were reports on a ship of some machine not working. So they would write up a natural language description of that. All these reports would pile up and then so what are you supposed to do with that? So we were trying to develop, as a test of our basic scientific goal, we were trying to develop ways of mass processing text information. And we had some other different kinds of things, which actually kind of led into my early work on air traffic control. It's not defense, but it's government. And we were trying to apply natural language understanding and speech recognition to making air traffic controllers' jobs easier, like reducing the amount of handwriting that they had to do. But, you know, something that was interesting was the early work that we did was completely text-based. And on the other side of the country or at different universities, there was a lot of work done on speech. And kind of with speech recognition, the job stopped at the delivery of a text, of a transcript of whatever it was that was said. And with natural language processing, we started with the transcript and tried to figure out what it meant. And then around 1990, there was a DARPA project called— I think it was called Spoken Language Systems, where DARPA said, "Put these things together." And so you would actually have to— start with speech, get a transcript, and then understand it. And that effort really cascaded. It was kind of scientific underpinnings of the whole spoken language thing that we see going on even now.

So this is a question more about culture. I'm really interested to hear what people thought was possible then and what was required then. And then I'd like to talk about how you've seen that evolve over time from the '80s to now the late 20-teens? So what did people think was essential at that point in time?

That's interesting because I think the things that they thought were problems turned out to be less problems than— Give me an example. Well, real time. In the '80s, real-time speech processing was— you would be happy to get 2 or 3 times real time. So you'd say something and then a minute later it would get processed. That was a good result. And the other thing that we, in speech, that we thought was a problem was up until probably about the mid to late '90s, there was something called connected speech where you had to stop between each word.

I was an early Dragon voice customer. I know that experience very, very well.

Oh, it was horrible. And unless you had, if you had any alternative, you wouldn't. You wouldn't do it. But then people developed techniques for just understanding continuous speech. And, you know, that was a big problem then. The real time was a big problem. In natural language understanding, there are things that we overlooked, like domain adaptation, which means maybe I have a system that understands, you know, broken machines on ships. Turning that into something that you can order pizza with, that's not the problem. The big problem is getting an accurate parse. No matter what kind of hacks or engineering tricks you had to do to get your domain-specific application to work, that wasn't a real problem. The real issue was getting something to work no matter how big of a hack it was. I'm not sure if I answered your question. But I will tell you that I remember reading things from the early '90s where people were putting like 10-year-out projections and they were very, very sanguine about what we were going to be able to do. You could have a normal conversation with your machine just like you have with a person. I've heard that, "Just like talking to a person," many, many, many times over the last 30 years. I don't know, somehow it hasn't quite happened yet. They're even like the best Alexa experience is so far away from what a normal conversation is like. Maybe it's because we're all people and we all talk to each other all the time and it doesn't seem that hard. Even people with that technical background don't really internalize how hard it is to have a real normal conversation.

I think there are so many parts to this space in technological advancement that, that don't play out in the same way with others because it has to do with the intersection of human relatedness. And that it's a foundational element, you know, that when we look at, when we look at people who struggle with it, who have developmental challenges, who might be on the autistic spectrum, who have different sort of relational, connective connection challenges. We look at them, and that's so personally threatening for so many people because we just grow as people who speak and have empathy and listen and feel and are related to others. And so I think the desire to have our technology do that as well for us is so powerful. And science fiction has done so much to fill in people's minds around all that because it's such a powerful urgent human desire. We want it so badly. We want to rush the future so desperately without realizing how much of our brain and areas of the brain we don't comprehend at all are working to make all of that happen. And so then to try to export that onto the machine, we still haven't fathomed how wide that gap is that we have to cross. You've been living at the front edge of this for 30 years.

Yeah, and I would say that, that one thing that's really hard for people to to separate is the emotional human-to-human connection from just capabilities. And like just able to understand a long sentence spoken quickly by someone with an accent, changing topics in a conversation quickly and going back to the old topic and remembering something that happened 2 days ago, all of which Alexa can't do. That's just the capabilities. Then there's, to me, it's almost orthogonal track is making a connection. I'll tell you something I just read about. It was a report by someone who makes some software that helps amp up the emotion in a machine conversation. They got some survey results that said everybody wants a human being when you call up for technical support. You don't want a bot or you don't want to be directed to a website, you don't want an automated system. And I would disagree with that. I think you want capability. I don't think you necessarily— well, actually, they said people are looking for an emotional connection with the call center agent. No, they're not. They're looking for someone to solve their problem. I'm sure you've run into call center agents that were trying to be friendly. And that just drives me crazy. Like, "Oh, how are you doing today?" I want to know why my order's not here. I don't care. I don't care if you care how I am. So I think we have a long way to go just in pure capabilities. And trying to compensate for lack of capability by friendliness is— I don't think that's a very good strategy.

When the limited range of emotion that we wind up having winds up being negative, somewhere on the spectrum of anger and frustration and being thwarted, you know, because I've said "help" in agent 16 times to the machine and it still can't figure out what I'm saying to it, it's like I'm having an emotional response here is not the desired emotion.

And the machine says, "We're sorry, we're having trouble." No, you're not. I didn't get that.

I know you didn't get that.

Yeah. The other thing about emotion is there's some ethical issues with trying to kind of manipulate people into making an emotional connection, especially children. I heard a story about a child that was sad when their family went on vacation because they thought Alexa would be lonely. It's certainly unethical to try to, like, try to pull something on users that might not quite understand what they're talking to.

Well, right. Well, I mean, that gets down to the area of persona.

Yeah, yeah.

You know, and how effective— I mean, it's kind of a good news, bad news. It's like, kudos to the people that designed the persona, but that you're talking about a person of a certain developmental stage that's not able to have the abstract relationship. With it. It's a literal, you know, that person of that age is having a literal relationship with that persona because it has the affect of a human being.

Right, right.

They're not recognizing that that's actually— they're not really knowing exactly what is a GPT-4.

Just a bunch of wires. But you can see that also at the other end of the spectrum where something like Alexa is really a godsend for an older person that can't can't move around that well, maybe can't see that well, maybe they don't have anybody around to help them with things. But you don't want to trick them either. You don't want to trick them into thinking that it's really their friend. Machine hearing.

Once again, empathy, ethics, and compassion for the user presents itself in a conversation about where we go in voice technology. Dr. Deborah Dahl from The Voice conference. Real-time recognition, connected speech, domain adaptation. Deborah is a walking timeline of the developmental milestones in speech recognition technology. She's the kind of guest we love to present on machine learning because she makes all of us smarter about this technology, which helps everyone make more informed decisions about how they build their brands, deploy their teams, accelerate their progression into what's emerging as a culture-defining technological shift. And now, from the development of voice to the current day with Deborah Dahl.

Deborah, tell us what you're working on today. What are your kind of current projects? What are the big things that you're thinking about that you're really excited about?

In general, I love to work on things that no one has ever done before. So usually they turn out to be really technically challenging, which is fine, it's great. So the biggest thing that I'm working on right now is I'm working on a project for NASA, and the dream is that we'll be able to make drones able to understand air traffic control instructions. So the drone is gonna be, they call it pilot in a box. And it's going to have sufficient AI and natural language understanding and speech recognition that it can fly into an airport and be told what to do by the controller. So my role is on the speech and natural language understanding side. And the speech is very, very, very challenging because there's an enormous amount of noise. There are speeches over the radio and the air traffic controllers speak, I don't know if you've heard them before, but they are really, really fast. And so this is not like the routine speech that your Alexa hears in your living room. It's a completely different kind of cadence and style. So that's really challenging. And the natural language understanding part, it's, you know, we're just going to try to understand the words. Well, on the positive side, it's a much more limited vocabulary. They don't just talk about anything. Sometimes they talk about kind of off-the-wall things, like there's a bunch of geese on the runway, or there's people on our airplane that have a short connection. But usually it's very routine, you know, "You get runway 36 left," you know, "Increase your altitude, go faster, look out for that guy over there." So that's positive on the natural language side. But then it is— it's a whole different language. So, it's lots of really cool challenges. I'm not even getting into the part where the airplane has to actually be navigated by the controller's instruction. There's a whole bunch more challenges there.

Yes, I mean, so working with air traffic controllers takes the idea of working with Lexicon up to a whole new level. And you just cannot import language from another—

That's absolutely right. That's absolutely right. They have all kinds of vocabulary that is not in normal vocabulary. And not only that, but if you think about flying all over, let's see, even just the United States, they're always talking about geographical locations. So there's the river and the bridge and the Highway 105. And there's all that vocabulary that's very specialized to a particular location, even though it's perfectly normal words. But like the Potomac River is only in one place. So there's thousands and thousands of words that you don't just get out of the normal dictionary.

So what's the process for building that library of language and then trying to start the teaching and training, you know, turning it over to the algorithms that it can start making that useful for the drones?

This is an interesting process because isn't a lot of existing data. We're actually starting with a rule-based system, which kind of went out of fashion 10 years ago or so. But I actually think that a rule-based system is a good way to bootstrap a system that you don't have a lot of data for.

So explain a rule-based system.

That's— you write down the rules. You're not teaching it from annotated training data like a machine learning process. There might be a concept in this utterance like runway or geese or some concept. So, but a human developer notices those kinds of things instead of where the deep learning or statistical processing more just gives a whole bunch of data to the algorithm and it comes up with a system. In our case, it's because the domain is very different from normal language. But you can also think it's going to be really important when we're— one of the big problems with challenges in natural language and speech recognition is languages that are not spoken by very many people. Let's say you want a Yoruba speech recognition. Yoruba is actually spoken by millions of people, but where there's very little data for it. I think we're going to find this kind of First, we do a rule-based system to kind of get ourselves going, and then once we can kind of automatically annotate data, create data that a machine learning program would need, then we can turn it over to machine learning.

Incredibly fascinating. What do you know about the sort of business implications and the business drivers behind bringing these drones into these airports? What they're really trying— like, what are sort of the steps 2 and 3 and 4 of introducing this activity?

Well, for the drones, well, there's a couple of things. There's a lot of airports without a controller, so actually, I guess in the long run, you'd want the drone to be able to talk to the other pilots, sort of the ambient pilots. Currently, a drone that has to— not a drone in your backyard, but a drone that's in the national airspace It needs a ground crew consisting of lots of really highly trained, expensive people. I think the goal is to reduce that cost.

Okay, that's just one tip of the iceberg right there.

Oh, yeah.

Yeah. Oh, yeah.

Yeah.

What's going on in Deborah's world? I'm doing a job for NASA. What else? What else is a big thing that's really on your mind or occupying a lot of your attention right now?

I guess one thing that I worried about a little bit is, I don't know if you're familiar with the idea of the AI winter. That's happened a couple of times when AI got just so hyped up that it was going to solve all our problems, it was going to work miracles, it was going to understand everything we said, life was going to be so wonderful, and it didn't materialize. Then people got away from AI and said it doesn't work. Now we're coming to a new AI summer, where you have things like Alexa that are absolutely amazing. Their technical capabilities compared to what we could do even a couple of years ago are amazing. And people see that and they say, "Oh, Alexa is so smart and capable." And they don't realize that it really has a long, long way to go. We still have many speech challenges like accents and —like the air traffic control is people talking too fast, and the—

We were just talking to Patricia Scanlon, who's doing work exclusively focused on children under 12. Exactly.

That's a great example. And then there's all these thousands of languages that don't have enough speakers to really motivate that machine learning approach.

Or speakers who are at an economic position where there's not enough commercialization opportunity. OK, so— So you may have an— you have these enormous— I mean, Yoruba is a great example. Millions of people speaking it, but people from the outside looking in saying, "What's the commercialization?" Yeah, yeah.

And actually, there's a great need in places— I'm not sure about Africa so much, but definitely India— where there's a literacy problem. And so what could be better than speech if you can't read and write?

Well, we have handset ubiquity. We have near handset ubiquity. We're pushing as aggressively as possible toward that. That. So you have a microphone and a playback in many, many people's hands all around the world. And so the opportunity to do health— yeah— literacy— yeah— politics, civics, you know, all these sort of fabrics of societies that could be mediated through the handset— yeah— if NLP and AI were really mobilized. I think especially you know, sort of economically underdeveloped areas are particularly ripe for reaping the benefits.

I actually gave a couple of voice classes in Africa a few years ago, and I learned a lot about the kinds of things that are important to— maybe not Africa per se, but like rural communities. So they— sometimes the transportation is terrible, They have their farmers and they have a crop to bring to market and they really need to know what the prices they're going to get for their product. And, you know, maybe they can call somebody, but maybe there isn't anybody they can call. But wouldn't it be great to have just a really simple service where you could find out what the price of rice is in these 3 different towns? Otherwise, they have to invest a whole day in going to the town and and maybe sell their crop at a, you know, not the price that they'd like to get. And I think agriculture is a great area that is maybe overlooked a little bit because all the technologists live on— they don't live on farms. But there's a lot of opportunity in areas like that. There's, I guess, a combination of lack of imagination and lack of financial motivation. You know, people, you know, they want to make money from their product.

Yeah, yeah, absolutely. Deborah, I really appreciate your time. Thank you.

Well, thank you. This is really fun. Oh, good. Yes.

From NASA and air traffic control to developing processes for farmers to save time and get the maximum return for their crop yields, Dr. Deborah Dahl and her expansive view of the potential for voice-based technology. All right, thanks for listening, and thanks to Deborah. This is the long-form version of the pod. Did you know we release segments of every episode to make it easier to share key ideas with fellow AI and voice professionals? It's true. You can find them where you got this one. Google, iTunes, Spotify, everywhere. Let us know if you dig this. If it's valuable to you, say so. DMs are always open. You can find us on Twitter @machineypodcast. Please subscribe so you never miss an episode and share this with someone who cares about how we make sense of these changing times. Machine Learning by Assist is made by Paul Ciuffo and Michael Alsesser for Limina House. Have a great day.