Next-gen voice assistants

I think a lot of people that aren't working in the field would assume that we're going to take a few years and kind of solve all this NLP and related stuff.

Yann Lecun famously said that he's going to solve NLP in two years.

How many years ago was that?

I think like six, seven at this point. He's made great progress.

Yeah, yeah, no doubt. Now I heard a similar thing. I think it was Eric Schmidt in a tweet. He was like, speech is a solved problem.

I think Eric Schmidt showed a lot more wisdom than Yann Lecun did there, despite probably less intimate technology. Understand?

Okay, but I gotta ask this now. I'm curious because you said that, and I know that it's not a static thing. It's done. I know that you're going to continue to make great progress. You've been doing that and you've been telling us about it through this. Where is this going and how does larger vision for the problem evolve over time to where you as a young man, a millennial, says this Gen Xer, who's significantly older. How is this evolving over your lifetime to where you are remaining impassioned about solving this problem in the long term? What does that look.

Big?

Thanks to our partners Linode, Fastly and launchdarkly. We love Leno. They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then ch out O'Reilly's online learning platform. Learn more and keep your team skills sharp@o'reilly.com changelog again o' rally.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible. To everyone, this is where conversations around AI, machine learning, data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're PracticalAIFM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host, Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How you doing, Chris?

I am doing just fine. How's it going, Daniel?

It's going great. No complaints. Took a few days off last week and short week this week, so I'm rushing to get things done this week, I guess is how how it is.

It's supposed to like recharge the batteries and make you feel all refreshed, but you just end up having to do all the same work and get it done faster. I get it. Yeah.

I have known people in my life that do a really good job of like front loading some of that before they le.

Yeah.

For some reason I've never been able to figure that out.

Neither have I. I think that's like a superpower that people have. It's a vacation superpower.

Yeah. Yeah.

It's not something I've ever acquired, sadly.

Yeah, maybe someday.

Yeah. I have to go into vacations highly stressed out because of everything that I was trying to get done. So I guess that makes the vacation all the more important.

Maybe so. Maybe so. Well, one of those things that I'm definitely working on right now and trying to get out the door is a couple of changes to some of our internal speech dialogue related technology, which is part of the reason why I'm really excited today about the topic because we've got a great guest, the CEO and co founder of Polyai, Nicola Merksich.

Welcome. Thank you for having me. It's great to be here.

Yeah. So before we get into all sorts of speech and voice and dialogue related things, maybe you could just share a little bit of dialogue with us about your background and how you got to do what you're doing now.

Yeah, for sure. So Look, I'm the CEO and co founder of Poly AI. I did a PhD with a guy called Steve Young back at Cambridge with my two co founders, Sean Wen and Eddie Hsu. And you know, we worked on building dialogue in an academic context for a long time, since 2006 when Steve founded the group. And you know, Steve started working on this stuff back when speech error rates were about 20%, which means that kind of like one in five words that you'd say would be kind of like misrecognized and typically it would be the one word that you needed to get right to understand what the meaning of the sentence was.

Of course.

So you know, kind of like building a formalism on top that would kind of like let you model the uncertainty, predict all the right things, know when you got something wrong, disambiguate, ask a question, implicitly confirm something. It's an art, right? And I'm sure we'll talk about all of it but the squat or the deadlift of, you know, just kind of like doing machine learning and nlp, right, because it involves natural language understanding, dialogue management, response generation, interacting with the external world. Its knowledge bases specific different tasks all the way into the natural language generation. Figuring out what to say in human language again and then producing it again in an audio format which is, you know, as human sounding as possible. So it's a real compound movement and you have to get it all right. So it's a really difficult task. It's why I got into it, it's why I stayed passionate about it. And you know, my whole team is really a group of people who worked on it long before this new hype of conversational AI came about. And we worked on it for a while. Our previous company, also spit out from the University of Cambridge, vocal IQ, was acquired by Apple in 2015 to make Siri more conversational, to give it a bit more of an ability to have a back and forth conversation. And that kind of like in academic terms, the multi turn task oriented dialogue is something we've stayed passionate about and apoly AI. We're building voice assistants for customer service, helping brands create something that's a super human customer experience, right? For anything that's short to moderate or even high complexity, putting in automated agents that sound at least as good as your best agents, they have answers to all the right questions, they're always current, up to date, and they're able to provide a superb level of customer service. And if they're not, they hand off to their human colleagues as kind of like supervisors or tier two agents.

So you mentioned one thing like a phrase, multi turn voice enabled dialogue I think is what it was. So maybe you could just kind of set that in context for some of the, like what are the categories of sort of dialogue and voice enabled dialogue that are out there? Like probably most people are familiar with Alexa is that multi turn voice enabled dialogue, what are the sort of categories of things out there that people are doing in terms of interacting via speech.

Yeah.

I mean there are many ways you can kind of layer out the taxonomy here. I think when you think about Alexa, Google Assistant, Siri and you ask if they're multi turn, if you can really have a dialogue, a conversation as opposed to just have many questions answered. The truth is primarily they're single turn question answering or kind of like simple task execution systems. But then again, you know, they're working really hard on making them multi turned. Now one reason why it's really hard to build a general multi turned voice assistant for consumers of all shapes and sizes is that they have very different requirements, they're trying to do different things. So like it's actually a task of enormous complexity when it comes to the things that we do, they're a bit less complex in scope because we build things to help you change your ticket for an upcoming flight or maybe you're making a reservation for a restaurant or you're trying to debug your router which stopped working and you're having like connectivity issues or you know, you're calling your bank and updating your address. So these are all things we do. And the one thing that's important about that task oriented bit of the nomenclature is that it lets you evaluate. And when you can evaluate, that means you're doing good science and you can improve. Now evaluating something that does as many things as Siri or Alexa, it's hard, right? It's building them is hard, evaluating them is hard. Knowing what you should be expecting and where product market fit for them is, it's one hell of a task.

Yeah. So when you're saying a turn just to kind of get into some of this jargon, a turn would be like you say something to your smart speaker and get a response. Is that sort of what you mean? And then in your multi turn, like if you're trying to debug your router or you're changing your flight ticket or something, that is likely going to take more interactions than a like uttering something and getting a response. Is that correct?

For sure, for sure. So yeah, I mean the whole reason we talk about turns is that for the most part the dialogue systems today, from kind of like the voice assistants of the large tech giants to automated customer service to chatbots, are built on what's known the turn taking paradigm. Right. So the assumption, and it's a strong one and it's not something that necessarily holds in human speech, is that you're going to wait for me to finish before you start speaking. And the assumption is that you've also absorbed all the information that I've tried to relay over to you before you've started speaking. Then we're taking turns speaking and yeah, a multi turn conversation is kind of like anything that takes more than one turn to achieve a task. Right.

I also have some. I actually want to take you back for a minute because you said something interesting. I'm curious about your perspective compared to someone like myself who doesn't have your expertise doing this. When you were talking about, you know, Alexa and Google Assistant being hard and you use the word hard associated with that several times, I really couldn't help wonder as someone who has been doing this as long as you have, you know, starting with those Steve Young days and moving forward to the present. When you say it's hard, I'm kind of curious how what you're thinking, what's, you know, you're compressing it all into a single word. But how are you thinking about that? As you were saying that, I kept wondering that.

Okay, okay. No, it's a really good question. I say it's hard because it's enticing, it's fun, it's a big problem. I expect to spend, you know, the rest of my life solving it and to be maybe, maybe a small cog in the wheel of how that ends up being solved. It's a hard problem on a pure, kind of like academic level because it's that compound movement of different NLP tasks that all need to work really well and they need to communicate with each other, which is something that we've not really yet cracked. Right. So say the thing at the center of a dialogue system. Language understanding is not a task that's fully well defined just yet. Right. So say, think of speech recognition in most languages. You say something, there's exactly one way, one way of writing it down. But natural language understanding, well, what does it mean? How do you choose to interpret it? What are the kind of like things you're choosing to take away from? Even like an order or something as simple as that. Right. So there a lot is left to the interpretation and that means that it stops really being, you know, a science or even a field of engineering where you have a clear metric to beat. Because really what we've shown over the past 10 years, especially with machine learning, is that you give something clear evaluation and you know, the share force and the intellectual power of the people working on it will crush it. Right. If you say, think about question answering and the squad data set. Right. I remember the leaderboard on the Stanford website, I believe. And, you know, first I think, like, the scores were like, pretty low. And then all of a sudden, like three, six months in, we got to the point where performance was unbelievable. I couldn't believe that it's that good. And it's because when you define a clear scope for a problem, we'll build the machinery to solve it. Right now, when it comes to building these voice assistants, we don't know what machinery to build. The truth is we've built a lot, right? Alexa is manned. I like to compare these assistants to aircraft carriers, right? And you think about Alexa, 14,000 people, right, building that thing. And if you think about the roi, they're not building it because they're making a ton of money on it. They're investing in the future because, I mean, Amazon always has a math around it, right? And, you know, I hope they're right. In any case, it's really good for us because they've actually indirectly funded a big growth in the area and they've allowed us to, you know, in turn, build a lot of stuff ourselves. But we were doing it before they got interested. Right. And the problem itself is. It's hard. Yeah, I mean, it's a good word to use because you have to solve a lot of these different problems. You have to solve things that are not just in the domain of machine learning, but also like human computer interaction, like the voice user experience is something that academics tend to overlook. They don't appreciate that sometimes just the tone or something like that is much more likely to prolong a conversation and imbued the caller with enough patience and goodwill to go through a conversation equally. People who are very good at user experience don't tend to be mavericks at machine learning. And then kind of bringing it all together to build something, it takes a lot of different Personas. People even that is hard. So, yeah, I guess that's what I mean.

You're talking about the human computer interaction. And maybe you could speak a little bit too. Like, I think the way in which people interact with, whether that be like a text chatbot or a voice assistant or something, is different than how they might interact with another human. What are some of those differences in terms of ways that people interact with those systems versus, like their friend and meeting them at the coffee shop?

Oh, for sure. I mean, I think that we could spend a lot of time just talking about the differences between voice and chat. Right. And how people interact there. I mean, people will dispense with pleasantries and they'll tend to use Shorter sentences. When they operate with technology, they'll swear more. A big chunk of input going into all the large tech companies assistance are.

I'm sure you have a huge database now of incredibly explicit and abusive language that people have, have said to their bot.

We've maybe got a few colorful examples. I can tell you for a fact that we have a lot less, you know, percentage wise than the large tech companies where the number of turns coming into these assistants with profanities can reach double digit percentages. Part of it is just human nature. Like what do you do when you can like use a new technology that understands you well? You swear at it because why not, right? Like let's see what it does. And you know, I mean, hopefully you've tried it at some point or hopefully, I don't know, I don't know why hopefully. But you know, things like Siri will have really good kind of like, you know, backup mechanisms to be sassy or to tell you off when you, when you've cursed at them or even when they think you might have, which can happen because speech recognition is not perfect. But yeah, I mean the other thing that's always interesting with technology is, you know, what do people build first when a new kind of like framework comes to mind. And I use this in one of our first kind of like investor pitches. But at the time the top four applications within Alexa were things that allowed the system I think to read recipes out loud, which is kind of cool. And then the remaining three were meowing, farting and barking. Right? And funnily enough, the top revenue grossing app when the iPhone was released was something playing one of those kinds of sounds. You can probably guess which one. And I guess it's a pattern of how technology evolves. But people tend to do these kind of like simple things that are just hacks where they have a bit of fun and then they go and they build like life changing things. Foreign.

This episode is brought to you by our friends at rudderstack. And we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs. Enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes. And because they're open source, you can see all their code so you don't have to worry about vendor lock in or black boxes and best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500,000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U D D E R S T A C K.

So Nicol, you're talking a little bit about different applications of voice technology, but also the way people interact differently with even chat versus voice. From your perspective as someone who's really, you know, hands on working with customers in this space, what makes for a good voice use case? From my perspective, people maybe don't have a great grasp on yet in terms of like, yeah, we all think voice technology is maybe gonna be like a huge thing and we can see really cool applications of it and maybe even really useful applications. But it might be hard for people to visualize, you know, what is a good voice application and what are the benefits of that as compared to, you know, creating text based search or creating like other things like when should I be thinking maybe voice.

As you answer that, can you kind of differentiate between voice and chat just for people who aren't intimately familiar with the use cases?

Yeah, for sure, for sure. So, you know, I think that there is a bigger question of like where you want to use voice as an interface to technology and then there's just the more narrow question of where you want to use voice or text when dealing with customer service. Right. And the only other interface you really have other than language are good graphical user interfaces. Right. So let's say you know, like smartphone apps and you know, kind of like the web. Right. I mean obviously the language based ones are better if you're on the move or if you just simply, you're not in front of a computer or if you want to do something really quickly. Now how the whole like AR VR space will evolve, it's hard to predict, but we know it's coming and there the role of voice in particular is going to be, you know, kind of like much, much larger than what you see when you interact on the web or with mobile. Mobile in fact is the worst one because you know, you're kind of holding your phone, it's a bit awkward. You're typically surrounded by people, it's awkward to speak into your phone. And really, you know, the place where voice on a phone has really been successful is kind of like hands free while driving. And there Siri gets a tremendous amount of usage. When you think about the web, I think that's where chat is a natural interface for customer service compared to speaking often. Because, you know, you might be at work, you might be, you know, speaking to your bank or dealing with something, and you don't want your colleagues to know that you're actually doing that at work. So chat is pretty useful. Or maybe it's early in the morning and you don't want to wake up other members of your household. But in reality, you know, 60 to 70% of all customer service interactions happen over the phone, right? And they happen with voice. Because, you know, in this day and age where you could easily transcribe all this and have all sorts of channels, we're doing this podcast, right? Well, while recording it, I see you guys and you see me. But the end product is just voice because people can consume it anywhere. It gives them a pretty good feel for kind of like, what kind of people we are, how we talk, our style. A lot of emotion goes through that voice. And it's also a really high bandwidth channel, right? I can probably type a bit faster than I speak, although it depends. I tend to speak quite fast. But really, I have a lot more fun when I speak. And, you know, it lets me kind of express myself a lot more fully. And, you know, if you think about just the need to capture that channel when it comes to customer service, when Covid hit, you know, everyone thought, big crisis like these tend to accelerate technology adoption, right? And for, you know, close to a decade now, companies have invested in digital transformation in order to push people to digital channels, mostly chat. Either web chat, where they're humans on the other end, or kind of like chat with an automated system. And the hope there was it's cheaper. Some people, especially those heavily invested in these projects, would tell you that, you know, it's the channel of the future. Younger people prefer doing it. Well, look, I mean, I'm a millennial. I've got a PhD in computer science. I grew up playing computer games and, you know, like, not seeing the sun. And guess what? When I need customer service, I like to call, right? And, like, I have a bit of anxiety calling in, right? Like most millennials do. I still prefer to call because it gets the job done. The alternative is you're typing and then someone responds in four minutes because they're actually speaking to 10 people at the same time. And it's not really a better experience. And with COVID you know, people thought, well, hey, now's the time for chatbots to take over. They'll go from there, 10, 15% of the market share Heavily augmented by the fact that you're being forced onto that channel. And the hope was now it's going to go to like a much higher percentage. Truth is, Covid hit call center volumes went up. Now the staff went down because of social distancing or mandatory lockdowns, but really people kept calling and it's dispelled that myth. Now in our case, it's been really great for poly because we build voice based systems for customer service. And it's been a big boon, especially in getting into those industries that previously might have hesitated to build this kind of futuristic technology. But it's not going away, right? And as time passes, you've got a smart speaker in every part of your house. At some point you'll have some kind of wearable that'll capture your voice really, really well. It's going to be really convenient to just say like, oh, hey, turn on my thermometer and order pizza. And I mean, we can talk about these scenarios.

I'm curious though, as we're about to dive into that next, do me a favor and set some context for me. For those of us like both of you guys are experts in natural language processing. I'm one of those interested people, but I'm not an expert like you. And we've talked about having this multi turn dialogue and these interactions as you're going and solving this for people out there and providing these capabilities that, that we're all getting excited about. Could you talk a little bit about what it is that you have to be thinking about in that pipeline? You know, as you're doing multi turn, what are the things that are part of that consideration for those of us who are not as intimately familiar with that?

Okay, so if you think about kind of like that, that cycle of building a dialogue system, especially if it's voice based, the first step in it is speech recognition, right? Kind of like transcribing what you think the user said. If you're doing it in the best way possible to maximize performance, the output of that is not a single sentence, but instead something that is a lot more complicated. A bit more complicated is an invest list. So kind of like maybe 10 different hypotheses of what you might have said. So let's say I want to get Serbian food and I said it fast. Was it Serbian, Siberian, Syrian? You're not sure, right? Or like I want to work for three people, right? Like did I say three or did I say three? Of course. What if we go into just like that technology, the language model there would basically Say, hey, it's more likely that they said three people. But then again, free people is also something that you tend to see quite often in text. So it's not impossible. So a good system will tell you, I think it's three people, but it might be free, right? And equally Serbian, Syrian, a few other hypotheses, right? So then the next thing that comes is natural language understanding, taking what the user said and parsing it and saying like in some ontology that I have previously defined, that I need to interact maybe with the external world. So let's take booking as an example. If I say, hey, I want to come in with me and my fiance, right? That actually means two people, right? And it's not nlp, as in like parsing who the entities are. Because while it could be useful in a composite task of counting up how many people there are in the request, really what you need to know is that I've initiated a booking request and how many people have asked for, and it's two people. And that's actually a really, really hard thing to do because like actually parsing those words and saying, uh huh, two people, that's complicated. Right? Then the next thing is like, let's turn it into. In very traditional dialogue system literature, this would be called a dialogue act. Where I've kind of informed that number of people is two. And that would then go off into a dialogue manager that would say like, okay, well do I have something for two people? And then what you'd know is, well like sorry, when? And then the system would have to go and say, look, well, request the time for the booking right now. Request time. If you respond like that, you sound a bit like. Not even like Jarvis, you'd sound like the Terminator or I don't know, you sound like a really bad voice assistant. Right?

Requested time.

Yeah, for sure. So you know, you need to turn that into like, hey, what time would you like to come in? Right? And then you have to sit. So that's kind of like natural language generation, another big subfield of nlp. And then finally, if you want to produce it in audio, you'd have to use a text to speech engine, right? And that would convert it into audio. You would play it back. And then the big thing kind of like in understanding everything and having a good conversation would be kind of like the bigger task of dialogue management. Looking at the whole previous set of things that were said and using it to augment the prediction in every subsequent turn. So let's say I said, you know, you might have thought that I said three people, but if I repeat three people in the next turn, then that kind of like alternative hypothesis is probably true and the system should figure it out. Like, hey, why is he repeating it? Doesn't sound right. Let's try that other one. Did you say three people? Like, so you can choose to confirm if you're uncertain. And there, there's a lot of machinery around how you handle that. Probability distribution, uncertainty. A lot of Bayesian methods that come into play. It's a pretty serious discipline.

Yeah. So now if you take that like series of steps that you've laid out, obviously Polyai is working in all these areas, but I was wondering if you could maybe talk about like where do you feel like you're having to spend most of your time? There's probably open challenges in each of those areas, but maybe where's the biggest open challenges in terms of advancing this field along that pipeline of things?

Yeah. When it comes to kind of like, you know, where we focus, one place where we don't spend a lot of effort is speech recognition. The speech recognition task itself, because that's one which is pretty well defined, commoditized. A lot of people are playing there. A lot of progress has been made. The big tech companies are pouring in millions and that's great for us because we're just getting a better product that we then get to build on. Right. So we typically use often several speech recognizers in a single deployment to get that variance out so we can extract the best possible prediction out of all of them. So the more uncorrelated Google and Amazon are, the better our performance gets. And we love that and we thank them for all their hard work. Now when it comes to the piece where we really excel, and this is where we're really, really differentiated from your 1500 chatbot providers that a lot of them claim to do voice, but their idea of doing voice is output speech recognizer and a text to speech engine there. It's going to be great. This is why they don't have many appealing voice applications out there. The piece that's really then exciting is what we like to call spoken language understanding as opposed to natural language understanding. So SLU versus nlu. And the difference there is you really have to consider the fact that, that there's a bunch of different speech recognition hypotheses that you can operate over to really figure out what's going on. The second bit is you also have to look at what happened previously in that conversation to know again how to kind of like Tilt the outcomes to improve the accuracy. And then finally, one thing that we do really well, and that's really important is as the conversation progresses, you can anticipate where the conversation is going to go. If I've asked you what for how many people are coming in, or if I asked you about, you know, like, for example, what our systems can do is parse my name, right? And Nikola Mirkcic will never be a common name in English speaking environments. It's a hard name by Serbian standards. But if you know that, like say I told you my phone number and you're authenticating me, then if you inform the speech recognizer that MERKSI is coming up well, then they're actually quite likely to parse it correctly. Even though it's an impossible collection of syllables in English, it's very unlikely even in Serbian. So. But if you know that it's coming, they can kind of wait out for it. Be like, ah, that was it. Right. So that's really important. And that spoken language understanding, that is what lets us do voice really, really well.

Could you also, just for those who are coming along with us, as you talk about spoken language understanding, which may be a new term for some people, could you also just real quickly define, you've kind of talked about some of the qualities of that. Is there more of a formal definition or is this more of an informal way of addressing it? I'm just kind of curious as you bring people into the terminology.

Yeah, it's a formal research problem and it kind of like touches on, on different ones. But if we attempt a formal definition here in a specific dialog task where we're trying to accomplish something, it is this problem of taking an audio stream and turning it into actionable, parsable kind of like slot value pairs typically, or something like that. So kind of like slots are things like maybe say date or location or number of people. Right. So kind of like extracting that structured information that in the backend your logic. So not AI like your pure kind of like business logic knows what to do. Either it sends a booking request, it sends a query for a specific kind of information or something like that. So rather than nlu, which is, you know, again, relatively complex to define when it comes to dialogue, is this again idea of extracting the same kind of information from a written sentence. Now the thing about a written sentence is that there isn't any noise injected by the speech recognizer, whereas in SLU there's an audio file which, you know, it's not only about a speech recognizer. That may struggle to recognize a particularly complicated word that may be, you know, from a pharmaceutical or, I don't know, a travel domain or something that doesn't come up frequently, or it's a problematic last name. But really maybe it's just background noise. Maybe it's the fact that the accent of the person is not something you're expecting. Your model is not very good at it. Or it could be that, you know, increasingly they're speaking from two rooms away and you're, you know, seven microphones in an Alexa device are insufficient to capture what they're saying. But you know, we're people, our expectations are growing and we, we expect that these things will, will work for us. And that's what makes the problem fun as well, because it's kind of like, you know, shifting goalposts, right? Just when we got it to work. When you normally speak on the phone, speakerphone. Once it works on the speakerphone, there's a baby crying in the background and then you're driving and there's a baby crying in the background and someone's talking over you, right? So, and then you, you might want to switch language, right? So it's fun, like it's hard.

Problem.

As I said.

We deserve a better Internet. And the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

As you were talking through some of the things about spoken language understanding, one of the things that you mentioned was things related to like specialized jargon maybe or particular accents. I guess my question is like, let's say you're onboarding a new client, they're in a specialized domain and they're trying to create this, you know, new voice assistant at this point, how difficult is it to kind of onboard a person into that? How much, you know, data do they have to provide and like how, how much are you able to sort of transfer things from your other use cases and common data that you have? Maybe both in terms of like restrictions between clients? Because I'm sure you can't always share data that you've gathered from certain clients. And use it to create things for other clients. How does that process work at this point and how much pre trained models can you use and that sort of thing?

For sure, for sure. Well, I mean I'm sure most of the listeners of this podcast know about the importance of pre training for deep learning applications, right? I mean it's typically like you figure out how to pre train well and then like really good things in that subdomain of machine learning follow, right? So when it comes to natural language understanding, I can tell you and I could talk about it for hours about collecting data sets of a thousand, two thousand training examples. And we did this back at Cambridge. My co founder Sean had this really good data set in revolutionary paper. He had one of the first papers on training an end to end dialogue system and that involved that whole compound movement. It's a really well cited paper. Really good piece of work. For that paper he collected a Data set of 600 training examples. And then for another paper of mine, I needed a bit more data, so he collected a bit more. And then I'd go through and annotate, he'd go through and check the annotations. I do it again. It takes about a collective kind of like one week of work and leaves you with permanent, you know, kind of like mental health problems. I don't mean to, I don't mean to crack jokes about mental health, but it is, it is a daunting task. It's no fun at all. So when we started Polyam, we were like, this has to stop, right? We're never gonna build amazing things if we're dependent on doing that. Because bear in mind, like, we're pretty highly qualified for creating this kind of data. So if it takes two people in the kind of like last stage of their PhD after years of doing this stuff to create that data set, like that's not scalable. So what we then started doing was pre training representation models for dialogues that would kind of like look at billions of conversations, things like Reddit, Quora, Twitter, and learn good representations for a dialogue. So that if I give you a set of kind of like, you know, turns like you spoke, I spoke, you spoke, I spoke. And then I say like, hey model, use the representation of the dialogue so far and the representation of a potential follow up to determine whether it's a good follow up to that conversation. So we would pre train in that way. And if you do it like that, then you get a lot of training data out of things like Reddit, Quora, Twitter and Reddit in particular. Is an incredible resource because people talk about all sorts of things on Reddit, right? And there are people from all over the world and in different languages as well. But also just like, let's think about English in all possible different dialects. You'll see anything phrased and rephrased there in a good way. And you know, you can train this thing for a long time. And this encoder converts that we have built is something that's in the family of models like BERT or GPT where really it's pre trained on a lot of data. But unlike those models, it's not really a language model, it's an encoding model for dialogues and it's purpose built and purpose pre trained for conversational AI. So then when you use this model to do things like intent detection, value extraction, all of these tasks that form that big compound movement of dialogue, it's a model that's really powerful, right. It takes much less data to get to a high level of performance than something being trained from scratch. And you know, there's been a lot of benchmarking of this model. Salesforce recently came out with a study that confirmed that this is the most accurate or rather the best thing to pre train with, to get the most accurate models with a limited amount of data or any amount of data really. So this is really important, right? And then what we're able to do and what we bring into all of our deployments is this model, right? And then in all of these deployments, wherever there's not sensitive information, it's just more conversations that are used to subsequently tune that model. But we don't need to use specific, you know, nitty gritty details of, you know, I don't know how you collect British postcodes or how you spell Serbian last names. Like that stuff is a bit more proprietary. We have a lot of different technology that's used to counter those specific sub problems. And the truth is we often have to solve some of these challenges that are a bit separate, often very engineering heavy. But when it comes to that data barrier that people think about, that first step of building a dialog system which you need a lot of data to train, we need a lot less data because we've already spent years pre training this thing.

Yeah. This brings me to probably my favorite subject before the interview. I was reading your post about the convert model. You mentioned that it's pre trained speech encoder in multiple languages. So. And I know that multiple languages is something that's emphasized on your website. And what is your thought process behind that? And maybe let's say that we're specifically talking about this language model. How have you gone about setting up that language model such that it enables you to solve problems in multiple languages?

This is a personal academic passion of mine. So quite a few years ago now, one of my best friends, Ivan Vulic, who is a great multilingual NLP researcher, he's Croatian, I'm Serbian, we met in Beijing over a few beers and we were like, well, how do we end up working together? Later he worked on multilingual llp, I worked on dialogues. We were like, hey, can we like do something multilingual in this context? And there we got them really interested. And this was a time where kind of like word level word vectors were all the rage. Things like the word 2vec model, glove and all those things, you know, Mikolov, Pennington, all of those guys. And like this was the first wave of like massively mindlessly data driven nlp, right? And like I stand for that, I love that, right? But I also love languages and its nuances, right? So the question is like, well, okay, you train something in English, how do you port it to another language? Typically older school NLP had like this pipeline of things running, you know, lemmatizers, stemmatizers, you know, parsing the sentence structure. And like that's different in different languages. Like the, you know, subject, verb, object, you know, works differently in different languages. The morphology murders you in different languages, right? Like if you go from word gender is a thing that exists in some but not in others. In one language, word order matters and others you can do whatever the hell you want, right? And I mean, it's sort of fun, right? But when you think about like then creating a dialogue system that works across all these languages, it's daunting, right? I mean you can't just go and translate because a lot of stuff is lost in translation. And you know, like just the multi sense words in one language could translate into something catastrophically different.

Different rhetorical questions.

Yeah, that's the tip of the iceberg. I mean even mundane things like the word bill might mean something. You know, like an account or a bill could be the same word in one language, but they're not in others, right? And that's just very confusing. Like they trigger different actions that both exist. Now what we started doing then was training word vector spaces which would embed complete vocabularies of different languages into the same high dimensional mathematical representation, right? So words like, I don't know, xion, kund, dog, suka, Hound, whatever, hund they were all like in this bubble in one place right. Now, of course there are problems with this because they're multi sense words, but you know, the multi sense ones tend to float away in a bit of a different direction. And then you have machine learning models trained to operate over the those mathematical objects instead of operating over a unitary representation of the word dog in Serbian or in English. Right. And if you do that, then the beauty of task oriented dialogue, when it's a specific task, is that you don't need to understand the nuances or the rhetorical questions. You need to understand that someone asked for a table near the disabled toilet. Right. And at that point the fact that you're just parsing a limited number of intents means that you're actually able to do it across different languages at once. And that's like a big thing that we really, really care about. It's again a place where pre training comes to our rescue and where we're then able to do these things very, very well. The other thing, again, a big thank you to all the cloud providers for the millions they've poured into speech recognition research across different languages. Because that's not a thing that we have the budget to do and they do it pretty well. And you know, that's a piece we don't touch, but it's provided by those companies and everything else we do at house.

Yeah. And so then like if you have this multidimensional space which embeds vocabulary from multiple languages, is the hope then that like when you add. So let's say that you support X number of languages, but then your next client wants to have a dialogue in a next language. Maybe that language is related to one of those you already support, but it's different. Is the hope then to sort of retrain that and add it in or transfer, learn from that existing model, which is faster than training from scratch in a whole new language or how do you approach that situation?

It's a really good question. You could do either. The transfer one seems cheaper and easier, but really in our architecture, the best thing to have is a unified approach that works across all the languages we want to support at once. So they're really retraining. Everything makes the most sense. Now bear in mind we have a single model that is kind of like the nuclear reactor of our system. So that model needs to be retrained and once it's retrained, we're done. Like that language is in there forever. Right. So on that front, yeah, it's a Lot more heavy lifting than like fine tuning 100 small models. But it provides this unified thing that will in the long run save us. This is similar to kind of like what PageRank did to search, right. They kind of created an algorithm that just indexes an incredibly large matrix. Right. And factorizes it. But once it does that, well, here you go. Search forever and Internet changes. I refactorize the matrix and then you know, I'm done. Whereas you know, previously we had like, you know, in older search engines you'd have to go to a specific kind of like industry and then you search there and there would be a small kind of keyword based model that would flag the right results and it had like special if statements on what's more relevant. Whereas now you have a unified approach and it works a lot better. And we're trying to do the same for language.

So back at the beginning of our conversation, you said something and I've been holding it onto because I knew it wasn't yet the point where I could ask. You said as we were getting into the topic and you were introducing it, that you expected to spend the rest of your life working in this area. That made me, as a non expert in your area, really wonder. I think a lot of people that aren't working in the field would assume that we're going to take a few years and kind of solve all this NLP and related stuff.

Yann Lecun famously said that he's going to solve NLP in two years.

How many years ago was that?

I think like six, seven at this point. He's made great progress.

Yeah, yeah, no doubt. No, I heard a similar thing. I think it was Eric Schmidt in a tweet he was like, speech is a solved problem.

I think Eric Schmidt showed a lot more wisdom Daniel Lecun did there, despite probably less intimate technology understanding.

Okay, but I gotta ask this now, I'm curious because you said that and I know that it's not a static thing, it's done. I know that you're going to continue to make great progress. You've been doing that and you and you've been telling us about it through this. So I'm fascinated about where is this going and how does the larger vision for the problem evolve over time to where you as a young man, a millennial, says this Gen Xer, who's significantly older, how is this evolving over your lifetime to where you are remaining impassioned about solving this problem in the long term? What does that look like? I'm really curious about that. I've been waiting the whole way through to ask you that.

Well, that is a tough question. I think that, you know, they're just like a lot of like tactical things to solve. Right. So like say, customer service. It's a challenge that Polya as a company is focused on. And I think we'll be working on it for a long time because we're really far from having a voice assistant that you speak with without kind of like getting that, you know, dinge of frustration at the start when you realize that, oh God, it's automated. Right. We want to make that like a non problem. Right. We want people to call in automated. Fine. The same way that, you know, like you, you, you log into a website. You've never seen the format before, but you figure it out. Right. You know, I think there's a lot of work to be put into those things becoming really, really good. It's not a small challenge. Right. It's going to be an adoption curve that is also partially just about shifting consumer behavior. Right. And voice assistants have done a lot to help there. They've shifted this into the realm of possible likely. And you know, the new generations, you know, kind of like people younger than any of us are growing up with these things. And that I think is really powerful because, you know, we might be the last generation which was really, really fluent in, you know, kind of like laptops, opening up a terminal, you know, the heavy intricacies of the web. Like, why would you do that if it's a lot more accessible with a more natural interface. Right. And you know, I think that people who, who are heavily reliant on the web might seem in 20 years time to those, well, to younger people, like, you know, those guys that are still using the terminal or you know, speaking about assembly. Right. I know that's the best and the most fortunate analogy, but to kind of like get into the meat of your question, like let's say that in 10, 20 years, 10 years, we've got voice assistants that are everywhere, right. They're endemic. They're just how you interact with businesses. You know, they've enabled us as humanity to move away from doing all of those mundane tasks. What's next? Well, I think like just a general interface with technology. The deeper answer to your question doesn't come without understanding what happens in AR and VR. What happens, you know, with things like neuralink eventually where like that's an absolute necessity. Right. Like, I really want that because that's really how we then kind of like transcend humanity. This technology then becomes a big interface of that. Right. Of just like communicating it, of understanding or absorbing all that information. And then you could go in full for singularity consciousness in AI. How are you going to communicate with all that? I don't know where that's going to go. I'm not super bullish on that, but I know that this problem is going to get deeper and deeper. It's a bit like the darkened conflict, right, where, you know, people sat down for I think a summer and they're like, we're going to crack this, you know, AI thing. It's like, yeah, right. You didn't even scratch the surface, right. So when we get to the point where we have technology where voice is completely natural, I think it's really hard to imagine what the world will look like at that point. It's going to be great. It's going to be really, really interesting. Right? We're not that close. There's a lot of work. I'll be past your age by the time that happens.

You got a ways to go, then we'll see.

I'm definitely glad to hear that perspective and also just get to hear from your great work with Poly AI. Yeah, it's been a pleasure and I know I'll probably annoy you with all sorts of speech related questions as, as time goes on and I'll look forward to seeing what, what Poly AI does. But yeah. Thank you so much for joining us.

It's been a pleasure and thank you for having me. Have had a lot of fun.

Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify or your favorite podcast app. Your neural networks will thank you. We are also on the web at PracticalAI FM. There you'll find recommended episodes, listener favorites and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors Fastly Linode and launchdarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

Sa.