Real-time conversational insights from phone call data

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog.

Linode is our cloud server of choice. Grab the Nanode plan for just $5 a month. Just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD and one terabyte of transfer. Let's be honest, you can go a long ways on that five. When you do need to scale up, their prices are predictable so you can put your calculator down, you won't need it. We've been running changelog.com on linode for years and we've always impressed by their award winning support team. Check them out@linode.com changelog Once again, that's linode.com changelog. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, printable, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're @practicalai FM. Okay, here's Daniel and Chris.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. How's it going, Chris?

Doing great, Daniel. How's it going today?

It's going great. It was a beautiful weekend after finishing a paper submission to a conference. So, you know, all was good. I got to relax and be outside a little bit and feel the satisfaction of having that done. So what about you?

Sounds good, just family stuff. Over the weekend I had the satisfaction of both. My daughter, who's in second grade, is into basketball and we raised the goal all the way up to the ten foot mark.

Whoa.

And so she's learning that the big time and we transitioned her up to a larger bicycle as well. So both big daddy moments for me. So not AI related, but it was a good weekend in that way.

That is exciting though. Yeah. Yeah. Cool. Well, I'm excited about the show today. We have a topic that is very interesting related to conversational data and extracting insights. We're joined by Mike McCourt, who is a data scientist at Invoca, to talk a little bit about that subject. Welcome, Mike.

Hi. Thank you for Having me.

Before we jump into that, would you just give us a little bit of a background to help us understand how you got into AI and data science stuff and then we'll take it from there.

Absolutely, yeah. So my arrival into the AI world was totally fortuitous. I'd not planned on it at all really. My original goal was to be a college professor and I really wanted to study astrophysics and to do research in that field and to teach studying how structure formed in the universe and how we came to be here. And so I worked for probably 10 years. Dutifully towards that end. I majored in physics and theoretical physics in college and got a PhD in astrophysics. And then I moved all the way across the country for a postdoc position, just like a research position in science, and then moved all the way across the country again for another one. And as I was dutifully working towards this goal of being a professor, I was getting less and less sure that that was the life I wanted. And at the same time I'd seen this new field of AI just take off and blossom.

And what year was this around?

Well, so I was in college 2004 to 2008. Okay. And basically never heard of AI in that time frame. I mean I knew of some people in the computer science department who were doing research in AI. And then like midway through grad school, 2010, 2011, I started to hear more and more about it. It seemed interesting and like by the end of grad school, you know, to an outsider, AI was like hot. And by the end of my first postdoc, AI was really hot. You know, I saw how fast the field was moving and new inventions were coming and it seemed like it was advancing by leaps and bounds. And eventually I got jealous and I wanted to make the switch.

So yeah, it's interesting. It seems like you're even the years. So I was also in college 2004-2008 and then went to grad school in physics. So we've got a lot of similarities there, except I guess you stuck. You had more perseverance than I did. I didn't make it through the postdoc. I jumped ship before then into industry. Yeah, but it is interesting, I guess there's really a lot of astrophysics and also high energy physics people getting into this space. At least that's my understanding now. I'm kind of more removed from it than you maybe.

As the non physicist in the group, I have to say I've noticed that it seems like the AI is breaking out with physicists these days. It's a good thing, by the way.

Yeah, totally.

Yeah.

Yeah. So you jumped earlier than I did. I think you could see farther than I could, but, yeah. I feel like almost all of my colleagues now from grad school are in data science or AI.

Yeah, well, it's definitely very competitive in terms of tenure track physics positions right now, but the demand for AI talent and data science talent is so huge that I think that's driving a bit of that.

Before we dive in and talk about what we were planning to. I'm just kind of curious with both of you here. Any thoughts on is there kind of a natural progression, do you think, these days for physicists to move into AI? Kind of natural synergy between the two?

I'll let Mike go there. Maybe he was in physics longer than me.

Maybe I made the jump more recently, I think. Yeah, I think for me, I felt like physics prepared me extremely well for the jump into data science. I think in physics, what you're trained to do is to take some observation about the world, some fact that you'd like to understand and explain, and then you come up with a model that you believe could explain it, and then you have to grind through the math and turn that model into something that can make predictions. And in the process, you have to collect some data, you have to inform your model, you have to fit four parameters in the model, and you have to understand your uncertainty and all those. And then at the end of the day, you have to make a prediction, and then you have to convince someone else to go run an experiment and test your prediction. And if your predictions come out well, then you believe that you've learned something about the universe and you can publish that. I think that process in AI is pretty similar. Yeah.

It's a great parallel there that you outline.

Yeah. So the focus is different in physics. We don't really care about the predictions on their own. The predictions are there to test our theories, and the theory and the model is really the product that we want to give to the world. And working in AI, the model actually is typically something that we. Well, sometimes we hide it, or at least it's not really the focus. The internal workings of the model isn't what we care the most about. That's the predictions that we care the most about. But I think that process is broadly similar.

Yeah, definitely, I agree, and I think you expressed it much better than I could. So it seems like you've also developed the important data science characteristic of being able to explain things and Frame things well. So I don't know if that's something you learned in physics or afterwards, but yeah, appreciate that. So tell us a little bit more about how you got involved with Invoca and your path to sort of what you're doing right now.

I got very lucky landing at Invoca. So like I said, I was kind of. I was very. I had a laser like focus on my physics career trajectory. I never really considered anything else until I kind of got towards the end and realized AI seemed more interesting to me. So I was not really aware of what was going on in industry and I made the jump a little bit blind. But, you know, while I was making major life decisions, I decided I wanted to work in AI. And I decided that if possible, I wanted to live in Santa Barbara. I'd always loved the city here and I didn't know if there would be AI positions in Santa Barbara. So I just started searching and I found this company called Invoca. And you know, as a scientist, Invoca does call analytics for marketing. I didn't know that much about phone calls and I knew almost nothing about marketing, but it looked like an awesome company. Especially what was really interesting to me was the data set that Invoca has because we study phone calls here. That's an unusual data set list. Most of the research, at least in AI now is not focused on phone calls. So it's kind of a unique problem in a big data set. And it looked like a fantastic company to work for. And so even though I came in knowing almost nothing about marketing, I was able to convince them that my skills as an astrophysicist would be useful to them and landed a fantastic job at a really great company here.

So you kind of led right into where I was about to ask you. And that is, I had looked at Invoco's website and it talks about Invoco being this AI powered call tracking and analytics platform. And so you've already kind of talked a little bit about this interesting data set. And so I guess for those of us who are not in that particular field, could you talk a little bit about what call analytics mean? And also I noticed on the webpage it talks about campaign attribution and it refers to some actionable data. And if you could kind of talk about how all those fit together and what they mean in this context, I'd really appreciate it.

Absolutely. So keep in mind I'm not a marketer, so I'll do my best at explaining the value proposition of what we do. I Think now, marketers, when a transaction happens online, marketers are used to getting all of this attribution information. So if you buy something online, they know who you are, they know what ads you've seen, they know what web pages you've been to, and they can use this information to really analyze their marketing campaigns and optimize them and try to make sure that they're spending money on ads that are driving business to the company and hopefully that are relevant and interesting to consumers.

Yeah, so it's not just like broadcast marketing. The trend, at least now is very targeted marketing, right?

Yeah, it's changed a lot in recent decades. But as soon as you pick up the phone, they lose a lot of that information about you. If you don't finally make your purchase online, but you pick up the phone and call them and make it purchase there, then a lot of that attribution is lost. And that's important because a lot of high value transactions still happen over the phone. If you're buying a new sweater, you'll do that online. But if you're getting an air conditioner installed in your house, you typically want to pick up the phone and talk to someone before you make that decision.

That's a great point in terms of how real life works when you're doing that.

Yeah. If you're shopping for mortgages or home improvement things, or if you're booking vacation on a cruise line, you know, big high value transactions, they still tend to happen over the phone. And Invoca works to close that gap so that marketers can analyze the transactions that happen over the phone and link those to their marketing campaigns.

And I'm just curious, I did spend a number of years as a technology person in marketing organizations. And how do you actually make the link? Or do you, in this case, if they're starting online and you know, traditionally marketing has focused so much on, on analytics with online activity, and then they've totally moved over into this new medium and you start your analytics there, which I guess we're about to learn about. And. But how do you actually make that connection between the two?

Yeah, that's an excellent question. And I have only a thin understanding of it. There's a whole. This is kind of the core of Invokus technology and I just work on the AI part, but essentially it works. When you sign up with Invoca, you get a number of phone numbers and then when ads are shown, a number gets swapped into each individual ad. And so that when someone calls the number, that number gets forwarded to your Business. But the number they called can be used to see what ad they called in from.

Yeah, I think together we're getting there. I know in a previous employer, actually the first data science position I had after leaving physics was with a telecom company. And I think most people don't realize this now, but it is super easy for you to, you know, spin up, you know, even a thousand, a hundred thousand, you know, numbers, unique numbers, and then destroy them right away after you've used them these days via API and things like this. So you can be very smart with your phone campaigns in terms of using unique numbers for even just a single purpose, like doing this link, which is pretty interesting.

Yeah, exactly. But then it starts to get complicated because you want to make sure it's a phone number that no one's used for a while. So there are a ton of people working on that here. I find it interesting.

Yeah, it is interesting. And I think, you know, also it's fairly transparent to the user, which is interesting in a similar way too. Most people don't realize that they're being, you know, tracked by Facebook when they're not actually on Facebook or Instagram.

Right.

There's people that have this like, embedded thing in their website or like most people should, that are doing any type of marketing that links their activity on that website, assuming they're logged in to Facebook or Instagram to their account. And I think a lot of that world is, is kind of transparent to most people. At least it was to me before I kind of got into some of these related things.

That point you made a moment ago about having so many numbers available, I don't think I have ever seen thought of using phone numbers in such a disposable way. It makes perfect sense. And I do do programmatic stuff with phone numbers. You know, there are, there are, there are companies out there, obviously, where you can, you know, get phone numbers for your business and program them into your, your stack and stuff. But I don't think I had thought about it in the, in the context that you guys are discussing. So it's an interesting, interesting thought there.

Yeah, no, I've had the same phone number since high school, so this, same here. This process really, really surprised me. But yeah, it's amazing to see it work. Yeah.

And I'm assuming, I mean, I don't know, it was interesting for me as a physics person coming into this world that, like, you were talking about some of these problems around, oh, you don't want to reuse the number too much. You know what's the optimum amount to do that? How many numbers do I need to spin up? What's the volume of that and the geography and targeting like? These are very complicated problems that. That aren't like the fundamental pure science problems of nature, but they are really interesting problems, at least for me. I found them interesting as a physicist just in terms of their complexity and also their connection to real life and how you could make a real life impact for people. I don't know if it was similar for you.

Oh, absolutely. Yeah. I think one of the really fun things that I learned as a physicist is that you can use math to make a model of something and then optimize it and then apply that to the real world and see it work. It's kind of amazing to me that I can listen to a problem, write something out on the chalkboard, grind through the math, and then end up with something useful.

So, Mike, I think we should probably transition to the more AI related things that you're doing at Invoca, since this is practical AI and we definitely want to hear about those things. And one of the things I know that was highlighted as we were talking before the show was this product called Signal AI and some of the things you're doing around that. Could you maybe just give us a bit of the motivation behind that product at Invoca and how it came about?

Yeah, absolutely. As I mentioned, we do. Well, the kind of core of Invoca is called tracking. So you can attribute a phone call to an ad that would allow you to see which ads were driving the most calls. But really, as a marketer, what you want to understand is what's driving revenue. And that requires understanding what actually happened on each of the calls. So you don't want to just optimize for a marketing campaign that drove more calls. You want to optimize for one that drives more calls of people who actually go on and buy something.

Yeah, you don't want vanity metrics similar to, like, you wouldn't want to optimize an ad for just like clicks of the like button on Facebook or something like that.

Yeah, exactly. At the end of the day, you need revenue. Signal AI is our answer to that. It's a product that will actually analyze what takes place on the call and tell you whether it was a purchase or whether it was an appointment or a cancellation or whatever it is that you happen to want to track to attribute to your campaigns. Signal AI is a supervised model. What that means is that the model is trained by labeled data that's provided by the user. So if you, you know, let's say that you're a dentist's office and what really matters to you is appointments. You're looking for new patients, new people to come in, and you want to increase the number of appointments you're getting per month. You would provide us with, say, 150 calls where an appointment was made and 150 calls where an appointment was not made. And you could train a signal AI model that would now listen to all of your incoming calls as they routed through Invoca and tag them as true or false for appointment. And then you could feed this information back into your marketing system to try to optimize not only which ad campaigns are driving the most calls, but which ad campaigns are actually bringing people into your office or appointments.

So are you primarily classifying the call from the AI perspective, or are you doing more than just classification with the AI stuff? Can you kind of talk about how that fits in directly?

Yeah, so signal AI. Exactly. It's classification. Was it an appointment, was it rescheduling, cancellation?

I'm just kind of curious, are there any other aspects of, particularly around nlp, natural language processing that you guys are focused on, aside from classification?

Well, I think for signal AI, a lot of the processing that we do, at the end of the day, what we want is classification. Classification is actually difficult on phone calls for a number of reasons that I find interesting. One is that phone calls range really widely in length. And so if you listen to phone calls made to businesses, a lot of them are three minutes long or even less than a minute in some cases. And some of them are can be two hours long. And so a classification algorithm has to be able to handle this wide range in lengths of phone calls. And that can be tricky because if the classifier is looking for a particular pattern in a phone call, you know, if the phone calls really, really long, the odds that that pattern is going to appear randomly increases. So it's tough to be able to handle that range of lengths and phone calls. And then another thing that I find really interesting about phone calls is that English is such a rich language that no two people are going to express themselves the exact same way, even for something as simple as scheduling. Sticking with the example of a dentist's office scheduling an appointment for a cleaning at a dentist's office, that's a relatively simple thing to do. But if you listen to 100 phone calls where people are all doing the same thing, they the same goal, no two of them are going to use the exact same sequence of words. You know, there's enough freedom in English that we can all express ourselves a little bit differently. A classifier algorithm that's working on phone calls has to be really sensitive at pulling these patterns out of speech. But at the same time, on phone calls, there are certain things that are exactly the same from one call to the next. You know, if there's a hold message on the call, if you call or I call, we're going to hear the exact same sequence of words in the whole message. And so if a classifier algorithm is too sensitive at pulling these patterns out, it'll get tripped up on these recordings that are present in phone calls.

Yeah. So I have so many questions. This is super interesting. I guess the first is the other thing that I was thinking about. When you were talking about challenges specific to phone calls, you were talking about the sort of, of structure of them, how there are maybe repeating things, but a lot of times there are varied links and there's a lot of variance in terms of language. You know, there's also the element of accent and all of that, I'm assuming. So one of the things I was wondering was are you primarily in the system analyzing the raw audio or are you converting that to text and then doing sort of text based methods to do classification? And along with that, I'm assuming that like phone calls, for example, the audio is not going to be as good of quality as our wonderful quality practical AI recordings or other audio recordings maybe. So are there also issues with that? Those are some of the things going through my mind.

Yeah. These are excellent questions. So by the time it reaches me and on the data science team here, the phone call's already been transcribed. And so I'm fundamentally working with text data, but they're transcripts of phone calls. And so I don't personally work on analyzing the waveform of the audio and turning that into text, but we do have, you see significant differences with dialect and accent. The way that people will explain things varies quite a lot across the country. And you know, again, doing classification like, you know, if I'm canceling if, let's say we have a model for appointment set and it's trained on phone calls from people in California. You know, me as a Californian canceling an appointment, that might seem more like the trues for appointment set because I'm from the same region than someone from, you know, Maine who's making an appointment. And so we really do have to be careful about how we handle dialect and regional differences. And then, yeah, as you mentioned, the Audio quality varies widely. You know, somebody may be making a call from home, and the audio quality is fantastic, and you can hear everything and understand every word that's said. Or they may be calling on a cell phone from, you know, driving on the freeway, and the audio quality is terrible.

What sort of real variance and impact does that make in the actual. Since you're kind of the consumer of the text and it's already been transcribed, do you see, like, an incredible amount of noise in the text because of those issues, or is it fairly clean? And then there's kind of outliers here and there?

You know, I think there's kind of a continuum from transcripts that are nearly perfect to ones that are almost all noise. So we get the whole range and we treat it almost as like a quasi linguistic phenomenon. When a word is mistranscribed due to noise in the audio, that's kind of like an artificial synonym, you know, so words like higher, tire, fire, higher, those might all be synonyms on, you know, for the purposes of phone call transcripts, even though they have different meanings in the dictionary.

So I'm curious, as you go through this process, and I've been listening to you and Daniel, and Daniel's also a real NLP expert, and so I'm trying to learn from both of y', all, are you using multiple models in your workflow as you're coming through this, or is that. Is there one master model? You know, how. Which approach are you taking, and how do you account for variability within the different speakers, given the fact I know Daniel mentioned accent a few minutes ago. So as you run into all these variations that you can have, I guess from an accent standpoint, that would be out from. Because you're only dealing with text.

Right.

But at some point, are you thinking about doing the raw audio? And maybe you do things like sentiment analysis or anything, and I guess I'm. I'm running all over the place here, but I'm just trying to understand.

Yeah. So are you using multiple models in your workflow for, like, different types of calls? Like calls of this accent type or of this dialect? It has one model, and then calls of a different accent or dialect have a different model. Also, you mentioned, like, segmenting things so, like, the whole music or message and that sort of thing. So do you have different models that analyze the different parts of the call, or are you kind of just saying we have one big model that has enough complexity to handle all those things?

Yeah, that's a great question, and it's a nice segue into a new product that we're working on at Invoca.

Oh, it was perfect then. Well done, Chris. Yeah.

So just to quickly answer your question about one model or multiple models, another thing that's a little bit tricky about working on phone calls in our industry is that we have customers that might work in the healthcare industry or in the financial services industry or banking industries where they're really, really concerned about privacy. And so we have a policy with signal AI where there's really one model per customer. And so each customer's model is only trained on their own data. So we might do, since we've transcribed millions of phone calls, we might do much better if we could say, take a whole lot of that data and pre train models that then get refined using each customer's data. But we've decided not to do that. And so each customer's model is trained only on their own data. And so sometimes there's not that much data. We might have to make a model with only a few hundred phone calls that has to handle all these different regional variations. And so the way that we try to make that work in practice is partly by recognizing and stripping out hold messages as well as we can. You know, often a hold message essentially plays advertisements and that can sound just like a purchase and that actually can be really confusing to our models. So we try to strip that stuff out as well as we can. And then once the recorded stuff is removed, then we try to make our models really, really parsimonious. And so they only include words and phrases into their predictions if there's solid statistical evidence that they really matter. I don't know if that answered your question.

Yeah, no, it did. And I'm kind of curious a little bit more on that, given the sort of privacy things that you're dealing with and all of that stuff. And your limitation to have this model per customer, like in your experience, how much data is 100 calls enough to create something useful or how much is needed? And then I'm assuming, are you kind of updating this over time? Is there some sort of human in the loop element from the client who's able to kind of help fine tune things over time?

Yeah, absolutely. So a customer can upload some data and train their own model. Another tricky thing about our industry is, or at least in Voca's models, is we have that. We decided to make them totally self serve. So our models are not actually trained by data scientists, they're trained by the customer. And so we really have to have to make them as bulletproof as we can. But a customer can upload some data and get their model up and running. And when their model makes predictions, every call that comes through in VOCA gets marked now as true or false, let's say for appointments. And then the customer has the opportunity to listen to some calls and correct them. If we mark a call as true, customer can listen to that and say, no, that wasn't an appointment. And they give it a little thumbs down. And then through the thumbs up and thumbs down gestures, we retrain the model and it should get better over time. There's also opportunities to collect more data and upload it. Let's say as time moves on, your business grows, the conversation drifts or changes over time. You can upload new data and update the model that way. So in practice, I think 100 calls, it's awfully hard to train a model on 100 calls. That's also often what people are willing to give us at first. It's a hard sell to say you can have this great model if you listen to a thousand calls and label them.

Yeah, who wants to do that?

Exactly. So even though that would make my job a lot easier, ultimately we serve the customers and we have to make do with the amount of work that they're willing to do.

Have you heard of our newest show called Brain Science? Yes, Brain Science. It's a different kind of show. I know, and it's probably one of the ones that reaches the furthest out from our typical listener audience. But this podcast is what we call for the curious. And what's cool about this show is we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and pretty much what it means to be human. If you've ever thought about why you do what you do or why otherwise others do what they do, then this show is for you@the changelog.com brain science to listen, subscribe, and learn more about this awesome show. Here's a preview of a recent episode called One Small act of Kindness, talking about empathy and mirror neurons. So it sounds like pliability and flexibility is a pretty crucial role too, in relationships. Because if you're not flexible, bendable, pliable, whatever, however you want to phrase that, if you're rigid, it's going to be difficult for you to flex to enable change or to what you've said before, recalculate, accept new data, analyze that data, make a new plan, and iterate towards a new action.

Yeah, one of the other things involved with this Flexibility would be what researchers have discovered as mirror neurons. And so mirror neurons are these neurons within the brain that help us sort of get access to another person's emotional experience. And so there's an action component in it that it was first discovered actually with monkeys, and this sort of mimicry that occurred by watching somebody else do an action. Well, in the same way I can sort of watch somebody else walk through something in terms of an emotional experience, and if I'm holding space for them in my mind, like my body, physiologically, these mirror neurons come come to play.

Is that why people cry when they watch movies or certain movies? Because their mirror neurons are firing? Because they're watching somebody go through a situation and they're empathizing with them and can't help but encapsulate themselves into their scenario and feel what they're feeling?

Is that why.

Yes.

Okay, so is that why anybody cries at anything when it's like, say, movie related? Because they're. That's what's happening.

Yeah. Think about it. Sort of like this emotional contagion.

Right.

So that's interesting to put it that way. We've said mirror neurons several times, but this emotional. Emotional contagion, I believe is actually a better subtitle for mirror neurons.

Yeah.

And so some of this emotional contagion or mirror neurons, like the research, has been rooted in aspects of pain. Because if I can recognize sort of the suffering of another.

All right, to keep listening, head to changelow.com brainscience9 that will take you to the episode titled One Small act of Kindness. Marielle and I dig into this thing called empathy. As a construct, we ask questions like what key brain structures are involved? How can we better understand empathy to be able to better navigate ourselves and our relationships with others, both at home and in the workplace? It's a deep subject, a very fun subject. Again, changelog.com brain science/nine or search for brain Scroll Science on your favorite podcast app and subscribe. We'd love to have you as a listener.

So, Micah, a moment ago you were talking about the fact that you have this customer self service part of your process, and I found that interesting. It's a. It's an interesting choice that you made there and not something we hear all the time. And I was really wondering if you would kind of take a moment and tell us a little bit about how you've made that work for you, what that customer experience is like for them to do it, and the fact that they can update and, you know, what does it mean that they can update their model.

From your perspective, when a customer signs up with us, they go through a process of determining what outcomes on calls are important for their business, whether it's appointments and cancellations or purchases and new customers. And then they curate examples of training data that they then upload. And once that happens, our models train on that data kind of behind the scenes. We typically have a few different varieties that we try out and train, and then we measure how well each of these models is performing.

So as they upload the data itself, are they able to initiate training, or is that essentially just kind of flagging that there's new data there and you or somebody else on the team would then address the data and prep the data and get it ready? I guess. How automated is it between the point where they're offering the data and you're actually training on the data? I was just curious, you know, in terms of. It seems like there's a whole bunch of tasks that would have to happen in there.

Yeah. From my perspective on the data science team, it's totally automated, so we don't initiate anything or, okay, anything. So the customer uploads the data, and the models train automatically. We do model selection automatically, choose the best performing one, and then offer that to the customer with some summary statistics. You know, this is how accurate the model is. If you're. Whether you're sensitive to false positives or false negatives, you know, these are the relevant criteria for you. And then once the customer sees this summary of the model, they can decide whether to go live with it or to wait and get more data to try to improve it.

So I'm curious, you know, coming from, I guess, a more. Less speech perspective and more of a text perspective. I know that sometimes with, like, text documents from, like, chatbots or assistants and that sort of thing, you can do kind of unsupervised topic modeling and determine, like, what, you know, computationally, what are the topics that the model is kind of finding on its own, you know, and there's a few different ways to do that, But I was wondering if you've explored those sorts of ways to ease the burden on the customer to do the labeling and that sort of thing, or maybe suggest to them, like, oh, it seems like there's different things happening here than over here in your data. Have you considered these things and being kind of proactive in that way?

Yeah, it's like you read my mind. So the signal AI process, where a customer determines what's useful to them, labels data, uploads it, and gets A model. That process works well for a lot of people, but we always want to understand what's difficult or what are the pain points for customers. And two of the main questions that people came up with was one, can I avoid having to label data myself? And two, the signal AI, it only tells me what I ask it to look for. You know, I want AI to tell me what I don't know about. I want AI to tell me what I should be looking for. And so we've recently come up with an unsupervised model, exactly like a topic model, as you mentioned, that will chew through calls automatically without any human labels. And it'll just ingest calls from a customer and kind of passively analyze them and look for recurring themes that take place on the calls, and then present those as kind of a summary to the customer. Like, hey, we analyzed 20,000 calls, and this is what people talk about when they call your business.

So, first of all, you're right in that basically every time I go into a project, I don't want to label data. And I'm saying, do I really have to label data? Most of the time in my projects, the answer is always yes. But the one thing I found, as I've kind of tried some of these unsupervised methods, it may be hard to kind of understand what the clusters or what the topics correspond to, even though it, you know, computationally, they're distinct. Has that been a problem as you've tried to develop these methods? And how have you kind of faced some of those challenges associated with unsupervised methods in terms of, you know, how many topics do I consider and how do I represent this to the user and that sort of thing?

Yeah, this really sent me down a road where I did a lot of math. So I think it's an interesting case that often you can run an unsupervised algorithm. And just to be clear, an unsupervised algorithm is one where no human is provided labels or provided any indication of what the model should be looking for. Model just chews through raw data and looks for patterns to identify. Often, you know, these unsupervised models, they can find patterns that make sense to a computer, and they're useful for prediction, and they're useful for clustering and all this sort of stuff. But if a human looks at what the patterns are, they're completely unintelligible. They make sense to the model, they make sense to a computer. But as a human, you can't make sense of them. And so, you know, we tried to design an unsupervised algorithm that would be where every element of it would be human interpretable. And we did that by imposing a lot of restrictions on the model. And so we looked at some linguistic features that seemed important for phone calls, and we made sure that we had a mathematical model that could reproduce those features. And then we put those constraints on our model at essentially every level.

So that's kind of like you, like you did have labeled data and you tried to recreate your labeled data with the unsupervised method to validate that you're creating clusters or topics that actually made sense. Is that the kind of strategy?

No. So there are no labels that are made by humans, but there are statistical properties of language.

Oh, I see.

Really important.

Could you guys, for just a sec, could you kind of address what may be an example of what a couple of those might be in terms of the characteristics you're looking for there?

Yeah. So are you guys familiar with Zipf's law?

So I'm guessing that you should probably just give a brief definition.

Yeah, I think this is fascinating. So Zipp's Law, it turns out that if you look at a big corpus of text and you count up the occurrences of every unique word that appears in the text, and then you sort them so you know how often the most common word is, and that word's typically D in written text, it's not in phone calls. So you take the most common word and you see how often that appears. The second most common word will appear about half as often as the most common word. And the tenth most common word will appear about a tenth as often as the most common word. And the hundredth most common word will appear about a hundredth as often as the most common word, and so on. And so language really sorts itself out into this interesting relationship, and that's called Zip's Law. And I think that's important. Well, for two reasons. One is that on the Wikipedia page for ZipSWA, there's this amazing plot where someone has analyzed all of the English language Wikipedia pages and they've made this plot or English. And of course it follows this relationship because that's how the relationship was discovered. But then they went and did it for Spanish and for French and German and Croatian, and I think they did it for 30 different languages. And they're, you know, Latin based languages and Germanic languages and Asian languages. And amazingly, all of them obey this relationship. And so there seems to be something really fundamental about human communication that creates this Behavior where there's some words that are really common and they appear all the time, and then there's a long, long tail of rare words. And the rare words tend to be really specific words that have a lot of meaning. And so language tends to sort itself out into this relationship. I don't think we fully understand what creates it, but it is clear that it's a common property across language. And so that's an example of something that we really wanted our model to be able to recreate. And it's also interesting because a lot of statistical models that are kind of implicitly assumed in some common machine learning algorithms are kind of under the hood. They assume that distributions are roughly Gaussian ish, that things roughly follow the normal distribution. And this distribution of words is completely unlike that. It breaks that assumption really, really violently.

And so with these patterns that, you know should be exhibited in language. So let's say that you have these 20,000 phone calls like you mentioned, and you could do some sort of topic modeling to break them up into however many topics. Let's say there's five topics, I don't know as an example. Would you then impose that sort of test on each of those clusters of calls on the transcripts to see if they follow that? Or how does that sort of test fit into the sort of clustering or the breaking up of topics?

Yeah, so the way that our model is laid out, and again, it's designed so that not only a computer can understand it, but it's designed so that a human could look into the model at any piece of it and try to be able to make sense of it. We sort of laid it out hierarchically. So if you think at the very bottom we have the data, all of the phone calls, and the phone calls have all these properties that make them hard to analyze. So the word choice follows Zip's law, this funny distribution, and there's all this uniqueness and dialect and all that. So the phone calls at the bottom are really messy. And then we want to move up through layers of abstraction. And so, you know, at the very top of the model, well, we want to, we want to say this is what has to be common across all of the calls in the data set. And, you know, if they're speaking English, that's going to be the dictionary. It's going to be the list of words that can be spoken. And that, you know, the dictionary may be an infinitely long list because people can invent new words all the time, but it's at least accountably infinite. List. So that's the stuff that's common across all of the calls in the data set. And then our model works by kind of starting from the dictionary at the top and moving through these layers of specialization. And so we have this list of words that make up the dictionary. And then we know while we're speaking language, and language follows this zip's law, where there's some really common words and some really rare words, but that are still important. And so we have to derive a probability for each word that follows this power law relationship. And so that gives us, I call it a lexicon, but it's essentially the words with their associated probabilities. And then where the topic modeling comes in is we say, okay, well, the words, the dictionary defines the language you're speaking. That lexicon now defines kind of what an average phone call looks like. And so that's going to vary from one customer to the next or from one industry to the next. You're going to use different words if you're calling a car dealership or a hospital. But then that lexicon, it's still too abstract. It's just telling you what an average call looks like. That lexicon is going to go through, again, several layers of specialization, and each of those are going to be these power law type distributions. But it's going to split that lexicon into, well, these are the words that you use if you're making an appointment or if you're canceling an appointment, or if you're a new customer who's enrolling for the first time or so on.

Gotcha. So you have the commonalities of the language and how words are used in the language. It sounds like your goal in developing the unsupervised method is then to take things those steps lower, where what you're really trying to develop are those probabilities of, or the power laws associated with the words associated with reservation or the words associated with purchase or with making an appointment or whatever the thing is. So you have these sets of calls that have a certain more distinct statistical relationship amongst themselves that allow you to separate them out. That's kind of the goal that I'm picking up on. Did I say that right?

Yeah, no, that's exactly right. We want our topics to kind of float above the data, the topics, because all calls are going to be different and everyone has their own dialect and their own individual problems. So none of the conversations are going to be the same. We want the topics to be something that kind of floats above that data set and represents the themes that are consistent throughout it. And we do this through this sort of process of hierarchically splitting the dictionary into these probability distributions. And then to try to make sure that our models are interpretable, we don't want to fit those topics directly to the data. So we take the lexicon and we split it into the topics that you talk about. And then each topic we say, well, you know, this is a phone call. You've got a caller and an agent and they have totally different roles in the conversation. And so if I'm calling to make an appointment, well, the words that I say are going to be totally different from the words that the agent says. And so we go through another exercise of the splitting and then to handle individual dialects and regional variations across the country. We also actually take a really extreme step where all of those caller side topics are then split and every caller gets their own personal variant of each topic.

Gotcha.

So that's the process by which we're looking for patterns that are consistent all the way across the data set. And so we've got the data at the bottom that's totally messy and totally idiosyncratic and has all this rich interesting information, but it's too much information. And then at the top we've got the dictionary, which is kind of uninteresting, we can all agree on it, but there's not that much information there. And then we go through these successive layers of specialization to get these two ends to meet. And then at the middle, that's where we have the topics and that's what we present to the customer. And so by doing that and by tuning the math pretty carefully to match this fibs law, I think we made something where we end up with really interpretable results.

So I guess as we wind up here, what are you excited about in terms of the future of unsupervised language methods? Methods and AI and conversational data or other related topics. What does it look like for you going forward?

Well, something that I'm actually really interested in now, you guys mentioned earlier, the possibility of having some labeled data and using that to inform the topic model. That's actually something we're working on prototyping right now and I'm really excited about. And that's a case where we could have a semi supervised table model where, you know, let's say going back to the signal AI problem, if someone says appointment set is important for me, but I'm only going to give you 100 calls, I may have 100,000 calls that are unlabeled and so designing a model that semi supervised that can accept some labeled calls and some unlabeled calls. Use the labels to inform the topics so that I get not just the words and phrases but all of the conversation topics that are relevant to this and using that to make a more powerful predictive model. That's the direction that we're headed at the moment.

Cool. That sounds exciting and something that I could talk with you about for many hours, but unfortunately our time has come to an end. Hopefully we can stay connected and keep the conversation going, but really appreciate you joining us to talk about this stuff and chat about how to develop insights from speech and conversational data. It was really interesting and appreciate you joining us. Mike.

Well, thanks so much. I really enjoyed it.

Thanks for listening to this episode of Practical AI. Do you know someone who's trying to break into the AI game? Send them our way. We appreciate every recommendation because word of mouth is how people hear about podcasts them a quick email or a slack message, put out a tweet, whatever, or get crazy, get up from your desk, walk across the room and tell them in real life, hey. Might start a good conversation. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo with music by the oh so mysterious Brake Master Cylinder. And we're brought to you by awesome sponsors. Support them. They support the show. We've got fastly on bandwidth, Linode on hosting and roll bar on bugs. If you haven't yet, hit up our master feed your neural networks. Well thank you. It's all change Dog podcasts in one easy subscription. Get it for the price of a free hot dog. Thanks again for listening. We'll talk to you next week.