The Promise and Perils of Voice Tech in EdTech

Welcome to the future of education. I'm intrigued by the conversation, nervous about what I might say today in the conversation with Patricia Scanlon over from Ireland. So we're going cross country or different countries today. But the conversation, she's the CEO and founder of Soapbox Labs, which has done some really important work around voice recognition for children. And as one with two six year olds in the house who talked to Siri and other devices in a variety of ways, I know the importance of tailoring to that age and making sure that you build tools that can understand them and so forth. And Patricia, you're obviously a world expert in thinking about this, but before we talk about Soapbox Labs and all the things you're doing in education technology right now, for those watching and listening, just personally, how did you get into this field of voice technology and working with children specifically?

Yeah, sure. And thanks for having me on the show. Delighted to get to talk to you today. My own history, I'm an engineer, actually by training. I ended up moving very quickly into research and doing a PhD in speech recognition. So I was working, started in Ireland over in Columbia University, then after the PhD, as it ended, I was in IBM research in New York and then in Bell Labs for seven years after my PhD. So I spent my whole career in the area of speech recognition. And then back in 2013 observing my own daughter, actually working with educational apps and services and just, you know, trying to find good content, good educational content, not just, you know, leave my guilt by giving her something quality. And just these ones that were teaching the emergent readers, you know, teaching the letter sounds, phonics blending, decoding, all this. And I really, really like what they were trying to do. And I liked that they were giving feedback to the parents about the progress of the child. And after a while I was getting these email updates thinking this is real quality. And I started, you know, quizzing my daughter about, you know, what's this sound and how do you blend that? Like, and she's like, oh. So when I was like, okay, weird, let's look at this a little closer, watching her to play with it. These apps are amazing at teaching. They're amazing at, like her passively looking. But when it came to the actual assessment piece, they were able to assess her passive but not her expressive skills. And it couldn't tell whether she had the ability to recall, couldn't tell whether she could pronounce because it was giving her multiple choice. And if she got it wrong the first time, she just figured the little monster, the little character she'd just do the opposite, like, and she was just playing. She just wanted to get the little outfits and little prizes and whatever. It was gone. It's cool and it's so close, but it's missing a very fundamental piece. And after spending my whole career in speech recognition, artificial intelligence, you know, it was very obvious to me what was missing. And it was that voice technology piece to listen to the child. So if you're going to truly have that personalized learning experience, if you're talking about reading or language learning, you need voice. So that was an engineer looking at a problem with my own daughter and realizing this is a gap. And I needed to take some time to figure out why there was a gap. So it was 2013, right. You know, we had Siri at the time, like, you know, we had, you.

Know, you could, you could feel Android Assistant and.

Yeah, yeah, you could feel where it was going. Right. We knew that this voice was beginning to take off and then it's like, well, why is it not here for kids? Like, what is the issue? You know, what's holding this back? Because it seems so obvious and it's not. It wasn't just obvious to me. Right. You know, things has been, you know, voice technology, speech recognition, has been around for 50, 60 years. People have tried this before using voice for reading and literacy and language learning, but it wasn't in any products. So my first step was to figure that bit out. Right. You know, and as a researcher, first was like, okay, we got to take a step back here and figure out what the problem is, because there's a massive gap and nobody's focused. And that was in 2013 when I founded the company.

Yeah. So let's dive a little deeper before we get into the company actually on what that problem was, because from a disruptive innovation perspective, I think it makes all the sense in the world that voice technology started in super simple applications, right. Of do you want one for an operator or two for pay your bill online and super simple things like that. And then obviously it's gotten a lot more complicated. Your insight, though, is that children, their speech patterns are actually very different from adults in the population. And so you need a very different solution accordingly. Can you dimensionalize that problem a little bit more for us?

Yeah, absolutely. So, you know, I'm a signal processing engineer. My history is in the audio signals and stuff. One of the first things I was really interested in is the kids voices are physically very different, Right. So they've got shorter, thinner vocal tracks, they've got smaller Vocal folds, They've got an immature larynx. And that's actually going to create a completely different audio signal than an adult or even an older child. Right. As the child gets older, these voice signals begin to change as they mature and they grow. So what we were seeing was initially of the bath floats. Wow, they're physically so different. And then you kind of really think, and anybody that's got kids, you know, it's not just the physical thing that's different. Their language is different. Right. So how a child of three or four speaks versus six year old, an eight year old, ten year old, as they develop, their language develops and it develops very rapidly and their behaviors. Right. Third, which is a huge factor as well, is that kids don't behave like adults. So if you want to engage with Alexa or Siri or Google, you'll, you'll straighten yourself up, you'll think about it and you'll go, okay, learned behavior. Now I know what I say need to say to get the best possible response. Kids don't think like that. So outside of, I would want to say a handful of very typical requests that kids give, these systems start to not work when they say unpredictable things or a little bit harder or outside of that, more, you know, that safe, predictable turn on timer music and a couple of jokes and stuff like that. So what we're beginning to realize was actually kids are so fundamentally different. People for years have been thinking about it. So we'll build our models for adults. We'll model adult behaviors and we'll model their language. Oh yeah, kids, right. So we'll do a side of kids. We'll just, you know, and, and it'll work, right? Like, you know, like I said, it works in very limited, predictable ways. But the minute you break outside of that, and most importantly for education, that doesn't fly, right? So because we're not. Education is not about getting to turn on a timer. This is about, did they read well? What was their fluency? What was their pronunciation? There's so many more components to this, and the bar is so much higher in education. You know, you don't get away with a canned response in education. You're going to lose people, you're going to lose the teachers, the parents, the kids. You know, the way we felt, there was a weight in this, right? This isn't something you treat trivially. This is something that was going to take laser focus because kids are very different. It's not just something simple. It's not another accent. You know, this is, this is something that we felt really, you know, deeply. We need to build it from the ground up. We need to build a proprietary system just for kids.

So let's get into that.

Sure, yeah.

So let's get into that because then you built Soapbox Labs and I'll put the site up there for people to take a look at and check out all the stuff that you are doing and have built in partnerships. But talk about the solution then that you built to start two. Solve that problem, you said. And I'll. Before you do, I'll insert just a little bit of humor, I think, which is my kids, when they talk to Siri now, and they're often asking me to call my mother. I love how they still say please to Siri. Right. Siri cares about the manners, it turns out, or not at all. But so it's wonderful. But I'm curious. So you saw this problem, you started to dimensionalize it and it is very different from adults. You're right. It's not sort of a use case. It's a very different system, if you will.

I like the way you put that, actually. Yes. It's not a use case. That's good.

So talk about what Soapbox Labs is. Yeah, right.

So we decided very early on, I mean, you know, I think some people have attempted this before and what they've done is to try and say, okay, we're going to tackle voice tech and we're going to build a product and we're going to put Voicetech in and we're going to sell the product. And what we very much felt like, this is a huge problem. You know, this is going to take a lot of time, many years, a huge amount of expertise to pull this off. So we're not going to even try and do the content piece, the education, the pedagogy, all that. There's so many amazing providers in the US across the globe who already do that. Why don't we stick with what we're good at? You know, what our expertise is, which is voice technology, that speech recognition and machine learning and AI piece and partner. Right. So we license our voice tech to third parties to integrate into their products and services. So we're like the oil and the engine. But what we really, from the get go, our mission was very much that we didn't want to just build a mediocre product. There was no potential point. This was education, it was kids and it was way too important. So we wanted to build tech that's state of the art. Right. So what we started looking at, well, what is the problem with the systems that are out there? You know, the accuracy, basically. Right. That was very fundamentally it. The accuracy for young kids. Right. So even when you're going down to age 2 and 3, you know, the accuracy was just getting abysmal. It was tanking. When you got kids under the age of 12 just started to decrease. Other things, we were hearing problems. We were. Having said, it started to break down in classrooms. You start putting it to the classroom, it didn't work. Or it needed a very expensive headset mic to kind of cancel out the noise. If you moved it to the home, it wouldn't work. You know, it was creating way too many false positives and false negatives. False positives. You tell a kid they're right when they're wrong. Right. That's not education. You're reinforcing errors. Right. No.

Right. That screws up the whole feedback loop, which is so important to good learning. Exactly.

Because. Because once people are over the novelty of voice tech, oh, it's cool. This is cool technology with voice tech. And then they start realizing, oh, hang on a second, tell my kid they're the right one, the wrong. Or the teacher's looking at them like, you know, or you tell them they're, they're, they're wrong when they're actually right. Then it's confidence thing. Like, you know, when kids, I don't know, your six year olds, you know.

I've got, I mean, and they call.

Right.

I mean, you think about the numbers of teachers that even when we were growing up did something like that and you sort of, you lost faith. Right. As a result. And if you lose faith in something that's scalable, that's catastrophic.

Yeah. And that is what happened on repeat, I think, in the industry. Like I said, it wasn't like people haven't thought about voicetech long before, thought it was a missing piece. It just didn't exist at the point where it was usable and scalable and deployable and it had utility and really to address the problems that teachers are having in the classroom or parents are having or kids are having. So all those things to us meant we needed to build that solution from the ground up. It needs to be stated by, it needs to be deep tech, as we call it. So we set ourselves like, you know, fairly high bar. Like, no, we were going to go young, we were going to go two, three, up to 12. We've spent seven years now. We have an amazing team of experts. We've over 130 years of speech recognition expertise and computational linguist expertise in the company. And that is just taking that dive. Not being tempted to take a mediocre solution to market too quickly. Because that's what can happen in AI or any machine learning system. You can take it to market whenever you want. Like it'll be ready at some point, but should you? And I think that's a very interesting problem with in all AI, because people love to love AI and they love the novelty of it. But, you know, you need to hold back and you need to invest a little bit deeper, go deep and make sure we're getting that quality of experience that is on par with what we're seeing with human evaluators. And that was really, really key. So we decided to take the time we really wanted it to work in real world noise environments. I know I was looking at my daughter at the time, who was not even four and I was really observed. She is never on her own. She's not going to take an iPad and go into a room on her own. She's going to sit in the room with us with the tv, with everyone else around, conversations, siblings, dogs in classrooms. Again, they're not on their own. Like, you know, you can maybe move them a little bit into a quieter corner of the room, but they don't sit in quiet rooms, so you have to accommodate all that. And a five, six, seven, eight year old, you put a headset mic in them, they're going to play with it, they're going to break it, they're going to, you know, we wanted to work naturally and be a real state of the art solution. And really importantly was one of our missions was this will be scalable, right? This will be cost effective and scalable. To be honestly, truly disruptive, you weren't going to get to do that, right? So the minute you start making it like, you know, bespoke and really time consuming to choose, tuned to a new client, we were going to make it that couldn't scale quickly to tens and hundreds and thousands of kids. You were going to make it unavailable, you were not going to get widespread deployment. And that was kind of against the mission of the company really. This is like, can we build tech? We're not going to just put in one product. We want to drive products across, you know, literacy, language, learning, screeners, you know, all kinds of assessments. There was so much we could do with assessments, technology, but we built it right. And that was really, really important. So we took the time to do that.

Yeah. So it's interesting. I mean, there seems to be a Little bit of a principle of universal design behind what you just described because it's really some of those use cases you were describing where there's noise in the background, it's a kid talking and competing inputs into the audio. Right. Those are the hardest cases I imagine to build really good fidelity around. And that's where you started. Which then I imagine when you start in the hardest case, it's a lot easier to extend it to other cases. Right. In this particular instance, I'm curious, is the basic strategy then for soapbox labs as you now, you know, you, you sort of built in some ways a Bell Labs or an IBM research laboratory. Voice recognition for kids, it sounds like, is the strategy to really partner with all the education technology providers out there so that you are the voice tech inside of their solutions or are you doing anything that goes direct to parents? How does that work?

No, what you said, we don't go direct to parents, we don't go direct to schools or districts. We want to work with the experts. And that's actually been really enjoyable because you get to power so many different products across a range of different industries. Like so, you know, speech therapy, dyslexia, screening practice. Right. Reading practice, fun, engaging reading practice, language learning practice, you know, kids learning English all over the globe. You get to do it in formative assessment, summative assessment. You know, we kind of felt that we wanted to be able to touch all parts of the child's reading journey. Right. So when you're learning phonics, right. My daughter would say that stage he was at blending, decoding, learning words, sentences, then moving up into fluency and comprehension and all that is possible with voice technology to be able to power. So what you're doing is, what we're trying to do is create a scenario where all we were hearing from teachers were that we'd love to be able to assess kids more, we'd love to be able to get like weekly updates on, you know, you've got 25, 30 kids more in the classroom, they don't have the time to sit with them. And all the research is telling them frequent assessment, frequent practice, you know, early screening, every. The teachers know the research and they know what they need, they, they would like to do to really progress kids literacy. But when you don't have the resources, when the resources all require an expert, right, A human evaluation, a human teacher to sit with the kid, listen as they're practicing, listen as they read aloud and they need to prompt, courage, encourage, correct, you know, all those things, whether they're assessments, whether they're screening, whatever, just, it requires human power. And that is just, it's not scalable. And what we really felt like very deeply that if we could just help teachers, help parents, let the system listen to the child. Right. And let the system prompt, encourage, correct and invisibly assess all the while that you take the stress out of the situation as well. The kids don't know they've been assessed. And if you made that playful and you made that engaging, like good content, engaging content, the kid is practicing because it's fun. Right. And they love using their voice. Kids love using their voice to engage.

Sure. They want to talk, right? Yeah, they do.

Right. They love hearing themselves as well. Right. But they, they, they could engage and we're getting this amazing data out of it that they don't even know they're being assessed. And the teacher can be surface to the teacher, it could be surface to, to the parent. And interventions can happen quicker, you know, so they say it takes four times longer to intervene with a child at age eight than it does with a four or five year old. You know, and what that does to a child in those four years, like, you know, if you could get them early and catch them up with their peers, you know, there's, there's great impact in this as well, like as well as being, you know, tool for their outcomes for the kids. Outcomes importantly.

Well, so I'm curious about that because one of your partners is Amplify. And they originally got their start as Wireless Generation was the original company before they went through a variety of changes and names and so forth. But their original product is Wireless Generation was this simple handheld device that teachers would listen to students reading and figure out their ability and figure out where are they making mistakes so that they could identify the weaknesses in their phonemic awareness, phonetics, whatever it might be. And it sounds like in some ways you are now I'm projecting and assuming here, but perhaps you're actually powering an automated solution for Amplify to be able to do that sort of work. Is that where you guys are on some of that?

It's been a really interesting relationship with Amplify. I love that story that those guys brought the Pampilot and they took it from the egg timer and the pen and paper to something a lot more quicker and made it easier on the teachers. And we feel that this is another massive, this is the next step change in that area. And we this great plan with Amplify. And when Covid happened, the pandemic happened. They Just really wanted to do something to help the teachers get back in September. So we accelerated, you know, progress got. Got in. So when kids now come back, came back after the summer holidays and an extended break, like. So it wasn't just the summer reading slide, people see, it was an extended period of time where kids were learning at home. But we all struggled. But they weren't getting the same education because most parents, you know, were, you know, not experts, didn't know what to be doing. So when they got back, the kids were being placed. So what they would do is they'd read a passage, our system would assess, and then it would give them a harder or easier passage. And it kind of just nicely leveled each of the kids for the teachers. And the teachers could see this. This information was surfaced to the teachers so they could quickly go through each one. They could listen back to the. The minute of audio, let's say, or whatever the passage length the children were reading, and they could quickly go in and see the errors. And then that gave, you know, the idea behind that is to give them reassurance as well about what the system is doing. It's not blind. It's like they can actually go in and listen to the whole audio or just the errors that have been surfaced by the system. And that's a game changer because that's like the first step in what we can do with this. This is like I was saying, you can build these in really regular assessments. You can do it in a practice. You can assess while the kids practicing. Doesn't have to be a separate activity. You can screen with this. You can do. There's so much more. It's just like you feel like it's just taking off now. And the possibilities here.

Well, and that piece that you just said of being able to assess in practice. So assessment is. And feedback is interwoven with the actual work as opposed to being a separate thing. That's sort of the golden ticket, if you will, to creating really short feedback cycles so that children are immediately getting the important feedback and improving against it as they get those metrics themselves. One thing I'm curious about is. And this may be a naive. Well, this is certainly a naive question, but I'm curious, how does this differ from the other approaches that are out there around voice technology? I mean, we see a lot of hype right now around having Alexas in school classrooms. Siri obviously is out there. Google has its answer. And then there's some education technology applications out there that use voice recognition, some of which I'M sure powered by Soapbox Labs, but some of which I'm sure are, you know, on AWS or some other engine, you can choose to throw people under the bus or not. Totally your call. But I'm just curious, like, how does it differ?

We like to lift the industry.

Yeah. How does it differ?

Anybody using voice is. This is a game changer, Right. So people are trying to find their feet in it. I think one of the things I've noticed is that what we're doing is voice technology. Right. What Alexa and Siri and Google are voice assistants. They're built to service a very particular need, which is to be a virtual assistant. They will get some answers and there are some skills, but they're not designed to do pronunciation assessment, they're not designed to do a fluency assessment, they're not designed to do comprehension questions. They're not customizable to different features, they're not customizable to different content. Like, you know, so they are great for asking general knowledge questions. And there's loads of cool skills. I actually think that are, you know, helpful. And I think anything that gets kids more engaged and using their voice will get them more engaged. But, you know, from my point of view as an engineer, when I started with this, voice technology is more than that. So when we looked at this 20 years ago when I was starting, you know, we were thought about voice assistants, you know, the sci Fi movies and all this, you know, voice assistants were a narrow use case to us. And I find really interesting, 20 years on that it's actually the voice assistants that broke open the market, right. A normalized voice, which is great. Like, that's really good for us. Because when we were Starting out in 2013, we were definitely having to educate people why voice would be important for education.

Now it's an important point, right? Because I think sometimes people look at voice and education and they're like, that sounds like a nice gadget or gimmick or nice to have. But do I. I mean, if I have written and a teacher's eyes and ears, do I really need it? You've made a compelling case this entire paw, you know, this entire conversation we've had about why voice and where voice can make such a difference. That's not at all gimmicky. It's core to learning.

Yeah. Because it, because voice is the natural interface. Right. And I think that's really key. So when people think voice and they think Alexa, that's, to me, that's, you know, it kind of jars a little because no, voice is what you and me are communicating. You know, there's a bit of visual here, there's a bit of context, but mainly we're communicating through voice. So if you strip away the assistant point of it, voice is what we will be using very shortly to replace keyboards, you know, mouse, swipe, touch. Because we've had to invent all these, you know, technologies hacks if you want to interface with machines because machines couldn't hear us. You know, as voice accuracy improves over time, it makes sense that we should use our voice more and more. And I'm not saying that a programmer or somebody who is a typist is going to sit there and like talk for eight hours, days. Maybe they will. But most of us, I mean 95% of the population and more probably use technology and its touch points through the day. And voice makes more sense. So if you kind of unlock that we've been seeing, you know, particularly for young kids who can't, who pre literate, and then you're talking about buttons on screens and menu systems like, you know, so then people are trying to design around the fact that the kid can't read and they're designing around the fact that, you know, that they can't handle the complex menu system. Voice just cuts through all that. And what you can do then is start having educational conversations with kids. What's two plus two? You know, how many legs does a horse have? Like, you know, you can introduce vocabulary, you can introduce questions. They can listen to a story. You could ask and you know, assess their understanding of did they listen correctly. Like you can do socio, social, emotional learning kind of exercise. There's so much more it can unlock in a really natural way with kids. And they say kids are the voice, you know, that generation, they're the voice first generation, right. So they don't have the inhibitions we have around voice. So it can do that very rigorous assessment, be it somewhat of formative, it can do the screening, it can do all those cool things, but, but it can do a lot more. It can kind of unlock interactions.

Yeah, no, that's super helpful way of framing it. So as we wrap up, I have many more questions, but I have two in particular that I'll focus if it's all right, and a bit of a lightning round, I guess, which is first, what are you most concerned about as voice technology spreads in education applications? And then my second question will be the flip side of that. What are you most excited about if you had to sort of do your.

Top two for each of Those concerns are that people will view, like I just said, they'll view voice technology as what they're experiencing, voice assistants and deem it not that useful to them without really understanding that bigger picture. Right. Secondly, that the privacy aspect has to be surfaced as an industry. It has to be surface. People are concerned, concerned when they are getting mixed messages or it's unclear where the data is going, what it's used for, is there competing business models like, you know, is the data secure and safe? And that is transparency is really, really key and you don't want to lose the trust of people. So what we're always advocating is that there is GDP or COPPA on top of that transparency about the data that's only ever used to improve the services, in certain cases not used at all. What we'd like to see and going forward is that more and more of this technology is taken offline, like. And I think that's very possible into the future, really. I mean, I think it wasn't. And everybody has this idea that cloud is king and if you want to do deep learning and this really state of the art technology, you need it in the cloud. That's changing and it's changing so, so quickly, you know, on device.

And that's because you're putting intelligence in the actual devices so that it's local. Is that okay?

Yes. What we're seeing is the processors are faster, they have more memory and the cost is dropping like stone. And what we're going to see is more and more opportunities. Now what you're also going to see is some of the clients, these like smartphones are getting lighter and cheaper, which is great too. And then there's balance. Right. So a lot is going to be done on device. And I think that if you think about all the use cases for a child, how many of them really need open access to the Internet? You know, they're not ordering Ubers or ordering, you know, checking out the football results. Like, no, they are looking at content, having a fun, engaging experience that does not need to always go to the cloud. I think that's where it's going into the future and I think that's where we need to keep an eye on because I think that, you know, that's a really exciting period where we're not going to have to worry about Internet access. We wouldn't have to worry about the privacy aspect like that. That's where, you know.

Yeah, that's huge because. Yeah. I'm a founding board member of a nonprofit called Imagine Worldwide, which is helping stand up autonomous learning solutions in developing countries where kids don't have access to school, period. Right. And so there the devices have to be offline. They charge by, you know, generators that are powered by solar power during the day, they charge during the night. And then you get your device that has to last. Right. For eight hours for multiple kids. And so it has to be offline. So what you just laid out is extraordinarily powerful actually.

Yeah, yeah. And deep learning, it doesn't have to be the. We don't go back 10 years in technology to put on these devices. We should be able to take state of the art that we're experiencing now in the cloud onto a device and cheaply and in a very battery power efficient way and cost efficient way as well. And that's where you get scale. Right? That's disruption. Global disruption.

Yeah. No, that's terrific. Well, look, you're doing a phenomenal set of work to empower children to be able to make progress in their education and not get those false positives or false negatives, which you are right is absolutely critical. But deeply appreciate the time today. Excited to see frankly where Soapbox Labs continues to go and who else you continue to power. And I'll put the site up there so folks can check it out and keep following along your progress. So thanks so much for joining us today.

Yeah, thanks, Michael. Thanks.

And we'll be back next time.