Speech tech and Common Voice at Mozilla

The definition that we've come up with is if you have a speech to text model that performs statistically worse for a certain demographic group, then that model is said to be biased. So using this corpus, the RD bias corpus, you can take any off the shelf speech recognition model and look at what is its accuracy or word error rate as we call it for a certain demographic group. Thanks to Jenny and everybody at Common Voice, we have information from people who opted in to self identify their gender, their accent and their age. So along any of those lines you can look at how well your model performs on every group versus every other group.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar.

Check them out@rollbar. And we're hosted on Linode Cloud Servers.

Head to linode.com changelog.

Linode is our cloud server of choice. Grab the Nanode plan for just $5 a month. Just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD, and one terabyte of transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable so you can put your calculator down. You won't need it. We've been running changelog.com on linode for years and we've always impressed by the re winning support team. Check them out@leno.com changelog Once again, that's leno.com changelog. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're Racticalai FM.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I've got Chris Benson with me who is a principal emerging technology strategist at Lockheed Martin. How you doing Chris?

I'm doing very well today, Daniel. How's it going?

It's going really good. This last week was a fun week. I've been writing a lot of inferencing API code which has been fun. It's been one of those weeks that I've got to be, you know, heads down in code a lot and I always enjoy that bit of it. It's like getting the various pieces plumbed together and working and then you have your models together and you actually see it work in a practical way. That's always fun for me. So what about yourself?

Last week was pretty interesting. I'll relay really quickly. Kind of what for me was a work related thing, but not in a direct way. So darpa, the Defense Advanced Research Projects Agency, had a competition and they had a bunch of autonomy focused companies, including my, just as a. We were there too, Lockheed Martin that were competing in what was called Alpha Dogfight. And so they had gotten, at John Hopkins University, they had gotten an F16 flight simulator and they had all the teams over the last few months getting their models of their F16s ready to go into dogfighting mode. And so they had the competition last week and they live streamed the entire thing on YouTube. So it was pretty cool to watch. And so in the end, a company called Heron Systems won out overall. Our own team came in second out of eight teams. But it was interesting is that Heron actually went up against a. If it was the Navy, it would be a Top Gun instructor. It was the Air Force. The call signs was banger. And within a set of constraints. It was interesting. The AI consistently beat the human pilot instructor.

Interesting.

Yeah, it was. Nobody was expecting that. So the AI models did very, very well. Much better than anyone realized.

When you first said that, I was thinking of like BattleBots, Boston Dynamics, like dogs going against. When you said dog fighting, I guess I'm not around, you know, aeronautics very much. And that seems way more violent.

And yeah, no airplanes in the air. Still quite violent in theory.

But yes, I guess that's true. Yes.

Yeah. But not adorable, furry little creatures, definitely. So. But anyway, it was pretty fascinating to watch. It was a moment where people went, things are definitely changing right now. So that was my week.

Interesting. That brings up all sorts of interesting questions and ethical things and all of that that I'm sure we'll get into as the podcast goes on as well.

I'm sure we will.

But yeah, I'm pretty excited about today's episode, as some of our listeners will know. I think I've mentioned a couple of times that I've been getting into speech technology a little bit more for my own work in recent months. And one of the things, like if you're going to train a speech recognition model and you're looking at what, what speech data is out there, one of the things that pops up fairly, you know, prominently in that is Mozilla's Common V. And Mozilla is doing some really cool things also through their fellowship program with speech Technology and so today we've got Remy Muhiri, Jenny Zhang and Josh Meyer with us who are going to talk a little bit about Common Voice and other things going on through the Mozilla Fellowship program. So welcome everyone. It's really great to have you here.

Thanks, Dan.

Thanks. Nice to be here.

Yeah.

So before we jump into Common Voice and speech technology and all the wonderful things that you're doing, maybe we could start out by just having a brief discussion about your respective backgrounds. Remy, do you want to start us out and tell us a little bit about your background and how you got involved in speech technology?

Thank you, Daniel. Yeah, so I think my background started as a software engineer. So I still actually write codes but not often and also contribute on some of open source library in Africa as well with couple of friends. Yeah, I would say I spent actually six years, this is by years, it's actually writing codes. And the journey actually started in 2013 learning about C Python and JavaScript and been working in actually a couple of startups in Kigali, especially startup Fintech Technologies. And before that how actually getting to voice technology was a project try to actually do with friends. So which were bought actually voice recognition. So how like an app would actually help people around speaking actually their native language is actually seeking for information. Yeah. But unfortunately we couldn't actually move the project since there weren't actually any Kenya, Rwanda data sets. Kenya, Rwanda is one of the local language spoken actually in Rwanda. In Kigali. Yeah. Currently working with Mozilla as a community fellow on voicetech.

Yeah.

Awesome. Yeah, that's great. And I'm really excited to dive into the details of some of that in a bit. Before we do, Jenny, would you let us know the same a little bit about your background?

Yeah, thank you. I'm a software engineer primarily and I have a strong focus in my work on ethical tech, data governance and privacy rights. And so I kind of found my way through Common Voice and open voice tech via those avenues. So currently I lead the engineering work on the Common Voice project, both in terms of the web platform and the data infrastructure around it and kind of help set the technical strategy.

Awesome. Yeah. And then Josh, what's your background?

Yeah, I got into speech technology from linguistics. Actually I was in academia doing more kind of cognitive science and phonetics, phonology, perception, kind of the acoustic side of things. And I started to learn more about computational linguistics and NLP and speech technology and that really sucked me in because that's something where you can see users actually interacting with what you're doing and that was really rewarding. So recently, just last year, I finished my PhD on speech recognition for low resource languages and, and during that, towards the end of the dissertation I was working with a machine learning group at Mozilla doing some things with Deep speech, trying to make it, kind of trying to find an approach that could scale to new languages easily for speech recognition.

Yeah. And Deep Speech, just for those that aren't familiar, that's a project coming out of Mozilla as well.

Yes, it's called Deep Speech, Baidu originally named it. But Deep Speech is an open source speech to text software stack. Includes code for training, code for inference and also pre trained models. And so today I am a fellow at the Mozilla foundation working on this project for African languages and Deep Speech and Common Voice. And also my normal day job is lead scientist for speech technology at a startup in Los Angeles called Arty Inc.

Very cool. I guess to get us started. Jenny, I'm wondering when Common Voice started, what was the state of speech data in general at the time and what were you able to get your hands on? What was the practical aspect of getting started at that point?

So Common Voice started in the summer of 2017. I don't know exactly what the state of open speech data was at the time, but I do know that it was not the norm to have data data sets. So for context, Siri came out in 2011, Alexa came out in 2014. So by mid 2017 there really hadn't been a ton of work on speech recognition and voice technology stacks in general that was available to the general public. The big companies obviously are not particularly interested either in releasing the training data that they have been working on or making those stacks open source. And so there had been a few data sets that have been doing a tremendous amount of work trying to find these open datasets and trying to basically look at large corpuses of voice data like audio, like open source audiobooks and those sorts of things in order to turn dataset that was initially intended for something else for voice recognition. But as is often the case with open source projects, and as is often the case with sort of Internet volunteer labour, it was heavily focused on English, it was not diverse and oftentimes there was not really a good sense of a roadmap to get from the initial 200, 500 hours of English to a place where the speech dataset can actually represent the user base of the Internet more broadly. And so that was kind of the initial impetus for Common Voice. We wanted to experiment with Having something that was crowdsourced and decentralized in the best way and look at how we can really democratize tech and bring people into the fold who were traditionally left out.

Yeah, and Josh, maybe this is a question for you since you're working in your PhD, but in terms of state of the art right now and what people are doing, what is the sort of, for something like speech recognition, how much data or how many hours of speech data do you need? Sort of on general, I know that's difficult because there's accents and gender and it's different for different languages and all of those things. But what sort of scale are we talking about in terms of getting something that's functional for speech recognition? What is the sort of scale in terms of numbers of hours of audio that's transcribed that we need?

Yeah, this is a very good question and it's kind of the million dollar question everybody wants to how much data do you need to make an application that's useful, that's based on speech to text? This is really dependent on the application. You can actually get by in some cases with zero data. I know it sounds kind of crazy, but if you have a really, really low resource language, you can actually take an existing model from a high resource language. You can take an English model, let's say, and basically just transliterate the words that you care about for the target language. And so you can kind of hack an English model to recognize, let's say Welsh. I know people have done this for Welsh, for example, but that usually works only when you have kind of limited domain tasks, when you have a small vocabulary. So if you're, let's say you're trying to recognize numbers, the single digit numbers from 0 to 9, you might be able to do that for a new language just by kind of transliterating what the English model thinks it heard. If you want something that's going to be more open ended, like a really robust speech recognition model that can recognize any phrase, any word in some target language. This is a hard to put a number of hours. People will usually say 2000 hours. But that number comes from some standard English corpora like Liberty Speech and Fisher. But that's a number that's, I think, a pretty safe one to say if you want something that's really open domain for some target language. And you're not assuming that you're going to do some kind of transfer learning from a source language model. I think 2000 hours is something people would say, but it really depends on what your target application is yeah, and.

I know that, you know, Jenny, you were saying that kind of state of speech data, at least how it developed, of course there was a lot of data available for English and maybe other major languages starting out. And I know that it was hard to put a number on that, Josh, but at least gives us a sort of scale. I'm assuming maybe Remy, you could speak to this. But I'm assuming that for many, for example, African languages, we might have kind of zero data available, at least in open data. In some cases we might have some more. In terms of the languages that you're working with, Remy, what is the sort of state of the data?

Okay, so yeah, I think like some of the language we're working on, we're currently working on with Kenya, Rwanda, which is actually a very well documented language. And so Kenya Rwanda actually has like over like 12 million speakers like in. In Rwanda. And to give you just like a background, most of the administration. Administrator, actually administrative government in Kigali. So use actually Kenya, Rwanda as actually language in their daily operations. So giving that like, so plenty of resources like consulting as well, like the Bible, which is the Bible which is actually very well documented in Rwanda as well. So a lot of newspaper around. So I think for Kenya, Rwanda, we have actually a very useful actually text corporate to use with.

So Jenny, I was kind of wondering, like at this point, as we're looking at Common Voice today, kind of what, what is the state of Common Voice currently and what kind of functionality is available in the platform?

Yeah, Common Voice, the platform has the core platform which you see when you go to commonvoice.mozilla.org really focuses on the data collection aspect. So on one hand you go to Common Voice and you click on Speak and you will be given any number of randomly selected sentences from our available text corpus for you to read out loud. And if you would like to validate, you will be given recordings that other contributors have recorded. And you can say this does or does not match the text. And generally speaking, we have a criteria for when we consider a recording to have been sufficiently validated by the community. And so the Common Voice platform is really the first step in the pipeline to getting to our data set. And that's just a web app. It's available on all platforms. There's a little bit of trickiness with Mobile Safari, you know, due to just the vagaries of Media Recorder API implementation, but we're doing our best to make sure it's as widely accessible as possible. And then the Common Voice data set, the Output of all of this volunteer effort. The last dataset we released in June had 7,200 recorded hours in 54 languages, of which about 5,600 have been validated by volunteers. And so when, you know, this is a constantly growing number. So that Data set had 54 languages, but the current common voice corpus overall has 56 languages. And, you know, that number is getting bigger every time. So right now we're looking at approximately, just a little bit under 400,000 unique contributors who have either done recording or validation. And, you know, over 130,000 unique voices in the dataset.

And quick follow up to that. And speaking as the probably the non NLP person on this podcast, when you say validation by volunteers, what exactly does that mean? Just for those of us who aren't familiar with it?

Yeah, so the common voice approach to validation, and Josh can correct me if this is wrong, is a little bit unorthodox when it comes to how we normally think about NLP annotation. So basically, for the purposes of training, you want to make sure that a piece of audio matches the transcribed text as much as possible. You know, without the ums and uhs of the usual speech patterns, without any punctuation or, you know, any of those sorts of things. And normally how that happens is you're given an audio and some the person who was doing the annotation would write down what they thought the audio was, or vice versa. Because all of our contributions are community led and the scale is such that we can't internally as a team, do QA on it, we're really relying on our community's members to listen to the recordings that have come through and say, do I think that this recording exactly matches the sentence that it's supposed to match? So, you know, if the sentence is the quick brown fox jumped over to the lazy dog and. And somebody reads a quick brown fox jumped over the lazy dog, that would be the kind of thing where we would say there is a mismatch between the sentence and the recording that the training model would be confused by. And so this is the kind of thing that we would want to filter out as not matching validation. Exactly. Does that about COVID it, Josh?

Yeah, I would say actually that it's not completely unorthodox. I think that in reality, I think with speech technology and nlp, what's orthodox really just comes down to what the application is and what the constraints are of the project. So there are some folks out there who would say the best corpus of speech for training speech to text models is conversational speech from humans that has then been transcribed. Common voices is a red corpus. In the past there have been red corpora that people have issues with, rightly so, because they're not seen as applicable to most projects. Needs for at inference time. So for example, concrete I'm talking about Libri speech. Libri speech is a corpus of mostly American English that was taken from was formatted from open books that were recorded from Project Gutenberg. So people who like to read books out loud for kind of the general public, a lot of the consumers of these books might be people who are visually impaired. There's this giant project, people read books out loud. And then some researchers, I think mostly from Johns Hopkins, they took this set of audio and they're not actually transcripts, it's the book. And they did some alignment and they used that. And that kind of became a benchmark for doing speech recognition, especially for English. However, the way that people read books out loud, they try to read it in a very quiet room. They try to make sure that there's no echo. They try to make sure there's no dogs barking in the background. That kind of audio is not what most developers are expecting for their applications that use speech to text. It's better to have actually kind of messy audio. It's better to have cars honking in the background and wind blowing and dogs barking, because that's a lot of times what you actually get at runtime at inference. So common voice is read speech. However, it's very valuable in the diversity of voices. It's just an enormous diversity of voices and also the diversity of background noises, the diversity of acoustic environments. You have people who record common voice just for fun on their morning commute when they're on the train. That is really, really valuable data because it's so noisy. So I think that people who might knock on common voice because it's red speech as opposed to conversational are also kind of. I mean, it's valid. People do speak differently. When you're reading something out loud that's linguistically proven, that's not really debatable. But the kind of noise you get in the background is just so valuable that common voice I think is going to be, if it's not already becoming the benchmark for speech recognition for all sorts of languages. Hi, I'm Matt and I'd love to tell you about pace.dev.pace.dev is a minimalist task management and async by default communication tool. Our screen recording feature is actually very popular wherever you can leave a comment, just like how easy it is to upload a file. You can record your window or the entire screen and upload it as a video to the team. Sometimes a screen recording is the perfect way to explain something, you know, whether it's a bug that only happens for you or maybe more optimistically, a new feature that you can't wait to show off. And the showcase feature takes that a step further and lets you highlight progress, which is a much more positive experience than trying to make up estimations out of thin air. So please learn more and start your free trial at pace.dev.

So, Jenny, I'm curious, you know, while we're still on the topic of Common Voice, I was curious, in terms of the Playbook, as you move into the future for expanding the set of languages that are available and maybe involving, you know, actively involving local language communities, what's the sort of playbook looking to the future for Common Voice in terms of expanding that and making sure you have, you know, more and more language communities getting involved?

Yeah, thanks for the question. It's something that the Common Voice team spends a lot of time thinking about because, you know, as many languages as we share between us, we can cover all of the 7,000 languages in the world. And so what we really want to do is make Common Voice as self serve as possible in terms of a platform for really any language community that wants to make use of it. And so the pipeline for getting a new language on Common Voice first starts with being localized. You know, if the Common Voice website can't be used by speakers of a language because all of the website copy is in English and that's not particularly useful. And at the same time that this localization is happening, the thing that we ask language communities to do is really find a corpus of sentences to be read. So our threshold for that is 5,000 sentences. That's kind of what we found sets a language community up for success, both in terms of having enough sentences to generate a sufficiently large corpus to start doing training work with, but also so that people who are coming to the site are not all just reading the same sentences over and over again. And these sentences can be from any source that the community thinks is appropriate, as long as the licensing is open source. So as long as the licensing is CC0, and then once the site has been localized and translated, and once we have the sentences, 5,000 sentences available, then the language gets activated on the Common Voice site and people can just go and go start contributing. And so to that end, we've built several satellite tools for making that happen. There's a tool called Sentence Collector which is primarily currently maintained by a community contributor, Michael Kohler. That is where anybody can go and just if they want to start typing in sentences to submit for a certain language, which will then also go through a community validation process to make sure the sentences are correct and appropriate and you know, all of these things. We also have a tool that scrapes large open source text sources such as Wikipedia in order to, you know, automatically put together a corpus so that you don't have to come up with 5,000 sentences to read. A lot of our largest languages have been compiled by scraping Wikipedia and scraping open source encyclopedia types. Sources, I keep saying sources, but this could also include things like transcripts from the European Parliament or you know, we're really very open to anything that the community deems will help them succeed is something that we're really interested in supporting.

I appreciate that. Hey Remy, I have a question for you. What can technologies like speech recognition enable, for lack of a better word, in local language communities in Africa and elsewhere? And what are the potential applications and are they different than those that we're finding in places in the west, in the US and in Europe and such?

Yeah, I think voice, like voice technology can actually enable like a unique opportunity to reach out to people who have like been excluded to the traditional, like to the traditional from digital services. And yet I think some of the kind of application will be especially fintech application regarding healthcare, just to give you an example. So two years ago I was working at this startup and we were actually building a product for people actually to contribute money into service groups. So those group were formed like with 20 to 30 people. So most of those contributors were actually farmers. And looking at the literacy level like in population in Africa, so you, you actually see that 30% of adults in sub Saharan Africa actually illiterate. And so it was very like difficult for those people actually to use like this application to contribute. And so my thought was like if you could actually have like a Kenya Rwanda language model, which is actually a local language in rwanda spoken with 12, over 12 million people. So this could have actually eased the way those followers contributes on their savings group. Yeah, so there also a lot of innovation happening especially in Rwanda. Most of the government services are being digitalized and so it's not very accessible actually by people, especially in rural areas because someone could actually live very like 10 to 30 kilometers from actually a siber a coffee, not a cyber cafe, Internet cafe. So where actually people go and seek actually for like services and they always go look for agents, agents who actually have them actually seek those government services. So I think, yeah, voice technology can actually enable a lot of solutions and helping actually farmers to actually access digital financial products.

Yeah, yeah. And that's super interesting. I wonder, as a follow up to that, if so the language Kenya Wanda, if people in maybe that language community and others, do you think that they would find more value in a sort of voice application versus a text application in the local language, what do you think provides the most value there?

So like, yeah, just giving you a context. So a lot of people, especially in urban areas, communicates like with voice over WhatsApp, like using WhatsApp. So there they tend to send more voice, voice notes to actually people they're chatting with in the local language. And yeah, and people will actually prefer actually communicating through voice and as well like seeking for information from voice. But just a quick example, so people will tend to call, especially call centers, like telecommunication call centers actually to seek information. So recently we've actually the crisis, there's this institution in Rwanda called Arabic is actually the Rwanda Biomedical Center. So they've been actually receiving over 1000 call like respected in getting information for Covid for the coronavirus. Yeah. And I think people will be seeking more value like with voice than actually texting.

Yeah, yeah, definitely. It seems like there's a lot of great potential there. And I know that of course one of the things like you were talking about some of the data that's available in the language you're working with. And I know Josh, you've worked a little bit with sort of bias and data sets. And it seems like of course, as data scarcity might be a problem with some of these lower resource languages, there's probably a heavy bias in those speech data sets for the lower resource languages towards whatever it is. Maybe it's only males in the corpus or only a very small representation of the person, potential accents in the data. You've done some work in this area. I remember reading a recent blog post about a data set that you're working with to help researchers diagnose bias in speech data sets. Could you give us a little bit of an introduction to that work and what you're doing there?

Yeah, I'd be happy to. I think that this question's a really important one and it's one that anybody working in language technology needs to think about, needs to face. Because whatever language technology you're working with, this bias problem is going to be present. So the work that you're referring to is the RD bias corpus. It's a corpus of English. Right now, it's a subset of the Mozilla Common Voice corpus that has been kind of filtered so that we have demographic data for every audio clip in the speech corpus. And, and also we've done extra steps of validation to really ensure that the transcripts are 100% accurate. So this data set is, it's about 1700 audio clips. So it's not enough for training, but it's enough for diagnosing, as we call it, diagnosing bias in your speech to text model. So this is, it's a hard problem to kind of get at when we talk about bias, to make it really concrete. So the definition that we've basically come up with is if you have a speech to text model that performs statistically worse for a certain demographic group, then that model is said to be biased. So using this corpus, the already biased corpus, you can take any off the shelf speech recognition model and look at what is its accuracy or word error rate as we call it, for a certain demographic group. And we have information, thanks to Jenny and everybody at Common Voice, we have information from people who opted in to self identify their gender, their accent and their age. So yeah, along any of those lines, you can look at how well your model performs on every group versus every other group. And it does. The kind of beginning of your question was this problem with more underserved language communities. This problem becomes more exaggerated if you have a small sample from any distribution that the sample is more likely to be biased, especially if you're collecting it in a kind of biased way. So a lot of the language communities we have, they know about these problems and they're working very hard. Like Remy and the folks on the ground working at Digital Umuganda, they have done a lot of work to make sure that the Kinyarwanda data set is as balanced as possible. And they've done some really great things with that, especially for gender diversity. It's a hard problem because you want to get data that's balanced, but also you want to, you need to be mindful of people's privacy. So you don't always have people reporting their demographic status because it's an opt in project. But even given that Digital Umuganda has been able to do some really great things through community recruiting to make sure that the corpus is as balanced as possible.

Yeah, Remy, could you talk about that a little bit more in terms of your strategy around data collection and making sure you have these kind of balance groups and involving the right people in the community.

Yeah. So basically Digital Muganda did actually an incredible job in actually collecting voice data sets. So the strategy was more like hosting like offline events and bringing the awareness to actually contributors. Why is actually voice technology very important and especially for underserved languages? Yeah, they really did a really good job and so far we have on the platform over a thousand hours of voice collected. This is have been done by setting up actually a community of commoneers, so communiers actually members who actively like organize local events in their areas, in their universities and in order to get people to contribute.

I was just wondering if you could say this Digital umaganda, is that like a, is it a company, is it a for profit, Is it a community thing? What could you describe a little bit of, of what it is?

So yeah, Digital um, is actually a company which actually work on AI products. And especially what they're trying to actually build is actually AI chatbots and like in, in Kenya, Rwanda. And the AI chatbot will be actually in charge of actually giving information related to Covid and also in like to other topics. So I've been facing the challenge of not having a chance actually. I. Can you run the actually data set? They actually started by actually collecting voice data on the common voice platform. And the next phase for them will be actually like working with the data sets. And Josh have been like working on the Kenya random model which they can actually now use to integrate to their chatbots. I think that's actually answer your question.

Yeah, it does. And I'm kind of curious, kind of going back to what both you and Josh were saying before. Really wondering as you're looking at what impacts performance in terms of getting these recordings from different people, different groups, Is it gender? You know, the sound of gender, accent noise, what is it that really affects, you know, what the end product is? Josh, could you address that one?

Yeah, this is a great question. What kind of demographic factors are going to have the biggest impact on performance for speech recognition systems? So I think accent and gender have been just hands down proven to have a huge effect. Accent depending on what we classify as accents versus dialects versus languages. I'm sure Daniel has thoughts on this too. Language is a continuum, right.

Everybody has a different definition.

Yeah.

Okay, a concrete example. Try using Siri or Google Home with a Glaswegian accent or any kind of Scottish accent. It might be better. Now I'm assuming they put more work into it, but pretty famously a few years ago it was just abysmal and so accent has a really big effect because I think the kind of continuum is so broad. And then after accent, gender has been shown to have a big effect, like a very reproducible bias. There's lots of different ways to think about why there's bias in a model. A lot of times the typical answer is, well, the training data set was not sampled correctly. That there are under sampled majorities, as it's been called in the data set. And that I think for gender might be one of the big problems. And also on a kind of technical level, speech technologies were developed basically in a way that makes men's voices, because they're lower pitch, they have a different kind of frequency range to make it easier to work with those kinds of voices. So there's a sampling problem and there also are problems that could be inherent to the technology itself. But at the end of the day, the demographic factors like accent and gender are usually much more pronounced than age. Age can also have an effect, but I've seen that less often.

Foreign.

News is the best way to keep up with the fast moving software world. We track, log and contextualize the coolest projects, the best practices and the biggest stories each and every week. Make changelog.com your daily destination or hit the snooze button and subscribe to our weekly newsletter that hits inbox on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0 and you can subscribe right now at changeloud.com weekly.

All right, so as we've got into talking a bit about the, the impacts of these demographic factors in terms of the performance of certain speech technology like speech recognition, I want to kind of circle back and see, you know, to Common Voice and Jenny and ask like if this work that Josh has been doing in terms of the demographic factors, if that's sort of influencing thought on the Common Voice team in terms of if I, I may be mistaken about this, but I think like you have some processing that actually when you download a data set, actually there's already a segmented out training test dev set. Is the work that Josh is doing or maybe other similar efforts, is that influencing your thought process around how you're forming those training data sets for people and segmenting out that data or maybe it's, it's making some changes to the collection interface to promote certain diversity. What's the sort of feedback from this kind of work?

Yeah, we work quite closely with Josh and rely a lot on his research and expertise in these matters to guide sort of our Technical roadmap. So I think from, I would say from the inception of Common Voice until this year, we've really focused a lot on language diversity with the idea that that was the thing that we could bring the most value to, especially around being able to mobilize Mozilla's international contributors and, you know, kind of counting on our reps and our local communities to do the kind of organizing that Remy was mentioning as well. And so now that we feel like we have a pretty good momentum going on the languages side, we're really starting to look more at the demographics inside of a language and gender is something that we are very cognizant of as something that we also need to correct. The kind of interesting thing about being pretty much the only voice data set out there that releases our demographic stats also means that we are very aware of all of the ways in which we could be doing better. And so the ways we've thought about remedying that do definitely include the segmentation that you had mentioned. Right now the segmentation optimizes for diversity of speakers, but there's certainly post processing we can do there to, you know, try and optimize for gender as well. But some of the other things we are thinking about on the platform side include things like, we know open source in general tends to feel less accessible to underwear representative groups and looking at what campaigns we can do what, you know, we could run challenges and events, specifically reaching out to underrepresented groups to make it very clear to them that we welcome and actively seek out their contributions. But also one of the other features that we haven't been building on the platform is the ability to collect data for a specific segment. So the thing that Josh had mentioned earlier on, you know, collecting a data set of just reading out the digits 1 to 10 and the words yes and no, that's a very specific target segment that we can then go out into the world and say, hey, this is a thing that we want to collect for the specific purpose. Can you help us? And we're looking at how we can use that segmented campaign ability to then drive other diversity as well.

I realize there's some good expertise with our guests on, you know, the ethical side and data governance side of things. And I wonder, like you had mentioned that only certain of the people that contribute to Common Voice, they self identify. Of course, it seems like, you know, you could, if you wanted to create some, you know, data augmentation method that would detect, you know, the, the accent or even the gender based on those who did self identify to then label those who didn't? Has this come across your sort of ethical discussions in terms of what you do with people's speech that comes to you and what you don't and how you form those principles.

So the Common Voice as a team, and I think Mozilla more broadly really takes a data minimalist approach wherever possible. I personally am a strong believer that the only data you can guarantee to be secure is data you never collect in the first place. And so what we as much as possible want to minimize what we are tracking on our contributors. You know, we respect, do not track. We ensure that if people don't want to be identified that they are not identified. This is also why we allow anonymous contributions on Common Voice. So that's kind of the first principles we're operating on in terms of using augmentation methods to gather more information on those demographic pieces from people who are not self identifying. I would say that any augmentation method that is available to us is in self also biased. And so I would be very concerned about introducing additional skew if we're using identification methods that are trained on uneven data sets to then further segment our own dataset.

Right.

You might just kind of be propagating a problem that is already there.

Yeah, totally.

There's something I'd like to tack on there. I definitely echo everything that Jenny said and a little bit, I guess, further, let's say that we do have some kind of gender identification or age identification or accent identification that is 100% accurate. Even in that case, I would not want to be using that at all. Because the fact that this is an opt in choice means that if the person didn't give you that data, they don't want you to have it. It's not that they just don't want to give it to you, it's that they don't want you to have it. So before I got into speech technology I spent a lot of time working in kind of psychology labs. And if I had folks coming in and doing some kind of cognitive tests and they have a sheet of paper and they're filling out demographic information and they choose not to fill out their gender. And then after they leave, I think, oh well, I saw that she was a woman, so I'm just going to fill in woman. That's something that would set off a lot of alarms. And I think that this kind of approach where you're filling in kind of holes in the data after the fact is the same thing when we're thinking about Kind of the ethics of using machine learning models, I think it really helps to make it concrete. If you say, okay, if it weren't a machine doing that, if it were a human doing it, would that be okay? And I think in this scenario it's something that I really do not feel comfortable with. Because the person doesn't want you to have the data, right?

Yeah. And I'm sure this is a problem. I think we're going to have a discussion pretty soon as well with one of the major image data set groups that's out there. Yeah, I mean you could identify a lot of things in images or you could probably attempt to identify even people through speech. I think I remember when I downloaded some of the data from Common Voice, I had to agree as a user of that data to not try to identify the speakers. So I don't know if that's always been there, but I definitely resonate with what you both are saying for sure.

So, question for Josh and Remy, as you were looking at the future of where your Mozilla fellowship can take you at this point, kind of, what are you thinking? Where, where do you expect things to go? What would you like to get accomplished under that fellowship?

So I think what I really want to accomplish is actually having, making sure that we have actually a strong model which the local startup ecosystem can actually use and good, actually relevant application for people who have been actually left out from the digital world. And yeah, I think that would be actually something really good for me to accomplish and as well as making sure like we have actually enough use case where we can actually use voice technology.

Yeah, that's awesome. I really like how you put that because one of the things that really been privileged to sort of learn a little bit more and grow in as I been doing my own work is just growing in my own desire to have sort of two way communication in the, in the digital world. So not so much that, you know, like we want to have for example, machine translation models that translate everything that's in English to every other language. You know, it's really great to, you know, on the other hand think about, well, there's so much value and amazing things happening in these language communities around the world, local language communities that we as high resource language speakers are missing out on. So that's really great and exciting stuff. What about you, Josh? What are your thoughts for the future of the fellowship?

Yeah, so right now, for at least the duration of the fellowship, there's a couple projects that Remy and I are working on. There's the the one that we've been talking about, the digital Umuganda kind of based project that is focused on Kinyarwanda and collecting Kinyarwanda data, engagement, community kind of mobilization and also model training for deployment and making a product that people actually can use and people care about. That's kind of one side of it. We're also working with some researchers out of Makera University in Uganda. It's also related to Coronavirus. It's kind of a project that's keyword spotting for broadcast radio so that folks in the Ministry of Health can know kind of more details on what's going on in the country with regards to Coronavirus because the broadcast radio is a really well used kind of medium to get up to date information. So these two projects, they're both about data and models and community engagement. And right now these are what's really driving and seeing these to some kind of useful product is the near future. But moving forward, what I'm really interested in is leaving a set of kind of best practices guides, kind of playbooks for how to do all the things we're doing right now. But, but without me and Remy, if a new community wants to start using Common Voice, using Mozilla's Deep Speech, they want to start along this path that ends in a product. How do they do that without needing some kind of expert? My hope is that we can make documentation and resources that are useful for developers everywhere so that they can do this at home. Common Voice is all about this democratization of data. Deep Speech is about democratization of tech. And I think that the work that we're doing right now is about this kind of democratization of expertise because for every project you can't get somebody to help you out. With voice tech, there's not a ton of us around. There's 7,000 languages in the world and within each of those languages there's, there's lots of projects. And so to make this really open, we need to get the knowledge out there too.

So were you always wanting to have products as the end goal here or developer tools that you could use or you seem to talk a lot about enabling other developers to build things through maybe a speech technology that you put out there? Was that always the goal or did you ever think about building sort of the end product like the FinTech application or the healthcare related application directly? What's the balance there in your mind?

So in my mind, these two projects we're working on, there's the Rwandan project and the Ugandan project. For the Fellowship. These are very, each, they're very product focused or end goal kind of focused. But the kind of overarching context, at least for me, is these are two examples of taking an idea in speech and going through the entire pipeline with data collection, data validation, community engagement, making something that can be deployed. That whole pipeline for both of these projects is something that's reproducible. So it's kind of the answer to your question, I think, is both. These are both in my mind first important, because each of these projects is kind of helping us hone the pipeline. And then once we have that honed pipeline, we can communicate that to a broader community.

So, Jenny, I'm kind of curious. You're right at the center of speech data gathering and governance, and I guess, are you hopeful and what are your kind of thoughts toward the practicality of being able to expand speech tech to more and larger communities around the world over the next few years? And what are you excited about in terms of Common Voice or speech tech in general?

Actually, that's a very interesting question. For Common Voice. The thing that I. So I've only been with Common Voice for about a year, and in that time, the thing that I've been repeatedly blown away by is the enthusiasm and the dedication of our various communities that says to me that there's really a need for something like this in the world, that Common Voices are filling a gap in both the marketplace and also just sort of the broader language community. You know, we've seen applications of Common Voice that we hadn't originally intended for. We were looking for it, using it to collect voice data for speech text models and those sorts of things. But we're also seeing that Common Voice is being used for language preservation. And I think there's potential applications for Common Voice for doing language acquisition. Even the thing that I'm the most excited on is just how can we broaden the reach and the diversity of Common Voice so that people who have ideas that would have never occurred to us internally are empowered to use these tools as much as they can. And I think at the same time, because Common Voice is our collection methodology, as Josh said earlier, is very consent minded, very privacy minded. And I think there's a lot of opportunity to do leadership in the data governance space around how we think about data licensing for something like this. I think to some extent, voice data is inherently biometric. If somebody downloads a Common Voice dataset and goes, hey, I recognize my friend, there's no amount of obscuring or anonymization that we can do for that data set that would prevent that from happening. And so I think there are going to need to be some very difficult conversations in the next few years on how we handle that in such a way that it doesn't get abused and it doesn't get used to create a deepfake or any number of nefarious and problematic applications. And I think being at the forefront of the governance question before things get to the point where they are now with Clearview AI, just to use the next example, is a really powerful way that we can show up in the world.

Awesome. Yeah, I think that's a great perspective to end with. And I just want to thank you all for taking time out of your really, really important work to talk to us about speech tech and data and bias in speech data. And you're really exciting work for African languages. We're just really thrilled to have this conversation. Of course, in our community Show Notes we'll link to all of the things that we've been talking about about our common voice and digital um, and the Fellowship program and all sorts of things. But thank you all for joining us. Really appreciate the conversation and hope to chat again soon.

Thanks for having us.

Thank you so much.

Do you have questions, praise or constructive criticism about the conversation? Conversation you just heard Comment on this and every episode of Practical AI on changelog.com just open your show notes, follow the Discuss on Changelog news link and let your voice be heard. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santo, that's me. And our music is provided by the mysterious Brakemaster Cylinder. We're brought to you by some amazing sponsors. Special thanks to Fastly Linode and rollbar and a special shout out to those listening on our ad free Changelog feed. If that's you, you're awesome. If that's not you, well, you're awesome too. But you can learn all about it@changelog.com that's all for now. We'll talk to you again next week.