NLP for the world's 7000+ languages

Bandwidth for Changelog is provided by Fastly. Learn more at fastly. Com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com what we love most about Linode is their independent and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock in open at every layer. If you want to learn more, head to linode.com open again linode.com open.

Welcome to Practice Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changehawk.com community and follow us on Twitter. We're at PracticalAI FM. Okay, here's Daniel and Chris.

Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm principal AI strategist at Lockheed Martin and with me is always is my co host Daniel Whitenack, who is a data scientist with SIL International. How's it going today, Daniel?

It's going pretty good. It's been a busy day of recordings. We were just talking about this before the episode started. I just shoved a bunch of cashews in my mouth as a snack and I think you had some breakfast troubles in between episodes.

Yeah, I had not had breakfast and so in the last few minutes I went to get breakfast and as I was putting it together, I have multiple dogs. One of my dogs did something. I think it was an organized thing. One dog was trying to pull me to the side while the other dogs went and got my breakfast. So I'm going to sit here and do this podcast. Hungry. That's right. You know what? We persevere no matter what we handle.

Exactly.

Okay.

These are the sacrifices we make for Practical AI.

That's right. That's right. We're hardcore podcasters, so, you know, it's interesting, I got to say. I know you live up north, but I'm about to head out today for the Denver, Colorado area, Littleton, where Lockheed Martin has part of its space division. And I looked at the weather for packing and it's like it's going down to, like, zero degrees Fahrenheit. And for listeners who don't know, I'm from Georgia, I'm a Southern boy, used to warm weather. I'm quite frightened to get on this plane and go to this place with. With such frigid coming. Yeah, yeah. I'm being tough in two ways, both with the dogs and with the weather. I'm just saying you're leveled up. There you go. I'm ready to go. Okay. Well, you know, we have a pretty interesting episode coming up here. It's going to be a little bit different, hopefully, I think so, from what our listeners usually hear. Typically, we'll have either a guest on to talk about what the guest is involved in, or you and I will do what we call our fully connected episodes where we will talk about a topic between the two of us. And we're doing a little bit of a blend of those today. So we're going to address AI with local languages. And today, instead of strictly being the host, you're here representing SIL International, which is a nonprofit in local languages.

I'm allowed to speak more than just questions this episode.

You're allowed to speak more than just questions on this episode. And also with us today, we have Dan Jeffries, who is the chief technical evangelist at Pachyderm. Welcome, Dan. How's it going?

It's going wonderfully. Thanks for having me on the show.

No worries. And for listeners up front, I'm going to try to go with. For Daniel Whitenack, we're going to say Dan W. And for Daniel Jeffries, we're going to say Dan J. So it'll be a little bit different since we're broken out with Dan's today.

Yeah. I suggested that Dan J. Go with Pachyderm Dan, but he informed me that he's not completely defined by his employer.

Oh, my.

Which I guess I understand.

Oh, and you've now put that out on this recording. I can't believe that. Well, well, well.

Dan W. Wanted to be fully defined by his employer and says that he has no other outside interests whatsoever. So I don't know if that's a.

Direct quote, but I'll. I'll let it slide through.

Okay, boys. Okay, let's get back in our corners now. O. We have a conversation to dive into here. I'm going to actually start for a second with Daniel Whitenack, whom our listeners probably know pretty well, but they know you mostly on the practical AI host side, and I'd like you to take a moment and talk about SIL International and what you do there. And then we're going to flip over to Dan J. In a moment.

Sure. Definitely. Yeah. So, as you know, I introduce myself as a data scientist with SIL International, which sometimes we joke as kind of SIL International is everywhere, but no one really knows about it. It's an nonprofit. We actually have people from over 80 countries working in 90 countries. And the vision and mission of SIL is to see people flourish in communities with languages they value most. So we do everything associated with language work, which involves a lot of things. So that involves things like multilingual education, it involves things like literacy work, like language development, even like language survey and mapping and other things, but it's also technology related. So one of our products is called Keyman, which is a keyboard for devices like phones and tablets, and it supports over a thousand languages, which the next biggest keyboard solution doesn't support even nearly that much. We also have a product called the Ethnolog, where we track, you know, what languages are being spoken, where, by how many people, how many languages there are in the world. We're also involved in the ISO standard process for the ISO standards for languages, little codes that represent languages. But I personally work on things related to AI and language, so SIL has actually gathered a lot of data related to languages. We've worked in over 2,000 languages. And so part of my responsibility is to help SIL develop programs and do experiments and research to push AI tasks like translation or sentiment analysis, speech to text, text to speech, these sorts of things past the languages that are currently supported into the longer tail of languages in the world, those local languages spoken around the world, where there currently isn't support for those things. So, yeah, that's what I get the privilege of doing.

Well, thank you for that unusual introduction for this podcast and kind of bring people in, I gotta say, as you finish, having worked with you now all this time on all these episodes, for listeners who don't already know, you are truly a natural language processing expert in AI. And I have learned a lot from you in the time we've been working together.

It's been fun.

Thank you. As we turn toward Pachyderm, I wanted to actually start off by noting that we have previously had an episode on Pachyderm entitled Pachyderm's Kubernetes based infrastructure for AI. It was episode number 23, and our guest representing Pachyderm on that was Joe Dulner, who everyone calls jd. He's the CEO of Pachyderm but with us today as we kind of dive into this story about local languages. Dan J. Can you kind of tell us a little bit about yourself and a little bit about how you arrived at Pachyderm?

Sure. I arrived at Pachyderm through a circuitous path. I've been a technologist for 20 years. I had an IT consulting company for a decade that I sold, and then I spent nine years at Red Hat. I designed some of their early artificial intelligence strategy before anybody was really thinking about it. And I am also a science fiction author with four novels, professional blogger. And one of the things that started to get me very interested in artificial intelligence early was a series of articles that I wrote called Learning AI if you suck at math. And I was taking it from an engineering perspective, trying to look at it from. I'm not a data scientist, but I was trying to look at it from someone who'd spent many years setting up huge SaaS infrastructures, gigantic web farms, office back infrastructure, and trying to figure out whether this stuff was learnable by someone who hadn't studied statistics and enjoyed all those things in school. And so that series of articles proved very popular. It was read by over 2 million people in seven different parts. And I was essentially teaching myself many of the concepts as I was going along. And at that point, I was starting to get talks around the world, both for artificial intelligence and other future technology. I realized at a certain point in time, after my beloved Red Hat was purchased and it was starting to change in terms of its structure.

Yep.

That I wanted to go back to someplace that was very innovative and that was doing fresh things in the industry. And that's when I came upon pachyderm, which falls very much into the mlops side of the house and does essentially version control for data science. So when your models, your data, and your code are all changing simultaneously, how do you keep track of all those things and create reproducibility? Because if you run a bunch of tests on a series of a million images and then an administrator comes in and crunches them all down on the back end to a smaller size, then it's going to be nearly impossible to recreate that earlier experiment. So I've been very excited to be with those folks. And everybody there is fantastic. And it's exciting to be in this amazing industry now and get to work with incredible people like. Like you two folks.

Fantastic. Well, thank you for that. So, you know, starting off, I know I've learned a lot from Dan W. About local languages myself, just from being associated with them. And we did a keynote together recently at Project Voice and he's been talking for a while about the partnership between SIL and Pachyderm on this. And. But I'd like to start off by asking, you know, now that you guys have been partnering together and working on this problem for a while, could you actually tell us, kick us off by telling us a bit about what local languages are and why they should matter to the rest of us.

Yeah, I can definitely jump in and give my perspective. Something I didn't know when I started working with SIL was really just about the language situation around the world, which is maybe something we should just start out by talking about. So a lot of people don't realize actually we track this very, very closely. Right now there's 7,111 languages spoken around the world. So these are living languages, not like dialects and that sort of thing, but recognized living languages that are spoken around the world. And certain countries, like let's say India or Indonesia, other countries have hundreds of these languages being spoken. Indonesia, for example, it's like about 700. And so what happens as a result of this is that there's some languages that are spoken by a lot of the world. So over half of the world speaks. I think it's 23 of those languages, which is a very small number. Right.

Compared to 7,000. It sure is, yeah.

The other half of the world speaks the rest of them. And so what happens is that these local language communities, which don't maybe speak one of these higher resource languages, are usually and often marginalized in some way. And so these marginalized language communities value communication in their local languages, but they aren't supported in a variety of ways. And this has implications for a lot of things. One of the ways that we think about this is in terms of the United Nations Sustainable Development Goals. So the United Nation developed these 17 goals about sustainable development and language impacts, basically every one of those. So if you think about education or humanitarian assistance, thinking about something like HIV or AIDS or Ebola, how are you going to be able to make progress in those areas where there's extreme language diversity? If you can't get out materials about HIV or other things in a language that people understand, value that they don't consider something foreign. And this goes across the board. In education, there have been studies that have shown if someone starts their education by going to school, so if they're at home and their family speaks a certain language and then their mother speaks it, their father speaks it, their whole village speaks it. And then they go to school and the first thing their teacher says is, oh, it's great that you speak that, but you're not going to speak that here. We're going to learn in this other language. Well, immediately, right off the bat, they form an association with education as something terrible and hard and it actually stunts their educational development. Whereas if they start education in their mother tongue, they actually have the same benefits as others in terms of their views of education and their forward momentum. So language impacts everything. And that's why we care about these local languages, because they actually make a difference for people's quality of life.

I think this falls into that AI for good category as well. That's something that's very close to my heart. I started Practical AI Ethics Alliance. That's practical AI-ethicics.org if anyone's interested in checking that out. But the basic concepts behind it are this artificial intelligence is a dual use technology. It reflects everything that is good and bad about humanity. So when we're looking at a problem like languages, and I think maybe the most impressive slide and the most impactful slide that I saw at your presentation, Dan W. Is where it showed that we were using maybe 100 languages for the vast majority of applications, whether that's translation or speech to text. And there were 7,000 other languages that half the population was speaking and we weren't doing anything with those. And that's a surefire way really to kind of continue to marginalize people or to ensure that, like you said, that people are going to find education difficult or that even finding basic services incredibly difficult. So I think it's wonderful that you're working on a side of the house that allows you to make a difference and have an impact in this part of the world. Because I feel like so much of the research sometimes gets poured into getting people to click on ads or all the things that make us money. And those things are certainly important. Economics are incredibly important. But it's also amazing to realize that artificial intelligence can make the world a better place. In some ways it almost sounds cliche or it sounds a little high minded, but it is actually true that certain types of things would never be able to be done without it. I remember seeing a translation for very old Japanese that only maybe 100 scholars in the world can speak now. And there are tens of thousands or millions of texts in a form of Japanese that's really not used anymore. And machine learning is able to augment the ability of those translators to scale what it is they're doing. So that those texts don't die out into the pages of history simply because we don't have anybody interested anymore in being able to translate them. So I think that's where this thing can make a massive impact.

Yeah.

And you've kind of gotten into the idea of the importance of applying AI to the possibilities. I mean, and is there any way, do either of you have any comments on kind of expanding that a little bit in terms of, you know, why apply AI to this long tail of languages? You know, you just identified one. Any others that come to mind?

Well, I think that, you know, AI and especially, I guess what I feel like we're on the verge of sort of new possibilities in terms of AI and language. Over the past couple of years, you've definitely seen like some people might refer to as an inflection point with a lot of new techniques, a lot of emphasis on transfer learning, a lot of emphasis on usage of monolingual data and, you know, things that really impact the languages in this sort of long tail of languages, the languages outside of those supported by the major tech platforms. So I think one thing to note is that, you know, whereas used to, we might have been stuck with some of these things like machine translation because of lack of data or something like that, that there are brand new possibilities where what if we could translate these HIV materials or this educational material into all of these different languages? What if we could enable people to be part of the global conversation in their mother tongue? That's a very interesting one for me, is oftentimes I think we feel like, oh, how can we get our great content as Westerners into the languages that people care about? Which isn't a bad thing. It's great to try to get those materials, those educational materials, scientific materials, entertainment media into local languages. But these people in these local language communities, they have so much to contribute to our understanding of the world, to scientific research, to all sorts of different areas. And so one of the things that I think AI could enable with things like speech to speech translation, with things like predictive text and other things like that, are enabling those local language communities to be part of a global conversation, not just to be consumers, but to actually contribute in a back and forth sort of way to global conversations around the things that actually impact their life and the things that they can contribute to like politics and education and all of those things.

I think it's interesting that you're talking about it from a two way street and I think that that's an amazing way to frame it because what is it that we can learn, not just what is it that we can translate from our own content into what other people can consume, but really what is it that we're missing on the other side of the equation? And historically, if you look at how language has been used, sometimes it's been used as a way to dominate other cultures or as a way to socialize. It's almost been used as a weapon. The farther that a language can spread, the more that people think. In your own way, if you think about something like the Etruscans and the Roman Empire, we know pretty much nothing about the Etruscans, primarily because they were completely consumed by the Roman Empire, not in small part due to language, and we've seen this in other parts of the world, but that sort of slash and burn mentality. There's a lot of things that are lost in the same way when we destroy a huge part of the forest. What medicines get lost that we would never have been able to find, what compounds were hidden in the species of plants that were wiped out. And in the same way, different types of languages allow us to think in a different way. And in fact, we just had the wonderful super bowl on the other day, and there was a commercial, believe it or not, that stopped me for a second, where they said that there were four words in Greek for love. And I went and looked up each of the words and they were fascinating in that each one of those words conveyed something very different about the nature of love. From Eros, which is more of a passionate kind of love, to Agape, which was more of a selfless kind of love or a love for country. And each of those words convey something very different. And so when we lose these languages or we assume that we have the words that perfectly convey things, we lose a lot of nuance and meaning and we lose people's ability to connect in a fluid way with us. So I think it's amazing that you framed it as a two way conversation. That's very important.

This episode is brought to you by Brave. We deserve a better Internet. That's why the team behind Brave reimagined what a browser could be. Brave is like Chrome. The good parts, even your extensions will just work. It has built in add and tracker blocking, easy anonymization with the Tor network. Earn tokens while you browse and use them to tip your favorite creators. And did I mention is lightning fast. Turns out the web is super fast when you remove all the cruft. Download Brave today using the link in the show notes and give tipping a try on chainsaw.com.

So, Dan W. What is SIL kind of doing in terms of AI for local languages these days? And what are you interested in doing to tackle in terms of, you know, the types of problems and issues that SIL is attending to specifically on your AI roadmap?

Yeah, definitely. So in the longer run, like I say, we want to see this sort of two way street that we've been talking about in terms of local languages being part of a global conversation. And I think maybe a natural place to start with that is the sorts of AI technology that have already been groundbreaking in our everyday life, maybe as English speakers or other high resource language speakers. And so if you do a Google search now, right, that's hitting an AI model now, integrated BERT into that. If you're writing an email, you've got predictive text along with that, that helps you. If you're dealing with a chatbot, you have these things like sentiment analysis and entity recognition. If you're talking to an assistant or a smart speaker or other things, you're using speech to text, you're using maybe those same assistant capabilities you might be using text to speech. And so these sorts of building block AI technologies are what we're thinking about a lot right now. And how could we take those building blocks which now only support sort of high resource languages, maybe up to a hundred, but as Dan J. Mentioned, that's kind of a drop in the bucket. How do we push those into the longer tail of languages? And what really excites us is, you know, how could we not just do that language by language like oe, add the next language and then we add the next language and then we add the next language. How could we knock out 40 languages at a time, right? And I think those are the things that get us really excited. And so some of the things that contribute to those sorts of advances, I think are first of all, multilingual models. So we've seen this shift recently into massively multilingual models that support things like Google Translate, where one model actually can process multiple different language pairs. And something people may not know is, you know, I think no one's been able to challenge me on this, but I think SIL and its partners have access to the most massively multilingual corpus that there is. So we've done work in over 2,000 languages. We have some type of data, whether that be text or audio, and depending what partners we gather together, maybe like 12, 1200 languages. So there's a lot of data there. And part of what we're excited about is what happens if we take the largest multilingual model that there is now, which I think the most multilingual one is around 103. What if we push that to something like 300? How does that affect adding the next language in? Does it make it easier? How should we structure these types of models into language families and other things? So we're exploring those things. At the same time, we're exploring a lot of the low resource machine translation technology that's been developed around transfer learning and fine tuning, iterative back translation. There's just a lot of different techniques out there now that allow you to maybe take a high resource language and adapt it to a lower resource language or even make use of multilingual data. And so those are all the things that we're interested in exploring first in terms of experiment and research, and then in terms of making strategic partnerships with tech companies, but also local institutions and governments to pilot out some of these possibilities and actually get them used.

And that really begs the question for both of you is, you know, as you talk about these partnerships, what specifically brought SIL and Pachyderm together to tackle these kinds of problems that you've just addressed here and why, from each of your perspectives, why did that partnership make sense?

Sure, I'm very pleased that Pachyderm wanted to work with us, so I'm really happy about that. Thank you to Janjay and the team who wanted to work on this. But I think that despite SIL having a ton of data and a very multilingual corpus and an amazing amount of language information and linguistic expertise, we're not a tech company, we're a nonprofit that has done language related work for a very long time, but isn't really an AI company per se and isn't operating a ton of computational infrastructure. So whereas we have a lot of this sort of data and language information, that side of the equation, part of what we want to do is partner with people that have a lot of expertise on the infrastructure side on the AI methodology and AI, you know, practical AI training side. And Pachyderm definitely fits into that component. So from my perspective, that's what I was excited about. Working with Pachyderm is actually building something useful that we can use over time and repeatably use and scale up. Because this is a large scale problem, right? 7,000 languages. We need something that's going to scale and something that's going to work. And so that's what originally got me thinking of partnership with Pachyderm and you know, Dan J. Can speak from Pachyderm's perspective, but I hope that they were excited about these sort of of AI for good problems.

Well, we definitely are excited about these AI for good problems, and frankly we've been looking for a number of these types of things in the field. So if folks are out there interested in doing those types of things, we want to talk. We're certainly not DeepMind or OpenAI or have infinitely deep pockets to be able to throw at some of these things, but we do feel it's of tremendous importance for us to help enable projects like this. And frankly, Pachyderm is, is more of the infrastructure side of the house. We recently launched the Pack Hub product, which runs on Google Cloud and allows people to automatically spin up clusters and add GPUs to them and parallelize their resources and those types of things. I think Pachyderm is one of the solutions that people don't realize they need until they start doing data science at scale. And I think we're seeing a development of a canonical stack probably over the next two to five years where the tools become codified that allow data scientists like Dan W. To really do their job easier. And if you think about the history of how these things have worked, a lot of times data scientists were just passing around a text file between them or maybe FTPing something somewhere and cobbling together infrastructure that's not really going to work as you get this technology out of the hands of the unicorns that have a billion dollars to just throw at things and create the infrastructure on the fly. So if you think about a company like Google or some of the research foundations doing their own work, they're all building all their pipeline tools and their training visualization tools and their explainer tools, all these types of things, and they're experimenting with lots of different frameworks and libraries. But over time we're going to start to see more and more standardization. And a problem like being able to version control your data and understand the entire data lineage of where things got from point A to point B to point C is incredibly important for being able to reproduce experiments. I read in VentureBeat the other day that something like 87% of data science projects never make it to production. And that's a massive number considering that we've spent upwards of 60 or 70 billion dollars, I think was the number. Or we're spending hundreds of billions of dollars more or hundreds of millions of dollars more in the coming years, and that means we wasted that much if we don't improve that. And if we don't improve that, we're in serious trouble. And one of the ways to improve that is by having that level of reprod and being able to work across a diverse team. And so getting our tools into the hands of people who are doing amazing things is definitely the way to get our name out there, but also really make a difference in the world. And I think both goals are incredibly important.

Yeah. And on a practical level, since I'm always interested in keeping this podcast practical, I'll kind of walk through our internal workflow and thought process on this. I mean, if you imagine, let's just take machine translation for example, which is one of the things that we're working on. Well, I can spin up a collab notebook and you know, develop the kind of pull the data together from a source that we have access to inside of sale, do some pre processing on that, do the training test, you know, do the testing, get the inference bit worked out. But now if, if we're really serious about like our goal of pushing this sort of thing into many languages at once, right now I have to think about other sorts of problems. And one of those, you know, on the data side is the data that SIL has access to and that we use is a big mix of data. So it's partly internal data and most of the time like formatted in sort of maybe non standard formats. As far as AI people are thinking, it might be data from partners, it might be a mix of public data. And so we have all of these sorts of data that we may want to bring together in unique ways and the combinations of those data for, let's say if we're targeting 40 languages, they might be different for the different languages. Right. And so there's this complicated issue of how do I combine all these things together in a sane sort of way with a bunch of pre processing. And then I've got the problem of, okay, well I need to kind of standardize those, I might, those data sets might be updating at certain times. And then I've got to connect all of those data sources with the correct pre processing, like I said. But then training, and that training needs to happen on GPUs where maybe the pre processing is happening on CPUs. And then I need to connect those output models, I need to actually export them and optimize them in a way where they can be exported to a certain place where they can actually be used in a, in a product. So, so all of those things for even a few languages or 40 languages or whatever we're looking at, that gets complicated fast. And so the ability to track all of that very rigorously, but also be able to scale it as we might want and do it in a sort of way that isn't like there's only so many technical people at sil, so we can write, you know, small bits of code to do these various things, but we're not going to write the whole infrastructure and logic around this. So we needed something that was able to handle those sort of data elements, scaling pre processing across lots of data sets and also scaling our training while utilizing certain GPUs. And so Pachydern project and the, and what's available in that project in the pipelining and data management allowed us to do those sorts of things. And it really comes down to the fact that, you know, we want to scale this out, we want to push it to many, many languages. And so to do that we're going to have to do it reproducibly and we're going to have to do it over and over and maybe scale it out horizontally as well. So there's a lot that goes into that and very thankful to get help on that front.

So, you know, I know Daniel W. When we were doing our keynote together at Project Voice, you had a really interest example that you as SIL and Pachyderm work together on around these sorts of problems. And it kind of outlined what both sides, how both sides approached and what you were able to do, including the benefits of doing the activities on Pachyderm. Could you kind of, kind of go through that example? And I would also invite Dan J. To pipe in, you know, so you guys can kind of relay that together a bit.

Yeah, sure. So the problem we were looking at was text to speech or speech synthesis and specifically adapting an existing text to speech model to a local language or a local dialect or a local accent. And so you could think of examples like there's a whole bunch of vernacular Arabics that are spoken around the world. There's a bunch of world Englishes that are different in certain ways. And so we took for example Singlish, which is a dialect that's spoken in Singapore. So it's actually a mix of like English and some other languages that's spoken together. This is kind of an interesting problem because as a dialect it has these elements from, you know, at least four or more languages, but it also has various kind of standard accents that go along with it. So there's like an Indian accent, it's English, Chinese accent, It's English and it's a nice kind of proving ground for some of this adaptation to accents and other things. And so we wanted to create some text to speech models for Singlish because for one these don't exist. And for two, we were able to access some data through one of our partners in Singapore to get some of this data for Singlish and to utilize it to test out these methods. The downside to that is like there was a lot of processing that had to happen here. So our partners were able to actually get us. I think it was like 800 gigabytes of data. So this was between our partner Wordly in Singapore and the government institution there, imda, that has gathered a lot of this data. All of that data is formatted in specific ways. Some of it's kind of noisy, it corresponds to a lot of different speakers, like 2,000 different speakers. And so there was a lot of like, how are we going to actually pre process this? And then how are we going to like make this efficient? So we're not running these models for like, you know, weeks on end without progress. So that's where we consulted a lot with Pachyderm and they were able to kind of guide us through like. Well, here's how maybe similar people have set up their pipelines in the past and the type of infrastructure that they've used and how they've scaled it. I know. So Dan Jay, I know we work through some issues with like data, like how do we upload that much data? How do we pick out like you don't want to load 800 gigabytes into memory. So how do you like access some of that data but not all of it to like figure out what you need? So I think these are, you know, from my understanding these are problems that other people are facing and they were kind of going to, you know, help us solve some of those. I remember the one problem of like accessing some data, but not all of it was kind of key to this whole problem.

I mean being able to split up the data and we rely a lot on. I think we made an intelligent choice in going with Kubernetes and Docker early so we could leverage a lot of the scaling that happens now. If you really think about the history of containers, Google was running billions of these containers even before Docker existed. And so the industry was moving in this, in this direction. And they had originally an internal service called Borg that was then shifted over into the Kubernetes open source project. And that really took off and allowed People to kind of build these massive infrastructures that scaled much faster than these virtualization infrastructures, where you ended up having an entire operating system that was built into this little box that you were processing things in. The virtualization error was very effective. But once we got to need lots of ephemeral machines or being able to quickly spin up 1000 different nodes to split up data and process it into little chunks so that you're not trying to load everything into a massive virtual machine and saturating the memory. We spoke with one customer recently. There's a case study coming out where they were doing a lot of of language processing and they built their pre processing tools and they were taking about eight to 10 weeks on the biggest possible node that they could spin up in Google or AWS or Azure. So they were basically grabbing the most expensive node possible to try to fit everything into memory and stack the GPUs in there and it was taking them about. And they were able to parallelize it with pachyderm and get it down to about six or seven days. So that is a massive increase. And that's basically because they're able to split it up across multiple nodes without actually really having to worry about precisely how they split it up. Pachyderm does a lot of the heavy lifting for folks in the back end and allows it to work across multiple nodes as opposed to having to try to figure out within your own code because you're already worrying about. A data scientist like DW is already trying to worry about how do I solve a problem like transfer learning or a noisy data set and how do I clean that up? Or I've got a number of different formats, how do I either use all of those formats or standardize on a different format before I can even do any of my work? The last thing you need to be worrying about is then figuring out how to also be an infrastructure engineer and auto scale lots of different nodes yourself and spin them down and suddenly you're forgetting about them and then your company gets an AWS bill for a million dollars at the end of it. So that's, I think, where we really make a big difference for folks doing this kind of work.

Gotcha. I have kind of a follow up question to the example itself that you guys have worked on. And that is, you know, if there's one thing that's become clear to me as the person not involved in this, it's what a target rich environment this is to work on in terms of, you know, 7,000 plus languages. And you know, you're even with as much work as you're working on at this point, it's still you're only hitting a fairly small fraction, at least at the moment. You have addressed this first example that you've kind of relayed to us. Why that one? Where can this particular approach that you've talked about lead to next? What do you envision as being kind of next steps for extending this?

Well, I think one thing to emphasize with this and one other reason why we were really interested in this partnership to enable this sort of thing is Pachyderm is part of the open source community and part of the Kubernetes community. And so anyone can run a Pachyderm pipeline and anyone can spin up a Kubernetes cluster in the cloud or on premises or wherever. And so we're thinking about this sort of work as a sort of template, right? So I created, with the help of the Pachyderm team, created this template for training speech models. Using this pipeline, you could plug in any sort of speech data set you want, assuming you could pre process it into the right format. And so the idea is we showed this for, let's say one accent and one speaker of this data set that we worked with, and we're working on others as well. But I could publish that pipeline on GitHub, which I have, and anyone could pull that down. Anyone could access the IMDA data if they knew where to look. And we put the links in, right? And anyone could generate their own speech model, run the same Pachyderm pipeline on their own Kubernetes cluster, because everything is portable and everything is built on this great open source community and people could collectively work on this for a greater impact than any one certain person. So we're not gonna, like you say there's, this is a target rich environment. So the only way that we're going to make progress here is if we make these sort of reproducible templates and enable people to run them for their own context and their own data and scale things up, up that way. So that was another really appealing thing to me about setting this stuff up is internally to sil. We could rerun this pipeline on any sort of, for any language where we have audio data to train our own text to speech, but we're not going to get to them all all at once, right? So other people could run this pipeline with their own speech data on their own cluster to create their models. And so by creating this sort of reproducible template, it's actually enabling A different sort of scaling.

I touch on the open source, the essential part of open source. And I'm admittedly a true believer in open source. I was, like I said, at Red Hat for nine years and saw the early days there where recruiters told me, why are you going into this Linux thing? Solaris is where it's at and where all the money is.

Solaris isn't where it's at, no.

And I told them, I said that it might not exist in 10 years. And they thought, what are you talking about? Right? And we spent a lot of time in the early days going in and saying, what is Linux and why won't it fall over? Why would I bet my future on something like this? And it's been amazing to watch the transition to an open source over the course of those years to become the default model for how things are done. It used to be that things would happen in a proprietary world and then open source would come along and kind of build a commoditized, almost good enough version. But nowadays everything, including most of artificial intelligence work, starts in open source. And there's a huge advantage, I think, with something like Pachyderm being completely agnostic to the tools that are built on top of it, especially versus some of the pure cloud services that have to, because of limited resources, take a fully opinionated stance on every project they have to support it for it to run, as opposed to us, which allows data scientists to really bring whatever tools they need to the project and then publish anything that they create. So we don't just have to explicitly support something like Pytorch. You know, we spoke with another, the group that I mentioned earlier was using a more obscure speech recognition toolkit called Caldi that they have had heavily modified themselves. And the chances of something like that being supported in one of the cloud providers choices with the limited resources, even if they are a billion dollar unicorn and they've got 1,000 programmers, they still only have so many resources. So something like Scikit Learn or PyTorch or TensorFlow and 50 different Python libraries are going to get supported where something like Kaldi is not going to get supported in the same way that the languages are not going to get supported across the world because of resources. So allowing people to do things with open source and bring whatever they want to the party, I think allows this kind of collaborative creativity to happen and allows a kind of scaling that wouldn't be allowed to happen with smaller projects or being able to move towards languages that might not be represented. And I think Those two concepts are intricately interwoven.

Yeah, I remember, Chris, when we were talking about our talk at Project Voice, one of the things that kept coming up is this sort of idea of collaboration for collective impact. And the problems that I have are actually not like I can get data for languages, I can get information for languages, I can get linguistic information for languages, I can get dictionaries, I can get grammars, I can get of these all, all of these sorts of resources. But I'm limited in certain areas as related to maybe, you know, some of these more infrastructure related things. I'm related resource wise. I work in an organization that's primarily a language related organization and we're doing a lot with AI now, but we're still kind of figuring out a lot of those things. Whereas you at Lockheed Martin, you have a lot of resources in terms of computation, you have a lot of AI knowledge, but you might not have those language related things that I have easy access to. So a great way for us to make an impact in the area of language is in this sort of collaborative way for collective impact, where we're not just kind of siloed in our own world. Like Dan J. Was saying, we're not limited to our own sort of implementations, but we can work together, we can open source things, we can bring all of our resources together to solve kind of larger, harder problems.

Yeah, you know, I think the collaboration that we did especially for Project Voice and really all along, as I've been learning from you over time about this, is it really brought home where local language is and how important it is. I know for me personally, I know I've mentioned before working on humanitarian assistance in disaster relief initiatives at Lockheed Martin, and I know we've talked quite a lot about educational impacts. You know, it's so critical that you be doing this kind of work, what your two teams are involved in, so that we don't leave behind enormous populations of people on the planet and grow that digital divide. It's, I think it's kind of a, a little bit of a stealth issue to many of us because it's not something we necessarily think about all the time. But if there's one thing that I've come to realize in my own introduction to this has been just how incredibly important this is to everybody going forward. You can't, in a humanitarian assistance or disaster scenario, go in if you can't communicate effectively in the languages of the people that you are trying to work with. And same with education, as you pointed out. So I think if there's anything I'm coming away with, it's that we're at a very special time for kind of the integration of artificial intelligence and language and that there's so many possibilities that could now be realized. And so I guess I want to finish up by asking you both, you know, what do you think the future is looking like for local languages and AI from this point forward? It's a remarkable moment. We're in at this kind of turning point. But what do you see ahead? And like, you know, as an example, how do you think SIL would work with Pachyderm and other organizations out there to enable all the possibilities that we have before us?

I have so many ideas. My problem is not lack of ideas. I'm so excited for, and I'm sure, Dan J. You probably have a lot of ideas, maybe you can go first. I don't want to still always be hopping in first. What are you excited about?

Dive into it. DAN J. Yeah.

Steal my thunder.

Well, you know, I think actually when we had dinner together the other night at the conference, we talked about a larger misconception in artificial intelligence, which is it seems like everybody's pouring resources into this concept of generalized artificial intelligence. And that's a noble goal. And I think we get there eventually at some point, maybe even in our lifetimes, maybe not. Maybe it proves to be a lot more intractable than we imagine. But you said that you were thinking more in terms of augmentation. And the way that I tend to think about artificial intelligence these days is very much in that augmentation model model as well, or that Centaur model, right? Where the artificial intelligence is helping humans scale their abilities and use their higher order learning and understanding and intuition, the types of things that are still intractable in machines. We see some of this behavior that we might be able to call intuition and something like AlphaGo as we combine three or four different algorithms. And maybe when we've got 20 algorithms working together, we could mimic it even more. But in the short term, this is really about scaling and about augmentation and about allowing people to do more. And if you think about something like language, where, especially when you're working with a language where there aren't necessarily as many speakers or there aren't as many experts in that field, or there isn't as much data, you absolutely have to have augmentation. You have to be able to scale what those folks are doing, and that creates more leverage. Right. If you think about trying to lift a giant rock by yourself, you can only do so much. But if you get a really long pole and put it under that rock, you've got a better chance of lifting it. And that's the way I think of artificial intelligence now in the speech side of the house. We need to be able to help all the folks out there if They've only got 1,000 experts in a particular field. If the work that you're doing is able to combine a lot of different data sets and look across 1000 languages or 100 languages or 50 languages, and find the similarities and therefore make the fact that there is not as much data for that largely irrelevant and still create a very robust translation model that could then make it easier for 1000 different texts to be translated and then a human to go over each of those quick and enhance them versus that person trying to scale themselves to do 1000 different translations and quickly getting burned out. And so I think that's really the wonderful part of this aspect. And from the packing arm side of the house, being able to scale the infrastructure automatically on the back end for, for folks to help data scientists do this kind of work so they don't have to also be specialists in machine learning operations and trying to figure out how to slice that data set up into 1,000 nodes so that they can train it and do preprocessing. We're very happy to partner and make a big difference and a small impact in what you're doing as well.

Yeah, I think you hit the nail on the head. I think what you got to there at the end around augmentation, but also this sort of idea of, of leveraging the language information that we have access to and things that are already out there to kind of grease the wheels and get things moving for local languages. I really am fascinated by the concept that like we have all these pre trained models out there, right. For certain languages and for certain language families, we have these open data sets, plus we have data sets internal to sil. We also have this ethnologue resource which is information about all of the languages of the world and how they're related. I'm just really fascinated by the idea that we could have all of that information together. So what languages exist, what are their populations, but also what languages currently have data and what languages currently have pre trained models. And that way when we have a pachyderm pipeline, let's say, and we want to train a new text to speech model, or we want to train a new machine translation model, if we put in the front end, like, oh, I want to train a model for this language, you know Kabundu, an Angolan language that doesn't currently have any support. What is the closest related language that has either a pre trained model or has a lot more data and kind of use tools like Automl and some of this automation to pull those resources in and sort of augment the development of that next language? Those are the things that really, you know, excite me and fascinate me. And excited to dig more into.

Well, this has been a truly fascinating conversation. I know I learned a lot. Thank you both for kind of diving deeply into local languages and how AI can impact and move that right along for the benefit of all. Truly an AI for good initiative. I'm pretty excited about it. And so Daniel Whitenack, Daniel Jeffries that thank you both for coming on the show. Thank you.

Thanks for having us. Really appreciate it.

Comment on this and every episode of Practical AI on changelog.com just pop open your show notes, click the Discuss on Changelog News button and let your voice be heard. Hey, do you have a friend who'd enjoy the show? Shoot them a quick text or an email. Slack em WhatsApp, send them a Snapchat, I don't know, write a message in a bottle, throw it into the seat. We don't care how, but we do appreciate the effort. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo. Our music is by the Beat Freak Breakmaster Cylinder and we're brought to you by awesome sponsors. Support them, they support the show. We've got fastly on bandwidth, Linode on hosting, and Rollbar on bugs. Are you receiving our free email every Sunday? If not, you're missing out. It's our editorialized take on this week in the world of software. What's interesting, testing and why? Head to changeweb.com weekly to check it out. Subscribe for the price of a free egg roll. Thanks again for listening. We'll talk to you next week.