#87 Inside DeepL and Welocalize VP Olga Beregovaya on Language Tech Evolution

They're going to launch live multi-language transcription and translation for Zoom calls.

They apparently captioned 16.8 million minutes in 2021.

If it's written in PowerPoint, it is AI, and if it's written in Python, it's machine learning.

And welcome everyone to the Pod of Slater number 87. Hello, Esther.

Hey, Florian, how you doing?

All good, all good. Fall's around the corner, rain's falling from the sky, and it's making me sad.

No, just kidding.

Today, we'll be taking a tour d'horizon on all things natural language processing and machine translation with WeLocalize AI Innovation VP Olga Beregovaya. Looking forward to that. She knows the space incredibly well. You know, she was, for example, the Head of the American Machine Translation Association in 2016 to '18, so that was a pretty crucial time, you know, all the NMT coming out, etc. And of course, now she's implementing NLP and MT for, you know, one of the biggest LSPs out there for WeLocalize, so we can talk about kind of practical use cases, etc. Looking forward to that, but now for the news. What are we going to talk about today briefly? We're going to talk about AI Media's, what was it, Esther? What is it, a conference presentation or like a financial update?

It's a, yeah, it's a conference presentation, something that is linked to the stock exchange in Australia.

Got it, okay. Access, like multilingual access services, transcription, et cetera. Keywords also released, half-year figures, just briefly plucking out what's relevant from a GameLog perspective, even though it's only 20% of the revenue, and Zoom is planning to add live translation, shocker, after they bought Kites, and so we'll talk about that briefly, but now for a machine translation giant, DeepL will be running a story at the time this is coming out, will have run the story on DeepL. Kind of the inside DeepL story. It was time for an update. Such a, you know, fast-growing company, but also super stealth, very publicity shy. They, you know, didn't talk to us for the story, but we, you know, published it anyway. And so just, you know, the figure that I keep saying also, keep mentioning also on the, on SlaterCon, like they're now like among the 150 most, most trafficked website worldwide, so that's incredible. I mean, it's right up there with some of the major media outlets. I think if I remember correctly, Canva was maybe around like 60, 70, or 80. So like some of the super top SaaS solutions, you know, used worldwide are in that range. So, you know, it's just incredible that they have that kind of traffic and yeah.

[Speaker:OLGA_BEREGOVA] I have a question. If you just leave the— maybe it's a technical question, but if you just leave the tab open, like you leave the browser open and you don't open a new instance, does that count as one visit? So for example, I always have, I don't know, Asana open or LinkedIn open. I always leave the tab up. So am I just one visit or do you know what I mean?

I think if you're logged in, I don't think it counts. Like when you're working on inside a SaaS solution. Like if you're in the logged in state, but that's beyond my web— web development.

I just thought, because it could be, it could be like if people are just always have DeepL tab open for their work, like I have LinkedIn open, for example, like, I don't know, maybe that it's not really—

that counts.

It's obviously very reflective. It's reflective of their usage.

Exactly.

So maybe it's not kind of counting every single time they use it. It's just counting when they open it. I don't know.

I know what you mean. Yes. No, so I think if you use it and it doesn't change like a publicly available URL, it's probably, I mean, yeah, if you're locked in, it doesn't count. So, but yeah, I mean, if even, you know, I think if people have it open, that means like, yeah, it's become part of your kind of day-to-day routine, part of your day-to-day browser routine. And I think a lot of people may also use it kind of as a dictionary or kind of a quick reference. I know a bunch of lawyers that do that. So yeah, lots of traffic. Average full-time staff was 86 according to filings in 2020, but now probably has ballooned or roughly doubled. That will be our estimate. There's some financial information that's available, but not revenue figures, unfortunately. But what we did find out, they have a balance sheet now, a total balance sheet of €21 million. Kind of a meaningless number if you don't have— not meaningless, but there's not much you can get from that. More interesting, they generated about €1 million in profit in 2019 and barely €140K in 2020, meaning tons and tons of investment in the platform. I mean, they didn't raise any additional rounds after the benchmark round, I think in benchmark being the VC back in like 2019, 2018, right? So it looks like now that they're roughly break-even in terms of their earnings means that they're probably investing a ton of I mean, well, not probably, they're investing a ton in just the growth, right? And they don't need additional capital like VC capital to, you know, get diluted and finance this. So going very aggressively into the enterprise in my view, also based on kind of publicly available data, you know, they hire for roles like, they have like 6 customer support roles, a bit more like account executives, managers. So, you know, more than a dozen people that like, interface with clients. If you go to their hiring page, they're like hiring things like pre-sales engineers, et cetera, meaning it's not just, hey, go to deepl.com, get our API, and like, you know, leave us alone and pay your $25 subscription.

So, you think they're building out the services side a bit more?

I mean, services, managed services, right? Managed, right, yeah. So, we spoke to somebody at Deutsche Bahn, the German railway network, and they're a client, so like she probably has a number to call, right? And if there's an issue, and then they probably don't pay just the a $25 subscription, they're a heavy user like inside the Deutsche Bahn enterprise. So yeah, going full on in my view into the kind of bespoke custom managed MT enterprise sale, which is a big, big development for the industry as a whole. They also opened an office in London, you know, near Old Street where you have all these tech startups. And looks like something in Amsterdam as well, and now their new mission statement is now becoming Europe's leading AI company, you know?

Okay, see.

They don't do it for less. Europe's leading AI company, so as many entrants into the translation industry are bound to do at some point, they want to expand beyond. I mean, how many times have I heard like, yeah, we want to expand beyond the translation space? Anyway, so they want to become Europe's leading AI company. And are, at least in the machine translation space, are well ahead of many of their competitors. So go ahead to our story. There's tons and tons of more information in there, so really excited about publishing it this week. So also something with AI in the title, at least, AI Media.

Yes, but this stands for Access Innovation.

No kidding, I didn't know.

Yeah, if you— I mean, I think on the ASX it's Access Innovation Holdings, or at least it used to be. So, because I was going to make that link and then I was like, well, it does have AI, but it's not AI, AI, although they do use AI in the operation.

Say more.

Right, so yeah, AI media, uh, we have a presentation coming out of the company, um, It's, I think maybe it's mandated by the ASX where they're listed in Sydney, potentially not. I'm not too familiar with it, with everything there, but Ai-Media has been listed on the Australia Sydney Stock Exchange on the ASX for 1 year. So this is sort of their 1-year anniversary of being a listed company. They are transcription, translation, and captioning provider for live broadcast, for live enterprise and for recorded media. So these are those different segments that they serve. And we've spoken a lot about video localization, having published Slater's report on video localization. And so putting that in the context of our framework, they primarily— Ai-Media primarily focuses on the text-based services. And so that would be subtitling for the prerecorded and then live translated captioning and live capturing for, uh, in, in live settings as well. Um, they've got 3 quality tiers. Uh, so they have this automated, like fully automated live captioning, semi-automated live captioning, and this premium, which I think is— they describe as AI-assisted, but like, you know, human-led, human, primarily human, predominantly human. Um, yeah, so they do use AI, uh, to some degree in their operations. They've got a couple of different software products as well. So they've got LexiTM and SmartLexiTM, which I don't think is not specifically a translation memory, as far as I can tell, they describe it as dictionaries, so like smart dictionaries, one of them being human-created and the other being, I think, AI-generated somehow. But yeah, so I dug a little bit into their presentation and found out a little bit about some of the highlights that they've been well, that they wanted to draw attention to from this past year. I mean, the financials we already knew, so they'd already published their full-year results for 2021, which ended in June. So just to recap, they generated $49 million Aussie dollars. That's around $36 million US dollars in 2021, and that's growth of 87%. So super high single, double-digit growth there from 2020. And they completed 3 acquisitions, so some of this is, um, inorganic growth, um, 3 acquisitions of 3 US-based companies. There's Caption Access, CaptionIT, or Captionit, I'm not sure how you pronounce it, and EEG. Um, so, and they also raised money back in April to fund the acquisition of EEG.

The valuation is quite quite ambitious. I mean, if they're at 49, I mean, roughly 50 million Aussie dollars. And so, you know, um, and basically barely any— I mean, no, no profit.

Well, so they said they increased the gross margin throughout '21 and they achieved positive operating cash flow and profitability in Q4 of financial year '21. So they are I think they've kind of been operating, well, non-profitable, and then now they're heading towards, you know, becoming profitable by the sounds of things.

Because their market cap is $200 million, $200 million Aussie dollars, right? So 4 times revenue and basically infinite times earnings. Interesting. So there's still a lot of optimism, I guess, in the, among investors that they'll turn highly profitable at some point. Any other highlights here? Client wins, you know, outlook, stuff like that?

Yeah, for sure. So on the client side, they mentioned a few. So they talked about securing a 3-year contract with Sky News Australia. That was for 24/7 captioning of news program. They also said that they delivered captions for the Tokyo Olympics for NBC's Peacock platform. And that was using both their Smart LexiTM and the premium live caption options, also on Channel 7 in Australia. They also mentioned new customers include Al Jazeera Media Network and the UK Parliament. So they've gone further afield. They've also diversified their geographic client mix quite substantially. So in 2020, it was quite heavily sort of Australia and New Zealand based, like 70%. In that region. Now in 2021, it's more like 36% Australia compared with 47% North America. So as we know, they acquired 3 companies, have grown quite, quite a fair bit in America and North America. Sorry. And 17% rest of the world.

They—

one stat that they normally put out that I quite like is the number of captioned minutes. So they apparently captioned 16.8 million minutes in 2021, and they have a metric which is revenue per minute. So they said they're making $2.9 Australian dollars for every minute captioned.

Pulls out calculator and fact checks.

Hang on.

Yeah. I mean, we'll get there.

16.8. Makes sense because, well, it turns out it's 48 million. It's 48 million and they reported 49 million. So, you know, yeah.

Well, one, um, one other thing I'll probably just close on this point is that they are planning to— I mean, outlook is, is positive. You know, they say it's a growing market, it's supported by regulations as well as commercial tailwinds. And they also want to put an emphasis, like many other people, on developing and growing their SaaS revenues, um, from the various products that they own.

Got it. Okay, so before we go to another transcription/translation offering, Zoom, a technology that all of us know only too well. I don't know how many Zoom calls I have this week, but it's a large, large number. Before we go there, let's go to the world of game localization, which we talked about also in last week's SlatorCon with Riot Games, Arkan, but what's going on with Gameloft and Keywords.

Yeah, well, so another listed company that we track, Keywords, based in Ireland, listed in the UK, in London. Well, so basically high-level figures, group revenue, etc., were in line with the previous trading update. So just to recap around, what is it, €240 million-ish, which is around $280 million. They had organic revenue growth of nearly 23%. So they had around 30, 30, 38% growth overall, of which organic revenue growth around 23%. Pretty good on the adjusted profit before tax level. So they've got around $47 million US in adjusted profit before tax. And that has increased around 83% since the year, since 2020. Sorry, this is for the first half, I should say. Yeah, so it's a first half sort of recap and results rather than for the full year. So we're comparing versus the 6 months to June 2020. And then diving into the localization units, so Keyword operates, I think it's like 6 or 7 different business units or service lines, but 3 of them are specifically involved localization services. So there's audio business line that provides voiceover, it does voice recording, it does other things that we wouldn't necessarily normally factor in to our market size, like music and sound design and things like that. But yeah, subtitling— sorry, not subtitling, but the dubbing kind of component falls within this bit. So, audio is the biggest of the 3 localization service lines, so they generated around €28 million or $33 million in the 6 months to June 2021. Then there's the service line that's entitled localization. It's the second biggest of the 3, which generated €23 million in the first half, so that's around $27 million. US dollars. That's the service line where they do sort of the, I mean, I guess, standard game loc, the text-based stuff like in-game text, audio scripts, new cultural adaptation. They do translation or localization of the packaging and marketing materials as well. And this is where they house the proprietary technologies. So for example, Kantan, the machine translation.

I already forgot.

Yeah, and then there's localization testing. So this is the smallest of the 3 units that does around €12-13 million or $15 million in revenue in the first half, sorry. And that, in that service line, so they do things like they identify out-of-context translations or truncations like grammar, age rating, and it's really sort of, yeah, testing and I guess quality control and localization of the game. So all 3 of those units grew. Yeah, you're going to say something?

No, no, go ahead.

All right, yeah, I was going to say all 3 of them grew, which is good news. Together, they— I mean, they had had a bit of a tricky 2020, particularly sort of in audio, and I think they had some kind of operational constraints as well in the testing business— part of the business. But combined audio localization and localization testing, those 3 units contributed around 27% to Keywords' top line, so to the total group revenues for the first half. It has dropped, the contribution of those 3 business units. I think it was around a third sort of last year, but that's kind of normal, or it's kind of to be expected since Keywords, you know, they've sort of openly said that they're focusing more on other aspects of of their business, like the game development and the marketing side. So that's where their acquisitions are being focused as well. So yeah, combined they have about $76, $75, $76 million in localization-related revenues. And so that has— quite big. Yeah, and so it is, as much as the percentage contribution declined, it's still around 20% increase from the first half last year.

So two final questions. How's the CEO search going and what did they think about like localization in the second half of their financial year?

Well, so CEO search is very well advanced. That's what they said. And they expect further to do further announcement in the coming weeks. So should be announcement around the new CEO. In terms of how localization is looking for the second half, so they say It's going to benefit because there are content, well, new content that is reflowing into the later stage services. So, because they do the full pipeline, if game development and yeah, if the game development and sort of art creation is strong, they call that the early stage, then it's gonna at some point flow into the localization parts as well. So pretty strong, they think, in H2.

Cool.

Another company that's pretty strong is Zoom, and they just announced that their Zoomtopia 2021 conference— what a name, Zoomtopia, come on, guys— Zoomtopia 2021, releasing the news that they are adding live, quote-unquote, live translation feature. Shocker, because they bought Kites, the Germany-based It's not really a, it's kind of a university spinoff startup that does speech. Well, okay.

It's conversational. It's the conversational piece that I think was interesting in particular to see.

Yeah.

Yeah.

Okay. Yeah, because that was a hard company to unpack. I mean, they're very, not very well connected. Again, Germany, please open up a little bit. But anyway, Kites. So about Kites. Now they say they're going to launch live multi-language transcription and translation for Zoom calls. Platform will use machine learning and natural language processing to first, and now wait for it, transcribe the spoken language and then each participant will be able to translate it into their own language. So, and that's, that beta is going to be available this month. So that's kind of a, it's a big-ish announcement, but I mean, the feature itself is not huge. I mean, like, like live quote-unquote transcription and then just using MT. Okay. I mean, that's—

well, I mean, there aren't many, there aren't many platforms that offer that built in. I mean, do you remember we did the tests for the video localization report? So there's lots that do trans— live transcription in whatever language is being spoken, but there are very few that actually translate on the fly.

But does it translate on the fly though? Well, okay, we don't know.

Okay, that's what I'd understood by this, but we'll have to go in and test it when the beta's available.

Yeah, we're definitely gonna, I'm gonna be the beta customer right out of the gates, and they plan to provide real-time transcription for as many as 30 additional languages by the end of the year. So they plan to provide real-time transcription, so I'm not, yeah, we'd have to go back. Sorry to the listeners that we should have probably prepared this a little bit better. I don't know, maybe right now it's kind of, it's you just transcribe it, not, like real-time and then you just get the chance to click a button and have it emptied because now I'm reading here that they're also providing real-time transcription by the end of the year. So it looks like that's a bit further down the pipe. So, but anyway, I mean, they clearly understand that this is an important feature and this is of course competition to whoever offers kind of a more bespoke and I guess human-centric interpreting service, we understand.

Well, interpreting and live captioning as well.

Live captioning for AI media, et cetera. So this whole field is developing so quickly and of course, we had Fardad from CUDA on the podcast. I mean, for many, many, many, many use cases. Tom Livni. Yeah, Tom Livni. For many use cases, I mean, the automated part is not feasible at this point in time, but I mean, there's probably a fairly, like with translation, like written translation, there's a fair fair amount of like volume or use cases that you can use automated solutions for. So to talk more about this and many other things, MT, NLP, and general language AI, we'll hop over and talk to Olga Beregovaya from Welocalize.

All right.

And welcome back to SlatorPod today with Olga Beregovaya. Joining us today to talk about AI, NLP, machine translation, everything. Hi, Olga. Thanks so much for joining us today.

Hi, Florian. Thanks for having me.

Hey, Olga. Welcome.

Excellent.

Thank you.

Great, great to have you on the podcast. So, you're the VP AI Innovation for WeLocalize. So tell us maybe a little bit more about your background and your kind of route into the language services and technology industry. I mean, you've been in this space for quite some time, and we also said that you were the head of the American Machine Translation Association in like 2016 to 2018, but I mean, you've had a few hats on, so just tell us a bit more about your background and language industry history there.

I'm just a linear case where like starting the age of 3, I just knew that I will go into languages and linguistics and I don't know how to do anything else. I've never done anything else. Well, maybe just a little bit of a side gig into acting briefly, but other than that, I have 2 advanced degrees in linguistics and my first degree is in linguistics as well. My first job in language industry, well, I taught for a bit, but my first job in language industry was actually building lexicons for rule-based machine translation systems and off it rolled. So I got into MT, then I got into localization and then frequently as you get into machine translation, you just learn more and more and more of computational linguistics on the fly. And then I was in localization on the buyer side for some time. And then I was actually CEO of Prompto, Prompto Americas for some time. Building, driving building of privacy enterprise systems and deploying them with enterprise customers. So that was the machine translation, exclusively machine translation for some time, and then joined Welocalize and have been driving first machine translation and then something I want to talk about further, machine translation broadening into NLP, NLP broadening so ML, AI, and leading AI innovation for Localized for the past few years. So as I said, it's just been this linear progression doing nothing else ever.

Well, so you've got this lovely AI innovation title, which is fantastic, but I mean, what is it that we mean precisely when, you know, in your title, AI, what does that consist of? And generally, you know, help us understand. What is AI?

Well, see, that's exactly what I hope we get to talk about in today's podcast. What is AI? And obviously it was, have we talked about AI as much as we suddenly are talking about AI over the past 5 years and obviously more than ever over the past 3 years, if we go by general Wikipedia definition. We'll say that AI is a branch of computer science that teaches machines to mimic human reasoning and human behavior, right? Tasks that are generally executed by humans. That would be the generalized definition. Now, obviously, I work for a translation or global content transformation company, right? Right, one of the world's super agencies. So everything I do in AI applies to human language or human content processing. So what I do at Wellocalize is exactly this. How can we apply what would be referred to as AI as it applies to NLP, natural language processing? How can we apply it to global content translation? And this is what I basically do at Wellocalize. Looking under every rock and having the vision for innovation. Where can we take AI? Where can we take NLP? Where can we take ML? And let's talk a little bit about the differences. How can we transform global content, generate global content faster? How can we help our customer— customers achieve their global content goals faster, less expensive, how can we deliver on time, and how can we chase and predict that constant threat? So if that makes sense. And I know that maybe the concept of AI, there is a lot of mythology around AI and part of my job is actually take a lion chunk of that urban mythology and actually make it applicable. And sorry, Florian, I interrupted you.

No, it's fine. All good. No, so maybe let's just take a step back from the language industry, maybe because you've seen that progression over the years, like, you know, you mentioned initially like working on rules-based, right? So, but outside of the language industry, maybe what are some of the— before we go back, of course, heavily into the language industry, what are some other applications of kind of broader machine learning, AI, and NLP that you're seeing?

Well, maybe even taking a step back from NLP, let's look at broader applications of AI. And maybe first, you use AI and machine learning in one sentence, right? And a lot of, I think there is a tiny bit of confusion or a lot of times we think that machine learning and AI are interchangeable. I loved one of, I don't remember who said it, but somebody said that if it's written in PowerPoint, It is AI. And if it's written in Python, it's machine learning. And I actually love this. I love this. I love this expression. And maybe, and I hope, I hope I don't sound too controversial when I say that maybe AI has become a little bit of a buzzword where machine learning is the actual technology behind it, right? When we say machine learning, what do we mean? We mean actually techniques technology that enables machine to teach themselves, right? Reach a certain point and then self-learn, right? And evolve. And that's what we probably mean when we say machine learning, right? Whereas AI is machine's ability to mimic human behavior. And now in the modern world, we see AI everywhere, right? AI powers logistics, right? And if we read, for instance, about one of the giants, about Amazon, right? AI is behind their warehouse, right? AI is powering logic behind their supply chain, behind their logistics. So if we look at that, AI is like, we go online to buy tickets, right? And AI is behind selecting the best flights for us. So pretty much every area, matching algorithms Personalization engine, you look at the everything, all the personalized choices, you go on Netflix, right? You go again, Amazon Prime, all the personalization engines are AI-driven based on our past behavior and based on our previous selections. Recommendation personalization engines are all driven by AI power. Now, if we narrow it down, if we zoom into NLP, natural language processing. Then the areas that, again, I hope we touch on later would be natural language understanding and natural language generation. All the conversational AI and chatbots. We log on to our bank page, we interact with AI-enabled chatbots. So practical applications of AI really have evolved over I would say past 5 years and even more so past 3 years. And maybe they were a little bit blurry previously and not as evolved. All of us, I'm sure, have yelled at virtual agents when calling our banks, right? Begging for an agent. And now we see those technologies evolving. So I would say that practical applications are way more visible and pretty much omnipresent. I hope, Florian, that answers your question.

Definitely. We're going to talk about natural language understanding and generation, the whole BERT, GPT-3 in a bit, but first maybe for the language industry, like in the earlier days, just taking a step back, right? So what were some of the first kind of attempts at like using AI and maybe like NLP and like CAT tools or just let's take a bit step back in the history. I mean, because it— I remember from personal experience, you know, when we used some of those rule-based engines, it was super tough. It didn't really add any productivity gains. I mean, just tell us about some of the earlier initial kind of bridges where AI had a use in the language industry and how it evolved from there.

Right, well, you were kind enough to say you've been in the industry for X years, not saying how many years I've actually been in the industry for, but yeah, I was there in the early days of deploying natural language processing systems in the translator's day-to-day, right? And if we look back, yes, first and foremost, there was introduction of translation memory, right, based on fuzzy matching algorithm. So those were not attempts, those were actually successful, pretty successful implementations of early days natural language processing algorithms, right? We look at Levenshtein distance matching algorithm. So those were translators' first introductions to actual productivity gains from natural language processing, right? So we look at translation memories, and I'm sure you guys remember then there was attempt at evolution of translation memories, sub-sentential alignments, right? And substantial fuzzy matching when we looked and chunking segments and then matching segments. And those obviously used more advanced natural language processing algorithms. So we looked at the evolution. Then we came to the whole terminology management universe, right? And that was, that was coming pretty close to more advanced natural language processing because we started using character-based, frequency-based algorithms extracting terminology. And I would say that terminology, a lot of those terminology extraction techniques, statistical terminology extraction techniques are still in use today. So then, to your point, Florian, we started plugging in rule-based machine translation systems, and no offense to rule-based machine translation systems, and actually current Machine translation engines still benefit from parsers and from some rule-based legacy, but yet rule-based systems did not render humongous productivity, right? Some of the rule-based legacy we are still using right now. So if we look back, like say 15 years, there was the foundation for the modern translator universe ecosystem was actually laid back in the day, right? You harvest your terminology, you harvest, fetch from translation memory systems, and you plug in your productivity tools such as machine translation. So what we know today, although I have to admit I'm the one speaking different forums about the gains, the current gains translation memory? Is it going to survive or is machine translation becoming so good that we might not even need fuzzy matching anymore? And I'm still actually, I'm still struggling with that idea. Where is translation memory going? And where is the whole app service as we know it going? Is it going to give space to actual AI-enabled translator work surfaces? And if we have time, I actually, I would would like to touch on that subject as well. But Florian, to your point, early days were the precursor to the modern-day AI-enabled translator environment as we remember it. I don't know.

How transferable is that expertise or was that expertise? Like from, like you mentioned that some of those parses and some of the rule-based, like the approaches to rule-based are still like useful today. What about the skills? So people that have been in this space for 10, 15, 20 years, how many times did they have to like reinvent their level of or their set of expertise, or was it kind of like just gradual? So if you knew rule-based, it was okay with statistical, and then some of this was useful for neural and whatever we're doing today, or was there a complete reinvention of the skillset?

In terms of the CAD work surface, and I mean, we're all exposed to CAD work, right? We follow the evolution of what the CAD environment, what the CAD environment, like the TMS work surfaces of standalone CAD environment look like. Frankly, and I haven't translated much in my life, but I follow the evolution of CAD environments. I would say that they look very similar from what we had known. I mean, obviously the UX is evolving, right? The whole world's UX is evolving. The Microsoft ribbon as we knew it 20 years ago has changed tremendously. So the CAT environment itself probably is pretty transferable. If you started translating 30 years ago, you evolve with the UX experience, but the whole cat experience might not have changed much. If anything, it's becoming simpler. The more load AI takes on, and again, something I would love to touch on is the whole role of human in the loop, how the role of human in the loop is changing. But in general, your cat interaction is not changing much. Now, how you interact with MT engine is definitely changing. Actually, at Will Localize, one of the groups on my team is the program management group, and they chase the evolution of MT because what you did with rule-based and how you interacted with rule-based output, right? It follows the syntax, right? It plugs in the new vocabulary as you customize the engines. You did not train rule-based, you customized, but you have more or less robust syntactic and morphological structure. And then statistical becomes way more random and way less predictable in morphology. So they needed to rewrite their training courses, right? And say, Tao's training courses were changing as well. And we actually contributed to Tao's training courses. So, okay, rule-based courses, they rewrite and they retrain our post-editors. And neural, again, fluency is there, but As we know, neural can generate pretty random meaning. And then suddenly you have your fluency, but what happens to post-editors with neural post-editing, you can be completely misled by the fluency and completely omit that your technical text is finished by, and the crocodile went to the river, right? And then you need to catch it. So the way you post-edit neural output is pretty different. So Florian, Yes, you have your post-editing and your interaction with a platform skill set, but you do need to reinvent your skill set quite a bit as the AI technology evolves. Not only is it my opinion, but it's actually empirical evidence from the field.

I think if we're thinking or delving into the human-in-the-loop workflow a bit more, whether that's post-editing or some other form of human in the loop. I mean, where do you see that heading? So you're talking a fair bit about post-editing, but is that the final stage? Is there other new different ways that we can interact as humans with machine translation?

Not just machine translation. Let's cast the net a little bit broader if we want to. It's the overall injecting AI. And Esther, you asked me what my role is at The Local, and that's exactly it. It's injecting AI into the global content transformation ecosystem, right? And I have to give credit to Alon Levy, formerly of Staffava, right? Then Amazon Translate and now Unbabel. Hey, Alon, should you hear it? The credit goes completely to you. The first time I heard the notion of content drift was from Alon when we were talking about training and retraining machine translation systems, right? And dynamic retraining of machine translation systems. They cannot stand, they fail because content is evolving, right? And your machine translation systems not only should play chase, but should actually predict where the content is going. And be prepared to handle that content. I would take it further and say that AI applications as they apply to natural language and human language, actually AI tools and AI applications should be there. And that's what I would say is happening to the world of NLP within AI as a subset of AI should be there to be able to capture and handle those new content types. Had we known how much data is needed? Did we know how much data is needed, for instance, to train conversational AI? So back to the human in the loop, human is not only interacting with machine translation, human is now interacting with annotation tasks, right? Post-editing and validating outputs of summarization tasks. Humans are, human professionals in our field, in language field, are interacting with outputs of AI in variety of tasks. So I would say, Esther, and I hope that's a valid answer to your question, professionals in our field, that whole profession, job of language professionals is evolving in so many directions, and since AI quality, the quality of AI output is still lagging a little bit, or not even a little bit, right? And as we get to touch on, and we can even say with the quality of machine translation, right? The quality will always be lagging or utility, right? Because quality is arbitrary. Utility is practical. How usable is your machine translation? The output of your summarization engine, the output of your sentiment analysis, the human translator or post-editor or let's call it language expert will always be needed to compensate for that lag. And this is the role that I would see. And this is the evolution, Florian, back to our previous conversation, the evolution of this platform. Will it always be the CAT platform? Or will it, or will it become that work surface that provides facilities for a human professional to interact with outputs of various NLP applications? So Esther, this would be my broader view where it would be going. And maybe back to MT, there is so much talk about human parity, right? All MT developers are talking about human parity. Yes, we still get a lot of feedback from the field, even more so when we are talking about domain-specific customized applications. Yet it sounds like human parity is still lacking when we talk about high visibility, high impact content. So again, very long answer. And guys, please, please Please do feel free to cut me off. I have a very hard time stopping.

It's super interesting. I think there was one thing that you mentioned there that I'd like to pick up on, and it's this idea of utility. So, you know, how useful something is, but what I want to know is whether you sort of see any NLP applications that are currently untapped. So applications for the language industry that you think we could really leverage, benefit from, I don't know, you mentioned Summarization, for example. What do you think has potential but is just not being utilized in the right way?

Within the language industry, within the language technology?

Yeah.

Uh-huh. Okay. So I think if we talk about say, if we take the translator occupation, right, the occupation of translator as a language professional, I would say that as of right now, for translator productivity, I think that if we just say transforming content from one language into another, I would say for translator productivity, as of right now, we are pretty much there if we just take subsets, localization and translation. From here, we should just be evolving the tools, right? Referencing terminology, enhancing translator experience when it comes to how terminology plays along with machine translation. Again, I'm not sure what the role of translation memory going forward is going to be. You probably know that a lot of companies are actually trying just ice matches and 100% matches and everything else goes to machine translation, right? And not even, not even giving space to fuzzy matches. So just evolving machine translation further, maybe thinking along the lines of what ModernMT is doing and Lilka doing in terms of dynamic training. So those things is where I would see evolution again, maybe plugging in more AI interaction into CAD work surfaces rather than making them traditional work surfaces, like predictive typing, for instance. I think it has huge potential and predictive guessing in the work surfaces, like things like this, making it more— making the actual work surface more AI-enabled. But that's when it comes to the actual translator occupation. Now, what I think is under tapped is services we as agencies, we as LSPs could be providing to our customers and things that our customers from the buyer side could be thinking about when it comes to multilingual global content or technology ecosystems, which I think are underused. And those would be, if you think, for instance, about digital marketing, multilingual digital marketing, or market analysis, what I don't see still enough of is multilingual sentiment analysis. I see very little of it happening. We do get some requests from the customers and we are having conversations with customers. Hey, We as agencies, we have those capabilities. We have learned enough about NLP to actually provide multilingual sentiment analysis services. Think how much you can learn from the field about how people in Malaysia actually feel about your— I don't know. I don't know what it is. I know peanut butter. I'm just thinking of—

Peanut butter. Yeah, why not?

Anything. Anything. I use peanut butter, but I don't know, Florian, what? Feel about your chairs, peanut butter, you name it.

Sure.

Television. I don't know, anything. But think about feedback from the field that you can actually get or internal communications or even a role of raw machine translation, I think, is seriously underutilized. We've learned to plug machine translation into our translation workflow, but all the work that we could actually do plugging in machine translation into corporate chat. So there is a whole gamut of NLP applications, say summarization, right? Just going into, for HR, going into employee comms, summarizing them, and instantly translating them for HR processing. So if we look at the whole spectrum of NLP applications, I think there is so much that translation industry can do that is not being done yet in terms of content analysis, not even as much content generation, but content analysis and processing that we're just scratching the surface with. So The services are there.

We can do it all.

We're just not doing enough of it. Yeah, go ahead, Esther. Sorry.

Sorry. Yeah, no, something I wanted to get your take on maybe just quickly so we can discuss some of the other language models and things, but is this idea of, well, the concept of chatbots, because you were talking about, you know, banking chatbots and things. So I mean, just briefly, how does one go about making chatbots into these multilingual chatbots and what kind of training data and requirements are there to be able to do that?

So that's another thing that, as I said, we didn't even think that we would be doing it, right? 10 years ago, who in our industry Let's first approach it from the service perspective. Or actually, no, let's— approaching it from the service perspective would be backwards. Let's approach it from the client perspective. We were building, and again, that would reveal how long I've been in the industry, huge user manuals, 1,000 pages user manuals, right? First, and online help, multi-page with super crazy complex links, online help systems, generated stuff, right? Then we go into single source, the blessing of single sourcing where you generate, and I'll get to chatbots. I'm not, I'm just starting from afar and then generating and then single sourcing. And then our customers realize that, well, we still can produce online helps, although then they transform into wiki pages, right? And then they're getting simpler, simpler, simpler, and shorter. And then they become interactive and then they become, they just become shorter and simpler. And then the realization comes that actually the same customer service, customer assistance experience can be much more interactive, much more usable through the help of chatbots, right? And the whole customer interaction is your tremendous online help systems and your— I'm not saying user manuals do not exist. They exist. Have they gone from this to this? Yes, they have. And then they started giving place to chatbots, right? More and more. And then you can log on your whatever you have bought yet again, whatever you have bought. I'm not gonna, I'm not gonna— let's just imagine you bought something and now you can log on to the page and then you have your virtual assistant, right? KBs still exist, but how many knowledge bases actually have now been converted to your chatbot experience. Had we known that instead of building huge online help systems, you actually will be developing training data for chatbots and how data hungry those chatbots are going to be, and that will become the whole line of service for us and a whole new way we would be providing services to our customers. We did not, right? And even more so did we know that those services would become multilingual and the chatbots would equally be data hungry for English and for global chatbots. Now, how do you make them multilingual? There are two schools of thought there. You can use machine translation, right? And there are multiple chatbot systems where you build your chatbot in English and then you plug in MT. But there are many, like for instance, I recently dealt with a Japanese request where it quickly became obvious that for Japanese, you just do not get the same level of user engagement with machine translation that you get with English, or you build equally and not even translate, but you build your, you rewrite your scenarios and intents and you build a whole library of utterances for the language you build your chatbot in. And basically you rebuild your multilingual chatbot from scratch. So there are two approaches there. Both generate pretty substantial amounts of data to train your chatbots, to train your global chatbots. But again, you see many companies that do nothing but generating and annotating data for global chatbots. Or super agencies like us ourselves have a whole line of business that actually does nothing but generating and annotating chatbot data. Time may come when chatbots will be trainable on unlabeled data. But that time is not there yet. You do need annotated data.

Yeah. So I recently spoke to some someone where they're using some of these text generative models like GPT-3, BERT, whatever, all these new models, especially now GPT-3 to create kind of artificial data, like mass artificial data to train some of these systems like chatbots, but that's just kind of a segue to what are your thoughts around GPT-3 and these models that are out now? I mean, You know, we had Jochen Hummel, you know, founder of Trados. He said it's a big deal. What are your thoughts and how do you use it maybe at a large agency like WeLocalize?

Oh, okay. So first, where do I start? Okay, first. It's a big one. It's a huge one and yeah, so first and foremost. January, I mean, NLU, right? Chatbots equally operate in NLU and NLG space, right? Natural language understanding and natural language generation. And there is a passionate debate which one is harder. Is it harder to comprehend human language or is it more difficult to generate? And so far, the sentiment is that understanding is more difficult. Generation. Uh, right now what I'm aware of is the synthetic data. Synthetic data is a huge deal, most definitely, right? But synthetic data is generally used to augment human data, right? So generated data in the modern systems— and is it a threat? That's a huge question, right? Huge question mark. Is it a threat to our industry that synthetic data will take over? But is it a threat to humanity that AI will take over, right? We can, we can be solving a much bigger problem here. So far, synthetically generated data is used to augment human data and then similar for machine translation training. And then that humongous corpus of human data augmented by synthetic synthetic data. Yes, there you can get pretty good results with data augmentation. But so far, it is more of an enabler than a replacement. And still, to validate the quality of synthetic data, you still have some room for human in the loop. In some instances, some instances, just synthetic data. So we talk more about data augmentation than the actual data replacement. Luckily for us. Then there is the whole paradigm of self-annotation, self-labeling. That's something that we are actually looking into as well. Can— how much human labeling do you need for the machine? And I'll get to, I'll get to GPT actually closing. How much human labeling do you actually need before the machine actually learn to annotate as well. Now, so part of it, so part of my answer, Florian, is complete replacement, maybe not as of yet. Augmentation, a definite yes. Now, my thought around initially BERT, and then we get into GPT. So if you think about what BERT was created for and most common applications of BERT. Enhanced Google search, right? That was the initial application. Roughly 10-15% of Google search algorithms rely on BERT. Summarization, some text generation. So BERT was not— probably the applications of BERT were not as aggressive when it comes to actually replacing human corporate generation. Now, obviously, GPT, GPT-2 was kind of a fairly big splash. Now, obviously, GPT-3 was a humongous splash. That's where we started talking about it writing poetry, right? It generating human-parity-level text and solving problems, big questions. Question mark. They tested it on 12,000 token text and it solved roughly 3% to 6% of problems. So problem solving, we leave with a big question mark. And the potential and the constraints of GPT. So I personally would be a little bit skeptical about GPT-3 coming in and solving all the problems of the humanity, and even less so problems of, say, chatbots and using GPT-3 as a practically applicable response system and text generation system. And a lot of And I would say I'm not alone in this. And I don't know, I might not be once published, I might not be making that many friends out there because I know that GPT-3 has a lot of proponents. It has huge potential, most definitely. And we're reading from OpenAI, Ilya Slatkov said that OpenAI GPT-4 will be trained on Trillian. Roughly trillion parameters and using some completely crazy, not even GPU, but I forgot what the W is, some, some completely crazy hardware behind it. So GPT-4 will likely have many more practical applications and practical capabilities, but I would see much more narrow application of the current GPT-3, maybe more around things like code generation, like the way Microsoft is using it. We know that Microsoft is using it for writing pretty substantial chunks of code, right? Maybe mathematical problems, maybe things like this, but plugging in GPT for solving human content problems, So far, again, to the best of my knowledge, so far, there are some success cases, but the failure cases are so alarming that I would say that we should be, should be pretty cautious. I mean, when the system tells you, yeah, right, you should go kill yourself. Maybe that's a warning sign.

Don't do that.

We should. You didn't, you didn't read about that one? Like, yeah, yeah, maybe it's a good idea. Now, I'm— and again, I, as I said, I might not be making that many friends, but I'm following it really, really closely. It can, it can generate extremely human-like text and extremely human-like response. But then you inject some questions, some content that it's not prepared to answer. You can basically, you can take it as tree fairly easily. And then the randomness and the bias of the response is going to be so high that it becomes dangerous before it becomes useful. And it takes very little.

I mean, it's good to have a bit of a—

so, but yeah, go ahead, Esther.

I was saying it's good to have a bit of a reality check sometimes on some of these technologies and to kind of remind ourselves of some of the dangers. As well as obviously the hype and the undoubtable potential that there is around there. But I wanted to ask one question, maybe just to close on, and it's kind of asking you to maybe get your crystal ball out a little bit and think if you had to identify, which I know people don't like necessarily, but I'm bound to ask, what are the key trends do you think in language AI that you know, as a super agency, we at Welocalize are looking at, what do you think is a really important sort of 2 to 3 trends for the coming years, coming 2 to 3 years or so?

First, as I said, most definitely, we're definitely looking forward to increasing translators' productivity. With advancements, further advancements in machine translation and for machine translation to coming closer to human parity and for neural machine translation to getting more enhanced when it comes to actually capturing meaning and accurately capturing meaning of the sentence and actually, yes, so I would say first and foremost advancements in neural machine translation And even more so when it comes to long-tail languages, because right now we're still lagging a little bit. And when it comes to, uh, economy is not staying steady, right? And the expansion into more remote regions automatically drives demand for translation in long-tail languages. So expansion of MT into long-tail languages and domains is definitely something that we're very interested in and things like, say, zero-shot translation between language pairs other than English is definitely something that we're definitely very, very interested in and following very closely and subsequently skill sets in translators from between various translation directions from skill sets. So machine translation is a definite and trainability of engines and dynamic, as I said, dynamically trained, not pre-trained, but dynamically trained machine translation and tight integration of machine, dynamic machine translation into what I would imagine should be NLP and AI-enabled work surfaces. Then obviously let's look outside of the box and let's look outside of just NLP, AI in smart resourcing, AI as a part of the overall digital transformation in our industry. AI by itself, if we're just myopic looking at NLP, it is just a tiny chunk of what we can do in AI, with AI in our industry. We started talking about AI driving supply chain, AI driving logistics, Same should be happening in our industry. And again, I don't know if I'm okay to mention names. I'm a huge fan of what, say, Unbabel are doing with their whole resource assignment and the whole, uh, crowd management. And that's something, it's just an example. I think it's an awesome example of, uh, what the industry should be looking at, at large, uh, assigning resources, selecting resources on the fly. Managing the whole logistics and managing like AI-enabled platform. And that's definitely something that we're doing at Will Localize and looking to further expand at Will Localize. And I know that a lot of other agencies, a lot of other companies are looking at the whole digitization— okay, an ESL issue here— digitization and injecting AI in overall processes. And then, as I said, data. It's kind of self-nurturing system, right? AI craves data, and you use AI to generate data. So the whole concept, Florian, something you touched on, which is very dear to my heart, synthetic data generation, using natural language generation to create more data to feed into AI systems. But then, Esther, to your point, not to forget about the role of human in the loop. We still need humans to generate the initial datasets to further generate data to create robust AI systems. And then one thing we did not get to talk about, but that's one thing that remove the bias and make generative models less biased and more accurate. And then inject one thing, inject reasoning into those language models, right? Right now they cannot reason and we want to teach them to reason and then we can use much more AI in our languages. So this much for Olga's crystal ball.

A lot of food for thought and thanks so much. That definitely lived up to the high expectations I had for this podcast. So Olga, thanks so much for joining us today. Really appreciate it.

Thanks, Olga.

Thanks so much for having me, guys.