#24: What Makes Machine Translation Work? - Olga Beregovaya — The Deep View: Conversations

When we talk about AI today, we're largely referring to generative AI, which is built on the backbone of large language models. The keyword here is language. So these models stretch their roots back through decades of research in natural language processing, whose goal was to advance work on machine translation. Now, LLMs today are used for a variety of purposes that far exceed translation alone. Some companies, like the enterprise translation company Smartling, use AI and machine learning and large language models and small language models for translation purposes. And my guest today, Olga Beregovaya, Smartling's VP of Machine Translation and Artificial Intelligence, is here to break down how it all works, what systems are being used, how they're trained, how they're verified, how they're validated, what the challenges are, what they're able to do, and how it has all changed and evolved. Over the past several decades. If you're new here, welcome. If you're not new, welcome back. Thanks for being here. This is the Deep View Conversations. Olga, thank you so much for joining me today.

Thanks so much for having me.

So there's There's so much to get into. This is a really interesting— I'm excited for this episode and kind of a deep dive into linguistics and the relationship between linguistics and artificial intelligence and machine translation. There's a lot there. But I do want to start with you and kind of you got your start in linguistics and now you're working in AI. And I'm curious about the kind of inspiration for you behind you know, jumping on the linguistics train to begin with and then jumping from linguistics into artificial intelligence work?

Okay, so as far as linguistics go, I guess, I mean, I started studying English and a couple other languages, actually English at the age of 3 and then a couple other languages right out of high school. So then I started, and as I studied linguistics, I guess getting into the structure of languages and language families was the inspiration behind me because I could have gone multiple directions from there. You could have been a teacher, you could have become a translator, you could have become a lot of things. But I started as I was studying linguistics at the university. I got very interested in how language families operate. There is this famous language family tree by Ferdinand de Saussure. And when you look at it, it's like, oh, are they all really Proto-Hindu-European? Do they really all stem from one root? And then in that case, how did Slavic languages and Germanic languages then gravitate and all those directions. So that was, I guess, the main inspiration for me to stay in linguistics. And from there, when it was time to choose, to make the career choice, it was, okay, how about I try myself in applying what was natural language processing principles at the time to different languages? So that brought me to rule-based machine translation, which was the state of the art at the time. And that's where I started getting more into relationships between mathematics and what you could describe as data science at the time, I guess, and natural languages. And that's where all the linguistic parsers were written, right? And that's where you started using the Interlingua. And then the trajectory was kind of, I don't think it was a choice at that point. If you wanted to stay in the field, it was kind of inevitable, right? Because then the natural language processing made the leap from rules to statistics and then from statistics to neural networks-based NLP. So that's, that's the path. I mean, you just cannot avoid it, even less so in the modern time and day.

You cannot avoid it. This is true. I want to kind of unpack something that you mentioned about the evolution of the architectures that machine translation has dealt with from when you started the kind of the rules-based machine translation to statistics and then to the neural networks that we, we use for it today. You know, since you've seen these changes and these evolutions and worked with them, how are they different from each other? I know structurally we're talking about different architectures, right? But in terms of the end result of better machine translation, what does one do that the previous one couldn't.

I think we should throw LLMs into the mix, if you don't mind, because that's kind of, you know, it's probably getting to state of the art, although I have some comments there. So the output of rule-based machine translation, right? You build a parser and then you map the structure of one language to the structure of the other language. First of all, to raise a translation direction in rule-based machine translation was an effort of a good year, if not two, because you need to populate the vocabulary and there is no worth training. There is really, all you can do is populate the lexicon. Now imagine yours. Stuck within this robust structure that has no idea of the context, no idea. It's really, I mean, it operates on like collocations and, well, what you would now describe, I guess, n-grams and sentence levels. So the output was, at the time of rule-based machine translation, we started using rule-based machine translation for machine translation post-editing, human-in-the-loop tasks. Apologies to everybody who've been involved in rule-based machine translation to date. I think we owe translators an apology for making them deal with the output of rule-based machine translation because it was clunky. It wasn't really, it was subpar. Probably those of you who've been around long enough probably remember the AltaVista Babelfish, which then later to become, I guess it was Yahoo Babelfish. Those of you who've been exposed to it probably have seen the quality of the output. Then we have a major breakthrough in statistical machine translation that operates, again, that really operates on n-gram levels and can use multiple n-gram models. That's where you saw a major breakthrough. The translation became much more fluent because it learns from the patterns within sentences, and then translation became, I guess, more accurate, more accurate and more fluent. Again, you could raise, this is where you could actually easily raise a single translation direction or bidirectional model within a matter of days. So then it also— the evolution of rule-based machine translation, I think, kind of stalled, although there were a lot of very interesting, and still are very interesting, models that combine the rule-based architecture with a statistical overlay. Yes, statistical overlay. And for instance, I was involved in one of those efforts at a PROMPT, or prompt, however you want to pronounce it, and you could get the best of both worlds from that hybrid architecture. And then off we go, there comes 2016, I believe, right? And attention is all you need. And that's where we graduate, you know, everybody knows that. Yeah, there come our friends Transformers and that's where things get in equal parts interesting, but just a tad less predictable, right? Neural machine translation is definitely superior in quality to its predecessors, but then neural machine translation started hallucinating and delivering unpredictable outputs. And there was a notion, that's where the notion was introduced of false accuracy or false fluency, where you basically start a sentence with, here is a construction crane and the crocodile went to the river, and you basically have two in the same sentence for no apparent reason because that's how the model learned. And the other thing was neural machine translation, which still seems to be more state-of-the-art, and I'll talk about it in a minute. Is that neural machine translation operates— most of neural machine translation systems operate on a string level, so subsequently within sentence boundaries. You know, there are some attempts to operate on a segment, sentence level, and document level, but in majority of cases neural machine translation is still constrained by sentence. Now guess how well it does with context? And then now, I think the last couple of years, probably the last 2 years, this is where people started using large language models for translation, and that's where things get really interesting. Neural machine translation is known for accuracy but somewhat lack of fluency. LLMs are exceptionally fluent in producing translators, but the hallucinations are really surprising. Like, you basically drop your content into something that resembles a black box, and then whatever comes out, and then it's a question of how well you manage it. And yeah, I guess, I guess that's the evolution. So you go from disfluent, hard to, hard to raise, hard to develop to something that seems to be extremely fluent but still has to learn, still has to learn the accuracy.

It's a really interesting evolution and particularly around like the false accuracy and false fluency. If you use any of these systems, I mean, I run into this very regularly when I'm like playing with and testing these models where they output text that looks very fluent. But if you read it or if you follow the links that it puts out, often it seems even at the level of state of the art that we're at today, it's nonsensical or it's not grounded in any sort of truth. And then we have to go in and we have to ascertain what's real, what isn't, what are you just saying? And it's really interesting. I wonder, in your kind of day-to-day operations, when you're dealing with state-of-the-art, when you're dealing with these models and kind of neural machine translation, how much your grounding in linguistics, how much that the fact that you come from linguistics is very helpful in working with these machines? Like, do you use that knowledge regularly?

Well, first of all, I absolutely should not be stealing credit from my team, right? There is a team on the ground that actually deals with the day-to-day. I guess, well, I'm definitely very much involved, but there is a group of equally data scientists and language engineers and linguists behind the scenes. So let me talk a little bit about where linguists play a role. And, you know, there is a lot of chatter about, is AI stealing our jobs? Is AI creating new jobs? The reality is that linguists this time and day need to reinvent themselves, I'd almost dare to say, like no one else, or like very few other professions. And the reason being, where you used to be, for instance, a professional translator, uh, now, yeah, first of all, you mostly let AI do the heavy lifting, and then you post-edit, fact-check, to your point, you post-edit the output. But most importantly, linguists these time and day need to be able to design prompts, even if those prompts have been templatized for them, validate the output for their language. I have no idea how the prompt is going to perform in Pika language, Swahili. I just would not know, right? So that's where you need linguists to design a prompt, and it can be as simple as thumbs up, thumbs up, thumbs down. And then once the thumbs up have reached a critical mass, you actually allow this prompt to go into production. Or you actually, in the ideal best case scenario, and that's what we ask for and that's the most sought after talent, is when linguists can also test the model and interpret the single signals from the testing. So when you show blue, meteor, comet, f measure, and anything else under the sun to a linguist, in ideal universe, the linguist should be able to interpret those single signals and continue iterating. So I guess that's the— And that's what we see quite often. And that's where I see like here at Smartling, we see quite often that engineering teams on our customer side often implement LLMs on their, on their side, on their end. And quite often they would come back to us and say, hey, we get performance on language X and A and B, but when it comes to XYZ, we actually cannot get the right performance. So I guess that's where linguistics actually, uh, means, uh, data science in the modern time and day.

Running a small business while keeping up with AI? Salesforce's Agentforce is bridging that gap with production-ready autonomous agents for sales, marketing, and customer service. We're talking real agentic AI here—not just chatbots, but agents that can actually execute tasks, make decisions, and scale your operations. The early results are compelling: businesses are seeing immediate efficiency gains and cost reductions without the typical AI implementation headaches. Salesforce has done the heavy lifting on model training, safety guardrails, and enterprise integration. If you're curious about practical AI deployment for small to medium businesses, check out their guide on AI agents. Head to tdv.co/s-f to see how autonomous agents are actually performing in real business environments right now. That's t-d-v.co/s-f. I have a whole bunch of questions just about like that exact point that we kind of jumped into. And one of them that I'm just like curious about, this is something I think about, in the different avenues in which translation happens. I think of live translation, for instance. You know, when like a politician is making a speech and they have someone kind of off in the corner doing the American Sign Language version of that speech, and they have like the idea of live translation. To the extent that, like you're saying, linguists need to kind of reinvent themselves and there's a role for them in analyzing outputs of these models, being the kind of editor of output. I wonder, is there anything like if we removed that human component, if it was just a language model as being a translator, is there something in the translation that's missed Like, is there a piece of nuance in conveying tonality, for example, that humans just have a better grasp of than a machine? Or is this a complete non-concern?

It absolutely is a concern, but maybe to, um, talk a little bit about the landscape there. It really, it's all about the content sensitivity to errors, right? And since we just, you and I just established that models tend to dream things up and hallucinate and lack factual accuracy. Again, if you are like, say you are an opinion portal, right? You are an opinion portal and they're okay, well, what's the worst that can happen? Okay, well, you kind of picked a hotel and kind of did not quite get the right information about the menu. It is still very, very, very important, but nothing like medical translation where your sensitivity to inaccuracies is tremendous. right? Or all of regulated industries. And maybe that's why if you look at the trajectory of how different verticals adopt fully automated, no human in the loop translation, you would definitely see a trend there, right? Like chatbots now, or as I said, opinion portals or short shelf life content such as blog posts. Again, if there is no possible repercussions there, then you are just fine with automated translation. Maybe I'll talk a little bit about training and fine-tuning, which helps you increase the accuracy. But the more— the less— the lower risk tolerance there is, the more we definitely recommend, at least for the time being, we definitely recommend human-in-the-loop for fact-checking, removing things like when the model delivers an explanation instead of translation for no apparent reason. And no, we deal with it quite a bit actually. I was just talking to a colleague about model hallucination as they were delivering a webinar and he gave me some extraordinary examples. So factual accuracy mistranslations are still definitely there, omissions and things that again for things like regulated industry or government are just detrimental. You just cannot allow them. Again, a lot of techniques are in place to mitigate those.

And it brings up an interesting point. So you said, for now, right? And so Smartling uses this kind of human-in-the-loop approach, and I think we have nailed the point on why that is necessary, especially your point about omissions. You know, that's a huge thing for me as well. But I wonder how the work that Smartling's translators do has changed over time, even in the last couple of years, as the models have gotten better, as we figured out different ways through fine-tuning, like you mentioned, to make them more accurate. I guess if you look at the, the kind of gap between what the language model did and what the translator did, how has that kind of shifted? And do you expect to see a point in time realistically that the human won't be needed in that loop?

How much time do I have? Let's just check.

As much as you'd like.

Okay, well, let's check back in 3 hours. I mean, it's such a huge topic. At Smartling, we have a grid, and the grid is translation for every use case. There are cases where it can equally be just fine with just an output of neural machine translation or a single language model, right? Be it a fine-tuned language model or be it just a generalized foundational model. And then there are cases where we do something that's probably— I'm sure you're familiar with the notion of automated post-editing, right? Which is where you basically harvest the best of both worlds or do a two-pass LLM translation where you do things like validating or reinserting the glossary, repairing leverage, and do smoothing. And then this same content again for more sensitive the same output for more sensitive content will be fed to a human translator. Or at this point in time, again, I guess it's human in the loop because the translation job really needed to reinvent itself. So there is still, I don't know if you would be familiar, it's probably more linguistic phenomenon, multi-dimensional quality metric or MQM is something that we use in our, universe to allow humans. And now actually there are also automated ways of automating this automated language quality assurance. And actually, if you take 100, which is barely attainable as the ultimate dream, then human in the loop would take it to roughly 98, 99. Automated machine translation with automated post-editing, whether it's done by an LLM or neural machine translation, can easily hit like 95, 96, 97, depending on the language group. And pure machine translation would like rest within 93, 95, again, depending on the language group. So the input to translator, again, just like the evolution of machine translation, the input to translate, the input that the translator receives has changed dramatically. Now, the way we measure it, there is a translation error rate, right? TER, which is also known as translation edit rate. I still don't know which one is the right expansion because if you look it up, both are rendered roughly in the same percentage. So we definitely see what is translation error rate or aged error, which is performed within the platform. You basically measure how much it took a human to achieve the human quality translation between the automated translation and the human effort. And we definitely see that TER delta shrinking. We definitely see that the translation edit rate is shrinking dramatically. And sometimes this is also a great way of measuring human parity. Sometimes we could see up to 70% of sentences not even being touched. Or when you start measuring ter, just like a character or a whitespace was changed. So obviously the level of effort is decreasing. Now ter is I mean, it's either Levenshtein at a distance or like slightly different algorithm. It's mathematical measure, right? And it's always referential. So just to make sure that we're doing the right thing, we're also doing, using semantic metrics like Laser and Labsit to make sure that we equally capture edit nature and edit distance. And again, we see all of those shrinking and semantic similarity going up. So that again, back to the role of a human in the loop, right? I mean, we're gonna be, The delta is shrinking and that's where we are talking to, for instance, kids in college that reinvent yourself, think data science when you think about translator job.

And, and well, and in that realm, those kids in college that you're talking to, if they do focus on that aspect of it rather than pure translation, as that delta shrinks, is there a risk that at the other end you know, in a couple of years, the focus on data science for linguistics and translators doesn't even help. Like, does the job— does the human job go away entirely or just not really? Sure.

Well, I think it fits. I mean, it's not just linguistics. It's not just translators, right? It fits into a much bigger scale debate about where are human jobs going. Right. And it equally applies to technical writers, like creative writing. And actually, maybe eventually we touch on generating target content without even needing the source. So I really want to tread very carefully in this territory because there is— there are two sentiments in the world of translation and linguistics. One, that eventually here we are, here comes human— full human parity, here comes singularity as we, as we would see it in the translation universe. And the other school of thought is there will always be a role for a human in the loop. But probably more on the model training side than on the actual post-editing. So at some point we expect accuracy. So when people go into linguistics or like say the Monterey Institute, right, the MISE, we usually advise them polish your math and data science skills because we're really, that's probably where the world is going. But do we see, do we foresee that post-editing and fact-checking will, might not be in the picture 5 years from now? Probably a strong yes. Interesting. Maybe one other thing I can add, a lot of money is still being spent on supervising and labeling data for, to train multilingual language models. And that would probably be the last thing that might, that would probably be something that's gonna last longer than maybe other linguistic tasks, right? Because unsupervised learning is only getting us this far and until it really catches up with supervised learning, probably, A lot of money will be spent there.

To me, like the big, the big inflection point, I guess, is if we could somehow, if hallucination, which if you're a listener of the show, I, you know, I don't like that term, but hallucination, if we were able to eradicate that verifiably and validate and know for sure that a model wouldn't hallucinate. Then I would find the more disruptive, uh, you know, broad and focused on linguistics job disruption more likely. In any environment where that can't be eradicated, I, I don't, I don't see the human element going away, especially as you mentioned in higher risk or higher stakes environments.

That and developing human quality, high quality datasets for specialized purpose-built models. That's again another thing because you can only get this far with scrubbing entire internet and then assuming the best. So definitely it's, I don't know, what's the better term if not hallucination? Let's use a word of your choice, whatever you prefer.

I've heard a lot of different ones. I mainly, it's, I just don't, I just don't like it. I'm wary of like anthropomorphizing what the systems actually are. And hallucination, I, I think that there's a larger problem to just completely run this tangent. Um, in industry where the industry uses terms that in industry make sense, but when you're communicating that to people who are not in it, it means something else. And so a hallucination means something for people and what's going on in the language models when they output incorrect information is not exactly the same thing. So I've heard a bunch of different terms, even like confabulation was thrown around, but we don't love that one either. Some people propose the idea of Frankfurter BS, which is kind of based around this notion that LLMs themselves can't distinguish between something that's true and something that's untrue. So it is Frankfurtian bullshit. And it's all semantics. But when we're talking about language models and we're talking about AI, which is kind of rooted in, at least the AI of today is rooted in natural language processing and language is a very big part of it. I find the semantics to be important.

I see, I see your point. I definitely see your point.

Yeah. So, okay, moving right along. I did want to, I know we've kind of jumped around a lot on these really interesting topics, but I did want to talk about Smartling and the models that you build and your approach to machine translation. And we've kind of alluded to some things, but just for starters, kind of broadly, how does it work? Okay. Very broadly, but for the systems that we're talking about, like what is your approach to offering machine translation? You mentioned a bunch of different specific approaches depending on the market you're dealing with, from healthcare to these other environments. So, you know, the whole human-in-the-loop method, building systems, how do you go about addressing each kind of specific market and targeted group in, you know, providing this kind of accurate translation approach?

Okay, maybe first of all, we as a platform imagine that Smartling has been around for X years and imagine how much labeled data we've actually accumulated over the years. So we have the luxury of actually having the access to labeled data and purpose it, apply it to different tasks equally. As I said earlier, and the notion of garbage in, garbage out has not gone anywhere. You need high-quality human data to— in the ideal universe, you need high-quality human data to actually train, build, train, or fine-tune your models. So maybe first things first, we get the best results still to date. We get the best results from trained neural machine translation, trained with domain-specific or customer-specific data. Of human quality, which actually does come from generations of human translation and fine-tuned large language models. Then the other part is you still cannot overestimate the role of terminology, domain terminology or customer-specific terminology. Our customers always are always looking for brand, tone, and voice. And again, if you look equally at neural machine translation engines or large language models, they equally cannot capture the customer-specific terminology as much knowledge as they have. So we highly recommend using fine-tuned custom models with specific language data. I mean, obviously you need BITEX for this. And if you are still— if you're not still, but if you're using neural machine translation that to date proves to be slightly superior when you apply automated metrics, then we go with customization. And overimposing glossary on top of the output. You can equally inject a glossary in the model and give it certain weight, or you can overimpose the glossary in morphologically accurate way. This is where large language models or small language models actually shine. The other thing is that there is a huge problem to date. There has been a huge problem in the translation industry that when you insert the glossary, when you apply the glossary, say to neural machine translation systems, it breaks the sentence fluency. It is not applied in a morphologically accurate way. And then here you are, and we often ask our customers, like, what do you prefer? Do you prefer a fluent sentence or do you prefer your brand terminology captured in the translation? And most of the time we hear like, hey, we really would want to capture our brand terminology. Now fast forward with automated post-editing with language models or AI-based automated post-editing, you actually capture both. You capture the right terminology and then you actually apply it in a morphologically accurate way so it doesn't mess things up. So what did we touch on? BITEQST for specific language, BITEQST for a specific domain or customer-specific, and application of terminology. The other thing that we do is fuzzy matches where you can leverage previously translated content to a certain extent, and we apply large language models to quote unquote repair fuzzy matches, or what we would call is build AI-based translation memory system where you actually leverage and then you use large language models with few-shot examples to compensate for the delta between the previous translation and the next generation. So there's a lot, a lot, a lot happening. And I haven't even started talking about quality estimation tasks, which is a whole different universe. But maybe let's see if you have other questions because otherwise I'll just be talking for hours.

Yeah. So before we get to the quality aspect, all the different things happening that you mentioned, you also, you mentioned large language models and you mentioned small language models. So I wonder, and fine-tuning and fine-tuning to customer data, does Smartling kind of develop and maintain specific model sets for each domain, like a small language model for this that is then fine-tuned to a customer? Like, do you have pre-existing models of large or small of some kind for each general domain? And maybe do you also have just a general massive model for translation as well? And then as you work with customers, You're like, okay, we'll pull up, you know, Domain 1 Subgroup 3 model and we can fine-tune based on that. Is that how it works?

Basically, to be fully honest, fully transparent, we get best results basically capitalizing on the models that are out there, right? And in equal parts open source models and commercial models and using smaller datasets to fine-tune those models and using few-shot examples and using RAG. For those models actually gives us— give us enough results to give us time to experiment with smaller domain-specific language models and see which approach wins. Also, as I said earlier, we do have quite a bit of labeled data. I mean, when I say quite a bit of, it's actually billions of tokens of labeled data that we are using to build our own translation-specific purpose-built model. But again, To date, models fine-tuned with labeled data, large language models fine-tuned with labeled data, task-specific labeled data deliver sufficient results for us. So maybe short answer would be fine-tuning for now, but experimenting with smaller language models.

Now for the experimentation stuff, in the effort to— so the interesting thing about all this is the models that you're talking about, the third-party open source or otherwise, or OpenWeights, whatever it is, there's so many options models out there are they're not purpose-built for translation. They're kind of these intended-to-be general-purpose models that excel in some areas and are weaker in other areas depending on training data. When you set out to experiment with building a purpose-built-for-translation model, what kind of data is good for— is it just anything that has words is good? Is it very targeted? Are we putting in like dictionaries? Like what, what kind of data is quality for building a purpose-built translator model?

Well, I mean, first of all, you need a— in the ideal universe, you need a sufficient corpus of parallel data. And there are a lot of techniques in preparing and generating synthetic data when it comes to developing datasets and curating datasets for training and fine-tuning those models. And And again, back to the concept of garbage in, garbage out, you run things like validating terminology between terminology datasets and the actual full sentence, full segment datasets. You do a lot of disambiguation and deduplication to make sure that the model actually delivers accurate translations. So I wouldn't go with everything with words, right? And then once you've built, like once you've built your general model, then you're actually applying weights. You can skew the output of this model to a specific domain. So again, there are several techniques. There is data cleanup, synchronization of data and terminology databases. And again, I mentioned already that sometimes, and that's probably one of the bigger problems in the translation universe, the under-resourced languages. And under-resourced, low-resource languages are— we definitely want to make sure, if you look at Facebook initiative of No Language Left Behind, you want to make sure that the models can provide equal coverage. For higher resource languages and for lower resource languages. So then the technique would be generating synthetic data, high-quality synthetic data to augment the deficiencies for lower resource languages. And then obviously there is a whole separate set of techniques such as paraphrasing or like a lot of other techniques that are applied to generating synthetic data. And right now I'm only talking about text-based because obviously the voice-based translation and interpretation is a whole separate universe. And if you want, we can also touch on ASR and text-to-speech models. So clean datasets, and that's where I'm saying that linguists who work on data, developing datasets for model training are probably gonna be employed for quite some time.

Why don't we talk about the, the quality, the quality side of things?

Okay, so there are two general directions in linguistics and in the translation or localization global content delivery industries or universes. There are two major tracks, and a lot of times people mistakenly think that just translation alone is what we focus on as the industry, and that's what academia is focusing on. And that's actually a misconception, and that's why I wanted to touch on the hallucinations as much as you dislike the word. Because first things first, you've produced a translation. How do you know whether it's good or bad? First things first, you need to be able to catch those errors and mitigate them. That's where probably the first layer of quality estimation comes in. At Smartling, we definitely pay a lot of attention applying more traditional BERT-based models and large language models to mitigate those hallucinations. Again, there are There are multiple techniques like self-policing models or one model that checks the output of another model or earlier generation, as I said, BERT and Roberta transformer models that actually can do that. So first step, catching and mitigating factual errors. Then the second step would be, if you think about the assembly line, that would be automated quality estimation where you actually, again, providing, and this is where labeled datasets are a huge help when you can actually provide direction to workflow or in the modern time and day, I guess, workflow agents, which direction, which, which process is applicable for this content type. Right. And this is where you estimate the quality of individual strings or of a batch of content and you decide whether you need human in the loop or this content is good as it stands or in the interim, what the human just really needs to briefly fact check. And then at the end of the process, I like to differentiate, and there is some disagreement there between LQE, language quality estimation, language quality assurance. The end of the process, you basically sign off on your content. And this is where we mentioned previously the multi-dimensional quality metrics. This is where data labeled for different— for the whole taxonomy of errors is of absolute necessity. Because you basically fine-tune the model on this data and then a model can replicate human judgment. So there are 3 stops, right? First, model output mitigating, catching mitigating hallucinations. Semantic similarity is of great help there. Step 2, using model and RAG for domain-specific— opening another can of worms here. So for domain-specific translations, obviously applying RAG is is a great technique here. And then the final trained on label dataset quality assurance. So that would be your stream, that'd be your workflow.

I did want to circle back to something that you mentioned a couple of minutes ago about the under-resourced languages and the challenge in addressing those languages in achieving equal weight, I think you mentioned, right, on the higher resource languages. I imagine English is the kind of big one. It's prevailing. Right. Just kind of because of the function of the internet and these models are trained on, if they're trained by crawling the internet, you're going to have a big skew. And the approach I think that you mentioned was you need to kind of turn to synthetic synthetic data to buff up those under-resourced languages. I wonder what that process looks like and what the scope of the challenge is. Again, in kind of in what you're just talking about, about the quality assurance stuff and the role of the human in mitigating that. And I— how do you ensure that in languages that are under-resourced that any synthetic data that you produce around it matches not just the quantity but the quality and factuality of the input for something like English, which as you mentioned is prevailing.

I mean, to your initial point, maybe we can touch on that first. If you look at the language coverage and the distribution of individual languages, I mean, obviously it's extremely uneven. Now maybe if we go back to smaller language models, a lot of in-country, language service providers and translation agencies that specialize in individual language, they basically just take individual language dataset and build language-specific models, and that's where they have great successes. But again, for that, you need to service 2, 3 languages. Now, if you operate at Smartling scale where we service 250, I believe, and maybe it's just what we service with the help of automated translation, it could be more than that. Obviously that's not a scalable approach. So also what happens in under-resourced languages or languages with smaller datasets, to your point about factual accuracy, that generalized foundational multipurpose models, they would often deliver, they might even deliver accurate vocabulary, but the phenomena they would reflect would be 1,000% English-centric, right? So to that effect, you really need a sufficient fine-tuning training dataset for covering those languages. But first things first, I think in absolutely every application of the natural language processing or be it multi-modal models, I think the requirements to develop a synthetic dataset are the same. You start with a clean initial dataset which has enough representation of different domains, sufficient number of vocabulary patterns and different linguistic patterns. And then from there, basically initially you absolutely need to start from clean dataset. And usually even for the most under-resourced languages, you would usually have something to start from. Something. Yeah. So some, you would usually have something to start from and sometimes you would deal with languages that don't even have the, they don't have much written content, but that's where ASR to your rescue. You can take the first pass on collecting audio data and then convert it, run ASR, and then produce text-based dataset. But from there, to your point about factual accuracy, I mean, you only want, I think the main thing is you only want to permute this much and only introduce certain permutations, right? For instance, you want higher number of sentences which capture different to capture different inflection patterns or capture, like as I said, for instance, convert negative to positive or introduce paraphrasing. But probably you would not want to do much about the meaning, altering the meaning of your sentence. So I don't know if it makes sense. Yeah. Yeah. Hopefully it does. But again, to generate a synthetic dataset, you need to be very careful about what you're actually willing to introduce and where you want to assume the risk. Relying on synthetic dataset.

For those, you mentioned that you offer or you serve roughly, I'll double-check the numbers on this, I'm sure you have them on your website, around 250 different languages. Is there, because we're talking about differing levels of data availability and as you kind of just explained, Is there a performance drop-off comparing your maybe English model to a model that's lower down on the list of, or higher up on the list of resource, of being under-resourced where the difference between like 94% and 89% or something, is that a phenomenon that you see?

100% because Everything, I mean, we start, when we started this conversation, we started talking about language grouping and it's not only the question of whether the language is high resource or low resource, it's also a question of language complexity. So maybe we don't even go into the dataset because, okay, there is a matrix and the matrix goes training data availability and language complexity. And it's very simple and very easy to, understand grid where, like, say you take Latin American Spanish, right? You have huge representation of different dialects and different flavors of Latin American Spanish in your training dataset. And to that, you also add the reasonable simplicity of Romance languages. So that's basically where you get the highest performance, but then you drop down, let's go down, not down, but then as you go, as you travel down the grid, let's, I mean, let's touch Estonian, for instance, and that's where you're dealing with Finno-Ugric language. That's A, from the most complex for any kind of natural language processing task language group, but also imagine how much data you would have for Estonian compared to Latin American Spanish. And absolutely, as you travel or say Southeast African languages or highly agglutinative languages, for those you definitely see degradation in performance. And this is where additional post-processing techniques and availability of clean data and maybe sometimes even applying rules on top of translation generated by AI-based systems. This is where it definitely helps.

I want to tug on that last point that you mentioned about the application of rules, which is an interesting thing. I think something that you mentioned right at the very beginning that I filed off in my head to follow up on and I'm doing it now. Is the kind of— the evolution in the machine tools that this field has used as it shifted into the neural networks, you got an added degree of unpredictability. And when you're talking about, like you just mentioned, the addition of rules sometimes, I wonder in the ways you work with these different clients, which as we talked about are, you know, at differing levels of acceptable error rates, depending on the high-stakes or lower-stakes environments that they live in. I wonder what that looks like in terms of offering or injecting layers of determinism or rules-based stuff into something that is not deterministic, or adding some sort of—

right, like, is that a thing that goes on? Absolutely. Where— okay, maybe first things first. I think the core principle of natural language processing, and I don't know, maybe other people in the audience would disagree with me, is do not use machine learning-based systems for entertainment's sake when actually pre-programmed, pre-created rules would do the task just fine, right? So sometimes for things like, for instance, mitigating very simple string formalism things like whitespaces or Lakale-specific conventions. You can either just apply rules and bypass any kind of AI-based transformations altogether, or you can rule, or you can apply rules to correct, to correct the outputs and introduce, to your point, introduce more predictability to the final output. So again, this is where, for instance, you have, and that's what we do at Smartling, you have a combination of machine learning-based techniques and rule-based techniques for the initial quality check. And again, those rule-based techniques would check for more obvious and more predictable things that are easier, easier handled by the rules. But definitely, and to your point, I mean, machine learning can introduce fuzziness, can introduce lack of predictability. So this is where relying on rules probably would be a more recommended approach, at least for the time being. Mm-hmm. Until we've learned to mitigate all those issues that are introduced by language models.

So it's kind of the application of the right technique at the right time.

Either the application of the right technique at the right time or combining both techniques and basically you run both. So you check for things like, hey, why is this string so long? What exactly did this model do that, like, why do we go from X number of tokens to Z number of tokens, then okay, something is absolutely off here. And obviously this is where you can apply semantic similarity techniques or you can just say, hey, something's dramatically off here in terms of punctuation, just in terms of number of words and whatever you want to count, words, tokens, characters, depending on, depending on the use case. So there are places where I would say that providing both and signaling from rules is, is definitely of great value. Also because we are dealing with language and language, I mean, thank God language, languages are structured, right? So there are a lot of things that you can learn about structure of languages that you can actually apply in the rules predictable piece of your workflow.

Language structure is very interesting to me as someone who's never really studied it. Right. But the, completely unrelated to everything we've been talking about, but I wonder the application of this kind of structure that you're talking about, the rules for a given language. Is that more challenging with certain languages? Like Romance languages maybe follow more given rules than something like English, which is kind of pulled from all these different sources where we have roots in Latin, but also Germanic, and we just kind of adopt stuff. Is it harder to apply, uh, structure-based rules to a language like that compared to other languages, or is it not really much of a problem?

I mean, first of all, it depends on how much investment you're going to actually put into those rules. And I mean, equally, you can obviously augment those rules, I mean, from all the available, all the available open source libraries. So you don't need to reinvent the wheel. But still, I would say that the level of effort that goes into developing different rules for, like, say again, I'll go back to my favorite Turkish and Finno-Ugric languages, and the level of effort that goes into more predictable languages is definitely is definitely different and the performance definitely varies. And then there are things like, for instance, how do you deal with— I'm just thinking about examples, like how do you deal with German compounds, right? Where you really, I mean, you attach X number of stems to each other and then, or like, I mean, I happen to be native fluent in Danish and I think the longest word in Danish is something like, I don't remember, like 30 syllables or something. So to parse out those notes, seriously, I can even say like, "Heuferlsam heißt Relais," something to that effect, something from electronics. So I think applying rules to those languages is definitely, it renders more complex results. And this is, I think this is a beauty, at least for me, I'm super passionate about what we do here at Smartling because you face completely different set of language, set of challenges for different languages. Pivoting back to the role of a linguist. You and I will never know things as much as I claim to have studied it and doing it for a living. You and I will not know all the rules and all the grammar and syntactic particularities of each individual language. So that's where you definitely need input of linguists that speak that language.

Yeah. This stuff is just Absolutely fascinating to me. As we're kind of wrapping up here, I want to leave off on this kind of point. You're dealing with state-of-the-art models. You're dealing with the kind of cutting edge of machine translation, as we've talked about. You know, you have worked with all these different model approaches to it, and now we're where we're at. What are you doing to ensure that you stay or Smartling stays on the cutting edge of what machine translation looks like. Is that even something that you can track? Is the rate of advancement just kind of hard to even keep up with?

Can I cry on air? Would that be okay? That's honestly, that'd be my first immediate reaction because never in my life would I have imagined that I will wake up to a new model or next generation of a previous model every single day, or it's one model on a good day. On more challenging days, it's 5 of them. So I think there are 2 things there. You absolutely want to, A, apply state-of-the-art techniques, right? Like, I mean, I already touched on RAG, and we can talk about a combination of knowledge graphs and machine learning-based techniques. But I mean, I think first things first, you want to be appraised, but you absolutely should not go wild west on those because chasing after new technique or new publicly available model every day is just not viable. It's just not realistic. So I think my first, my immediate comment is choose your battles, study what is innovative and what is the delta between what you could do on Monday and what you could do on Friday is, and measure the impact. So that's the first, that's like an intellectual effort, right? And again, beautifully, you can use AI to summarize it for you so you don't need to read every paper on arXiv. That's one. And second, again, I'll talk about Smartling because that's my day-to-day. You evaluate the models for different tasks. And this is, you know, there are a lot of commercial and publicly available model evaluation frameworks. We rely mostly on our own suite of automated metrics. And this is something you can run. I mean, we have a quality model, quality model performance evaluation suite that we run in-house that combines 8 or 9 metrics. And since we're pretty confident with our tasks, and since we deploy variety of different models, our prompts are reasonably portable. And I go reasonably portable because I mean, still at the end of the day, different models are different. Depending on different parameters. We can fairly easily press the button and have— we have a team here at Smartling that does this. Basically, their job is model evaluation and model benchmarking for different tasks. And since you rely on automated metrics as a first pass, you could just rule out, okay, well, the impact is minuscule, I don't need to do that. Or you start experimenting with a new technique, and that's obviously going to take longer time. So A, analyze the impact, don't chase shiny objects, and apply automation to whatever this model can do for you. In our case, throughout the entire translation quality estimation pipeline.

Right. Yeah, no, it—

even if you get like for quality estimation, you get the right precision and recall ratio, you're good. And if the ratio, I mean, if you get the right precision and recall numbers, And then with the next iteration of whatever that is, whatever the differences between the models are, like, hey, I'm hitting like 80 on both, golden, deploy. Right.

The, the, the pace of deployment has been crazy with the new model every day. I, I think just, you know, over the weekend Meta drops Llama 4. But then I, I, I think your point about not chasing the shiny objects, but evaluating what the actual difference is in the, the way that you're using it is a really important one because there are a lot of new models. There's a lot of hype and a lot of marketing around them. And, you know, at the end of the day, is it really changing? Is this one new model really changing what you're able to do with the model that came out two weeks before that? Right. It's a more kind of analytical or thoughtful approach than just new, better, put it in, use it.

And at the end of the day, Smartthing, I mean, we're a business, right? And we also need to look at ROI, right? And we need to look at, you know, okay, so what's either in-house computational cost or cloud computational cost or implementation of a new model if it's a model that you need to pay for. That you cannot deploy locally and it charges, right? It would charge per, per token. So there is also this factor, like, hey, it might outperform it, but it takes a 3,000 token input prompt.

That is another really interesting point too. But we are gonna cut it there. I don't want to take up any more of your time, but Olga, this was, this was really, this was really interesting. So thank you so much for coming on. Thanks for having me.