AI Translation the truth Smartling learned

You burn a lot of tokens when you deploy third-party large language models. API plug is not going to do it for you. You need to write targeted complex prompts. You're very likely not to get to the right place and you're going to burn a lot of tokens and a lot of money.

Welcome back. Today I'm joined by Olga Beregovaya, VP of AI at Smartling. We're talking about one of the hottest topics in our industry. Can AI really deliver human-quality translation at scale? Olga, thanks for being here. Let's start with the obvious question. When somebody says, "Plug in GPT and you are done," what's the reality?

So, what does it actually mean when somebody says, "Let's plug in GPT and call it a day"? First of all, I would say yes, there are many universes in which you plug in GPT and you're golden, right? Which is you need to understand some inbound content or you have some very quality content with very low sensitivity to quality, or you need to just quickly— usually just— usually it would be for inbound. Yes, there are a ton of scenarios where that would work. But now when we talk and we operate in the enterprise universe, When we talk about implementing AI and getting translation at scale, I will be a very strong advisor against just plugging in GPT, thinking you got your silver bullet and call it a day. There are so many considerations. It looks perfect on the surface, but when you start digging in, there are so many things to be mindful of if you want to produce enterprise-based AI translation. I don't think we are all necessarily aware of it.

You have mentioned before that you have a small slide with a couple of things. Shall we put it up on screen?

We absolutely should. Give me just one second. So, this is what I would call the silver bullet syndrome, right? We can assume that, okay, plugging in an LLM is going to solve all the world's problems. Anecdotally, I was just talking to a friend. One of the technical leaders in her company said like, hey, you know, I need those Figma files, 4,000 Figma files in 52 languages in 3 days. 'What do you mean you cannot just plug in GPT?' Okay, and here is pretty much what you mean. 'I will just plug in an API. Why? It is cheaper.' And, 'Is my company directive to use AI?' Yes, but you need to write targeted complex prompts. Just translate it into language X, you're very likely not to get to the right place and you're going to burn a lot of tokens and a lot of money. Next one, LLM delivers higher quality translation and we've done a lot of research at Smartlane. We consistently use machine learning to benchmark different models, and we are yet to see that LLMs beat trained NMT or NMT plus automated post-editing with LLM. So yet again, deliver higher quality. Why do we usually get into that sentiment? Somebody runs a very simple general domain sentence, and yes, it would look fluent, and because it's simple and general domain, it would look accurate. But does— is it always the case? Research shows that not necessarily. Be deployed across multiple languages with minimal effort. Yes, but this is where platforms like Smartling, your platform partners, come in. Where are you going to get the infrastructure and where are you going to get all the functionality to run your translations across multiple languages and multiple content types? You've got to get it somewhere, and just a simple API plug is not going to do it for you. And then, "LLM integrations learn from individual users." Very much true. We see a lot of Copilots learning locally. From talking to enterprise buyers, I learned that it actually is great, but it generates as many employees as there are, as many voices are generated across the company. And how good is it for the brand, question mark?

That's it. It's not that simple as most people might consider, right? So what are the most common mistakes you see teams make when they try to rush into AI translation, Olga?

I think one most common, and I've been thinking about it a lot, and again, we at Smartling, acting as partners to our customers, often make sure that we walk in and we say, are we solving the real problem? Do you really need to plug in AI or are you following an executive mandate? And if it's the latter, let's see, let's unpack and let's see if you really need that. So I think one of them is just jumping into blindly art for art's sake and implementing AI because you have to. I think that's one of them. The other one, and I'm probably not going to be saying anything new, is hype. Often enterprises, when implementing AI and AI for localization industry, just follow the hype of, okay, AI is going to solve all my problems. And I'm a strong proponent, and again, we, we, plural, are strong proponents of measured deployment. Set your goals, set your ROI, and pause on ROI for a second. Mitigate your risks, understand your risks, and then proceed with a measured deployment, understanding all the pieces of it. Just jumping into it is not going to get you far. Now, why am I talking about ROI? What we see customers do and where we come in, and it goes back to prompting, you burn a lot of tokens when you deploy third-party large language models, right? And see, do I really need to make this AI investment to solve a, I don't know, a small problem of my SKU listings or a problem that doesn't warrant that kind of investment? So I think it's jumping into it just because you have to, not measuring what you're trying to get out of it and not counting the money you're going to put into it. These are probably the 3 biggest mistakes I would think about. And again, the assumption that it's just going to do it all for you.

Yeah, this is like the magic button idea, which some have in mind. We saw it the other day on LinkedIn as well. Miguel Sepúlveda mentioned about it. So this is something people need to understand that the reality is just a little bit more complex than the simple thinking that it could, should work. Right? So, Olga, let's talk about data. In Smartling's 2024 report, you just published a report on AI and you saw a 30% jump in translation volume. Why is that happening and what does AI have to do with it?

Okay, so the jump in the volume was accompanied by a tiny jump, a dip on spend. Right? So what we do see is customers tend to spend even less, a little bit less on translation, right? Like 3% less on translation, I believe the data point was. But yes, at the same time, we see the huge volume jump. And I would compare it exactly to the machine translation wave, right? When statistical machine translation finally beat over rule-based, people realized that they can open a lot more content. A lot more content opens up. Right? Tap into the untapped. And that's when I think AI plays the pivotal role. And it's very similar to the machine translation wave, except the order of magnitude is significantly higher. It is now we finally can get our hands on much more content and be much more relevant in countries, in geos, because it's not only the jump in the volume. If you look at the content types, it's jump in the variety and diversity. Of content types because now you have multiple quality levels. If you look at the Smartling platform, there are several quality levels, all of them fit for purpose. And if I can get away with MT here, hey, I'm pouring billions of words in. And if I need MT plus large language models-based automated post-editing, hey, it gets me reasonably close to humans so I can get even more. So I think it's really the means have become so sophisticated and advanced that there is a lot of excitement and understanding that you can really do more with less without it being a buzzword.

Some of our audience are more deep into the topic and you just mentioned it. So there's MT is rather low cost nowadays. It's very standard, standardized. It's for huge volumes and AI is sort of a little bit slower. Can you give a little bit more context about how at Smartling you work with a typical AI workflow? Is it MT first and then AI on post-editing, or what is the secret sauce that you guys apply?

Okay, to that effect, let me actually share another visual because I think a picture is probably worth many words. So as you rightfully said, machine translation tends to be fairly cheap, right? We all know how much we can buy machine translation for per million tokens, per million words from different suppliers. But again, the world and Smartling as a part of the world, we're all learning the same thing. Machine translation is more or less guaranteed usability, utility, and accuracy. Whereas LLMs will give you the beauty of context, will give you the beauty of additional linguistic fluency. So how do you actually get the best of both worlds? And again, all of our benchmarking shows that the best of both worlds gives you better results. At this point in time, even better results than purpose-built language models for translation only. So this is something that we've got. That's called Smartling AI Toolkit, and a part of the Smartling AI Toolkit is AI Hub, which is basically the way of getting into our AI Toolkit. And here are the toolkit features that really give you the best post-translation experience. Say you are done with your neural machine translation step. Great, done. And even better so if you trained your neural machine translation system Still to date, they show the best results when automated metrics are applied. What can you do with that? A, AI-adaptive translation memory. Perfect. You take your translation memory, you elevate your fuzzy matches by up to 35%. In production, we've seen even up to 60%, but okay, let's go with conservative 35% average. So hey, you just got more accurate TM matches, and obviously you pay less for them because they fill into a different fuzzy match grid. Glossary insertion. What is the problem the industry had been struggling with for years? Glossaries not being morphologically aware. You insert the glossary, beautiful, you're stuck in the land of infinitives. Do you want it? No, it just killed your sentence fluency. So AI can actually make a sentence terminologically accurate and fluent. Formality. And we, I mean, we both come from very culturally rich languages, I would say, and you know that formality can mean a lot in a lot of languages. So this is a great place to actually say, hey, I'm writing to such and such audience. I'm selling to backpackers today and I'm selling to luxury travelers tomorrow. I'm the same hotel chain. Help me. LLM Tag Aligners. Again, all of us from loc industry know that tags can play vicious tricks on us after they've gone through automated translation workflows. So here you are, your tags are aligned. And then something, uh, AI post-editing agent that's being released now is something that would take it all together and produce a smooth, fluent, accurate, on-brand, on-point sentence. Now, there are only two important things I believe when it comes to language. It is how good is your automated translation and how good are you at predicting and evaluating the quality of this translation. So this is basically the toolkit in a nutshell. Tackling the content from all the angles, and then telling the workflow, "Hey, this is what I've just learned about the content. How about we route it to the right process?" So, that's it in a nutshell.

We will see all this later on in this video because this channel is about that. I don't want only topics, but, you know, see the actual product. But before we dive into it, The report also says that only 26% feel trained to use AI. How do you coach companies through that gap?

I think that's something that's inherent in the DNA of our industry. We often come to companies that are centered on— no offense— but often companies are centered on producing everything in the language of, in the culture of wherever the company is based, right? And most often it's English-centric. So I think that role of trusted advisor is built in and inherent in the localization industry. So I would say it's reasonably easy for us to come in as trusted advisors on yet another critical topic when we have already been those advisors on internationalization, globalization at the source, right? Continuous loc. There are so many things we already have done with our customers. We are adding yet another dimension. And you walk in as an educator already having your reputation as, hey, these guys know their business, they mean it. They just took me to Africa and Estonia. So I think it's— we develop a lot of talk tracks, do a lot of proof of concepts and benchmarking. Obviously there are a lot of webinars, but really there is nothing better than show and tell. So grab your customers' content, sit down in the room, look at their content types, analyze their goals. Again, I say Laying ops are all about plans. Build out the proof of concept plan and show them, hey, it worked. Or how about we fine-tune some models? Or how about we play some more with linguistic assets? So I think it's a super interactive process. You cannot be didactic, but you absolutely capitalize on the trust that you already have been as an industry player.

Got you. Partnership mindset and showing how it works. Really work.

Showing, I mean, there's only this far you can get with 2 slides PowerPoint, right?

Totally. And as we talk about showing, can you show us what it looks like in practice? How does, you know, the Smartling dashboard look right now and how a manager can track quality or cost savings in real time?

Okay, so let's go to the toolkit dashboard. So let me share what it actually looks like in practice. I will show a few screens. I will show the workflow, I will show the dashboard, and I will show what insights a business manager can actually get from looking at this dashboard. So let's look at the Smartling AI Toolkit dashboard. This is the place where you actually see all the savings, all the efficiency gains, and all the quality improvement that you have just exercised and you've just obtained during running the project through the Smartling AI pipeline. Smartling AI system. So we already touched on the concepts. Here we are. Here is your translation memory leverage. You can see that some of the, some of the matches were not leveraged at all. Okay, great. They went to machine translation and AI translation. Then you get some leverage through the general fuzzy match leverage, but then some of it is actually obtained through fuzzy match repair, our adaptive AI. So you can see how much you actually have gained just by running AI-based translation memory optimization. And then the rest of it, obviously you always want to know how well are your smart matches, also known to other players as ICE matches, in-context matches, how well your smart matches are actually contributing to your savings. And we know that smart matches would be complete, accurate match. And we're looking at the data from just 7 days. So as you can see, you can also see, okay, how much of a bump did I actually get from applying Smartling Adaptive Translation Memory? And you can see the dynamics, you can see the leverage dynamics through the day. So that's a great visual dashboard for the manager to see, okay, where am I? What are my gains? Is my investment actually paying back? And obviously it is. Then there is this module that we spoke about earlier, Linguistic Quality Estimation. Again, if we go back to the notion of, There are only two things. How good is your quality? And how good are you at predicting this quality? You can see that you have high quality, medium, and low quality segments, and you see the breakdown of segments that go into each of the categories. What does it tell you? You make your own— you're the master of your own domain. You make your own decision. Do I want to take my high segments, feel confident about them, and push them straight to production? Absolutely go for it. Medium, how about I have some light review of that, a validation review of that. And low, yeah, you'd be better off sending it to full post-editing exercise. So again, you have all this data and you see how your workflows were working for you. Also, needless to say that all the data that you obtain from the dashboard can be exported and can play great with your BI system. So whether you're using Tableau, Power BI, or anything else, obviously you can take this data ingest and slice and dice it on your end. Now let's look quickly at how do you actually get here. You can set up the AI-based workflow for the toolkit, and the workflow can actually let you make the decisions. As you see, there are a couple of decision tree places where you decide how exactly you want to handle different quality segments, and you have a lot of settings. There are a lot of ways you can actually manage the step, and you can make your decisions on whether you want to use Fuzzy Match Repair. Do you want Do you want to use our entire AI post-editing agent? Do you want morphologically inserted glossary terms and all the other options? So all the other options are at your fingertips. Sometimes it's relevant, sometimes it's not relevant. So here you are, all your options are laid out to you. Then you make the decision of how you want to go about it. Go to a human, go straight to publishing. And again, everything is extremely modular. I'd probably even go with the word agentic, and you can make decisions on how you want to design your AI-based workflow for perfect, for the best outcomes. And this is what it's going to look like for the translator experience. So as you can see, the translator will get— this is your usual CAT tool, but the CAT tool is enhanced with the AI functionality, which actually tells you what, not only what kind of a match it is, but what kind of a quality level it is. And this one, the one with high, is going to go straight to publishing. The one with low is obviously something you need to pay more attention to. So the translator is also informed about how AI performance actually impacts their level of effort. And we get a lot of great feedback on that because it's much easier for the translator to know where they need to focus their attention.

Got you. Love that. This makes it very tangible. And this is also the aim of the channel, to open the door to the innovation we have in our channel. Now about the models, you're not using GPT only, right? What is the mix of multi-LLM setup you're using at Smartly?

Well, we all wake up to the same world now, right? We all wake up to the world of new model pops up. There isn't, or 5 new models just pop up. So first, initially, yes, all our experiments started with GPT because that was obviously top of everybody's mind. But now I think we've nailed the approach of having all the right machine learning-based metrics to quickly filter out what's good, what's not good, what's fit for purpose. So we are, I would say that we at Smartling are very hype immune because we have the right guardrails in place to always make sure that we deploy state-of-the-art and best in class and consistently monitoring. When I'm saying monitoring, it's not that people are sitting there clicking buttons. It's basically a monitoring benchmarking machine learning-based machine that can quickly tell you, does this model hallucinate? Does this model perform well for such and such task? Does this model work well for this, for this language? So if I gave you the landscape of the models now, we would go anywhere from the GPT family to Vertex AI family to open source models. The thing is that I will be giving you a completely different answer a month down the road. Because the benchmarking would show us that, again, without following the hype, being super conservative, measured, and having trust in the model performance, we always plug in what at that point in time works best and usually can see the future where this model— what's this model— what this model's trajectory is. Within the platform, we have the mechanism of managing those multiple models, so it can be pretty seamless and painless if you see that you need to move from model A to model B. Does—

is there some automation happening in the backend or is it that the user has to configure it? You know, I want to use this engine or that engine, but for the whole project or is it language-based? What is it?

I think it is just like machine translation. We can do it all on our users' behalf because obviously our research, both for model performance and best-performing prompts, And again, I do not want to undermine the prompt complexity. We could do it all and we could black box it for you and deliver the best outcomes and be confident about those outcomes. Now, if a customer wants their own model configuration, like sometimes a customer would be OpenAI or Azure or Google shop, then they can make their own selections and they can make their own selections of which models they want to use. So there is a lot of automation, but need there be, be there a need for a manual step, we can also allow for manual configuration. So really all about the degree to which the customer wants to be involved and wants to make their own choices. But at the end of the day, all the workflows and selections are automated.

Good. Finally, Olga, for everyone watching, if their boss said, deploy AI translation this quarter, what's the thing they should do first?

I speak to so many people where the boss says, deploy AI translation this quarter. My first response is like, hey, are you guys using neural machine translation? Go tell your boss that you're already using AI because neural machine translation is based on deep learning and on AI. And many customers actually very happy like, hey, Yeah, really, I'm using machine translation. I can do that. But generally, if the boss says deploy AI, I would say go to your local instances. I don't know, use Copilot, whatever. Grab low-hanging fruit and report back that you are using something locally. Like, we're all using GPT or Perplexity for lookup. So my first thing would be like, yeah, we're using it. We're deploying AI to improve our processes because you cannot mindfully deploy AI within a quarter on an enterprise scale and make it deliver. That would be my first response. So my advice would be grab some super, super low-hanging fruit, such as Copilot, report back, and buy yourself time for the proper measured calculated deployment.

Olga, thanks so much. This was practical, real, and exactly what the industry thinks and feels about talk with you. So can't wait to see how SmartLink keeps pushing the limits from now on. And I hope we two will be on another show whenever there is innovation coming from SmartLink. And of course, for our audience, from all the other players. I'm today here with Olga Beregovaya, VP from SmartLink. It was a pleasure to talk to you and all the best until the next time.

Thanks so much. And thanks so much for having me.

Thank you. Bye-bye.