Olga Beregovaya - Benjamin Loy LocDiscusstion

All right, good afternoon from sunny Texas. I'm here with two Smartling stars, Olga Beregovaya and Benjamin Loy. I hope I'm saying that somewhat right. I'm more concerned about Olga than Benjamin, but I'm in the right register. They're both nodding. So Olga and Benjamin, why don't you go ahead and introduce yourself real quick? That way I don't forget anything important. And we are sure to get in what you want to be known public. Go ahead, Olga.

Okay, let's try ladies first. So Olga Beregovaya, I've been in the industry for over 25 years now. My entire career in natural language processing and language technology. Joined Smartling 2.5 years ago and totally thrilled to be there on the team with brilliant researchers like my colleague Benjamin Loy.

Benjamin.

Hi, I'm Ben. I've been in the industry for only about a decade, so nowhere near as much as Olga, and I've actually been at Smartling that entire time. I came up on the engineering side of things, and now I'm running research for AI and all things translation here at Smartling.

You seem like the right people to be talking to at Smartling, and I've entitled this session Smartling Beyond Words: Engineering the Future of AI-driven localization, and we'll be talking about topics on the cusp of AI and localization. And in the first chapter here, I would like to talk with you a little bit about segment translation and how in the world of large language models, there's a lot of talk nowadays about the fact that large language models are better suited to handle text in its entirety, especially with content keeping content in mind, so to speak. And so as localization software providers were reimagining their localization workflows a little bit, I would imagine, how has Smartling had to rethink the workflows in light of LLMs being less segment-oriented?

All right, I guess I can go first on that. So yeah, we're at an interesting point in the industry. Traditionally, you get a file, you break it up into segments, you have segment pairs when you go to a target language and you compare them and it has a lot of useful things. We've been really trying to move towards a hybrid approach where you take the document, but we still break things into segments. Imagine you have a website, you still have buttons and menus and dropdowns and all those things you really want to have in these segment pairs still, right? You don't want them to like move around or do anything else. But we are kind of moving in a direction where the content itself, the like the heart of say like a marketing site, could be vastly different between languages. They're no longer translations or transcreation or other things. So we're trying to think of ways to move beyond, you know, these segment pairs and really think about how to do things at this kind of higher level.

Yeah. What's, what's so useful about segments, Olga, and what's obsolete, uh, in the age of generative AI? How do you parse the difference?

Well, I mean, first things first, just parsing. And if you think about the linguist cognitive load, I mean, we do not think on entire document level, right? We will still think in terms of smaller chunks of text. And obviously your segmentation preferences are your segmentation preferences. Like you want to see a sentence, you want to see whatever line break, you want to see a paragraph. I think it's still very, very useful. In terms of how the linguist sees their work and how the linguists operate. So that, no matter how we redesign the UX of linguist experience, I don't think that's going anywhere. So that definitely is still very much useful. And to— maybe we'll touch on that later. Yes, you definitely want to do document-level translation, but you still need to see individual segments. That's one. Two, TMX, our TMX standard. I forgot the number was not 1.4, maybe. Maybe my knowledge is older. You still need to, I mean, interoperability is a big thing in our industry, right? People migrate between platforms, people migrate from, I don't know, standalone tools like Trados. You still need to be able to import, store, and maintain those segments just to honor the standards, right? And mitigate the potential leverage loss, right? We still want to save the money and be able to reuse as much as possible. Again, it's probably interim. We definitely, there will be other ways of looking at content, but I would say these are the two things I can think about, caring about linguists and caring about how much you can actually reuse without losing.

That makes sense. If you were building a localization system from scratch nowadays though, would you even start with segments or would you turn that on its head and start a different way? You guys have any vision on that?

I mean, I personally would still preserve segments for the hybrid approach. Things like TMs, as Olga was mentioning, are infinitely valuable, right? If you have the same button, the same sentence all throughout a website, you don't want to pay for that, you know, 10, 20 times. So those types of alignment issues, I think, are still going to be around. But again, like a more holistic document view, I think would be incredibly valuable. Like today, you know, TMs are based on sentences or segments. Could they be based on higher-level contents like documents? Maybe. Could you do more like, um, LLM RAG based on documents instead of segments? Yeah, sure, possibly. So there, there is a lot of room there to kind of go up and out, uh, outside of segments themselves. But I think just the fundamental unit, like, like again, to Olga's point, if you see a document, You could highlight what you want to change, change it, sentence, move on to the next segment, highlight it, change it, move on, and so on and so forth. So I think segments are still pretty valuable.

Yeah, and of course, one of the challenges that I see in the future is that we'll probably move in a direction of sourceless content creation where LLMs maybe create things on the fly. What do we do with translation memories, and how do we preserve the value of translation memories if the source isn't entirely clearly segment at all? Segment at all? Have you guys thought about, about that?

I don't know, Ben, you want to take this? I can take this.

Sure. Yeah, it's, to be frank, it's tricky, right? Yeah.

That's why I'm asking you.

Yeah, yeah, yeah, yeah.

So this is, this is one of the reasons why I like starting with segments, even though you're in this document view.

Yeah.

You can still catch the pairs and then you can still leverage and gain value. But I definitely believe the future is going to be more along the lines of like you looking at like say an English source, you're translating to German. You can be like these things naturally have pairs, you mark them as pairs, but this pair may be up here and the other pair may be down here. Some things you don't want to bring over and you start doing transcreation. Some segments will just be one-sided, they won't be paired. These type of things I think we're going to look at more and more and more. Especially again with LLMs becoming more and more prevalent in the actual content generation. I think it's going to be much more kind of this hybrid mix of things.

Yeah, there's so many aspects to this, this as well. I was just thinking about LQA and what that would look like if your unit is no longer a segment and it becomes a paragraph or an entire document. How do you make that palatable for linguists, for example? Olga, do you have any ideas about that?

Yeah, I have. And, uh, yeah, maybe I could just add one thing to what Ben just said to the previous, previous conversation topic. There is also, I mean, first of all, it's not unheard of empty, uh, you have source and then you have empty target. Nobody said that it could not be the other way around.

Mm-hmm.

So it might not even take the fundamental redesign. And then also every time you hit an LLM, whether it's your own or whether it's third party, you spend tokens. Right? So if you manage to store it, even if it's generated token and you can fetch it from previously translated or generated content, it's just financially, it's just economically more, it's just helpful, right? You spend much less money if you manage to store your content, however you chunk it. Now let's go to LQA and what it's going to look like. There's a lot of talk in the industry about what LQA even means now.

Right?

What quality means now and we'll go out of traditional linguistic quality and assess content impact, right? And assess content suitability. I will take it, I will take it twofold. First of all, if we follow, and I don't see standards like MQM going away, right? Or if we follow any similar metrics, be it DQF, MQM, or any iteration thereof, you still start looking on the sentence level. Right? It's very hard for you to highlight syntactic issues or sentence structure issues on the document level.

Yeah.

What helps and what we've learned at Smartling for automated LQE, you absolutely need to see context because a lot of things like mistranslations are easier caught in context. So I think the segment and the sentence are going to be of equal importance because certain errors belong within sentence boundaries. But expanding it, expanding the process of LQA to more context-driven assessment, that would be equally important. And there will definitely be new metrics introduced, as I said, such as overall document relevance and overall document fit for purpose.

Yeah. What's really relevant in the industry nowadays too is the symbiosis between LLM automation and human in-the-loop type of interaction. How do you feel the design should be for this partial automation, and where do you see LLMs coming in and flowing things over to humans to control the message? What's your vision on that?

Yeah, that's definitely a big topic because you can get beyond the translation space into localization project management, which case, you know, your jobs being consumed can be evaluated by LLMs. Determine what it is, the levels of quality that, that particular content you would want to go for. You know, fit for purpose can be vastly different depending on what the content actually is, routed to different workflows, routed to different people. Obviously you can use automation to generate suggestions to start with. You can use automation to potentially generate LQA or other quality estimations or assessments. But ultimately, at the end of the day, for a huge amount of content, creative and otherwise, you still need the humans there to like do the work. So the future is definitely for localization managers a much more, I would say, fluid flow with these systems where a lot of their decisions can be automated away. But, you know, at the end of the day, you know, you have a human on top and you have humans at the bottom doing the actual localization work.

Yeah. Olga Beregovaya, could alignment between source and target become a post-processing step rather than a design industry, you think?

Absolutely so, right? The more, and even more so when we talk about the content that generated content or transcreated content. So absolutely, it could be a post-processing step where the translation again is performed on the entire document level. And then to Ben's earlier point, you align what's alignable. And now in the modern time and day, there are so many alignment NLP-based, LLM-based alignment techniques. You align what's alignable and then you figure out how to handle the rest. But I think segmentation as we know it is at a preprocessing step is more of a constraint than actually of an advantage. And soon, soon it will be changed. Now we're back to how do you align? I mean, and that's the problem the industry has had for a long time. How do you align transcreated content?

Yeah.

Anyways, you've chunked it up, then you go into transcreation and we've seen so many failed attempts of shoveling transcreated content into the standard. TMS paradigm. So I think it would— if alignment was a post-processing step for storage and reuse and reference and RAG, that definitely opened a lot of opportunities.

Yeah, are we pushing towards a new form of contextual translation memory, you think?

I mean, I think it would be super useful if you had the ability to not only see, you know, fuzzy matches on a segment level, but actually see the entire document. Like LLMs can take that and run with it and do other things. Like, for example, the transcreation case, you— segment pairs are weird, if not completely unnecessary or potentially even harmful, right? Because the two sides may not have any overlap of meaning whatsoever. But if you can do a RAG on top of a document, bring it in, then the LLM can actually look at, like, what you're doing with the transcreation, try to, like, understand what's happening in the language, and then reproduce it in a way where the linguists can really benefit.

I'm glad that in 10 minutes I could navigate you guys towards destroying the CAT tool stack more or less already. So since we're pulling things out, what do you think will survive in this paradigm shift? What parts of the CAT tool stay? What parts go?

I was just having a conversation with somebody where we said like, hey, CAT tool as we know it has not really changed dramatically over the past 20, 25 years, right? I started in the industry, there is a set of CAT tools. You open a CAT tool now and maybe you have things like MT suggestions, AI suggestions, but in general, the CAT functionality as we know it has not, has not changed much. So what I think, first things first, we look at other industries. Chatbots are very present in other industry user interfaces. Right. And obviously some kind of interaction with external generative AI or external sources and being present in the world, in the work surface. That's one, right? There definitely has got to be a way of pulling context into, into the CAD tool. And I think things like concordance search, they might become obsolete, the concordance search as we know it right now, right? Because it's at the end of the day, it's like super, super, super rudimentary precursor of rag, right? So that one will definitely go. So we throw in the chatbot, we take out concordance. Splitting and merging segments, again, back to the segment segment. I think that will probably still stay, right? Because you make a decision of what kind of chunk of text you want to work with. Probably right now, most CAT tools we're looking at, they would have a separate tab for machine translation and AI suggestions, and that's definitely something Once it fully becomes AI-first, you would not have the need for separate references, nor would you need to have a need for suggestion 1, 2, or 3, because you would have the ranking mechanism behind the scenes that will actually already have ranked machine translation and AI for you and furnished you with the best choice.

We've talked primarily about the human linguists in the loop. Up until now, but there's project managers at all LSPs. What happens when they're no longer assigning segments? Are they going to be orchestrating flows? What is their role in the future, you think?

Yeah, I think this is going to come back to kind of what I was referring before. I think everything's going to change with AI, and a lot of the linguistic project manager role stuff. Uh, it's just gonna be automated away, right? Yeah. Again, fit for purpose. Uh, if you can build an automated system to determine that and then determine the proper workflow, like maybe this is just MT, maybe this is a human writing it from scratch with an editor, it's probably something in between, you know, making those type of distinctions quickly and efficiently would be great. Automatically tracking when things will be due, you know, letting you know if there are issues. And really just kind of having these kind of chatbot or AI helpers that kind of like bring the process along, I think would be exceptionally useful for lock managers.

Yeah, it, it, a lot of, a lot of the new capabilities sound fantastic on paper, but when you turn them into practice, uh, they're, they're still not, not that optimal at, at times. I want to talk a little bit about style guides for, for LLMs. What happens when, when you drop a 100-page style guide into an LLM prompt and why does it fail so far? Why is that complicated?

It's just a lot, a lot of stuff. Context windows in modern LLMs are rather large, but they're not necessarily created equal. There's a lot of research that suggests the farther into the context you go, the worse the results. It just degrades even though the context window is available. So one, trying to get all that in in one shot is difficult. Two, if you need to shove a 100-page style guide in a million times, that's gonna be a lot of money you're spending on tokens. That is kind of— and three, human instructions and LLM instructions can be different sometimes. Like humans have common sense, LLMs don't. So sometimes you just have to be much more precise with LLMs or even just trust the LLM to do the right thing. While with humans, you can express all the different nuances and everything else you really want, and a human can take their time and really understand what's happening.

Yeah, we really need to learn how to speak LLM a lot better. And I think that's something that all of us that deal with the regular chatbots, the AI chatbots on a daily basis, I've seen my language adapt over the last couple of years and my techniques improved that way. Wouldn't you agree that that's an ongoing process?

Maybe I could comment on that. Yeah, there's a lot of research about how interacting with AI and with GenAI in particular is actually impacting the way we speak, not just to the machines, but the way we speak to each other. Right? You learn not to use, as you said, your language, your own language is changing, right?

Absolutely.

You learn not to use certain clauses. A former colleague of ours says that you speak to a model like you speak to a toddler. So I catch myself speaking in simpler, clearer constructs, whether I'm talking to a model or whether I'm talking to somebody else. So obviously it's going to go two-way. We use human language to train the model and model says, hey human, let me train you. So I think it's bidirectional, extremely interesting.

And it's something that I've seen evolve too from the perspective of really talking to the LLM almost like it's a person and complimenting when it does well, kind of like when you're training a dog and seeing it improve performance alongside that. Have you guys done that too, like said thank you or great job to your chatbot?

Well, first of all, it's positive reinforcement, right? So I mean, it's a, it's a part of the process. But yeah, absolutely. I, I'll try to pronounce this word. I might not do good at it. Anthropomorphism, right? Did I say it right? Like when you actually treat an LLM as if it was a thinking, thinking entity. And I think that's a, that's a pretty widespread phenomenon though.

At times it's very, very difficult not to have some kind of almost human-like interaction with the thing when it really does something remarkable. But let's get back, let's get back to our style guides and to what we were talking about. How do you identify and resolve contradictions in legacy style guides when using them in LLMs? Do you have any strategies for that, Ben?

Yeah, yeah. So, We did an analysis of style guides versus published strings in a translation memory, and we discovered pretty quickly that there's a lot of common sense happening with our linguists. They, they'll have an instruction in the style guide, say, in French, France, make something informal, while the linguist is like, oh, this is not appropriate for this audience at all, and then they'll kind of naturally ignore it and make it formal. So we, we see these type of things all the time where There's— the linguist has this common sense that they apply to the rules and they're able to modify it a little bit to be appropriate for the audience. And LMs are just robots and they'll just do exactly what you tell them to do. If you tell them to use the informal register, they'll do it and then it can result in some problems. So to combat this, we actually started going to the translation memory itself to try generating our stylistic information. So you can analyze the TM, what our linguists are doing, you know, they're being evaluated by MQM and other rubrics. Make sure that it's good stuff, and then actually generating stylistic information from that that we could feed into LLMs via RAG to kind of solve these types of problems.

Yeah, that would be an interesting question, I think, for us to you as a scientist who are more aware of the technical intricacies. What's the biggest difference between how human translators and LLMs interpret and instructions? What do we need to pay attention to when we talk to the AI?

I mean, some of the stuff you were just mentioning, right? Like, you are the best possible translator. Like, please. Compliments and talking it up really do work. It's pretty amazing. So one thing that comes up is different models do have slightly different characteristics of this. The GPT model family, for example, is much better at like listening to more intricate instructions, while the Gemini family tends to just kind of find it on its own and just kind of naturally understand the issue and then go for it without in-depth instructions. So, so kind of making sure your prompts and your RAG are kind of adjusted towards that are very important. And again, like, this is kind of like humans, but just don't overdo it. Like if you have a 10-page prompt, like you're, you're probably not going to get the results that you want out of something like that.

Yeah. Have you explored the fine-tuning versus prompt engineering for encoding stylistic nuances? Is that something that you play around with?

Actually, maybe I'll talk about it. I mean, first of all, fine-tuning, talking about ROI, fine-tuning is not cheap, right? So it's rather expensive. And to Ben's earlier point, we do so much work with RAG once you've already extracted the stylistic information. Plus, the— I mean, being a TMS platform, it's internal RAG within the platform. So you already have the glossaries, you're already able to pull relevant content from translation memory. So if a customer— and our research shows that if a customer has quote-unquote healthy, clean linguistic assets that are consistent stylistically and relevant, like substantial volume-wise, usually RAG techniques are rendering enough results for us not to bother with fine-tuning. Now, where we're thinking fine-tuning would help, and we're experimenting with that, is customer might not have enough linguistic assets, customer might not have enough stylistic preferences extractable. Maybe fine-tuning for a domain and help a customer jumpstart and kick off their localization program could be— that's where you could get better bang for your buck. And the other would be obviously fine-tuning for long-tail languages.

There's a lot of talk about unstructured data all of a sudden now too, because LLMs have shown promise in being able to handle not structured data. And then we're diving right into things like knowledge graphs and vector embeddings and the new infrastructures that we're dealing with. How do you imagine the next generation of term bases and glossaries and vector clusters and ontologies to work? Do you have any vision of the future? Are you more leaning towards knowledge graphs? Are you big believers in vector embeddings? You think both have a role? This is a loaded question. Go ahead.

Fair enough. Fair enough. Like everything, I think has its place, right? So vector embeddings have been an absolute savior for us when it comes to hallucination prediction.

Mm-hmm.

Or in the translation space, we naturally have a source text and a target text. Um, you can just compare them via cosine semantic similarity with these embeddings. So you can kind of determine accuracy pretty okay, which is great. So if the LLM goes super off the rails, you can catch it pretty quickly. So vector embeddings has been absolutely phenomenal in that kind of area. Um, I am definitely not ready to throw kind of the lexical TF-IDF style search lookups into the waste bin quite yet.

Yeah.

We still have post editors. I still think a lexical comparison, you know, if you have 100 characters and 90 of them are the same is more valuable than having, you know, 80 of the same characters, but being semantically more similar. Like I don't think that's a good trade-off today. But there may be other things like domain clustering and other things that vector embeddings could be good for. Knowledge graphs, really, the more I think about it, could possibly be our future of what does Terminology 2.0 look like, right? Like, you're getting all these documents in, the documents naturally have relationships, you have named entities all over the place, named people, and now you're like, how, how do I know the pronoun for this person? How do I know all this other information? So you can use LLMs to generate your knowledge graphs. That can be informed back via RAG and then really bring it up to the next level to solve— just sorry, my voice is going— all of these kind of classic problems in the space of like, who is this pronoun? What is the gender of you? What is, you know, all that kind of stuff.

Yeah, I was— one of the things that I've been becoming aware of is that vector embeddings tend to be more focused on nowadays because they're maybe a little bit less expensive, as where knowledge graphs maybe have better results, but they require a lot more work upfront and are more expensive token-wise. Do you have any ideas about that? Where— how practical is it to implement knowledge graphs versus vector embeddings? Interested in getting your opinion about that.

Yeah, so this is still pretty early days for us. Um, for the due to knowledge graph stuff that we want to do requires a lot of LLM processing of, you know, you're looking at a document, you're trying to break it down, you're defining ontology, you're doing all that kind of stuff. Um, and embeddings are super cheap, like $20 for a billion characters, like, level of cheap. So, and that standpoint, embeddings will probably be cheaper ignoring like storage and databases and all that other stuff. But the, the power of the knowledge graph is still the thing that we're really investigating to see if the ROI is there or not for that technology stack. Um, super excited to figure out.

Yeah. What, what role could metadata-rich knowledge graphs play in disambiguating named entities, you think, Olga?

Well, but I would say, I mean, I, I think Ben already covered most of it. Right? If an entity is a part of some interdependent structure and a connected structure such as knowledge graph or ontology, obviously it natively leads to easier disambiguation because suddenly you have much more knowledge about a specific named entity. So that's exactly what we're talking about. Probably named entities and terminology are one of the top shining examples of where knowledge graphs can be, uh, can be, uh, most contributing, right? Because you immediately get information on the relevance, you immediately get information on where it sits in the entire ontology tree. So I think one of, actually probably one of the main application cases for knowledge graphs could be the disambiguation and taking care of different kinds of dichotomies.

Since we're geeking out nicely here, I'm going to ask you a couple more geek questions, then we'll turn away from this, but indulge me. What's the biggest mistake localization teams make when thinking about metadata and could metadata itself become a proprietary differentiator for LSPs or platforms? Olga, I have a feeling you have something to say.

Yeah, yeah, yeah, I'm spinning on my seat here because I definitely want to say a lot. Look, traditionally, what have we thought about as an industry around metadata, right? Usually the information would be what you get from your TU, right, from your translation unit, and usually you would have like your traditional set of whatever the metadata knowledge and then some, whatever, whatever travels with the project. So I think the mistake that we're making is limiting our use of metadata to more traditional localization use cases where in reality the metadata can inform so many things when you work in the AI world. And again, we're going back to what style to translate in, what domain, and in reality, I think localization teams are underestimating the value of the metadata that we're sitting on. Now, when it comes to differentiation, and I think again, thank you, thank you, our standards. Our standards contain a lot of great information. And now, if you look at the outside market, people pay tons of money for labeled data, right? And again, to Ben's earlier point about human in the loop, labeled data is super expensive. And we as an industry, be it a log department or be it us as providers, I mean, IP conversation aside, we sit on a goldmine of metadata that really can help us inform our translation on multiple levels. And can it be a differentiator? It's a huge differentiator for sure.

Yeah, this is the perfect segue. I just wrote an article a couple of weeks ago about data annotation and its future for the localization industry. So let's talk a little bit about how we can move upstream. Why do most localization tools still treat translation as something that happens after the content is written? Now we have a chance to be involved much higher and earlier in the process. Why are we still making that mistake, Ben?

So I guess the main thing is, you know, we for the longest time identified ourselves as translators, right? We talked to our localization managers, we gave them tools, we kind of kept linguists a little bit away in our LSPs, and that kind of created this distance of, you know, somebody creates copy in a company, it goes to a lock manager, it goes to an LSP, it goes to a translator and back out again. So really, I think we need to start breaking that down and really, you know, being in the industry, talk to the VP of marketing, talk to the VP of product, really start going more upstream to where the copy is being created and then figuring out how to meet those people's needs.

Yeah, I think that the biggest threat for the localization industry is probably coming from outside of the industry because somebody's going to make a button that just says translate this and it'll be just a feature, one feature within the many of something else. What would it take for for companies like Smartling to build tools for marketers and product teams, not just for linguists?

Maybe I could start and take that one. I mean, first of all, I'm not sure why you're using future tense. Those buttons are there.

Yeah, yeah, yeah, they're there.

Those buttons are very—

I'm trying to be diplomatic.

No need to be. I mean, we're not oblivious to the outside world. Yes, yes. Translation as a feature, right? There's a new acronym out there, TAF, right? Translation as a feature. It already exists there. Now, the question, the real question is how much can you get out of that button, right? If you just integrate it, whatever, use API, use MCP, if you just integrate external API or external provider into your tool, then the real question is, do you get the consistency, right? Do you get the consistency? Do you get centralization? Do you get reuse? So we started dissecting it internally and we quickly realized that the capabilities of that button are exceptionally limited compared to what a system that has all those different file parsing, being able to store, being able to cache, being able to retrieve, preserve the brand tone and voice. That's where the main value is coming from, right? The quality of translation, those models eventually are gonna become They're going to become good. Generalized models are going to become better, but how much can you actually do with that translation? This is where I think companies like Smartling will be becoming more and more relevant.

Fair enough. How can localization tech position itself as a co-pilot to content creation though? And I think that's an important thing to be thinking about, wouldn't you agree?

Yeah, like I, to everything that Olga said for sure, but also there's the accountability part too, right? Like, We're about to enter the danger zone of AI being right, you know, 90 to 99% of the time, which means if you do something 100 times a day, it's probably going to go wrong at least once, right? So if you're at a marketing— in a marketing department sending an email out to millions of people, you trans— you hit the button, it translates it to a language you don't understand. Do you really want to hit that send, right? Like your, your, your very job is dependent on it. So. Us being in a position to help solve those problems would be super important for the future as well.

Yeah. What kind of content? Go ahead, Olga.

Sorry, sorry, sorry. I guess the way it works is just my thought process.

Yeah, yeah, no problem.

Might be one question behind you guys, but I just wanted, I just realized that there is another huge value add until LLMs become self-policing, right? And loop in terms of The quality, to Ben's earlier point, how can I know what I'm pushing into Swahili? I have no idea. And this is where systems like Smartling with sophisticated AI-based LQA tools and LQE tools can actually come to play because that can be that mitigation. We can manage that mitigation of potential hallucinations, errors, and test for quality because before something goes to publishing, so the publisher that CMO is much better informed about what's going out than if button is just pressed.

Let me connect that to the moving upstream thought that we were discussing before. What content do you think, Olga, is right for upstream multilingual generation the way we were discussing it in the last 5-10 minutes?

I think we're a little bit back to the conversation that we had in the onset of machine translation. Right? When we were talking about the content that's more suitable for machine translation and direct publishing than content that required human review, and that debate is ages old. I think it's the same paradigm. I think you really want to assess the risks and high liability content. You probably still would not want to generate, probably high liability content. You still want to translate and adhere to the source. And inherit the compliance and the properties of the source that are essential for translation. The content, first of all, the content that requires more creativity, I would say that grab this idea and generate a blog. Like creative yet low-risk content is a fantastic, fantastic content in my opinion to be generated. Knowledge bases. Again, if you have your products, aspects. If engineering is doing a good, good job documenting, generating from engineering documentation and UI and generating KBs and potential help systems, I think is another avenue. And I think we see a lot of successes there. And then possibly the entire support infrastructure, right? And support systems. I think they're another, another great candidate. I probably would not go into, like, I mean, into regulated industries. And again, if the content is— if it's high-risk content, I wouldn't put it into agentic systems. I wouldn't put it into generated content types.

Yeah, let's talk a little bit about the risk, uh, to end our conversation here this afternoon. Let's talk a little bit about the risk and the hallucinations that are still part of the AI Techscape, so to speak. How do you explain these risks, Ben, and, and what can we do to avoid them?

Yeah, so for, for example, Smartling, we offer kind of a range of translation solutions, right? You have your tricked old MT, AI, post-edit, no, nothing at all, AI, just humans and editors. And the best thing that I've seen is really breaking the types of content that fit in those buckets. Down in a very concise way and then explaining those to end users. So, you know, user-generated content, reviews, great for pure MT. Your knowledge bases, your support pages, great for automation without humans. You know, your marketing stuff, kind of low creative but kind of high impact, you know, post-edit. And then your creative content, you really want to have just kind of a human going for it. So if you kind of build out these buckets and you can really explain it to your clients of like, here's what you need, this will give you the best possible quality, you know, like go for it. And if you need any help, we'll be here to help you out.

Yeah. Do you, do you guys believe that the human-in-the-loop validation is a permanent requirement or just a transitional phase? What's your opinion about that?

I'm in the— well, as permanent as permanent can be. So if people really need human parity, right? Like a perfect translation would require superhuman intelligence, right? Like even the best translator on earth makes a mistake. So that's not going to be available for forever. And even if you take the best possible LLM today, generate translations, it's still going to be subpar to a professional translator for that. So if you really want the best, the only way to get there is with this human kind of in the loop. And I just don't see a solution against that for at least the immediate future, especially for content types like creative where LLMs just struggle immensely.

And maybe I could add to that. Nobody can— the idea of gold standard, you still need to benchmark against something and actually back to where localization departments and maybe researchers make a mistake just saying like, hey, you know, machine labeling is going to solve all your problems. You still need golden standards to benchmark against. And I'm not aware of any other source, at least not, not at the moment, to develop those referential source of truth, direct assessment-based gold standards. The other thing I want to say, look at what's happening outside of Locke. There was this naivete around probably 2, 3 years that like, hey, you know, you can take unstructured data, unlabeled data. Look what's happening now. How much investment? And how much acquisition is happening in the labeling and annotation space in the last month or two months. If we look at the activity, the discovery is being made by scientific and commercial community that, hey, without human in the loop, you really have hit the wall with the current generation of transformer models and there is no way. So I think the rise of labeling is speaking for itself.

Yeah, there's a tip in there for the localization. Industry for sure, and we have the right people to deal with that kind of labeling. So there's a bright future for the people that are going to jump on that opportunity. It's been a fascinating 40 minutes or so already that we've been talking, and I don't want to be too rude and take up too much of your time. I'm going to give you one parting question I would like to hear an answer on for both of you, a little curveball. What are you most excited about in the localization industry for the future? Is there something in particular that you're— that really sparks your imagination and makes you dream of the future? Who wants to go first?

Go, Ben.

Oh man. Okay. Like, I mean, for me, like, I love solving these hard problems and like everything we've been talking about, knowledge graphs, these LLMs, like what they're good for, what they're not good for. Like exploring the space. Like, like, I honestly love it. Like, I'm in there doing stuff. What, what the linguistic experience is going to be in the future, like, when you talk to linguists themselves, a lot of them are actually really into the idea of this future. They want to be more involved. They want to be, you know, make the next generation of CAD tools. They want to be closer and do the stuff that we were talking about earlier about, like, really being up there with the people making copy and like doing stuff. So like seeing how that's going to evolve out, seeing how, yeah, yeah, how we're all just trying to solve all these really interesting problems and how we can all just kind of lift each other up to get the best possible translation experience.

Awesome. Olga, you want to put a bow on it? Go ahead.

Ah, let me try. I mean, first of all, I've said it a few times in different forums, large language models are called language models for a reason. And the reason may be that language is at the center of it as much as we're like in the world of multimodality. So again, I was having a conversation with one of prominent industry figures. I was like, hey, the time is ours. It's our prime time. Localization industry as an after— as a translation afterthought is a thing of the future. And we actually are becoming the solution to bridge, if not erase, the linguistic divide. So I'm thrilled and super excited. I mean, all the, obviously all the scientific innovation, that's very important, but I'm thrilled to see that global content and global communication that we empower is actually front and center now. So basically we're reinventing the world and it's our time.

Thank you for that beautiful, positive parting thought from you, Olga. I really appreciate it. Thank you, Ben, as well for all your technical specifications here this afternoon. And I can't wait to spend some time with you again at a conference in the not too distant future. Thank you for spending time with us. And until then.

Thanks so much for having us.