SOTA machine translation at Unbabel

We have started a project on this. It's to combine these systems, these quality estimation systems with the machine translation itself. So that is something that we started working on this. But I believe that you can work on this for the next few years and there is a lot of things that we can improve there. Yeah, that gets me really excited. I think it's a direction that it's going to be really nice.

This is the Quality Aware decoding project. That is basically what I just mentioned about what we have been talking about of having these quality predictions about the hypothesis translations. The idea behind this project that Riccardo is talking about is what if we bring the quality estimation or comment already to inside the MT process and then we can make the machine translation aware or more aware about its quality having a signal from a different model. So this is what this project is about.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO we deploy our app servers close to our users and you can too learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined this week by Ricardo Ray and Jose Souza from Unbabel here at EMNLP 2022 in Abu Dhabi. How you doing, Gu?

Hi, we are fine.

Hi. Good. Yeah.

How's EM NLP for you?

So far we have been mostly attending WMT workshop.

Yeah. And what's wmt? What does that stand for?

Right. WMT stands for Workshop on Statistical Translation. Workshop on Machine Translation on Machine Translation. But this is an historical acronym because it's actually now a conference. I would say that it's the main conference of machine translation and it has been happening for several years and it's always co located with EM nlp. So it's nice because it's one of the biggest NLP conferences together with the biggest MT conference.

It's mostly attended by researchers, so not so much by people in localization industry. But it's interesting to know what's happening in terms of research, the latest approaches and methodologies for evaluation as well.

Yeah. And is that the industry that UNBABEL is in? Could you just give people a little bit of an understanding of what UNBABEL is?

Sure.

So UNBABEL is a translation Company, we provide translations trying to unite the best of both worlds, which is using machine translation and professional translators to provide these translations and the best of both worlds. Because if you only rely on translators themselves, it's very difficult to scale this process of translation to different volumes of content. And that's why you use machine translation to speed up this process. And then we use the translators to correct if necessary. And that's I think, the biggest difference of UNBABEL to other companies, which is we are the pioneers to use something called quality estimation to actually decide whether if we should post edit or not the translations. And I guess we are big on also evaluation, technology evaluation. And I think Ricardo can talk about comet.

Yeah, like what Jae just explained about the difference between combining humans and mt. So if you have a mechanism that tells you that your machine translation output is perfect, then you don't need a human. But for you to do this, you clearly need a very reliable quality estimation system, a system that receives that translation and is able to give you an accurate score for that translation. And that's why NBABO has been focusing for so many years on specifically quality estimation and also evaluation. Evaluation, it's a little bit more general. It can also include things like metrics where you compare the translation output with a reference translation that you believe to be perfect. And it's what people typically use when training models and stuff like that. For the past few years we have been developing a metric that is being widely adopted by the research community and also the industry, which is called comet. Comet has been very successful in the last two years. And yeah, it was developed by us. We also developed a quality estimation framework that was also gained a lot of traction three years ago, I think.

Yeah, 2019 it was.

Yeah, called OpenQE, which is basically similar in terms of the model approach and everything, but it does not rely on a reference. So it's what we use internally for performing quality estimation. Yeah, I think this sums up a little bit.

That said just one thing is that all of this is only possible because over the years UNBABEL established some quality controls for the translations. And this started by using a framework called mqm, which stands for multidimensional quality metric, which is basically a typology, and then guidelines on how to use this typology with different phenomena that happens when translation is made, that goes from accuracy, whether the translations are adequate, if they're fluent, and then there's a whole taxonomy about that. So this kind of evaluation enabled us to accumulate data about the quality of translations over time that we can then use to train quality estimation or metric evaluation models.

Yeah, so this seems different. I think some listeners probably in their experience with modeling and other domains or with other data are probably familiar with a confidence score or a probability. So this goes way beyond that. Right. So just to clarify, this is not just a confidence score coming out of your model of translation, but this is actually a metric that you're running on the output of your model, is that right?

Exactly, yeah, exactly.

Yeah. So explain maybe Comet a little bit because that has gained so much traction. What is maybe different about comet? Another popular one I know for machine translation is called blue. So what distinguishes Comet as different from maybe that or like other metrics that are out there?

So like we were saying, BLEH is a very well known metric, but blue is a lexical metric. And this means that BLEU will take the empty output and it will compare with a reference that was created from a human. And usually the typical setup is that we only compare that empty output with a single reference. And as we might know, there are multiple ways to translate a specific sentence. So a lot of times BLEU will give a very low score for a very good translation because of that. Sometimes it also gives you a very high score for a very bad translation because of another aspect of bu, it's that it's going to give the same weight to all worlds. So if you have a named entity that is not correctly translated, it's going to be like one word that is missing from being perfect. And BU will give a very high score. If you miss like a punctuation, the score penalty will be exactly the same, although the errors are completely different in terms of severity.

Just one thing to differentiate between, just to explain a little bit more Bleu, is that the way that it looks at both the translation hypothesis and the references, looking at each word and trying to understand if there is an overlap of each word with the reference. So and it does that for combinations of, for one word or for combinations of two, three and four words, usually which we call N grams. And then it has a brevity penalty that is basically to penalize if the translation is too small, too short. So that's basically the rationale. And there is a class of metrics like that I think we are calling lexical metrics.

Yeah, lexical metrics, yeah.

So tr, which is translation error rate, it's similar to that. Chrf, it's similar to that, but CHRF goes at the character level. So this is a class of things that is very different From Comet.

I think yeah, Comet takes advantage of the representations coming for large language models like XLM Roberta. We have been using XLM Roberta and basically those representations allow you to compare words in an embedding space. So two words that might not be exactly the same but have the exact same meaning, the COMET will use those representations to output the score. Now the other thing that we add on top is that we train those representations to be more suitable for the specific task of machine translation evaluation. And I'm saying this because this is a very important difference from other metrics that have also been proposed like BERT score, where because of the fact that you don't have any fine tuning on top, if you use Bird score and you say I love you or I hate you because love and hate will have similar embeddings, the score will be very high when in fact they are the complete opposite. So we start from a pre training model, but then by training the model with some supervision from human labels on errors, the model learns that I love you or I hate you on for this specific task they are complete opposites. And I think that kind of splits apart Comet from all the metrics that were being proposed before that either fall into the lexical category or into the embedding category.

Yeah, that's great. And you also mentioned, just in passing, like there was another kind of category of quality estimation that didn't require a reference. Could you talk about that a little bit?

Yeah. So the idea is very similar to the idea of comet. So the difference is that when you have access to a reference, which is the case of comet, when you create the embeddings for the empty outputs, they will be perfectly aligned with the embeddings from the reference because they are on the exact same language. On quality estimation you are comparing it directly to the source so the embeddings will not align perfectly. And still what happens is that during training, using human supervision, the model learns to what is correct and what is incorrect, only comparing the empty output directly with the with the source.

So quality estimation serves a different kind of application than the metrics like bleU, chrf and comet, which is usually I want to know what is the quality of specific sentences or translations given their source sentences for Comet, usually what you're more interested COMET or the other metrics, you're more interested in understanding the difference between models or MT systems. So you're evaluating at some sort of try to understand at some sort of test set level or evaluation set level so that you can decide whether I go with Model empty model A, B or C. And then in quality estimation is basically to take decisions on the fly at real time in which I cannot wait for someone to make a reference or a post edition and decide, okay, can I trust this translation? If I don't, should I throw it out? Is that bad that they should do it from scratch? Or I can still give it to someone that can repurpose this and rephrase it, you know, so they're slightly different in the applications, but this is something that you can talk about, about trends they start to like. I think Ricardo was teasing to intersect themselves a bit.

I would say that the metrics field, so the evaluation on the metrics side was stuck with BL for a long time. Quality estimation on the other hand was. I feel that there were more research and more innovation on that field. Actually that was our motivation when we built Comet. We tried to replicate what was being done. The state of the art of what was being done on quality estimation, we tried to bring it to the metrics field and now the modeling approaches are very similar, but it was viewed as two completely different tasks for years.

So just to give insight on a bit of context on what Ricardo said about the progress in quality estimation, so I did my PhD working on this kind of problem and I finished in 2015. So I was working from 2012 until 2015 on problems around this. And the approaches back then they were basically using feature based approaches like classical machine learning and with deep learning and access to embeddings and now large printing models, this very, very fast shifted to this kind of approaches and the performance of these models, of these approaches also are much better than when I used to first work on this. So the quality of these quality estimation models nowadays that they are very useful, you can actually do a lot of things with them, like I was saying. And yeah, I just wanted to compliment that because for me it was. I was not working on the field specifically this problem for, I don't know, three years. I guess when I came back to it it was, whoa, now it's really up to everything. You know.

Could you explain a little bit? So you mentioned how like in Comet or in these other models you might be comparing like the embeddings of, of words, but words don't always map like one to one between languages. And sometimes I don't know if you're looking at sentences or other things. But, but could you describe some of the, I guess what are the main challenges looking forward that aren't solved yet in terms of next steps with Quality estimation and things that you're looking at now that you see as open problems.

Yeah, you actually touched a very nice point. I wouldn't say that it's not that the words don't align very well, but sometimes what we see is that the embeddings themselves for certain specific words are not discriminative enough. And we have seen some. For instance, if you translate the sentence this ipoh costs 50 cents, you translate it to Portuguese and the translation needs. I'm not going to say it in Portuguese but pretend that I'm speaking Portuguese. The perfect translation would also be 50 cents. But for some reason DMT might have hallucinated and say that it's $0.500. So it's basically changing the price of an Apple. And this is critical error in much scenarios. But if you look at the embedding space of the 500 or the embedding of 50, it's gonna be very similar and it's gonna be very hard for the neural network that is trying to differentiate these two things, it's gonna be a very hard task because there is not enough signal. You also see the same thing with some named entities. And currently there has been some work, some progress in trying to look at the quality estimation and metrics and try to figure out why they are not working for this kind of very specific phenomena. Actually yesterday we had a lot of presentations about challenge sets that tried to test metrics for these specific phenomena. So in WMT we have several competitions, several what we call share tasks and inside the metrics share task where people are trying to compete to create better metrics, there was also a shared task that we call this challenge set subtask where people submit examples that are challenging for metrics and then the participants from the metrics task have to score those examples and then we get the scores back to the developers of the challenges for them to analyze. And a lot of people looked into this and tried to make some suggestions for future work in how to improve metrics for this. So if you guys are interested in this, I take a look at the findings from the metrics task because they are interesting findings and pointers for future work in this area.

One of the problems of this model based MT evaluation approaches is that first they are based on the data that the pre trained models were trained on. So there's everything there, there is bias and there is the limited amount or it can be a lot of data as well, but all the idiosyncrasies of that data are encoded in the pre trained models. Then when you fine tune this for the specific tasks that they need to work on, namely quality estimation and MTE evaluation, they also are limited in data in the sense that we have orders of magnitude less label data for this fine tuning process. So this can have its biases and it can have. Also like taking the example of Apple, for some reason you never seen Apple, the company, but you saw only for the fruit. So you start to, every time you see Apple, you translate that to the fruit. You know, you actually say that it's. If the model translates that to the, to the fruit, the evaluation thing is going to say it's fine because in the evaluation data that you used to train the model, you never saw for some reason the brand. And this is related to the named entity problem that Ricardo was saying. So I think one step, we are giving the first step as a community to understand that now and really poke it and see, okay, there's a hole here. And now the next step is how to, you know, alleviate that problem. I don't think it's possible to alleviate completely solve, but we are for sure we'll try to alleviate this for these models. Now there is a. And there are a lot of complaints also of people, not complaints, but you know, even us when we are using different models, not only ours, we see that these models fall short sometimes. And this can be very bad in a commercial setting or even in sensitive scenarios in which if you get 2 cents and the model that translated this to, I don't know, 2 million, that's not very nice. Right? You might have some legal implications with that. So, yeah, I don't know other open problems. I think for me, one big problem is that, and this is also a trend that we see in the metrics and in the quality estimation task is that bigger models have better predictive power. So people usually what they are doing is just throw more GPUs at it and just train a bigger model. And this seems to be giving improvements as well. But the problem is that not every practitioner can actually use these models once they are trained because they need bigger and bigger GPUs which are costlier even at inference time. So we actually had a paper in emt, the European association for Machine Translation Conference, that was actually making comets smaller. And it's like diminutive in the name of the paper. The name of the model is Cometino, which is a diminutive of comet, like Portuguese, very Portuguese way to say it. And it was also a first step towards that. But I think there is a lot to be done for all the other models and also for Comet?

Yeah, definitely. I think Cometino was just the first step into that direction with there is a lot of things that can be improved in distillation of these models, even the evaluation models like we did for Komatting and not just for evaluation. We have been focusing this podcast a little bit on evaluation, but on machine translation you have the same problem on machine translation. Bigger models have been achieving impressive machine translation quality, but it's very hard for everyone to develop those models and it's even harder for people to deploy those models. We face this at unbabel. We develop our own machine translation systems and we have seen this trend. We get improvements if we keep scaling our empties, but then we have difficulties serving those empties. And it's also we know that not every company has the capacity to build some such big models like big tech companies develop. So yeah, it's not just in the evolution side but also in machine translation side. It is something that people should look forward to. It's without losing performance how to make these things smaller and easier to deploy.

Yeah. And would you say on the model side specifically like Jose, you mentioned sort of models getting bigger and bigger. Some people might have seen nice giphys about an encoder decoder and one language coming in and one language coming out and transformer models. But what are some things others are exploring, maybe yourselves that are either different approaches or you mentioned distillation and all these other things to make models smaller. But are there different architectures or techniques being explored? I think I saw one of your papers, something about KNN or something. I don't know if you can speak to that.

But yeah, we just at this moment there is a poster on the usage of KN MT for the chat share task. So this is something called, I think this is broadly called dynamic adaptation. And one approach to that is doing NMT that rather than actually fully fine tuning one base model like one of these large pre trained models, you actually just do some data retrieval approach in which you combine the contents of a data store that has relevant data for the use case that you're trying to serve with machine translation. And then at decoding time, when you are assembling the translation using the translation probabilities of the model, you interpolate these probabilities with the probabilities of words or expressions contained in the in the data store. So this way you avoid having to fully fine tune a model for each use case that you have. And this is something that we started to research and approach at unbabel. But I just must say that this doesn't solve the problem of the base model being big. You just avoid fine tuning it completely. So there's still the problem of, okay, how do I shrink or compress this model so that it can reliably and cheaply explored it for translation? And this is like you said, distillation, quantization and other compressing techniques just to.

Complement what Jose was saying about the key and nearest neighbor approach. Another very big advantage of this is that it's very easy to combine with translation memories, which we know that they are wildly used in a translation industry. And this is a seamless way to basically take the MT and make the MT work with those translation memories because you can build this data store that will help the model to translate the content accordingly. So just to add that also, which I believe that it's very important for the localization industry in general. Great.

Yeah.

Well, we've talked a lot about challenges, I guess, which is fun to talk about at a research conference for sure. What are some things just generally about the machine translation industry or UNBABEL or other things that make both of you excited and optimistic about the future? What are some of those things that excite you? It doesn't have to be an MT or things you've seen at this conference or things that you're following that give you some encouragement and excitement about the future of the space where we're working.

Actually, I'm very passionate about evaluation in general. I think that shows up in my work because I mostly work on evaluation. I've been getting very excited with the progress that we have been doing in evaluation. I think we have started a project on this that it's to combine these systems, these quality estimation systems with the machine translation itself. So that is something that we started working on this. But I believe that you can work on this for the next few years and there is a lot of things that we can improve there. Yeah, that gets me really excited. I think it's a direction that it's going to be really nice.

Yeah. This is the quality Aware decoding project. That is basically what I just mentioned about what we have been talking about of having these quality predictions about the hypothesis translations. The idea behind this project that human Picard is talking about is what if we bring the quality estimation or comment already to inside the MT process and then we can make the machine translation aware or more aware about its quality having a signal from a different model. So this is what this project is about. So we have a paper at NACO this year describing that. So yeah, this is pretty exciting. And I think in terms of more broad challenges, what I find interesting is that I don't believe that translation is solved. I think a few years ago some people claimed that there was human parity between MT systems or some empty models and humans and translators. But then it turned out that the actual translators that were used were not really professional translators. Like I know English, right. But I'm not a native speaker and I cannot translate everything. So I'm not a subject matter expert on different topics. So I cannot actually, if you give me some chemistry content to translate into English from Portuguese, I cannot do it. Right. So I think what's exciting is to see that the technology is allowing us to translate better and better. Maybe compared to me as a non native speaker when I'm translating some content, but still there is a lot of, there are a lot of challenges to actually translate very well, very specific content that requires very specific terminology and very specific way of actually building the sentences. And what is much better is actually the fluency that these machine translation models are giving nowadays. But what remains still a challenge is that sometimes the translations, they look very good, but they are not on point. So they are not adequate. They are talking about something slightly different or completely different. So I think this is exciting. I mean, not everything is solved, but at the same time is encouraging. Is encouraging in this sense.

Yeah. Great. Well, as we close out here, where can people find out more about UNBABEL and specifically maybe some of this really research that's going on. And also you mentioned beforehand that Unbabel was possibly hiring as well. Where could people find out about that?

Right. So we have our website like unbabel.com and we have our Twitter handle like Unbabel. You can follow our news from there. We just put up a research blog in which we are going to be writing about our research. This is going to be possibly in the links in your info box. I don't know.

Yeah, we'll put it in the show notes for sure.

And yeah, we are also hiring soon. Like we are starting to accept applications for the next year for research scientists in different levels and different geographies. So Unbabel, we didn't talk about it, but it was born in Portugal, in Lisbon. But now we have offices all around the world. We have offices in the west coast, in the the US in the east coast, London and some other places in Europe. And we are going to post this also to give an email for contact for people who are interested in other research that we're doing and other works. We have open positions not only for research scientists but also for the engineers and other positions that are not technical.

Well, yeah. Thank you, Jose. Thank you, Ricardo. Really appreciate you taking time. I know there's a lot of good posts, posters around to see and all that, so thanks for taking time.

Thanks, Daniel.

Thank you.

All right, that is our show for this week. If you dig it. Don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to Fly IO for backing our dynamic requests to Breakmaster Cylinder for the Beats and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.