C5W3L06 Bleu Score (Optional)

One of the challenges of machine translation is that given a French sentence, there could be multiple English translations that are equally good translations in that French sentence. So how do you evaluate a machine translation system if there are multiple equally good answers, unlike, say, image recognition, where there's one right answer, you can just measure accuracy. If there are multiple great answers, how do you measure accuracy?

The way this is done conventionally is.

With something called the blue score.

So in this optional video I want.

To share with you, I want to give you a sense of how the bleu score works.

Let's say you are given a French sentence, le chi e sur le tapis, and you are given a reference human generated translation of this, which is the cat is on the mat. But there are multiple pretty good translations of this. So different human, different person might translate it as there is a cat on the mat. And both of these are actually just fine, perfectly fine translations of the French sentence. What the bleu score does is given a machine generated translation, it allows you to automatically compute a score that measures how good is that machine translation. And the intuition is so long as the machine generated translation is pretty close to any of the references provided by humans, then it will get a high bleu score. Blue, by the way, stands for bilingual Evaluation. Understudy. So in the theater world, an understudy is someone that learns the role of a more senior actor so they can take over the role of the more senior actor if necessary. And motivation for blue, and motivation for blue is that whereas you could ask human evaluators to evaluate the machine translation system, the bleu score is an understudy could be a substitute for having humans evaluate every output of a machine translation system. So the bleu score was due to Kishore Papineni, Salim Rukus, Todd Ward and Wei Jing Zhu. This paper has been incredibly influential and it's actually quite a reasonable and it's actually quite a readable paper. So I encourage you to take a look if you have time. So the intuition behind the bleu score is we're going to look at the machine generated output and see if the types of words it generates appear in at least one of the human generated references. And so these human generated references will be provided as part of the dev set or as part of the test test set. Now let's look at a somewhat extreme example. Let's say that the machine translation system abbreviating machine translation is mt. So the machine translation or the MT output is dddddddddd. So this is a clearly A pretty terrible translation. So one way to measure how good the machine translation output is is to look at each of the words in the output and see if it appears in the references. And so this could be called a precision of the machine translation output. And in this case, there are seven words in the machine translation output. And every one of these seven words appears in either reference one or reference two. Right? So the word the appears both references. So each of these words looks like a pretty good word to include. So this will have a precision of 7 over 7. It looks like there's a great precision. So this is why the basic precision measure of what fraction of the words in the empty output also appear in the references. This is not a particularly useful measure because it seems to imply that this empty output has very high precision. So instead, what we're going to use is a modified precision measure in which we will give each word credit only up to the maximum number of times it appears in the reference sentences. So in reference one, the word the appears twice in reference to the word the appears just once. So two is bigger than one. And so we're gonna say that the word D gets credit up to twice. So with a modified precision, we will say that it gets a score of 2 out of 7, because out of 7 words will give it a 2 credits for appearing. Um, so here the denominator is the count of the number of times the word the appears seven words in total. And the numerator is the count of the number of times the word D appears. But we clip discount. We take a max or clip discount at 2. So this gives us the modified precision measure. Now, so far we've been looking at words in isolation in the bleu score. You don't want to just look at isolated words. You maybe want to look at pairs of words as well. Let's define a portion of the blue score on bigrams. And bigrams just means pairs of words appearing next to each other. So now let's see how we could use bigrams to define the blue score. And this would just be a portion of the final bleu score. And we'll take unigrams, or single words, as well as bigrams, which means pairs of words into account, as well as maybe even longer sequences of words, such as trigrams, which means three words appearing together. So let's continue our example from before. We have the same reference 1 and reference 2. But now let's say the machine translation or the empty system has a slightly better output. The cat the cat on the mat. Still not a great translation, but maybe better than the last one. So here the possible diagrams are, well, there's the cat gonna ignore case, and then there's cat D, there's another diagram, and then there's dcat again, but I've already had that, so let's skip that. And then cat on is the next one. And then on the. And the map. So these are the bigrams in the machine translation output. And so let's count up how many times each of these bigrams appear. Um, the cat appears twice, cat D appears once, and the others all appear just once. And then finally, let's define the clip count. So count and then subscript clip. And to define that, let's take this column of numbers, but give our algorithm credit only up to the maximum number of times that that diagram appears in in either reference one or reference two. So the cat appears a maximum of ones in either of the references. So I'm gonna clip that count to one cat D. Well, it doesn't appear in reference one or reference two, so I'll clip that to zero cat on. Yep, that appears once. We give it credit for once, OND appears once, give that credit for once, and DMAT appears once. So these are the clipped counts. We're taking all the counts and clipping them, really reducing them to be no more than the number of times that bigram appears in at least one of the references. And then finally, our modified bigram precision will be the sum of the count clips. So, so that's 1, 2, 3, 4 divided by the total number of bigrams, that's 2, 3, 4, 5, 6. So 4 out of 6 or 2 thirds is the modified precision on bigrams. So let's just formalize this a little bit further with what we had developed with on unigrams. We define this modified precision computed on unigrams as p subscript 1. So p stands of precision, and the subscript 1 here means we're referring to unigrams, but that is defined as sum over the unigrams. So that just means sum over the words that appear in the machine translation output. So this is called y hat of count clip of that unigram divided by sum overall unigrams in the machine translation output of count number of counts of that unigram. Right. And so this is the what we had gotten as, I guess as two out of seven, two slides back. So the one here refers to unigram, meaning we're looking at single words in isolation. Um, you can also define PN as the N gram version instead of a unigram for you have N grammar. So this would be sum over the N grams in the machine translation output of count, clip of that n gram divided by sum over n grams of the count of that N grammar. And so this, these precisions or these modified precision scores measured on unigrams or on bigrams, which we did on the previous slide, or on trigrams, which are triples of words, or even higher values of N for other N grams. This allows you to measure the degree to which the machine translation output is similar or maybe overlaps with the references. And one thing that you could probably convince yourself of is if the empty output is exactly the same as either reference 1 or reference 2, then all of these values, P1 and P2 and so on, they'll all be equal to 1.0. So to get a precision or a modified precision of 1.0, you just have to be exactly equal to one of the references. And sometimes it's possible to achieve this even if you aren't exactly the same as any of the references, but kind of combine them in a way that hopefully still results in a good translation. Finally, Finally, let's put this together to form the final BLEU score. So P subscript N is the bleu score computed on N grams only, or also the modified precision computed on N grams only. And, and by convention, to compute one number, you compute P1, P2, P3 and P4, and combine them together using the following formula. Um, it's gonna be the average, so sum From n equals 1 to 4 of pn and divide that by 4. So basically taking the average um, by convention, the bleu score is defined as E to the this, and exponentiation is a linear operate. Exponentiation is a strictly monotonically increasing operation. And, and then we actually adjust this with one more factor called the. Bp penalty. So bp. Stands for a brevity penalty. The details maybe aren't super important, but just give you a sense. It turns out that if you output very short translations, it's easier to get high precision, because probably most of the words you appear, probably most of the words you output appear in the references. So, but we don't want translations that are very short, um, so the BP or the brevity penalty is an adjustment factor that penalizes translation systems that output translations that are too short. So the formula for the brevity penalty is the following. Um is equal to 1 if your machine translation System actually outputs things that are longer than the human generated reference outputs. And otherwise is some formula like that that overall penalizes shorter translations. And the details you can find in this paper. So once again, earlier in this set of clauses, you saw the importance of having a single row number evaluation metric, because that allows you to try out two ideas, see which one achieves the highest score, and then, you know, try to stick with the one that achieved the highest score. So the reason the BLEU score was revolutionary for machine translation was because this gave a pretty good, by no means perfect, but pretty good single real number evaluation metric. And so that accelerated the progress of the entire field of machine translation. I hope this video gave you a sense of how the Blue School works. In practice. Few people would implement the blueschool from scratch. They're open source implementations you can download and just use to evaluate your own system. But today blueschool is used to evaluate many systems that generate text, such as machine translation systems, as well as the example I showed briefly earlier of image captioning systems where you would have a system, have a neural network, generate an image caption and, and then use the BLEU score to see how much that overlaps with maybe a reference caption or multiple reference captions that were generated by people. So the BLEU score is a useful single row number evaluation metric to use whenever you want your algorithm to generate a piece of text and you want to see whether it has similar meaning as a reference piece of text generated by humans. Uh, this is not used for speech recognition, because for speech recognition there's usually you have one ground truth and you just use other measures to see if you got a speech transcription on pretty much exactly word for word. Correct. But for things like image captioning, multiple captions for a picture could be about equally good. Or for machine translation, there are multiple translations about equally good. The BleuSchool gives you a way to evaluate that automatically and therefore speed up your Alphabet development. So with that, I hope you have a sense of how the BLEU score works.