#29 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 5]

Let's take a look at some ways to improve the consistency of your labels. Here's a general process you can use. If you are worried about labels being inconsistent, find a few examples and have multiple labelers label the same example. In some cases you can also have the same labeler label an example, wait a while until they've hopefully forgotten or or technical term is wash out. But have them take a break and then come back and relabel it and see if they're even consistent with themselves. When you find that there's disagreements, have the people responsible for labeling. This could be the machine learning engineer, it could be the subject matter expert, such as the manufacturing expert, that's responsible for labeling what is a scratch and what isn't a scratch, and or the dedicated labelers discuss together what they think should be a more consistent definition of a label. Why? And try to have them reach an agreement and ideally also document and write down that agreement. And this definition of why can then become an updated set of labeling instructions that they can go back to label new data or to relabel old data during this discussion. In some cases the labelists will come back and say they don't think the input X has enough information. If that's the case, consider changing the input X. So for example, when we saw the pictures of phones that were so dark that we couldn't even tell what was going on, that was a sign that we should consider increasing the illumination, the lighting with which the pictures were taken. But of course, I know this isn't always possible, but sometimes this can be a big help. And then all this is an iterative process. So after improving X or after improving the labeling instructions, you would ask the team to label more data. And if you think there are still disagreements, then repeat the whole process of having multiple labelers label the same example, repeat measure disagreement, and so on. Let's look at some examples. One common outcome of this type of exercise is to standardize the definition of labels. So between these ways of labeling the audio clip you heard on the earlier video, perhaps the labellers will standardize on this as the convention, or maybe they'll pick a different one and that could be okay too, but at least this makes the data more consistent. Another common decision that I've seen come out of a process like this is merging clauses. So if in your labeling guidelines you ask labelers to label deep scratches on the surface of the phone as well as shallow scratches on the surface of the phone. But if the definition between what constitutes a deep scratch versus a shallow Scratch barely visible here I know is unclear. Then you end up with labors very inconsistently labeling things as deep versus shallow scratches. Sometimes the factory does really need to distinguish between deep versus shallow scratches. Sometimes factory need to do this to figure out what was the cause of the defect. But sometimes I found that you don't really need to distinguish between these two classes and you can instead merge the two classes into a single class, say the scratch class. And this gets rid of all of the inconsistencies with different labelers labeling the same thing deep versus shallow. So merging classes isn't always applicable, but when it is, it simplifies the task for the learning algorithm. One of the technique I've used is to create a new class or create a new label to capture uncertainty. So for example, let's say you ask labelers to label phones as defective or not based on the length of the scratch. So here's a sequence of smartphones with larger and larger scratches. So not sure if you can see these on your display, but let me just make them a little bit more visible here. And I know that all of these are really large scratches if this is a real phone you're buying. So this is just for illustrative purposes. And maybe everyone agrees that the giant scratch is a defect, tiny scratch is not a defect, but they don't agree on what's in between. If it was possible to get them to agree, then that would be one way to reduce label ambiguity. But if that turns out to be difficult, then here's another option which is to create a new class where you now have three labels. You can say clearly not a defect, or clearly a defect, or just acknowledge that some examples are ambiguous and put them in a new borderline class. And if it becomes easier to come up with consistent instructions for this three class problem, because maybe some examples are genuinely borderline, then that could potentially improve labeling consistency. Let me use speech illustration to illustrate this this further. Given this audio clip nearest grocery, I really can't tell what they said nearest grocery. And if you were to force everyone to transcribe it, some labelers will transcribe nearly go, some will say, maybe they'll say nearest grocery. And it's very difficult to get to consistency because the audio clip is genuinely ambiguous. To improve labeling consistency, it may be better to create a new tag, the unintelligible tag, and just ask everyone to label this as nearest unintelligible nearest grocery. And this can result in more consistent labels than if you were to ask everyone to guess what they heard when it really is unintelligible. Let me wrap up with some suggestions for working with small versus big data sets to improve label consistency. And we've just been talking about unstructured data or problems where we can count on people to label the data. For small data sets there's usually a small number of labelers. And so when you find an inconsistency, you can ask the labelers to sit down and discuss a specific image or a specific audio clip and, and try to drive toward agreement. For big data sets, it would be more common to try to get to a consistent definition with a small group and then send the labeling instructions to a larger group of labelers. One other technique that is commonly used, but I think overused in my opinion, is that you can have multiple labelers label every example and then let them vote. And voting is sometimes called consensus labeling in order to increase accuracy. I find that this type of voting mechanism technique, it can work, but it's probably overused in machine learning today where what I've seen a lot of teams do is have inconsistent labeling instructions and then try to have a lot of labelers and then voting to try to make it more consistent. But, but before resorting to this, which I do use but more of a last resort, I would usually first try to get to more consistent label definitions to try to make the individual labelers choices less noisy in the first place rather than take a lot of noisy data and then try to use voting to reduce the noise. I hope that the tools you just learned for improving label consistency will help you to get better data for your machine learning tasks. Will one of the gaps I see in the machine learning world today is that there's still a lack of tools, MLOps or machine learning ops tools for helping teams to carry out this type of process more consistently and repeatedly. So it's not, you know, us trying to figure this out in a jupyter notebook, but instead to have tools help us to detect when labels are inconsistent and to help facilitate the process and improving the quality of the data. So this is something I look forward to hopefully our community working on and developing in terms of improving label quality. One of the questions that often comes up is what is human level performance on a task? I find human level performance to be important and sometimes misused concept. Let's take a deeper look at this in the next video.