Andrew Ng covers label ambiguity and data-quality issues in ML systems: speech recognition with multiple plausible transcriptions, structured data challenges like User ID merges and fraud detection, and noisy inputs like dark images or low-quality audio. The lecture emphasizes improving labeler consistency, selecting informative features, and protecting user privacy.
In the last video, you saw how the right bounding boxes for an image can be ambiguous. Let's take a look at some more label ambiguity examples. We briefly touched on speech recognition in the first week of this course. Here's another example given this audio clip. Sounds like someone was standing on a busy roadside asking for the nearest gas station and then a car drove past. So did they say something right after that? I don't know. So one way to transcribe this would be nearest gas station. And in some places people spell with two M's, so that would be a different way to spell it. And we could have used or ellipsis instead of the comma as well, which would be another ambiguity. Or given the audio has had noise after the last word. Nearest gas station. Did they say something after nearest gas station? I'm not sure actually. So would you transcribe it like this instead? So there are combinatorially many ways to Transcribe this with 1M or 2 EMS commonal ellipses. Whether to write unintelligible at the end of this. Being able to standardize on one convention will help your speech recognition algorithm. Let's also look at an example of structured data. A common application in many large companies is User ID Merge. That's when you have multiple data records that you think correspond to the same person and you want to merge these user data records together. For example, say you run a website that offers online listings of jobs. So this may be one data record that you have from one of your registered users with the email first name, last name and address. Now say your company acquires a second company that runs a mobile app that allows people to log in to chat and get advice from each other about their resumes. It seems synergistic for your business. If you run a listing of online jobs, maybe you merge or acquire a second company that runs a mobile app that lets people chat about their resumes. And from this mobile app you have a different database of users. So given this data record and this one, do you think these two are the same person? One approach to the user ID merge problem is to use a supervised learning algorithm that takes its input to user data records and tries to output either 1 or 0 based on whether it thinks these two are actually the same physical human being. If you have a way to get ground truth data records, such as if a handful of users are willing to explicitly link the two accounts, then that could be a good set of labeled examples to train an algorithm. But if you don't have such a Ground true set of data. What many companies have done is ask human labors, sometimes a product management team, to just manually look at some pairs of records that have been filtered to have maybe similar names or similar zip codes, and then to just use human judgment to determine if these two records appear to be the same person. Because whether these two records really is the same person is genuinely ambiguous. They may and they may not be different. People will label these records inconsistently. And if there's a way to just get them to label the data a little more consistently, you see some examples of how to do this later. Even when the ground truth is ambiguous, then that can help the performance of your learning algorithm. User ID merging is a very common function in many companies. Let me just ask you to please do this only in ways that are respectful of the user's data and their privacy, and only if you are using the data in a way consistent with what they have given you permission for. User privacy is really important. A few other examples from structured data if you are trying to use the learning algorithm to look at a user account like these and predict is it a bot or a spam account? Sometimes that can be ambiguous. Or if you look at a online purchase, is this a fraudulent transaction? So has someone stolen an account and is using a stolen account to interact with your website or to make purchases? Sometimes that too is ambiguous. Or if you look at someone's interactions with your website and you want to know are they looking for a new job at this moment in time, based on how someone behaves on a job board, website or a resume chat app, you can sometimes guess if they're looking for a job, but it's hard to be sure. So that's also a little bit ambiguous. In the face of potentially very important and valuable prediction tiles like these, the ground truth can be ambiguous. And so if you ask people to take their best guess at the ground truth label for tasks like these, giving labeling instructions that results in more consistent and less noisy and less random labels will improve the performance of your learning algorithm. So when defining the data for your learning algorithm, here are some important questions. First, what is the input X? For example, if you are trying to detect defects on smartphones for the pictures you're taking, is the lighting good enough? Is the camera contrast good enough? Is the camera resolution good enough? So if you find that you have a bunch of pictures like these, which are so dark, is hard even for a person to see what's going on, the right thing to do may not be to take this input X and Just label it. It may be to go to the factory and politely request improving the lighting because it is only with this better image quality that the labeler can then more easily see scratches like this and label them. So sometimes if your sensor or your imaging solution or your audio recording solution is not good enough, the best thing you could do is recognize that if even a person can't look at the input and tell what's going on, then improving the quality of your sensor or improving the quality of the input X, that can be an important first step to ensuring your learning algorithm can have reasonable performance. And for structured data problems, defining what are the features to include can have a huge impact on your learning algorithms performance. For example, for user ID merge, if you have a way of getting the user's location, even a rough GPS location, if you have permission from the user to use, that can be a very useful cue for deciding whether two user accounts actually belong to the same person. And of course, please do this type of thing only if you have permission from the user to use their data. This way, in addition to defining the input X, you also have to figure out what should be the target label. Yes, and as you've seen from the preceding examples, one key question is how can we ensure labelers give consistent labels? In the last video and this video, you saw a variety of problems with the labels being ambiguous or in some cases the input X not being sufficiently informative, such as an image is too dark. Let's take these data issues and put them into more systematic framework that will allow us to devise solutions in a more systematic way. Let's go on to the next video to take a look.
