A lesson on Human-Level Performance as a baseline in machine learning. When labels are inconsistent, measured HLP drops, making it easier to beat but harder to build reliable systems. Examples span medical imaging, speech recognition, and fraud detection.
Questions this video answers
66.7% (as reported for the inspector on the six-example set)
100% (HLP raised from 66.7% to 100% on those six examples)
Chapters
I think the use of HLP in machine learning had taken off partly because it helped people get papers published to show they can beat hlp. But it's also been a bit misused in settings where the goal is to build a valuable application, not just to publish a paper. When the ground truth is externally defined, then there are fewer problems with HLP when the ground truth really is some real ground truth. For example, I've done a lot of work on medical imaging, working on, you know, AI for diagnosing from X rays or things like these, and given an X ray image, if you want to predict the diagnosis, if the diagnosis is defined according to, say, a biopsy, so a biological or medical test, then HLP helps you measure how. How well does a doctor versus a learning algorithm predict the outcome of a biopsy or a biological medical test? I find that to be really useful. But when the ground truth is defined by a human, maybe even a doctor, labeling an X ray image, then HLP is just measuring how well can one doctor predict another doctor's label versus how well can one learning algorithm predict another doctor's label? And that too is useful, but it's different than if you're measuring how well you versus a doctor are predicting some ground truth outcome from a medical biopsy. So to summarize, when the ground truth label is externally defined, such as the medical biopsy, then HRP gives an estimate for Bayes error and irreducible error in terms of predicting the outcome of that medical test, the biopsy. But there are also a lot of problems where the ground truth is just another human label. The visual inspection example we had from the previous video showed this, where the inspector had 66.7% accuracy. Rather than just aspiring to beat the human inspector, it may be more useful to see why the ground truth, which is just some other inspector compared to this inspector, don't agree. For example, if we look at the length of the different scratches that they label, say on these six examples, these were the lengths of the scratches. And if we speak of the inspectors and have them agree that 0.3 mm is the threshold above which a stretch becomes a defect, then what we would do is, then what we realize is that for the first example, both labeled it 1 totally appropriately. For the second example, the ground truth here is 1, but is less than 0.3, so we really should change this to 0, then 0.5 gets 1 1, 0.200, 0.1. And this example has a stretch of 0 0.0.1. But really this should have been a 0. If we go through this exercise of getting the ground truth labor and this inspector to agree, then we actually just raised human level performance from 66.7% to 100%, at least as measured on these six examples. So. But notice what we've done by raising HLP to 100%. We've made it pretty much impossible for a learning algorithm to beat hlp. So that seems terrible. You can't tell the business owner anymore you beat HLP and thus they must use your system. But the benefit of this is you now have much cleaner, more consistent data and that ultimately will allow your learning algorithm to do better. So when your goal is to come up with a learning algorithm to that actually generates accurate predictions, rather than just prove for some reason that you can beat hlp, I find this approach of working to raise HLP to be more useful. To summarize, when the ground truth label Y comes from a human HLP being quite a bit less than 100% may just indicate that the labeling instructions or labeling convention is is ambiguous. On the last slide, you saw an example of this in visual inspection. You also see this in speech recognition, where the comma versus ellipsis, that type of ambiguous labeling convention will also cause HLP to be less than 100%. Improving label consistency will raise human level performance, and this makes it harder, unfortunately for your learning algorithm to beat hrp. But the more consistent labels will raise your machine learning algorithm performance, which is ultimately likely to benefit the actual application. So far we've been discussing HLP on unstructured data, but some of these issues apply to structured data as well. You already know that structured data problems are less likely to involve human labors and thus HLP is less frequently used. But there are exceptions. You saw previously the user ID merging example where you might have a human label, whether two records belong to the same person or I've worked on projects where we would look at network traffic into a computer to try to figure out if a computer was hacked. And we asked humanity, human IT experts to provide labels for us. Sometimes it's hard to know if a transaction is fraudulent and you just ask a human to label that. Or is this account a spam account or a bot generated account? Or from gps, what is the mode of transportation? Is this person on foot or on a bike or in a car or in a bus? It turns out buses stop at bus stops and so you can actually kind of tell if someone's in a bus or in a car based on their GPS traffic trace. And for problems like these it would be quite reasonable to ask a human to label the data, at least on the first pass for a learning algorithm to make such predictions as these. And so when the ground truth label you're trying to predict comes from one human, the same questions of what does HLP mean? It is a useful baseline to figure out what is possible. Sometimes when measuring HLP you realize that low HLP stems from inconsistent labels and working to improve HLP by coming up with a more consistent labeling standard will both raise HLP and give you cleaner data with which to improve your learning algorithms performance. Here's what I hope you take away from this video. First, HLP is important for problems where human level performance can provide a useful reference. I do measure it and use it as a reference for what might be possible and to drive error analysis and prioritization. Having said that, when you're measuring HLP, if you find that HLP is much less than 100%, also ask yourself if some of the gap between HLP and complete consistency is due to inconsistent labeling instructions. Because if that turns out to be the case then then improving labeling consistency will raise HLP and also give cleaner data for your learning algorithm, which will ultimately result in better machine learning algorithm performance. Here's what I hope you take away from this video. HLP is useful and important for many applications. For problems where I think how well humans perform is a useful reference, I do measure HLP and and then use that to get a sense of what might be possible and also use HLP to drive error analysis and prioritization. Having said that, if in the process of measuring HLP you find that HLP is much less than perfect performance, much lower than 100%, then it's also worth asking yourself if that gap between HLP and 100% accuracy may be due to inconsistent labeling instructions. Because if that's the case, then improving labeling consistency will both raise HLP but more importantly help you get cleaner and more consistent labels which will improve your learning algorithms performance.
