When machine learning algorithms approach human performance, it becomes unclear whether to reduce bias or variance next. The speaker uses examples like team-of-humans error at 0.5% versus single human at 1% to explain why. ML beats humans easily on structured-data problems like ad prediction and loan repayment, but struggles with perception tasks like speech and vision.
Questions this video answers
0.5% error.
1% error.
Training error 0.6%; dev error 0.8%.
Training error 0.3%; dev error 0.4%.
Chapters
Long teams often find it exciting to surpass human level performance on a specific recognitional classification task. Let's talk over some of the things you see if you're trying to accomplish this yourself. We've discussed before how machine learning progress gets harder as you approach or even surpass human level performance. Let's talk over one more example of why that's the case. Let's say you have a problem where a team of humans discussing and debating achieve 0.5% error, a single human 1% error, and you have an algorithm with 0.6% training error and 0.8% dev error. So in this case what is the avoidable bias? So this one is relatively easier to answer. 0.5% is your estimate of Bayes error. So. So your avoidable bias. You're not gonna use this 1% number as reference, you're gonna use this difference. So maybe you estimate your avoidable bias as at least 0.1% and your variance as 0.2%, so there's maybe more to do to reduce your variance than your avoidable bias. Perhaps. But now let's take a harder example. Let's say team of humans and single human performance is same as before, but your algorithm gets 0.3% training error and 0.4% dev error. Now, what is the avoidable bias? It's now actually much harder to answer. That is the fact that your training error 0.3% does this mean you've overfitted by 0.2%? Or is Bayes error actually 0.1%? Or maybe as Bayes error is 0.2% or. Or maybe Bayes error is 0.3%. You don't really know, but based on the information given in this example is actually, you actually don't have enough information to tell if you should focus on reducing bias or reducing variance in your algorithm. So that slows down the efficiency which you can make progress. Moreover, if your error is already better than even a team of humans looking at and discussing and debating the right label for an example, then it's just also harder to rely on human intuition to tell your algorithm what are ways that your algorithm could still improve the performance. So in this example, once you've surpassed this 0.5% threshold, your options, your ways of making progress on the machine learning problem are just less clear. Doesn't mean you can't make progress. You can always. You might still be able to make significant progress, but some of the tools you have for pointing you in a clear direction just don't work as well. Now, there aren't many problems where machine learning significantly surpasses human level performance. For example, I think online advertising, estimating how likely someone is to click on an ad, probably learning algorithms do that much better today than any human could. Or making product recommendations, recommending movies or books to you. I think that websites today can do that much better than maybe even your closest friends can. Or logistics, predicting how long it'll take you to drive from A to B, or predicting how long it'll take a delivery vehicle to drive from A to B, or trying to predict whether someone will repay a loan and therefore whether or not you should approve a loan offer. All of these are problems where I think today, machine learning for faster process a single human's performance. Notice something about these four examples. All four of these examples are actually learning from structured data where you might have a database of what ads users have clicked on, database of products you've bought before, databases of how long it takes to get from A to B, database of previous loan applications and their outcomes. And these are not natural perception problems. So these are not computer vision or speech recognition or natural language processing tasks. Right? Humans tend to be very good at natural perception tasks. So it is possible, but it's just a bit harder for computers to surpass human level performance on natural perception tasks. And finally, all of these are problems where there are teams that have access to huge amounts of data. So, for example, the best systems for all four of these applications have probably looked at far more data of that application than any human could possibly look at. And so that's also made it maybe relatively easy for a computer to surpass human level performance. Now, the fact that there's so much data a computer could examine, so they can better find statistical patterns than even a human might. Other than these problems today, there are speech recognition systems that can surpass human level performance, and there are also some computer vision, some image recognition tasks where computers have surpassed human level performance. But because humans are very good at these natural perception tasks, I think it was harder for computers to get there. And then there are some, you know, medical tasks, for example, reading ECGs or diagnosing skin cancer, or certain narrow radiology tasks where computers are getting really good and maybe surpassing a single human level's performance. And I guess one of the exciting things about recent advances in deep learning is that even for these tasks, we can now surpass human level performance in some cases. But it has been a bit harder because humans tend to be very good at these sort of natural perception tasks. So surpassing human level performance is often not easy. But given enough data, there have been lots of deep learning systems that have surpassed human level performance on a single supervised learning problem. So if it makes sense for an application you're working on, I hope that maybe someday you manage to get your deep learning system to also surpass human level performance.
