Why ML teams use human-level performance as a benchmark. Deep learning gets competitive with humans, and human-labeled data speeds development. Progress curves show rapid gains until surpassing humans, then slow down as you approach Bayes optimal error, the theoretical floor you can't beat.
In the last few years, a lot more machine learning teams have been talking about comparing the machine learning systems to human level performance. Why is this? I think there are two main reasons. First is that because of advances in deep learning, machine learning algorithms are suddenly working much better. And so it's become much more feasible in a lot of application areas for machine learning algorithms to actually become competitive with human level performance. Second, it turns out that the workflow of designing and building machine learning system, the workflow is much more efficient when you're trying to do something that humans can also do. So in those settings, it becomes natural to talk about comparing or trying to mimic human level performance. Let's see a couple examples of what this means. I've seen on a lot of machine learning tasks that as you work on a problem over time, so the X axis time, this could be many months or even many years over which some team or some research community is working on a problem. Progress tends to be relatively rapid as you approach human level performance. But then after a while, the algorithm surpasses human level performance and then progress in accuracy actually slows down and maybe keeps getting better. But after surpassing human level performance, it can still get better. The performance, the slope of how rapidly accuracy is going up, often that slows down and the hope that it achieves some theoretical optimum level of performance. And over time, as you keep training the algorithm, maybe bigger and bigger models, on more and more data, the performance approaches, but never surpasses some theoretical limit, which is called the Bayes optimal error. So Bayes optimal error. Think of this as the best possible error, in that there's just no way for any function mapping from X to Y to surpass a certain level of accuracy. So, for example, for speech recognition, if X is audio clips, some audio is just so noisy it's impossible to tell what is in the correct transcription. So the perfect error may not be 100%. Or for cat recognition, maybe some images are so blurry that it's just impossible for anyone or anything to tell whether or not there's a cat in that picture. So the perfect level of accuracy may not be 100% and Bayes optimal error, or Bayesian optimal error, or sometimes Bayes error for short, um, is the very best theoretical function for mapping from X to Y that can never be surpassed. So it should be no surprise that this purple line, no matter how many years you work on the problem, you can never surpass Bayes error or Bayes optimal error. And it turns out that progress is often quite fast until you surpass human level performance, and it sometimes slows down after you surpass human level performance. And I think there are two reasons for that, for why progress often slows down when you surpass human level performance. One reason is that human level performance is, for many tasks not that far from Bayes, optimal error. You know, people are very good at looking at images and telling if there's a cat or listening to audio and transcribing it. So by the time you surpass human level performance, maybe there's not that much headroom to still improve. But the second reason is that so long as your performance is worse than human level performance, then there are actually certain tools you could use to improve performance that are harder to use once you've surpassed human level performance. So here's what I mean. For tasks that humans are quite good at, and this includes looking at pictures and recognizing things, listening to audio or reading language, really natural data tasks humans tend to be very good at at. For tasks that humans are good at, so long as your machine learning algorithm is still worse than a human, you can get labeled data from humans. That is, you can ask people, ask hire humans to label examples for you so that you can have more data to feed your learning algorithm. Something we'll talk about next week is manual error analysis. But so long as humans are still performing better than your learning algorithm, you can ask people to look at examples that your algorithm is getting wrong and try to gain insight in terms of why a person got it right, but the algorithm got wrong. And we'll see next week that this helps improve your algorithm's performance. And you can also get a better analysis of bias and variance, which talk about in a little bit, but so long as your algorithm is still doing worse than humans, you have these important tactics for improving your algorithm. Whereas once your algorithm is doing better than humans, then these three tactics are harder to apply. So this is maybe another reason why comparing to human level performance is is helpful, especially on tasks that humans do well and why machine learning algorithms tend to be really good at trying to replicate tasks that people can do and trying to catch up to and maybe slightly surpass human level performance. In particular, even though you know what is bias and what is variance, it turns out that knowing how well humans can do well on a task can help you understand better how much you should try to reduce bias and how much you should try to reduce variance. I want to show you an example of this in the next video.
