Kyle Balmer argues humans overestimate their abilities while underestimating AI. He points out that 80% of people think they're above-average drivers, and that most tasks don't require 100% accuracy. We should measure AI and human error rates rigorously instead of trusting our gut.
Chapters
Why do you assume you're smarter than an AI? We humans tend to overestimate our abilities. For example, 80% of people think they're above average at driving, which mathematically doesn't work. Ethan is making a really good point up here. We spend a lot of time worrying about hallucinations in artificial intelligence models, but we're not really asking, well, is that any different from humans? Humans make a lot of errors in their work. They're certainly not infallible. However, we are used to those kind of errors. We expect humans to make mistakes, but we get mad when an artificial intelligence does. So. These questions up above make a lot of sense. First, ask, how vital is it that the task is done with 100% accuracy? For the majority of tasks, we do not need 100% accuracy. In fact, it's a fantasy. We then ask, how accurate is AI at the task? But then we have to compare that to how accurate a human would be at the task. And then, importantly, how do we actually know? How are we measuring accuracy here? Is it just a subjective feeling of, oh, the AI didn't do very well. I could do better, or do we know how? Have we measured the inaccuracy? Does this mean that AI is actually potentially more accurate than humans? No, not necessarily. But we need accurate measures rather than just our gut feeling about who's doing a better job.