A Bitter AI Lesson - Compute Reigns Supreme!

Dear fellow scholars, this is Two Minute Papers with Before we start, I'd like to tell you that this video is not about a paper and it is not going to be two minutes. Welcome to Two Minute Papers. This piece bears the name the Bitter Lesson and was written by Richard Sutton, a legendary Canadian researcher who has contributed a great deal to reinforcement learning research. And what a piece this is. It is a short article on how we should do research and ever since I read it I couldn't stop thinking about it and as a result result I couldn't not make a video on this topic. We really have to talk about this. It takes less than five minutes to read, so before we talk about it, you can pause this video and click the link to it in the video description. So in this article he makes two important observations. Number one, he argues that the best performing learning techniques are the ones that can leverage computation, or in other words, methods that improve significantly as we add more compute power. Long ago people tried to encode lots of human knowledge of strategies in their GO AI, but did not have enough compute power to make a truly great algorithm. And now we have AlphaGo, which contains minimal information about GO itself and it is better than the best human players in the world. And number two, he recommends that we try to put as few constraints on a problem as possible. He argues that we shouldn't try to rebuild the mind, but try to build a method that can capture arbitrary complexity and scale it up with hardware. Don't try to make it work like your brain, make something as general as possible and make sure it can leverage computation and it will come up with something that is way better than our brain. So in short, keep the problem general and don't encode your knowledge of the domain into your learning algorithms. The weight of this sentence is not to be underestimated because these seemingly simple observations sound really counterintuitive. This seemingly encourages us to do the exact opposite of what we are currently doing. Let me tell you why. I have fond memories of my early lectures I attended to in cryptography where we had a look at ciphertext. These are very much like encrypted messages that children like to write each other at school, which looks like nonsense for the unassuming teacher, but can be easily decoded by another child when provided with a key. This key describes which symbol corresponds to which letter. Let's assume that one symbol means one letter, but if we don't have any additional knowledge, this is still not an easy problem to crack. But in this course soon we coded up algorithms that were able to crack messages like this in in less than a second. How exactly? Well, by inserting additional knowledge into the system. For instance, we know the relative frequency of each letter in every language. For instance, in English, the letter E is the most common by far. And then comes T, A and the others. The fact that we are not seeing letters but symbols doesn't really matter because we just look up the most frequent symbol in the ciphertext and we immediately know that, okay, that symbol is going to be the letter E. And so on. See what we have done here? Just by inserting a tiny bit of knowledge, suddenly a very difficult problem turned into a trivial problem. So much so that anyone can implement this after their second cryptography lecture. And somehow Richard Sutton argues that we shouldn't do that. Doesn't that sound crazy? So what gives? Well, let me explain through an example from light transport research that demonstrates his point. Path tracing is one of the first and simplest algorithms in the field, which in many regards is vastly inferior to Metropolis Light Transport, which is a much smarter algorithm. However, with our current powerful graphics cards, we can compute so many more rays with path tracing that in many cases it wins over Metropolis. In this case, compute reigns supreme. The hardware scaling outmuscles the smarts. And we haven't even talked about how much easier it is for engineers to maintain and improve a simpler system. The area of natural language processing has many decades of research to teach machines how to understand, simplify, correct or even generate text. After so many papers and handcrafted techniques which insert our knowledge of linguistics into our techniques, who would have thought that OpenAI would be able to come up with a relatively simple neural network with so little prior knowledge that is able to write articles that sound remarkably lifelike? We will talk about this method in more detail in this series soon. And here comes the bitter Doing research the classical way of inserting knowledge into a solution is very satisfying. It feels right. It feels like doing research progressing, and it makes it easy to show in a new paper what exactly the key contributions are. However, it may not be the most effective way forward. Quoting the article, I recommend that you pay close attention to this. The bitter lesson is based on the historical observations that one AI researchers have often tried to build knowledge into their agents, 2 this always helps in the short term and is personally satisfying to the researcher, but three in the long run it plateaus and even inhibits further progress and four breakthrough progress eventually arrives by an opposing approach based on scaling, computation by search and learning. The eventual success is tinged with bitterness and often incompletely digested because it's success over a favored human centric approach in our cryptography problem from earlier. Of course, the letter frequency solution and other linguistic tricks are clearly much, much better than a solution that doesn't know anything about the domain. Of course, however, when later we have a hundred times faster hardware, this knowledge may actually inhibit finding a solution that is way, way better. This is why he also claims that we shouldn't try to build intelligence by modeling our brain in a computer simulation. It's not that the our brain approach doesn't work, it does but on the short run. On the long run we will be able to add more hardware to a learning algorithm and it will find more effective structures to solve problems and it will eventually outmuscle our handcrafted techniques. In short, this is the when facing a learning problem, keep your domain knowledge out of the solution and use more computer More compute gives us more learning and more general formulations give us more chance to find something relevant. So this is indeed a harsh lesson this piece sparked great debates on Twitter. I have seen great points for and against this sentiment. What do you think? Let me know in the comments. As everything in science, this piece should be subject to debate and criticism and therefore I'd love to read as many people's take on it as possible. And this piece has implications on my thinking as well. Please allow me to add three more personal notes that kept me up at night in the last few days. The bottom line is whenever we build a new algorithm, we should always bear in mind which parts would be truly useful if we had a hundred times the compute power that we have now. A corollary of this thinking is that arguably the hardware engineers who make these new and more powerful graphics cards may be contributing the very least as much to AI than most of AI research does. And note number three to me it feels like this almost implies that best is to join the big guys where all the best hardware is. I work in an amazing small to mid sized lab at the Technical University of Vienna and in the last few years I I have given relatively little consideration to the invitations from some of the more coveted and well funded labs. Was it a mistake? Should I change that? I really don't know for sure. If for some reason you haven't read the piece at the start of the video, make sure to do it after watching this. It's really worth it in the meantime. Interestingly, the non profit AI research lab OpenAI also established a for profit or what they like to call capped profit company to be able to compete with the other big guys like DeepMind and Facebook reality Labs. I think Richard has a solid point here. Thanks for watching and for your generous support, and I'll see you next time.