This interview excerpt (Lex Fridman with Jeremy Howard) outlines the discovery and importance of "super convergence"—a phenomenon Leslie Smith found where using much higher learning rates (up to ~10×) can train certain neural networks roughly 10× faster and improve generalization. Howard discusses practical recipes (start very low, gradually increase learning rate), barriers to publishing unexplained experimental results in deep learning, and recent trends toward optimizers and techniques (discriminative learning rates, fewer hyperparameter dials) that automatically set per-parameter update behavior.
Questions this video answers
Leslie (Les) Smith
About 10× faster by using a roughly 10× higher learning rate
October 8, 2019
There's some magic unlearning rate that you played around with. It's kind of interesting.
Yeah. So this is all work that came from a guy called Leslie Smith. Leslie's a researcher who, like us, cares a lot about just the practicalities of training neural networks quickly and accurately, which you would think is what everybody should care about, but almost nobody does. And he discovered something very interesting, which he calls super convergence, which is there are certain networks that, with certain settings of high parameters, could suddenly be trained 10 times faster by using a 10 times higher learning rate. Now, no one published that paper because it's not an area of kind of active research in the academic world. No academics recognize this is important. And also, deep learning in academia is not considered an experimental science. So unlike in physics, where you could say, like, I just saw a subatomic particle do something which the theory doesn't explain, you could publish that without an explanation, and then in the next 60 years, people can try to work out how to explain it. We don't allow this in the deep learning world. So it's literally impossible for Lesley to publish a paper that says, I've just seen something amazing happen. This thing trained 10 times faster than it should have. I don't know why. And so the reviewers were like, well, you can't publish that because you don't know why.
So anyway, that's important to pause on because there's so many discoveries that would need to start like that.
Every other scientific field I know of works of that way. I don't know why ours is uniquely disinterested in publishing unexplained experimental results, but there it is. So it wasn't published. Having said that, I read a lot more unpublished papers than published papers because that's where you find the interesting insights. So I absolutely read this paper, and I was just like, this is astonishingly mind blowing and weird and awesome, and why isn't everybody only talking about this? Because if you can train these things 10 times faster, they also generalize better because you're doing less epochs, which means you look at the data less, you get better accuracy. So I've been kind of studying that ever since, and eventually Leslie kind of figured out a lot of how to get this done. And we added minor tweaks. And a big part of the trick is starting at a very low learning rate, very gradually increasing it. So as you're training your model, you take very small steps at the start, and you gradually make them bigger and bigger until eventually you're taking much bigger steps. Than anybody thought was possible. There's a few other little tricks to make it work, but basically we can reliably get super convergence. And so for the dawn bench thing, we were using just much higher learning rates than people expected to work.
What do you think the future of, I mean, it makes so much sense for that to be a critical hyper parameter learning rate that you vary. What do you think the future of learning rate magic looks like?
Well, there's been a lot of great work in the last 12 months in this area and people are increasingly realizing that we just have no idea really how optimizers work. And the combination of weight decay, which is how we regularize optimizers and the learning rate, and then other things like the epsilon we use in the atom optimizer, they all work together in weird ways and different parts of the model. This is another thing we've done a lot of work on, is research into how different parts of the model should be trained at different rates in different ways. So we do something we call discriminative learning rates, which is really important, particularly for transfer learning. So really, I think in the last 12 months a lot of people have realized that all this stuff is important. There's been a lot of great work coming out and we're starting to see algorithms appear which have very, very few dials, if any, that you have to touch. So like, I think what's going to happen is the idea of a learning rate will. It almost already has disappeared in the latest research. And instead it's just like, you know, we know enough about how to interpret the gradients and the change of gradients we see to know how to set every parameter there.
