Most Research in Deep Learning is a Total Waste of Time - Jeremy Howard | AI Podcast Clips

Foreign.

So much of fast AI students and researchers and the things you teach are pragmatically minded, practically minded, freaking figuring out ways how to solve real problems and fast.

Right.

So from your experience, what's the difference between theory and practice of deep learning?

Well, most of the research in the deep learning world is a total waste of time.

Right, that's what I was getting at.

Yeah. It's a problem in science in general. Scientists need to be published, which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area. So that means that they all need to work on the same thing. And so it really. And the thing they work on, there's nothing to encourage them to work on things that are practically useful. So you get just a whole lot of research which is minor advances in stuff that's been very highly studied and has no significant practical impact. Whereas the things that really make a difference, like I mentioned transfer learning, like if we can do better at transfer learning, then it's this world changing thing where suddenly lots more people can do world class work with less resources and less data, but almost nobody works on that. Or another example, active learning, which is the study of how do we get more out of the human beings in the loop.

That's my favorite topic.

Yeah, so active learning is great, but it's almost nobody working on it because it's just not a trendy thing right now.

You know what somebody. Sorry, sorry to interrupt. He was saying that nobody is publishing on active learning, but there's people inside companies. Anybody who actually has to solve a problem, they're going to innovate on active learning.

Yeah, everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think, gosh, this is taking a long time and it's very expensive. And then they start thinking, well, why am I labeling everything? The machine's only making mistakes on those two classes. They're the hard ones. Maybe I'll just start labeling those two classes and then you start thinking, well, why did I do that manually? Why can't I just get the system to tell me which things are going to be hardest? It's an obvious thing to do, but yeah, it's just like transfer learning. It's understudied and the academic world just has no reason to care about practical results. The funny thing is I've only really ever written one paper. I hate writing papers and I didn't even write it. It was my colleague Sebastian Ruder who actually wrote it, I just did the research for it. But it was basically introducing transfer learning, successful transfer learning, to NLP for the first time. The algorithm is called ULM fit, and I actually wrote it for the course, for the FAST AI course. I wanted to teach people nlp and I thought I only want to teach people practical stuff, and I think the only practical stuff is transfer learning. And I couldn't find any examples of transfer learning in nlp, so I just did it. And I was shocked to find that as soon as I did it, which, you know, the basic prototype took a couple of days, smashed the state of the art on one of the most important data sets, a field that I knew nothing about. And I just thought, well, this is ridiculous. And so I spoke to Sebastian about it and he kindly offered to write it up, the results. And so it ended up being published in acl, which is the top computational linguistics conference. So, like, people do actually care once you do it, but I guess it's difficult for maybe junior researchers or I don't care whether I get citations or papers or whatever. There's nothing in my life that makes that important, which is why I've never actually bothered to write a paper myself. But for people who do, I guess they have to pick the kind of safe option, which is make a slight improvement on something that everybody's already working on. SA.