ResNets are back baby!!! ...and they're gone | Machine Learning Monthly March 2021

Writin' ML code on the weekend like usual, way up on the deep end like usual. Oh, I didn't see you there. Hello, welcome back. I guess you're here for another episode of Machine Learning Monthly. Now, let's not, let's not, let's not, let's not wait around. All right, let's get straight into it. It's been an exciting, exciting month in the world of machine learning for March. Now, if you're new here, Machine Learning Monthly is a monthly newsletter that I send out, so the link will be below if you want to sign up. And it's just some of the coolest things I've found, the latest and greatest of machine learning, but not always the latest. I send it out in a little newsletter once per month. That's why it's called Machine Learning Monthly. And this is the video version of that, so we're going to go through that. If you've been here before, you know the drill. So without any further ado, let's get into it. We always start off with a little bit of dancing, by the way. And so, um, let's begin. First of all, my work. I've launched a new course, which is TensorFlow for Deep Learning. All the links you need for that, I'm not going to spend too much time on this, but if you want to try it out, the first 14 hours of the course are available on YouTube in a beautiful little playlist. If we come here— wonderful. 14 hours worth of video. I would have done it in one video, except YouTube's video upload limit is 12 hours. It used to be higher, but it's 12 hours now. And of course, if you want to see materials being developed, uh, you can go to the GitHub page. And of course, I've now been streaming things on Twitch. So if you go twitch.tv/mrdburke, you'll be able to see me live coding, making materials for the course. And of course, I later on, I plan on making many more live streams of different things to do with machine learning. And one last thing, there is— I know I'm spamming you with stuff that I work on. I want to get to other people's stuff. One last thing is I now have an archive channel, Daniel Bourke Archive, which is just a history of all of my past live streams. So if you can't make it live, I'm just going to publish them to Twitch and you can— I mean, sorry, YouTube on the archive channel and you can check them out there. But enough about me. Let's get into from the interwebs. This is the latest and greatest of the machine learning world, but of course not always the latest. One of my favorite articles this month is "Why Machine Learning is Hard." This is a great example of why it's not always the latest, because this came from 2016, but the best materials are things that ring true over time, and there's nothing different from when this article was written to now. Of course the field has changed, it's advanced, There's new ways of doing things, but the main premise is what makes machine learning hard compared to normal software development. And that's not to take away from software development, traditional software or software 1.0. Software development, you would have say an input and an output and you would create the steps to that. However, machine learning, well, this is the way Zad Enam, I'm not sure if I'm saying this correctly, but Zad Enam writes, that machine learning adds a lot more dimensions to the problem that you're working on. The main one of those being data. Now, so it goes through, it makes these beautiful graphics on what traditional software development would look like. And then for machine learning, it adds in a whole bunch of things. So this is one of my favorites, is when you add in data to the system, you add multiple dimensions, you add another variable, and there's different dimensions to that particular variable. So if you know machine learning, the curse of dimensionality, the more variables you add to your system, the harder it is to control that system. And so machine learning, if you add in data, all of a sudden you've got questions you have to ask. Do you have enough correct data? Do you have enough data just in general? Do you have weak labels? Are your labels— if you're working on supervised learning, we're going to get to a really cool part of self-supervised learning in a second. If you're working on supervised learning, are your labels correct? So go check out this article. It's a great read, especially if you're new to the world of machine learning and want to figure out what the main difference is between traditional software development and machine learning software development. And I'd also— a bonus— I'd recommend looking up Software 2.0, which is a great article by Andrej Karpathy, which talks about the difference in machine learning is you're programming data. Not necessarily just steps in a program. You're programming data. So go check that out. Two of my favorite articles for the week— or for the month, sorry. On to the next one. Now this is a great article from the team at Hugging Face, which is one of my favorite machine learning companies. Companies? Saying things correctly, am I? Correctly. Now this is simple considerations for simple people building fancy neural networks. So This is— was originally published September 2020, but it still rings true to today, and it's by Victor Sahn. So thank you very much, Victor, for publishing this article. I read through it and I got a very heavy dose of confirmation bias. So it goes through practical tips that you should consider when you feel like you're building fancy neural networks. What if you started by putting machine learning aside, adhered to Google's number one rule of machine learning, and that is if you can build a system that doesn't have to use machine learning, you should probably build that system. But Victor argues that you should ask questions like, are the labels balanced? This is sort of doing exploratory data analysis. In other words, becoming one with the data. Are the labels balanced? Are there gold labels that you do not agree with? Gold labels meaning, I'm assuming, ground truth labels that you don't agree with. How were the data obtained? What are the possible sources of noise in this process? And then of course, continue as if you just started machine learning. So one of the main premises of this point is to continually experiment and go back to basics. Ask, what is the fundamental question I'm working towards? Don't be afraid to look under the hood of those 5-liner templates. Yeah, I've been guilty of this before. At the moment, we're blessed with all these beautiful machine learning frameworks that we can just import a pre-trained model, such as Hugging Face's Transformer and use it to write all the code. But what if you wanted to slightly alter that model? Well, you're gonna have to look under the hood. Don't be afraid. It takes some digging, but with time you get familiar with it. And then of course there's some great, great, great other readings. My favorites also are some of the resources linked at the bottom of this post by Victor. Especially a recipe for training neural networks by Andrej Karpathy and a checklist for debugging neural networks. So make sure you check out this post, read it in its entirety. And if you want an extension, check out the extra resources below. Onto the next one. Airbnb's multimodal deep learning neural network. Yeah, this is what I'm also very excited about. Aside from the learning paradigm that we're about to get to, it's this multimodal modeling. And what is that? What is a multimodal model? Now my dyslexia is kicking in, and when I try to read something and speak it at the same time, I get a little bit confused, but that's all right. So what is this? Well, Airbnb developed a model. I'll go down here. I want to find the beautiful picture of the architecture. Am I— have I gone too far? I think this— there was a really concise one. There we go. This is the one I was after. So what they have done, Airbnb, if you're not familiar with it, it's— you can list your place on there if you have an amazing space or not even amazing, you just want to put it up to the world. You can take a photo and you say, hey, if you want to sleep on this fancy looking coffee table, you can. You just pay me $25 a night. And so Airbnb, uh, for their business, it's in their best interest to classify what a photo of a room is. That way, if you or I were going to Airbnb.com and we wanted to search for a specific type of room, maybe we wanted a room with a fireplace and an ice bath because we want to do some hot and cold therapy because we were on a training weekend and we want to heat our bodies and then cool them down rapidly. So we're like, hey, let's find me a room that's like a sauna room or a recovery room or something like that. Now, how could Airbnb do that? Of course, they could just use an image classifier on just the image. But when you upload an image to Airbnb, you have the option of giving it a caption. You also have other features that are associated with it, such as where the image was taken, and you have other computer vision problems— sorry, computer vision features such as amenity detection, meaning maybe this image also has a couch here and another couch and a table and a TV. So potentially those features could contribute to what type of room that is. And so the really cool thing Airbnb have built here is a multimodal model which takes in multiple sources of data. So not just one channel. If it was just an image classifier, it would only be this. That would be one source of data. Multimodal is taking in multiple different sources because let's be real, A lot of data you experience or you come into contact with in the real world comes from multiple sources. So in Airbnb's case, WideText takes in text, image, and dense features, which are like structured data, and combines them all together to decide what type of room type there is. Now this is a really great blog post, so I'd highly recommend checking it out. In fact, check out all of Airbnb's engineering and data science blog. Why am I not following that? Oh, I'm not signed in. Because the thing I love about this— oh, there's amenity detection. I actually had a YouTube series on this about this time last year. I rebuilt a model to do this with just images. Now, the beautiful thing about Airbnb's system is that they publish work that they use in production. So, you know, it's not just like a research thing, like they're using this for their business. So that's why I love Airbnb's blog. They post things that they're actually using in production. Go check that out, multimodal modeling. And speaking of multimodal modeling, can you think of a data source other than videos which has more different types of data? So a video could be images, sound, time, different frames. It could be— there could be text within the video. So Facebook are launching, um, how would you say, a research goal to learn from videos to understand the world. So you can imagine how much rich information. I mean, this is the way I think of it. So this is a blog post from Facebook AI of talking about how they, how they're going about leveraging or finding information in videos. And a lot of it has to do with self-supervision, which we're going to get onto in a second. But the way I think of it is that at the moment, if you wanted to search something such as what is deep learning, you've kind of, you've been trained of how to use Google in a text sense. Now, the search that we're using right now is, as I just said, it's sort of, it's based on you looking for text. But we, we are yet to sort of fully leverage the power. And I mean, Search on text is getting good, and I think we're only still just scratching the surface of what we can do with text. But we haven't started to think about how we could search for something in a video. So for example, what if there was like a 3-hour-long video and at timestamp 1 hour and 30 minutes, there was a specific phrase of, say, Daniel Bourke talking about machine learning for business. And— but that video is not searchable. I mean, you might have seen some videos becoming searchable on Google recently. But I think like imagine just AI search systems in about 10 years once we, or who knows the timeline might be shorter, might be longer. Once we start to really leverage the power of searching long-form video and long-form audio. So podcasts and whatnot. It won't be just, I think our whole paradigm of searching for information will change such as Facebook used the example of grandma. Yeah, here we go. So imagine you could search all of your videos. Show me every time we sang happy birthday to Grandma. So there's a lot of multimodal information there. So first of all, the search query has to understand that you want to find happy birthday, and then it has to go find a video where the song happy birthday has been sung. And then visually, it has to understand that you're looking for Grandma. So it has to also know who your grandma is in that video. So, you can imagine how you could expand on that search query once video and audio has become searchable. So, go have a read of this. This is a great insight into what's to come. So, there we go, what's next on how we can start searching for information in videos or learning from videos to understand the world. I just read that little URL extension there. Thank you for helping me out. And then, onto what I said we're going to get on. And this is one of my most exciting, or one of the most things that's most exciting me in the world of machine learning right now. And that is the dark matter of intelligence. Self-supervised learning. To go along with this blog post, Facebook's learning from videos, there's actually a few amazing things from Facebook AI this month. Of course, they're always publishing amazing things, but this month especially I've found some Awesome things. Self-supervised learning, the dark matter of intelligence. So this is a blog post. It's about 20 minutes or so read, or it took me 20 minutes, probably longer if you're going to look into all of the links and whatnot. Self-supervised learning is predictive learning. So it goes through, it's written by Yann LeCun, which is, he's the founder of convolutional neural networks, but he's also the vice president, chief AI scientist all-around AI hero, and he believes that self-supervised learning is the new frontier of machine learning and deep learning and artificial intelligence. And most of all, it's been used at Facebook, so, or in production. So these are the things that I love, is seeing companies who are putting things out into the world and publishing their results or publishing their work so other people can, can learn from it. Now, what is self-supervised learning? I'm not going to read it from here. I'm just going to, I'll let you know how I understand it. So if you imagine when you're first born, you, uh, but before you even know the name of anything, right? So before you even know, let's use the example of a car. When you're 16, 17, 18 years old or about that, you start to learn to drive a car. It takes you about 20 hours and of course you get a little bit better from the 20 hours to 100 hours. But maybe not so much better from 100 hours to 1,000 hours. By the time you've done 100 hours, you could safely say you're a pretty competent driver. However, you only started at 16 years old. So, before that, in all that time, you had— you were experiencing driving, but not necessarily hands-on. Now, what I mean by that is you were watching your parents drive, or maybe you're walking and you cross the street, or you watch another car turn a corner. Like this morning I was sitting at a cafe and I was watching cars and I was— a little boy saw a car and he liked the car and he watched it drive past. So, he's interpreting the physics of driving without necessarily being hands-on. And that's to me what self-supervised learning is. So, oh, I'll show you the bonus. This is a great example, is SIA, a self-supervised The start of a more powerful, flexible, and accessible era for computer vision. So, SIA, you can have a look at this article. I'm not going to dig through it, but it's a self-supervised model which achieves results on ImageNet, which is capable, which is, yeah, outperformed the most advanced state-of-the-art self-supervised systems on ImageNet, but it's also very competitive with supervised learning systems. So what SIA did was it looked at 1 billion images of Instagram. So just random images, public images, and it did it without any labels. So there was no labels of these images, what they were. And so how I understand how it works is that it learned the fundamental patterns in different images. And how did it do that? I believe it used Suave, which is— we covered Suave in a previous Machine Learning Monthly. But if you'd like to read about SUAVE, make sure you read the CIA article and then you can check out SUAVE. But basically learned the, the structure that underpins images and it's not, that's very akin to how, how I imagine, of course this is, I don't know this for sure, but this is just how I imagine a child from a baby to say whatever age is having millions of visual experiences with the world. Without knowing what each of those things are. So, another example, I was— my, my, one of my best friends has a little girl and she's almost 1 year old and she doesn't know the name of things. She doesn't know that a coffee cup is a coffee cup. She doesn't know that a table is a table. She doesn't know that a tree is a tree, but she's, she's gaining, she's looking at these different things and gaining a visual representation of the world. And so, when she does reach an age of like 3 or 4 or something like that, and she can start to say, point at a tree and say, "That's a tree," and a dog and say, "That's a dog," she's already had a visual experience of these things for a few years now. And this is how I imagine the same thing with supervised, self-supervised learning, that is, is that the pre-training on 1 billion Instagram images is akin to those several years you spend visually experiencing the world without knowing the name of things, but just knowing, like, that a tree has branches or that a table has legs or that a dog has fur or stuff like that. So, again, this is a new frontier I'm still learning about, but it's very exciting and it really, it makes sense for the future of artificial intelligence because, I mean, self-supervised learning, although it achieves amazing results, labeling everything is is not scalable. So go check out these 3 blog posts from Facebook AI. Really, really cool stuff on the future of self-supervised learning and potentially the future of artificial intelligence. Onto the next one. Now this is paper of the month. This is a funny one. I was getting all excited to call this episode ResNets are back, baby. 'Cause if you're not sure, in computer vision, ResNets were a model that were introduced in 2015, and they performed amazing on image recognition. So identifying or classifying a different type of image. However, they got kind of replaced by EfficientNets in 2019. But then, 2 weeks ago, there was a paper called ResNets Revised. So if we have a look at this, this is— I can't believe how fast it fits. Field moves. Now, oh sorry, revisiting Resonance. And this is 13th of March, so literally about 2 weeks ago. It's April 3rd today, my little brother's birthday. So happy birthday, Sam. Uh, 21st birthday. I'm very excited to see you later today. We're going to have a great party. So if you want to say happy birthday to my little brother, leave a comment below and I'll be sure to show him all the comments. Um, but yeah, there was a paper 2 weeks ago, Resonance Revisited, and it basically said that They improved ResNet to state-of-the-art performance through a bunch of, uh, because it— the model, the architecture ResNets was originally released in 2015. A lot has changed since then in terms of training and regularization techniques and whatnot. And so the authors of the paper added those changes, the modern training changes, to the original ResNets plus a few little tweaks. And their main— this is actually a beautiful paper, by the way, so if you want to just read this paper I'm about to get to why it's kind of how fast the field moves in a second. But long story short, they found the new ResNets with upgraded training techniques and whatnot were to perform faster than EfficientNets and get the same results. So yeah, 2 to 3— can you see that? 2 to 3.3x faster on GPUs and 1.7 to x faster on TPUs. And so one of the things I loved about the paper— this is why it's really cool to check it out— is this little table here. They broke down how the model was improved through different methods. So we got a few training methods here, such as cosine learning rate decay, then regularization methods. My favorite one here was label smoothing. I got introduced to that through TensorFlow's categorical cross-entropy. I used label smoothing to gain some improvements on my, one of the models I was building on a recent Twitch livestream. And where were we? Over here. And architecture improvements. So they broke it down into how each of these different paradigms added improvements to their model. So really beautiful and well-written paper. However, I told you, I was so excited that ResNets are back, baby. I texted all my friends saying, or machine learning friends saying, ResNets are back, baby. Of course I wouldn't text my non-machine learning friends, they might think I'm a weirdo. But if we go EfficientNet, this is how fast the field moves. EfficientNet V2. There we go. Smaller models and faster training. Are you ready for this? So this paper was literally 2 weeks ago. March 13th or something like that. Yeah, 13th of March. And now we have EfficientNet V2. So remember this paper ResNets brought back the original ResNets to make it faster than EfficientNet V1. And 1st of April, 2021 could be an April Fool. I'm hoping that it's not. EfficientNet V2 is out. And so you remember the new ResNets that we just discussed. Well, they're here. And this is the new EfficientNet. V2. So that's, that's how fast the field moves. And what's confusing to me, so this is Kwok Leung. So Kwok Leung is an amazing researcher, I believe one of the researchers on the original EfficientNet. And of course, not to take away anything from Mingjing Tan, I just have seen Kwok Leung's name appear fairly often. Now, both of these papers were from the Google Research Brain Team. Google Brain, UC Berkeley. Now I'm not sure, are they talking to each other? Like, it seems like there's a lot of work went into this ResNet paper, but then 2 weeks later EfficientNet v2 comes out and beats it. Now that's not to take away and say that this work is completely rendered pointless. I mean, a lot of the things in this article, I really love how this one is written. I have— I'm yet to read this EfficientNet V2 in full, but I'm going to read this this weekend. But long story short, EfficientNet V2 trains 5 to 11 times faster than other models while using up to 6.8 fewer parameters. So amazing, amazing, amazing. In Machine Learning Monthly, I link ResNets are back, baby, but I'm going to leave EfficientNet V2 in the description so you can also read this paper as well. That's crazy to me. So, 2 weeks ago I was like, "Yes, ResNets are back!" And today, Fisher Nets are back. Alright, what's next? Superpowered speech. Oh, this is very, very exciting. So, 2 things here is Coqui by Josh Mayer and team, and SpeechBrain. So, Coqui, if you are working on a speech problem, you probably want to look into these two libraries. Coqui, I hope I'm saying this correctly, my Puerto Rican friends out there. I believe coqui is a sound that a frog makes from Puerto Rico. Let's actually look that up. So, YouTube, coqui, coqui frog, Hawaiian forest coqui. There we go. Can you hear that? Cookie. Cookie. Cookie. That's beautiful. I could listen to that going to sleep. Coqui is a library, state of the art for text-to-speech as well as speech-to-text. I remember when I was working at my last machine learning engineer role, we tried to work on a speech problem and it didn't work too well. This is about 2 and a half years ago, maybe. Didn't work too well because a lot of the stuff that we found was outdated libraries, none of them up to scratch. But these two libraries, SpeechBrain and Coqui— I hope I'm saying that right— SpeechBrain, a PyTorch-powered speech toolkit, set out to help you with a whole bunch of speech problems. So speech recognition, speaker recognition, speech enhancement, processing, multi-microphone processing, research and development, as well as Coqui and Coqui is by Josh Mayer, who is a great follow on Twitter, is very active in the speech space. It does amazing, amazing work in the artificial intelligence and machine learning for speech. So open speech technology, that's what it's all about. So he's co-founded Coqui. So if you're working on a speech problem, go check out Coqui and SpeechBrain. And if you want to follow someone who's very active in the speech world, Josh Mayer is who you want to check out. So big shout out to that. And what's next? Oh yes, finally. To generalize, speed up online test error and slow down offline learning train error. When I first read that, I was like, hmm, that's a whole lot of words that I don't really understand. But then I read the article called The Deep Bootstrap Framework. And what this was is a new lens on understanding generalization in deep learning. In other words, when you train an algorithm on a certain training dataset, generalization is how well does that perform on a dataset it hasn't seen before. And so this paper introduces a deep bootstrap framework. Good online learners are good offline generalizers. And now keep reading. It took me twice to sort of understand this. But long story short, the way I understood it, this graphic is great. So the ideal world test is where you have unlimited data. So there's just constant stream and this is, this is not too far away from how you could imagine like a big service like Facebook. They basically have unlimited data. So when it goes to their learning from videos, they have videos being posted and same with YouTube and same with other areas. There's too much data to ever go through, whereas a real world is you sort of you have a static dataset. It's just, you download, say, 1,000 videos and it's just going to stay those 1,000 videos. Online, you're getting continually more videos every day, every second, every hour. And so, what they found was that when you're looking at generalization through a lens of optimization, you want your model to optimize on ideal world data, so unlimited data, as fast as possible because there's an unlimited stream of it. Whereas if you have a limited stream of data, you want your training error to optimize as slow as possible for generalizability. Does that make sense? The way I intuit it is when you have an unlimited source of data, you want your model to, to learn as quickly as possible because there's always more data coming in. Whereas if you have— that way it will generalize better to the, the new data samples that are coming through. Whereas if you have a static dataset, a real-world test set, and they describe both of these here. You want your training error to reduce as slow as possible to give your model the best chance at understanding as many patterns as possible in that training data so it can generalize to unseen samples. And so they really break it down here. I'll let you read through the whole thing to get a more understanding. But here, the principle is whenever one makes a change that affects generalization in the real world, such as the architecture, learning rate, etc., one should, one should consider its effect on one, the ideal world optimization of test error. So when you're optimizing on the test error of the ideal world, unlimited data, you want to optimize as fast as possible for the best generalization. However, when you optimize on real-world data, so a limited dataset, you want to slow down the optimize optimization of training error. So slower is better for ideal generalization. So again, this took me a few, few passes to fully understand, but where it is— here we go. This means that the generalization of the model can be understood in terms of its optimization performance under two frameworks. So online optimization, how fast the ideal world test error decreases. And offline optimization, how fast the real-world training error converges. So what can you do with these? Well, it's, it's important if you want your model to generalize, if you want to make any changes to it. So you can use these two frameworks to base your changes to your architecture or your model. So based on this observation, good models and training procedures are those that one, optimize quickly in the ideal world, and two, do not optimize too quickly in the real world. Now, I know that's a lot of, a lot of jargon there, but trust me, read through this. And if you've had some experience training neural networks, you'll, you'll start to understand. Okay, I see this. So my design choices have to speed up training in the ideal world on the test error, and they have to slow down training in the real world if I have a limited dataset. So go and check that out. But I think that is it. A massive month in the ML world in March. We saw how quickly the machine learning field can move. I mean, within 2 weeks, the state of the art in terms of training models on image classification changed. And of course, a whole bunch of other beautiful things such as why machine learning is hard, multimodal models, learning from videos, the future of artificial intelligence, 'Self-supervised learning of the potential future.' 'The dark matter of intelligence.' That sounds, that's a really cool title for a blog post. SIA, the best performing self-supervised vision model out there. We've seen self-supervision in language such as BERT and GPT. Now it's coming to vision. If you want speech, go check out SpeechBrain and Coqui, some amazing new speech libraries. And finally, if you want to generalize your networks, go and check out the Deep Bootstrap framework. Otherwise, if you want the next issue of Machine Learning Monthly, don't forget you can subscribe. It'll be delivered to you at the start of every month. The video like this one, the one that you're watching, comes a few days after. And if you have any submissions, send me an email at daniel@mrdbourke.com. Whatever you've worked on in the machine learning field, or you found something cool and you want to add it to the article, like for example, Ashik— Ashik is the one who sent me ResNets, the new ResNets paper. And you know what, Ashik also sent me the new EfficientNets paper. So thank you so much, Ashik, for the contribution. But yeah, if you want to send me anything, leave a comment or send me an email, daniel@mrdbourke with the title of the subject Machine Learning Monthly Submission. But as always, keep machine learning, keep creating, keep moving, keep dancing. I've got to go party with my little brother because it is his it is his 21st birthday. Once again, happy birthday, Sam, and I will see you all next time. Far out. I love this field.