One Neural Network to Rule Them All (and more) | Machine Learning Monthly June 2020

Let's have a bit of fun, shall we? So every month I write an article. Here it is. I actually write a few articles every month, usually. But every month I write an article called Machine Learning Monthly, June 2020. Well, this one, this one's for June. It's July now, but this is for last month. And so basically the start of every new month, I'll publish this article of the coolest things in machine learning from the previous month. And so this is on zerotomastery.io where I'm, I'm an instructor on there. I teach machine learning on there. Um, it's on the blog there, but if you go to this, you can see ML Monthly. It actually happens every month. So ever since January, so June was January, February, March, April, June is issue 6. And so I figured, you know, what would be fun is if we made a video version of going through this. Look, see, there's yours truly. Need to reflect the mullet though. That one, uh, this photo is a bit more professional. But I thought it'd be fun. Why don't we go through— there's about 10 things here. Why don't we spend this video going through all of the little tidbits that I found in June 2020. You know what, I might actually start making video versions of these. If this is something you'd like to see more of in the future, leave a comment below. Again, if you'd like to submit anything, maybe something you've worked on for the next issue of Machine Learning Monthly, be sure to check it out. I'll leave a link in the description or send me an email directly of something you'd like to share. Anyway, let's start off. So I started off with this cool little tidbit. Welcome to Machine Learning Monthly, one of the best ways to get started in the industry. The first part of this series is about solving problems in machine learning. The second part is about how to use machine learning to solve problems in machine learning. That's a bit weird. Why did I write that? In this post, I'll introduce you to a number of new machine learning algorithms that I've been working on for the last year or so. Well, that's a, that's another lie. I haven't been working on my own machine learning algorithms. I'll also cover some of the more common problems and techniques that you'll find as you learn about machine learning. Well, I usually do that. I will also be discussing the various problems that you'll encounter in machine learning. Well, that's kind of the same thing that I said here. And the ways that you can learn them in this series. You know what? You know why that doesn't really make grammatical sense? Because I'm, frankly, I'm a better writer than that, or at least I tell myself that, is that this was written, this entire paragraph was written 'By a machine.' See there? Whoa. 'The above paragraph was completely artificially generated by GPT-2,' which is actually number 2 in this list, but we'll get to that. There we go. AI TextGen. Is my head covering this? Not really. You can still see that. AI TextGen. So if we go to this, if we click this one, boom, GPT-2. I actually used this OpenAI GPT-2 model, which is a natural language processing model, to generate that entire paragraph. All I fed it, all I fed the model, was this text here: "Welcome to Machine Learning Monthly," and it spurted out all of this stuff. But again, we'll get to that. Anyway, let's go to the first two things were my posts. So, how, how can a beginner data scientist like me gain experience? If we check that one out briefly, little blog post here. Um, so if Manuela sent me an email saying, hey, how can a beginner data scientist like me gain experience? And I went through a bunch of different cool little points here. You might, might want to reference that if you're going for some kind of role. If we go back, how I passed the TensorFlow Developer Certification and how you can too. So I made I've kind of got like a little style going on with machine learning and content creation is I write an article and make a video to go along with it because I find myself, I like to read some things and I like to watch some things. And I thought, well, if I'm making things for other people, maybe there's some people out there like me who like to read, who like to watch, who like to listen. A lot of the time I'm watching a YouTube video, I've got it running in the background while I'm cooking food or something. I usually like just watching people talk or hearing people talk. Anyway, this is an article, a guide, if you want to go through the TensorFlow Developer Certification. You can check out how I did and how I passed the exam. There's a video to go along with it as well. Anyway, finally, into Machine Learning Monthly. Oh, this is a little bit of a spoiler for what's to come. This is the machine learning roadmap for 2020. I'm actually really proud of this. If you check it out, we're not gonna— I'm not gonna read straight through the courses, I mean straight through the post. Oh, there's a little issue there. Maybe my browser's not showing me something. But check this out. This is, as I said, this is a spoiler of what's to come. I haven't announced this, but because you're watching this video, I made this really cool interactive mind map that you can go through. Starts here, machine learning. You probably want to go here, machine learning problems, process, machine learning tools, machine learning mathematics, machine learning resources. And then they all branch out. I mean, look at this, process goes out. Stay tuned, I'll leave a link below that you can access this and interact with it. Um, oh, this is in the way here, isn't it? I'll move this down. But there will be in the coming, in the next couple of weeks, or maybe next week actually, by the time this video comes out, the video walkthrough, I'm going to walk through, actually made like a little documentary walking through all of this. So stay tuned for that. But that's machine learning. Again, this is Machine Learning Monthly. This is what it's all about. Those first 3 things were all from me. So I kind of hog the spotlight there. But if you, if you want to get this, this article, there's a lot more we're going to go through. If you want to get this article delivered to your inbox, you can sign up. If not, you can just go to the zeromastery.io blog or potentially come back to this YouTube channel. If you like this sort of video of me just walking through the coolest things I found over the last month in machine learning, I might keep making them. But that's— I'm really proud of that. I spent like a week putting this together. Kind of connects all the different concepts of machine learning. So if we go back to the article, now number 2 was AI TextGen. Leverage OpenAI's GPT-2 architecture to generate text. So if we go to— let's check it out. Again, we're not going to read through this because you can do that in your own time. I just want to play around with these things, you know. So AI TextGen, by the way, what's this? What's his name? Max Wolff, I believe. Max Wolf, where is he? How do you get to a GitHub profile these days? Max Wolf. GitHub just updated their, their layout. So I'm still getting used to it. It does look pretty nice actually. Max Wolf, awesome, awesome creator. I love, I love his work. And this is, this is phenomenal. AI TextGen. So it's a robust Python tool for text-based AI training generation using OpenAI's GPT-3. GPT-2 architecture. So if we go to this, this is OpenAI's blog, another blog that's really cool to read. So better language models and their implications. So you can read through that of GPT-2. Long story short, it's this language model that came out how long ago, maybe 18 months ago now. And it, you feed it a whole bunch of text. So Basically, to train the model, it's read through the entirety of Wikipedia and a whole bunch of other texts. Maybe I'm getting Wikipedia wrong, but let's just say it's read a whole bunch of online text, learnt what words should come after the other words, statistically, of course. And then you can use it to generate new text based off what it's learned. And this is what Max Wolf has created. Ai-textgen to make it really simple to do. So you can go just pip3 install ai-textgen, run a few lines of code here. That's basically what I did in my Colab notebook. I literally just went through this, fed it a prompt, um, this one here. Where is it? Welcome to Machine Learning Monthly. I just fed it those 1, 2, 3, 4, 5 words and it produced all of these. So have a play around with that. You'll probably scare yourself a little because some of the stuff that it comes up with are pretty real. Like, I read that. Yeah, this paragraph, probably not. You could probably decipher that was a bit fake, but some of the other stuff, I mean, especially if you train it. One of the things I've thought of doing is training my own version of it using transfer learning on all of my past blog posts. And then seeing what kind of things it can come up. So maybe that could be a future video, who knows? I just, 'cause I've got a lot of blog posts out there and maybe I could get it to write one for me. Anyway, that was really cool. That was number 2. Number 3. Now we've seen AI generate text with AI TextGen, but you can also use AI to play it like Bach. And if you don't know who Bach is, I think it's Johann Sebastian Bach. Yeah, J.S. Bach. Google's doodle, so that little graphic that you see on google.com, the one that usually goes here, a while ago was to celebrate Bach's 334th birthday, was allowed you to play music. Let's see if we can— —in before that gets copyrighted. But apparently that's generated by an artificial intelligence model. So you can read through this of how it was done. Pretty in-depth blog post of teaching you how to do it. But you can also play along. Where was that? Today's doodle. We go here. There was somewhere where you could play along. Why won't you let me play along? Anyway, check that out. But this is really cool, right? So these— I like the idea of generative models because I never really learned to play a musical instrument myself, but I feel like now that I know how to code and I know some machine learning, I could actually code up my own band. That'd be so much fun. But this is just one example of how it's done. So you can, you can, what you can do is you can, if I found the demo, I'm not sure where the actual demo is, you can put in your own notes and then Google's Coconet will check out the notes that you did and will make it sound, will add some more instruments and make it sound like a symphony made by Bach. But of course it's just us sitting at our keyboards and clicking different notes, and then it makes it sound really cool. That was number 3. And then number 4 is data science, expectations versus reality. So let's check this out. This is from Dan Friedman, really cool blog actually. So check out Dan Friedman's or dfrieds.com. Again, all of these links, the article itself, this article will be below. And then all of the other links that I talk about will be in the description so you can check it out. Some great points. So Dan's story, the synopsis of this, he learned data scientist right at the peak, or learned, went to a data science bootcamp right at the peak of ML and AI on the news and all the headlines and that sort of stuff. And he got a job in the field and then found out it wasn't quite what he expected it to be. I mean, yeah, you see all these posts that say like, "Data scientist is an amazing role," like, and of course it is. Like, for some people it's truly amazing. But again, it's, this is the thing, when you set your expectations really, really high, well then if you get into the actual experience and you find out, well, hey, these things didn't suit, fit my expectations. And so what he's done is he's He's taken his experience. And this is the kind of post that I actually like. It's like when someone's actually set out to do something, got into it, and then realized, you know what, it wasn't what I thought it was. Here's what I've learned. I really like those type of posts. So the points are people don't know what data science does. So people think different things of people's, like you just ask someone to define machine learning. How many different definitions are you gonna get? Data science leadership is sorely lacking. Can't always be built to specs. See, a lot of data science and machine learning projects stay in proof of concept stage because it's quite easy to build a model these days, but actually getting that into people's hands, that's, that's another challenge in itself. You're likely to be the only data person, especially in— so as in, if you're like working at a small startup Maybe there's a lot of people who are engineers, but you might be the only data person. And so, because think about this, if you're working on a project, it's likely that the project needs engineers more so in the beginning than it does any data science. Because if you don't have anything to build, you don't have, or any resources to build something, if that something doesn't exist, then you don't have any data, then why do you need a data scientist? So hiring a data scientist first before you even have any data is kind of the cart before the horse. It's that, that little, uh, metaphor there. Number 5: Your impact is tough to measure. Data doesn't always translate to value. Data and infrastructure have serious quality problems. I like this point. Let me zoom in. Actually, that's probably a better idea. In data science books, online classes, online tutorials, and Kaggle competitions, the problems are faced are radically different from what exists in industry. In those resources, you'll have clean data that's easily available, well documented, and structured in an outline that allows you to apply data science techniques to answer the problem. Again, you'll find once you start getting on working on actual projects, it's not quite as easy as downloading a CSV file. Sometimes it is. Not quite as easy as downloading a CSV file, um, and then just going model.fit. Here's a good little tweet. How much do you— how do you spend your time? Cleaning data and moving data. Yep. And then another one, analyzing/presenting data. So look at this, picking features and models is 6.2% of your time. This is from 2,111 votes. Vicky, actually she's also amazing, so you should check her out. But here we go. Data work can be profoundly unethical, moral courage required. Yeah, you're right. This is This is true, especially in today's world, like a whole bunch of privacy stuff to go along with data. So what if you're asked to do something with data that's, that's against your ethics? I mean, one time I was asked to build a machine learning model that would be, that would help sell more pizzas. That wasn't the exact thing that it was doing, but it basically would end up doing that. And I could not, it was against my ethics because I'm a proponent of health and I can't use code to to sell food that I wouldn't personally recommend to someone. So I had to say no, I can't work on that project. We go here. That was number 4. Number 5 is what the hell is MLOps anyway? So you might come across this term, and as we said, a big problem of machine learning is, is getting a model into production because at the end of the day, machine learning is an infrastructure problem. Like what— setting up the tools around your model. So if you've ever heard of DevelopmentOps, MLOps is like the machine learning version of that. And if you're not sure what DevelopmentOps is, check it out. Um, I think if we go here, DevOps: The Rise of the DevOps Mindset, another great blog post here, which was actually part of the, the Web Developer Monthly, which is on Zero to Mastery, my, my friend Andrej writes that one. So I'll leave this in the link below. Anyway, where were we? Here we were. Chip Huen, another great blog. Check this one out. What I learned from looking at 200 machine learning tools. To better understand the landscape of available tools for machine learning production, because I don't know about you, but maybe that's what you want to do. You want to get a machine learning model out into production. You want to get a product into people's hands. And now of course, machine learning may not be the right tool for the problem you're trying to solve. But if it is, you probably want to check out these different resources to figure out how you can get that model into production. I actually really liked this first link here too, Full Stack or Production Deep Learning. Because if you see here, zoom right in on this one. If you wanted to There's your modeling code. So these are all the steps. This is like MLOps, all these steps around your model configuration, data collection, serving infrastructure, feature extraction, data verification, monitoring. You may not have all of these, um, but you'll probably have at least a few of them in whatever product you're trying to build. But this post by Chip is phenomenal. Like, it goes through all of the different MLOps tools So the landscape is underdeveloped. There are many AI startups, most of them are application startups, so using AI in some sort of application, but a lot of them aren't building tools for that sort of thing, tools for AI. Um, I could find only 200 AI tools, which sounds like a lot, but if you search for Python application tools, you're going to get a lot. Problems facing MLOps. So some of the problems facing ML applications: deployment, monitoring, data labeling, CI/CD test deployment, model compression, inference optimization, edge device privacy. We spoke about that before, privacy, big thing. Personally, I think privacy is a little bit overrated, but that's, that's my opinion. And I plotted the number of tools by the main problems they address. This is a great blog post. So check this out if you want to look at MLOps, or if you want to understand MLOps a little bit better, and what Chip learned from going through 200 machine learning tools. These are, again, these are the type of blog posts I really like coming across, where someone has gone, basically just surveyed the landscape and distilled them. And even better is Chip, I think, What we got about— anyway, I think she's a computer scientist as well, has taught machine learning at Stanford, so part of— is part of the industry. One neural network to rule them all. This was actually outstanding research called the Once for All network. Now, a real big thing when training machine learning models is— let me, let me zoom in again. This is specialize it for efficient deployment. I think this is really where the next, uh, the next big phase in machine learning. We've kind of reached the point of— this is again my opinion, my observation— reached the point of, all right, yeah, it's fairly easy to get a pretty decent model off the ground if you've got the data. The next thing is again getting it into people's hands. This is where machine learning and software engineering is really merging. If I had my time again, if I was learning machine learning again, I'd probably spend 80% of my time learning, uh, software engineering and the other 20% learning machine learning. So that's probably a video in itself or an article of some sort, but that's how I'd start again. So if you want to take that, me going forward at the moment, I'm practicing, I'm learning software engineering. Now what the Once for All network is, OFA, is train once and specialize for many deployment scenarios. So If you're trying a machine learning model on the cloud, say you're using a cloud GPU like Google Cloud, that GPU is probably— I mean, sorry, yeah, Google Cloud or Google Colab— that GPU is probably pretty powerful, but that model might not be able to run on a mobile device, say an iPhone, and probably definitely can't run on a tiny AI, so like a small, small camera that, like a Raspberry Pi or something like that. And so what Once for All does is it gives you a training framework for training a model that can be not only used on the cloud, it can be used on a mobile, and it can be used on a tiny device, an Internet of Things device. And it also has far less carbon emissions, because these are things we actually have to really start considering. If you know what's going on with the amount of carbon that's been released into the atmosphere. Training a machine learning model, although you can sit here, I can, you and I can sit here and type in things, click run, we don't see all the carbon emissions and all the electricity that our models are actually using. So things like an MNASNet, so I think it's Model or Neural Architecture SearchNet or massive, I'm not sure what M is, but Neural Architecture SearchNet uses a lot or emits a lot of CO2. Whereas Once for All, because it's so efficient, uses— look at that, what's that, 3 orders of magnitude less, or over 3.5 orders of magnitude less, and it still gets amazing results. So check out that Once for All network, um, very handy if you want to train a machine learning model once and deploy it to several different device types, which is— I mean, these days if you're building a machine learning model, you probably want it accessible as much as possible. We go back. Number 7, all of the matrix calculation or all of the matrix calculus you need for deep learning. So one of the biggest questions I get for learning machine learning is, Daniel Bourke, how much math do I need to know? The truth is, it's how can you measure that? How much math do you need to know? You can't measure that. And so matrix calculus is actually a really big part of deep learning. It's actually all of what backpropagation is, how a neural network improves its, improves its, uh, guesses, or improves its, the patterns that it's learned. And so Jeremy Howard and Terence Parr— Jeremy Howard's the founder of fast.ai— wrote a book, and it's available for free. This is all of the matrix calculations, or calculus— why do I keep saying calculus? Calculations. This is all of the matrix calculus you need to know for deep learning. And you probably look at this— and don't worry, I look at this and I go, look at all these Greek symbols— but this is— treat this as a reference to come back to. The way I say things is when you need to learn things, when you need to learn them. So if you find that you, you start to realize, okay, I'm building my machine learning models, um, and I'm kind of at a breaking point, I want to really improve them, I'm trying to read the latest state of the art research papers, but I'm really, I'm really getting stuck with the math. This is where you come in and you read things like this. You could start, spend a week reading it end to end, even if you don't understand all the symbols. Just read it end to end. Don't even bother like trying to read the symbols and then come back, leave it settle for a week, come back, start reading it again. That's how I learn things. I find it really helpful to basically just get a textbook, read it end to end, and then refer back to it when I need. So we come back, check that out. Number 8 is, can AI play Minecraft? So this is actually a really cool challenge using reinforcement learning. Personally, I haven't— I've only really worked on one reinforcement learning problem, but there's a big competition going on here. Can you train an artificial intelligence agent to play Minecraft? Now, I have really no No interest in playing video games, but I do have an interest in training AIs to play video games for me. So this would be a project that I might check out. And again, it's really cool to— I mean, if you did well in this project, you could probably apply to OpenAI or DeepMind and be pretty, pretty suitable for a role there. Like, I mean, you gotta, you gotta do well on the project and of course document all your work. It of course is going to take a lot of work. It's open for 87 days. I think they've got left, but that'd be some prime experience to, to get ready for. I wonder what prizes they have. To be determined. So here's the timeline. Just started July 1st. Check that out. A couple of days ago, you got to the end of the year. So that's a good 6-month project. Might be able to even join a team as well. But check that out, AI Crowd. We go here. YOLOv4, number 9, state-of-the-art real-time object detection model. You only look once is back, is back. YOLO's back for a fourth time. Let's check it out. We go YOLOv4 and the Medium article to go along with it announcing it was also incredible. Oh, they both linked to the Medium article. If we go here, code, darknet, there we go. YOLOv4. Where's the demo? Demo. I want a video. Hmm. Anyway, there's a code on GitHub. I think the video might actually be in here. I love this paragraph. This is how you announce some state-of-the-art research that you've done. Darknet YOLOv4 is faster and more accurate than real-time neural networks Google TensorFlow EfficientNet and Facebook PyTorch Detectron RetinaNet/MASK-RCNN on Microsoft COCO dataset. So it's like basically going, hey Google, hey Facebook, I've built this thing and it beats you guys. I love that. Stick it to the man, Alexey, what a legend. If we come here, there's a little demo of what YOLO does. Real-time object detection. Here we go. I love it. How cool is that? Might turn that down again, copyright rules. But that's pretty quick. Try out YOLOv4. There was a little bit of a controversy. Someone also released YOLOv5, but I'm pretty sure from my research YOLOv4 is Someone who wasn't part of— so the story with YOLO is it's from, uh, darknet, um, from PJ Reddy, I believe. But PJ Reddy has left the AI field. Um, long story short, some— Alexei released YOLOv4, someone else released YOLOv5 at the same time. I think it turns out that YOLOv4 is actually better than YOLOv5. Correct me if I'm wrong. Number 10 is end-to-end object detection with Transformers. On the topic of object detection, Facebook just released a model. So Transformers are really taking over. They might be, they might be the universal architecture, maybe. By universal architecture, I mean one architecture that works well on a lot of different problems. Convolutional neural networks actually work pretty well amongst vision problems and text problems because convolutions are actually quite efficient calculations. But Transformers have kind of taken over the NLP world, natural language processing, but now they're making their way into object detection, computer vision. One would have thought maybe this was once ruled by convolutional neural networks. I think they are actually still a part of it. Yeah, there we go. So this is their new model, DETR. I think it's detection transformer, so object detection transformer. They use CNN features with standard transformer encoder-decoder and then predict boxes and classes. Now this is a more traditional object detection model here, Faster R-CNN, and it has a bunch of different steps in there. So using transformers kind of simplifies things. But again, this is a blog post, there's a paper to go along with it. Um, there we go, Advancing Towards a Unified Approach for NLP and Computer Vision Tasks. One of the cool things about using transformers is that— where is it? There we go. DETA's novel architecture can also help improve the interpretability of computer vision models because it relies on attention mechanisms, which is a part of the transformer neural architecture. Attention— I'm not going to go through that, but check it out. Basically, it's what does a model pay attention to when it's figuring out different things. It is easy to see what parts of an image the network is looking at when it makes a prediction. So that's a real big point in explainability, is if your model makes a prediction on an image, what part of that image caused that prediction? So if we come back, the last one is 10.5. Speaking of Transformers, which is what this post was kind of about, end-to-end object detection with Transformers, if you want to learn what a Transformer actually is, uh, apart from Optimus Prime, This blog post is a pretty good overview, and it also links to different resources about other transformers. So there we go, classic fully connected neural networks, CNNs, RNNs, kind of talks about different other architectures, and then breaks down— there we go, more on attention, attention and CNNs, the transformer, attention is all you need. So check out this post if you want to understand more about Transformers. That's actually a really cool one. So there we go. I'll see you next month. I mean, that was it for June. If you did— we'll go back to the main screen— if you did enjoy this video of me going through, I'm not sure how long it went for, but if you did enjoy this kind of video of me just going through some of the coolest things over the last month, let me know in the comments below. If you think, well, Daniel, I just prefer to read the blog post, Well, that's cool too. But again, if you'd like to see more of these, let me know. If you have anything you'd like to see in another issue of Machine Learning Monthly, also send it to me or leave it in the comment below. Otherwise, if you'd like to get that post, it comes out at the start of every month, roundabout-ish. Some of the coolest things that I've found over machine learning and why I found them cool. Be sure to subscribe on the zeromastery.io blog. But otherwise, as always, keep learning, keep creating, and I'll see you next time. Take it easy.