I've got a new favourite machine learning book | Machine Learning Monthly October 2020

5, 4, 3, 2, 1. Oh, we're on here. Um, well, hello and welcome to this video. I'm, I'm assuming that you're here for Machine Learning Monthly for October 2020, and if you are, well, that's very exciting because I'm very excited to— I mean, put it this way, as you'll see in the video, it's been a massive month on tour. If you're new to these kind of videos, what Machine Learning Monthly is let me just show you rather than talk about it, is I write an article once per month. There's my pretty face on the zeromastery.io blog. It's called Machine Learning Monthly. You can sign up for it for free. There we go. Just chuck your email in there. And every month at the start of the month, it's usually within the first few days, you'll get this email of the best or the latest and greatest things in machine learning, but not also just the latest and greatest, you know, because there's a lot of stuff that is evergreen, has stood the test of time. And so I include that in there. So if anything that I've found curated by yours truly that I think you should be aware of, I put it in this little Machine Learning Monthly. And so what we're gonna do is if you just wanna read along, the link for this article will be in the description below. But I'm gonna just add some commentary over the top of each of these things. So without any further ado, let's get started. So what you missed in October as a machine learning engineer. So I put out two relatively cool pieces of work in a— you need to be your own biggest fan. Come on, let's be real. In October. So one of them was an article, The Ten Commandments of Self-Taught Machine Learning Engineers. And basically it's a setup of a— I remix the actual Ten Commandments. I legit did this. I took the Ten Commandments from the Bible or religion and whatever, and then sort of change them to be, have a machine learning flavor. Because if you're going to be a high-quality machine learning engineer, it's good to have some principles about you to sort of guide your path. So for example, number 1 is math, code, and data are your holy trinity. If your data is of poor quality, it does not matter how elegant your mathematics or efficient your code is. So I won't go into all of those, but I will, I will let you read through them. My favorite one is number 10, you shall not covet. So you should seek to build your skill at applying the Trinity and answering the questions of those you seek to serve, but you should not do so with desire. Desire places a curse on you to forever take the future too seriously rather than enjoying what you have now. Remember, the journey to learn anything, including machine learning, especially machine learning, It's a long one. So what can you do to cure it? Develop a love for learning. That's why my motto is— we'll wait till the end for that one. Anyway, check out that article. I also made a fairly lengthy kind of documentary commentary video. Oh, so excited. Oh, I say that I'm excited a lot, right? Um, this is a review of the State of AI 2020. So I believe if we go down here, State of AI Report 2020, we'll have a look at that in a second. But I also, if you want to check it out, this is a fairly lengthy video, added some commentary over the top of what was in the State of AI Report 2020. So they're my two pieces of work. But now that's enough about me. Let's get into the community. So Kevin actually sent me He emailed me this. And by the way, if you have any of your own work that you would like to see in a future issue of Machine Learning Monthly, whether it be a blog post or a tutorial that you've created or some sort of prototype app that you've built, whatever it is, if it's to do with machine learning, send it my way. I'm interested in hearing from you. So this is exactly what Kevin did, and he made a mask detecting Telegram bot. So if we go in here, let's, let's check out the demo. And he also did a great job of sharing his work. So this is a really— first of all, thank you, Kevin, for sending me this. And second of all, really amazing effort on what you've done. So if we go here, let's zoom in. Can you see that? There we go. So what Kevin's bot does is you chat with it on Telegram, which is a chat app. You take a photo of yourself. And the bot will tell you whether or not you're wearing a mask. Looks like you're not wearing a mask. Please put a mask on. And then Kevin puts a mask on, takes a selfie, sends it to his Telegram bot, and then it says, Looks like you are wearing a mask. Woo-hoo! And so beautiful tweet and little video that you shared there. And he also did a— now the reason why I love this sort of— this actually inspired me to sort of my, for my own prototypes going forward in machine learning is to do it with a Telegram bot. Now the reason being is if you have an app that, for example, Kevin's one, you could take a photo and send it through to a Telegram bot. It queries a machine learning model that I believe Kevin hosted on Heroku. By the way, this is a great article if you want to build your own machine learning bot prototype. The reason being why Telegram is, is a great option is because you can use it on basically then any device that can run Telegram, because rather than developing a pure iOS or a pure Android app, if your phone, if your users or someone wants to try out your app and they have Telegram, they can just message your bot. So that's a real cool way to sort of get a, get a prototype out the door to test it out, test its functionality out. Kevin also did this amazing blog post, which is linked in the Machine Learning Monthly if you want to see how you can build your own Telegram bots powered by machine learning. And he used inspiration. So this is what I'm saying. If you want to work on your own machine learning projects, I really like, like finding, finding something that exists online and then replicating it yourself. So I'm going to mute that. So Uber announced this mask verification app, um, as part of their Uber app. So I think if you wanted to, uh, drive Uber, I think they had some sort of policy because of coronavirus and whatnot that it was, you should wear a mask while you're riding an Uber. So they had this little application built into their Uber app. Um, so yeah, Really cool from Kevin. Thank you so much for sending it my way. Now let's get on to what's been on the internet. This was an absolutely massive month for natural language processing. I mean, it's been a, it's been a, ever since the release of BERT, which is, um, what was that? The end of 2018. It's been an absolute NLP landslide. So I've got how many? 1, 2, 3, 4, 5. Look at this. I've got 5 resources just for natural language processing. So let's go through all of them. So we'll get out all of these. So the first one is Modern Practical Natural Language Processing by Jonathan Mugen. If we go here, now this is an NLP course. So it's got 4 notebooks. It's going to get you started on, on basically how to do modern natural language processing. So 4 notebooks, 4 videos. If you learn well using code and video, I personally like to learn that way. You might want to check out this course. It's all on GitHub. So if we got here, number 1 starts with overview converting text to vectors, a.k.a. how do you turn your text into numbers? Then learning with vectors. So once you've got your text in numbers, how do you classify those? How do you visualize the text in numbers? And then how do you generate sequences? So for example, if you take a body of text, so say you wanted to generate a blog post, you could take all of your previous blog posts, train a model to generate text based on your previous blog posts. That's what sequence generation is. And extracting pieces of information from text. So really great little NLP tutorial here. And by the way, I've said here, in this NLP firesale is that this is, this is actually a really important point is that you're going to find a lot of, of resources everywhere scattered all over the internet. But a really, a really, really valuable skill that you can develop is to be able to figure out what matters most to you, um, what you want to learn. Say for example, you wanted to learn NLP and of course you can't learn them all. So a really, really valuable skill is just going, you know what, I'm picking this one thing and I'm going to do this for the next month and then ignoring, ignoring the rest. You might be going, Daniel, what's the best thing? Well, truth be told, the internet is pretty good at putting out good stuff. So it's very rare that you're going to stumble upon something that's really bad. And the beautiful thing about machine learning is that it's code and math. So it either works or it doesn't. Now we've also got the Getting Started with NLP by Elvis Saravia. Saravia. Oh, that's a— that comes off the tongue nicely. Actually, I would recommend just subscribing to Elvis's blog. Oh, there's my email. You can email me if you want. My actual email is daniel@mrdbourke.com. Is because Elvis just posts— I've been reading his posts for the last couple of years and he posts some phenomenal stuff on latest machine learning research. But this one is a full-blown recommendation for getting started with natural language processing. I'm not going to go through all of these resources. Again, there's going to be too many resources for you out there. So, it's up to you. Every individual has to be in charge of their own education. But I totally recommend checking this out. Probably one of my favourites here is, and I would highly recommend this course, is CS224N. It's Stanford's natural language processing course. It's all free. It's from Stanford, one of the best universities in the world, and it's on YouTube. So yeah, that's probably my favorite out of the whole resource. All right. And then if we go back, we also have the Super Duper NLP Repo by QuantumStat. So if we come in here now, this is, I believe QuantumStat have another few resources, datasets. There we go. Oh, my head's in the way. I'm going to move me. Boom. So if you see over here, we've got datasets, models, notebooks, etc., etc. So QuantumStat have put together a phenomenal collection of all of these are Colab notebooks that are ready to run with different, different problems in NLP or different examples of problem solving. So really, if you want a demonstration of how to do some sort of problem, that's how I usually start. If I'm solving a project is I kind of start with what someone's already created rather than reinvent the wheel, and then I tailor it to my own needs. So I'm still getting— I'm still in that stage where I'm being able to sort of implement my own things from scratch. But for now, that's how I approach each project. So for example, if we go text generation with an RNN, I'll put my face back up here. This is going to open into a ready-to-run Colab notebook. Loading. Boom. Text generation with an RNN. And so if you go through here, there's going to be a whole bunch of code. That's going to help you to— there we go. Now run the model, see how it behaves as expected, check the outputs, etc., etc. So really recommend this. That'll be bookmarked. Bookmark that under your NLP resources. All right. Now if we go to the next one, when I said NLP firesale, I really meant it. There's a lot of NLP resources here. This is another phenomenal resource or newsletter that you subscribe to, should subscribe to. Apart from Machine Learning Monthly, of course, your number one source for the latest and greatest in machine learning. This is Natural Language Processing News by Sebastian Ruder. Sebastian, again, I've been following his work for the last couple of years, is a phenomenal, phenomenal researcher in the NLP space, and his newsletter collects machine learning resources, but they're focused on NLP. So I actually got a lot of these resources straight from Sebastian's newsletter. So big shout out to Sebastian. Thank you for all the work that you do. We have the NLP, ML and NLP Starter Toolkit. So if we come down here, I'm not going to go through all of these because again, it's, it's, it's rather than just sort of talk through it, I'm just going to show you what's going on. So one of my favorites in here, I believe, was not only the low resource NLP toolkit section, because that's where we're going sort of with models, right? Is rather than you see all these like big GPT-3 models and they cost— these things cost like $10 million to train and need a big bunch of servers to run and whatnot. So low resource NLP, so things that could run on your device or your local machine without requiring millions and millions of dollars of training budget. That's, that's what excites me in the field of machine learning at the moment. And this is what we want, the NLP tutorial. I believe we were going to go through that in a second. But anyway, so this is again some phenomenal work and I'd highly recommend if you want to like sort of create your own. I mean, I'm really inspired by this one. So this is by Tae Hwan Jung. So currently an undergrad student. And what he's done, or what Taiwan has done, is collated many of the most important models and techniques in natural language processing and created tutorials for all of them. So this is, this is a phenomenal project. And I would highly recommend if you're getting into the field, so you could do this for, say, computer vision. So if we dive into one of these, if we want to go into Let's go into FastText. So FastText is a really cool natural language processing library made by Facebook. It's actually incredibly fast, hence the name FastText. So if we go here again, these are all runnable notebooks. Install FastText. So this is how I would start an NLP project if I wanted to use FastText. I'd probably grab this. So that's a lot of install code. Make a simple dataset. So FastText takes, uh, strings with the label at the front in that format, followed by the string here. Make your dataset, train it, and then predict. How easy is that to get started? And so this is a great reference to sort of make sure you start this one. Give Tae Hwan some, some love and bookmark this under your NLP resources. So I believe that is the end of the NLP firesale. Now let's go on to the rest of the ML Monthly. So we have the State of AI Report 2020 by Nathan Benaj and Ian Hogarth. I won't go through this too much because I've got a full video dedicated to this one. But basically what this is, is State of AI Report analyzes the most interesting develops in artificial intelligence. So we aim to trigger an informed conversation about the state of artificial intelligence and its implications for the future. And that is exactly what it did because I read this report. I got so excited when it got released. It took me a couple of hours to go through. I'm not going to lie. It's, uh, 200 or so slides. There we go. So if you want sort of a one-stop shop of everything that's happened in the AI space, I mean, if you wanted to cover everything, it would be a couple of thousand slides. But, um, this is what you should check out. Set aside some time on your weekend and go through this. So research, talent, industry, politics. Beautiful. So big props to Nathan Benh and Ian Hogarth. Thank you for putting that together every year. It's something that I look forward to. It's become a yearly tradition for me to go through this. Um, anyway, oh, this is another one, which is, see, I think I mentioned it in here later on, but MLOps I'm starting to get obsessed with. And so to me, the model building phase of machine learning. So once you've sort of developed what your inputs are and your desired outputs to sort of connecting the inputs to your outputs with the model, the pipeline in between that, that's sort of starting to get to a fairly mature state. So if you had a computer vision problem, you're probably going to use a CNN, but maybe spoiler, you might use a transformer in the future. So now we're sort of moving towards how do you run those models at scale? And this is where a lot of my time over the next sort of 12 months is going to be going towards is learning this, this material. And don't worry, I'll, I'll share it as I learn. But this is a great guide to sort of getting your head around what MLOps is and data and infrastructure. So when I say MLOps, I mean running machine learning models at scale in some sort of application. So I would read this in its entirety, Emerging Architectures for Modern Data Infrastructure. What that means is, okay, you've got your model. That's actually a small part of many machine learning pipelines, but how do you ingest data? So take data from the world, store it somewhere. How do you transform it so that it's ready to be used in your model? And then once your model makes some sort of prediction or whatever on that, how do you then retransform that and deliver that back to the user and then create a feedback loop so that whole system is working synergistically? Logistically. So let's go down. I really like this. Uh, so there's some, some big dog companies here which has sort of shared their views. Oh, that doesn't look very good. Can I click on this? There we go. Um, so one of my favorites here was UPS save 100 million driving miles and $350 to $400 million per year with Orion Route Optimization System. Netflix generate more than 80% of content views through ML recommendation system. I would say this is a trend almost on all content platforms. I mean, when's the last time you actively searched for a YouTube video? So we need to go back. A unified data infrastructure architecture. So this is a very— when you first look at this is a pretty complex diagram. But once you break it down, it's like these are your sources of data. This is how you ingest them and transform them. This is how you store that data. I'm not sure entirely what historical means, but this is how you make predictions and this is how you make outputs. And so in a working full-blown ML system, you would— this is kind of in a whole, whole loop. So again, I'm, I'm still learning about this space, but resources like this from Andreesen Horowitz are really great to sort of get a kind of a bird's eye view of the whole ecosystem of data infrastructure. Highly, highly recommend that article. And speaking of putting ML into production, Made with ML, Goku Mohandas, the founder of Made with ML, are releasing an incredible course called Putting ML in Production. If we go here, a free course, by the way, I'm going through it. I've watched a few of the videos on YouTube already. So putting ML in production, highly, highly recommend this. So if we go here, we've got lessons. Boom. So this is what we're going to go through. I'm going to go through product. So objective, solution, evaluation, iteration. Sorry, this is what you'll go through if you go through the Putting ML in Production course by Made with ML. But I guess you could say we'll go through it too because I'm going to go through it as well. Then you have data collection, annotation, exploratory data analysis, splitting, preprocessing, augmentation, modeling, scripting, API production. So what a phenomenal, phenomenal resource. This is nothing short of inspiring. There's also a YouTube series to go along with each one. So I've gone through the first 3. I need to go through evaluation. So be sure to subscribe with the Made with ML YouTube channel and make sure you check out this course by Goku Mohandas, Putting ML in Production, as Goku goes through and builds Made with ML's first machine learning powered feature. Really, really cool resource. All right. Book. Oh my goodness. So what do we got here? I actually just finished this bad boy, Machine Learning Engineering. So if we go here, we click on this by Andrey Burkov. You can read the whole thing for free, by the way. Um, so this is Machine Learning Engineering. I feel like it's got my last 3 years of, of Google searches in one book. Legit. So when I first read the table of contents, it gave me goosebumps. So we've got introduction and before the project starts. So how do you set up a machine learning project? How do you collect data and prepare it? Feature engineering. I'm not going to go through all of this because I'm going to do a review video on this book. So stay tuned for that. But I kind of went through it and read it nice and simply. If you can see that there's a— I went through about 10 pages per day. So that's kind of how I like to, to go through things is of course you could read down and sit in one day, but this is the higher quality of the book, the slower you want to go through it. So this is a reference book. It's going to sit on my desk for, for the foreseeable future. Um, so go check that out. Phenomenal book by Andrei Burkov. Again, you can read it all for free. It's becoming— it's easily within my top 5 machine learning books. The other book by Andrej, the 100-page machine learning book, is, is also in that top 5. So I'll probably make a video on that in the future too. So make sure you check that out. If you want to read it for free, do it. If you want to buy it, also do it. I just like having a physical copy to carry around with me. Now, We go NumPy, the manifesto. So if you're in any type of data processing in terms of data analytics, data science, machine learning, you've probably used NumPy. Or even if you haven't directly used NumPy, unlikely. If you're using Python, you're using NumPy. If you're using Pandas, it runs NumPy under the hood. Same with Matplotlib. And what they've done is there's now NumPy the paper, which is kind of it discusses NumPy's whole thought process around what the NumPy array is, because again, NumPy is just completely based on the array. So you've got the data structure, you've got the indexing, indexing copy, vectorization, broadcasting, reduction. It's actually beautiful that NumPy, the whole entire of NumPy is based around a few fundamental concepts. So if you wanna read up the paper of NumPy, like where it's heading, what it's, where it's related with the ecosystem, get a bit of the history of the programming language that you're, or programming framework that you're using when you write all your code, I'd highly recommend this paper. Because not only is it important to sort of pay tribute to those sort of who have come before you, especially the NumPy people who have, the amazing developers who have built the NumPy library and created it as open source so that people like you and me benefit from it. It's also good to know like where it's heading and sort of how you can potentially, I believe down the bottom is, here we go. But importantly, for NumPy to meet the needs of the next decade of data science, oh, we got too zoomed in. But importantly, for NumPy to meet the needs of the next decade of data science, it will also need a new generation of graduate students and community contributors to drive it forward. So again, if you're a graduate student or you're looking to contribute to some sort of open source, potentially NumPy, if you want a library that's going to be used by hundreds and hundreds of thousands of people around the world, you might want to look into contributing to that. So if we go here, boom, we've got another resources GitHub. I really like these. This one is for the incredible PyTorch. So if you want basically anything to do with PyTorch from tabular data to tutorials, visualization, explainability, object detection, missing data, medical, sorting, GANs, VAEs, unsupervised learning, etc., etc. You're going to want to bookmark this, this PyTorch repo. So again, if you're getting into PyTorch, I'd highly recommend going through this. I'm going to give that bad boy a big star. Let me move my head again. There we go. Starred. So we come up here. So this is phenomenal work from Richie Ng. Thank you so much. And I believe— why might you be interested in PyTorch? Well, if we come here, if we go resource research, PyTorch is also more popular than TensorFlow in paper implementations on GitHub. Now that's, that's research focused. TensorFlow, I believe, still wins out in the industry. But again, competition is good, right, between two deep learning frameworks. So if we come here, check out that if you want to, if you want a one-stop shop for everything to do with PyTorch. And now we have one of my favorite innovations of the last year or so is using Transformers. The Transformer architecture, the architecture that has transformed— pardon the pun— NLP is now coming for computer vision. So if we go in here, so an image is worth 16 by 16 words. Transformers for Image Recognition. Again, my face is getting in the way. So this is a great paper on explaining how— and I believe it's still anonymous, so I can't, I can't attribute to who, who created this paper. But if you did, thank you very much for your work. I really liked reading through this paper. If we go here, it's really well written. So this is basically talking about how they use transformers for Image classification, which is— or is it classification? Yeah, image recognition. Sorry. Well, it was classification because they were doing it on ImageNet. If we come down here, the biggest improvement here was boom, ImageNet Real was, I believe, that it had far less training time. So yeah, this is a, this is a real important point, is that it was able to get state-of-the-art results. All of the best results here are in bold, but it had far less compute time than previous methods. So see here, I believe this was BitNL. So I think this is Big Transfer, took 9.9K TPUs v3 days. If you're not sure what a TPU is, a TPU is a, a very fast computing chip. So if we go here, there we go. If you have a look at that, so instead of— of course, all these days, these numbers, um, because it's— you're running on basically— this is like, let's just say, arbitrary number of 115 computers in one, in one little section. So one day is actually— if you run for an hour, it's actually running for 115 hours. Again, my math there is probably pretty bad, but that's how you can imagine it. So instead of taking 10,000 hours or 10,000 days, it took 2,500 days, which is still a very long time to— if you're— well, I don't know how long that was on wall time, but it sounds like a very long time. But the important thing is that it's, it's a quarter and still got better results than the previous state of the art. So watch out convolutional neural networks. Transformers are coming for you. I'm really, I'm really excited to see what's going to happen in the next sort of 12 months with Transformers in the computer vision space. So if you want to be on sort of the cutting edge of what's going to potentially happen in computer vision, make sure you read that paper. There's also a couple of resources I attached here for learning more about Transformers. A great annotation here from the paper Attention is All You Need, which was the— that's how I like sort of learning new concepts as well now is the past me used to avoid going to the original research because it was too complex to read. But now if I'm trying to learn a new concept, I like going to the source of where it originated from. So the transformer architecture originated in the paper Attention is All You Need. So I like going to the source and trying to, even if I don't understand it the first time, just, just practicing being a little bit uncomfortable trying to understand these things. And so let's, I think we've got a couple more. Let's wrap this up. Two brilliant resources here. So this is RoboFlow. I just totally, I mean, subscribe if they have a blog or something like that, subscribe to this because they're doing amazing work in the computer vision space. So how to train a custom mobile object detection model with YOLOv4 Tiny and TensorFlow Lite. Why is this important? If you wanted to build an on-device object detection model, you're probably going to want to use something like YOLOv4 Tiny and TensorFlow Lite. And guess what? RoboFlow have a tutorial of how you can do that. We go through here for this case. I believe they built an Android application to identify different blood cells from blood microscopy. I'm not sure what it's called, like blood pathology scans. And they also have a YouTube video to go along with it. Phenomenal, phenomenal work. And also full disclosure, Roboflow sell a computer vision product so they can help manage your datasets and build models and whatnot. But this is, if you are going to start a business in some sort of field of machine learning or whatever business it is, you got to remember, okay, you can have a product, but whatever business you're in, you're also in the education business. So you have to teach people how they can use your product. And that's what RoboFlow do here. Of course, I'm sure you could adapt the code if you don't want to use their, their product. You could develop your own database and just plug in the data to the model that they share. But the way I like to think of it is that if you're a business, You gotta think about what is your, what's your cooking show? So if you imagine all the famous chefs, right, they reveal all of their secrets. They show you how to do, they show you how to cook. And so if you're building some sort of business, say it's a machine learning product, show people how to use it. Or whatever app you're building or whatever business you're running, show people how they can gain value from your business. Anyway, speaking of gaining value from a business, Gradient Descent is a podcast by Weights and Biases, one of my other favorite machine learning companies. I would highly recommend subscribing to this podcast. It, it contains interviews from people all throughout the industry. I kind of, I love to listen to this one on, say for example, if I'm going for a drive or if I'm I'm doing some, some chilled out coding, I'll listen to this in the background and just hear what it's like from people in the field because there's nothing to me that's inspiring when I can hear about what other people are doing in the machine learning field. I get excited to upskill myself and potentially work on things that they're working on. So one of my favorites recently, I still— I've bookmarked it. I haven't listened to it yet is Building industrial strength NLP pipelines with Ines and Sophie of Explosion AI. So if we go here, where it is, boom. It's on YouTube as well. So you can subscribe to the Weights and Biases YouTube channel. And so Explosion AI are the creators, another one of my favorite AI companies. Explosion AI are the creators of of spaCy, Prodigy, and Think. So just an amazing team. And so for me and you, of course, to be able to hear with a couple of the founders is— how phenomenal is that? So really recommend checking out the Gradient Descent podcast by Weights and Biases. Woo! But anyway, we are, we are done. That was the Machine Learning Monthly for October 2020. A massive month on tour. We had a whole bunch of NLP. We saw Transformers for Computer Vision. Um, what else do we see? We saw the incredible PyTorch, NumPy: The Manifesto, The Machine Learning Engineering Book. So highly recommend checking this one out. This is easily one of my favorite books by Andrey Burkov. Thank you so much for the amazing work. Putting ML into Production by Goku Mohandas from Made with ML. A 2020 Guide to Data and Infrastructure and the State of AI Report for 2020. So, oh, and of course, Kevin's amazing prototype for building a mask-detecting Telegram bot. So inspiring work. Thank you so much to everyone who, who sent stuff in. If you have something for a future episode or article that you'd like to contribute, please send me an email. My email is Daniel@MrDBourke. Otherwise, Keep machine learning, keep creating, and I'll see you in next month's monthly. Peace.