It's going to be a big (and fast) Summer of learning | Machine Learning Monthly August 2020

You know what time it is. We're a few weeks late this month, but as you might be able to tell if you've seen a previous video, I'm in a new space. I kind of moved place, so I needed to reorganize my studio. I'm still figuring out a few little kinks in terms of, like, there's a road, you can't see it, but there's a road, like, just there. So, I'm kind of figuring out how the best way is to record audio. So, if you have any feedback on, "Hey Daniel, this video sounded like crap," leave a comment below. But otherwise, What we're going to do is every month I write an article called Machine Learning Monthly. Let's have a look at that. Let's get that on screen. Boom! There it is on the Zero to Mastery website. There's the handsome chap that wrote that, yours truly. And it just collects all of the, or about 10 or so things of the coolest stuff that I found in machine learning over the last month. And because it is September, it is September 19th today, we're going to go through last month's. We're going to make a video version of this and we're just going to have a little bit of a chat. And so I can catch you up on some of the latest and greatest in machine learning field. And it's not always about the latest and greatest. There's also stuff in here that could be a little bit old now, but it's, it stood the test of time and it's amazing and it's worth checking out if you want to, of course. But that being said, let's get into it. So this is the Machine Learning Monthly August 2020. I'm not actually sure what this one will be titled yet, but usually we title it based on something in here. If you want to get this article as a newsletter in your email inbox once a month, go to this website. It'll be the top link in the description. You can sign up and then you'll get this every month. Otherwise, you can just watch these videos on the channel and just hear me talk about it. Though the article is probably a quicker read because I tend to be a bit verbose. Alright, so what you missed in August as a machine learning engineer. So this is what I did. I wrote an article. Called "How I'd Start Machine Learning 3 Years In." So 3 years ago, there's my old room. Brings back a tear. If you want to know more about the new place, leave a comment below and I'll, might do a video or something like that about it. I really like it. Anyway, I started learning machine learning in this room and 3 years on, it's about actually 3 years to the day today. I think it was September 17th. Anyway, that doesn't matter. I know a lot more than when I started, that's for sure. But let's be real, I'm still figuring this out too. So, anything I say on all my videos, I always say this, take it with a grain of salt. Trust your own intuition. And so, in this article, I put together a few points, things, mostly pieces of advice that I tell my past self. So, things like, The curse of the engineer, the technology nerd, is use less tools. If you want to build things, that's the number one thing that I would change. I won't dive too deep into this because you can read the article. I would spend more time building things, so getting projects into the hands of others. That's my sort of nature, is being more practical, and so that's the major thing that I would change. There's also a video version to go along with it. There's my old haircut. Thankfully, I've grown some more hair since then. Anyway, that's not what you're here for. You're here for the best from the internet. Now, is my face getting in the way there? Nope. I'm not going to read straight from the article. I'm just kind of going to click on the resource and then talk about it. So, there is nothing more that I love— oh, this is going to play out loud. I'll mute that. Check out this. Look at that. I need to figure out how to do animations like that. If you know a resource where I can learn to do these things, I might actually reach out. So, this is an amazing series on a visual introduction to neural networks. So, if you're like me, I like to learn things with multiple senses. So, with my hands, I'll write it out, with my eyes, with my ears. I like to listen to things, especially if I'm learning something for the first time, is I like to hear someone else talk about it for however long and then try to reinterpret it in my own words. But this is a little series by the YouTube channel vCubingX, definitely worth checking out, subscribe to them, on neural networks. Just look at that, the inside of neural networks. So, I mean, it's one thing to write neural network code, it's another thing to to know just what's happening inside. Look at this. See, I'm making things with Keynote, and these guys are making it with full-blown animations. So, note to self, step up your education game, Daniel Bourke. Really big shout out to this series, so go check out vCubing X. I'll leave a link to all of these things in the description, by the way, or you can just jump back to this article and they'll all be there. Anyway, that was That was number 1. Number 2 is using machine learning to detect deficient coverage in colonoscopy screenings. Now, I thought this was pretty damn cool to see— is my face— I might move my head up there. There we go. I thought this was pretty damn cool to see Google, or how machine learning is being used in healthcare. To aid, like, physicians. So I don't think it'll ever fully replace— like, this is what a lot of people get worried about, is will machine learning or AI replace my job? I would say if it's a continually repetitive task, chances are that AI will eventually be able to do it better than you. But in terms of being able to provide care for a person, I think AI is a very long way off there. So what my take is in the healthcare space of machine learning in healthcare is Let computers do the computer stuff, like the calculations, the finding patterns in data, etc., etc. Let the people do the people stuff, like just simply being there for someone who is unwell and just needs the space or the energy of someone else around them. That's what people are good at. I think we're a fairly long way off for that of computers. But essentially what Google did was they used a machine learning based algorithm. Let's get some photos. So, what happens with a colonoscopy is you get, usually get a camera shoved up you know where, and what it's going through, close your eyes if you're not willing to look at some graphic stuff, it's going to go through and take photos of your intestines, so this is what we all look like inside, and then what it's looking for, what the camera is looking for, what a doctor usually looks for is something called a polyp. Now, I don't know if there's an, oh, maybe, yeah. See, a motorbike just went past, and that's the kind of thing I need to figure out in this new studio, is that there is a road right there. So this is the kind of thing that it's looking for, but the problem is, is that when the camera goes through— so I'm not sure if we go here— yeah, so it can detect where the coverage is deficient. So you imagine, right? You're getting a camera, it's going through your intestine. Your intestine is fairly large, there's a chance that the doctor might miss some portion. Let's just say, I can't, I don't know which is missing, which is not, but let's say it missed the red portion here. And then, in that red portion was actually a polyp. And so, that polyp turned out to be dangerous, so even, or cancerous, so even though you got this scan, Right? There was still missing potentially cancerous tumors inside you. And so what the— they've called it SeeDooD2, C2D2, SeeDooD2, C2D2. What this does is it uses an unsupervised learning method and it kind of figures out, hey, we're going through your intestine, or like the camera, we've looked at all the camera footage, this is the algorithm, right? Looks at all the camera footage and goes, you know what? Hey, we've reconstructed this, this person's intestine and we think all of these red parts you've actually missed, so it's probably worth going back through and checking out these red parts. So we can find something like this. Now, I think the metric here that was worth paying attention to is this is performance on synthetic videos, so of course, take that with a grain of salt because it's synthetic. The real metric is real videos that you want. Is that mean absolute error, which indicates how much the algorithm's prediction differs on average from the ground truth, we find the mean absolute error is below 0.1. That's amazing. So meaning the prediction of C2D2 is within 7.5% of the ground truth on synthetic videos. But on contrast, a group of physicians, so a group of doctors, given the same task achieved MAE 0.177, so almost, or over double, 2.4 times. So the algorithm on synthetic videos is 2.4 times better than physicians. So that's not, to me that doesn't replace physician, that just gives them another tool to use in their arsenal to provide better care. Um, and then we got here performance on real videos. Uh, I think I remember they said it was difficult. Yeah. Difficult to assign a score directly. The task of verifying— so what the test here was, the physicians verified the algorithm's predictions. And so C2D2's score matched up with physicians 93% of the time. Now that's, that's pretty darn good. And so here's some just demonstrations of what it looked like. So here's a poor score. You can see that they're pretty poor images. Here's a fairly okay score, and here's a really good score, right? So they're pretty clear, right? That's like a full-blown tunnel through your intestine. That's crazy. So check that out if you're interested in machine learning and healthcare. This is, this is cream of the crop stuff for you of how machine learning can be used in the healthcare space. Let's go back. Of course, and why that matters is because colorectal cancer results in nearly a million deaths per year in the US alone. So that doesn't even include my home country of Australia, that doesn't include if you're not from the US, I mean, colorectal cancer could be pretty prevalent in your country too. Anyway, on to number 3. Oh, I'm very, very excited about this one. Now, this is almost a month ago, so it seems old, but it's, as I said, these things are, these things are evergreen. Fast.ai courses are some of the best online to, well, I would actually, if I was to say, if you were to do one course on AI, it would be fast.ai. Now, I haven't personally gone through this yet, but I just know how quality it is. I do actually have— so this is, this is a multi-part one. So if you want to learn deep learning, check out fast.ai too. There's actually 4 big releases and I've got one of them. So not only do we have fastai v2, which is a library, we have 3 other mini libraries, and we have another course, and we have this book, Deep Learning for Coders with fastai and PyTorch. This just arrived the other day. So excited to go through it. I think this is going to be my summer study. So Australia, our summer is December to January. So I think this is what I'm going to go through fastai version 2 and read the book over the summer. So I'm really excited about that. So check it out. There's a new library, brand new. I remember my first real interaction with fast.ai was when I was working at a Brisbane tech company as a machine learning engineer, and we were stuck on this natural language processing problem. And I watched one fast.ai lecture and then applied the technique from that fast.ai lecture to the, the problem that we had in natural language processing. It was text classification. And the, the technique that I applied from fastai wiped the floor seriously with everything that we'd just been spending a week working on. And that ended up being the, the model that we deployed in practice, the fastai model, which is built on top of PyTorch. So incredible resource. Jeremy's my favorite teacher out there. Go check him out. Go check out Fast.ai's new stuff, all of their new stuff. And of course, read their blog posts. They're actually real good quality blog posts and examples of how you can share your own work. And if you don't have your blog to share your own work, use Fast— is it FastPages? Yeah, there's a bonus. FastPages. Boom. Here's how you can make your blog. FastPages. All right, we're getting distracted. And there's a couple of bonuses in there, like the podcast that Jeremy did with Lucas Biewald, who's the CEO and founder of Weights and Biases, another one of my favorite machine learning tools. And there's also the Zero to Hero with Fast.ai series by Zachary Mueller. Oh, it doesn't work. And anyway, So that might be— I would just Google that. Google Zero to Hero series with Fast.ai. Zero to Hero with Fast.ai. Boom. Oh, there we go. So yeah, Google if you want that. That's a, a great series of blog posts by, um, who was it, About Zachary Mueller. I just said his name. So, big shout out to Zachary Mueller. That's some amazing work of deconstructing, say, a new tool or a new way of doing things into his own words. So, not only did he take the time to sort of learn it himself, he took the time to put it in a way that you can understand as well. And that I really find, like, that was one of the points in my last video was if you want to learn something, teach it. So, give that a go. Anyway, number 4 is MinGPT by Andrej Karpathy. So if you've been browsing the internet of machine learning, you've probably found that GPT-3 is phenomenal. Like, the results it's getting, it's just, it's just hard to even comprehend. I mean, in a test where GPT-3 was asked to produce a fake article, or just to write an article of about 500 words in length, The full version of GPT-3, so I believe it's 175 billion parameters, was producing articles that were almost or basically indistinguishable from a human-written article at 500 words of length. I think it was a team of humans were, were asked to decipher, was this GPT-3 or was this a human article? And they got about 52% accuracy, which is basically a coin toss. Anyway, MinGPT, MinGPT is a pure PyTorch implementation of a mini version of GPT. And this basically explains it. So this is GPT-3, big warship, and this is MinGPT. But if you want to see how a giant language model is built, well, then this is the repo for you because I think it's about 300 lines of code or something. Something like that. If we go in here, model, there we go, 10. Look at that, 200 lines of code. I mean, there might be some, some, uh, extra things here, like this is a trainer.py. I think this is just boilerplate PyTorch. Yeah, there we go, that's about 100 lines. You know what, a good little project would be to just basically, if you want to understand GPT, because GPT is the architecture that sort of builds GPT-2, GPT-3, would be just to go into this repo and rewrite the entire thing or copy, just copy it line for line. And so you can understand it yourself. That's what I find really helps me if I'm trying to understand something. I just literally look at someone else's codebase and retype it myself. So let's go into here. And what do we have? Number 5, Less Supervised Computer Vision. So that's what I'm noticing, a trend. Lately is that I think that was the last Machine Learning Monthly, the trend was, or the video was called, The Future Will Not Be Supervised. And it makes sense, right? So if you think about supervised data, how much effort does that take to label it all and just collect it and whatnot? So now with GPT-3, that's an example, that's a completely unsupervised model, yet it's getting results that are better or if not equal to supervised deep learning methods. And so I've got a few links here on different weekly supervised methods in computer vision. Learning with limited labels tutorial from NVIDIA at ECCV. I think that's the European Conference for Computer Vision 2020. And then deep learning with limited labeled data seminar by Colin Raffel. If we go here, So there's the tutorial on weekly supervised computer vision. So this is what I'm interested in, I think, over the next, I would say, year or so maybe, is two things: is uncertainty in machine learning model predictions. I really like that. And models which use less data but still perform really well. So check out that if you're interested in computer vision learning with less labels. Same, same with this similar topic. There's a lot of stuff here on videos. Look at that. So limit labels in a label-less world, self-supervised learning. So that's really, really, really— I think this is, this is the future of deep learning, machine learning, is self-supervised and unsupervised. And then here's a great little seminar by Colin Raffel. On a whole bunch of different— here we go, the content here on basically all of the recent advancements in weakly supervised learning, semi-supervised sequence learning, big self-supervised models are semi— are strong semi-supervised learners. I think maybe it goes through like the most recent papers. Yeah, there we go. So This is a reading list, but if you check out the GitHub repo, I believe there is all of the materials for the, for the course as well. So big shout out to Colin Raffel there. And now number 6 is ML Ops. So this is a tutorial series. Let's check it out on Made with ML, by the way. That's— I'm a real big fan of Made with ML, so make sure you check them out too. What is MLOps? So if you imagine, so machine learning is not just building a model in a Jupyter Notebook. It's what happens after. So say you did build a model, what happens after that? How do you get that into the hands of people? How do you, how do you build an application around that model so that someone could use it, say, on their cell phone? So if you wanted to take photos of dogs and identify what breed it is and you train the model, how would you get that onto a mobile phone? So this is where MLOps comes in. It brings all the practices from DevOps, which is a big thing in software engineering, into machine learning. So if you come here, we've got here number 1, it's a great YouTube tutorial series, beautiful, by the people at dvc— dvc.org. Daniel Bourke. Beautiful. I believe DVC is also data version control. So that's a great little tool. But check out this. If you want to learn more about MLOps, I definitely go through these videos here. It also utilizes GitHub Actions, which is a great tool. I'm not going to dive too deep into that, but definitely something worth checking out if you're looking into getting into full stack ML, which I definitely am. So maybe I'm going to— I need to, you know what I need to do? I need to create myself a summer of learning curriculum. And I think this might be part of it alongside the full-stack deep learning course, of course, and the fast.ai course. We've got a— it's going to be a big summer. Anyway, that was number 6. Number 7 is how to deploy and host a machine learning model with FastAPI and Heroku. Speaking of deploying machine learning models, this is something that you might want to check out on testdriven.io blog. So basically FastAPI I'll just, I'll just let this read it for me. FastAPI is a modern, high-performance, batteries-included Python web framework that's perfect for building RESTful APIs. It can handle both synchronous and asynchronous requests and has built-in support for data validation, JSON serialization, authentication and authorization, and OpenAPI. My goodness, I'm not even quite sure exactly what FastAPI is, but just that sentence sentence alone makes me want to use it. But if you read this tutorial, the goals are: develop a RESTful API with Python and FastAPI, build a basic machine learning model to predict stock prices, deploy a FastAPI app to Heroku. Now Heroku is a cloud provider, and what that means is that it's some, some computers in a big warehouse all around the world. And if you deploy your application to there, what it means is if you've got a script, say a Python script that's it's a machine learning model that predicts stock prices. If you put that onto Heroku, if you do it in the right way, it means that, say if I did it, I could then share a link with you to go and check out the code that I've written in a website. So really cool tutorial there if you want to learn how to deploy a machine learning model. Looks like it's all in pure Python. See, look at that. How cool is that? An extension to this project would be figure out how you could do it with— take the machine learning model and build a Streamlit application around it. But anyway, that was number 7. Number 8, I actually really liked this one. So this is a great— another example of how machine learning can be used. For me, it's in a healthcare setting or just in a, in a product-based manner. And speaking of products, this is on-device supermarket product recognition. So you imagine, I think it's not only just— so where's the demo? There we go. That's a really cool demo, by the way. This is using augmented reality, computer vision, and a bunch of just normal software engineering processes. So what it does is it uses— it looks at what product it is and then brings up information about that product. Product. So what they said at the top, Google is one of the greatest challenges faced by users who are visually impaired. So of course this, this app would definitely be helpful for someone who's visually impaired, but I think it's not even, not even for those who are visually impaired. It could be for anyone who just wants to learn more about the food that they're eating. So I think this— I've spoken about this a a lot before, but I think this— I want to work on some sort of project like this in the future. Maybe that'll be like my next, uh, end-to-end series, like the Airbnb series. Um, because at the end of the day, like, food labels or just food in general could be quite confusing. I mean, you've been told all these different things of what to eat, what not to eat. So this is where I see a use case for this has been really helpful in not only helping people who are visually impaired, but just helping people who just want to know more about what they're eating in general. So check out this blog post. I think it's all built with MediaPipe as well, which is an open source framework. Yeah, look at that. MediaPipe, that's really cool for building. I think it's, yeah, an open source computer vision framework for putting together multiple computer vision models in, in succession. So if you imagine, I believe the article says, it goes, to accomplish this, Lookout, so that's what the app is called, includes a SuperMatic product detection and recognition model and an on-device, with an on-device product index along with MediaPipe object tracking and an optical character recognition model. So yeah, MediaPipe is creating a pipeline of different computer vision models character recognition models, the product index is on device, so that makes sense. But really cool. And so the resulting architecture is efficient enough to run in real time entirely on device. So important, because if you're in a supermarket, you might not have that great an internet connection, and you want it to be quick. You don't want to be sitting there waiting for information from, from your phone. So check that out if you're interested in building products like that. And finally, number 9, I'm really excited by this one actually. So my last video, or one of my last videos, you saw that I built a deep learning PC using a NVIDIA Titan RTX. Well, turns out that a couple of days later, I think it was the 1st of September, NVIDIA just announced their RTX 30 series, which are just phenomenal GPUs. Like, they're— where's the graph? We got benchmarks here. So, and not only are they phenomenal, they're also at an incredible price point compared to the previous models. Um, compare specs. Okay. I think it was, maybe it was in the keynote. Anyway, I'll let you check that out because, um, unless it's somewhere here. Anyway, if you want to learn more about these, I do have a great blog post. Tim, there we go. Which GPU— if you just Google, I think it will just come up— which GPU for deep learning. This is, this is what I'd recommend if you're looking to build your own deep learning PC. You do not need one to get started. Use Colab until you do. You'll know when you do. Yeah, this post by Tim Detmers is probably the best online for it. Yeah, look at this. Look how much— this is so in-depth. If we go here, the most important GPU specs for tensor processing, tensor cores, of course. But there was somewhere— where's the graphs? Here we go. So, boom. If you want to build it with one GPU, so the performance per dollar The RTX 3080, so 10GB, is— I think by this graph it's the best. Uh, this is the new GPU here. And where is it? So that's this one here, RTX 3080. So $699 US. So what's that in Australian dollars? It's about $1,000 Australian dollars. Um, so check those out, some amazing new GPUs. What this just means is that Building state-of-the-art hardware is getting more and more accessible. I mean, that's the trend that it's always going to take, right? But at the end of the day, it's not the tools that you have. You don't want to be the guy who's got all the gear and no idea. It's how you use those tools. So check out NVIDIA's new GPUs and also check out— where's Tim's post? Uh, which GPU for deep learning? But all All those links will be in the description as well as if you want to get this, this article, Machine Learning Monthly, make sure you go to the link in the top of the description and subscribe here. That'll just go straight to your inbox. If you don't want that, you can just check out the Zero to Mastery blog. It'll come up there. It's usually the first or second day of every month. But yeah, this video will usually come out 3 to 4 days after this article comes out, but it's late this month because I was moving place. Anyway, it's been a massive month on tour in machine learning as usual. Let's switch back to the overall view. As I said, I'm still working out the new studio, but, and as always, keep learning and keep creating. And if you— what was your favorite? Leave a, leave a comment below. If you have any resources you'd like me to put in a future episode, it can be yours. I'd actually really like it if if you sent me something. You can leave a comment below or email me, daniel@mrdbourke.com, and I'll see you in next month's episode. Peace.