Data scientists should be more end-to-end | Machine Learning Monthly September 2020

You know what time it is. I can't believe it's been a month already since the last episode of this. If you're new to these kinds of videos, we're going to go through Machine Learning Monthly, which is a little bit of an article that I write rather than talk about it. Let me just show you. It's Machine Learning Monthly, the September 2020 edition. And of course we are October 10th. So this is for last month. So that's usually what happens. I write an article collecting the best things from machine learning, or at least in my opinion, over the last month. And it's not limited to the last month. It's actually because there's so much out there. It's just collecting the things that I found most valuable and potentially you might find valuable too. So without any further ado, this is the article that we're going to go through. There'll be a link if you prefer to read to this article. Uh, it'll be the first link in the description so you can read along. As we go through it. I'm just going to riff on it. So let's get started. What you missed in September as a machine learning engineer. So at the top, I usually put out my work. There was a video I did. I'm not going to show you the video. I'll just show you 3 years ago. There we go. There, how I'd start learning machine learning again. So there's a video. If you haven't seen that one, go check it out. It's got some helpful tips. Uh, if you're getting started into machine learning, I spent a lot of time studying in my bedroom. So it's just basically all the information that I've gathered from learning machine learning over the last 3 years in a— or how I would do it again. So the things that I've learned that I would improve on. Also another one here. So this is not my, my own work, but you could sort of say that it is. It's from Siobhan. He sent it to me. Now this is what I love, love, love to see. Like seriously. So Siobhan, what he did, he actually sent me an email saying, do you mind if I do a data science project or a data analytics project on your YouTube data? And I'm like, sure, go for it. That'd be amazing. And he actually, he didn't ask for permission. He was more like, I've already done this. Here's what I've done. So Siobhan, big shout out to you. And this is his first data project, Exploring Daniel Bourke's YouTube Channel Part 1. And so a whole bunch of different data analytics here using the, I believe it's the YouTube API. Yeah, there we go. YouTube Data API. Can you see that? We'll zoom in a bit. So a lot of cool things that you can pull from that. I even learned a few things about my own YouTube channel here. So go check this out. Give Siobhan some, uh, some support. There is nothing more than I love seeing than someone getting after their own thing, whatever project it might be. And so this is a little bit of tip for, for you. If you're working on your own stuff, you might look at something like this or you might look at anyone else's work and you might think, wow, there's no way I could do that. But what you have to realize is that everyone starts learning whatever it is at their own point in time. Everyone has to be a beginner at some point and you might be thinking, oh, I'm not going to share this because it's not that good or whatever, right? You need to get that mentality out of your mind because I know what it feels like. I had to tell myself that, you know what, I'm just going to create this regardless of what other people think because I think it would be helpful. If you're ever stuck, create something that would help your past self. So what would you have liked to have known 6 months ago? 6 months ago? That's the type of stuff that I, I like to create. So again, big shout out to Siobhan. Make sure you check it out and show him some love. And if you have something you want to share with me, something that you've worked on, please send me an email. It's not hard to find. It's on my website. I'd love to share it in a future video/article of Machine Learning Monthly. Anyway, Let's get into the best from the internet. So in the article, if you want to read along, I've got a little what it is slash why it matters. We're not going to read them out verbatim in these videos. I'm just going to sort of talk about it. So that way you sort of, it's not just repeating what exactly is in the article. Now, advanced NLP with spaCy. So if you're not sure what spaCy is, let me just show you. SpaCy, spaCy.io. So spaCy is an industrial-strength natural language processing library built in Python. The actually underlying code is Cython, so C Python. It's really, really fast. When I was working at the machine learning company I was working for, we, we tended to sort of start with spaCy using— so it's used by a whole bunch of different companies for natural natural language processing task, we would start with spaCy as a, as a way to tokenize text. Tokenize text is take some natural language like this and turn it into, turn it into numbers. So spaCy has had a massive amount of upgrades over the past few years. And now if you want to learn how to do NLP with it, there's a course here. It's completely free. From the founder or one of the founders of spaCy. So chapter 1, you'll find words, phrases, names, and concepts. You'll learn how to do large-scale data analysis with spaCy, processing pipelines, training a neural network model. And it's by Inés, I believe her name is. Yeah, Inés. Apologies if I'm pronouncing that wrong, who is one of the core developers of spaCy and co-founder of Explosion. One of my favorite AI companies, actually, Explosion.ai. Make sure you check them out. I take inspiration from, from this company. So if you want to learn NLP, I would highly suggest at least knowing some spaCy. If you're, if you're doing NLP in Python, I would say spaCy is a requirement. So if you want to get familiar with it, check out this free course. That's number 1. Number 2, speaking of natural language processing, this is a beautiful article that I came across, which is it compares essentially 3 different ways of, uh, so NLP, natural language processing, the first major step in any NLP project is converting text into numbers. So often referred to as tokenization. And there's a few different ways that you can do it. So if we zoom in here. So this article is brilliant. Text classification with NLP, TF-IDF versus Word2Vec versus BERT. So TF-IDF is term frequency, TF-IDF, inverse document frequency. I believe what it does, it What does it do? Where's the actual formula? So basically it takes into account the count, how many times a word appears in a set of text. So suppose we use the word suppose, it's going to count how many times the word suppose appears in here. And then I don't know the exact formula off by heart. I usually just call the function. It figures out how the ratio of that, that term across the entire document corpus. So if we had 100 Wikipedia articles, How many times does suppose appear in each article? And then how many times does appear across all of those articles? I may be butchering that, but that's why I'm linking this article for you. Word2Vec creates a word embedding of words, so it turns each word into a vector. And then BERT also creates a word embedding, but it's usually now these days referred to more so as a language model. So if you want to learn how each of these different ways to tokenize text work, I would definitely read through this article in full if you want to. See, the reason I love this article is because it hits, it hits a number of, of different points. Not only does it talk about the theory behind something, it shows you coded examples. So they're my favorite, favorite resources is not only Okay, here's the concept. Here's how you actually do it with code. So getting hands-on with it. Let's check out that article. We're sticking with the theme of NLP for number 3, Super Efficient NLP Projection: Advancing NLP with Efficient Projection-Based Model Architecture. So this is from Google. They've just— last year they published a neural architecture called Now, if I believe we go to here, it's going to take us to the maybe the, the paper. There we go. Projection, projection attention networks for document classification on device. So this is the keyword here on device. Whenever you see this, it usually means that whatever machine learning model they're talking about has some sort of computation constraints. And what I mean by that is if you imagine a smartphone, its processor isn't as powerful as, say, you had a data center and a full-blown deep learning computer setup. So although you, you might train a model on your deep learning computer or data center, if your end goal is to get that to work on a mobile phone, like on someone's native device, so they don't need an internet connection, the model just lives on the phone's storage. It might not necessarily perform as well as it does on the Deep Learning PC on the phone. Because one has far more compute power than the other. And so the, the graph you really want to have a look at here is this one here, the model comparison. So I believe they tested it on Civil Comments dataset, which is a classification dataset, um, trying to classify— Civil Comments dataset provides primarily 7 labels, so 7 labels, so multi-class classification. And if we go back, the results were the area under the curve for this new model, which is called PQRNN. So the beautiful thing about this as well is that BERT, which is a transformer model, very popular in the NLP world over the past— I think it was released in 2018— um, BERT uses a transformer architecture, whereas the cool thing is that PQRNN uses an R— a modified RNN. It's a recurrent neural network. And the main figures here you want to pay attention to is if you look at the model size on the log scale, BERT has 440 million parameters. You can just think of parameters as, as patterns or numerical patterns that a model has learned, whereas PQRNN has 1.3 million. And if we look here at the model size PQRNN is over, just over 1 megabyte, and BERT is getting close to a full gigabyte. Now a little quick story here is we were working, uh, on a machine learning project, um, to— sorry, an NLP project to, to classify insurance claims. So whether or not it was a car accident and whether or not the person submitting the claim was at fault or not at fault based on the text they said of the description. So say for example, I backed my car out into a pole in the car park. In that case, you would want the machine learning model that we were training to classify that claim as being— it was that person's fault. So they have to be charged the excess amount. And we tried to use BERT, and BERT was getting great results for, for classifying the text. But the problem was when we tried to deploy it in practice, it was far too large a model. So it was, it was almost defeating the purpose, even though in in like our Jupyter Notebook that we were, we were getting these amazing results, something like 98% accuracy. When we tried to deploy it, we found that it was taking far too long to process these claims. And if you imagine, when I was processing the claims, I'd do it on just 1 or a dozen examples at a time. But this insurance company that we were working with had to process upwards of 3,000 per day. And so if they're waiting for them to come through, Well, then it's, it's not a great experience overall if BERT is taking forever to process these. So we ended up going with a smaller model that performed almost just as well, but it worked far better in practice because it was a small model. And now if this PQRNN is performing again almost as well as BERT, but is something like 300 times smaller. Well, then that's pretty good, especially when you want to work on device. So check out that article for some pretty cool advancements in NLP for on device. Now, this is a really cool one. Machine Learning from Scratch book. As I said, nothing impresses me more than seeing someone share their work and what I believe it's by D.A. Friedman. I'm not sure what. Their first name is— where is it? I can't find it, but shout out to D.A. Friedman '97. Is that essentially I've done a little bit of background research on what this book— essentially this person's a student studying machine learning and they decided to put all of their, all of their knowledge, the things that they'd learned into a book. And the book, the beautiful thing is I believe it is made with Jupyter Book. Yeah, powered by Jupyter Book. So you can write a, you can write a whole book using Jupyter Notebooks. How cool is that? And the thing is, another reason why I love this book is because it's got, not only does it have some really popular and useful machine learning algorithms here, it's got 3 parts to it. Concept. So it shows you here the model structure. This has got all the math. All the theory behind the model, and then it has construction. So how would you write it in code? And then finally, so like from scratch, like this is coding ordinary linear regression from scratch in Python, and then an implementation. So scikit-learn of course has the linear regression algorithm implemented. So it goes through the, from the very foundations using pure math and then writing the code of how to build that model with Python and then implementing it with, with an existing library. So this is a phenomenal resource. And I believe— there we go, Danny Friedman. So big shout out to Danny Friedman. Make sure you check out the ML from Scratch book. I definitely— I'd actually bookmark this page. What you could do is if you're learning machine learning is spend 25 minutes or half an hour a day just reading through here. And when you get to the code parts, go through the whole thing and write out the code by yourself. So just write out, just copy this. Don't copy and paste it. Write it out by yourself. That would be, that would be a really cool 6-week project is to just reimplement this entire book. Seriously, copy it. And if you want the framework for that, go to mrdburke.com/bookframework. 42 days. This is what a 6-week project is. Spend a week on each of these. Where is it? Where's the book? Spend a week on each of these and just reimplement them. Give that a try. So yeah, big shout out to Danny Friedman. Inspiring work, my friend. Check out the Machine Learning from Scratch book. It's definitely on my, my list of top machine learning books now. We come back. Number 5, speaking of learning the math for machine learning, this one, this one wasn't published in September 2020. So we're going against the trend here. But that's what I said. Again, I like to share content that is evergreen. So things, of course, latest state-of-the-art research. But when it comes to fundamentals, some of the things that are 10 years old are still well and truly relevant today. So a lot of people email me asking, Daniel, how do I learn the math for machine learning? And the truth, like, there's no like set real way to learn it. One of my favorites is, is when you run into a roadblock and you're writing some code. So I'm code first. I like to see things being run. A Jupyter Notebook is like basically built for me because it's, it's got the, it's got the motto of the data science practitioner built into it. If in doubt, run the code, see what it does. If it doesn't work, try something else. I really adhere to that framework. But if you want to learn the math that's going on behind the scenes, behind all of that code that you're running, I would highly suggest reading this article. So it talks about— I love some of the points in here. A note on math anxiety. It turns out that a lot of people, including engineers, are scared of math. To begin, I want to address the myth of being good at math. So, if we come in, zoom right in so you can see this. The truth is people who are good at math have lots of practice doing math. Shock! As a result, they're comfortable being stuck while doing math. A student's mindset, as opposed to innate ability, is the primary predictor of one's ability to learn math, as shown by recent studies. And of course, you can check out those studies if you want. But that's— it's all about the story you tell yourself. See, a lot of the time, the people asking me, Daniel Bourke, how do I learn math and machine learning, the answer they're really looking for is that if you want to learn something, you can. If you spend enough time putting in the practice, testing yourself, going over concepts, you'll start to figure it out. Imagine when you were 3 years old and you could barely walk, you could barely talk. What did you do? You just kept practicing. And now I'm talking without even thinking here. We go for a walk down the street and I don't even have to think about moving my legs. So math, treat it like another language, just as you were like learning to code or learning to talk. At the beginning, you're probably terrible at it, right? And even now I still say some things out loud and they don't really make sense. And same with code. I'll look at a body of code and be like, I don't know what's happened there, but with enough practice and going over it, you start to build that confidence with going forward or running into a problem and getting stuck. Well, you, you know, hey, at least I've got this experience here. I've come up against one of these problems before. I can figure it out. Um, and there's another one I think somewhere down here. I really like these 4 questions. So define your system. What are the inputs, outputs of your system? How should you prepare? Not exactly. This is not exactly related to math and machine learning, but how should you prepare your data to fit your system? How can you construct features or curate your data to help your model generalize? How do you define a reasonable objective for your problem? So this is amazing. Learning math as you need it. That's my favorite approach. Anyway, go check out the article. Highly recommend. If you want to figure out, uh, or have some sort of idea of how you should approach math for machine learning. Number 6, speaking of beautifully written articles, this is another one that I think is going to be evergreen. So, unpopular opinion: data scientists should be more end-to-end. So, where— let's find the definition of end-to-end so we know what we're talking about here. So we have some different definitions here of a generalist, focus on roles, product manager, business analyst, data engineer, data scientist, machine learning engineer, full stack, in a unicorn. Um, there we go. An end-to-end data scientist can identify and solve problems with data to deliver value. Oh, I'm biased. I, I really do like this approach, so I agree with this, this opinion. To achieve the goal, they'll wear as many or as little hats as required. So this is the main, the main one here in bold, identify and solve problems to deliver value. And what the article goes through is it talks about some different points here. There's, there's a, there's a big one here. Yeah, here we go. For some data scientists, it can lead to increased motivation and job satisfaction. So what end-to-end means is imagine you've got a machine learning project and you need a way to collect data, you need a way to model data, and you need a way to deliver the, the model/service that you've built based off those data. You have 3 points: data collection, data modeling, data model deployment. And so what the article is arguing is that you should have knowledge of, of, of at least all of them, essentially, if you want to be a practitioner. And I agree with that. And I also There's— the article talks about the other side, right? The other approach is just to be an absolute expert at collecting data and storing it in an efficient way, efficient way. And then you might be an absolute expert if you've got data that already exists and you just know how to model it really, really well. That would be something like a Kaggle competition expert or something. A dataset already exists. You're just really good at modeling. And then finally, Um, there's arguments for just being really good at deploying a service, so that's more the, the DevOps side of things. And now what being end-to-end offers you is autonomy, mastery, and purpose. Again, these are points from the article. So by being able to solve problems independently— so this is if you had end-to-end knowledge— instead of waiting and depending on others, end-to-end data scientists are able to identify and define the problem build their own data pipelines and deploy and validate a solution. Mastery: in the problem-solution-outcome from end to end, they can also pick up the domain and tech as required. So again, learning the things you need to learn as you go. And purpose: by being deeply involved in the entire process, they have a more direct connection with the work and outcomes, leading to an increased sense of purpose. So I remember another story. From the fields is that I was probably one of the experts who could do the modeling part. So once the data was there, and I think this is where a lot of people start actually is if you're learning machine learning, usually start with an existing dataset, you build a model and then that's, you just keep doing that over and over and over again. Where you sort of miss out is the collecting data phase and the deployment phase. So in another project, similar to the NLP one that I talked about, is that I built this incredible model, but if I wanted to deploy it, I had to wait for another person on my team who might have been working on another project to be able to do it. Because we had— I was a machine learning engineer, but I was probably more so a— how would you say— a machine learning modeler. And then we would have a DevOps engineer who would deploy our machine learning models into practice so that we could show the clients what we built. And so I remember we had a few projects on the go, I built this amazing model, but then I had to wait a couple of days to get it deployed and show, show the client a proof of concept, because the DevOps engineer was working on another project. And so if I had more end-to-end skills, which I kind of do now after more practice, I I would have been able to not only build the model and then deploy it myself and not have to wait for that buffer of waiting for someone else. So basically being end-to-end helps you to remove roadblocks. Now, if you want specialists, yeah, of course, if you want to build something like GPT-3, you want to be really specialized in, say, a language model or something like that, The best way— I love this— is to pick it up via learning by doing. If you ask most software engineers or machine learning engineers that I've met, of course they've studied the right books and courses, but they've also done their own projects end to end. So I'm going to leave that here. I highly recommend reading this section. Read the article in its entirety. It's only about a 10-minute read. It'll be well worth your time. So yeah, great little article there. Big shout out to Eugene Yan. And so Eugene has written this with experiment— experience as being an applied data scientist at Amazon, at Alibaba. Look at that. There we go. Great article. That's something that's got— I think that's, that's an evergreen article. So you could read that in a year's time and it'd still be valuable. Number 7, using GitHub Actions for MLOps. Speaking of building an end-to-end machine learning system, GitHub Actions is probably a tool you might want to look at, especially if you're hosting your code on GitHub. What GitHub Actions allows you to do is— where's a diagram? Here we go. Say you updated you did a pull request to your GitHub repo. So you uploaded some code to GitHub. What you could do is add a comment here and then it automatically triggers GitHub Actions to do some sort of ML workflow. And in this case, it's testing the new model. So model statistics are dropped back into the PR with links back to the experiment tracking system. So that's really cool. So say for example, you updated your model code and you're like, oh, I changed a few parameters. On this new model. I'm trying it out. I'm going to upload it to GitHub, push it to the model.py file. GitHub Actions automatically watches that model.py file. And then as, as that model.py file gets triggered, say you add this, this comment here. I'm sure you can, you can adjust these steps as needed. If you want to test that new model.py file, type in the comment run full test. It kicks off some, some tests on your new model and then it drops into the PR scores of that model and whether it's better or not. And so if it is better, if it runs faster, if it's got a higher validation accuracy, well then you might want to merge that new model.py into the, um, into the master branch or something like that. So really cool tool. Check out GitHub Actions. I haven't personally used it enough myself, but Hamal Hussain is a machine learning engineer at GitHub, writes some great articles. Big shout out to Hamal. Here we go. Number 8, deep speed. Again, this is probably actually, yeah, this does help the, the, even if you're at a, so this one is, man, I'm sweating. That's how hot it is in Australia and that's how excited I am to go through all these amazing resources, working up a sweat. That's how you really know that we're getting things, we're going through We're having fun. So when you're working up a sweat, there aren't many times that you work up a sweat that you're not having fun. So this is from the Microsoft Research blog. So DeepSpeed, it's a framework for extreme-scale model training for everyone. The main 4 points are here. So trillion-parameter models. So if you imagine GPT-3, which is probably the best language model that's ever been built so far was 175 billion parameters. So this is saying that we're now, we're now making it— Microsoft basically— we're now making it possible for trillion parameters, an order of magnitude more than GPT-3. We're now making it possible to, to start training those type of models, which will be absolutely crazy. Um, 10 times bigger model training on a single GPU with zero offload. So That's, that's another really big point here is that say you're someone like me, you don't have access to massive amounts of cloud compute. I do, but it would, would cost a lot. I have a deep learning box over there with a single GPU on it. So if I wanted to train a big model on it now with DeepSpeed, DeepSpeed hooks into PyTorch, by the way, I could train a 10 times bigger model and I believe it's, yeah, models of up to 13 billion parameters without running out of memory. So that's massive. Powering 10 times longer sequences and 6 times faster execution through DeepSpeed sparse attention. So if you imagine you have a natural language processing model, you can process up to 10 times longer sequences. So you say you had a sentence of 100 words, you could now process 1,000 words. with up to 6 times faster. So that's, that's pretty insane. And then 1-bit Adam. So Adam is an optimizer. So if we go tf.keras.optimizers.Adam, Adam is very popular. I find Adam just works for almost any neural network. Put a, put a comment below what your favorite optimizer is. Mine's, mine's Adam. But so essentially what this does is it one-bit Atom, if you're using the Atom Optimizer, makes it more efficient so you can get 5 times communication volume reduction. So just, just some massive things. Check out this article. It's actually pretty in-depth. Um, I read through it and kind of, there's a lot of things in there that I didn't understand, but if you're if you're— especially if you're, you're wanting faster compute on a single GPU, or if you're working on really large-scale models with PyTorch, you probably want to check out this DeepSpeed article. So shout out to the Microsoft Research team. This is another one, number 9, The Ultimate Guide to Object Detection from the team at RoboFlow. I actually really like what RoboFlow are doing. So this is, this is a great example of how I would build an AI startup is, so their product is basically, I don't wanna say basically because that sounds like I'm simplifying it. Let's just go to product overview. RoboFlow, so it helps you preprocess your image data. Yeah, transform raw images into trained computer vision model in minutes. So it helps you, not only store your images, preprocess them, it now helps you train computer vision models. So I think their tagline was making computer vision accessible for anyone. Yeah, we help innovators like you apply computer vision. And so what this guide is, is this is the link that I was sharing in Machine Learning Monthly, is an Ultimate Guide to Object Detection, October 2020. So look how fresh this is. This is fresh off the press. This is what you get here with Machine Learning Month. You get the freshest stuff. So this is going to be— if you're— if you've done image classification before, assigning a label to an image, object detection is a little bit different because not only does it have to recognize the image, it has to figure out things like the edges of the boxes where that boat is. So sure, you could take image classification, this image, and say it's a photo of, of water, it's a photo of forest, but where, where are the boats or something like that? That's what you're trying to figure out with object detection. And so basically it turns into a regression problem, regression, because you're predicting a number, you're predicting the point of where, where is it? Somewhere in there. There we go. Anchor boxes. That's what it's called. So you're trying to predict each of these corners around not only where, where they are in the image, but also what the image is of, but where the corners should appear. So check this out. It's got a whole bunch of different things that you can do for object detection, a whole bunch of different models for object detection and how to train them and how to prepare your data, how to label data. It's just amazing. And so the case study here for RoboFlow is not only are they building an amazing AI tool, they're teaching people how to use it. And not only just how to use the RoboFlow tool, but how object detection works. So that's what I would do if I was building an AI startup. Not only would I build a product, I would also educate people on how to use it. So shout out to RoboFlow, amazing product, and thank you so much for your incredible guides. Speaking of, AI in the wild. How about AI in agriculture? So this is another phenomenal article. Just sometimes it's hard to understand, like, where— if you're learning these things, like, say, for example, in your bedroom, studying in a Jupyter Notebook, it's hard to understand, like, where, where the code that you're writing can be used. And so this is a great example of how computer vision models are being used to detect weeds. So if you imagine you've got this tractor going across this whole field, and I'm sure this, this field is not— sorry, this photo does not show probably even a quarter or a tenth of the field that this tractor has to drive over. Now what happens is, as it goes over, it's got cameras on there. I'm not sure what you'd call it, but let's just say cameras on the tractor. And instead of spraying the actual crops with pesticide, it sprays only the weeds using the computer vision. So, the benefit of this is twofold. Not only do people like us, the consumers, we— I don't really want pesticides on my food or on my cotton, on the t-shirts that I'm buying. So, the twofold here is that the pesticide doesn't go on the actual plant, It only goes on the weed. So again, that's saving us as end users from not having pesticides on our, on our food as well as in our clothes. But it also uses less pesticide because instead of just spraying the whole field and just going gung-ho with, with pesticide, it's really targeted on where it needs to go. And so I can't remember the exact Oh yeah. And then this article, by the way, talks about how they set up the pipeline to train the model, what sort of tools they're using, Weights and Biases, PyTorch. Shout out to Weights and Biases. Amazing tool. Check that one out. I'm not sure there was somewhere in here where they said how much they saved. Reproducible models. Yeah, this is just a great case study on how machine learning can be used in practice. So if you're interested in not only just AI in agriculture, but just how, how you go about implementing a machine learning model into, into the real world, read this case study of an article and just copy it. Just, just build your own system that's like this. But yeah, there's somewhere in there of how much they improved. I'm not sure how much they saved, but it was a very significant amount saved on pesticides. Sides every, every year or every sort of timeframe because they weren't spraying the whole field. They were only targeting just the weeds. Whoo! Well, looks like we're done. So again, as always, September was another massive month for machine learning. I hope you like some of the resources there. I'm not sure what my favorite one was because I mean, they're all so so great. I think, I think it was the end-to-end article just because it's a whole bunch of confirmation bias. I really like that mentality of just being able to— someone who goes and looks at a project and you know what, I can— there's a problem here and I'm going to try and build the whole solution myself. Just give it a go. So that's probably my favorite one. Leave a comment below on what your favorite one was. Um, if you'd like to, Actually, I'll just leave it, leave myself on here. If you'd like to receive that article in your inbox, it comes out about the first of every month. Make sure you go to the link in the top of the description and sign up there. Completely free. Just some really cool resources of machine learning. But otherwise, if you have any resources that you'd like to see in a future article slash video because we're doing articles and videos now, feel free to either leave a comment or send it to me in an email or something like that. But otherwise, as always, keep machine learning, keep creating, and I'll see you next time. Peace.