Forget models, build a data flywheel | Machine Learning Monthly January 2021

5, 4, 3, 2, 1. Hello friends. It's that time of month again. Oh, I'm very, very, very excited. If you're new here, this is Machine Learning Monthly and it is for January 2021. So it's a previous month. The show where we go through the latest and greatest, but if you've watched this before, you know it's not always the latest because imagine a timeline. We've got the beginning of time here and the end of time here. Some of the things are right close to the end and some of them are a little further back, but that's okay. They're still amazing nonetheless. And so every month I write an article. Let's have a look at it. It's called Machine Learning Monthly. There's my face in a really formal profile picture with a blurred background and whatnot. But this just came out. If you'd like to get it, or if you'd like to just read this and not listen to me talk to some amazing machine learning things, you can subscribe to that, or the link will be below. And you can get this, you can just read it on the zeromastery.io blog page. Now, I usually start off by telling you what I'm working on, but I've kind of done that the last few episodes. And it's a TensorFlow deep learning course. But before we get into that, I have something really important to announce. And that is FreeCodeCamp are building, I tweeted this out, I need to find the link. FreeCodeCamp, where is it? Um, here we go. There should be the link here somewhere. Boom! FreeCodeCamp are building a data science curriculum with advanced mathematics and machine learning. This is so, so, so amazing. If you've ever used FreeCodeCamp, I have. I learned them to— I basically use FreeCodeCamp to learn to code and they're building a full-blown— look at all this. What will the data science curriculum cover? Can you see that? I'm gonna zoom in. It will cover scientific computing with Python, foundational mathematics, algorithm. Amazing, amazing. FreeCodeCamp build world-class resources and it will all be for free. However, they need your help. They need some donations. So they need $300,000 over the first 2 years, but they're launching a $150,000 fundraiser because this amazing person, Darryl Silver, is going to match everything up to $150,000 so Free Code Camp can make $300,000. Uh, it's now at $28,690. There's 26 days remaining. I personally donated, um, where was it? $777 of my own cash monies. That was about how much I made off YouTube last month. So all going to FreeCodeCamp's curriculum. I'm very excited for this. I'm gonna be pushing it. So make sure you check that out. That's more important than almost anything you're gonna read in here today. Because FreeCodeCamp build amazing resources for people all over the world and make it free. So let's get into it. I'm working on a TensorFlow course. The thing is with courses, right, and I can understand why FreeCodeCamp need about 2 years, Because this is the TensorFlow course I'm working on. There hasn't been a status update in a few days because I'm working on something else that you're probably going to see soon. So it's like a little bit of an extension of a course. Of course I signed up to, to do two more things when I'm working on one thing. Should always work on less but better. You can check out the GitHub repo that's going to come out very, very soon. I've recorded about 100 20 or so videos. It's probably the biggest project I've ever worked on. But the thing with courses is that you need— you make things 3 times, right? You need to learn it yourself, then you need to make the materials, and then you need to teach them. So at minimum, everything gets done 3 times. But enough about me. From the machine learning community. So this is amazing from Nicholas. Forgive me, Nicholas. I can't pronounce your last name, but I'm going to give it a shot. Muehnhoff? Muehnhoff? So in May 2020, Facebook AI kicked off the Hateful Memes Challenge. Let's have a look at this. Hateful Memes Challenge winners. Beautiful. So what the competition was, was to detect hateful memes. But the beautiful thing about this competition, and this is kind of a theme, as we said at the beginning, I think, I think, but I wanted to say this at the beginning. The theme of this episode, one of them was combining multiple sources of data. So visuals and text. So this is what you had to do for this competition was to take in, say, a meme, read the text off the meme and the image and detect whether it was hateful or not because there's too much. We don't want hateful speech online. But remember, there's, with hate speech, there's a difference between hate speech and speech that you hate. Big difference there. But look at this, 3,300 participants from around the world and Nicholas finished second. Congratulations Nicholas. That is an amazing effort. Um, it's a $100,000 competition, so I hope you're in the money there. Um, and Nicholas shared all of his solution on GitHub. So if you want to see, he called it Viliyo. And can you see that? State-of-the-art vision language models in PyTorch and PaddlePaddle. Actually, I haven't heard of PaddlePaddle, but I'm assuming it's some sort of deep learning framework. And the beautiful thing, Nicholas also created his own self-driven learning, self-learning AI in a year. How amazing is this? So there's a great list from Daniel Bourke on the top resources from creating. That's me, by the way. Your own artificial intelligence master's degree. The idea is simple. Do some great courses for free online, like FreeCodeCamp's upcoming course. Please get behind that. That's, that's very close to my heart. Do an independent project, and it all costs about $0. So Nicholas got involved with AI about 8 months ago. Look at this. And this is what he did. Coursera AI for Everyone, Harvard CS50, Udacity Git and Shell. Very important. That's another theme of this Git and Shell. Another theme of this episode of Machine Learning Monthly is MLOps, so machine learning tooling, because machine learning is— machine learning engineering is like 90% engineering and 10% ML. So you better learn some tools to do with machine learning if you want to get into engineering. Udacity Python, Harvard Udacity Deep Learning, Coursera Project Hateful Means Competition, But the beautiful thing is he spent the most time, Nicholas spent the most time on the project. That's what I stress the most is that you will learn courses are great for building foundational knowledge. However, you cannot replace the knowledge you gain from working on things on your own. So massive effort, Nicholas. Thank you so much for sharing that. Uh, I really appreciate your submission. Um, so go check out Nicholas' work and then another one from the community. Aurelien Pedens, I hope I'm saying your name correctly here. I believe you have the same name as one of my favorite books on machine learning. Aurelien Garonne, you can't really see that, but this is Hands-On Machine Learning authored by Aurelien Garonne. I hope I— whoa, come back to me. I'm going to put that book on the ground. Treat your books nicer than I just did. And so my mouse has disappeared. I'll be back in one moment. My trackpad is dead. That's the blessing and curse of wireless. And we're back. Trackpad is plugged in. But Aurélien Peden's Deep Thoughts blog. So as I said here, there's nothing more than I love seeing someone sharing their work and interpretations of the world. Aurélien is a third-year computer science student from France writing about machine learning. Let's go to Aurélien's blog. This is one of the best things that you can ever do, in my opinion, of course. But I believe if you start a blog, you've got your own little corner of the internet. That's where you can share your work, you can send people to look at it, they can figure out who you are, see what you've done. I highly, highly recommend starting your own blog. And that's what Aurelien Pedenne has done as a third-year software engineering student at Ensi, whose passion for programming started at the age of 10. Amazing. I started learning about machine learning and deep learning 2 years ago. So what Aurelion is doing is here you will find tutorials, course notes, and personal thoughts. I really liked this article on a deep dive into residual networks, which we will see later throughout this episode. But look at this, right? This is the sort of stuff that you can work on. So big shout out to Aurelion. Thank you for submitting your work to Machine Learning Monthly. If you would like to submit your work to Machine Learning Monthly, send me an email, daniel@mrdbourke.com with the subject Machine Learning Monthly Submission. And who knows, might be in a future video or a future article. But let's get into the internet. So last month was all about OpenAI. There was some massive plays from OpenAI. I think last episode, well, I don't think I know. Last episode we looked at DeepMind. Now we're looking at OpenAI. So there were two big ones out of there. I might zoom in a little bit more just so it's easier to see. Beautiful. DALL-E creating images from text. So this was a model which is able to create images from text, e.g., a small— a snail made of a harp creates an image of a snail with a harp as a shell. Yes, seriously, this happens. And then there was CLIP, connecting text and images, a neural network which learns visual features from natural language supervision. For example, learning about the photo features in a photo of a dog via its caption, a photo of a dog. I told you there was a trend in this episode, vision and text. Let's have a look at DALL-E. This is, this is one you're just going to have to check out for yourself, right? I believe one of these— DALL-E doesn't have a paper out yet as I'm recording this. It might by the time you watch this video. There we go. Creating images from text. DALL-E. Let's zoom in. DALL-E is a 12 billion parameter version of GPT-3 trained to generate images from text descriptions using a dataset of text image pairs. Now, I did read this article and I couldn't quite find what they meant by a dataset of text-image pairs. Was it— they said they're freely available on the internet, but that's kind of broad. I mean, there are a lot of images and text pairs on the internet. Like, are they— I know OpenAI are partnered with Microsoft now. Are they getting them from Bing? Doesn't matter. But let's see. Oh, and by the way, DALL-E comes from the combination of Salvador Dalí, an amazing artist, with Pixar's WALL-E. So I really like that. So where are some of my favorites? Let's go down here. And I really liked— so all of these are incredible. But if we come down here, I liked— you just have to look at these images. Here's one. So what it takes is a text prompt and an image prompt, and then it tries to fill out the image. So you see here we've got the first half of the image is a kettle or a teapot, sorry. And then it's blank. So what DALL-E will try to do is fill out the rest of the image. So if it comes in with a text prompt, it goes the exact same teapot on the top. So ideally you want this teapot with GPT written on the bottom. Okay, what comes out? Look at that. That is unreal. This is the start and this is the image we ended up with. That is unreal. And then let's change this teapot. This is wild. This is absolutely wild. Uh, and I believe, what is it? It's a transformer. Here we go. Summary of the approach and prior work. DALL·E is a simple decoder-only transformer that receives both the text and the image as a single stream of 1280 tokens. So that means it takes that text prompt and it takes the image and it converts it into numbers or tokens. And so it has 256 for the text. So that, that stream of text. So if we come here, this text prompt gets turned into 256 tokens and then 1024 for the image, right? And so it takes this image prompt as 1024 tokens. Does it? Or maybe that's during training. Hmm. Anyway, the attention mask at all of its 64 self-attention layers. So this is a big theme here throughout this. This is actually 2, 3 themes. We got vision and image, Transformers in computer vision just taking over. And machine learning operations. But I think that's enough for DALL-E. Go and have a look at that. Just, just wow yourself with what it does. I spent a while just going through all of these different images. Where's the one of the room? Oh, there we go. Here's a snail. A snail made of a harp. A snail made of the texture of a harp. What is this? This is incredible. Look at that. That snail is a golden snail made of a harp just by saying with text. Far out. Let's go back. So now we're into Clip. Now I wonder— oh, there's another one of my favorites. The text prompt passed to a model. Yeah, DALL-E, just draw me a living room with two white armchairs and paint and a painting of the Colosseum. The painting is mounted above a modern fireplace, by the way. So this is the future of interior design. It's just let's get a neural network to design the room room I'd want. You know what, I'd like a beautiful painting of flamingos up here. Um, so Darlene, if you could get on that please, I would like that. Anyway, let's get into CLIP, which was another— so the big month for OpenAI. And of course it's not just this month, all this research is, um, probably taken a while to come out. Contrastive Language-Image Pre-trained aims to tackle the main problems we face with computer vision today. Costly datasets. So if you're labeling a large-scale image dataset, that requires a lot of work. I mean, ImageNet took 20,000, 25,000 annotators. That's not really that scalable in the long term. Narrowness. That's another big problem in computer vision. An ImageNet model works really well on the 1,000 ImageNet classes, or however many there are. But show it something out of those 1,000 classes and what happens? Actually, I have an amazing meme to show you for that. Here we go. This is what I'm talking about. Machine learning. This is machine learning in a nutshell. There's a baby. Daddy, right? Dad's butt sees some fruit at a shop. Daddy sees some boobies. Daddy and see some hot air balloons. Daddy, right? So that's the narrowness problem of machine learning. And then of course, Poor real-world performance. Deep models often report superhuman performance on benchmarks. However, the real world doesn't have benchmarks. So optimizing for a single benchmark is akin to a student optimizing to pass exams for knowledge instead of learning the principles they can apply after graduation. And so how does CLIP address these? So for costly datasets, CLIP trains on an image and text pairs readily available on the internet. Oh, okay. This is the, the OpenAI one where I was like, where does the data come from? So that's what I'd like to know, right? When they release these amazing papers, it's just, they say it's trained on this sort of data. It's like, yes, but where did this, how did you get this data? Like, this is just, is it just magical? Is this available to people like you and me outside of OpenAI or is this an OpenAI proprietary thing? Maybe the paper will reveal more. And so that's for the costly datasets. So that's, remember CLIP is with images and text. So if it's lacking sort of just images without, with labels, the text is part of the label of the image. And then for narrowness, because CLIP is trained on language-based descriptions of images as well as the images themselves. So remember, that's the important point, text description of an image and the image itself. So like a dog photo and the caption, a photo of a dog. It can be adapted to other tasks by telling it— this is, this is wild— telling it what to do. Instead of gathering more data for your avocado image classifier, just tell it, CLIP, give me an avocado classifier. Wild. For poor real-world performance, CLIP can be evaluated on several benchmarks, e.g., ImageNet, ObjectNet, ImageNet adversarial without being explicitly trained on those benchmarks, and it performs amazing. We'll check that out in a second. But that prevents it, because it's not been explicitly trained for those particular benchmarks, it prevents it from over-optimizing for one particular thing, which is something that plagues deep learning models at the moment. They optimize for one particular thing instead of being— that what's the ideal goal of deep learning is to be generalizable, to be applied to to things it— what they weren't really explicitly trained on to begin with. So let's go into the blog post. Connecting images, text and images. You really have to read these blog posts. They're absolutely phenomenal. Here we go. So this is a dataset, ImageNet. So there's a ResNet-101, CLIP, Vision. I think that's Vision Transformer. Large. So there, there, as we said, the theme Transformers taking over vision. ImageNet, it equals ImageNet ResNet-101. But where CLIP starts to pull away is that ImageNet V2. CLIP beats it here. ImageNet rendition still gets amazing results where ResNet-101 starts to fall away. ObjectNet, CLIP still getting over double. Right. And then ImageNet sketch, CLIP still doing really well. And then here's probably one of the most important is ImageNet adversarial, which is a dataset that explicitly tries to trick traditional computer vision models by putting confusing images in there. So this is an image of a banana, but it's cut up. So that's going to confuse the model. Look at CLIP, just 35 or whatever that is, that multiplier, 35x better performance. Than ImageNet ResNet-101. So make sure you check this out. We want to have a look. So this is, this is what happens. Contrastive pre-training learning. So here we go. We've got a photo of a dog and its caption that goes along with it. Text encoder, image encoder. We stack those tensors, create dataset classifier from label text, and we keep going there. So This is phenomenal. I'm a real big fan of the two things that came out of OpenAI this month. However, CLIP did take a substantial amount of compute. However, they did also figure out a way to improve. There we go. CLIP is 4 to 10 times more efficient at zero-shot ImageNet classification. They did— there was an experiment they ran. I don't quite know it off the top of my head, which made it more— there we go here. The second choice was the adoption of the Vision Transformer. This is what I'm saying. Turns out Transformers are more efficient than, than a standard ResNet, which uses convolutions. Wow. So check out that massive month on tour from OpenAI. There's the paper, there's the code. One more thing I really liked was the model card, which is kind of like a nutrition label for machine learning model, which I believe a fair few models are going to have to start coming out with as they get more and more powerful. Another one in the terms of— this is another theme of visuals and text. Does this paper include video? So Amit has built an amazing— this should be Papers with Video Chrome extension. Let's go have a look at this. It's very easy to install. Do I have it on mine? Why haven't I? Oh, I've installed it on my other Mac. Damn it. That's all right. I'll just explain to you what it does. A Chrome extension that adds video explanations to research papers on Archive. So if you install Amit's extension and you go to a paper, say Transformer XL, it'll automatically— the extension will show you whether or not that paper has video. Why is this important? Because whenever I've read a paper to begin with, a lot of the time it's like I'm reading something in a different language. And so I like to hear how someone explains it So I'm a big auditory listener. So I like to hear how someone, someone else interprets something and then explains it in their words and then try to see how that matches up with the words that I'm telling, or the explanations that I'm telling myself in my head. So it works for 3,700 papers right now. Very easy to install. I installed it in my, on my other computer. Really wish I had it here. But Amit also has a phenomenal developer blog that I'm a big fan of. So if you want to make sure, if you can sign up to— yeah, there we go. Sign up to Amit's research and engineering blog. It is absolutely phenomenal. Knowledge transfer and self-supervised learning, interactive analysis of sentence embeddings. Amazing. Amit is someone in the machine learning community who inspires me to do better work. So Thank you so much, Matt, for both your Papers with Video Chrome extension and the amazing blog posts that you put out. I believe I've mentioned them in a previous Machine Learning Monthly, but this is another shout out. Here we go. We're up to No More Bottlenecks: Transformers Take Over Vision. So a new paper just dropped, Bottleneck Transformers for Visual Recognition. Remember I said there's a massive theme going on right now. Previous readers and watchers of Machine Learning Monthly will know that we've been talking about Transformers taking over vision for a while now. And this PDF is taking a while to load. Anyway, there's a paper, you can read it. But essentially what they introduce is bottleneck Transformers. So what they did is they made a simple change to the popular ResNet backbone, as we talked about before. That's kind of what CLIP did. They replaced the convolutional layer with multi-headed attention. There's another beautiful paper, Attention Is All You Need. And this spurred off the, the transformer architecture. Attention Is All You Need. Hopefully this paper is loaded. Yes, here we go. This is what Bottleneck Transformer does. So this is where attention was introduced. So this is how this all connects. But this is what happens. The ResNet bottleneck, um, goes in here, and this is a convolutional layer. Can you see that? I'll try zoom right in. So original ResNet convolutional layer bottleneck transformer, the new architecture has multi-headed attention. So a ResNet bottleneck, right? And then on the right is a bottleneck transformer, BOT block. The only difference is the replacement of the spatial 3x3 convolutional layer with the multi-head self-attention layer. So this is this layer. That's the only difference. The structure of the self-attention layer is described in Figure 4. I'll let you read through that paper. But what's the important point here? Here we go. Finally, let's zoom in so we can see that. Finally, we present a simple adaptation of the Bottleneck design, Bottleneck Transformer, for image classification, resulting in models that achieve a strong performance of 86%. 84.7 top-1 accuracy on ImageNet benchmark while being up to 2.3 times faster in compute time when compared to EfficientNet models. EfficientNet models are supposed to be really efficient. It's built into the name. And here we go. Bottleneck Transformer. So this is why I've said, um, no more bottlenecks. Transformers are beginning to take over vision. I really need to learn about the transformer architecture. Architecture. Let me know if you'd like to see some videos describing Transformers and how they can be used for vision. So I need some things to work on after I release a course. Um, why this matters— again, two things benefit the deep learning world: better results and faster performance. If you can't get one, get the other. And here's— we've got a bonus in this, this year's, uh, or sorry, this month's episode, is that if you want to learn about Transformers, you're like, what's going on with Transformers? I want to learn about them. Well, The Full Stack Deep Learning team have released a Colab notebook that describes Transformers, how to build them yourself with PyTorch, more specifically PyTorch Lightning. So if you want to have a look at this, I like to learn things by code. I mean, I look at Transformers and I look at the math equations that I see in papers and I get scared. But when I code them out, I go, oh, okay, I see how the math and the code lines up. So check this out. Amazing, amazing resource. We'll get back to Full Stack Deep Learning in a moment. Now we're on to Made with ML's pivot. So Made with ML, if you remember a few months ago, used to collect different resources from around the internet. So it was kind of like this newsletter, but it was scalable because people all over the world could submit their work and have it upvoted by others. However, it recently pivoted, and it was previously one of my favorite websites for machine learning in the whole world, and now it's even more of my favorite websites for machine learning in the whole world. Now, Made with ML is focused on less but better. So first and foremost, teaching you the fundamentals of machine learning, see the ML Foundations on GitHub, and teaching how to apply what your machine learning knowledge, or how to apply, sorry, your machine learning knowledge to real-world problems. See AppliedML on GitHub. But let's have a look. Made with ML. It's a beautiful, beautiful website. Inspires me to do better work. goku-mohandas/madewithml. Look at this. Features: intuition first, hands-on engineering, comprehensive. So this is where we disagree. This is intuition first. We'll never jump straight into code. Instead, we'll develop an intuition for the concepts first. My style is kind of a bit more reckless. I like to do the code first and then develop the intuition after. But everyone learns differently, right? So ML Foundations. So this is the type of resources that you get in here. Learn the foundations of ML through intuitive explanations, clean code, and visuals. So Goku Mohandas, absolutely amazing person. I really, really, really, really respect what he's doing for the ML community. So there's basics if you want to learn notebooks, Python, NumPy, Pandas, PyTorch, modeling, linear regression, logistic regression, neural networks, data quality, deep learning. Oh, here we go. Attention and Transformers still to be done. See, this is what I mean. There's a lot of resource on this, but it seems like attention and transformings are taking over. Attention and transformings? Attention and Transformers. And then of course, once you've learned the foundations, you can learn how to apply them into a production-grade product to deliver value. So if we go into the GitHub for AppliedML, look at this. This is wild. Have I starred this? Yes, I have. Made with ML. So Goku is releasing— if we click on this objective. So this is what you start off. If you're going to start off with a machine learning project, you should always start off with an objective. What's your objective? What problem are you trying to solve? Releases a video. I don't know if I can show this in my own video, but just go and check this out, right? This is, this is phenomenal. I want to structure my future work like this. Look how beautiful this is. This is a work of art. So go have a look at that. And it all originated from the pivot. The problem, if you're not familiar with the old ML, Made with ML platform, it used to be like Product Hunt of ML. However, too many things lead to overwhelm. Less but better. Yeah, I'll let you read that. So let's go into— oh yeah, so follow Goku on Twitter for a treasure trove of machine learning content. So I'm a huge fan of the website Goku and learning a lot. Thank you very much. Speaking of production ML, Awesome Production Machine Learning. If you wanted everything, all the best tools and resources for productionizing your machine learning models in one place, check out the awesome there's a typo there, that should be an A. Production Machine Learning Repo by EthicalML. Now a warning, there's too many things here for you to go through. Best to work on whatever you're working on and then come back, come here, bookmark this page for machine learning tools, star it if you want, and then just search it. All right, search it for what you need. Learn what you need to learn when you need to learn it. Otherwise you risk overwhelming yourself. Follow your curiosity, and if you find you're stuck with something, come here to see if there's a resource to help you out. Now, the missing piece of full-stack ML. So that was, oh sorry, that was the Awesome Production Machine Learning repo. Amazing work by the people behind EthicalML, and everyone of course who contributed to that. The missing piece of the full-stack ML. Now this is, so this is by Josh Turban. Josh is an absolute beast in the machine learning world. Come in here. A researcher working at the intersection of machine learning and robotics. So he also co-organizes the Machine Full Stack Deep Learning program and has a PhD in computer science and was a research scientist at OpenAI for 3 years. Whoa. Um, so Josh, amazing person first and foremost. So previously at OpenAI, now working at a stealth startup. That sounds very Batman-esque. Did an amazing talk the other day. It's called The Missing Link in the ML Infrastructure Stack. If we go here, this kind of went out of the radar. I really liked this talk, but it's only got 695 views. So go pump that up. But this is the whole concept that it introduces is kind of the concept of the data flywheel. Now I've got this image here. But this is essentially what your machine learning products want to work towards, right? It's creating a data flywheel. If you're not sure what a flywheel is, it's like something that, it's like a perpetual motion device, keeps itself going. So what you want to do is collect data, clean and label it, train on it, test, deploy, monitor. And then in your system, your system actually collects more data as it goes. So if we have a look at this, Tesla do this really well. Tesla data pipeline images. Are we going to see some images on this? Nope. I know where it is. And Andrej Karpathy Autonomy Day. I found it. So this is from the Tesla Autonomy Day talk on YouTube. Go check that out. Hopefully I can show this video without getting wrecked by YouTube. But anyway, here's what happens. This is the data flywheel. So the data source is the cars. We find something that doesn't work. We want to improve that with other examples. So this is a tunnel. And if it doesn't work, we want to show it, show our model images of more like scenarios that are inside a tunnel. Then we label those, retrain it, redeploy. So that's the data flywheel. The whole objective is to find samples in your dataset and collect more where your model is performing poorly and then improve that model through more training. I want to build one of these. I'm going to build one of these this year. So yeah, go check out that talk by Josh Turban to— oh, and he— sorry, the whole talk is that throughout this entire process, the data flywheel, you need some way to evaluate how your model is going. So he proposes the idea of the evaluation store. So at all points you have a metric to evaluate how your model has gone from collecting to cleaning to training to test to deploy to monitor. That way you can find out where is, where's the bottleneck of your data flywheel. How, whatever stops the motion of that data flywheel, you should try to remove that roadblock. It's a very amazing talk. Speaking of Josh Tobin teaching Full Stack Deep Learning, the Full Stack Deep Learning, if you want to learn how to get your models to people, so build something to get your models in the hands of others, you should check out Full Stack Deep Learning. So the Spring 2021 version of it is starting. This at least looks the same as Made with ML. This looks really good. What's this made with? Material for MKDocs. It's very pretty. But look at this, week 1 fundamentals. So the notebooks and labs are out for that. CNNs, 3, RNNs, Transformers. If you know the architectures, you can probably skip the first 4 weeks. I believe that's what they say. But then we start to get into week 5 ML projects, infra and tooling, right? So what should you do for your machine learning infrastructure? What sort of tooling should you use? How can you troubleshoot it? What sort of data management should you do? What are your AI ethics? All right, if your model is predicting on certain things like, What do you have to be mindful of, of that model? How can you test your model? How can you deploy your model? Very, very important. And then Week 12, research. So where can you go? Research directions, teams, projects. Far out. This is amazing. The course is about to launch. You can enter your email to sign up. I've signed up. I'm going to be doing it. I believe it starts on— where's the time? It starts on March 1st. Put it that way. Yeah, there we go. ML Projects, March 1st. That's when it officially starts. But this is the first 4 weeks are like the foundations to get you ready. So check out that you can follow along for free online. So they're going to publish all their materials online. And then finally, let's finish off. Speaking of full-blown machine learning systems, this, this stuff, this I picked up this from Josh's talk. This is Apple's system. For delivering ML-powered features to over a billion devices. They are probably closer to more than 2 billion devices now. Apple published a paper in September 2019. So this is what I'm saying, right? It's describing their Overton system. Now I'll be upfront with you. I've read through the paper, but I haven't quite fully deduced what it does. However, I really like the approach that they take. So let's have a look at the— it's probably easier to take at the diagram, but this is what I was saying. September 2019. When we say Machine Learning Monthly is latest and greatest, sometimes things aren't the latest. This is almost 18 months old, but this is how Apple deliver machine learning to billions and billions of devices worldwide. So what does Overton try to do? Well, they want an Overton engineer to focus on fine-grained quality control. This is coming back from the evaluation store that we just talked about with Josh. Which subsets of the data are performing the most poorly? How can these be improved? And so this is really cool. Their system, Overton, helps engineers build machine learning-based applications without writing any code in frameworks like TensorFlow. Instead, they focus on this: support for multi-component pipelines. So which part of your machine learning pipeline is causing the most issues? How do you tackle that first? Because remember, what are we trying to create? We're trying to create that data flywheel. So you want to remove roadblocks as much as possible. Once you've got your model code ready, you can set that up in a script. And so what you focus on is, okay, my model's really good. I'm using, say, an EfficientNet B4 or something like that. It's really good at looking at images, we'll leave that in place. Now what I'm going to focus on is all of the pieces of the puzzle around that model that's stopping it from, from being better. That's how I understand it. And then finally, updating supervision, again dealing with data. So labeling data is typically performed by annotators, but this doesn't scale very well. How can you— so this is what the Overton system does. It, it has a high emphasis on weak supervision. Which is often, as they report, which gets very similar to results to like transfer learning on unlabeled data. And because Apple is so privacy-preserving, they want to increase programmatic supervision. So even if they get data from a user, no annotator is, is even looking at it. It's, it's, it's, uh, being labeled programmatically. Again, we're trying to introduce as many engineering features to the machine learning engineering pipeline as possible, rather than having someone just sitting there and label data. Overton paper is easily one of the most beautiful papers I've ever read, and it goes through the scenario of building a machine learning system to answer the question, how tall is the President of the United States? So if you imagine, that could be a question that someone asks Siri. How does Siri use machine learning to answer that question? And so this is probably one of my favourite figures from the Overton system. So what it starts with is supervision data in the form of a JSON and a schema, which includes like what model it should use and a whole bunch of other stuff. And then what it does, it passes it to Overton. Overton combines it in a supervision system. Overton trains and tunes models automatically. And then delivers a deployable machine learning model. So this is important to speed up or to reduce the time between experiments. Overton has an emphasis on creating deployable models. So any model that comes out of Overton can be deployed into an existing system and then it, it reports to, to an engineer. And then the engineer's job is to add/augment slices. So slices, they mean slices of data. So which slice of data should be improved the most? Add labeling functions. So what programmatic labeling functions can you add to your system to improve your weekly supervised data? And add synthetic examples if you need. So go have a look at that paper. If you're looking at building a full-blown machine learning system, check out Apple's Overton. And the best part, Apple have battle-tested their framework for over a year before publishing it. So the theory comes from a practical grounding. So the theory discussed in the paper is based in practical means. We're talking about Apple delivering machine learning devices, or sorry, machine learning powered features on their devices using this system. So that's the best kind of theory. Far out. That was a huge month for Machine Learning Monthly. We had the theme, of vision and text. We started off with Nicholas' hateful meme detection. We went through OpenAI's amazing, amazing work over the last month. And we finished with a bunch of full-stack deep learning resources. And of course, I'm going to emphasize this again, is I'll leave the link below. FreeCodeCamp are building a full-blown data science, all from the ground up. Math, Python, you name it, they're building it. So if you can afford to donate, Please donate. There's 25 or 26 days left. Darryl Silver is going to match all of your dollars up to $150,000. FreeCodeCamp are amazing. I'll leave the link to this article, Machine Learning Monthly. You can have a look at all the results yourself in the description below. Sign up if you want. If you want to submit your own, see it in a future episode, my email is daniel@mrdbourke.com. Put in the subject Machine Learning Monthly submission and We might see it in the next one of these. But as always, keep machine learning, keep creating, and keep dancing. You know, did we discuss at the start how my friend said my tip for dancing is to just, you just gotta let your wrist flow, right? And not just your wrist, all of your joints. So you start wrist, ankles, feet, hips, to beat. Anyway, that's enough. We'll see you next month.