This book will help you get a job in machine learning (ML Interviews Book) | ML Monthly June 2021

Friends, welcome back. It is that time of month again. I'm super, super excited for this episode. Now it's a beautiful rainy day outside. I did a rain dance this morning. And it looks like it must have worked. This is Machine Learning Monthly, where we cover the latest and greatest of machine learning for the previous month. So this is the June 2021 edition. It's the latest and greatest of machine learning, but not always the latest, because there are some things out there that are evergreen and have stood the test of time. Without any further ado, this is the article. We've got a lot to cover today. If you'd like to read it, you can jump down the description, the link will be there. Uh, of course, if you'd like this delivered to your inbox, you can put your email in there and then you'll get this at the start of every month. And the video version, which is what you're watching now, comes a few days after. Let's jump into it. So, my work, the Zero to Mastery TensorFlow for Deep Learning course, has launched. Yes, yes, yes! I'm super excited about this. The biggest update from the last month is that now all of the materials are available in a beautiful book. Look at this! So if we go lerntensorflow.io, boom, jump into it. Look what you get. You get this amazing, amazing formatted book. We have 00, 01, 02, they're the fundamentals. Then we have notebooks on computer vision, transfer learning, a massive, massive important technology and deep learning, natural language processing, and finally the time series notebook. Beautiful, beautiful, beautiful. Go to learntensorflow.io, it'll redirect you to here. This is, this is a lot easier to read than a Jupyter notebook on GitHub, which is where the current materials live. So if we jump in here, and if you click on one of these, sometimes it loads, sometimes it doesn't. Yeah, see, we're waiting for that to load, versus this resource also has beautiful headings. We jump down into here, uh, say we wanted to learn about uncertainty and dating. I'm not going to spoil this, but you can read that if you like. But learntensorflow.io, my new TensorFlow course is now available as a book. Beautiful. So now let's jump into the internet. We have some amazing resources here. Do I have a booger in my nose? Maybe. Hopefully my faces are small enough that you can't see that. Anyway, this is a beautiful blog post by Uber. Uber's journey towards better data culture from first principles. So let's jump into this. If we have a look here, did our notebook even load actually? Yeah, see, look, this is what happens when you try to load big notebooks on GitHub. Something went wrong, reload. When you can just now go to this version and it's going to be really quick. Fundamentals, zero, zero. We jump into this. Boom! All right, matrix multiplication, so important. Anyway, we're talking about Uber. So this is a great blog post on how Uber are becoming a data, well, they already are a data company. All right, so a lot more companies are sort of heading towards that data first. And if you imagine a timeline of data companies, Uber's probably one of the biggest big dogs, or one of the elders. I mean, they're only 12 years old, but they've basically, from the get-go, Uber's business depends on data. And so they've shared some principles here, which are really great. And if you're looking to build some sort of company or at a company that wants to start approaching data from first principles, you'd be wise to take a note out of Uber's book. So one of the first ones, I like these, data is code, data is owned, data quality is known. So we go back to data as code. Data should be treated as code. How did it get created? How did it get changed? Who owns it? So if you have to go to talk to someone about a certain dataset, who's the best person to talk to? These things should be tracked, especially in a large organization like Uber. Organize for data. I really love this one as well. Teams should be aimed to be staffed as full stack. So that means that everyone in the data engineering pipeline or all of the steps should be covered that way within individual teams. There's no bottlenecks. And then if we go back here, I also linked why this matters because it relates back to Software 2.0, which has been covered in a previous issue of Machine Learning Monthly. But it's— I'd highly recommend reading this article. Software 2.0, the main distinction between Software 1.0 and 2.0 is that Software 1.0, you would write a set of instructions for computer program to complete. Whereas software, software 2.0 is you're not necessarily writing a full set of instructions. Of course there are going to be some instructions there, but you're programming data. So that's the big paradigm shift. And this is what essentially, this is what I got from this article is that Uber are really heading towards that software 2.0 milestone. Let's jump back in. So the next one up is Test Text-Based Video Editing Research Paper and Demo. I got sent this demo. I wonder if we can check out the video demo without getting in trouble. I might just put a clip here. Let's watch this together. We introduce a practical tool for iteratively editing talking head video. Starting from this video. This was easy for us. We added the transcript and within 40 seconds we obtained this. This was entertaining for us. What have you— when I was first sent this, I was like, no way, no way. And as someone who's in the machine learning field and sort of looks at these things daily, I didn't even— and someone who also makes videos, I didn't even think of this when they sent this to me. They sent this amazing research to me and sent this demo video. When I edit videos, I don't even think of editing them with text. I— because it's just such a different paradigm to what we've done in the past. If I make something wrong, usually now I just roll with the punches because mistakes are going to be made and I just keep talking. But if I make something actually wrong and I need to edit it out, I usually just cut it and re-record it. But this is a massive paradigm shift to how I think about editing videos. And I think there's a beautiful company as well called Descript who are doing something similar to this with video editing. They use transcription so you can upload a— this is not sponsored by Descript or whatever, I just like their product. You can upload your audio or video. Yeah, there we go. A video word processor. Edit video by editing text. So they— this one's more like of a cut So if you, if you say the wrong sentence, so I tell you to don't think of a green elephant, what are you going to do? And I want to cut that sentence out. Well, I can do it with that. However, this demo video goes a step further. And if I said the wrong word, if I said don't think of a purple elephant, but it should have been don't think of a green elephant, I could just cut that out and then it would generate a video of my own face saying that word. Now, of course, this is still research, but if you'd like to check it out, the link to the script is there. But there's a project website for the paper, there's a paper itself, and of course the video demo, which is this video. Check that out. That's wild. If you make videos, that could be big for the future. All right, the next one. This was a massive one. So GPT-3 came for writers and now it's coming for coders via GitHub Autopilot. I was about to say autopilot because because I'm thinking about Tesla's Autopilot. But let's have a dive in. You've probably already seen this. I mean, it's all over the internet. Your AI pair programmer. RIP all of the programming jobs. No, I don't think so. Let's have a look at this little demo. So what it is, is you enter some natural language describing what you'd like the code to do, and then Copilot writes the— this is Python— the code for it. Now there could be— I can understand why some people are like, wow, this is going to take all my jobs. However, I think, well, it's saying more than autocomplete. For me, it's like, yes, it's a very smart autocomplete. However, I think this will be most helpful for things like helper functions, debugging and whatnot. You'll get Copilot to help you out because You can't— you still can't yet go to Copilot and say, write me an app that replicates Airbnb's functionality. Or maybe you can. I'm not sure. I haven't tried it out. I've signed up for the test, but that's what I think. It's like, it's, it's— yes, it will be able to probably write code better than me for small little helper functions. But the overall logic of a program from top to bottom, where data goes in, how it gets manipulated, how it comes out, You'll still have to put those pieces of the puzzle together. Copilot will just help you out. It's similar. Imagine this. Those tools like Logic Pro. Logic is a tool. There we go. Logic Pro, a digital tool for creating music. And of course, it can produce digital sounds of like an electric guitar or something like that. But you still have to put the pieces of the puzzle together to create a beautiful sounding track. That's how I imagine Copilot's going to help you out. Yes. Wow. I've got a little bug flying around here. Come on, buggy. Don't want to kill you live on camera. Yeah, you're still going to have to put the pieces of the program back together. But this is really cool. And a little tidbit from this is that I think this is a beautiful synergy happening between when Microsoft bought GitHub. So Microsoft Look at this, we've cracked the code. Microsoft buys GitHub. Microsoft acquires GitHub. Look at that. So GitHub, notice that Microsoft partners with OpenAI. OpenAI forms exclusive computing partnership with Microsoft. And then the final, final nail in the coffin is, oh, we don't even have to search for anything. How is Boom! GitHub Copilot is partnered, uh, powered by OpenAI. I believe it's called Codex. Codex, which is a cool name. Yeah, so GitHub Copilot is powered by Codex. I, I think it'd be very similar technology to what GPT-3 is using, so some sort of code language model. It's just an amazing synergy there. We see it all comes together now. Microsoft buys GitHub, Microsoft partners exclusively with OpenAI, and now GitHub is partnering with OpenAI to create Copilot. That's a pretty cool setup. But now let's jump back in. Go check that out, sign up to the waitlist, and we'll try it out. The next one— oh, this is absolutely amazing. This is probably the book I wish I had when I was looking for machine learning jobs. It is an introduction to machine learning interviews book by the amazing— is it here? Yeah, Chip Hewitt. Chip's an absolute and complete living legend and one of the OGs of machine learning. So there's Chip's website. Graduated from Stanford, teaches a course, Machine Learning Systems Design. We've seen this in a previous Machine Learning Monthly where yours truly did the guest lecture for machine learning deployment. Boom, there we go. Guest lecture by moi. But back to the book. So this is an amazing book that if you're looking for a job in machine learning, uh, Chip has interviewed at a dozen big companies and startups, uh, got offers for machine learning roles at companies like Google, NVIDIA, Snap, Netflix, Primer AI, Snorkel AI. Just keep naming big ML companies, Chip's probably been there, um, as an interviewer, also involved in designing and executing the hiring process at NVIDIA and Snorkel AI. So NVIDIA, You probably heard of that company, pretty big. And what she's done is she's collated all of the things that she's learned over that process of both being an interviewer as well as an interviewee. Is that the same thing? I'm not sure. You get what I'm saying. And then collated it. So such as some of my favorites from here is describing the different machine learning roles. This is all online and free, by the way. So you can read through this if you're going for a job. I'd highly recommend this resource. I get asked all the time, what should I do for an interview? I don't really know. I've never had a traditional job interview. In fact, I'm not even going for a job in machine learning. I'm sort of, I would rather build my own business, but I understand that's not for everyone. So if you like to go for an interview, read this book. And so some of the roles that she breaks down, machine learning engineer, data scientist. If you're new to the field, it can be, you see all these role titles and you're like, I don't even, personally, I don't even know what all these people do, right? So when you run, when you do sort of the things that I do, you kind of a little bit of everything depends what the day is, but it breaks them down. My other favorite one was 4. Yeah, 4, where to start. Oh my gosh, we've got so many tabs already, team. Where's number 4? Look at this. Look how comprehensive this book is. Where to start. But that's the beautiful thing is that it's all HTML so you can just jump in and out. I love this one. How long do I need for my job search? Well, this is, this first sentence is completely true. Getting a job is a skill in itself. It's kind of strange, right? The skills you need to get a job are different to the skills you would actually use on the job. Right? So by the time, when you, when you get like, you get good at trying to look for a job, but then you get and you get an actual job and then those skills wane for looking for a job because you don't have to look for a job. It's kind of counterintuitive, but it is a hard problem to start. Now, this is my favorite one. Can you see this? Am I blocking this out? I don't think so. Boom. How to become a machine learning expert in 3 months. You can't, right? If I was to advise someone on how to become a machine learning expert in 3 months, spend the first 87 days building a time machine. Day 88, take a rest because you just spent 87 days working straight. Day 89, go into the future 10 years or so, trick your future self to swap with you, and then there you go. How to become a machine learning expert in 3 months? You can't, right? I've been doing this stuff for 4 years and I'm still nowhere near an expert. I'll be the first one to tell you where I can improve. But becoming an expert in anything takes years, if not decades. Yes. At best, they teach you bad machine learning. At worst, it's a scam. So anyone who claims to give you a shortcut to becoming a machine learning expert— I got trapped in this. I got trapped in this in the past. I thought I could do everything in 3 months, 6 months, or whatever. That's the curse of the internet. It like speeds up everything. But it is a scam. You take enough shortcuts, you end up taking the long cut. I love this article by Peter Norvig, which is Teach Yourself Programming in 10 Years. We're kind of, digressing here, but this is worth it. Read this article. It's just one of the many hidden gems that you're going to find in this book. There's stories of how other people did it, and then there's dos and don'ts for ML interviews. I love this one. Dos: start your preparation early. I like this one probably the most. Number 2, get someone to refer you. All of the jobs I've had in the past have been through someone else. So that's a big thing. But how do you do that? How do you get someone to refer you if you don't know anyone? Right? But this is the power of the internet. We go down to Appendix B. Of course you could read through these, these are some more great tips, but this is a summary. B, building your network. If you didn't think that having a network mattered much before, I hope that this book has changed your mind. Having the right network can open you up to new opportunities and bring you closer to your dream jobs. How do you create a network? Well, if you're like me and you're in a bedroom and you just want to make things online, You just make things online. And then at the start, yes, they'll be crap, but people can tell that you've just been honest. You've just been upfront. Hey, I'm learning. Go back through all of my original videos and all of my original articles. Bunch of crap, right? But it was at the time, it was the best I could produce. That's what I'm doing now. I'm still creating the best I can do with the skills I have now. And then eventually people will find that if they can see your honesty, interact with other people's work. That's how you build a network. And it's really Network is to me a scummy word. It's just you're building friends. To refer someone, this is like, you'll read back through, there's a story of like the best applicants are the ones that are referred by someone else. That's what you're doing. You're building friends in the field by sharing your work, sharing your ideas and interacting with the same that others have put out there. So highly, highly recommend this book. I'm now going to send it to anyone who asks the question, what should I do about getting a job? My favorite is, um, Daniel Bourke, uh, how does a beginner data scientist like me— yeah, here we go. My favorite is, I always say this one, start the job before you have it, right? That's my best piece of advice. I'm now going to say that along with this book, so go check that out. Thank you, Chip, for creating the amazing work that you do. Up next, we have Tesla inching towards full self-driving. This was an amazing talk by Andrej Karpathy, who was the head of Tesla AI. Oh my goodness, pause. Um, but it's a beautiful video. It rehighlights the use of the data engine, and it also goes through, uh, what Tesla are now using as, um, they've gone vision only. Oh, look at this, their supercomputer. It has 8— no, 720 nodes of 8 times A100 80GB. So 5,760 NVIDIA 80GB GPUs to power their Full Self-Driving model, which is absolutely insane. Where's the, where's the architecture? I believe it's— yeah, here we go. There we go, neural net architecture. It's got some backbones, uh, in the talk Andrej also mentions that these are sort of ResNet-50-like. Of course, Tesla are going to customize them for their own use case, and they combine them and they have multi— multiple different outputs. So it's just one big architecture, um, with a whole bunch of image feature extractors that go into different trunks and put out— so this is a detection output, attributes, attribute output, kinematics, future trajectory, direction, etc., etc. So I would highly recommend looking at this talk. It's absolutely amazing that people like us can, can see how the best companies in the world are building their tech stacks. And of course, can we replicate this in our backyard? Probably not. But it gives us an insight into the skills that we have to learn to work towards potentially getting a job at Tesla or creating your own machine learning startup. Go check out the that beautiful talk by Andrej Karpathy. Next. Oh yes, this is another thing that Tesla, why they're leading in self-driving is because they've created a data flywheel, a concept that we also covered in a previous Machine Learning Monthly, Build a Data Flywheel. So if you'd like to check that out, that's Machine Learning Monthly, January 2021. Now we are up to hit me baby two more time. That should be times, times series. And there we go. Okay, so that's— I worked out what I did with my own writing. Hit me, baby, two more time series analysis and forecasting libraries. Doesn't quite work as well. But these are two libraries that have come out of Facebook and LinkedIn for time series analysis and forecasting. If you're— why, this is the beautiful thing I love about machine learning as well. Is that a lot of the best practices get released as open source, and then it's your skill, or it takes you to sort of apply those to your problems. So let's have a look at Facebook Kats. Cool name. Maybe it's an acronym for something. Is a kit to analyze time series data. Oh, okay. Kats, a kit to analyze time series. Cool name. We just worked that out on the fly. But there's some great, great examples here. If you'd like a tutorial, it goes through time series basics or Kats basics, forecasting. So if you'd like to predict a number in the future, detection, which is like time series classification. This is one of the extracurriculum resources that I've linked for— if we go back for the time series module of the TensorFlow course. Of course, the course is— of course, the course is focused on TensorFlow, but But there are other libraries that I've put here. If we go other kinds of time series, this is extra resources. If we see here Facebook Cats, beautiful. We can leverage the technologies used by the best in the business, as well as LinkedIn Greykite, which is very similar. Um, we have a flexible intuitive forecasting library. We have great Greykite quick start tutorials, etc. If you're doing anything to do with time series Bookmark both of these and I'm sure you'll find something useful in there. Speaking of open source best practices, we have all kinds of data augmentation with Augly by Facebook. Now this is a beautiful library that offers over 100 different kinds of data augmentation for text, image, video, and audio data. And it's open source, of course, all in Python. I used it here. Example workflow of Augly. We have an input image. 10 lines of code, barely. And then I turned my book into a meme. This book is awesome. Very biased meme here because I wrote the book and I said that it's awesome. But this is the type of input and output that Augly offers. And now it's geared towards a lot of different problems that Facebook have with content copyright breaches and stuff like that. So if you imagine people reposting photos that have copyrights, but changing the colors on Facebook's timeline, of course, that still infringes copyright. So Augly was created to help train models that can detect those type of images. So look at this beautiful image of a turtle, and we have some screenshot augmentation, overlay text, saturation, crop, rotate, brightness, blur, etc., etc. Once again, some beautiful tutorials in here as well. We go into the examples. This is how I created my book meme. If we jump into this, this is the ugly image example. And of course you can open this in Colab and try it on your own images. There's over 100 different augmentations you can try out. So bookmark that if you would like to build more robust AI models. To finish off, let's finish with the Fantastic Four rapid-fire resources. Pew pew pew pew pew pew. We have 2x MLOps. So this has been covered a few times in Machine Learning Monthly, but there's a reason, because it's amazing. This is the MLOps course made by Made with ML. I'm personally going to be going through this as I start to work on my next project. Stay tuned for that. It's by the incredible Goku Mohandas, who has some amazing machine learning experiment, uh, experience. Experiments is a big part of machine learning. I've been saying that a lot recording videos for the TensorFlow course. But you can check out Goku's awesome name, by the way. Dragon Ball Z, am I right? Awesome experience here. But the course, the main part of the course is, look at this. We have product, data, how do you label data, how do you preprocess it, modeling, baseline, scripting, how do you organize your Python scripts into a package, How do you take your things from being a notebook to being something like a Python library? Interfaces, how to use a command line, a RESTful API, testing, reproducibility, production, how do you deploy stuff? Dashboard, infrastructure, CI/CD workflows. So continuous integration, continuous deployment. That is actually how the book is built, with a little bit of CI. So if you go to the GitHub, This is how this book has been built using GitHub Actions. So if I make a commit to this repo, change one of the notebooks, it runs this, this code here, uh, copy Python notebooks to docs, which is another folder over here, docs, and then it hosts these docs on a GitHub Pages website, which is what you see here. So check this out, go through it. The less— the lessons are all done with beautiful code. Let's have a look at CI/CD. Intuition, GitHub Actions. Wow, that's a beautiful graphic. That's like graphically showing you what I just kind of told you. Components, we've got a YAML file, workflows, events, jobs, steps. Look, this is, this is beautiful work. This inspires me to do better work. So check this out, bookmark it for all of your MLOps needs. Speaking of MLOps, we have mlops.toys, which is another curated resource. So like, I like these little URLs that are easy to remember. mlops.toys. If you want training orchestration, you've got tools here. Tools for data versioning, boom. Have we got experiment tracking? Is my favorite Weights and Biases here? Neptune, Weights and Biases, boom. There we go, Weights and Biases. All different tools on mlops.toys. If you got something, you can contribute as well. So big shout out to MLOps.toys. And finally, we have two beautiful blog posts, Training Compact Transformers from Scratch in 30 Minutes with PyTorch. So when Transformers first came out, I remember we had a problem in my machine learning role is that they took way too much data. But this beautiful research out of the University of Oregon and PixArt AI Research goes through an incredibly well-written tutorial, which first of all defines what attention is and replicates it in code. Attention is the main algorithm behind transformer architecture. Many different flavors of attention, but they use, I believe it's self-attention. So there's the PyTorch code for self-attention. Vision Transformers. Vision Transformers is a recent paper from— yeah, there we go. An image is worth 16 by 16 words. Which achieved state-of-the-art results on image classification and whatnot. It beat out convolutions. However, of course, they're data hungry. So what this paper solves, Compact Transformers, is allows people like you and me who don't have great big clusters of GPUs to train our models on so that we can train it. Where is it? Here we go. Vision Transformers for Mere Mortals: Compact and Efficient Transformers. This is my favorite kind of research. Oftentimes you read a research paper and it's like, yes, we trained this state-of-the-art model, but it took 300 TPU days or something like that on across 240 TPUs. I don't have that. I have a GPU and I have a Google Colab. So I like the fact that the researchers have allowed us to train— where's the stats? 30 minutes. Boom. Well, it's in the title, isn't it? 30, 30, 30. There we go. You can train on a CPU, 30 minutes to 80% accuracy. But go check that out. Beautifully written blog post. There's a paper to go along with it. That's how I like to read research in this sort of style. Finally, we have a beautiful blog post by Inés Montañí, who's one of the co-founders of Explosion.ai, the founders or the creators of spaCy, spaCy NLP, which is an industrial-strength natural language processing library we've seen in the past. But this blog post goes through how you should think about applied NLP. As you could imagine, or you might not, you might not, applied NLP, which is applying NLP to a setting in a production setting, is a little bit different to research. So in research you might have— I like this table, by the way. How versus what. Find the best trade-offs. That's my favorite highlight from the whole thing. What are the trade-offs? So if you imagine in research, you can build the biggest model you can ever imagine, and it can use all the compute you ever need. Well, maybe depending on what your limitations are, but in production, that's often not the case. So I remember when BERT came out, 2018, we were trying to deploy that as machine learning engineers, and we didn't have the resources for it. So we ended up going with a smaller model that performed at 95% the accuracy of BERT, but was like 50 times smaller. I'm making those numbers up, but you can imagine that that's the, that's the about the scale. So find the best trade-offs. Potentially the biggest model doesn't perform as fast as you need it to. It may have better accuracy performance, but its speed is slower. So just keep that in mind. You can read through this blog post, phenomenal for developing a mental model about thinking about how to put your NLP models into production. With that being said, we have reached the end of Machine Learning Monthly. Once again, a massive month for the ML world in June. If you'd like this article delivered to your inbox, sign up. Otherwise, the link will be below. And this article, of course, includes the links to everything that we've covered. If you have anything, let me, uh, Let me put me back to full screen. If you have anything that you would like to, or if you'd like me to share your work, it can be whatever you want, beginner tutorial, any level. If you have something you'd like to see in a future video and, uh, what's this, text, text version of Machine Learning Monthly, send me an email to daniel@mrdbourke.com with the subject Machine Learning Monthly submission because I'd love to see it in here as well. With that being said, Keep machine learning, keep creating, and most importantly, keep dancing. Pew pew pew pew!