Using Machine Learning to Replicate Airbnb's Amenity Detection Part 2: Data Exploration

Welcome back everyone, Daniel Bourke here, aka the Bedroom Engineer, and today is day 4 of the 2020 following Airbnb's amenity detection pipeline. If you're wondering what that is, basically Airbnb's engineering team wrote an article on how they're using computer vision to scan the photos of listings that get uploaded to their platform and figure out what amenities they have. And an amenity is a kitchen, a bathroom, a table, a toilet, a jacuzzi, or a swimming pool. And so I thought it would be fun to replicate that article. Today is day 4 and this is video 2. Let's have a look over here. This is the project outline. We're taking 6 weeks to kind of work on this. The worst case that happens if we take up all 42 days and it doesn't work out, It's only 42 days. Anyway, let's come over to the whiteboard. So since it's, uh, day 4, I better update this. You want to do day 4. Now this is what I wanted to get done in week 1. So build a small working model, get data sources, set up project outline, and get workspace set up. So we have actually done this. Oh, trying to draw an outline while talking on the camera. Not very not working very well. So we've got this set up in— this is set up in Notion. I'll show you that in a second. And the workspace is set up on Google Cloud. This is what we're focusing on now, particularly this. This will be first and then this. So we need to get the data ready and then build a small working model using Detectron 2. That's what we're starting with. At least I, I think so. I'm pretty confident in that part, actually. My bad there. I kind of had to fight with the little stand that this camera setup is on. But let me show you my workspace/notion document where I'm kind of tracking this whole experiment. So I'm using Google Cloud, in particular a deep learning VM. VM stands for virtual machine. And so basically what this is, is a computer that lives on Google's server that I can access that has a lot more power than my little computer here. So that's, that's basically what I'm doing. And so I have to pay for this. I'm renting a computer off Google using their Deep Learning VM, gives me access to things like PyTorch, etc., etc. And the beautiful thing is that I can just connect to it via terminal, via SSH, which is just some sort of connection interface. I don't actually know what it is. And then that gives me access to this JupyterLab server. And JupyterLab is a place where I can run Jupyter Notebooks, code in them, and store my data and whatnot. And the beautiful thing is about renting it off Google— by the way, Google aren't sponsoring this video, it's just what I'm— the cloud service I'm most familiar with. So this isn't a plug for Google, it's just what I'm using. And the beautiful thing is if I run NVIDIA SMI, what this means is that I'm running a GPU, in particular a Tesla P4. Which is apparently a pretty powerful GPU, which we're going to need to run computations on our, on our images, on our model. And we've got over here our project outline. So this is, this has been a bit refined in the last couple of days. I'm using Notion for this. Uh, as you can see, start last Monday, that was last video, end March 30, 2020. We're day 4 of 42. My criteria, I've filled out a little bit of criteria here. So it has to— these are the things that I want to be able to do by the end of the project. So one of the pain points of Airbnb's model was that they couldn't download it from Google Cloud's AutoML. So I'm going to try and fix that pain point. I've got my resources here in a nice little table. And we're up to day 4. So this is where I'm just keeping like a, like a log of exactly what I'm doing. So 1:38 PM, that was 20 minutes ago. I've created a workspace in Google Cloud. I can boot it up, connect via SSH, and the first thing I'm probably going to do is try this little script for downloading images from Open Images. So the focus of today, as you can see here, is getting the images I need from Open Images. And let's have a look, what's Open Images? We'll go here, open images, open images dataset. It's moved to a new site. Beautiful. There we go. So basically there are millions of images on this website that I can download that have bounding boxes on them, which are going to help me with Airbnb Amenity detection. So I'm going to download these images, or at least the subset of images that I want. Airbnb only used about 30 different object detection classes to replicate and build something that is able to do something like this. And now, what 30 classes did Airbnb use? If we go to my Explorer notebook These are the classes they're most interested in. So toilet, swimming pool, bed, billiard table, sink, fountain, oven, ceiling fan, television. So what my goal of today is, is because Open Images, this dataset has millions of images on 600 categories, I realistically only want 30 categories. So that's my goal for today, is to get all of the images from Open Images V5 that are in these categories. I don't care about the rest, I just want to get these ones because these are what the categories that Airbnb list as their most important categories somewhere in this article here. This image here, that's all I've done. I've taken this image, I've copied the categories into my Jupyter Notebook, and Now I've got to figure out how to get these particular images from Open Images because that is how you do machine learning projects. A lot of the time when you're first beginning, uh, there'll be datasets available for you to use, but those datasets are often small datasets like this. The entire— if I was to download every single image from this website, I think it's like 560-something gigabytes. And that's too many for me to use for now. So my whole goal for today, as I said, is to get the subset of images and work on that. And that's another thing to remember, is when you're working on machine learning projects of your own, a lot of your time will be spent on getting your data collected first and then building a small model to begin with. That's enough talking. I'm going to go, uh, do what I just said I'm going to do. And we'll see if we can get a small model with him today or tomorrow. Fingers crossed. 346 minutes later. Woo! Oh, you know when you stare at a computer screen for too long and you start to go a little bit crazy? Well, that's where I'm at at the moment. But good news is, and I will say it until I go hoarse, Machine learning, or at least a lot of practical machine learning, is about lining up the inputs you have, aka the data you're working with, with some sort of machine learning algorithm, and then with the outputs that you want from that algorithm. In my case, the algorithm I kind of want to use is Detectron2, the bee's knees for object detection, and my input data is a whole bunch of images from Open Images. Where are they? Here we go. So this is the, the dataset that I'm working with, subset with bounding boxes, 600 classes. So what I'm trying to do now is instead of these 600 object classes, we spoke about this before, is get the only these 30 classes from Airbnb. And so what I've done is I've written some code to see whether those classes exist on Open Images, or if Airbnb have like, for example, kitchen and dining room table, did they just make that themselves, like, or is that an actual class in Open Images? And so hopefully instead of downloading all 1.7 million images, I download a subset of like 200,000 which, which fulfill the, the 30 classes. Or I think Airbnb ended up with like 35,000 images. Don't quote me on that. So come here, my JupyterLab. I started off with this subset. These are the, the classes directly just taken from here. I took this, turned it into a Python list, started exploring the Open Images. So I downloaded the class descriptions boxable, which is just a CSV, turned that into a pandas DataFrame, did some matching. So, um, created a match column. So if the name is in my subset list, I just had a boolean column there, value counts. And so this is where my first little issue kind of arose. Was maybe there's only 20 out of 30, because there's 30 classes in the subset. Maybe there's only 20 out of 30 of them that exist in Open Images. But then I did some digging, I did some exploring, I did some dancing. Well, not yet, I haven't done some dancing. I will dance though once Detectron 2 starts to run, hold me to that. Um, I wrote some code here, just a whole bunch of matching strings and all that sort of stuff. And then I figured out that if we look at actual missing classes, there's none. Because the only difference between Airbnb's classes is they have underscores, whereas when you download them from Open Images, it doesn't have underscores. So just removing the underscores meant that all the classes lined up, which is amazing. We have a lineup. So that means I then went and used the script from LearnOpenCV, this GitHub repo, this script in particular, downloadioi. I was typing that downloadio because, you know, I/O, input/output, but it stands for download_open_images.py. Just a little script that's going to hit up this database or this Open Images and go, hey, give me these classes. You can run a code like something like this. I'll just show you the exact code that I ran. I tried out the Jacuzzi class because there's not many Jacuzzi images. Here we go. You run something a little bit like this. So python3 downloadoai.py classes Jacuzzi mode train, which is going to go, hey, go to the training dataset and download all the images in the Jacuzzi class. So I did that, but then, and it worked great, so This script works amazing, but we ran into another issue where if you— if an image has multiple objects in a scene, as you could imagine, like a bathroom has a toilet and a bathtub, aka this example, running that script downloads the image twice and a text label, sort of a text file with separate labels. So as in one label for the bathtub and one label for the toilet, but two images. So let me just show you. If I download it into validation, it comes into the validation folder. And then if I go into toilet, uh, there's a particular example I want to show you. Which one is it? I think it's this one, right? So there's the label toilet, and then these are coordinates for a bounding box. There we go, right? Fairly pixelated, so hopefully it works. There's a toilet, but you can see there's also a bathtub in there. So if we go back into bathtub, and if we open up— this is the same image file name. Here we go. We compare these two file names. Let me close some of these. There we go. So you can see these are the same image, but then if we look at this label, it's going to be bathtub, but these are the same file names. So my next steps are— is potentially before I download a a bunch more images with all the other classes, is to figure out how Detectron 2, as I said, remember, practical machine learning, when you're using something for some sort of certain application, you need to line up your inputs with your outputs with your machine learning algorithm in the middle. So my next step, what I'm thinking of doing tomorrow, is to figure out the inputs to Detectron 2 and then see how compatible my images and labels are with that. What I'm suspecting is that I will have to— Detectron 2 will probably take one image but can handle multiple labels. But I may be wrong, I may have to check that out. I've got to check Detectron 2. So that's what I'm going to do next. Explore Detectron 2, see how it takes inputs, and then download all the images for the Airbnb classes into my into my workspace, into my Google Deep Learning VM, and then I should be able to write some code in here to start building a small model and see how we go. By the way, one last update before we go off. I created a GitHub repo for all the code, so you can expect this GitHub repo to start be updated as well as this little, this little Notion document to be updated. This is the Notion document, remember, is just like a diary entry. Literally, I'm just spilling out everything that I find. This was, this was day 4. I'm going to wrap it up here. It's time for a walk, you know. Um, but the GitHub repo will have code and, and other tidbits. All right, so tomorrow, figure out the inputs to, uh, Detectron 2 and try and build a small model. It's walk time for me. My puppies are, uh, mate, they're knocking at my door. I can hear them scratching like, "Please take me for a walk, it's getting to 5 PM." I gotta do what I gotta do. See you tomorrow.