Replicating Airbnb's Machine Learning Powered Amenity Detection Part 3: Data Manipulation

Hello everyone, Daniel Bourke here, aka your friendly neighborhood machine learner, and I am very pumped up. One reason, because I've got coffee. The second reason, because it is a Monday and I'm super pumped to begin the second week of the Airbnb, or the 42-day project that I'm working on, aka replicating Airbnb's amenity detection pipeline. And if you're wondering what that is, well, I read an article a few weeks ago about how Airbnb uses computer vision, and I decided it'd be a fun project to try and replicate that, to practice some tools I've been wanting to use and to just sharpen up my skills as a machine learning engineer. Now we're officially into week 2, and today is day 8. Let me show you over here, we've got our little 42 Days project. If you're wondering what this is, there'll be an article below describing what 42 Days is, but Let me take you over to the whiteboard because this is where we are planning things out. And as you can see here, whiteboard is day 8. Wonderful. So if we look at week 1, we ticked these off in the last video, and now we finished off in the last video getting our data sources together. So we got our data sources together. However, we failed at building a small working model. So we're going to bring this over into this This week. Boom. So that's now down there. So the goal for this week is to refine our data sources that we found, AKA Open Images, and to check whether public data is enough. But the way we're going to figure out, and by public data being enough, I mean, can we replicate Airbnb's MVP, AKA achieving 50% mean average precision? We'll jump into that once we've built a small working model. I'm getting attacked by puppies! I was going to show you that I've got my 3 little goals for the week ready, but then my puppies decided we need some attention. So the way I start out each week is I kind of just, if I'm working on a project, I put down 3 key things that I want to do. And here are the ones for this week. We've got week 2, data manipulation, dataset preparation, Wait, let's try that again. Dataset preparation and small model building. And so this week I want to convert all of the images and labels to Detectron2 style labeling. We'll go into that once I get it done. Uh, download all of the images from Open Images belonging to Airbnb's target classes, the 30 target classes we've talked about in the previous video, and then build a small working model on a custom dataset based on Detectron2. So I'm going to do a few hours of coding and trying to work towards these steps. As you can see, this is day 6, converting all images and labels to Detectron2 style labeling. That's my current roadblock. I've got a way to download images, but right now they're not of the same input shape to the model I'd like to use. So the goal for today is to fix that little roadblock and see where we end up. So as I said, I'll spend a couple of hours working on this, and once I've made some progress, I'll I'll let you know what's going on. You know, in the last little section of this video, I said I'll see you in a few hours. Well, it's actually been a few hours plus a day plus a few hours. And if you think I'm looking fresh, it's because I've just woken up from a nap, so thank you. And the reason why it took a bit longer is because I got stuck. And you wouldn't believe, I got stuck, stuck on some simple pandas column matching thing for hours. So basically, I was just butting my head up against a wall, but luckily I've got my secret cure. Eventually we got it fixed and it led me to create like a little database in Notion of code snippets. So quite often I find myself running into a similar error that I've been down before, like a rabbit hole. Instead of just, as I said before, banging my head up against a wall, I've decided to start documenting those errors I quite often run into. So one of them was that simple pandas column matching thing, like just creating a new column based off another column in a pandas DataFrame. Simple problem. I've done it probably over a dozen times before, but I keep running into it. Anyway, let me update you on what we've been doing. Because I'm using Detectron 2, I'm coming in to the Detectron 2 documents and I'm looking up, or I've been looking into using custom datasets because that's what I want to do. I want to use Open Images and Detectron 2 was trained on COCO, which is Custom Objects something, something, basically another open source dataset of images. So basically I'm trying to take the pre-trained Detectron 2 model and train it on my own custom Open Images dataset based on the classes that Airbnb are most concerned about. So I read through the documentation here and it was quite a bit cryptic, right? So, okay, it says here Um, for standard tasks, we load the original dataset into a list of dictionaries with a specification similar to COCO, so COCO's JSON annotations. This is our standard representation for a dataset. Okay, awesome. And then it goes through, each dict contains information about one image. That was the, the key point there. Um, the dict may have the following fields. The fields are often optional. So if you look at these fields here, We've got filename, semseg filename, semseg, height, width, so the shape of an image, image ID, blah, blah, blah. So a fair few files here, or following fields. Well, I can't even read. So a fair few fields, but we don't know which ones are optional, which ones aren't. And so what I did instead of just looking at this, 'cause I look at this, I look at documentation sometimes and it's almost gibberish to me. So what I prefer to do is basically write the code out, visualize it myself. So that's what I did in the Detectron 2 tutorial. I made my own version of it. Um, so here's a little pre-format function that's going to format some data, and then I found out that this is the style. So if you were to compare this— actually, I've got an example right here. So Detectron 2 example style. You can see annotations. If we come back, um, annotations is its own list of dictionaries. So we've got bounding box, bounding box mode, category ID, which is the category of class of a particular image, the file name, uh, the height, the image ID, the width, and then segmentation. I'm not going to be using segmentation because I'm just starting with bounding boxes. If you're not sure what that is, just look it up, object detection bounding box versus object segmentation. Long story short, one's like a square outline, the other one is outlining the entire object itself. So I figured out the inputs to detection2, which is amazing. And then I used this downloadio.py or oi.py script to download the images of specific classes that I was after. However, I had to alter it. So this amazing blog learnopencv.com provided the script. Long story short, this script used to download images as well as create label files in the form of text that we looked at in the last video. I quickly realized through a bunch of trial and error, as you can see, this notebook is pretty long, and you might be wondering why is there 00, 01, 02. Well, basically I wanted a demonstration of to go, hey, look, this first time that I was exploring the data, this notebook is just a bunch of janky code that runs through different things, no real structure, just me trying to figure out different stuff about the data. So exploring the data. And then this 01 is just a refinement of the original notebook. So I took that code and cleaned it up a little. And then 02 is where I'm currently working at, and now we've got it down to about 3 different functions. So if we have a look here, I download the images into one file, aka a training file or a validation file, and I get all the image IDs, which is just going to return a list of unique image IDs. So that's one function. Then I have another function to format the image annotations So with OpenImages, you can just download the training set bounding boxes. So this file here, all the validation bounding boxes, that's going to download a CSV. And now the caveat to my functions, and I've got this here to do— note, this function could definitely— oh well, it— well, it could— I know it definitely can be faster because I use Pandas to do some manipulation, whereas I think if I just use the CSV file to do the manipulation rather than importing into a pandas DataFrame, it'd be way faster. So if you have, uh, tips on how to do that, well, leave a link below or leave a comment or check out the GitHub repo and improve this notebook for me and faster code. And then finally, there's another function, get_image_dict, which mirrors the Detectron2 notebook that gets balloon dicks. So basically turns the annotations I download from OpenCV, so all of the bounding boxes for training or validation set, into this style code. So a list of image dictionaries. So if we come down here, this is what one of my images looks like. I've downloaded this from Open— Open Images. So there's the training file— or there's a file name there. There's the image ID. There's the height. There's the width. There's the bounding box. Again, we're using just the square bounding box, not the full image segmentation yet. Um, and then if it has multiple bounding boxes, it has different category IDs. And the category ID is just the class name converted to an integer. And if we have a look at the detection2 example style, It's— they're a little bit out of order, like in terms of the dictionary. Remember dictionaries? I don't think dictionaries have order. Maybe they do, maybe they don't. But they contain the very similar information. So what are my next steps? If we revisit the Notion document, my second brain for the time being. Remember this week I want to convert all images and labels to Tektron 2-cell labeling. So we're getting very close. I can't tick that off yet because I want to, if we come up here, view an annotated image. So that's one of my next steps, is to make sure my labels are correct by visualizing one of the labels that I've created. So, aka, grab an image like this and turn my label into a visualization, much like you see in the example detection 2 notebook, something like this. So that's what I'm going to be creating next, but for my dataset. And then if we come back to the Notion document, so I'm getting really close to ticking this one off. Uh, after that's ticked off, I can download all of the images from Open Images, um, into my, uh, VM, so my virtual machine. So I have all of the target images I want to work with, and then I can build a small working model on a custom dataset. So as you can see, we're up to day 10 now. We did get stuck. But we're polishing the data script, becoming one with the data. As you'll find out if you read this blog post here, I'll link it below. It's a great recipe for training neural networks by Andrej Karpathy. And as you'll see, step number 1 in the recipe is become one with the data. And as for, like, someone asked in the last video, like, what's my style of documenting things? I kind of just treat it like, uh, this is literally just a conversation with myself in Notion. So 2:49 PM, having a nap and then getting back into it. Whenever I take a break, I try to at least write one thing to do next. So as you can see, next, write a function to visualize my images and labels. Um, and then next is to check to see if Detectron2 works with my labels. So that's That's what I'm up to next. Let's do it! Side note, I just realized that little update clip was about 9 minutes long, so to avoid this video from getting too long, I'm going to do the next stage in the next video. If you like that sort of style, let me know. If you don't really care about length, or if you want— if you have any suggestions or advice for the style of these videos, because I'm going to be making them for the next 4 weeks or so, leave a comment below of things you might like to see or would really like to see. Actually, yeah. Leave a comment below of some things you'd really like to see in the upcoming videos, because there's going to be at least half a dozen more in this project, because we're at day 10 of 42. Exciting times! I'm gonna go visualize some data.