Using Machine Learning to Replicate Airbnb's Amenity Detection Part 1: Overview

What's going on everyone? Machine learning engineer Daniel Bourke here, aka Mr. My whole body is covered in sweat because it is extremely humid where I am, but that's all right because I love the heat, aka Dr. Delts because look at that deltoid, you know, that was my nickname when we first started gym. I was going to be a doctor, but then I decided against it. And I am very excited today because it is a Monday and I mean, I live a life at the moment where it's like every morning's Monday morning and every night's Saturday night. That's like my, my motto. But it is Monday and it is the start of a new project. Woo! I am absolutely pumped and I'm calling it 42 Days. This is kind of like, it's like an experiment and this is the first video. And the reason being is 42 days is 6 weeks and I go on a trip with some friends to Vietnam in 47 days, give or take a few. So I've kind of I'm going to fill that little space. Someone's decided to do a little burnout outside my place. But what is 42 Days? Well, I wrote a blog post on my website. I'll link everything I mentioned in these videos in the description, by the way. I wrote a blog post on my website called 42 Days and another one called How to Communicate and Share Your Work, which is a part of my machine learning course, which just finished up. And there was a section in there called "Start the Job Before You Have It." So the, the whole premise is if you wanted some role, let's say in the technical field, instead of just going through and applying with a resume in some online portal, which is most definitely going to be clogged, you figure out what that role, what that person in that ideal role does, or you design it yourself, and then you start doing it. And so the whole idea of— oh wait, I've got a framework for this, like a little cool poster. I'm going to print that out and come back. Alright, we got it printed out. So I'm going to stick this on my wall, and as you see there, we've got 6 weeks. That's 42 days. Yeah, and so all I'm going to do is this one little project, so I'm going to introduce you to that in this video. I'm gonna do that project. I'm gonna write it up here and then we're gonna work through this section together. And today is, 'cause it's Monday, it's like a research day. So, and it's day 1 of this project, so I'm basically going to just be outlining the rest of the days today. So let me write this project out and then we'll get back into what I'm actually doing. Alrighty, so you see here, we've got it there. I don't know if you can read that, but it says Project 2025: Airbnb's Object Detection Pipeline. Now you're thinking, what is that? Well, I read this amazing article by Airbnb Engineering, and when I think about it, of companies I'd like to work for, Airbnb, there's not very many, right? Because I'm not searching for a job, but Airbnb would be pretty high on that list. And I read an amazing article by them. I'll probably put it here. And it's about their object detection pipeline. And I decided what would be fun is if I just replicated that. And there's a few tools that I want to try out, a few new things, a few shiny new things. And so working on this project for the next 6 weeks will give me something to, to work towards, aka replicating and potentially improving Airbnb's object detection pipeline. And try out all these new tools. So I don't know how this is going to turn out. It's an experiment, but that's the whole idea of it. The beautiful thing about 6 weeks, aka 42 days, is that if it doesn't work out, it's only 6 weeks. So that's enough talking. Let's get into it. I'm going to spend the next couple of hours doing some research, putting together a Notion document of everything I'm doing, and then we'll come back here. Alrighty, so as you saw, I've kind of spent the last hour or so building up this little Notion template. And how good is Notion? Ocean. How beautiful is this? Look at that little house, that Airbnb photo I found on Unsplash. Anyway, so this is the 42 Days template, 2020, of finding Airbnb's object detection pipeline. And what I did was I went through this article here, and of course everything will be linked below in the description. Um, I'm kind of developing a central ground truth of the project that I'm going to be working on for the next 42 days. And so the last hour was basically me doing research and planning, going through this article that I want to reproduce at least, or improve, or just put my own little twist on it. And I was just putting the subjects here. So I've broken it down: problem, data— because all machine learning problems come with data, right— modeling— what kind of model did they use— and then any additions and learns. And so if we go back up here, I'm developing some criteria for myself. So if I'm doing a 42-day project, what is success for that project? And I haven't quite figured that out yet. So I looked at the article reading in there, and a little tidbit is that when you're trying to build like some sort of project, if you find that there are pain points— aka one of their pain points was They ended up using AutoML, so Google's AutoML service. They can't download the original model, so that's potentially something that I could fix. This is just early days still, but I've got some tools and techniques I'd like to cover and explore. But basically, let's, let's sum this up. What is their problem? Well, if we look here, basically this is what they want to do. They want to automatically do this on any image that gets uploaded to Airbnb. So if you've ever used Airbnb's app, they've got a whole bunch of different properties that you can stay in, and they want to figure out what amenities, or like tables and chairs and ovens, those, those properties, those listings have, so that they can put that in the description on their app. So that's their problem in one line: automatically label an uploaded image with what amenities it has in it. What data did they use? They had 25,000 total images with 30 labeled classes. So This is where I'm thinking I'm going to go first. 32K from open source, 43K from their internal dataset. That might be where I get let down, is because I don't have access to their internal dataset. It just means that I'm going to have to be crafty. Um, there's a bit of a distribution of the classes. This is how they set up their data labeling. So basically they used open source labels plus, or open source data, plus Google's Data Labeling Service. So they pay people to label their own data. Modeling, they tried out using some transfer learning with TensorFlow Object Detection, aka MobileNet V2, and Faster RNN Inception ResNet V2. What a name, Faster RNN Inception, CRNN Inception ResNet V2. One of them was fast with lower accuracy, the other one was slow with higher accuracy. But in the long run, they ended up going with Google's AutoML Object Detection API, which got the best results. And is this— is my lens fogging up? I guess it doesn't really matter, I look kind of clear. Um, and it achieved— so their evaluation metric is mean average precision. We'll jump into that later. So that's 68%. I calculated with some off-the-hand calculations that when they were doing transfer learning, they spent about $51 in GPU compute, which is pretty cheap. 5 days of training there on AWS. But if they used Google's AutoML service, um, I got 2 figures. So they said 3 days wall time. If it's 3 days wall time, it costs about $1,800 US. If it's 3 days compute time, because of the way Google's AutoML service works, it was $226. So there's some metrics and, and, uh, costs that I will be trying to keep in mind when I'm working on my own project. And there's some more classes. And the takeaways are: when they use their own customized model, it performed way better than some third-party model. That's really important. So that's their business getting solved, their business use case. And additions in one line: Object detection is one thing, but how good does a photo look? So that's another thing they're looking at for when they advertise their properties. Does a photo look good? So when someone uploads a photo of their property, does that look good? Like, is it blurry or is it just a crap photo? And, uh, learns. So this is some good takeaways. Right down the bottom of the article is some amazing takeaways. I like these ones. Um, basically what it says is data is very important and don't be afraid to try experiments. So we're going to go to the whiteboard and I'm going to take what I've learned from here and basically try to structure out the next 6 weeks and see where we get to. You know what? I think that's enough to begin with, that's for sure. We'll keep it, we'll keep it simple. Let me, let me walk you through this. Let me take you off, off this here tripod. And so what I What have we got? Well, we've got something simple. Project: 2020ifying Airbnb's object detection pipeline. It is day 1 of 42. We have 1, 2, 3, 4, 5, 6 weeks broken up. And as you see, the columns aren't that evenly spaced, but it doesn't really matter. Again, we are cooks, not chemists. So this is Airbnb's, like, metric that they were using for their proof of concept. That's how this all started out. So when you're, when you're starting a project like this, like a a machine learning or data science project, it's, it's always good to kind of set yourself an evaluation metric. That's step 3 of the machine learning framework in, in my machine learning course. A little plug there. But that's kind of just something that you can work towards to know what, what defines success for you. So if you work— if I did spend 6 weeks on this project, which I'm going to do, What is success for me? So I want to at least equal or better Airbnb's MVP, uh, or 50% mean average precision. We'll get into that. I'll leave a link where you can look that up. That's just a metric to evaluate object detection models. But I want to at least equal that. And I, as I said, I want it to be— this is week 5— deploy and make it accessible. Now of course all of these will change, but let's go through them. Week 1, which is now, we'll build a small working model so something works. We'll get the data sources together. So remember, data is a really important point. We'll set up a project outline. We're already starting that in Notion. Week 2, we're going to refine the data sources because one of the takeaways in the Airbnb article that we saw was that data makes up a massive part of the project. So you'll spend 90% of your time, and I've put 2 weeks here. To do, to do, to get data ready. Maybe that not, might not be enough. As you see here, is public data enough? Who knows? Week 3, I'm gonna start scaling up the model with different data sources that I've collected. So this build small working model is going to take the data from week 1 and week 2. We'll scale it up in week 3. Week 4, hopefully we can finish modeling. 5 and 6 is going to be deploy and make accessible to you. Airbnb used a REST API. Or at least that's what Google's AutoML object detection model was. So just basically you can take a photo on your phone, upload it, and get some results back. That's what I would like to have by the end of this. And then week 6 is going to be evaluate next steps. What would I do if I took another 6-week block using this original project? And is there something I can do with the retraining pipeline? And another question that I'm really, really concerned about is what doesn't my model know. So AKA, we'll dive into this deeper later on, but AKA which samples or which images, if we did upload them in bulk, which images is the model most confused on, which is it most uncertain about? Because at the end of the day, we're not really too concerned on the images our model is getting right or the guesses our model is predicting correctly. We're most concerned about the ones that it doesn't really know. But we'll look more into this later on. Now I've got a little bit of a rough outline on both the whiteboard and in Notion. I think it's time to, to build a small model and get into it. And as I said, this, this video here is an experiment. It's the first one. I haven't done this sort of 42 days, 6 weeks thing. So let me know in the comments below. We're going to wrap this video up here. But let me know in the comments below if you like this sort of style. I'm trying to keep them relatively short. You know, I have a tendency to go too long, but I'd like to make at least a video or two per week of the entire 6 weeks, basically just giving an update of where it's at, what I'm going to be doing next, and what I've done so far. Interesting things. If you do find anything interesting, please leave a note in the comment. If you'd like to check out the 42 Days thing that I created I'll leave a link to that blog post as well. As you can see here, I've got the project on my wall now. So we've got the start date, which is today, the 17th of February, and the end date is going to be the 30th of March. So as I go through, I'm just going to tick off those little boxes. This is fun! I am very, very excited. So Thanks for watching, and again, because it's an experiment, I'm not quite sure if it'll work out, but that's part of the fun. I'll see you, uh, in the next video. I think it'll be either in a couple of days, probably at the end of this week. It's Monday today, so maybe I'll get 2 videos in the first week, and then we'll see from there. That's enough. I gotta go build a model.