Results from 18-hours of training a machine learning model | Airbnb Amenity Detection 7

It is week 5, day 32, and this is video part 7 of the Airbnb, or replicating Airbnb's amenity detection. Try and say that 3 times really fast. Now, according to the whiteboard, we're a little bit behind on things, but that's nothing to worry about because this is a project of fun. So we have, it is day 32, we have 10 days left. This week, the main goal, the main focus is to finish all the modeling. So that's what's gonna happen in this video. Roll that intro. As you just saw in this video, we're going to be building a big dog model, a.k.a. a model with all of the data, all of the Airbnb subset of the Open Images data, close to 35,000 training images. Yeah, that's right. A big dog model. Well, let me, let me catch you up on what I've been up to. But before we can build that big dog model, Here's week 5. This is the Notion. This is basically the— my personal assistant when it's come to this, this project, because I just log everything in here. And so, as I said before, we can build a big dog model, we need to build a small dog model. We've already done that with a couple of experiments on Weights and Biases. We'll look into that in a moment, but we've narrowed it down. So we've tried out a few models from the Detectron2 Model Zoo to 7 that we'd like to try in this. I've just narrowed it down to 2. So what I'm going to do with those 2, I'll show you in a second, but first let's go through the Notion board. So we've got week 5 here. I've got to pick one of the models I've tried and balance them for training, speed, and performance. So that's an important thing to remember. If I'm working with a limited time frame, I can't just train a model for days and days on end. Airbnb mentioned in their article that they trained a model for 5 days and 3 days, and I don't really have that. I'd rather just do enough small experiments so I know that the one I'm going to train works, and then just let it run for however long it has to run to, to get a result that I'm satisfied with. And that's where the training time/performance trade-off might come. If I don't train a model for X amount of days, so say 5 days, I train it for 1 day instead of 5, is the performance I'm going to get of 1 day, is that say 90% of the performance I would get after 5 days? Well, then I'm willing to make that trade-off. Get GitHub organized. Jesus Christ, this should have been done a while ago. Well, the good news is the GitHub is fully organized. So you come in here, MrDeburg/AirbnbObjectDetection. Of course, it could do with some refining, but for now, here's all the notebooks I'm working with. As you can see, I'm kind of iteratively working through this project, but by the end of it, this GitHub will contain all the code that I'm working on and a way that you can replicate everything that I've done. So I'll clean it right up, but I'll I'm just keeping everything there in one heap so you kind of can see if you wanted to. I probably wouldn't if it was me, go back through each and every step. So as I said, when it becomes public by the end of the project, I'll refine everything so it's, it's easier to understand. So we can tick that off. Let's come back to the Notion. Tick that off, get GitHub organized. Now pick a model from Weights and Biases. Let's have a look at what that means. So in the last video, in part 6, Week 6, we ran— well, we finished off with an experiment trying several different models, aka if we have a look, we have a list in one of these called models to try. Models to try. Is this going to play along? There we go. So remember, we went through the Detectron 2 Model Zoo, picked out some object detection models from that because that's our problem that we're working on, object detection. And now this is a a dictionary of pre-trained models, and we ran some experiments tracking them in Weights and Biases. The experiment was: for model in models_tried, do 3,000— or choose a model and do 3,000 iterations on a subset of the data, which was 3 classes from the training dataset: coffee maker, bathtub, and treehouse, and then measure the performance. And so because we're measuring average precision, which is a measure to measure object detection models. Here are the results from those 7 experiments. How creatively named are they? See, if you track something in Weights and Biases, it will track your model experiment with names like this. So I know for a fact that this is the number of experiments I've run so far. Nearly 40 total experiments. So there's a little takeaway for you, is that deep learning is highly— and my dog's going crazy. Deep learning is a highly ex— empirical process. So, it means try something, see if it doesn't work, see if it doesn't work, just keep going like that. And now, these are the 7 models that I tried: Silverbird, Sage-C, Gentle Deluge, Daily Fog. And so, as you can see, higher is better. Or I'm going to just let you know that higher is better. My dogs have decided that it's time to sing. Thank you, Dad. Alright, back to where we were. Average precision. It measures a bounding box and it compares that to the actual bounding box. So if my model makes a prediction of a square that's on my face here, but the actual square in the truth labels was over here— probably wouldn't be because it's a truth label— It just compares how much, how much of those two boxes are aligned. Now, you might see here that Earthly Cloud 39 and Fine Haze 37 are the two front runners. Now, Clear Eon 38, this orange one here, falls out a little bit short. And now, in terms of saving time, I could go back through and try these out between each other, but I'm just going to take the top two. From this. So I've narrowed it down from 7 models to 2. And remember that these experiments were all run with everything kept the same except for the model that I was using. The data was the same, the seed was the same, the hyperparameters were the same, except for the pre-trained model weights. So the difference here is just the difference in how each model performs on the same set of data, hence why I'm taking FireHaze37 and EarthlyCloud39 for the, for for the experiments that I'm about to run, which is the segue into the next part to catch you up. So pick a model from Weights and Biases. I haven't picked one single one yet, but we've isolated it down to two. And what I've been doing today in preparation to see which of those two models should be scaled right up and to be the big dog model, aka— Caught myself about to say aka. In other words, running on all of the data I've created a 10% subset of the data. So another subset so I can run experiments faster because if I train a model on all of the data, that may take hours and hours and hours that I don't have. So what I've done, let me show you. I've started putting everything in Google Cloud, by the way, because I've moved away from Colab. I need to run some experiments just, just on my own accord rather than using Colab. I might be training a model for hours and hours and hours on end. Colab might timeout. Can't afford that. So I have a GCP bucket with all of my data, my labels, and new splits. So the experiment I'm going to be running probably by the end of today or tomorrow before I start training up into a big dog model is testing those two models. We come back to Weights and Biases, testing, come back to our runs. I'm going to be testing the model in EarthlyCloud39 and FineHaze37 against each other on 10% of the data with the same training distribution or the same distribution as the entire training dataset. So I went through and I found the training class distribution. So these are all the Airbnb's target classes. These are the number of images that are in the training dataset. Stairs, couch, swimming pool, porch, television. So that's how many— you can see right down the bottom here, there aren't many examples of jacuzzi or dishwasher or treehouse. Dishwasher, I would have thought there would have been more. Treehouse, I can understand. Jacuzzi, I can understand. You can imagine not too many houses actually have those things. But if we come down here, I've also made a random 10% subset of the training classes, making sure that it was of the same distribution of the original training class. So that's a very important point. You could just take 10% of the training data, but is it— if it's not, if it doesn't have the same distribution— so see here this curve, if you were to line up this curve with the one up here, they're quite similar. They won't be exactly the same, but they're quite similar. So it has all of the similar characteristics as the original training dataset. So I can— the experiment that I run on this 10%, this random 10% of training classes with, with one of the models I'm going to choose, will be hopefully representative of the big dog model that I eventually train. And then I also, because the, the way Open Images downloads, you download it in a train validation test set, but the Airbnb article stated that they used a holdout set of 10% of the images. So that's what I created of my own. So rather than using a train, validation, and test set, I've combined the test and validation sets of the Airbnb subset of, of data, and then I did the same thing I did with the training dataset and created a val_test 10% classes distribution. Let's break that down. Take the test dataset, the validation dataset, combine them, and take a random subset of 10%. So now I have a 10% version of my full training dataset and a 10% version of my full test dataset. So what I'm going to do, I'm going to compare, looping this back, Earthly Cloud 39 and Fire Haze 37 on those two 10%, and whichever one wins out of that, I'll tune its hyperparameters a little bit, and then I'll scale it up to use the entire dataset, and we'll be finished modeling. Whoo! That was a fairly— how long has this been going for? Almost maybe 5, 10 minutes, I don't even know. But that is what I've been working on so far. And so by the end of day 32, I'm going to set up the code. By the end of today, I'm going to set up the code to run those experiments that I just mentioned. And then if we come back finally by week 5, by the end of this week, which is a couple of days, we'll train a model with tuned hyperparameters on all the data and make sure all the data is in a GCP bucket so I have ease of access, which is already there. We can actually probably tick this off, but I'll probably put here one last to-do is make sure trained models also live in GCP. Anyway, I've got some experiments to run and I'll catch you up once I've made some progress. Oh, this is some good timing. Check this out. So I'm setting up a couple— it is Friday afternoon, 5 PM, and usually we hit our big breakthroughs on Friday afternoons, but today we haven't. And I want you to just check this out. We've got some inference running. What I'm doing at the moment is I'm running two final quite extended experiments before I train on all of the data and train a big dog model because that's going to take a fairly long time. I would have liked to have had that all done for today, but I kind of ran into some, some more issues such as figuring out what learning rate to use, figuring out how many steps I should use, aka steps through the data, a whole bunch of things. But what I've got set up at the moment, I'm trying to train a Detectron 2 model from scratch, RetinaNet R101. So if we come up here, this is, this is the inference that it's doing. So I've got it set up— let me just show you the little experiment that I've got running. I've got it set up that every 2,000 steps, so about epochs or something like that, it will evaluate its performance on the evaluation set, on the holdout set, with a maximum iteration of 20,000. So every 2,000 steps it's going to evaluate it, and that's when I get this result here. You can probably hear through the microphone, my laptop is going mental right now because I've got two— not one, but two browsers running Detectron2 models. They are running on cloud servers, and I'm also doing some uploading to, to Google Cloud. I've got a whole bunch of things going on right now, but they're all tracked in Weights and Biases. That is the important point. So I don't have to worry about All I have to worry about is making sure they kicked off correctly, and then all of the details will be tracked in Weights and Biases. So this experiment running in Google Colab is a model training from scratch using the config. Where is it? Up here. There we go. This one here. So if we go here to detection2_model_zoo, I'm using the R101, and this was from my preliminary experiments, the ones that I found to be the best. COCO object detection baselines, there we go, R101. So I'm using this, this model here. Now I'm training it from scratch in this Colab notebook. This is the Weights and Biases run here. And then in this JupyterLab notebook, I'm training one on transfer using transfer learning. So using the Detectron2 pre-trained model, model weights. So that was trained on the COCO dataset. So COCO dataset. So this one, what I'm going to do with that is, this is what the pre-trained weights are trained on. What I'm going to do is once these two, they're both running for 20,000 iterations, everything kept the same except for the initialization weights. So what I'm going to do with that is, because I'm having, I read back in the Airbnb article and they said that they had some issues with transfer learning. So I'm thinking, well, maybe I might— in one of my previous experiments running for 10,000 iterations or so, the average precision wasn't improving as much as I'd like it to. It did end up at about 24%, which the goal is to get 50% or higher. Now that was only with 10% of the data, so it might improve even more when we scale up to 100%. But I just want to sort of make sure before I kick off, because I'm only really going to get one shot at this. You know, I'm only going to get really one shot because I've only got a limited amount of time before this project ends to scale up and train on all the data. And I want to avoid using Google's AutoML, which is what Airbnb went to, to go and use. But that, take note, that is where I see the future of deep learning and machine learning going. It's just automate, automate all of this training process automatically with a service like AutoML. It's a shame that Google don't allow you to download the model and/or view the model that AutoML ended up building. Well, I guess if they did allow you to do that, you'd probably be able to reverse engineer AutoML and take away Google— some of Google's revenue. Doesn't matter. I'm going to wait for these two experiments to finish. This one's got probably— it's up to 12,500 iterations. It's going to take probably another at least an hour, whereas this one in Google Colab is not as far along. I think it's only done yet just about 2,500 iterations. So they're not— they're probably going to be finished tonight, maybe. It is Friday, so you know what that means. So around, hopefully around 10 PM or something like that, and then all the results should be tracked in here. And then I'll fill you in over what the direction is based off these two experiments, where I'm going to go next. Hopefully by the end of this video we'll have a fully scaled-up model. You remember the most important point: while my models are moving, I've got to be moving. So I'm gonna let these train, go do a workout, come back and check the progress. Rest. That felt like about a million. Far out. You know, I think it can get about 100 iterations per minute on there. I'm running on a Tesla P1000. GPT-4 GPU. I believe Colab is using the same GPU as my Google Cloud Platform. Colab is free, Google Cloud is paid for. We've got kicksets. Exercise. Would you look at this! Look at how many iterations there are, over 100,000. It should be— I think it's finished. I think the full machine learning model is finished. We're going to scroll right to the bottom over here. You can see Weights and Biases, but let's keep going. I just set it up for 180,000 iterations. 140,000, 150,000, 160,000. Oh, this is some scrolling effort right here. 170,000. Oh, inference done! And there we go, the full machine learning model has finished training. And sadly, we only reached an average precision of 43.297 or 43.3. So we didn't quite reach Airbnb's MVP minimum of 50% mean average precision. And I think I have an idea why. I mean, we could, as you see, this line isn't really going up anymore, sort of flattening out. We could try train our model for longer, but as I said, it's— we're getting towards the end of it now, and we really need to start building some sort of application around the, the model rather than just training the model for longer. I think if we did have the, the double the dataset, so Airbnb's internal images, the, the 30 or 40-something thousand internal images, see We're only working with the 35,000 or so training images from Open Images. If we did have that internal dataset, we might be able to— I think we actually will be able to reach a metric like that Airbnb got from Google's AutoML, which I believe was something like 68.7% average precision or somewhere around that. But we did manage to beat what Airbnb got with their training their own models. So that, that is the takeaway from here. And it took less time. Let's have a look here. They said they trained a model on AWS for 5 days. Remember, they have double the data. So let's say our model would have taken twice the time to actually train. How long did it take? If we come into here, look at that, finished March 29th, 2020, 18 hours, 37 minutes. So let's say we did double that, we get to 36 hours. On a single GPU, a Tesla P100— it doesn't even say it here, does it? Maybe it does. Oh no, we can check here. This is a single GPU instance. So a Tesla P100— I'll figure out how much it actually costs in some sort of write-up, or maybe I'll put a slide over here— for 18 hours. So we did— we not only got better results in average precision, than what Airbnb trained on their AWS instance. We did it in less time and we didn't have to use— we now— what the other benefit is, the trifecta here, the benefit is that now we have access to the model and we can use that. See, I've got some model checkpoints here. So this is the model checkpoint at 180,000 iterations. We can now use that in our own application rather than it being stuck in Google Cloud's AutoML, which is where Airbnb finished up with their article. So we've got some improvements. We're going to go back over them probably in the next video. But now that we have a fully trained model, the next thing to do— I'll probably put it down here, a little note— the next thing is build inference code to test model inference. So I need to build a function to test model inference. And let's come here. It's the end of week 5. Let's get Notion up. What have we got? Pick a model from Weights and Biases. We did that. We ended up going with transfer learning for 180,000 iterations. We could go for longer. We could do some more hyperparameter tuning, which was, which was the next point here. But in the interest of time, we haven't done too much hyperparameter tuning. I've got it all, all the parameters that I used are in the config. So I'll be sharing all of this publicly once the project wraps up. So we can't really say that we did that. We didn't use sweeps, high weights and biases. Maybe that's for next time. Train a model with tuned hyperparameters on all the data. We did do some hyperparameter tuning, but again, it could be, could be extended. Make sure all the data is in a GCP bucket. Make sure the trained models also live on GCP. So that is my next, my next step to do is make sure all of my trained models and artifacts are living in Google Cloud Storage so I can access them later. But as I said, we now have a fully trained model, which is exciting. 18 hours running on a GPU somewhere in, I think it's in Sydney. I live in Brisbane. That's so amazing. Now it's time to, to see what sort of tools that we can build, what little mini application we can build with the trained model and get it deployed. I'm most excited about this. So that's That's week 6, it's the final week. Let's do this, I'll see you next time.