Kaggle Grandmaster Yaojian Babahin walks through the three-stage pipeline for competing on Kaggle: problem definition, model development, and leaderboard evaluation. Using the NYC Taxi Fare Prediction competition, he shows how to download data, identify the target variable and features, and understand the submission format.
Chapters
Hi all, welcome to the course on Kaggle competitions. In this course you will develop the overall pipeline for successful participation in machine learning competitions. Also, you will learn some practical tips and tricks that can be used in any machine learning project. I will be your instructor for this course. My name is Yaojian Babahin. I have a Master's degree in Applied Data Analysis and over five years of working experience in Data Science. I am also a Kaggle Competitions Grandmaster, having gold medals in both classic machine learning and deep learning competitions. First of all, let's discuss what Kaggle actually is. Kaggle is a web platform for data science and machine learning competitions. It allows us to solve data science challenges and compete with other participants in building the best predictive models. The list of Kaggle benefits is pretty long. Note that this platform could be useful for everyone from beginners in data science to experienced professionals. We could get practical skills working with real world datasets, develop own pet projects, meet and grow with a great Kaggle community, get experience in new domain or model type and also keep up to date with the best performing machine learning methods. The general competition process consists of three major stages. Firstly, Kaggle gives us a problem definition and data to resolve this problem. Then we are developing a machine learning model and preparing the submission file that is uploaded to Kaggle. Finally, our submission is shown on the so called leaderboard together with a position relative to other competitors. To start competing on Kaggle, we should perform three simple steps. Firstly, go to the Kaggle website and select any active competition we are interested in. Then download the data available in the competition. That's it. Now we are ready to start exploring the data and build machine learning models. As an example, we will work with a past Kaggle playground competition called New York City Tax Affair Prediction. The goal of this challenge is to predict the fair amount for a taxi ride in New York City given the pickup and drop off locations. The typical data structure in Kaggle competitions consists of two major train and test datasets. Our goal is to prepare a model on the train dataset given some labels. Afterwards we should make predictions on the test set. Lets read the train dataset from New York Taxi Competition using Pandas Library and look at the columns available there. The first column is an ID variable called Key. The fare amount is the target variable we'd like to predict and the rest of the columns are features we could use to build the model. Now let's move on to the test set. It has the same list of columns except for the fair amount. As this is the column we should predict after we have built a model, we could make predictions on the test set and save them as a CSV file. This CSV file could be submitted to Kaggle. Every Kaggle competition provides a sample submission file. This file shows the correct format and structure of the submission. Let's take a look at the head of the sample submission in the Taxa Fair prediction challenge. As expected, it consists of two columns, the ad column and fair amount we are predicting. Alright, now let's explore train and test datasets from an.
