A tutorial on defining a machine learning problem before building a model. You're predicting home sale prices from 50 attributes like location, size, and amenities using 2017 sales data from one region. The video shows how to load data in PySpark and verify it with basic checks like row counts and data types.
Chapters
What's the point of doing an analysis if you aren't solving the right problem? In this video, we'll define our problem in the context of our data. We are going to build a model to predict how much a house sells for. This question can be interpreted multiple ways, which is why it's important to take the time to formally define it. Let's assume we are real estate tycoons looking for the next best investment opportunity for a given house on the market. With a listed price and a series of attributes describing the home, what is it likely to actually sell for, AKA the sales close price? The data set we have is a sample of homes that sold over the course of 2017. Using this sample, we are to provide a quick proof of concept of whether it's worth investing in more data for the 5.5 million homes that sold in the US in 2017. To do this, we need to understand some of the limitations of the data we have. First, we only have a small geographical area, so to apply our model to new areas poses serious risk. We know that we only have residential data, so we shouldn't expect to predict how much a business location is worth. Lastly, we only have one year's worth of data, which will make it hard to draw strong conclusions or about the seasonality in the data set. The original data set has hundreds of attributes available, but in order to start simple, we've already worked with our client to identify around 50 attributes they think are likely to influence the price of a home. These attributes generally fall into these groups. For dates, we have date listed and the year the home was built. For locational data, we have the city that the home is in, its school district, and its actual postal address. We also have many different metrics to gauge the size of a home, like number of bed and bathrooms, as well as the area of living space. For prices we have the listing price and we wouldn't be able to predict anything without the sales price. We also have a lot of data available on the amenities that a house has, like a pool or a garage, as well as the construction materials that were used to build the house. Big data means a lot can go wrong. When loading data, make sure you have the right number of records and columns. We can use dfcount to get the row count, dfcolumns to get the list of columns, and we can take the length of DF columns to get the number of columns. When we used parquet, it set the data types for all of our fields, which is a huge advantage over CSV. It's still worth checking, especially if you weren't the one defining it. Here we can use dtypes on our dataframe to create a list of tuples containing a column name and its corresponding data type. In this video, we learned about the data set we will be using and the problem we will be trying to solve. Additionally, we learned how to check to see if our data loaded properly by checking rows, columns, and data types. Now it's your turn to apply what you've learned in the exercises to verify our data.
