John Hogue covers the basics of PySpark, starting with loading Parquet files into DataFrames. He emphasizes feature engineering, matching Spark and Python versions, and why Parquet's columnar format reads faster than CSV. Data science is iterative: explore, set goals, handle uncertainty.
Chapters
Hi, I'm John Hogue and welcome to Feature Engineering with Pyspark. Easily one of the most important aspects of applied machine learning is feature engineering. It is the process of using domain knowledge to create new features to help our models perform better. In this course, we will look at a real data set and work our way to building a regression model in Pyspark. Before we dive in, it's important to note that while the techniques you'll learn in this course are invaluable, that data science cannot be applied as a cookie cutter. You'll need to research your data and become your own expert. There's much to be said of the dangers of not understanding your data, especially where our outputs are increasingly being used to make decisions and inform policies. Before you dive into modeling, spend time to define what your goals are and how the output might be used. Take time to research your data and its limitations. Oftentimes you may be tasked with explaining what is and isn't possible. Lastly, remember that data science is about being curious, asking questions, and applying new ways to solve problems. Every project and dataset is different. Data science is an iterative process that requires comfort with uncertainty, as at any point you may have to go backward or even start over. A good project may inspire further questions that set the goals for your next project. As we progress through this process, this course will have extra emphasis on a lot of the art sides of data science, exploring data, cleaning IT and engineering it for use in a model. Before we get started as a cutting edge technology, Spark changes fast and frequently. Make sure you're looking at the right version. You can always go to the latest URL by using the latest or put the version number, major, minor and patch to get a specific version. Programmatically, you can check your version of Spark with these commands. That way you can ensure you are looking at the right documentation and not using deprecated methods. For this course, we will be using a parquet file like most data in Hadoop, the platform that Spark runs on. It is write once, read many times format. Parquet is columnar, meaning that it is organized by columns, an important feature for huge data sets, as it is blazingly fast to read in only the data you need. CSVs, on the other hand, have to read and parse the whole dataset to read a single field. Another difference is parquet fields are defined and typed, saving users from defining data types like dates, booleans, or strings. For this reason, parquet is relatively slow to write since it's not delimited by characters. It's less likely to be right and wrong if those characters exist in the data. These are just a few of the advantages that are causing the industry to adopt Parquet quickly. We have many format readers to choose from for converting various file types to a Pyspark dataframe. Here we will use Spark ReadParquet and put the results into variable df representing a data frame. In this video, we covered off on some important considerations when starting any data science project. We also learned about Parquet and and how to load it to a Spark data frame. In the exercises, you'll verify the versioning of Pyspark and Python, and finally, you'll.
