Python Tutorial: PySpark: Spark with Python

In the last video you were introduced to Apache Spark, which is a fast and general purpose framework for big data processing. Apache Spark provides high level APIs in Scala, Java, Python and R. In this video you'll learn about pyspark, which is Spark's version of Python. Apache Spark is originally written in Scala programming language to support Python. With Spark high, Spark was developed. Unlike previous versions, the newest version of PySpark provides computation power similar to Scala. APIs in PySpark are similar to Pandas and SciKit learn Python packages. Thus, the entry level barrier to PySpark is very low for beginners. Spark comes with interactive shells that enable ad hoc data analysis. Spark shell is an interactive environment through which one can access Spark's functionality quickly and conveniently. Spark Shell is particularly helpful for fast interactive prototyping before running the jobs and clusters. Unlike most other shells, Spark Shell allows you to interact with data that is distributed on disk or in memory across many machines and Spark takes care of automatically distributing this processing. Spark provides the shell in three programming languages. Spark shell for Scala, Pyspark for Python, and Sparker for R. Pyspark shell is a Python based command line tool to develop Spark's interactive applications in Python. PySpark helps data scientists interface with Spark data structures in Apache, Spark and python. Similar to ScholarShell, PySpark Shell has been augmented to support connecting to a cluster. In this course, you will use pyspark shell. In order to interact with Spark using Pyspark shell, you need an entry point. SparkContext is an entry point to interact with underlying Spark functionality. Before understanding Spark context, let's understand what an entry point is. An entry point is where control is transferred from the operating system to the provided program. In simpler terms, it's like a key to your house. Without the key you cannot enter the house. Similarly, without an entry point, you cannot run any Pyspark jobs. You can access the Spark context in the Pyspark shell as a variable named sc. Now let's take a look at some of the important attributes of SparkContext. The first is the version. This attribute shows the version of Spark that you're currently running. In this example, SC version shows that the version of Spark is 2.3.1. The second is the Python version. This attribute shows the version of Python that Spark is currently using. In this example, SC Python var shows that Spark is using Python version 3.6. The final attribute is the master. Master is the URL of the cluster. A local string run in local mode. In this example, SC master returns local asterisk, meaning the spark context acts as a master on a local node using all available threads on the computer where it is running. You can load your raw data into PySpark using SparkContext by two different methods. The first is the SparkContext parallelize method and a list. For example, here is how to create parallelized collections holding the numbers 1 to 5. The second is the SparkContext text file method and a file. For example, here is a way to load a text file named test.txt using sparkcontext file method. Now that you understand Pyspark, let's write your first spark code in.