Spark DataFrames are immutable: each transformation creates a new logical plan instead of changing data in place. Spark doesn't actually compute anything until you call an action like count, at which point it optimizes all the accumulated steps and runs them together across the cluster.
Chapters
Welcome back. We've had a quick discussion about data cleaning, data types and schemas. Let's move on to some further Spark concepts Immutability and Lazy Processing Normally in Python and most other languages, variables are fully mutable. The values can be changed at any given time, assuming the scope of the variable is valid. While very flexible, this does present problems anytime there are multiple concurrent components trying to modify the same data. Most languages work around these issues using constructs like mutexes, semaphores, etc. This can add complexity, especially with non trivial programs. Unlike typical Python variables, SPARK dataframes are immutable. While not strictly required, immutability is often a component of functional programming. We won't go into everything that implies here, but understand that Spark is designed to use immutable objects. Practically, this means SPARK dataframes are defined once and are not modifiable. After initialization, if the variable name is reused, the original data is removed, assuming it's not in use elsewhere and the variable name is reassigned to the new data. While this seems inefficient, it actually allows Spark to share data between all cluster components. It can do so without worrying about concurrent data objects. This is a quick example of the immutability of data frames in Spark. It's okay if you don't understand the actual code. This example is more about the concepts of what happens. First, we create a dataframe from a CSV file called VoterData CSV. This creates a new dataframe definition and assigns it to the variable name voterdf. Once created, we want to do two further operations. The first is to create a full year column by using a two digit year present in the dataset and adding 2000 to each entry. This does not actually change the dataframe at all. It copies the original definition, adds the transformation, and assigns it to the voterdf variable name. Our second operation is similar. Now we want to drop the original year column from the dataframe again. This copies the definition, adds a transformation, and reassigns the variable name to this new object. The original objects are destroyed. Please note that the original year column is now permanently gone from this instance, though not from the underlying data. For example, you could simply reload it into a new dataframe if desired. You may be wondering how Spark does this so quickly, especially on large datasets. Spark can do this because of something called lazy processing. Lazy processing in Spark is the idea that very little actually happens until an action is performed. In our previous example, we read a CSV file, added a new column, and deleted another. The trick is that no data was actually read, added or modified. We only updated the instructions, AKA transformations for what we wanted SPARK to do. This functionality allows SPARK to perform the most efficient set of operations to get the desired result. The code example is the same as the previous slide, but with the added count method call. This classifies as an action in SPARK and will process all the transformation operations. These concepts can be a little tricky to grasp without some examples. Let's practice these ideas.
