Learn to load CSV files into Spark DataFrames, convert them to SQL views, and query them with spark.sql. The tutorial covers schema inspection with SHOW COLUMNS and DESCRIBE, and explains how Spark distributes data across servers to handle large datasets.
Chapters
Hello and welcome to this lesson about Spark SQL Spark provides a programming abstraction called Dataframes and can also act as a distributed SQL query engine. In this lesson, we will create a SQL table from a dataframe and then query it to load a comma separated file into a dataframe called df. Let df spark read CSV where the first argument gives the name of the file. If the first line of the data gives the column names, set the header argument to true. Now, to create a SPARK table from this data frame, do df createor replacetempview with the first argument giving the desired name of the table. Next, to run a query, do SPARK SQL with the first argument giving the query to inspect the column names of this table. Set result SPARK SQL with the first argument set to the string show columns from table. Another way is to use the query select from table limit 0. Yet another way is describe table query. To see the columns of result SPARK SQL query, we can either do result show and visually inspect the result or do print result columns. The dataframe is a fundamental data abstraction in spark. A SPARK dataframe is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database, also called simply tabular data. We could have two dataframes having the same types of columns and containing different data. We could then concatenate the rows of data in these two tables into a single dataframe. Recall that a SPARK dataframe is a distributed collection of data organized into named columns. What do we mean by distributed? SPARK can split this dataset into parts, then store each part on a different server. In this case, SPARK is partitioning the data and distributing it automatically on our behalf. This is one technique that SPARK uses to handle large data sets, even though each server may not have enough storage to hold the entire data set on its own. What's more, SPARK allows us to treat a dataframe like a table and query it using SQL. SQL stands for structured query language. A query tells the computer what to fetch. What's useful about the SPARK SQL table is that it allows us to take the data set that is in a data frame, namely a distributed collection of rows having named columns, and treat it as a single table and fetch data from it using an SQL query. We often use an instance of a sparksession object. By convention, this is provided in a variable called Spark. Some implementations of Spark, such as PySpark Shell, automatically provide an instance of a SparkSess. The following Spark command reads delimited text data into a data frame from a file one of its options allows it to use the first row to define the names of the columns. It automatically splits each row into columns using the delimiter, which by fault is a comma, but which can be changed. Let's load some data into a dataframe, convert it into an SQL table, and query it.
