In this introductory video, the instructor outlines the fundamental responsibilities of a data engineer and the primary categories of tools they use. The data engineer’s workflow follows an ETL (Extract‑Transform‑Load) pattern: extracting data from various sources (databases, APIs, files), transforming it using parallel processing frameworks, and loading it into analytical databases. The talk emphasizes three tool families: (1) database systems (SQL and NoSQL, e.g., MySQL, PostgreSQL) that store operational and analytical data; (2) parallel processing and cluster computing tools (Spark, PySpark, Hive) that provide pandas‑like APIs while abstracting distributed execution; and (3) scheduling/orchestration platforms (Apache Airflow, Uzi, Cron) that ensure jobs run in the correct order and at the right intervals. The speaker notes the importance of understanding tool abstractions and limitations, and acknowledges that organizations may use alternative or custom solutions. The video serves as a high‑level roadmap before deeper dives into each tool category in later chapters.
Chapters
Hello again. Great job on the exercises. You should now have a good understanding of what it means to be a data engineer. The data engineer moves data from several sources, processes or cleans it, and finally loads it into an analytical database. They do this using several tools. This video acts as an overview to get a feeling for for how data engineers fulfill their tasks using these tools. We'll spend some more time to go into the details in the second chapter. First, data engineers are expert users of database systems. Roughly speaking, a database is a computer system that holds large amounts of data. You might have heard of SQL or NoSQL databases. If not, there are some excellent courses on datacamp on these subjects. Often, applications rely on databases to provide certain functionality. For example, in an online store, a database holds product data like prices or amount in stock. On the other hand, other databases hold data specifically for analysis. You'll find out more about the difference in later chapters. For now, it's essential to understand that the data engineer's task begins and ends at databases. Second, data engineers use tools that can help them quickly process data. Processing data might be necessary to clean or aggregate data or to join it together from different sources. Typically, huge amounts of data have to be processed. That is where parallel processing comes into play. Instead of processing the data on one computer, data engineers use clusters of machines to process the data. Often, these tools make an abstraction of the underlying architecture and have a simple API. For example, have a look at this code. It looks a lot like simple pandas, filter or count operations. However, behind the curtains, a cluster of computers could be performing these operations using the Pyspark framework. We'll get into the details of different parallel processing frameworks later, but a good data engineer understands these abstractions and knows their limitations. Third, scheduling tools help to make sure data moves from one place to another at the correct time with a specific interval. Data engineers make sure these jobs run in a timely fashion and that they are run in the right order. Sometimes processing jobs need to run in a particular order to function correctly. For example, tables from two databases might need to be joined together after they are both cleaned. In the following diagram, the join product order job needs to run after clean product and clean order ran. Luckily, all of these tools are so common that there is a lot of choice in deciding which ones to use. In this slide, I'll present a few examples of each kind of tool. Please keep in mind that this list is not exhaustive and that some companies might choose to build their own tools in house. Two examples of databases are MySQL or PostgreSQL. An example processing tool is Spark or Hive. Finally, for scheduling we can use Apache Airflow, Uzi, or we can use a simple bash tool, Cron. To sum everything up, you can think of the data engineering pipeline through this diagram. It extracts all data through connections with several databases, transforms it using a cluster computing framework like Spark, and loads it into an analytical database. Also, everything is scheduled to run in a specific order through a scheduling framework like Airflow. A small side note here is that the sources can be external APIs or other file formats too. We'll see this in the exercises.
