In this tutorial, the instructor introduces Singer, an open‑source specification designed to standardize data ingestion pipelines. Singer defines how extraction scripts (taps) and loading scripts (targets) communicate via a JSON‑based message format sent over standard output. The talk explains the three core message types—schema, state, and record—and how they are organized into named streams, allowing logical partitioning of data (e.g., per table or error channel). The speaker details how to describe a stream’s data using a valid JSON Schema, including field types, constraints, and optional ID/schema identifiers, and how to wrap that schema into a Singer schema message with the write_schema function, specifying stream name and primary‑key fields. Finally, the tutorial covers Python’s JSON module, contrasting json.dumps (string output) with json.dump (file output), and shows how these tools serialize Singer messages for downstream consumption.
Chapters
Hey, welcome back. Now that we know a bit about major concepts in a data platform, we should learn about getting data into its data lake. We'll be exploring Singer, an open source library that can connect with many data sources. We spoke of data pipelines in the previous lesson. They bring data from one place to another like water and gas pipes. To get data into your data lake, at some moment you you need to ingest it. There are several ways to do so, but it is convenient if within an organizational unit the process is standardized. That is the aim of Singer as well to be the open source standard for writing scripts that move data. At its core, Singer is a specification that describes how data extraction scripts and data loading scripts should communicate using a standard JSON based format over standard out. JSON is similar to Python dictionaries, and standard out is a standardized location to which programs write their output. Because Singer is a specification, these extraction scripts, which are called tabs, and the loading scripts, which are called targets, can be written in any programming language and they can easily be mixed and matched to create small data pipelines that move data from one place to another. Taps and targets communicate using three kinds of schema, state and record, which are sent to and read from specific streams. A stream is a named virtual location to which you send messages that can be picked up at a downstream location. We can use different streams to partition data based on the topic. For example, error messages would go to an error stream, and data from different database tables could go to different streams as well. Imagine you would need to pass this set of data to a process with a Singer spec. You would first describe the data by specifying its schema. The schema should be given as a valid JSON schema, which is another specification that allows you to annotate and even validate structured data. You specify the data type of each property or field. You could also impose constraints like stating that the age should be an integer value between 1 and 130, as we've done here, or that a phone number should be in a certain format. The last two keys in this JSON object are the ID and schema. They allow you to uniquely specify this schema within your organization and tells others which version of JSON schema is being used. They're optional, but highly recommended in production grade code. You can tell the Singer library to make a schema message out of this JSON schema. You would call Singer's Write schema function, passing it the JSON schema we defined earlier. With stream name, you specify the name of the stream. This message belongs to. This can be anything you want. Data that belongs together should be sent to the same stream. The key properties attribute should equal a list of strings that make up the primary key for records from the stream. If you're processing data from motorized vehicles on a parking lot, for example, this could be the license plate or a surrogate key. If there is no primary key, then specify an empty list. As you can see, the Write schema call simply wraps the actual JSON schema into a new JSON message and adds a few attributes. The message will be printed on a single line, but we've added line breaks here to fit the screen. JSON is a common format, not just in Singer but in many other places. Python provides the JSON module to work with JSON. To get objects in your code serialized as JSON, you would call either JSON dumps or JSON Dump. The former simply transforms the object to a string, whereas the latter writes that same string to a file. Okay, now that we understand what JSON schema is and how Singer wraps that into messages, let's put this in.
