Creating Data Pipelines with Airflow | Design, Execute & Monitor Pipelines

Welcome to the webinar, everyone. Okay, so today we're talking about Airflow and at work there are many situations where you want to run some code repeatedly. For example, if you want to run a weekly report or you want to provide the latest data to a dashboard, and often that means running several scripts in order. So perhaps you have some code for cleaning or processing some data, you have some more code for creating some models and. And you have yet more code for creating some visualizations and all these bits of code in order that forms a data pipeline and then in order to manage the scheduling of them to run in the right order at the right time. That's called orchestration. And this is really fiddly and tedious to do manually. So it's best done with some software like Airflow. Today you're going to learn how to create and run a data pipeline using Airflow. Our guest is Mike Metzger. So welcome, Mike. Mike is a data engineering consultant at Flexible Creations. His work focuses on data engineering and analysis using SQL, Python and Apache Spark. He's got 20 years of experience working in the data networking and security space. Mike is also the instructor of five Data Camp courses, including Introduction to Airflow in Python. Welcome, Mike, over to you. Actually, before we get to that, for everyone in the audience, we are going to be using datacamp datalab today, so please log into that. I'm going to give you a link so you can code along with us. Now I'm going to leave you to it, Mike.

Awesome. Thank you, Richie. Good morning everyone. Basically, to get started, as Richie mentioned, I've developed a course for Introduction to Airflow in Python. It is what I recommend you go through for a lot of detail on exactly how Airflow works and why we do some of the choices that we do. In the meantime, for this intro, what I want to show you is kind of creating a pretty basic data pipeline with where we're going to access data from Amazon aws, which is basically a list of their IP addresses that they use for any of the various services. And then we're going to do a little bit of filtering on that and kind of illustrate how things would work if you wanted to move forward with this. We did set this up so that it would work within the Data Lab environment, so everyone should be able to work along with this if you like. One thing to note, this is a little different than a lot of Data Lab setups because we're using more command line tools than we would otherwise. And I'll kind of illustrate how that works as we, we go through, if there's any questions at any point, please, you know, put them in the chat. We'll, we'll pause as need be and kind of answer things if you're having any problems. I'll try to illustrate a few gotchas here and there for how this works within the environment. And let's go ahead and get started. Okay, so this is the Data Lab workspace for this project. As I mentioned, we're going to create and test an airflow dag, which, if you haven't heard the term DAG before, it basically means a directed acyclic graph. This is extremely common within data engineering scenarios. You'll see it in Airflow, you'll see it in Spark, you see it in various other tools. The important thing to know about a DAG or a directed acyclic graph is basically you have a set of steps that, that have to go in a certain order and you can specify what that order is and the tool will figure out, okay, here's how I need to make all these components run together so that everything works in the way that you expect. For this one, we're doing a pretty simple setup. We'll go through that a little bit more in a minute. But basically the idea is that it's not a looping thing. It's that you go through this process in what's called a DAG run, where it goes through and processes the set of steps that, that you've defined as well. So for this example, we are going to collect a list of IP addresses that are used by Amazon Web Services. This is a file that they post within a JSON format. This very easily could be accessing a web ui. It could be, I'm sorry, not a web ui, a web API. You could be downloading files. Something could be uploaded to a location on a server that you control. There's varying ways to handle what's going on. Basically, we're going to collect that information, run some basic cleanup on that, and then save that data for use in later down the pipeline. We are also going to test the DAG within dareflow and see a few different ways that you can work through those and see how everything behaves. But first, before we start that, we have to actually get airflow set up within the Data Lab environment. If you're running this at the same time, your output may be a little bit different than mine. That's okay. Basically we're just checking to make sure that all of the content returns a green status update on the cell so let's select our first one. The first thing we need to do is actually install Airflow via pip. And before we actually do that, I do want to mention that there are many ways to install Airflow, and there are many ways to use Airflow or its derivatives. Things like Google's code composer is actually based on Airflow. Amazon has one as well. I believe it's mwaa. I can't remember the exact acronym for what that is offhand, but it's their Managed Airflow service. I believe it's actually Managed Workflows with Apache Airflow. But in this circumstance, we're actually just going to install the Airflow Python libraries, which will allow us to create DAGs, use the appropriate code that comes through with Airflow, and we can also test the dags by doing a few other steps that we'll. We'll cover here in a moment. So first we need to actually run PIP install, And this will return quite a bit of output. Basically, this is going to go through and install all the needed components for the Airflow. Installation usually takes about 30, 45 seconds to complete, and we'll review some of the content as we go through. Basically, it's just downloading and installing the assorted packages. Wait for a second because it's just going to refresh. Okay, here we go. So you can see that we're basically collecting the objects that are required, figuring out which packages are needed, and then downloading and installing those. And this effectively installs everything that's in here. You don't have to worry too much about what the the individual components of this are. One note, if you haven't run this notebook before, chances are it will throw some different messages again. That's okay. The Data Lab environment will save different things based on what you've done in the past. So now that we have the base Airflow packages installed, we have to do some environment setup. So I'm going to go run that. This simply just adds where we installed Airflow to our path so that we can run some Airflow commands. Now, Airflow itself, as I mentioned, there are varying ways to run and install this. It can be things like this, where we're running it within a specific packet, running the specific package within our own installation. But you can also set it up for production environments where you have differing systems handling parts of the Airflow components. Airflow itself uses a database to manage all of the DAGs that are listed, handle the scheduling, and work through the like, provide the access to the web ui. There's also an Actual API that you could use in the airflow 2. X environment. We're not covering any of that here. Just note that there are different ways to install Airflow depending on what your needs are. For this purpose, we do have to have a basic database, and that's what this airflow db init command does. This will actually creates a SQLite backend for the airflow configuration. And this is pretty quick. Basically this is just creating a SQLite file with the appropriate tables that are needed. And finally, we have a couple of configuration modifications to make. This just makes Airflow work better within this environment. As a note, you probably not need to do this kind of change within if you're installing Airflow on your own system, it's needed for Data Lab, but it's not necessarily needed for if you're running it locally on your own Linux box or, you know, within a container or something like that. All this is doing is modifying a couple of configuration lines and we'll go ahead and run this. It returns within a few seconds and we should be ready to go. Okay. To actually verify our airflow configuration, we want to run a few commands to check the output, make sure everything is set up correctly. Airflow itself, in addition to being a library service, an API, it's also a shell command that's available within from the command line and allows you to interact within the Airflow environment. And note that for any shell commands that we're running within Data Lab, we do have to proceed it with an exclamation mark. And there's several commands that we can run to actually verify our airflow installation. The first one, and one that I still use to this day, is the airflow help command. This shows us a list of all of the Airflow entries, the commands that are available, etc. So help you see, that shows us basically the usage for airflow. So if you're running these commands on your own, you're going to see things like Airflow space config or Airflow space DB as we did earlier, and then you can have further configurations from there. There's also specific commands that don't have underlying group commands or sub commands. The terminology varies a little bit depending on the version of Airflow you're working with. But the one we're interested in right now is the Airflow version command. And this simply just shows us the version that we've installed. So let's go ahead and run Airflow version, and we see that we have 2.9.2, and this is what was expected for the Type that we installed. Given that we have installed a specific version, this is obviously what we're doing. We were expecting you can actually install Airflow kind of just manually without specifying the version. I don't really recommend that because the way Airflow works does change quite a bit between, not necessarily the, the revisions, but within the minor versions even. There can be quite a bit of configuration change or behavioral change within there. So if you're following a tutorial or you're working through something, I would advise trying to go with the same version of Airflow that's there. Otherwise you may run into a few issues. Okay, before we get too much further, are there any quick questions on setup or configuration?

We don't have any questions from the audience yet, but if. For anyone who's watching, if you do have any questions for Mike, then please do ask them in the chat. Mike, for now, I think you can continue on and if we get some questions, we can pause again later.

Okay, perfect. Okay, as mentioned, the dags within Airflow there, these are the directed acyclic graphs. These are the basis for data pipelines within Airflow. It's how you tell Airflow what you want it to do and how you make it go through and configure everything accordingly. There can be many, many dags per airflow installation. You may write some yourself. You may have them written from colleagues, you may download them from somewhere and modify them accordingly. These dags, it's a very common process for an airflow administrator to run a DAG that they did not write. They may update it accordingly to work within your environment or within, you know, assorted changes. But it's not, you know, it's very common to not necessarily understand everything that happens within a dag. I do also want to mention that while you see the term DAG a lot within data engineering tools and tasks, they don't always mean the same thing. In general, they do, and they. They're defining the set of processes or the set of tasks to do. But the way they're implemented varies quite a bit between tools. For anything that I'm saying here with a in regard to a dag, please just apply that directly to Airflow. It would be different for something based on Spark or things like that, though, amusingly, you can create an airflow DAG that also runs Spark. Dags cover that here, but it is certainly an option. Okay, so we can run a help on a sub command to see what options are available. So let's take a look and see what we have within The Dags tool, and you'll see that we have airflow dags and now our commands have changed. We have things like backfill, delete, list. One that happens and is really useful is the list import errors, but we'll cover that one a little bit later. You can see things like list the jobs that are specified, see reports on it, show information, and so forth. So let's take a look at the DAGs that are available on our system so far. Scroll someone mentioned there was. Okay, so for this configuration we have a simple DAG or a single DAG configured that is called Simple Dag. The. We'll look at the actual DAG in a moment, but I just want to illustrate that this is a way that you can see all of the DAGs that are defined on the system. If you do a normal installation of Airflow without some of the configuration changes we made earlier, you will see probably 40 to 50 dags listed in here that are actually quite interesting. They provide examples on how you do different things within Airflow, including working with Python, working with Bash, working with SQL operators, working with virtual Python environments. There's a lot of different options available in there that don't want to cover here just because it's. It will take quite a bit of time, but it is definitely something to take a look at when you're working with this on your own. Okay, now, to actually run a dag, they've. They've improved this considerably within the later versions of Airflow, but originally you had to kind of set things up to schedule a DAG within the Airflow ui. And the. That's actually another good point. At the moment, we're not covering the Airflow user interface within this tool, partly because that requires access to a different web port to view that information and interact with the system. In the Airflow 2. X versions, they've made it so that you can interact with a lot of the tools from the command line, which is what we're doing here. So we can do our testing without necessarily having to have all of the. The underlying Airflow setup. If you go through the intro to Airflow course, it does cover quite a bit of the Airflow UI and how you interact with that. In addition, all of the tutorials for Airflow will also cover the basics of how do you create those, how do you interact with dags within that environment and see how things are working. But to actually test a command from the command line or test DAG from the command line, we can run a simple command that's basically Airflow Dags test and then the name of the dag. So we're going to run this against a simple DAG and see the output. And note that our DAG ID is the same thing as the DAG name. So in this case it's going to be simple underscore Dag. So let's test this accordingly. Scroll this up because the output gets a little bit long. Okay, so this is the, the output that would normally be stored into log files for your, your DAG information. There's quite a bit in here. Basically when you see this option that says filling up the DAG bag. This is. It's effectively saying, okay, I'm going to execute the stag when it's appropriate. Airflow Scheduler is not absolutely deterministic. It kind of will set run things based on when it has availability and based on the. The scheduling that you've defined. Scheduling is another thing that we go into in a lot of depth within the course. Basically, there are varying ways that you can schedule a DAG within airflow. You can run it, you know, hourly, you can run it monthly, you can run it, you know, on every other Tuesday. It kind of just depends on what your needs are. Ok, so then we have our DAG run id. Basically we're running the simple Dag. We get some information about how it's started, it's configured, and then we see that we actually have our, the test of our DAG starting. We're running the individual task. Tasks within airflow are basically the components of a dag. Again, when we look at code, we'll see this, this will make a lot more sense. But basically it's the individual components that you want the airflow DAG to run as it creates. There's a lot of variables that can be configured to make a DAG more generalized so that it will pass information according to a common one is like the date or the code that's being run against it. Various variables can be passed in with the airflow Dag. That's basically what all this shows. And then we see that we're running an actual command. It's User bin bash. And then we're saying just get date. And then we start seeing the output. In this case we actually have the date according to what we're running. And the main thing that we're looking for here is that we see the task is successful. Then we see the end of the task and see here it's always fun to find. Then we see marking run successful. Obviously, if you have a failure it's going to show a failure scenario. But for this, this is basically showing that our airflow configuration is working. And we have a DAG that is running as expected. To illustrate what the DAG itself looks like, In our Dags folder over here we have the simple underscore dag PY file. And. This is effectively a Python file that is running the. The configures everything that that's needed to run the dag. It's important to note that in general, a DAG is not the tool that is actually transforming data. It is telling Airflow how to do, like what to run in order to do those steps. So for illustration purposes, we're showing a lot of code in these that you wouldn't necessarily see within a DAG file. It'd be more of import it from a library and then run either certain operators or components from there. But to start with, we're actually going to just get some date time options for date time and time delta. And then to actually run a dag, we have to import the Airflow models DAG library. And then we're going to run what's called a BASH operator. An operator is basically the way that you tell Airflow how to do how to interact with something. In this case, there's bash operators, there's email operators, there are Python operators, there's kubernetes operators, there's SQL operators, all sorts of things depending on what you need to do to get information. To actually define our dag, we start with a context manager, which is the with dag, and then we give it a name. For simple dag, we define some arguments, can provide a description. We provide a start date, which is just the date in this case is the date time. You can also provide text objects, a couple other options, and then we define this as DAG. I want to note that this is newer for airflow 2. X. In the earlier versions of Airflow, you had to create the DAG and then assign objects to them manually. This just makes things quite a bit simpler. You may still run into some older code where you see the older style of it. See, okay, so then to actually run our task, we create our task object, which in this case is a BASH operator. We provide the task id, which is basically print date, and then we show the actual BASH command that's run, which in this case is just date. Once that's in place, then at the end of it we don't have to do this, but we're effectively defining the dependencies you'll see in the other DAG that we're going to create in a moment that this can get more interesting. So that's effectively what that previous DAG or that DAG test was executing is showing the output of the dag, the bash operator. All right, let's switch back to our notebook for a minute. I also want to note, if by some chance you're not seeing this file section, you can go into file and you'll see either hide workbook files or show workbook files. That took me a minute to find the other day. All right, let's get back to where we were. Okay, any questions before we create our own?

We have just a couple of questions from the audience so far. And again, if anyone wants to ask a question, please do ask them in the chat. This first question comes from Sinathemba saying, are these dags, are they Python scripts that have different tasks for the whole process? Do you just want to cover exactly what a DAG is and how that relates to Python scripts?

Sure. They've modified this a little bit recently, but effectively, yes, you're going to have a Python script that is defined as a dag, and within each Python you can have. You can actually have multiple. You could create multiple dags within a single script. I wouldn't recommend it just because if you have any troubleshooting, it's going to be fun to try to figure out how to where that problem is and work through those issues. But otherwise it's a pretty clean one to one mapping of your. Define the DAG within that script and then the script is copied to a certain location on the airflow installation and if it's able to read that file, it will go from there.

Nice. Actually. So for dags, I always think it's a really complicated term for just saying it's a tree, like structure. So I think the place most people see this is like in a file system on your computer. So you've got the drive at the top and then you got folders beneath that, and then you got more folders beneath that, and then you got files at the bottom. So the DAG is kind of the same thing, it's a hierarchy, but instead of folders you've got, in this case, you've got Python scripts. So it's like it's doing stuff in order and that's the structure of things. Does that sound about right?

I think that's a great option. Another one that I've used for as an analogy for this is kind of like a recipe. So you could have something where. Let's say you want to make a pie where certain things have to happen in a specific order, but certain things you don't really care about, like what order they happen. Like, you might be able to make the crust separately from the filling and have both of those going on at the same time if you have multiple people or the time to be able to do it. But you have to have both of those ready before you can actually create the piece. So it's just a different way of thinking, kind of the same process of you're just telling airflow what you want to do and it will figure out how best to do it and it is up to you to tell it. Okay, you have to run these before you do these other steps, otherwise you're going to get errors. But it's basically that.

I love that analogy of a recipe. That's very cool. All right, we got one more question. This one is from VU saying I what is staging in etl? So, yeah, do you want to talk about like, different environments? Because you mentioned staging.

So, so it can be, unfortunately, and I think everyone knows this in data engineering especially, terms are, are overloaded. You're going to see a lot of different usage of it. The, the basic version of staging is going to be that you have a development environment, you have a staging environment where you put content to test, say, your processes beforehand. And then maybe from that staging environment that gets loaded into production, and then the production only happens after you've done a set of steps to verify that everything works the way you expect. It can also mean that is your staging environment for where you're going to load your data. So your data might go into a staging setup and it needs to be modified or cleaned before it gets used for by, you know, let's say other people in your organization. You know, it could be like the C level execs need a report, but that has to be staged from the data that you have that, you know, if you handed them, you know, like in the example we're going to do here in a minute, you know, there's thousands upon thousands of entries of these IP addresses. They don't want to see that. They may just want to see like certain, you know, information that's appropriate. So this, the staging thing is a little overloaded. It kind of depends on your context. But that's some common definitions for it.

Okay, that's interesting that there's multiple meetings, but generally it's like, it's just a separate environment where you're going to put your stuff, where you don't necessarily want to break stuff either for your users or let them see stuff they shouldn't be seeing. Okay, cool. Sorry, was that a dramatic over simplification?

No, no, no, no. It's, it's again, it's so dependent on the context of what you're, you're describing. And you know, I can go on for a while on, you know, exactly what you mean. You know, if we want to get into databases, that's a whole different beast. But it's really kind of try to. The best example, I guess I would say, is beyond your definition. Just try to get some context on to where you're seeing that and then work forward from there.

All right, yeah. If you want to provide more context in the chat, then we can maybe take another look at this at the end. All right, so we've got a couple more questions just come in. So this one is from Krishna saying basically just talk through what are operators and can you have like, what do they do? And are there any custom operators?

So operators are. Operators are not the only tasks that are defined within Airflow. There are other ones that we'll actually cover one in a minute, something called a sensor. But basically an operator is a task that you want something to happen. So in our examples, we're using the Python operator, so we're going to run some Python code. We've just used a BASH operator, so that's running a command within Bash. There are SQL operators, there's Spark operators, there's email operators, there's more than I can count. They've kind of, you know, mixed that up accordingly. You can absolutely create your own operator, Especially starting out. I don't know that I recommend that. Unless it's something that you're going to do over and over and over again. And it's helpful to share that with say, another group. Like if you have some custom software or something else that you want done, you can create the own operator and then kind of create parameters to make everything work from there.

All right, nice. Oh, before we go to the next question, just there's a comment on staging from Travis saying the new Data Camp calls on data engineering, provides coverage of staging that I found really helpful. I believe that course is Introduction to Data Versioning with dbc. So that came out recently. Staging is actually also covered in Data Warehousing Concepts as well. And also our introduction to Snowflake Core Course and in fact MLOps deployment and lifecycle. There's about four different courses that cover staging. So yeah, please do have a look at the Data camp, data engineering curriculum. At least one of those courses is going to give you all the answers you need. All right, so one more question then we'll, we'll move on with this. So this comes from Oladipi, sorry, OLO Depot, saying, uh, what's the difference between mlflow and Airflow and which one do you care about when it comes to MLOps?

So I will defer a bit on this. I do not have a lot of experience with MLflow. My understanding of it is that if you have a situation where you're trying to test different models and trying to verify that you have everything in ready for production to actually, you know, verify that a change doesn't either make things worse or, you know, hopefully improves your setup, MLflow is probably the better choice for this because you get a lot more feedback in regard to actually setting that up and making everything work appropriately. Airflow is designed as more. More general tool of. Okay, I have the set of steps that I have to do, so maybe the Airflow side might be more useful for actually creating the training data that you're going to feed into the model of. All right, I have to download this set of API, you know, this content from the API. I have to combine that with a set of databases. I have to then filter out some stuff that, you know, for whatever reason is not meaningful and then kick off from there. So these tools aren't necessarily mutually exclusive. You can absolutely have a DAG and Airflow that would create a source data or something that then mlflow could run the appropriate content to figure out what's happening from there and kind of tie them together. Again, I'm not that familiar with mlflow, so I would. Look, I know we had, There was an MLflow code along at one point and then I think, I can't remember if there's a course available on that yet or not.

Yes, there is an MLflow course. And well, remember that we do have a, we did have an MLflow code log. Yeah. So in general, MLflow is all about managing models. So if you're doing things like a B testing and you've got multiple versions of models that you want to see how they compare, then yeah, that's, that's what MLflow for. So if you, if you're doing machine learning, the. So the clues in the name bit that you want, MLflow, if you're doing ETL stuff, data engineering. Analytics. Engineering, then yeah, Airflow is what you need. All right, so with that we can have more questions at the end, but do you want to carry on with.

With the code long, definitely. Let's create a DAG and see what's involved with this. Okay, so we've successfully gone through Airflow itself. We've tested a dag, we verified our configuration is there. So let's go ahead and actually create one of our own. And I will. Basically what we're going to do here is I've got a template DAG file. We're going to fill out the section, and once we have that filled out, we'll rename that. So I'll pop back here in just a second, go into our DAGS folder. Let's take a look at this. Note that because we don't currently have this labeled as a Python file, Data Lab isn't quite sure what to do with it. So we can just go ahead and say open as a text file. The reason for this is again, just kind of a Data Lab setup. We're trying to make sure that we don't get errors when we're running the Airflow command during this process. We'll rename this to a Python file in a moment, but first we're going to. We need to import our Python components. In this case, we just need the DateTime component, and we're also going to use Python requests to actually access some information. Then we need to actually import our DAG component. So we'll do from Airflow models dag. And this is required anytime that you need the actual DAG object, but you don't. You can create tasks outside of a DAG object, and so you could theoretically have a library of tasks that you then assigned to a. It's not really tasks, it's more of a set of object. You know, the. The operator configurations that you might have. We'll cover that again in a minute. So let's then import the file the Python operator from Airflow Operators. And just to illustrate something, I'm going to skip the next one for just a second, but most of the operators follow a convention. So for the BASH operator, it's actually going to be airflow dot operators bash then import bash operator. And if you see an object that is referencing airflow operators, it pretty much follows this convention. These are included within Airflow itself. There's a whole nother set of operators available that I believe are from modify this sometimes, but it's from airflow contribution. Basically, these cover any kind of operator that you can think of. So there was the question earlier about are the custom operators? You may not necessarily have to write a custom operator, but you may Be able to find a library that references, that handles the operator that you actually want to. You want Airflow to run before you write your own for something, especially if you're talking about working with a specific tool or database format or file format or something like that. You might want to take a look and see. Just do a search for airflow and then operator and whatever that type is. We're also going to use something called a file sensor and I'll explain what a sensor is in a moment. Sensor. So a file sensor basically is a very simple way to just check to see if a file exists. And airflow can be configured to wait until that file is present and go through. And it's more or less just kind of like a pause where you can say, all right, there might be another process that is pushing a file to a location that I care about. And in this case, the file sensor is looking for a file on our local system. This very easily could be something like an S3 sensor where it's actually looking on Amazon's S3, you could look for other types of cloud storage or anything along those lines. As long as there's a sensor for it, once it exists, then continue from there. Okay, so we've done our basic imports, so now we need to actually create our context manager. So we have with Dag, we'll give it the name AWS Data Dag Type. Today we're going to give it a quick description. And we have to give it a start date. There is a lot of nuance for what how airflow handles things based on this component, especially like it doesn't affect us for what we're doing now, but if you are actually trying to create something in production, the start date and the. There's what's called the scheduling interval has a lot of effects on how the system will run your code and it will do things in ways that you do not expect. I do cover this quite a bit in the course. It's again, unfortunately just a little too much to cover here. But in this case we're just going to say start this for today, Then we can reference this as DAG as we need to. Ok, so our next component is we actually have to define a Python function for our callable or, I'm sorry, to tell the Python operator what to do. So in this case we're just creating a Python method. We have a URL that we need for aws. We're simply doing request get for the AWS URL and then we're going to write that out to a location on the file system. This is not necessarily the best way to do this, especially in a production scenario. I would do this in a more generalized fashion, where that directory is kind of passed in as a temporary directory, a staging directory, if you will, and use that accordingly. But it makes it quite a bit more terse as to figuring out what's actually happening on the code. So in this case, we're just going to write it out to a specific location on our file system called source data, and then print a quick note. So this is the actual code that gets run, but we have to tell Airflow that we want to create a task to actually create this. So we have our task name or task Object, which is DataPoolTask, and creates the actual Python operator itself. And that requires two components or two attributes. The first is a task id, and this is what shows up within Airflow as to what it's running. And it should be noted, one of the whole reasons why you want to use Airflow in the first place and create these dags, especially if you've used, you know, like Unix or other tools for a while. So instead of like scheduling something with a cron job or a scheduled task in Windows, you can actually create these settings and Airflow will not only figure out what the dependencies between them are, but will show you where the run is at the time. So you can see, all right, I've run this task, or I've run this task, this task is waiting to run, or hey, this task failed and have different things happen based on that. So what we're doing here is we're kind of putting our work into individual units and telling Airflow, hey, I want you to do this, then I want you to do this, Then I want you to do this. So the task ID basically says the doing the pull AWS data. There's also a way to provide some description information that's visible within the ui. We're not covering that here, but that is an option. But the main part we need is this Python callable, which is simply the method that you want the Python operator to run, and that actually creates our task. So it's not going to do anything yet, but the task itself is actually specified. And then we want to create a file sensor to verify that the file is actually available. This basically handles scenarios where if we're running something in the background that isn't completely done, or we're not sure is going to be ready yet, we can actually create the file sensor to kind of handle things for US and just give it a little bit extra time. Or you can also have a completely different process that would create this file and have the DAG wait at that point. So again, we're going to give this a task id which we have as wait for file and specify the file path. And then finally we're going to use a bash. We're going to use a BASH operator to filter out this file a little bit so that we only see specific entries from the content that we get. Depending on your level of knowledge with Amazon Web Services, there are different regions that they run data in. So to just kind of illustrate this, we're going to filter it for only looking at the US East1 region using a tool called JQ. This is basically the ability to query JSON accordingly. You don't have to worry about what this is actually doing. JQ itself is really interesting. I'd advise you take a look at it. But it's not something I'm going to go into a whole lot of depth here to actually run or create the task. In this case, we're going to do a BASH operator. Again, we have a task id. We'll say filter for. And then we'll do our bash command. And this one I'm copying and pasting. Okay, so once we have the operators actually defined, we have to do. We have to define what's called the task order or the dependency setup. So the first thing we need to do is specify the task that we want to run first. So that's just going to be the data pull task. And then we have to define what's called the downstream task. And airflow has what's called the downstream operator. This is basically the shorthand version of it when you see the two greater than symbols. So we say data pull task and then we're going to do our verify file present and then we again show this for the filtered data. And this basically just specifies that everything's going to run in the specific order. You could also do this in the opposite method where it said something along the lines of filter data, which needs the verify file presence and then the data pull task. If that makes more sense to you, great. The straight, the left right dependency order makes more sense to me. You can use either one. Once you have the airflow configuration, like once you have the airflow UI set up, you can actually view what it's going to do and see, okay, which order will be defined. If you do not define a dependency within there, Airflow will just kind of run the tasks at any step that it can. Dependencies are something we again cover quite a bit in the course, encourage you to take a look at that. But in the meantime, this is everything that we need for the actual code. So let's now rename our file to actually a Python file. And we can go back to our notebook. So now we need to take a look and verify that our syntax is correct. And you can do this within the UI itself. So we can just do something like this Python 3, say Dags, And if we're lucky, then it doesn't actually return anything. And this is what you want. Basically this handles looking for syntax errors within the the Python setup. There's another command we can run if there are problems as well, I.e. airflow tags list import error, So it doesn't actually return anything. Whereas just to illustrate, if I come in here and let's say we mess with our import a little bit, Then we see that we actually have a problem with the specific Dag file and then that there's no module found. Name requests. Put this back. My apologies for bouncing around. I'm just trying to illustrate a few different options here. Okay, so to actually test our Dag again, we have the option of airflow dags test and then the name of the Dag. But before we do that, let's actually verify that our Dag is present. So now we have our two. We have the two dags listed, we have aws data dag and we have our simple dag. So if we want to actually run the airflow dag. Again, we start seeing quite a bit of content. The important thing is we'll start seeing because we have more tasks. In this case, we have the task instance for AWS file written. So the Python component ran here, the file we have our wait for file task id, which ran successfully, Marking tasks as success. And then finally we do our base filter. And this is a set of info. But if we come over here into our data folder, we can see that we have our source data now. I think I just locked it up. Sorry. The JSON file was quite a bit. Quite large. That's basically. While that's running, basically, this is what's happening. We have our appropriate source data and we have our filtered data. It's something that you can look in on the terminal side of it. All right, while that's doing that, I know that was kind of a quick run through. Are there any. What questions can I answer for everyone? Richie, I think you're muted.

Apologies. You think I'd be good at that by now, okay, so because we're coming to time, we're going to run a little bit over. If you have to jump, then please do check out the, the recording for the rest of the Q A before everyone dashes off. Before we do the rest of the QR and A, I would just like to say please do sign up for our Radar virtual conference. There's a QR code at the top left. So this is happening Wednesday and Thursday next week. It is going to be our biggest ever virtual conference. I think we've got like nearly 30, 000 people signed up already and we've got so many like top speakers. So the whole theme is around AI and there's like a, about 10 different CEOs coming along to speak. And then we've got some top speakers, like top authors and things. So it's, it's going to be huge. It's two days of, of AI carnage. Please do sign up for that. And beyond that, I'm just with the QR code. So we have got some upcoming webinars. So please do click on the QR code or take a photo of the QR code and sign up for those. All right, so we got time for a couple more questions. So yeah, let's do two questions and then we'll, we'll wrap up. So Nameless asks, can you talk a bit more about the ETL roadmap? I'm not sure what the roadmap context is here. Oh, is this data cat roadmap you're after? So if you go to. We have a trello board. Let me give you a link to that in the chat and then I'll, I can, you can see what courses are coming up. So I'll send a link to that in the chat in a moment. I'm not sure otherwise. Mike, are there any trends in ETL that you want to talk about? What's happening in the wide world of etl?

What's that?

Easy question, right?

What's that saying, everything old is new? Again, it depends on when you ask. So the. Trying to think of a good option here. The tools vary quite a bit. You know, I see things like just, okay, here's some examples from some of my own clients. There's a lot of things that I have had to do with Spark in order to get tasks run. But as powerful as Spark is, it is one of those things that is kind of a pain to use. So anytime that you can get rid of it honestly works pretty well. And I've noticed that kind of some of the tasks that I created a few years ago to work within Spark don't necessarily have to be run that way. Now you can use things like DuckDB or other tools to run a lot of processes and get a lot of better behavior and performance from the tools than what you would have otherwise. So I guess one thing I would always say with ETL is don't necessarily think that it's done, especially if your data is constantly growing like pretty much everyone's is. It's something to at least, you know, at some point go back and reevaluate. Okay, what have we worked on here? What have we, you know, is there a part of this that we can swap back out and use in a different way or try something different? So, you know, I've been at IT related things for well over 20 years at this point and some things come back around, some things are, are brand new and different. But it's. Yeah, it's the always learning scenario that you have to deal with this to kind of keep on top of things.

As always, Wade, there's always more to learn. One of the reasons Data Camps here. So we always give you some new courses to take. All right, super. Thank you. Let's do one more question. This comes from Martha. So Martha says, does Airflow also work with R code?

So you cannot if you are writing. Okay, let me modify this a little bit. In general, if you're creating dags in Airflow, you have to use Python. However, you can have other tools. I don't know for certain that there's an R operator. It would surprise me if there wasn't. But even if there was was not an R operator, you could also create it where you have a BASH operator that is running the R commands to generate what you want. So there's usually a way within Airflow to make something work and compartmentalize the task down to a certain setup, but it's not going to create the individual. Like you can't create a DAG directly within R. Now you may be able to do something crazy with the, the API. I don't know that I'd recommend that you do that, but I would love to hear about it if it works.

So I can actually confirm that it works. So I don't know how many people know, but Data Camp started off as a purely R training company and so we had a lot of people who were very good at R. And in fact, in the early days, Data Camp's data engineering was built almost entirely around R. We used Airflow to orchestrate all this, but it was running R scripts in the background. I can't remember whether exactly how it works, but you can definitely get Airflow to run R scripts and do all your data engineering that way. The reason we switched over is just because it's very difficult to find enough people who are good at R and data engineers, especially these days, as Python's eaten everything, we had to move on from that. But it is actually possible to run your entire company's data engineering using R and Airflow. So, yeah, certainly possible. All right, wonderful with that. That was brilliant. And I have to say, like, as my experience as a data scientist is like, whenever you give someone like a dashboard or something and they get an answer they don't like, they're like, I think this data might be a bit out of date. And so with Airflow, you can make sure all your code's running and your data is fresh, and so you can avoid that problem entirely. So it's one less complaint from your users to have because all your data is taken care of. That was brilliant, Mike. I really enjoyed that. So thank you so much.

Thank you. I enjoyed the process.

Wonderful. And thank you to everyone in the audience who asked a question. Lots of great questions there, some good technical stuff. I'm impressed, audience. Well done. And thank you to everyone who showed up. I hope to see you all next week for Radar. If you haven't registered already, then please do so now. It's going to be brilliant and I hope to see you all again in future webinars. All right, thank you, everyone, and goodbye.

Goodbye.