Full Stack Data Engineering with Python | Code Along Using Python!

Hello everyone. So we're going to talk about this full stack data engineering. I'd like to share with you my. Yes, my screen. So let's go. It's a full stack with Python. We're talking about some analysis. And this is me, I'm Blender, I'm from Brazil. More specifically my city is sifi. Right. Right now I'm a senior data scientist and I have a master in applying computing. So what are we going to talk about? So the data we're going to deal with is the data that comes from LIGO detectors. I'm going to talk about Lego better. We're going to try to manipulate HDFI files with Python. We're going to extract transform this data. Secondly, I'm going to show you how to store some different type of files inside those different type of data inside that files and why are we dealing with that kind of file, etc. But we're going to also talk about something very scientific which are gravitational waves. But science and exploration in this data is available and we can explore a lot of it. It's free, it's open, you can always go look into it if you have like the mean of do a research or just do a research in your garage too. So let's go. I'm going to talk. What are those files? What are gravitational waves? A little bit. Not that, not that much a little bit. Where to find the data that I'm going to talk about how to manipulate. You know this file is in Python again and I'm going to show you how to store in data frames inside it. And if you have anything else to ask, I'm. I'm here for you. So what are those files? You know those are HDF 5 stands for hierarchical. Hierarchical data format version 5. So you can store a big quantity of data and it's Ethereum status. So you can have no data text files and you can have binaries and you can store like image. You can have data sets and you can have your metadata also into it and it's very easy to share, it's just the file itself. Also if you add a data set in that group over here and you want to add another data set in another group inside that file, it's going to be very easy. The new data does not no touch in the all the thing that is over there looks like you have like a full sets of folders inside it. You're gonna see it and who does use it? You know, it's using academic research, aerospace, oilcast in a bunch of Industries you can find it. And how is this organized? So this is HDFV or you can go to the website and download it. I'm using is for Windows so you just have to register. And I did a screenshot. So basically over there it looks like we open a folder in our computers. So we have a lot of data types. We have different folders. We call it groups. So we have the data sets, we have groups. So data sets data, they resembles the arrays collections, anything you want to put on. And the groups looks acts more like folders. It also can contain attributes that we can call them like we use like a metadata for it and provides additional information about the data set. You can also access those attributes with key values going to show you that too. And we can store tables, that's how we're going to store our data frames over there. That is very similar to data set but you can have the structures of rows and columns. So the data we are going to do here are gravitational wave thetas, the signals theta low gravitational waves itself. So what are those right? What are those gravitational waves stuff? Gravitational waves is a very exciting subject. It's a phenomena that produce for every accelerated and no perfect spherical object. So right now when I'm moving my hand, I'm having. I'm actually doing gravitational waves. But I cannot feel it, I cannot see it. I don't have the instruments to feel it right now because they are very, very, very small. They are in the scale of minus 21. You know, the atoms is the square from minus 9. So which is really really small. I can even compare it anything else. So there can be compared with the propagation of, you know, throwing a rock and a lake. If you throw a rock in the lake you're going to see the small wave is being formed around it and then they disappear. They. But they do not need a medium to be propagated. And actually they are the band of space time. It was, you know discover. It was written described by Einstein and those on his relativity studies more than 100 years ago. But the first detection, the first direct detection only happened 2015 by the LIGO laboratories which is very, very interesting. Like 100 years after. And I can show you a little bit. I'll have a video over here. Let me show you. This is a simulation. So this is a simulation of the phenomenon that was detected directly detected signal which is two black holes just spinning around each other. And they are really really fast. They are really really big. And they are able to bend the space time. So if it goes there's gonna be a moment then gonna go collide. It's a little slow, sorry for that. They're going to collide. They're going to faster, faster, faster, faster, faster and they become only one. And so those waves that are quite big right now, then they're going to calm down and that's it. Those are gravitational waves. So why are we looking into those big things, you know, black hole collisions and not for just my hand moving over here because as I said, the magnitude of it is quite, quite small. And we need some event that is huge so we can detect something that is so small. So as big as the event, as high as going to be under the magnitude of it. I can also tell you those scientists research, let's come back and the lab itself, I want to talk about the lab itself. It's where this data came from. The data we're going to do is for the first detection. We had orders, you know, from 2015 to here we have others. But this lab, it's in the United States of America. We had our Hanford lab and the state of Washington and Livestone in Louisiana. Again the data recovered. The first detection it led in 2017 the Nobel Prize office for the three main scientists over there. Okay, so where to find the data that I'm talking about? It's a free open data you can use. It's a collaboration. We have the gravitational wave open science center you can access with stage URL. It's a collaboration between four different labs. LIGO, FICO, JUS 600 and CAT, KGRA I guess. So you have this streamlane for multiple observations that happen and it's unified in just one place. So for that data specifically you go in the portal is more like this one and you can download the file over here. HDE is the file that we are talking about. Again. So let's do the coding for this section. What I'm gonna try to show you guys is basically we're going to get the data, we're going to see how is the data organized inside the file and then we're going to use the strain, the streaming data and we're going to apply some processing, we're going to apply some whitening and bad pass filter. Don't worry about the calculations of it. I know it's a lot, I don't even remember that much. But I'm trying to explain what I've been doing, applying those methods inside the data and we're going to see the final result. That's going to be the data after we do the fine tune of it. Okay. For that notebook itself, it's here or set up. I'm going to import some libs. I'm going to import numpy, I'm going to import the scipy like lib2 import signal interpolate just over here, you know, see we can have a whole section just talking about it. But matplotlib, so we, we can see it. And finally the lib that we have that help us to, you know, deal with those files. Okay, I'm gonna run here if it's everything fine? Yeah, it ran fine. And I have two files. Let's see if I have the files here. Okay, I have, I have three actually, but the other one I'm gonna use in down below. So those files are from the detection. So they are 4 kilohertz. So what happened is 4096 dots each second. Okay, so each second is going to have 4,096 dots. And the first one here we have for the L1 I'm going to call it for the limestone lab and H1 is going to be for the handful lab. So LIGO has two labs. Actually physically there's only one lab but has two detectors. So they are very good for comparing if the event happens and one has happened in the other one. So we compare both of those data. Okay, I can explain that more specifically. So we can see the keys, the names of the groups. Okay, back to this, the files. So how to get the file you call the lib h5 file. The name of the file and the R is for reading. Now we want to see what are the groups that we have on the file. So let's see how we can do it. So I'm going to get just the L1. Okay. L1 kiss. Oh, I didn't run that one, sorry. Okay, now we can see the keys. There are the groups that we have inside it. We have meta, we have quality and we have a string. Okay. To access the groups you have to pass a name inside brackets. It's like we're dealing with the name dictionaries. Dictionaries. Okay, so I'm going to go for string because I know because I look at before L1. Sorry for that. I look at before and strains has another group that has a data set is where we want to get the data. Strings, keys. So we have a string inside it. So it's going to be like access library, dictionary inside dictionary, copy past. Over here I'm going to have string strain. Looks like it's automatically called for. String. Okay, so I have a data set inside of it here. Data set string. The shape is 131,000. So if we get into a data set availables, if you want to get availables, you can access ice. It is numpy array. Okay, so I'm just going to put the bricks L1, let me copy here. Paste and the data we can see there right now. See, you can see there. So since I have those, those are the time series that I want to look into the signal, the detection. I'm going to put those inside a variable just for make my life easier. I'm going to call this train underline L1 for this one. And string H1. I don't even call those underlines, but hopefully this. Okay, so now I have those two variables. Let's look into those two variables. They're just being numpy arrays. Okay, I'm going to look at the types strain L1 and I'm going to also going to look at the. We can look at the shape. String L1 shape. See, it's a numpy array. The data set was actually a numpy array inside. So we can have many, many, many data sets inside of it. Many numpy arrays inside of it. Managing. You have a collection of time series that you want to store. You can just go ahead and do it another way. We have like attributes. So inside that group called strain we have attributes. So for access to attributes and get actually the names of them, we go ATTRS keys. So we have where do you want to access the attributes? And you have the keys would be the names of them. Okay, so we have those endpoints, X labels, etc. And to actually see the attributes, you go for dot ttrs and you pass the name of it. So I'm gonna put like the X spacing over here. Okay, so if I'm, if I'm correct, the X spacing is the space time between each dot. So each number that we have inside those numpy arrays. So every time we, so 0.0002 seconds, you get 1.11 little bit of signal. So for the task one, right now we're going to organize all time and start to plot, you know, this strain. So what we're going to do, since we do not have the timing, I look into it. We don't have the time. Inside our file we calculate is using the available information. So the available information is. Let me put the attributes here so you guys can see the available information correctly. For in. I'm trying to do the same thing that I did before, Maybe like that. Print. And we're going to print the attributes, we're going to print all the attribute values with that. Let's see if it works. Okay, so we have the start time here, we have the spacing. One thing that I didn't know if I told you in the beginning of the beginning, the statutory here means it last 32 seconds. Okay, so do the 32 seconds. And each second has 4 kilohertz. So how do you figure out the timing? So we want the data, we know that the data has 32 seconds. Oh, it's 32, not 23 and 32 with 4K frequency. What means we have 4096 dots data points for each of the second. The GPS time start is this one. So in storage, this attribute over here, string, string again, attribute x start. Okay, so we know the spacing for each of the time. The interval for each of the dots that we have is that one over here. Okay, so forget the time. The time is going to be important for us a little bit later to get the time, we can have the start time, the end time, and we need the time for each of the dots. So what do we have here? We can have arrange array in numpy. So let's do it. Okay, start time is equals output here it's going to be the start time attribute. Start time attribute. On the end time, it's going to be the same thing plus 32. And the spacing is the spacing attribute. It's that one over here. I'm going to put in another variable just to make it easier to read, copy, paste. And the variable is going to be this one. Okay, so for the time np arrange, so we have, we will have the array start time. And time and spacing. Let's run. Okay, according to the waveformation we have previously me looking around, the phenomenon lasts for half of a second and it occurs exactly at that moment. You know, September 14, 2015, 9, 50, 45 GMT. So that's the GPS time, this dot 443. I actually don't know how to explain that to you. Dot 443. Oh my gosh. But this is the time, this corrector time. Okay, so we're gonna, we're gonna tell it, call it mid, like it's exactly time. And for the moment that second happens is the moment we start the the event. So if we put the time and compare the time with this train that we have, we have the time, we have the array of the times. And we have array of the dots. We have it for the same, the same length. And if we find where in the time the event started, we will find where in the strain in the dots the event started. So we can look into it. Okay, because we have 32 seconds. But we are looking for just half of a second. So to find it I'm gonna go for mid NPY and where time is bigger than bigger vehicles. Okay, can it's going to be array. So to get this array. Let me just print it for you. Okay, so we start in 63,000. Okay, I want to find this exactly moment here put 00. I could, I could just call it start event with a book need. Okay, so I want to see the event now inside those 32 seconds I want to find that moment the six tree. But I want to see two seconds before and two seconds after. Just for, you know, just for the sake of it. Let me see if I can undo it. Okay, let's just comment. This one, This one. Okay, so we have here the start time to visualize timing. And we have this train that we want to visualize. It's already copied on your notebook. This string with the start time and the end time. The start time is going to be start time. This equals to mid minus 2 seconds. 2 seconds R4000 4000 dots plus and int time. This equals mid plus 4096. It's going to be an integer like the end of the array. This beginning of the array and the end of the array. So let's print to see roll data in the middle would be, you know, the time series itself. But as you can see, that's not the graphic we're looking into. It's a signal. It's the signal, but it needs some more filter some more things to happen to. We can actually see the very beautiful wave we're looking for. Okay. So that's when the task two comes. So we're going to actually use whitening that is like we have a lot of noise in this event. Okay. So we're gonna try to make the noise look the same everywhere. And we're gonna make it even harder for the, you know, especially for the low and the high frequencies and astrophysics and whitening is crucial and because it is country state significant data amount of data without prior knowledge of noise noise sources. So after whitening calls something called sigmas. So indicator of how far from the average is is the, is the actual data that we're looking at you. And also we're going to do Something way simpler. That is a band pass filter. It's going to cut off something with the frequency below. I believe 30, I believe 20 and after 300. And those methods are already here. You only have to uncomment it and use it. Let me run it. Okay, it went. So let's apply. Okay, how do we apply for. Or have the string L1 and string L2 where we have to apply those two methods. Okay, the first method we need to put the DT with. What is the dt? The spacing. The X spacing. The space time for each of the tote that we have. Okay, so let's go. We have the string. I'm going to L1. String L1 is going to get the. The L1 when I have written. Okay. If not, just, just tell me. Okay, so we're gonna use this method called written. We're going to input this train L1 and I'm going to put the. I think it's spacing that I call up the spacing time. Okay, it is strain H1Y. It's going to be written strain H1 spacing time. Okay. And after that I want to do the band passing. And for the band passing I can use, you know, the same, but I'm going to use a new one. L1 L1BP. Okay, equals to this data that is already written. But we have to use the bed passing. Bed passing method. And also I put something called is L1 since it's true, that's L1. And let me explain that part for you. L up. Okay. And here's going to be an H stand file and he is a false. Okay, this line over here, if L1 do that. So what it's going to do, it's going to do a shift, a little shift for the side. Because. Because it's basically the distance in time that we have from the signal. They are far away. Remember, they are far away. Let me go here and share with you. So the labs, the interferometers, they are far away from each other. One is in here from the south, the other one is in Washington. So they have a distance when the signal travels, you know, the gravitational wave signals goes travels on the speed of the light. So that's the distance in in time. That's 0.0.007. That's going to, you know, need to get from one to the other. For our eyes is instantaneous. But for the interferometer it's not. So we need to do this, you know, shift. Let's run this. If it works. Yeah, no problem. So now let's plot it. So I want to plot the exactly moment. Okay. So I get the time over here and the subtract. You know that same. Values that we have for the. Oh my gosh. The same moment. The exact moment of the. Of what happened. For each of the elements of the array, I'm going to subtract the exactly moment of the event. And okay, it's going to look like it's zero. When I, when I print it's gonna. You're gonna see it. Oh, I print something wrong. Okay, add one within baby. Okay. Because I didn't put the N over here. If it works now, please work. Okay, now you can see that's beautiful. Okay, that's the gravitational wave signal. After we filter, after we bend passing and after we do a lot of stuff. That's the signal. It's beautiful, isn't it? So about that shift. If it wasn't a shift, the red. The green line would be a little bit earlier. Okay, so I can close the files. So we did know the processing through our data. I could close these files over here because I don't. They are only reading. But yeah, you just use close. It's like you're dealing with any files and in. In Python. But what if I want to store that new data, that's the beautiful new data inside the same file that we have from the detections. Imagine that I will work. I already did this organization. I'm going to pass this for my data scientist or whatever who's going to screenshot it. I'm gonna try to use L1 copy because you guys gonna see that you have here one file that is called the L1GWOS and it has underlined copy in the end. Copy in the end. That's the file that I'm going to try to work with and show you that you can insert data in this HDF5 files and the old data is going to be still there. And it's very easy to do it. Okay, so I'm going to put H5 file and the name of this file. Yes, that one copy. Back. I just had to get the name. Okay, put here the name of the file. Yeah, that's the path. You have to put inside this, which I forgot the name in English. And you're gonna put a A It's for append. And so now you can read and you can write inside it. If it works. Okay, it works. Let's go now for. We're gonna put. It's the same file, right? It's just A copy of the file. So I already know what the string is. String and inside the string group, remember we have it's the same group. Okay, let me just show you again that it's the same file. Anyone under copy. So inside it I'm going to use the method create data set and that's it. And I'm going to put the name of my data set. The other data set was called strain only. I'm going to put strain, white and bend passing. Okay. And use the attribute name data to pass. My data is over there. Strain underline L1. Here. This data over here that is already been passed. String the line L1 my tape. Okay, let's see if it works. Works. We created a new data set over there and I want to show you just to verify that it's already over there. We don't have to click save like we do with pandas, etc. Anything else? It's already written inside. Okay, I'm going to use another. Okay, another file I'm going to read again the same file. Just read this one. And I'm going to see if I can find this data set over here. Inside it. I'm going to put the name of the data set that I just created and see if it's here. Copy. Let's see if it works. Okay, it's over here and I can even use my Equals. It's a HDF5 data set. But if I put remember I can treat it as a numpy array. If I put, you know, the brackets and two dots, I can just have the numpy array and I will compare with the actual file. Actual data if the data is correct. This one I use Comment. Yeah, true. See the same data. Okay, in the end of this notebook, I want to create a new HDFIVE file. How do I do it? It's absurd. The same thing as creating a text file file User lib file. Put the name of the file, you know, new file H5 and I can create as a. So let's see if this file just comes and appears here. For me it's going to be shown here. What is here? It's here. Okay. It's in reading mode. Richie, do you have any questions at the moment?

We're not many questions from the audience at the moment. If anyone wants to ask Linda anything, please do write your questions in the chat. I actually have one question for you. It seems like when you're trying to find what the right bit of data you want within an HDFI file, it's quite Complicated. You've got to spend a lot of time just looking at the keys and then going backwards and forwards and gradually drilling deeper into it. Is there an easy way to see the whole structure of the HDF5 file using Python, or would you recommend using a separate HDF5 viewer alongside your Python code?

I would rather use a viewer. But if you don't have anything, we can create, create just only one method to go inside and see that we have like so big groups or anything. But in the meantime, if you get this, this file, you need to have, you know, you can, you can have a picture of the structure of the file inside the file itself, but it's difficult. I'm not saying that's easy to navigate, etc. But, but you can. Even as you like printing the keys and etcetera, you can just create your own method to just navigate into it. Actually, I was researching just. Is that a way that we can print the tree itself? If I find out, I'll send back to you.

Okay, that sounds like it would be a useful feature. Maybe we have to file a feature request with whoever's creating the hdfi.

You can always add in the Lib itself. Thanks. Lib. It's open. Yeah. Okay. Do you want me to go jump to the. The second one is going to be like the completion of the. What we did here. It's the same, the same structure over here that we have, you know, the same import of the leaps, the methods. The methods are more organized. I'm going to run the methods and here I'm going to insert the code for how to. We will create like a pipeline for it basically, but without a pipeline from the Scikit Learn created. So we would have to first of all get the files extracted and later we have to create the time and then we have to calculate the dt, but we can extract it for X spacing attribute. Then we have the string itself, the whitening, the bed passing and we finish. Okay, so let me just do it right away so you guys can have it. The H5 files I was using for, I choose for H5 file, put the name of the file. It's going to be F for E or F in this list over here. Okay. And now we need to get the time. Time is going to be defined time method that we have and I'll pass one of my files. After that. We need the spacing time. It can be called DT and we just have time X and time like 8 and time 7 here will be the same thing and we're gonna have like the calculations string. I can call it GW if you want. So get the string. UW stand for gravitational wave. Get the string. I will pass one of the files to get the string. Still GW is going to be the whitening written GW and dt. Okay, agw with the band passing. Okay, so we have here everything we did before. Now we have like the same data. Let's run if it works. It did not work because DT I put df what makes no sense. And okay, we did a transformation. Now we can store node GW anywhere you want to the same file or in another file. And you can also get all the detections that you have in the Lego laboratory or the GW open and transform all of this. Finally I want to show you that we can even store in our Pandas file a data frame, sorry, data frame inside the ADF5. So I'm gonna have this data frame, this very small data frame for you. And I can use the pen to notify. So we know that Panda has a lot of tool to SQL, to CSV, to Excel, a lot of tools. But it doesn't does have to hdf. Okay, so we put the name of the file HF. And I'm gonna put the key. It's going to be the group that's going to be in and the subgroup is going to be data itself. And the data columns equal true. And the format is going to be table. I want to talk about this format table because I try to put, you know, just get the data frame and get the values from the data frame and put inside a data set instead of a table inside the H5 file. But it doesn't work because it. I believe only we can have one type. And if you have a known PI array that is stored, but you can use table and you can store it. So let's do it. Oh, okay. Common here. It worked. Okay, we can have something called gravitational wave history, you know. Here because this one is at our detections. And then you know the solar mass before and the solar mass after. Okay, let's read this file as we are reading before file. And the name of it. The name of it and read. Page five keys. We have gw and inside gw5 and we have gw. Let's. Okay, and we have data in data. Let's do it again before. We have items here. Two members. How do I get access to it? I believe it's case again the same thing that Rich was talking about. Okay, we have the tape in table two. Okay, three sharps and that little trick that is not easy to find. I spent a lot of time looking into it. It used to be just put here sync value, but does not work anymore. You have to put the brackets and parentheses inside. Now you can see the data. Okay. To create the attribute too, you can do the same thing. To create an attribute, you go for the same file and you can put here attrs. And like you creating a dictionary, you can do, you know, equals. Here we go. Oh, my gosh. It's wrong. Today. Yes, let me. Okay. The problem is, is I used R here instead of A, so I cannot write inside. Okay. Oh, I can.

Just want to remind people of the difference between RNA attributes.

Richie, are you there?

Oh, yeah. Just want to explain the difference between the R and the A file attributes.

Okay. So over here, I use the. Open the same R. I'm going to put a R. R is for reading, A is for appending. Let's see if it works. Fire is already open. Okay, Blender, you want to make GW pulse, please? Yeah, it worked. So I have to close my files. So I could see, can write and I can read. And after I am able to write inside the file, I could just come here and assign a new attribute and put the name of the description. See if we can see. So we can put in the value is out of here. You can put the description. You can put, you know, the age of the file, where it was come. Where does it come from? And put the metadata inside it. And I believe that's it for me.

All right, super. Thank you very much, Blender. That's really cool. I like that. I feel like I've learned a lot about physics and gravitational waves and also about the signal processing there as well. So in addition to the working with HDF5 files, I've got some side knowledge there as well. Excellent. We've got a few good questions from the audience. Before we get to that, I just want to say we've got a lot of very fun webinars coming up. So there's one for managers tomorrow on setting up a data academy. There's a session on Thursday for Data Camp classrooms. So this is primarily targeted to professors and other people who are teaching. Next week, we've got a session from Nvidia on understanding how LLM inference works. So if you're into generative AI, then please come to that. And then Wednesday, Thursday, next week, we've got two sessions on creating data products. Click on the QR code at the top left to register for those. All right, so Pretty simple question first from Sylvia saying how do you uncomment multiple lines? So in workspace there's a keyboard shortcut for that which is it's either control forward slash or command forward slash in depending on which operating system you're using. There's a more in depth question from Kevin here saying why would you actually bother to use HDF5 files rather than using single file locations? So can you talk through the benefits of HDF5.5, sorry, HDF5 files.

Can I read it? And can you put again.

Oh yeah, sorry. The question is, what's the benefit of using HDF55 files?

Yeah, both performance and compression benefits. It's a quite small one. Imagine that you want to allocate different time series inside of it, or you can put your whole research inside it. Actually I did it. You can have all the images that you have over there. You can put like a binary files and you can put inside of it only numpy arrays inside it. You can have neural networks completely inside of it. So it's a single place. But of course it depends on your application if you want to do something more broadcast and depends on what you actually want to do. If you want a single place to put everything up, that's where you go. I guess it was quite difficult to work towards using Pandas actually for that specific case of mine. But of course there is a lot of applications. You can use both of them. It depends on what you actually want to use, what to do. Okay.

Putting images and associated numeric data with them sounds like quite a nice use case if you're into computer vision, type things. And certainly, yeah, one of the big benefits is the HDFI file can be bigger than the amount of RAM you have because you don't load all the data at once.

So we can use these groups that can be translators as folders. It's like you, if you have a lot of folders in your file that you want to put in just one place. That's the goal. Of course, the applications that we deal with, companies in the industry, sometimes we're using a lot of cloud also. So sometimes we don't see the applications because we're not into that industry itself.

Okay, so if you're using a cloud based system then that maybe negates some of the benefits because you're probably not.

Memory constraint there right now. I wouldn't because my actual, my life work, my actual work, I will not use it. But if I'm doing the research and it will probably be a candidate for next my SAP std. I already thinking about it. Okay.

All right. Time for one last question before we wrap up. This question comes from Balder saying what you have to do if you're working with categorical data in HDF5 files. So we're most looking at numeric data today. What about categorical data?

I believe it's the same thing. Yeah. No, if you want to like put different types, prepare the same thing. Like if you put. You could put like there are different types and where the data frames, like each column is one type of the. Would be. Yeah. Would be the same thing.

Yeah. It's been a while since I've done any work with HDF5 files. Remember that it is possible to work with categorical data in there, but it was originally designed for numeric data, so it was slightly more of a pain. Have a play around and report back to us. So we can't be more helpful there with a precision answer. But anyway, Blender, that was brilliant. I felt like I learned a lot today. Thank you for instructing us also. So thank you to Rhys for moderating. Thank you to everyone who asked a question, and thank you to everyone who showed it today. I hope to see you all again in future webinars.