Data Engineering on Google Cloud Platform Specialization Progress | Learning Intelligence 47

Woo! We're up to our first lab of the Google Cloud Data Engineering certification, otherwise known as Serverless Data Analysis with BigQuery. Now, you're down there, I've been standing up. Let's come down to the same level. Sounds like the standing desk needs a bit of oiling. Now, first of all, I want to give a big shout out to— whoa, how'd I end up here? This is not where I'm supposed to be. I'll be back later, bud. Coursera for sponsoring this video. You know, I love Coursera because they partner with people like Google. Just have a look at this, right? Got that little Coursera— you see that Coursera for Google symbol there? I'm going through Serverless data— what's it called? Serverless data analysis with BigQuery. I even said it before, I can't remember it here. Now, what exactly is BigQuery? Well, according to the formal definition, it's a no-ops data warehousing and analytics. Now, I just read that because I purely copied, copied it down, but to me, I understand it as it's a place Well, you can, if you have a big amount of data, you can send it up to the cloud. And if you want to store it and leave it there in a safe manner, you can. And it's, it's, it gets cheaper over time as you store it there for a longer period of time because Google do something smart with their, their storage to, I mean, if you're not accessing it all the time, you don't have to use that electricity and all that sort of stuff. So storage gets cheaper over time, but it means that if you want to do a, do analysis on that, like run some queries on it, which is literally why it's called BigQuery. You can do it fast and you can do it effectively with SQL, SQL-like queries. Why is that so handy? Well, the fact is that it's serverless, right? And to me, I don't know anything other than data warehousing that's serverless, which means living in the cloud, you don't have to necessarily boot up a computer to run stuff on it. Whereas in the past, you'd have to have your own— Sorry. "Daniel, there's nothing to repeat." "They're here." Siri's just going crazy. Anyway, in the past, my phone should be in the drawer. That's a note to self, right? Keep your phone in the drawer. In the past, you'd have to have this big computer on site and run all this data stuff. You'd have 3 different people to do this, to do a job that you and I can do right now in the bedroom, I guess. But We're going to jump into the first lab. Now with this, with this course is the reason why I love it is because it's structured labs. It actually shows you what to do if you actually want to start running it on your own data. So I'm going to, I have a plan. I'm going to get 3 labs done. There's 6 labs in total, 3 labs and then break. And then we'll do another 3 labs. Why don't you come show me good— All right, we just did our first— it's really dark in here. I think there's a storm brewing. We'll get to that in a second. Actually, there's enough light for this. We just finished our first quiz on the The Google, uh, certification Data Engineering. We've, we've been through 3 labs, so we've done the goal. We've done the goal for now. It's, it's, it's break time, I think. But Lab 1, you can see, was building, building— I think that should be 'and'— and building a BigQuery query. Lab 2 was loading and exporting data. So how do you, how do you get data from, from your computer to, to upload it to BigQuery, you can do it. I, the two methods we went through was the command line interface, so terminal, and then otherwise just that regular upload button. And then it was a lab on advanced capabilities, so advanced SQL queries. Now finding SQL, SQL seems a bit like, a bit like Python and Pandas, right? But it's, it's, it's more to me seems— no, I don't want to say simple. Simple's not the right word for it, but it's, it's like you're literally going select this from here, where this, and group it by that. So I mean, it's similar. And there's a saying when I went to Bali once, it's like when you, when you go over there and you buy like a t-shirt, but it's a, it says it's a Supreme t-shirt, but it's not Supreme. They would say it's same same but different. That's what I feel, uh, sequel is. I think we've got a storm coming. Check it out, that sky is looking grey, it's looking dark. The rains are coming, or actually the rains are kind of already here. Seven, you got a new toy there? You got a toy there too, Belle. We were gonna come and play in the backyard, but I guess since it's raining Let's do a workout inside. Yeah, that's a good idea. Now I went for a bike ride earlier this morning in the sun. That was amazing. How good is the sun? Check this out. We just finished some writing, writing, not riding, we're riding now. Now we're up to, we've got BJJ training and the Google Cloud certification on Coursera. It's a big day. So I've done some physical exercise today. Oh yeah, I had BJJ training. I got a tournament in like 10 days, which I'm really excited for. But I like to break up study with movement. So I've been sitting down for a bit now. It's time to move the body. The puppies love moving too. We're gonna do something nice and simple. We'll start off 5 minutes of stretching and then we're gonna do 100 push-ups and 100 squats. And then we'll get back into the other 3 labs that we have to do. Let's go. We'll start off with one of my favorite ones for the hip. You know, sitting down for so long studying all day, sitting on a computer screen, your hips can get real tight, and that's, that's not a good time. So what you want to do is get one leg back like this, a leg up forward, hands shoulder width apart, and then you're just going to straighten your back leg, thrust your hips towards the ground. All right, and if you really want to step it up You can just hold it here. You can go down like this, right, keeping that same posture. Or what I like to do is get an upper back stretch as well. Lift arm up like that. We're going to do this for 5, 5 each side. Should feel it all through the hip, all through the leg. Really make an effort to keep that back leg straight. All right, for this one, we're doing an opposing movement to what you usually do when you sit down, right? What do you usually do? Usually sitting at a computer desk, hunched over like this, right? So the spine is curved and the shoulders are rounded forward. So this one is to, to counteract that. I'm going to come in just like this. You can start on your knees if you want, or I prefer start flat. On your stomach, and we're just going to pretend like we're on a surfboard. Keep your hips on the ground, push up like this. So now we're doing the reverse of that curve that happens when you're sitting at a computer desk for a long period of time. I'm just going to hold this in. I'm going to make the arms nice and straight, spread the fingers, and hold it there for about 20 or 30 seconds or so, just to feel that stretch. Should feel it right through this spine, right through the hips. All right, push-ups and squats. Let's do it. We call this one the lounge room workout because you can do it anywhere. So to do these 100 squats and 100 push-ups, we're going to do 20 push-ups straight into 20 squats, and then we're going to have a minute rest. Do that 5 times. All right, I like breaking up things into intervals. Study intervals, workout intervals. Let's go, set 1.

For the record, I'm done trying to make y'all comfortable. For the record, you ain't trying to grow, then it's done for you. For the record, lab on me going all the way. For the record, ain't trying to link, no time to waste. Shoulders on ice for y'all. 86 all the hate. I woke in a ball today. Got lost in the ball today.

100 push-ups, 100 squats, 5 minutes of stretching. You like that, Belle? Yeah, you had fun, didn't you? Feeling good now. A little bit out of breath, but that's good. It's definitely raining outside now. Check this out. Woo-hoo! Seven, you made a Made an absolute mess there, didn't ya? All right, we're gonna get some water, wipe this sweat off, and get back in. The last 3 labs, we'll finish this certification tonight. Oh yeah. My gosh, is it dark in this room. Allighty then. You get it? That was like Jim Carrey. All righty then. Oh, it's not that good? Don't say that again. I thought it was pretty good. Yeah, alright, alright, alright. Okay, let's get back into some Google Cloud Platform dot engineering labs. I wonder what we— I can't exactly remember what was next. I think it's MapReduce, which I'm gonna have to look up the definition to as well because there's a lot of terms in here. A lot of this stuff is new to me, right? And I think it's important to recognize that even though it's all new and you're going through, I'm going through these labs, or you might be going through these labs or whatever course, you're not gonna get it the first time round. Like, I find that a lot in jiu-jitsu, right? So I go in, I do a move, and I'll stuff it up. And then the next time I'll go in, it'll be the exact same move, and I make the same mistake. And my coach is always like, he's a black belt, so he's wise. Awesome dude. And he's like, you'll make the same mistake 100 times, 500 times, but eventually you'll realize, hey, I should probably stop putting my arm there so I don't get choked out next time. Anyway, let's get back into it. We've got 3 labs left. Data flow, data pipeline. Oh, there we go. Lab data pipeline. There was MapReduce. That's what I was talking about before. I'm gonna have to look up that. I think it's reducing the dimensionality of your data or If you know what MapReduce is, leave a comment below and explain it as best you can, and so you can help me and others understand. And then we have another lab, MapReduce in Dataflow. So we've got Dataflow pipeline, MapReduce in Dataflow, and then a final lab, which is lab— no, lab inputs, side inputs. And then we finish up with the last quiz. Easy money! It's 6:42. We're going to finish this in time for dinner. All right, catch you on the flip. We're in the last lab. What do we got? Side inputs. Boom. So side inputs are like different sources of data into your main main pipeline. So imagine you've got the, the stream of data here, side inputs, another one coming in over here. That's how I'm thinking of them anyway. All right, let's get into it, finish this lab, and then we go get some food and wrap it up. All right, I'm up to my last attempt on this quiz for the last 8 hours, or for the next 8 hours if I don't get this quiz right. I always— I wrote an article the other day how I always stuffed up in exams. Exams get me, man. They get me. All right, so which of the following statements are true? I think it's this one. Dataflow executes Apache Beam pipelines. And data flow transforms support both batch and streaming pipelines. And this one I know is BCA. If not, we're completing this tomorrow morning. Jump in. Yes! Oh boy, oh boy, oh boy. Some resources here, you can check those out if you want. Actually, there's a great Medium article which is here, I'll link that below so you can check it out. It's, it's a, it's one of the sample projects, right? It's like Google put all this, all this GitHub data on BigQuery so you can test it out. So it's 2 billion files, and then we did one before which are like which Java programs need help. So the Java open source libraries which have to-dos or fix-me's in them. And there was 2,196,247 files. Something you don't really want to process on your little baby laptop, all right? You want to use BigQuery for that. So we've gone through a lot of stuff today. Let's see if we've completed it all. From Dataprep, well, Dataprep's a new one actually that I just learned about. BigQuery, Dataflow, MapReduce. Have we completed the required stuff for today? Oh no, what is this? We haven't passed one of these things. I'm going to have to go back. It was that dang side inputs lab. The last lab, best lab. All right, well, we're going to have to go back. I'll see you in a sec. All right, we're back. So I don't know what's going on. There must be an error. I need to reach out to Coursera. 'Cause refresh this page, saying I haven't completed this lab, then we go into here, the exact lab, lab completed, elapsed time 7 minutes. It's 7 minutes because I've done it 4 times now and I'm really quick at it. You don't need the certificate right now. You think so? Yeah, yeah, you can fix the bug tomorrow or whatever, but what's most important is you got the skills. Yeah, you're right. You got an incredible platform like Coursera, It helps out 35 million people worldwide, partners with incredible companies, incredible universities. What you want to use it for is to get the skills that will take you to where you want to go. The thing about studying is you can spend the whole weekend in this room, hours in that chair there. Then you go to work on Monday or you see your friends, no one knows about the hours you put in here, but that's okay because you're on the journey. You're on the journey to where I'm from, 2048. I want to thank you. I want to thank you for taking care of your health and taking care of your knowledge. Step by step, I mean, you may not notice it now, but it all adds up over time. Trust me. Keep learning, keep moving. Too true, future Dan. Yeah, I gotta keep that in mind. You heard the man. Keep learning, team. Special shout out to to Coursera for sponsoring this video, and special shout out to my boy Lach. Lach's back, baby! Or Lock, or Lach. I still get his name wrong. It's Lach. It's definitely Lach. Lach, you're the man, man. See you, team. I gotta get some dinner. Keep learning. Oh yeah. And if you want to try the course, there's a link below, as well as a link to that Medium article, which was a tutorial on BigQuery. But I'm enjoying it, except for this little one part that's not letting me pass, but we'll get through it. Have fun. I mean, you can do it tomorrow, but the certificate's not the be-all and end-all.

Let them take advantage. I was wildin' on record, off record deals. Tell them talk to Khaled for the quote. On record, off record, I still want the act, not the Coach. For the record, I'm done trying to make y'all comfortable. For the record, you ain't trying to grow. Then it's not for you. For the