Daniel Bourke finishes the Google Cloud Data Engineering specialization with a 94.6% quiz score. He compares BigQuery's high-throughput streaming to Bigtable's millisecond reads for fraud detection, then reflects that perfectionism shouldn't block progress. Next up: the official Google Cloud exam.
Chapters
Morning! It's time to get into some study. I got a friend's barbecue at like lunchtime, so I'm halfway through the Google Cloud Part 5, the final part. Get that up. Crawl down in here. Turn the monitor on. Hopefully it's still open. Alright, where were we? Here we go. So these are the lectures I need to do, all them there. I got these to go. So I wanna finish them before I go to the barbecue so I can enjoy it and come home and not have any study left for today. Let's freshen up and we'll get into it. Oh, got the morning coffee. Look at that, look at that, it's morning coffee. All right, let's get into— now we're a bit fresher, got that, got that morning bed hair. Look at that morning look. That whatever. Um, yeah, anyway, that's enough for this morning. We're ready to go. 3, 2— oh, actually, I've got a few hours, so fingers crossed we can get it done. I'll get it done. I'll get it done. Check that out. Can you see that? That's a spider. Taking on a fly. It's like caught it in its fangs. Nature is epic. I had to come outside to wash my shirt. Check it out, look at that stain. Oh yeah, fix that up. So we've got one quiz, one lab left of Part 5. I'll come up, I'll show you my progress, and then, then I should be able to get it done within the next half an hour or so and then go lunch. Alright, so we've done 3 Pomodoros so far. Windy. And that's been plenty of enough time to get a large chunk of what we had to do done. Let's come in. I've got a couple more videos here, so there's 1, 2, 3, then a lab, then another video, then a final quiz, and then we'll be finished. And here's where I'm up to so far. There we go. 2 more pieces of assessment to go. Boom, boom, boom, boom, boom. Did, did I think from here to here so far today. And we got, we should, should be one more Pomodoro finish to that. Last 3 have been about essentially Bigtable and BigQuery. The main takeaway I've got so far is BigQuery will work for the majority of what you need in terms of streaming data, because it can, it can ingest 100,000 rows per second. So if you imagine a table, it can be updated 100,000, with 100,000 different rows every single second. And then if you wanted to query that table, so if you run it, wanted to run some code on those 100,000 rows, you can get a response back, which is called the latency, in a matter of seconds. Whereas Bigtable, sometimes you might need quicker, like seconds might not be fast enough, right? Like maybe someone's making a fraudulent trade on a credit card, so you might need milliseconds for that kind of thing. That's something you want to use Bigtable for. Similar to BigQuery, more expensive because of that extra speed. But I'm going to finish this off and I'll see you in 25 minutes or, or when we're done. We should have a new certification after this. Yeah! Alright, I'm We're up to the final quiz, the final quiz of the Data Engineering Google Cloud Certification. Did I say that right? It's, it's always been a mouthful. All right, let's get into this. Hopefully we get it, we get it right on the first go. So there's 3 responses here. Which of the following are true about Cloud Bigtable? Uh, offers low latency. Oh, I'll take my headphones off so I can hear myself a bit better. It's warm, so my ears are sweaty. Offers low latency in the order of milliseconds? Yes. Ideal for greater than 1 terabyte data? Yes. Uh, great for time series data. And it doesn't have support for SQL, it's NoSQL. True or false, Cloud Bigtable learns access patterns and attempts to distribute reads and storage over nodes? Yes, true. Cloud Bigtable does, it adapts. It's actually pretty cool, like it's years of engineering experience from Google and it It— once you put your data in there, the advice is to let it run for a few hours on a big dataset because it will slowly adapt to optimally run on that dataset, whichever— whatever you've got in there. And then of course you can tweak it a bit depending on how you want to store your data, depending on how you want to do compute. Do you want SSD? Do you want hard drives? And then finally, which of the following can help perform me— improve the performance of Bigtable. Um, don't use HDD instead of SDD. Change schema to minimize data skew. Yep. Clients and Bigtable are in the same zone. Same zone means, say for example, I had a big, Bigtable data in Australia, maybe it's in Sydney instead of being in the US. Or if you're in the US, maybe it's in California instead of being in Sydney. So you want to make sure it's as close, literally physically, as possible because that will reduce the amount of time data has to travel. Add more nodes. If you have 3 nodes and it's not working very well, you can scale up to 5. Fingers crossed. Boom! Passed final assessment. Now I've got one more summary video to go and then we should be certification time. Be back in a sec. Woo! Alright, let's check, let's check the grades. Where we at? We'll refresh this. Please have picked up. Oh yeah, we passed it! Grade is 94.6% out of 100. That's awesome. I missed out on these two. Of course, we could go back and make them 100, but I'm not, I don't really need it to be perfect. I don't need it to be 100. I like the fact that there's a little bit of error in there, right? Because I think in the past, what's definitely held me back, and I got an email about this actually the other day, it made me think, is someone was being held back by trying, working on different things, by trying new courses, by working on their own product projects, et cetera, because they wanted everything to be perfect. And a big lesson for me was, 'cause I'm a perfectionist as well, right? I like everything to be perfect, to be great, to be awesome. But perfectionism can definitely get in the way of progress. And so, today, not entirely fussed at getting all the results perfect. Having a few little errors on here is perfectly fine with me. Well, perfectly fine with me, kind of counterintuitive. But don't let perfectionism get in the way of progress. Now, I think what's next is I check out— I get actually Google Clouds— Google Clouds. Google Cloud certified by doing the data engineering certification, um, actual exam. So I need to go do an exam, pay some money for it. Hopefully we'll see if Max Kelsen will sponsor that for me. Um, and then yeah, that'll, that'll be actually Google Cloud certified. That way I can, uh, say, hey, if you need a qualified Google Cloud and data engineer, I'm your guy. But let's get this certificate, hey? My computer doesn't have a printer, so I need Mum's help. Hey Mum, can you check your emails and print the little attachment that's on there for me, please? Which email did you send it to, Daniel? Uh— Whoa, look at that! What's it say? Daniel Bourke has successfully completed Building Resilient Streaming Systems on Google Cloud Platform. Bit of a— An online non-credit course authorized by Google Cloud. An off-brute Corsa. Corsara. Corsara. Bit of a mouthful, eh? Oh yeah, here we go. Ta-da! Good morning of study, if you ask me. Would have been nice if it was in colour. Now, I'm going to go make some lunch. Actually, I'm making a salad. I'm bringing salad and some nice bread to the barbecue. If you ever go to a food event where your friends are hosting, be nice and show up with some food. But, otherwise, good morning of study, and we're finished just in time. I've got about 15 minutes to make that and then head off. But keep learning. If you want to try out the Google Cloud Specialization Platform, I'll leave a link in the description below. I've really enjoyed it. Now I've got to go and get actually certified, so— oh, Slick! How are you, bro? I just finished the Google Cloud Specialization. What do you reckon? Excellent, excellent. I wish you would have seen it. Thanks, mate. Keep learning, team. Peace out.
