LeadDev Berlin 2022 Michele Riva

All right, thank you so much. So, explaining distributed Systems like I'm five, it's gonna be hard in 10 minutes. We can try. Before we start, I have some news. I just been promoted to staff engineer. So you. Yeah, Yeah. And this talk is actually part of my growth path into transitioning into a staff engineer role. And that's because I wanted so bad to be when I first joined Nearform. Just for you to know, we are a professional services company and we work a lot in the open source software area. We maintain a large portion of the Node JS runtime, we do a lot of great stuff in that sense. And I wanted to get on the leadership of the company, so I wanted to get promoted and I wanted to find a way to, for example, learning how to mentor people, you know. So I started with a dear friend of mine giving mentorships every Monday. And the idea for this talk started during this course. This is me with Gabriele, dear friend of mine and start engineer and VMware in the RabbitMQ and Kubernetes teams. One of the best engineers I've ever met in my life. And we came up with the following idea. So a very, very simple question. What does? What problems do distributed system solves? Okay, not so easy, but let's make an example. We asked our the people in the course to create an API that uploads a large CSV file, let's say greater than 10 gigabytes in size to a server, then read the file, normalize the data, store the data into database, and then, you know, API server, stuff like that. And of course, these are junior developers, so they started with the most simple architecture ever. So an API server, you make a request, you put some data, the server starts to send data to the database, et cetera. If you visualize that into a different kind of diagram, you start realizing how synchronous this is and how dangerous this can be. In fact, what if we have dozens of concurrent uploads? As I was saying, I work a lot on Node js and especially if you use something like Node JS for dealing with input, output and files, et cetera, it's not the best choice ever. I'm sorry, guys, but especially one problem you might encounter is that once you start uploading and processing more and more files, you. You'll just run out of memory. Right. When pointing this out, the more advanced engineer in the course said, okay, well, we just use a load balancer, split the traffic amongst multiple servers. Yes, this is part of the solution, but doesn't really solve the problem. So I learned the hard way that if we are talking about a very complex topic, you have to explain this to me like I'm five. And if you can't explain to me like I'm five because I can't understand, please explain to me like I'm three. Let's see if we can go any further then. So we introduced the ice cream scenario. As a typical Italian guy, I talk about food, so let's dig into it. So let's say we have an ice cream truck. And what happens in Italy when you have an ice cream truck and it's really hot during summer, is that people start to come to the ice cream truck and they want ice cream. But we are Italians, so we want ice cream. A lot of ice cream. That's the problem. So the more time you spend in a city, the more people will come. They all want your damn ice cream. But of course there's a problem. How do we currently deal with the fact that there are a lot of people, everyone wants the same thing at the same time. And let's say this is a cool scenario. How do we deal with that scenario in real life? With queues, of course, Pretty easy, but that's not enough. What if we have a lot of people? So the problem with people, typically speaking, and it's not just about computers, this is also about people, is that when we are in queues, there is people wanting to skip the line, you know, cut the line. People trying to steal ice cream from babies and kids, whatever. Ice cream is beautiful. We all know that. We all want the ice cream so bad. So of course we hire a security guard. So that lady in that case will make sure that the first person staying in line will be the first person to get served with their ice cream. So first in, first out, right? Ok. Of course, one ice cream truck might not be enough. So we are in a lucky business. We buy another one, then another one again, and that's it. We end up having multiple ice cream trucks and multiple queues. Okay? So when it comes to software architecture, things tend to be a bit harder than that. Of course. In fact, a distributed system is a collection of independent computers that appears to its users as one single computer. So that is telling us that it doesn't really matter if we have like three lines, for example, three queues, and it doesn't really matter if there is one ice cream truck per queue. We might have one ice cream truck, 10 ice cream trucks, 100 ice cream trucks, and many different waiters and waitresses that bring the ice cream to the people in line. We also have guards that will make sure that the first person in line will be the first one to get served. So starting from there we can start understanding what a distributed system is. So first of all, to be considered as distributed, a system must respect three laws. The first one is computers operate concurrently. And yes, this is the case. So computer number one ice cream truck number one is totally independent from the number two. And the number three waiters are completely independent and the queues are totally independent from them. So law number two, computers fail independently. And if an ice cream truck catches on fire, I mean frozen stuff gets on fire all the time, right? But of course it's not a real problem because we have multiple trucks and you see we can still keep different lines. And one cool thing about lines is that they are self organizing. So if I come and I see three lines and one is less crowded, I go there. So they will self balance in that sense. And if one ice cream truck is on fire, waitress won't go there, hopefully so no one is injured. And of course one ice cream truck on fire shouldn't affect waiters and other trucks. So that's a security reason mainly. But computers do not share a global clock. This is one very last thing to keep in mind when designing a distributed system. And this is a bit more tricky to explain, so I have to change an example for a tiny bit. But don't worry, we will be back to the ice cream really soon. So let's say we have a source of truth, which is a golden watch in that case and two different watches. We want to synchronize these watches to a unique time. There is a little problem which is called laws of physics. So for example, we make a sync request to the source of truth. There is some latency in the request and then there is some latency in the response. And if you observe that well enough in real life, I have a watch, I look at it and I have to wait for the light to get reflected back to my eyes. And this is latency. So I will never know the exact moment I'm observing my source of truth. Right? So clocks are not reliable in that sense. Never trust them. No, I'm sorry, I'm kidding. We are trying to get around these problems by creating standards such as a network time protocol NTP and precision time protocol ptp. If you look for NTP versus PTP on Google, you will find an amazing article from geekforge. I guess it's beautiful. Well explained, highly recommend that. So the Solution we required to our students was basically a solution that could respect any of those rules, or let's say all the three rules altogether. So first question is, is uploading a large file to a server a good option? I don't think so. At least not in a distributed system and not if we have such large files. In fact, there are multiple ways to upload files. For example, using object storage, I don't know, AWS, S3, Google Cloud Storage, Azure Blob storage, whatever. It's not really important. It's the concept in that case. And as soon as we put something into a storage, then we can notify someone like a Rabbit or you know, SNS on aws, whatever that calls a consumer that takes, that does basically the heavy lifting in normalizing the data and inserting data to a database which gets connected to an API server that lets you run queries on the data. And I know that this diagram, it's really, really bad, let's be honest, but it's like overview for people to understand that we may want to respect those rules with that diagram. So are computers operating concurrently? Yes, let's say RapidMQ fails, which is really unlikely because it's a really resilient software. But okay, we won't be able to upload stuff for quite a long time. Not a problem. Everything else will continue working. Apiserver fails, no problem. We continue uploading stuff and we will be able to resume our queries later and so on. So you can see this not being a real concern. So when they fail independently and they all operate concurrently about global clocks, I'm really sorry, but it's more for people implementing this booted software rather than people implementing architecture at this specific case. So what problem did we solve? My idea is that first of all explaining very simple stuff like that during an interview, or asking people to grow their skill set, pointing themselves in front of these kind of challenges can be interesting because by itself it's really, really simple, but can lead you to any kind of surprise when you start looking at the problems that can arise from Synchronous systems specifically. So that's my idea, that's what I wanted to share with you. And one very last thing, I'm sorry for the self promotion here, but it's free and open source software that I've been developing. It's called Lira. It's a full text search engine capable of running on edge networks. So if you want talk about distributed systems and algorithms behind full text search, I'll be around. I have a very good green sign on my badge. So I'm hug ready and. And I'm here to talk with you. Thank you so much for being there. I will be mainly on Twitter, so let's.