INSIGHT | Making the existing web real time - Guillermo Rauch (Socket.io & Cloudup)

All right. Thanks a lot for being here, guys. I understand this is the last talk of the day. We just witnessed the reprovisioning in real time of a new projector. So that was pretty cool. That held us back for a little bit, but no more. So what I'm going to talk about today is essentially some of the fundamental realizations that I've made about real-time systems after working on them for a long time. Um, and some of the things that I've noticed that can help you guys turn existing legacy systems that are not real-time, uh, into more modern, faster self-updating systems. So, uh, you probably usually hear, uh, real-time being thrown around a lot, and it's usually conflated with the technologies that we use for implementing that. So you think of Socket.IO and you think, oh, it's a real-time framework. And then you think of maybe a RESTful API, and you think, oh, maybe that's not real-time. So I think it's hard to debate the merits of a real-time system or how to implement them without specifically— that just turned blue— without specifically talking about what it is and what are the defining characteristics of that system. So I've come to the conclusion that it has to do with two fundamental things. So one is showing data updates to the user with minimal latency. This has to do with the networks that are in place and the communication channel that is established between a user and the server. So the user has to think that the application is alive, that the data that they're looking at is not stale. Sometimes you might be chatting with someone in the same room. It happens a lot at offices these days where everyone is on a system like Slack or IRC. And as soon as you are done typing, you're expecting that that information will be conveyed to someone else with minimal latency. Uh, the second aspect of a real-time system is that you will usually find is that not only is it fast at retrieving data in the sense that if you take a specific message and you measure how long it takes to get delivered, it's not enough to look at just the delivery of that message and measure its latency. You also need to take into account that the system that presents the information has to get the data on behalf of the user. So if I'm just looking at the data source, if I'm just looking at your web application, it should be able to absorb data updates without the user being involved. So this basically means the elimination of things like pull-to-refresh. So if I'm looking at a news feed, I shouldn't have to press a refresh button. I shouldn't even have to think about whether there is a synchronization problem, uh, with— between my dataset and that of the server. So once these two conditions are, uh, satisfied, we can think of a real-time system as something that's fast and self-updating. So this is a very good guideline to go by when you're designing your own systems, even when it comes to designing your UI. It doesn't matter if you're, working the network architecture, the topologies, if you're— or if you're thinking about what the application looks like or how it behaves, uh, it's useful to go by these two premises. It has to be fast and self-updating. An example that I like to give, uh, in terms of how this reflects in a day-to-day programming task, it would be implementing a very simple stock ticker. Uh, the most naive implementation that we can think of is starting with some basic markup that would represent a widget. And then we create a JavaScript function that is in charge of getting the data from some RESTful API. And it'll do this by basically setting a timer to get the data every 15 seconds. And on the right-hand side, I'm going to present a log of what would happen over time if we consider every second after the user has loaded this page, what would happen? So let's say that we get the information about the price. It's initially $500. Then we request the stock again. We render that price. We know that the source of truth of the server in this case is saying the price is $500. That's what the Apple stock is today. After we get it, we're going to wait 15 seconds. So every second that goes by after this, the price could have changed at any point. The user, meanwhile, is waiting until the next cycle of refresh. So in this case, there's going to be an added artificial latency of, let's say, about 10 seconds, because we know that from the perspective of the server, the price had already changed. So the user just waits needlessly until the request happens again. And the price is rendered. So this is an example of actually a self-updating UI. And in a lot of cases, whenever we're evolving a legacy system to make it real-time, the most naive approach is to do this. Just make it poll in the background and try to update some UI as soon as that response comes back, and then schedule it for some time in the future. Obviously, it's not fast enough. And that'll make most users or real-time geeks like ourselves pretty sad. So we can think of a naive optimization to handle this scenario, which is why not just schedule the update right away? In fact, a lot of systems that to the user look real-time are actually executing some sort of polling cycle if we examine how most of the architectures in our current systems work. And that would satisfy both premises now. So if we schedule it right after, it would be fast-ish, because we're also accounting for an extra round trip. So we're accounting for the user having to ask. So it needs to send a message to the server to get the latest information. But it does become somewhat fast and self-updating. The problem here is that the server is the source of truth of this information. In that example, the stock API or the servers or brokers will know about the price before the user actually thinks of asking. And we know that the user had the intent of knowing or was interested in that information. We can consider it as subscription of sorts. So basically— we can eliminate that extra step of having to ask because the server is the one in charge of basically managing the information. And the other downside of this approach is that by basically making a lot of GET requests over time, we're increasing the load significantly because we're creating a lot of messages that servers need to handle. So your load on the server will look artificially higher because you're just asking and asking and asking again. And it's much better to actually have the server push data to us with a technology like what I created, or any other mechanism for essentially establishing a bidirectional socket. And that essentially solves the problem of making a system that's fast, that's self-updating, and that minimizes round trips between server and client. An analogy that I like to give to discuss this sort of system is that If we focus on the language of what we're discussing, when we talk about servers, when we talk about clients, when we talk about requests, when we talk about fulfilling requests, an example that comes to mind is that of, for example, a restaurant. So let's assume that the— let's see, there is some latency in the presentation. Guillermo Rauch here. You can sit on this side. So let's think about someone going to a restaurant and performing a request. In the case of, for example, a cafe, you go, you sit down, and you ask for the menu. So we can essentially establish that as a GET request from the client to the server saying, get me the menu. So that's the first step of how most of our applications are architected these days. And usually, there is some sort of handshake process by which the server says, OK, 200 response, here's the options that you can subscribe to. The next step would be the actual request of an item. So in this case, I'm going to ping a— let's say a get bagel endpoint. And the server will say, okay, accepted, please wait, while it goes to the backend and talks to other services. In this case, let's consider the kitchen a backend web service. And let's say, uh, in this, in the sense of like most of your day-to-day systems, this would mean implementing some sort of job queue and the petition goes in and eventually that work is gonna be done. And we're expecting some sort of result later on. So between that handshake of web services internally, there's— it could also be RESTful. It could be based on some other protocol. But essentially, the work is scheduled to be done and fulfilled at some point in the future. So what this means is that the user is now in a waiting state and needs to find out about the progress of the action "So what, what's happening to that order/subscription that just happened?" And this is where usually we find that most APIs implement some sort of status endpoint that is meant to be queried over time. And for this example, it would mean for going to the server and the server asking its internal service, "What's the status of the order?" And this would happen again and again obviously over time, and the frequency with which the client requests can increase. So it can result in a lot of bottlenecks, it can result obviously in a poor quality of service, and essentially, basically you're not designing the ideal services to convey information as fast as possible. The problem becomes worse as more clients join this service. So they will start queuing up status requests with higher rates of frequency than actually the server can keep up. And this results in server congestion. So this is basically the main downfall— or main fallback— sorry, main disadvantage of trying to evolve a RESTful type API into a real-time system. Usually it's going to result in non-ideal levels of speed and increases in server congestion, which will make scaling slower. The most trivial solution to this problem is actually, one, obviously push to the clients without asking, so refactoring your entire backend to perform data push. But an alternative to this, which would be the kitchen says, OK, here's the food, and emits a message that gets directed all the way to the client, aware of that subscription. So we can think of a system like events or Socket.IO to fix this. But if we look at our previous characteristics that we defined, that the system has to be fast and self-updating, we actually find that if we reduce the latency between the client and the server— so for example, in our analogy, it would mean that the server is actually very, very physically close to the person that's asking. And therefore, even if the client is asking very frequently, the responses will be fulfilled very fast. Or in order to avoid server congestion, we can also increase the server capacity. So we're also addressing our second problem, which is if we have a lot of people asking, we actually have a lot of nodes in their proximity. This is the most traditional and most effective workaround that there is in order to make a system— and you'll find that in the real world, a lot of web applications have actually evolved in this way. They haven't necessarily completely revamped their architectures, but they have made sure that congestion and proximity/latency is a solved problem. So although not ideal, this would make our second refactoring actually work in practice. And it's worth reminding ourselves that real-time becomes about the user experience. Experience, uh, and not so much about the underlying implementation. And the implications of this are actually quite profound when it comes to designing our systems. The speed of how frequently the data updates, so the, the speed of that first fast, um, point that we made will actually determine a gradient of how real-time your system is. An example that I like to give is, When Google Docs first came out, their implementation was quite different from what it is today. And Etherpad, an alternative to it, came out in 2009 saying, "We're actually really real-time, and that's our main differentiator from Google Docs." It wasn't that they were contesting the definition of real-time, they were just saying that they were more real-time than Google Docs. And they went on to describe that Google Docs was actually implementing a polling cycle of 5 to 15 seconds. So this meant that character-by-character synchronization was actually impossible, because you actually had to wait 15 seconds. So Google Docs violated our basically two principles in the sense that it was self-updating, but it was not fast enough. And they actually had to revamp and reconsider their architecture. By actually reimplementing their synchronization mechanism and their concurrency algorithm. They also, in terms of minimizing the latency, implemented long polling when it actually used to be polling. So today, we can think of the equivalent as WebSockets. So if we try to approach a definition, I think it's a really good idea to stick with the language of what we're discussing. So it took me a while to realize this, but thinking about the actual words that we're discussing or mentioning was really helpful in me understanding how to simplify this problem. So we're talking about real-time, real as in staying truthful to the source of truth of our application, and time means being aware that there is different steps at which the data is changing at different, um, clients and servers at the same time, and we have to reason about how to, uh, make the timelines merge so that we give the results that the user expects in the fastest possible way. So implementing a real-time system becomes a synchronization problem as opposed to, how it's usually thought of, which is a messaging problem. It becomes a lot more useful to think about synchronizing data and what are the challenges as far as concurrency and fault tolerance, as opposed to thinking about what transport I'm gonna utilize to transport, to essentially deliver those messages back and forth. So a trivial example that I came across recently was while I was at Internet of Things conference, we were discussing how, for example, the world will become connected through the internet, even in its most fundamental ways of your everyday life. So using a light bulb can actually be— a switch to a light bulb can actually work over the internet, as opposed to the traditional wiring that we think of today when we utilize it. And there's actually been a lot of examples of systems like this recently that you can control from your phones, et cetera. And I try to stick with this example because it marks a fundamental challenge that we have with our internet systems today that we actually don't have in the real world. So let's assume that a user wants to turn on a light bulb that— and this light bulb is actually an internet-connected light bulb. Wired directly. It's going over your Wi-Fi connection. And it costs you $100 on Kickstarter. So let's now consider a switch, which is going to be how the user interacts with this system. The first approach that we can think of is we connect this switch to the internet, and we use something like Socket.IO to send an on event when the user flips it. Or we can think of sending a POST request that says, just turn it on, or any sort of transport mechanism that you can imagine. As a matter of fact, like I mentioned earlier, transports are not as interesting to reason about as opposed to a synchronization problem. Now let's assume that— and this happens often— that message fails to be delivered to the source, to the other side. There is a failure, and the user still is not aware about it. But one thing that we had to keep in mind is the flip of that switch happened whether that failure occurred or not. It would be really weird for the user to have to be stuck in moving that switch until the server responds. In fact, if you guys or gals remember about the early internet, this used to be the case for UIs like starring or voting or any sort of thing that meant making a quick binary switch that the user had to get immediate feedback on meant refreshing the entire page. So that transition was not fast enough. Nowadays, we're thinking about optimistic changes of UI, which means if you actually flip that switch, even if that message gets delivered or not, you want an immediate response to your intent. So in the sense of common web applications, you will find that they've evolved to update the state of their UI immediately in an optimistic way so that if the data arrives later to the server, the response has been given to the user immediately. So a better way to think about this problem is actually about synchronizing state. The flip became on at a faster rate than that light bulb is going to become on. And the approach to solving this problem is over time ensuring that that source of truth that becomes the switch is synchronized with the light bulb and the light bulb is faithful to it. In, in the event of errors, what you're going to want to do is retry over time. Until you get an acknowledgment that the intent of the user has been fulfilled. Another example, maybe less trivial, that is helpful to think about is real-time text collaboration. So when it comes down to implementing a system like Google Docs or— excuse me. My switch is not working. So in order to implement a system that basically exchanges characters in real time, we face a similar issue in the sense that We have to synchronize data that gets optimistically changed, which means you're allowed to type in and nothing is blocking you from typing in. You're not waiting until the server gets your character until you see it on the screen. But we're dealing with a problem of higher complexity because we have simultaneously different sources of truth. So we have different people that could be editing on the same data source. And they're having the same expectations simultaneously. So for them, the collaboration has to be low latency. It has to be convergent, which means your changes and the changes of everyone else's that is editing simultaneously need to end up in the same consistent, uh, worldview. Uh, the, uh, intention has to be preserved. So for example, if you decide to erase a portion of the text, it cannot show up later because someone else made a change to that paragraph and that paragraph is somehow restored or flickers back to you. And causality has to be preserved, which means if you're making a series of changes that depend on each other, they have to be carried all the— preserved and carried all the way through to the eventual state of the document. In order to guarantee low latency, we can think of the last two points that we mentioned. In order to make sure that we have an accurate list of everything that happened, we have to make sure that every single change, every single character, every single movement that the user does, every part of the input needs to be preserved. And we have to do the best that we can to make sure that all those changes, no matter how small, are carried all the way through and preserved on the server side. So these three aspects of the implementation actually end up being related, because in order to minimize latency, which means also minimizing bandwidth, and also preserving every single little change that happens to a grove of text, we have to ensure that we capture changes in a very minimalistic way. So let's say for this example, I'm going to change— I'm going to edit this text, and I'm going to change the city from San Francisco to New York City. The most minimalistic way to express this change would be saying that there is a remove operation from position 7 to position 20. So we'll express it in some sort of encoding. We'll say, D7 to 20, and we're going to insert New York City in its place. So we're going to create another operation, I7 New York City. Obviously, this could also be merged into a replace operation to make it even smaller, but this is sufficient to preserve an entire log of everything that occurred from the user's standpoint, and also ensuring that these changes are as atomic and little as possible. In fact, those of you that use vi might have noticed that the way that we can encode changes that happen to a document happen to be very similar to how we interact with our editor every day, in the sense that we have commands that are very— or I would say at least very small compared to other editors, very precise, so that you're in charge of saying how the text is edited. It's sort of a low-level way of editing text. And actually, there's an editor that takes this one step further, which is called ed. But unsurprisingly, we have this approach even today, even though vi was created in 1976, because It was created at a time when bandwidth and latency was actually very essential to think about. So from an interview to the vi creator, there's an interesting excerpt that talks about how the shells that he was working on that led to the creation of this editor were so slow that he was thinking faster than the computer was reacting. So in a sense, since when you're logged into a remote shell, none of the changes are ever optimistic. So you want to make sure that the commands that you're writing over the wire are so small that when the response can come back as soon as possible so that you can resume your thinking. And basically this is the reason that today we have these commands. It wasn't so much that they were trying to create the best user experience, it was that it came from the standpoint of minimizing latency and bandwidth. So in our effort to address the other two points, which are convergence and intention preservation, we have to think about how can we take things that happen simultaneously As an example, a user might be editing at the same exact moment that another user is editing, and they don't know about each other's changes. But we always want to converge into a unified state of the world. So let's consider a document that starts as "Hello, world!" And one of the users wants to say, "Hello, JS world!" And simultaneously, the other user wants to say, "Hello, big world!" Guillermo Rauch. Again, the changes in their data sources were made optimistically. They were allowed to type, and on the screen they were getting the result of that input immediately. But when it comes down to conciliating these changes, this is fundamentally an undecidable problem by the convergence models that we usually work with. So if they happen at the exact same time, how can we make a sentence that both people are happy with. In terms of convergence, the problem is actually quite easy because we can just say we're going to have both words in the order that the server receives them. The server doesn't actually care about who typed first. Whoever had the fastest internet connection or was closest to the server would win. So it'd be a convergence approach of whoever types fastest wins. So in this case, we're actually sacrificing intention a little bit in the spirit of making sure that everyone that loads the document later, including those users, see hello JS World or hello big JS World in whatever order is decided. It just has to be consistent along time. And finally, the other aspect that we want to consider is preserving causality. So it's helpful to think about people that go offline. And this becomes really important in— in text editors, but actually it's also important in any sort of web application, because users, even when they have a WebSocket connection established, that connection might actually fail, data might have changed, and then a connection has to be resumed later. So an example of violating causality would be User A and User B start typing. Then User B types something that depends on the previous sentence. And that individual packet gets broadcasted to all the other members, which apply them to their own editors immediately. So in the case of User C, it's not enough to get the latest message as soon as you reconnect. It's also important to make sure that when you get that message, you verify that the state that you were in at that moment is perfect to apply that new message on. So in this example, User C should have waited to get the messages that new line depended on that User A and User B actually had. The way that this is usually implemented is by considering a revision number. So, bo— every user starts with a document and an integer. So, usually this is, uh, retrieved from the server and they say, "We all start from revision number 1." Now, if we consider the example where both users were competing to insert js and big in between hello world, we can think of them emitting two operations simultaneously. When those operations are sent to the server, we can attach that clock, um, that says, okay, I'm at— this is my message, but also attaching the information of what's the state of my world when I'm creating this operation. So both would send their insert operations, but also would attach what version of the dataset they're looking at. By the time the server receives them, it now becomes trivial for the server to say, OK, those two operations actually happened simultaneously, for they have matching revision numbers. And what the server can do now is apply a transformation function, return new operations back to the users that will result in a consistent and convergent view of the world for them, and then everyone is now at revision number 2, but since they got the operations back from the server that decided on an order, they now all have a consistent view. So they would have hello JS big world. There's a typo in my slide. I apologize. So it's worth now coming back to the idea of time and how we can think of synchronizing the clocks of the different users as they start making changes to the data. We can think of a real-time system as one that is actually rapidly synchronizing these, uh, sources of truth that are actually separated across distance and time. But in order to implement this effectively, we have to consider some caveats. The expectation is that the data is always coming through, and in the example, for example, a rich text editor, operations are always being uninterruptedly coming through the wire. But the reality shows that it's actually likely that, for example, the user can go on a disconnected state for a very long period of time. So something that applications need to be really mindful of is what I'll call state reconciliation. A quick test that you can do on your own applications to verify this is what happens to your application if you decide to close the lid, um, on whatever state your application was in, you close the lid of your laptop, and then a week later you resume that session. So what happens to your application? Does it stay stay in the state that it was at by the time it was closed? Does it try to refresh itself completely? Does it try to notify the user that he's been disconnected or she's been disconnected for a very long period of time? So it's very important to recognize that the approach has to vary. In the— in the example of Rich Text Editor, we can't just synchronize every single operation that happened because it could have been tens of thousands, for example. And then by the time you try to reconnect, you're basically synchronizing a very large data set. State reconciliation means that, for example, we would get a new starting point, and we would get only a portion of the operations in the past that led to that new starting point. So it's very important to think about the time that the user spends offline, not just about the idea that a message, doesn't make it to the other end, which would— can be thought of as a short disconnection. We also have to think about what happens if there's been a disconnection for a very long period of time so that the application would have a very hard time reconciliating, or it would spend a lot of time getting the new data in. The frontend also has to handle scenarios related to obtaining that data. If we now consider again that it's been, it's been a week that you've been disconnected, what happens by the time you try to get the data again? Maybe your session had already expired and you need to make sure that you handle the scenario of a complete UI change. For example, in this case it would be a logout state from, you go from an editor to a scenario where you no longer have even permission to write on the data. Um, a different user actually could have been logged in, so this would happen in the cases of you've actually started a new session on a different tab. Uh, again, uh, we have to be mindful that a single user can also pose a concurrency problem even when interacting with their own dataset, and this is becoming more and more true as you start interacting a lot, uh, with datasets from multiple users. Multiple devices. So you have one piece of data that belongs to you, and there is not other collaborators like in the example of a multi-user rich text collaboration application. But even in the case of a single-user application, it can happen that you've edited your own data set from a phone, from your watch, or other users have. So it's important to consider— concurrency problems even in a single-user application. And again, for example, if you're looking at a newsfeed, the delta between a disconnection and a subsequent reconnection can be so big that the approach to getting the new items has to— has an impact on the UI and the user has to be notified accordingly. The main conclusion to derive from this is that you can never make your frontend assume that you're going to have a complete understanding of every single operation that happened or every single event that happened to determine the current state of the application. And this ties into a final concept that I'm going to talk about today, which is rendering your applications on the server and then having the client side pick up. We can think of that scenario in a very similar way. In order to think about synchronizing data sets, we cannot only consider the application once it's been loaded. We also have to consider the user's initial intent, which is basically getting some sort of data, going to a URL, and downloading some information. How long is it taking you to download that initial state of the application? With the advent of single-page apps that promise fast and self-updating frontends, we've sacrificed that initial time that gets to get the initial data set. So in the case of most applications, you're going to find that you might have implemented a stack like React.js and Socket.IO. And in addition to your app code and styles and JavaScript code, Uh, so in an effort to guarantee fast and self-updating applications, you've written a large client-side application that actually doesn't give you the data right away as soon as you go to the URL, and several round trips are spent downloading that code. If we think about how the TCP connection works underlying that, you're gonna find that it's actually very, very bad to download really large, uh, code bases right away because TCP connections work in a sliding window. They're only going to absorb information progressively. So in the beginning, the first few segments that are allowed to be sent over a TCP connection means that in a single round trip, the user can get only 5 kilobytes of your application. Over time, maybe with the first 8 segments that most servers are configured figure to send nowadays, you— you'll find that you only have a 14-kilobyte window to send that initial state back to the user so that they can start processing the information that they were interested in. It turns out that the most optimal way to create a real-time application starts from the very second that the user has the idea of requesting the information and going to a certain URL or opening an application. And a very important caveat is the stage that we— the stage and the time spent downloading that initial code base. So you will find that one of the main advantages to this model is If we— if you render server— if you render data on the server and then you have the client take over, now you have a disconnection situation once again. So by the time the client loads the JavaScript code of the data that's been dumped by the server, by the time a WebSocket connection, for example, is established, data might have changed. The other important idea is the client is allowed to edit, uh, the application optimistically. This is why you don't want to just do server-side, um, rendering. You also want client-side rendering because that way you'll— you're gonna be able to respond immediately to the user changes. So the solution to this problem means dumping the state, uh, initially and then having a, a codebase that la— that loads lazily that is able to reconcile changes and then enable subsequent changes to the state without the connection necessarily working at a very fast pace. So for example, once your text editor loads, changes can be made immediately. And maybe the connection is really slow, and synchronizations will happen over time. But that shouldn't hold the user back. In summary, reconnection and reconciliation become two very important aspects to think about once we want to create this ideal real-time system. It's useful to consider that initial page load download as the total time that you're considering when you measure your latency. A problem that I see usually is that when you're measuring latency, you consider it in isolation. So you think about, for example, how long is it taking me to— get the— in the case of an API, get the Apple stock information. Oh, it takes me 200 milliseconds. But it's actually way more useful to think about how long did it take from the moment the user was interested in the information of getting that stock, which means, for example, loading your website or making a search, to when the information is actually given to them. And subsequently, how long is it taking to deliver updates? To that information that the user has just acquired. The main conclusion is then that we have to focus on accelerating the synchronization of ideas and intents. We have to make sure that the user is able to get an immediate response to their input, and the subsequent reaction from all the other data sources are applied to that user's state as soon as we can. And for that, we have to take a global view of our architecture, optimizing everything from the networking stack, making sure that push is enabled through a technology like Socket.IO, but also considering the problems that occur related to synchronizing data, network disconnections, problems related to long latency that can potentially be masked, and so on. So hopefully you guys have gotten a few important lessons, and you can apply them to the design of your own real-time systems. Thanks a lot.