Jay Kreps | Kafka Summit 2018 Keynote (The Death and Rebirth of the Event-Driven Architecture)

Hello, I'm gonna be talking today about something I'm incredibly excited about. This is something I think is missing from a lot of the systems we build today. It's something Kafka has been really oriented around and part of the vision for that. I'm gonna be talking about events. And events are something that is so simple, it's easy to miss them. And so I should start out and just say, what is an event? Well, it's the simplest thing you could imagine. It's something that happened. It's just a bit of information, an immutable record of something that occurred. And it's so simple that it almost disappears. We know that we have these things happening in a business. There's sales, there's invoices, there's trades, there's customer experiences. All of these you can kind of naturally think of as events. But the world of events is actually a lot bigger than this. And this is what makes it powerful. Events don't just capture that kind of thing. They can also capture state. And they can capture the change in state. So if you have a database and a row is updated, you can think of that update as an event. You can go from old value to new value. And that change event, that change in state, that's kind of a very fundamental thing. And you can think about this type of state change as being a representation of what's happening in that table. So instead of just being a static data set, you could see it as a series of change events. And this is common in databases and in distributed systems where this change log of events is used to take a table and turn it into a stream and replicate it elsewhere. And this is a really powerful concept. It allows us to take that stream and materialize it in a different system. It allows us to take that stream and process it and turn it into something completely new and derive from it and keep it in sync. And this notion of a stream version of tables, it occurs everywhere. So it shows up in event sourcing. It shows up in database commit logs. It shows up in database replication. And it's what makes it so that you can kind of represent the full spectrum of data with events. You can represent the kind of things that are happening. You can represent the communication between microservices. And you can represent even these kind of tables of data as streams of events. And this is what makes stream processing such a powerful paradigm, the idea that you could take all of these different things and have a way of responding to it, reacting to it, transforming it in real time. And part of that power comes from a characteristic of events, which is that they really have two functions. They function in part as triggers, something that makes an application go do some work. So we would think of this much like a request-response. If I issue a request to your HTTP service, and the service goes and does some work and sends back a response, that's kind of a trigger. I've triggered some work. And events do this. So if, if, if I issue a sale event, and some service goes and fulfills that order and, and sends something out, um, it's acted as a trigger. But that same sale event is also a piece of data, and this is the other function of events. And, um, if you had that stream of sale events, that's actually a dataset. It's an infinite data set that's always growing, but it's a data set and you can compute something on it. You can compute how many sales have we had. You can compute any fact about that data. And so the world of events, actually, it blends this idea of triggering work with this idea of data and computation and transformations on data. I'm gonna come back to this idea as I talk about this. So if events are so powerful, You know, where are they? And the answer is they haven't really shown up in infrastructure. This is in part what we're trying to solve with Kafka, but they haven't really had a proper home. So, you know, they don't show up in databases. Relational databases have no real concept of events. They don't really show up in RPC systems. They aren't really there in the infrastructure that we built. They're kind of hidden, they're implicit. You have to hunt for them. They are there in your code somewhere, but you don't really see an object that's called event most of the time. And yet, in some sense, events are everywhere. Like, if you think about what a business is, it's a process that generates value, right? That's what a business is. It's something you do over and over again that creates some economic value. And the way you, you know, actually model a process most naturally I don't think it's with static tables. I think it is actually with events, things that happen and how those are reacted to. And if you ask somebody to draw out kind of how a business process works, even a non-technical person, they'll often, you know, kind of do it in this terms, like, hey, this happened and then this, we do that in response and then the sale happens and then we ship this out. It's a very natural way of thinking about what a company does. And this is something that's starting to show up much more in the world, these event-driven architectures. So it's not a new term, but it's something that's gaining a lot more prominence. And you can see this in all different types of companies. An example I like to use is retail. We all kind of understand retail, 'cause you buy things from stores. And you can see how events show up in the operation of a retail business. So imagine this is kind of the architecture of a large retailer, where we have, you know, online sales, you know, through a website or a mobile app. We have brick-and-mortar stores. We have distribution centers that are maintaining inventory. And our goal is to get the inventory to where it needs to be, either by shipping it out to customers or putting it into stores. And how do you, how do you build and operate a company like this? Well, the easiest way to visualize it is that retail kind of happens in response to the stream of events. The stream of events is the sales. That are occurring in the business. It's the shipments of new products that are arriving. The backend of retail is basically placing new orders as this occurs. It's repricing products, it's moving things from distribution centers to stores, it's fulfilling orders and shipping out. That whole backend state machine, you can see as a kind of stream processing that's happening in response to these core streams that are the operation of the business. And when you try and build something like this, what, what do you have to build it with? Well, with traditional systems, mostly what you have is tables. You have data stores, you have caches, you have places to stick state. And that's actually not enough. You need tables. Tables are where you keep the current state of the world, and that's incredibly important. But you also need streams. You need streams of events. And these two things interact. If the tables are kind of the current state of the world, and the events are what's happening in the world, those things are closely related, right? Like, the things happening in the world, they affect the current state of the world. And the current state of the world is what causes the things to happen, right? And so those things are, they interplay. And part of the power of modern stream processing is actually bringing these two things together, giving you tools that allow you to model both tables of state and streams of events. And this is what has been going on in Kafka over the last 3 years, adding stream processing primitives, adding the Streams API to do declarative stream processing that combines tables and streams of events, and with KSQL, the SQL layer that Confluent has been building to do SQL processing on streams of events in a declarative way. The fact that you need both of these concepts, that they're both really fundamental to how you would model a business, I think one argument for that you could find in data warehousing. So for people who know data warehousing, they're probably familiar that there's really two kinds of tables in a data warehouse. There's dimension tables and there's fact tables. And so dimension tables, they're kind of just like the tables you're used to. They're tables that have all the entities of the business in them. So there's nothing too special about them. But fact tables are a very particular type of table because in a fact table, you have a bunch of time-ordered immutable series of things that happened. That's what a fact table is. It's a bunch of facts. And so what does that sound like? Well, it sounds like an event, right? So in order to model a business, you have to basically, reinvent kind of a half-baked version of events in a relational database to be able to do that analysis. And that's because events are so fundamental to modeling a business. Now obviously, you don't quite have the full power of events in a data warehouse. You can't subscribe to the facts as they occur. So you don't get the trigger usage, but you do get the data side of it. And I think the fact that you have to have that shows why it's so powerful. And you can see this, these two views show up in how people talk about this area. So people will talk about stream processing as a way of building event-driven microservices. This is one view. It's a way of building applications that respond to events and they may issue new events. So in the example I gave, I was responding to a stream of sales events. I was keeping track of inventory and I was issuing new orders to place to get more inventory so I don't run out. And I can think of that, that's kind of a microservice, right? It's a microservice that consumes a feed of events and produces an output feed of events. But the other view of stream processing is as a kind of data transformation tool for data pipelines. So I can imagine doing something like a Customer 360 where I'm gathering feeds of data from different systems that have bits of data about my customer. And I'm performing a join on all of these. So I'm taking the change feed from all these data stores that have customer information, I'm stitching it together and creating a unified view of my customer. And this example is much more of a data pipeline. It's less a microservice, it's more a data pipeline, right? And yet both of these two things are actually the same thing. They're actually just event stream processing. I'm taking in streams of events, I'm combining that with possible tables of data. I'm outputting streams of new events. And so the technology to do this powers both of these use cases. The fact that you can put both of these together is what makes this idea of an event-driven architecture powered with like a streaming platform where all these streams of events can come together. This is what makes this so powerful. This is our vision for, you know, how a modern digital company would be built is that you could, capture these streams of data. It could be the events generated by microservices, it could be the data feeds captured from existing data systems that might come via Kafka Connect, and you can have all of those represented in one place and they can feed out to other systems. You can compute derived things on top of those streams, you can do real-time analytics on top of those streams, and this is what can connect up all the different parts of the business. And this is a very different way of thinking about the communication between parts of a company. It turns it inside out in some sense, right? Instead of me having to know everybody who I need to talk to, I just say what happened in my area of the business. I just publish my stream of events. This is what occurred in my area of the company. And anybody else who is interested taps into that stream and does what they need. And this, this scales so much better because It truly does decouple the different systems from each other. I don't need to know everybody who reacts to a particular event. I just know that this event happened. That's all I need to know. And so, you know, if you've been around in the industry for a while, you could be forgiven for thinking that none of this is really new. Like, we've kind of seen all this before. We've heard about events for a long time. We've heard about systems, you know, that would process events. We've heard about event-driven architectures. None of this is really new. So, what's different this time around? Is anything really different this time around? Is it all just kind of the same stuff? And I'm gonna claim that there are some things that are different. I'm gonna claim that Kafka has some fundamental changes in how it's built that makes it truly different from the things that came before, and I think this is why we're suddenly seeing so much success in the use of events and event-driven architectures at scale, you know, in terms of companies' whole architecture. So I'm gonna go through 3 examples of things that are fundamental to Kafka that make this possible in a way that it wasn't before. So the first is, I think Kafka gets the semantics of events right, and without that, it's very hard to work with event data, and it does this in a number of different ways. The first is that Kafka is built around a fundamental primitive that in distributed systems is called atomic broadcast. This is the idea that I can publish something, say something, send a message, and have any number of receivers get that message, that it won't be lost, and that each receiver can get it just once and take action on it or process it or internalize it. And this is exactly the kind of communication you want in an event-driven system. This is exactly the primitive that you want to build around. In the absence of this, you have to do a ton of work to make up for it in your app, to correct for possible loss and retry and failures. Having this done for you is extremely helpful. Kafka also acts as a proper storage system to store the stream of events, and this is really important for that use of events as data. I talked about the use of event streams as a data, as a dataset you can compute off of. Well, obviously that is only possible if you're storing the data. If it transiently disappears as soon as somebody reads it, then you don't have that usage. And so these two things really help create a primitive that is much more useful for building event-driven applications. So that's the first advantage. The second is that Kafka is providing much more powerful primitives. So a traditional message bus, is a pretty low-level piece of infrastructure. It says, you know, hey, here's a message, go, you figure it out, go do something with it. And that's great, that's useful, that's a useful thing to do, and if you looked at Kafka, you know, maybe 3 years ago, it wasn't that far from that. It would say, hey, here's a stream of events, go do something with it, but it didn't really get much beyond that. But if you think about what Kafka does today, the integration of stream processing into its protocols, The integration of stream processing APIs that are available, things like KSQL, this significantly raises the level of abstraction that you can work with. Instead of saying, here's an event, go figure it out, you can say, here's an event stream, go do on. And you can do declarative operations on top of these streams. So rather than kind of imperatively mucking around with each event, you can say, hey, this is what the output stream looks like based on its input streams. Compute that. And that's a much more powerful way to think about data. To me, it's kind of analogous to the difference between, you know, a file where you can muck around with individual bytes and a relational database, which gives you these kind of powerful and declarative ways of doing computation on top of rows and columns without having to say exactly how it's computed. And this, uh, these primitives are strengthened by the ability to combine two things that were separate before. So what we've had before is we've had databases, and they kind of do tables only. They have no notion of events. The events are kind of implicit somewhere in the app. The app figures that out. And we've had messaging systems, and even the more primitive stream processing layers, and all they've really had is messages or events. They have no concept of, of of tables or state. But I said these two things are really closely related, the state of the world and the things happening in the world. You know, the whole problem is to do computation on top of those two things. That's what one of these event-driven applications is. And if you can't do that in a way that's correct and easy to reason about, it's gonna be really hard to build an application in this space. What you need is something that does both of these two things. And this is a critical part of the stream processing features in Kafka, is this ability to take tables, represent them as a stream of changes, capture them using something like Kafka Connect, and then do stream processing that combines that table, even in its stream form, with events and emit new derived tables and derived event streams. That's a really powerful thing. So this is, this is the second advantage that Kafka has, is these kind of powerful primitives that move the thinking up the stack. And the third, and maybe the most important, is the rise of modern distributed systems has let us build systems that work at a different scope. So instead of being a piece of infrastructure that supports an application or a few applications, you can build something that really works at company scale. And that's a problem where the devil is in the details. So if anybody tells you they've done that, don't believe them until you actually see somebody use their system at company scale for like, 100 or 1,000 applications and 100 or 1,000 engineers working against it. It's a hard thing to do, but if you can do it well, it's incredibly powerful. This is what enables you to have the set of all the streams in a company and actually get them in the same place, and that's a really powerful architectural primitive to have. So those are 3 of the differences in Kafka, but that's not the only thing that makes the world different now. I think that the world has changed, and so one of the biggest drivers of events and event streams has nothing to do with the technology. It has to do with what's happening in the rest of the world. And there's a set of changes which are motivating large-scale event streams at scale. The Internet of Things, the rise of microservices architectures and the need to decouple big monolithic applications, the rise of mobile, these machine learning applications that tap into bits of data from all over the company. All of these are either big producers or big consumers or both of event streams. And if you think about it, all of these are really part of kind of an uber trend, which is the digitization of business. Business is becoming something that is digital. It's going from a world of kind of people and paper and processes, and we're moving into a world where business is kind of built in software first, where the architecture of a company is something designed in software first, and we're— all companies are trying to get there. Startups are trying to build that from scratch, Big existing companies are trying to evolve themselves towards that. And I think that's where this kind of streaming platform vision comes in. This is what we think the architecture of a modern company looks like. We think a modern company will be built around data streams. And this streaming platform will be the central nervous system that ties all these pieces together. And if that's successful, and if this is able to take on the world, as it has in a number of companies, That's gonna be a major data platform. It's gonna be, I think, bigger than relational databases. You can think of this as, in a sense, kind of a generalization of what databases do, where you're having not just tables but also streams and bringing those together in a platform that really works at company scale. And so one of the really exciting things about a conference like this is we're pretty early in this process. So we're just getting to see this come about, and you can hear from companies who are in different stages of that journey where they're either have just done their first project in this fashion, or they're actually already kind of have the central nervous system vision, and they have hundreds of applications, and they have all the problems that come along with that. And you can see, how are they addressing those problems? What part of this vision have we built and succeeded at? What part is not done yet? And how can you kind of guide yourself on that path? And I think it's going to be the people in this room that help make this happen. So I think that's an incredibly exciting thing. I'm really looking forward to the rest of the conference. And thank you all so much for coming and being a part of this.