Jay Kreps | Kafka Summit SF 2018 Keynote (Kafka and Event-Oriented Architecture)

Today our first keynote is, uh, gonna be Jay Kreps. Now he's kind of a man here who needs no introduction, but I don't care, I'm gonna do it anyway. Jay, um, Jay was, uh, lead architect for data infrastructure at LinkedIn years ago, where he was co-creator or creator of projects like Voldemort, Project Voldemort. Anybody, any Voldemort users? Yes, shout out in the back. Okay, Apache Samsa, Samsa users? More hands, look at that. This other one he co-created, Apache Kafka. Please, no, don't put your hand up. That's why we're here. Anyway, it's my pleasure to introduce Jay Kreps.

All right, Voldemort. That's really bringing me back. Welcome, everybody. I hope you're all as excited to be here as I am. So I'm gonna be talking today about events and about event-oriented architecture and how it's affecting the systems that we're building, software applications. This is not a new idea, but it's something that I think is really coming from the background into the foreground now. And so the title of my talk overcome by events. This is actually a military acronym, OBE. And so if your plan is overrun by the course of events, then you can say it's OBE, right? And I think, I think that's actually something that's happening in the world. It's happening in a lot of different areas and systems. I'll try and outline some of those. They're kind of becoming OBE. And so let me start and say, what, you know, what is an event? This is an idea that is almost so simple that when you explain it, it sounds silly. An event is just something that happened. It's just something that occurred. And that seems pretty fundamental. Like, if you think about what's an event in a business, it's a sale or an invoice or a trade. It could be something low-level, like, you know, some low-level thing an application does, or it could be something high-level like a customer experience, something very rich. But if you think about where are events, where have they been, they haven't really had a proper home in infrastructure. They've been implicit. They're kind of hidden. So if you look in your code, there wasn't really a lot of events. If you look in data infrastructure, there's no real representation of this concept. What we've had is, you know, databases and programming languages and RPC layers, but we haven't really had events anywhere in our stack. And I think increasingly they're becoming visible and they're coming to the foreground. And I'm gonna go through a couple areas where I think this is happening. So I think it's happening in microservices. I think it's happening in monitoring. I think it's happening in data pipelines and ETL. I think it's happening in analytics. And I think that the fact this is happening across a lot of these areas, actually there's some synergy between these. And so I'll go, I'll go through a few examples. So if we start with microservices, I think this is one of, maybe the most important project in software engineering is how can you get the same productivity when you have 1,000 engineers working on something that you had when you had 5, right? How can you scale software engineering? And nobody really knows the answer to that question, but it kind of fundamentally has to involve chopping the problem up into pieces, right? It has to. And so the only question is, how do you do that? How do you make all those pieces work together? And the way we started doing this was really with these very synchronous request-response, typically REST-based applications. And that works pretty well for a certain type of problem. It works well for UIs. It works well for interactive UIs, something that is itself synchronous. And it's no surprise that this was kind of the early technology stack for microservices, because a lot of this movement came out of a bunch of web companies that were in large part big UIs. They were kind of big CRUD applications. And it worked. It actually worked to scale engineering. But it only goes so far. And I think as companies try and interact with the real world that happens at a different pace, right, as they're no longer just a CRUD interface, as there's asynchronous things that happen in the background, as there's a more complicated company that becomes connected, I think synchronous doesn't really scale. And that's, that's kind of led to the emergence of these event-driven microservices. So, so microservices that react asynchronously to what's happening elsewhere. And this allows, like, a much greater level of decoupling. It allows services that don't need to directly connect to the things upstream of them. And it allows the producers of these streams to not need to know everything that reacts to it. And this lives kind of in harmony with the synchronous services. If you're taking a frontend application and decomposing it, you're probably doing that with REST. But a lot of the backend parts of a business are fundamentally acting in response to what happens. and this event-driven paradigm is really taking over. And we've seen the rise of this very rapidly. So people have only been talking about this for really a matter of years, a few years, but already if you look at a poll Pivotal did a few months back, they found that Kafka was half as popular as REST already as a layer for implementing microservices. And that's a huge change in how people are going about this, and I think this is a huge rise of events in service design. And you're also seeing it as part of what's happening with these serverless systems. So when you think about serverless, the, the thrust of it is really more about getting rid of kind of unneeded infrastructure concepts, not having to manage individual machines and containers and so on. But what goes almost unnoticed is a lot of the applications people are building are fundamentally reacting to events. They're fundamentally processing some stream of events. And the use of events doesn't just go between the services. It's actually trickled into the internals. So there's been an emergence of event sourcing architectures, where the internal data representation is based on the events that it's built off of. And this doesn't make sense for every application. For a simple CRUD application, you probably don't need something like this. But for really complex domains, this is absolutely necessary, and it's become popular. And this is, this is happening naturally in the stream processing space. A lot of the stream processing capabilities in Kafka are kind of internally event sourced, and the natural representation for those events is a kind of stored commit log of what happened. That's a lot more valuable than stitching together a queue and a store and many other things. Feeding off of that log of events is the way that this is most naturally built. And so that's in the area of application development, but events are happening elsewhere as well. So the monitoring stack is changing. So if you ask people, "What is monitoring about?" Most people say, "Well, we have metrics, and we have alerting, and we have log aggregation, and maybe we have this new thing, these kind of service tracing frameworks." But in a sense, a lot of the smartest people looking at this have thought, you know, this is actually the old school. And so there's kind of an emerging idea that maybe there aren't really 4 disconnected products. Maybe there's really something underlying all these, which is events and event streams and event processing. And maybe each of these are almost an interface to that in different ways. And so what do I mean by that? Well, if you look at a log and you think about a log aggregation, something like Splunk or any of these systems that are trying to collect your logs. What is a log? Well, it's basically a time-ordered series of things that happened. It's basically a bunch of events, and it's events about what your application did. And if you think about metrics, what are metrics? Well, most people would say, well, it's kind of a graph. It goes up and down, and it shows me what my application did. And so what's the connection to logs? Well, traditionally there's not much of one. But if you think about it, metrics are really just an aggregation on the stream of events. You could be counting the number of errors or you could be measuring the request latency. And so here's a, here's a KSQL query. This is something you can actually run in Kafka that's doing a type of metric computation off the raw events. And so what's the power in thinking this way? What's the power in thinking about events? The power is if you have real infrastructure around events, there's all kinds of things you can compute that you couldn't before. So instead of just defining low-level metrics on a single machine, you can actually do a much better job of defining what the real SLAs of the system are. What does it really mean to do things correctly? And checking that, like, in real time. And that's a very powerful thing. And this translates even to alerting. So most people would think of alerting, you know, you get a phone call. But what is an alert? It's, it's an event. That occurs in response to some criteria. And again, the benefit of thinking in terms of events is the ability to actually define something very rich. A lot of the movement in systems is not to alert off of low-level things like, "Oh, the CPU was too high," but to alert when there's real customer impact. That's when you want people to wake up. Measuring a bunch of low-level things typically just creates a lot of noise. If you can measure what you really care about, That's much better. But measuring what you really care about is more complex. You, you actually need a way of expressing what you care about. And so the emergence of events and infrastructure around this is really a powerful idea. And this applies even, I think, to distributed tracing. LinkedIn built a system that was oriented all around collecting all the Kafka streams from all the machines and measuring not just the latency, but the cost of those systems. What did it cost to serve a particular page view? Measuring the errors and how that propagated through these call graphs. The ability to do this came from actually thinking about these as event streams, and we were able to join on not just events that were operational, but also events that were about the user experience and what people were doing and the business impact. And that's the power of these things, is once you start to think in general terms, a lot of things that seem different suddenly become connected. Connected. And I think you can see that as we move to the next topic, which is data pipelines. This is an area with Kafka that we've talked about so much that I don't know that we even need to go into great detail. But if you, if you think about what's happening as you pipe data between systems, this has traditionally been this kind of batch data dump, this big squiggly mess. And the power in this area comes from making it real-time, making it scalable, and thinking about a database not as a data store, not as something you dump out to a text file, but as a stream of events, a real-time stream of events that anything can subscribe to, and being able to move to an architecture where all this integration can happen off a central platform. And this reduces this kind of rat's nest of one-to-one connections. And the power of this is now, as you combine these different areas, you can think about, well, what happens if I apply these monitoring-like primitives to the changes in my data pipelines? Can I monitor that? Or what if I have an event-driven application that triggers off of database changes? Or what if I have an application that feeds data out to these monitoring systems? These areas can actually interact off the same streams. So you can, you can apply monitoring primitives not just to low-level things, but to the kind of rich customer experiences, the things that actually make you money, that you care about, and likewise, you, you can apply them to these kinds of data pipelines. And so the final area I'll go into is, is analytics, and this is an area that's actually been built around events for a long time. So one of the fundamental tenets of data warehousing is how you define your data, and if The first thing you would do if you were setting up a data warehouse is you would build out a star schema representation, and right at the center of that star is something called a fact table. And what's a fact? Well, it turns out a fact is an immutable record of what happened with a timestamp. And so does that sound like anything you've heard of before? It turns out, yeah, that's, that's basically exactly the definition of an event. And so a data warehouse is actually built around an event stream. And even the dimensions kind of evolve over time. You usually keep all the versions of it. And the ability to do analysis is really dependent on this. And I think this is interesting. I think when you see people come up with the same concept with different names over and over again, there's usually something going on there. There's usually some idea that actually makes sense. And so I think this is actually kind of a proto-event definition. Now obviously, a data warehouse is a— a pretty slow event stream. It only updates maybe once a day. But still, the fact that they have a similar data definition tells us something. And as this starts to move towards a much more event-driven model, you can suddenly do real-time analytics. And the difference between real-time analytics and monitoring is actually a little bit blurry. It's actually not clear which is which. You know, monitoring typically means on data streams coming from low-level application data. And analytics typically means on higher-level business things, but to a certain extent, the primitives don't care what the data is. And this becomes even more important as the definition of monitoring and— or as the definition of analytics itself shifts. So, you know, it used to be that analytics meant making a report, and somebody would look at that report, and then they would make a decision or take action. But increasingly, the point of the analysis of data isn't to enable some human to make one decision once a week. It's to actually automatically make these decisions all the time, continuously, using machine learning or even simple rules that kind of optimize the customer experience. And this kind of closed feedback loop, it's not even clear, is this analytics or is it a production application? I mean, the answer is yes, it's kind of both. And the demands of this type of application are very different. It's, it's very, very data hungry. It requires you to be able to tap into, you know, primary data from databases, but also activity data that is about what people are doing. What are they interacting with? What was the outcome? That kind of event data. And so the mixture of all these data streams is very difficult, and building this around event streams makes it much easier to maintain this type of application. You'll see in the architecture diagrams of people who are building production machine learning applications, you'll see something like this picture very commonly. I think events are actually everywhere, and they're kind of coming out of the woodwork in all these different areas. I think this makes a lot of sense. I think if you think about it, a business is a series of events and the reaction to those events. That's really what it is. In fact, if you ask a non-technical person to diagram out some business process, they'll often say, well, okay, this happens, and then we do this, and then that happens, and then we do this. They're fundamentally describing a series of events and the reaction to those events. That's how they would explain themselves. And so this is something that's really, really essential. And that brings us to the infrastructure for events. What's the right way to represent this? How have we done this traditionally? Well, traditionally, we haven't. Traditionally, there hasn't been much. What we've had, really, is a bunch of databases with tables. And databases are important. That's not something that's going away. It's not something that's gotten less important, but it's only half the picture. And so, you know, a table can store the current state of the world. That's how the world is right now. And that's an important thing. But events are about what's happening in the world. And there's some interplay between these two concepts. So the state of the world impacts what happens in the world. And what happens in the world impacts the state of the world. and that's, that's a statement that's true, you know, not just in technology. That's like true about the world, right? These things have some interplay between each other, and you need both of them, and you need to represent both of them. But we've only done half, and the other half we've kind of been faking it in the application space on our own. And by bringing these together, it's actually really powerful. And this is, this is what's often called the table-stream duality, which sounds a little fancier than it is in the stream processing space, where If you look at the stream processing features in Kafka or KSQL, they actually allow you to do this. You can, you know, connect up to databases and treat them as a stream. You can transform that stream. You can materialize it back into other tables. You can take those tables as a stream and transform them again. And so this interplay is built really, really into the heart of how stream processing on events works. And as you put together all these different areas I talked about services and microservices. I talked about monitoring. I talked about analytics. As you put all these together, the power of this is the ability of all these to connect. When a sale occurs, or a trade is executed, or there's some user experience that occurs, all the different things that can key off of that, whether it's a graph that has to update, an email that has to go out, compliance or regulatory stuff, security systems that need to analyze that, The ability for all of those things to happen in reaction, that's the promise of this world of events, and that's really the origin of this idea of a streaming platform, and that's the vision that we've been building towards with Kafka, the idea that you could have all these event streams in one place, that all these different systems could hook up to this, and this could be the interchange for all the things that are happening in your company, and so I'll go through a couple of key areas in Kafka that have actually helped make it the de facto standard for doing this. We've seen companies all around the world adopting this type of architecture, and they've built around Kafka, and so I'll give a few examples of what we've tried to do in Kafka that have made this work. So the first and the best known is this kind of immutable log-centric architecture. I think this is, you know, really the data structure for a stream. You have these, this sequence of time-ordered events. It's multi-consumer. Anybody can tap into it and subscribe to it. You can go back and reload from the beginning of time as you want. I think this is a really powerful abstraction, but it's not the only thing. So built into Kafka is really pretty deep support for tables and log compaction, and this, this supports that idea of the table-stream duality, the mutable data, the static state that you're computing, and this is really important to bring together these two worlds so that you can take the state of the world as well as what's happening in the world, and the next thing is that Kafka's been built to be horizontally scalable, and the reason this is important is not just because there's really big applications out there that have these massive data streams.

There are.

I mean, if you look at IoT applications, some of them are massive, It's because it can grow from a platform that supports one application to something that supports a number of these. It can do this gradually, right? So one application kind of brings its data streams, and other applications come to get this, and it happens organically. And this can grow until it's really used across the company, is really kind of the central nervous system that all these things plug into. And the ability to do this bit by bit is incredibly important. I think no system gets rebuilt all at once. And so this ability to scale behind the scenes as, as people plug into the platform is very important. And then some of the areas we've worked on a lot in the last few years is the connector architecture, the ability to actually plug in off-the-shelf systems that don't know about events, don't know about Kafka, and be able to suck out event streams from them, to be able to get off-the-shelf connectors for databases and data systems and SaaS applications. This is incredibly important to be able to just plug this in to the existing things that you have. And finally, the transactional semantics for stream processing. This has been a big deal. This is the ability to get exactly-once semantics, which really means just getting correct results as you do stream processing, and this is something that cuts all the way across Kafka from what's stored in the log, to the protocol, to the stream processing libraries. And what this allows you to do is, in a distributed architecture, actually take input streams, transform them, maybe in a stateful, rich way that goes across messages, and produce output streams, and do that in a way that's correct, right? So one of the huge things that's held back this area of working with streams of data is you're not really guaranteed to get the right answer in the same way that you are with a batch system. The reason people were running big batch jobs at the end of the day is you either get the output and it's correct, or it fails, in which case you rerun it and you get the output and it's correct. And moving to a streaming world really requires having similar kind of guarantees that you can build around. There aren't that many programs that we can expect to get the wrong answer and, and still be happy with. And the ability to do this in a general way is in a way that doesn't require the application developer to have to do all kinds of hijinks to make it work is incredibly important, and so this has played out in Kafka Streams, it's played out in KSQL, it's literally just a setting you change, it's available to any of the other systems that wanna integrate, and I think this is an incredibly fundamental thing. So those are 5 of the things that I think we've done in Kafka that have helped it become this kind of de facto platform for events. But it turns out there's something I think is even more important than any of those 5 things. Something that I think is actually the most important thing of all, which is the community and ecosystem around it. So this type of platform, it isn't just a piece of software. It's something that people build around, that they're actually architecting their companies around in many cases. And so, you know, all of you who are here today, all the people who have helped submit code, All the books that are out there, the blog posts, the websites, the tutorials, the Stack Overflow answers for the obscure error messages, all of those are important. If you go onto GitHub today, there's 20,000 repositories that mention Kafka in some way. That's a ton of code. That's clients in different languages, it's connectors for a zillion systems. That's an incredibly powerful thing. A lot of that is new or experimental, But some of it isn't, and it's code you don't have to write if you build around this system and this ecosystem. And there's now over 100 meetup groups worldwide with 28,000 members. These are all people who are trying to figure out how to make this new way of thinking work, who are figuring out all the problems and all the solutions. And so I think as we look at this vision of where we're trying to go to and what we're trying to build, I think it's that community that's really the most powerful thing. I think if you want to make some big change in the world, if you want to actually rebuild companies around this type of architecture, you're going to need help, right? You're going to need a lot of people who can help you figure out how to do it. And I think that's something that we have. And it's what I'm most excited about in this conference over the next few days is getting to learn from people who have done this and everybody getting a chance to share ideas and best practices and make connections in the world around how this is progressing. And so with that thought, I hope you all enjoy the show. I'm incredibly excited to get to meet all of you. Welcome, and I hope the next two days are everything you hope for. Thank you very much.