Episode 393: Jay Kreps on Enterprise Integration Architecture with a Kafka Event Log

This is Software Engineering Radio, the podcast for professional developers, on the web at se-radio.net. SE Radio is brought to you by the IEEE Computer Society, IEEE Software Magazine, online at computer.org/software.

For Software Engineering Radio, this is Robert Blumen. I have with me today Jay Kreps. Jay is the co-founder and CEO of Confluent. Prior to founding Confluent, he was the lead architect for Data Infrastructure at LinkedIn. Jay is the author of a book on event data and stream processing, and has appeared as a guest on SC Radio episode 162, Project Voldemort. Jay is the author of a LinkedIn engineering blog post called The Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction, and we will be talking about that blog post today. Jay, Welcome back to Software Engineering Radio.

Thanks so much for having me.

Anything else you'd like to tell the listeners before we start the actual topic?

No, I think that's a great introduction.

We're gonna work up to the idea in your blog post about an enterprise data integration architecture organized around a log. I wanna establish some building blocks first. Let's talk about when we have a large enterprise with many systems, if you looked at that enterprise from 30,000-foot view, what does it look like?

It usually looks like a big mess. I mean, at least, you know, in that blog post, in a lot of the work we've done at Confluent, we usually kind of draw this spaghetti mess of interconnection between applications and data systems with, you know, ETL processes that kind of shunt data around in real-time messaging and point-to-point kind of RPC communication. And usually most people look at that big kind of spaghetti mess And they say, "Yeah, that's right, but it's about 1,000 times more complicated than that in reality." You get some idea of sizing.

So how many strands of spaghetti and how many meatballs in a big organization? We're talking tens, hundreds, thousands?

Yeah. I mean, each company is different, right? But I think one of the key points is in some sense, if you're connecting everything to every other thing, then, you know, if you're going to have n things, you're going to have like n squared connections between the things in that kind of fully connected world. And of course, no organization is fully connected, but they're always striving towards it. You would always hear, oh, you know, this needs to be integrated with that and so on. And, you know, I think that kind of scalability problem is maybe at the root of a lot of the challenges that haven't really been solved in the prior generations of technology in this space where they were you know, they're kind of helping you integrate thing A with thing B. But, you know, as you do that and as you go down this path, it's not really scalable, you know, across the whole company.

When you're talking about thing A needs to integrate with thing B, what do you mean by that?

You know, it could be all kinds of things like you see all kinds of these kind of data copy processes where data is getting pulled out of a relational database and put into some other relational database or data store. You see integration between applications. That could be as simple as this service calls that service, or it could be something more complex than that. You see a whole set of kind of messaging systems, message queues that send off messages and are trying to support more complicated patterns of routing and interconnection. You see essentially all these kind of somewhat ad hoc data integrations or pipelines and technology to support it. But if you step back and you look at a company and you say, "Hey, do we have access to what's happening in the company in some kind of standard way that anything can plug into?" Usually the answer is no, or not really, or only partially, or it's relatively complicated to get at what you want.

You use the term data integration in your blog post. Do you mean something in particular by data integration that's more specific than integration in general?

No, I really just meant plugging things together. How do applications interconnect? How do data systems interconnect? I think maybe if I step back, if you think about the history of software engineering and software development and a lot of these software platforms, the problem that was focused on for a lot of history was how to build one application. What's the database to support one application? What are the programming frameworks? What do we need to run one app. And I think a lot of the interesting things happening in the world now are when you kind of zoom out. Now people are like, hey, we, we don't have one application and that application doesn't exist in a vacuum and it has to exist as part of a whole kind of interconnected company. Like in some sense, we're not building one app in software, we're building like a whole company in software that needs to somehow make sense. And when something happens, there's many, many things that need to, to react. Like one example I often give to make this concrete is if you think about a big retailer. A lot of people can kind of understand retail because you understand stores and buying things, and so it's kind of an intuitive domain. And you think about, well, how many different systems need to update or integrate the information if something sells, if they sell a product to a customer? And it's clear there's analytics systems and there's all kinds of coupon and discounted logic and there's pricing logic and there's whole backend logistics for managing warehouses, and there's, you know, reporting systems, and there's things that look for, you know, fraud and bad transactions. There's all these different systems that all have to somehow take into account that something sold and update in different ways. And so that, that process of kind of taking, you know, individual chunks of software and, you know, plugging it all together, that, that's kind of what I mean by data integration. I don't know if it's a perfect term. I, I don't just mean you know, kind of copying data from one place to the other.

Is there any way to quantify in terms of either number of updates or volumes of data, uh, how much data are we talking about that needs to move around between applications?

Yeah, you know, again, I think this, this kind of depends on the scale of the company and then how much of that company is really kind of modeled in software. And so when you look at a lot of these big tech companies and you look at their usage of Kafka, they actually have billions or hundreds of billions or even trillions of these events that are flowing through the system each day. And so that's obviously massive scale, and I think it took years. And then those companies are more or less fully modeled in software. And if you look at other companies, they're often not all in software. They have many things happening in the real world that have only disconnected, non-integrated support and software, and so they might have much less. But if you think about how those companies would probably exist 10 years from now, I do think companies are getting more and more to where there's a complete supporting software infrastructure, even for the things happening in the real world. There's instrumentation, there's knowledge of that, there's support for that. Even the things people are doing, there's associated apps and so on that help kind of track it and monitor it, and that all needs to come together into one company that actually works as an integrated whole.

Before you went to the solution we'll be talking about based on event log, you tried other approaches to integration at LinkedIn. What else was tried and why did it not work?

Yeah, I mean, the context was, and this post is actually relatively old, right? So I think I wrote this maybe at the end of 2013 or beginning of 2014, right around that. Timeframe. And, you know, at that time LinkedIn had been through this journey of kind of rebuilding all the internal infrastructure for scale, and there wasn't much that was available off the shelf. And so when we talked before, it was about a key-value store that we'd built internally. And so we had all these different data systems, and we were trying to be really smart about the use of data and how that would, you know, feed back into the applications and customer experience. And, you know, I felt like what what we were doing was, was trying to solve that kind of n-squared integration problem of how does this application trigger off this thing and how does this data flow into this analytics system and how do we monitor this? And I felt like we were trying to solve it, you know, one, one integration at a time. And so we were kind of building a big team to do that. But it was— we were never going to finish, you know, we weren't, we weren't even scratching the surface. I think at one point we were like, hey, what do we really— if we think about what happens in the business, And we try and say, what do we really have coverage for that you could tap into? I think we said maybe we have about 10% of everything, and we were working very hard at that point. So we were like, okay, that— so that was the, that was the context. And then we thought, well, you know, what would a solution look like? And it's actually not hard to imagine. You would say, well, whatever happens, you know, somehow that data, that like, you know, stream of things that are happening, it should be available and any other system should be able to tap into it. and they should be able to react or respond or calculate something or do something with it. And then the devil is of course in the details. So when you say everything, that could be actually a lot of data. And when you say any system, of course, the needs of different systems could be very different. If you have some offline data warehouse, it may read things only once a day. So it may take updates very slowly. If you have some kind of very real-time application that feeds back into the customer experience, it may have pretty low latency requirements. And so then we were like, well, you know, is this really possible? Can you, you know, can you create something that actually fulfills all those demands? And do we need to? Like, our assumption was there must be something out there in the world that solved this problem, and we were just, you know, too dumb, like we didn't know about it. And so we kind of went through this journey of trying these different off-the-shelf messaging systems, and our theory was maybe we could kind of stitch them together in some way, you know, and you see this in the database world where people will take, you know, lots of MySQL databases and try and stitch it together into some, you know, some kind of sharded something or other. And, you know, we thought, oh, maybe we could do the same thing, like we'll, we'll get some of these, uh, off-the-shelf messaging layers and we'll try and, you know, stitch it together into some platform for this. And, you know, it was, it was kind of remarkably unsuccessful. And so then we came back at it and we thought, well, look, you know, we have internal capability to build stuff from scratch. And we had some ideas that came out of the database world for what you really needed to do. And that was kind of what led us to start working on Kafka.

We'll go more in-depth into that. I wanna break this down into stepping stones. These bits of information that are getting shared, let's call them events. What is an event? Give some examples.

Yeah, it's one of those things that's so simple, it's almost hard to explain, right? So an event is just something that happens. And in that retail example, it could be a sale. So, okay, we sold something to somebody, that's an event. The representation of that in a computer, well, it could be some snippet of JSON that says what was sold and where was it sold and who was it sold to and how much did it cost. The data about what happened, but it's, you know, obviously the event isn't really tied to that JSON. You could represent it in some other format. And so if you think about it, it's a type of data. An event is just data and it's immutable, right? You know, something that happened, it doesn't unhappen and it happens at a particular point in time. And so it's just a fact actually. It's just something that occurred and you could imagine representing all kinds of different things that happen in a business that way. I talked about sales, but just as easily you could represent some change to some state about a user. So at LinkedIn, maybe somebody comes and updates their user profile. You could say, okay, my profile is now this, where this is the contents of all the fields. That would be an event, like a profile update event. And if you were making a ridesharing app, you could say, okay, you know, my car is currently here, right? That's an event. It's just this, you know, it is true that at this point in time, this car was at this point, this position, and it was this driver. And, you know, this is the ride it was part of and so on. So, so that's all it is. It's a very different way of thinking about data. You know, typically we've thought about databases which are full of, you know, the current state of the world, right? And so we think about state, we think about, okay, You know, your database tables, it might have the set of products or the, you know, set of customers or whatever, and they, you know, could get updated. But it's really modeling the current state of the world. And events are about, you know, what's happening in the world, what occurred. And so that could be something that's very high level about the business that occurred. It could be something very low level. At LinkedIn, at some point, you know, they were monitoring even very low level things like traffic through network switches and, you know, so on as events. And you'll, you'll see this actually if you kind of search around in software engineering channels for events, you'll find it actually springs up all over the place. So in the world of application monitoring and observability, there's a whole movement towards events. It's very much about low-level mechanics of watching software do its job. If you look at front-end development, there's a whole movement towards events, which is about capturing what's happening and having UIs that react and respond. And then of course, in the world of microservices and application interaction between applications. There's a movement towards events. And so, you know, it's actually kind of, you know, coming out of the woodwork in all these different areas and they're all talking about the same thing. Events are just things that happen in the world. And, you know, it's a very different way of thinking about data, but ultimately kind of a very simple concept. Once you understand that, you're like, oh, you know, what's the big deal?

They may all be conceptually the same, but if you have events which involve something like a customer bought something, in the old days you'd store that in your relational database. Then other data like monitoring, your security system detected an anomaly, maybe these things are transient, low value, you don't want to go to so much expense to capture them, maybe you don't care if you lose them. So are all these events all treated the same way as if they were all equally valuable, or is there some ability ability to tier them based on how much cost you're willing to incur in relation to the business value of that class of things.

Yeah, I mean, I think the first mental jump is just understanding that there's such a thing as events, and then we can think about, hey, how should we build the right infrastructure for it? You know, I think you're 100% right that it's not like this concept didn't exist before. I think it just wasn't well supported. And so you would see all these cases where Yeah, people are trying to use a relational database for streams of events. And maybe the most prominent example of that is for people who know the world of data warehousing, they have such a thing as fact tables and they have dimension tables. And dimension tables are kind of what you'd expect. It's your set of products and the set of geographies and so on that you operate in. But the fact tables are actually the set of things that happened and it's always ordered by time and it's immutable. And it's very much like an event. You can't subscribe to it, you can't get it in real time. It just gets dumped in at the end of the day. But it's kind of an interesting thing that in order to analyze a business and try and say, okay, what happened in the business yesterday, you basically have to reinvent this event concept in a relational database. And so, yeah, it's nothing new. I think the new thing we added was, Concretely, what would the data structure for events be? If you said, hey, this is an important thing, how should we represent it in infrastructure? And is it possible to do that in a way that actually makes it easy to build around? Because anybody who's tried to build real-time event streams where you trigger off of it and process it in real time on top of a relational database has come to the same conclusion that this is really quite hard. You're abusing a piece of infrastructure that was built for one use case for something else, you know, that's, that's a common thing to do. But, but it just gets harder and harder and harder to do well. And so, so that was kind of our idea was, yeah, you know, you can't solve all problems with one piece of infrastructure. But if you take this idea of events seriously, how far can it go? You know, what— how can you make something that's really built, you know, to do that from, from the ground up? And if so, hopefully it can go from, very low latency to kind of longer-term, slower storage and processing and be a basis for all these applications and so on.

We are gonna get to how that infrastructure works, but one more concept I wanna cover. If we're going to integrate many different events through data and enterprise, do you need all of the apps to agree on a common schema for the events?

Yeah, you do. In a simple way, so let's say that I'm trying to keep track of all the sales occurring, and there could be many ways that we sell things, like maybe we sell them online and maybe we sell them in brick-and-mortar stores, right? So all of those people, it will be much more convenient if they can all agree, this is how we represent a sale, and then we can send it all to the same event stream. And then likewise for the people who are trying to react to that or respond or tap into that or take it and store it somewhere, if they haven't agreed with the people upstream about what the bytes mean, they're not going to be able to make any sense of it. So they have to agree on what the representation of a sale is. And then we have to agree, like, how can it change over time? Maybe we'd all love it to be the case that we just say, oh, this is what a sale is. It can never change. That's the definition. But the reality is we work in businesses that are evolving all the time. And so the attributes of a sale, what can we capture about a product? How can we sell it? The different things that we would record are going to change as the uses change. And so we have to agree on that, how that can evolve so that things can, you know, software upstream can be updated and software downstream can be updated. But also we don't require the whole organization to stop and get on to the new thing with, you know, some downtime. So there has to be some agreement around that. And so in a sense, it's actually, I think, more important than even the schemas in relational databases because typically a relational database is a implementation detail of one application. So the application just kind of needs to agree on the schema with itself. And even there, the schema can be helpful to make sure that you're checking constraints on your data and so on. But once you get to multiple applications all sharing, then it becomes that much more important that we agree on the structure of data. Otherwise, you know, you could have somebody broadcasting, you know, malformed things out to the rest of the world that they can't react to or handle. And then, you know, chaos. Chaos ensues.

What are some of the schema definition formats or languages that can be used?

Yeah, at least for Kafka, we've tried not to reinvent the wheel there. You know, there's a bunch of these serialization layers that are common also out in the world of, you know, building RPC services or RESTful services. So, you know, there's things like Protocol Buffers and Avro and JSON, and they all have different proponents of different kinds. You know, for Kafka and for Confluent, we've tried to make our stuff work with as many of these as we can. We provide out-of-the-box support for something called Avro, which is kind of a binary format with a JSON schema that has good support for the evolution of data. But we're not religious about it. People tend to be the most religious about these, you know, kind of formats and standards. And our job is just to build tools that people work with.

All of these formats have in common that they are supported by many different languages. And that is probably essential because every app in a big company is not going to be written in the same language.

Yeah, that's a really important part of this, is you want to have some definition that's agreed on that anybody can plug into. And so the kind of things that are baked into the programming languages like Java serialization or whatever the old school pickling in Python, I mean, it's really inappropriate. It would work if you were just sending data to yourself with the same version of the software doing the reading and writing, but this is something where you're gonna have many different applications potentially in the future, even in the present in different languages, and those applications will all change independently. So you can't assume a single version of everything, you can't assume a single programming language, you wanna have as much independence in that data as you can.

We're moving in these stepping stones toward this concept of the Kafka event log, In your blog post, you introduce the idea of an event log which is borrowed from the database world and you're extending it to the enterprise. Let's start with how logs work in single-node databases.

Yeah, that's, that's a great question. So, you know, our thought process was we were trying to solve this problem of when, you know, when something happens out in the world, how does it get integrated into all these different, you know, systems and applications? So if In my retail example, something is sold and all these downstream systems need to kind of react on their timeframe. And we were looking for where are examples where this happens. And the people who were thinking about this were a set of database engineers. And so one of the ideas that is kind of under the covers, it's basically an implementation detail of databases, but it often leaks out, is this concept of a commit log. And that's used to keep all the different parts of the database in sync. And in a distributed database, it's often used to keep these different independent nodes in sync. And so the idea is you just take every update that you want to make, and that update, you know, maybe you're saying, hey, I want to update my LinkedIn profile to this new value. That, that might require changing multiple things, right? So that row has to change, but different database indexes may need to update. There may be things that need to, you know, trigger off of that if the database supports triggers. All kinds of things may happen in reaction to that. And so the idea is It's very simple. These commit logs are really just a file where you just append the change, and then taking that change and integrating it into all your data structures happens only once it's been committed. And so the idea in databases, if you want to have some notion of correctness and consistency and fault tolerances, in some sense, that log of committed data, that is the database. That's the real truth of what happened. And all the tables and indexes are just representations that you could actually recreate off of that log if you had to, right? They're kind of like a cache of the information that's been structured for a particular type of query. And so, so that's the kind of basic idea of, you know, how a commit log is used. It often becomes very important as an external factor because people often tap into that commit log. So in MySQL, people use the kind of binlog to get the sequence of changes out of the database as well. and so it becomes almost a, you know, a public API, but it's kind of an accidental public API, I think, in most databases where it's an internal detail that ends up getting exposed. And the things that we knew from database engineering were these commit logs are the fastest part of the database. So you can write, you know, a series of things to disk in a, in a linear order extremely fast. And we also knew from distributed systems that there was a way to do this at scale across many nodes, that there was consensus protocols and ways of committing data that would extend beyond a single machine. And so we thought that could actually be applied to the enterprise at large. You could actually think about your kind of whole company instead of data systems and applications as kind of like one big data store, and you would commit to some log and then it would be applied elsewhere.

Formulated this question in terms of a single-node database, and now you've talked about how it can be generalized to a distributed system by which you might say multi-node Cassandra cluster, where every node is running Cassandra. Now, it sounds like the next step was you generalize that a little bit further to the enterprise where you have these heterogeneous systems, but you're still thinking about the log in the same way. As if the entire enterprise was one big database. Did I get that right?

Yeah, I think that's exactly right. So in a traditional single node database, the log is usually just, you know, on the file system and then it would be used to keep replicas of that database in sync. So they would just take each change that was applied on the, you know, primary or master node and they would apply it on the kind of secondary follower. Nodes, and then other applications might sync off that. So, you know, that was the kind of typical architecture of a single node database. And then in distributed databases, people came up with, you know, techniques for maintaining this type, this type of distributed log across multiple machines. And this is figured pretty prominently in a lot of the architectures of distributed databases. So things like Amazon's Aurora, Yahoo early on had a paper on a system called Peanuts that worked this way. I mean, that was the name in the paper anyway. And so it clearly generalizes to a distributed system where the, you know, the individual nodes in that distributed database all work asynchronously. You know, they don't all work in lockstep. And so it wasn't a huge jump for us to think about like, well, what if those nodes are doing different things with the event? You know, that would work as well.

Two very important properties of these logs, which you've mentioned, and I'd like you to expand on why are these properties important. They are immutable and append-only.

You know, there's even a third property, which is, you know, kind of maybe obvious just from my description, which is they're ordered. And I think you actually need all those things. So you need it to be the case that when something happens, you know, new things can happen, but that past history doesn't change in sneaky ways out from under you. And that's, that's really important in a database. The database needs to be able to take each of these changes, integrate it into its tables. If, you know, there's random changes happening in the past of history where you're kind of rewriting history, it's very hard to integrate that in a correct way, especially if you imagine the database crashing at some instant in time. Like, what, what state are we really in? You know, are— have we even made a full atomic change? And so This idea originally came out of the idea of how do we get atomicity and how do we have transactional commits and how do we handle correctness in a database? But it's actually arguably even more important when you get into a big diverse organization where you have different parts of the organization have different teams that talk to each other less than they should and are building independently. It needs to be the case that when something occurs, that there's a really principled way that that propagates out to all the other systems in the organization.

We're at this point, we started talking about the problem of integration, the complexity of point-to-point integrations. You simplify that, you put the log at the center of the architecture. I think we have all the pieces now where you can answer the question, what does the enterprise architecture that you're talking about look like?

Yeah, well, the, the idea is that it's based around events and that you can think of, you know, events representing both the, the kind of activity of the company, but also the state changes. So if I update my profile, that's an event. And so you could really think about kind of all the different types of data in an enterprise as events. And then the question is, well, what's the, what's the data structure for an event? What's the abstraction that, you know, you can give to a software engineer that they can work off of that? And we, we felt this kind of commit log was really the right way to do that. And so the idea is that you, you know, you commit to some kind of log and you say, okay, this occurred and this occurred and this occurred, and you can have many people writing to that log. And then you can have all, all these independent readers who, who read off that log and apply the change, and they can do that kind of on their own timetable. So slower systems can do it slowly, real-time systems can do it, you know, as quickly as they can, and they'll all get the same sequence of things, they'll all get them in the same order, and they can get them kind of on the timeframe that they want. But there has to be strong guarantees around this, right? If data can somehow be lost, or oh, if a system goes down for a small period of time, it can't be that data, you know, you're missing updates and everything gets out of sync and you have to go do some scavenger hunt to figure out what happened to your data. It has to be guaranteed that when something happens, it will propagate out to everything that's downstream. We felt like that was actually really clarifying if you stepped back and you just thought this way, you think, okay, look, the things that happen in a company are events, and we need infrastructure that allows us to propagate events at a global company scale. If we think, hey, the way you would build that is this commit log, then it really is just this distributed systems problem of how can we build that, how can we make it accessible? And then how, you know, what are the programming abstractions to make that really easy, which is a lot of what we've done, you know, since that blog post was written was really, and at Confluent, is really focus on the kind of up-the-stack tools that allow, you know, people to build against this in a really simple way.

We did an entire show on Kafka. It was 215. It was about 4 years ago. Listeners who want to get some introduction to it could listen to that show. So I don't want to to do a whole basic intro to Kafka here, but how has the Kafka platform changed in the last 4 to 5 years since we last covered it?

Yeah, it's, it's changed in a couple ways. So that the evolution of this was, you know, we had a pretty big vision around, you know, events and commit logs and how it could really be kind of the central nervous system for a company, but we couldn't build it all at once. And so the The first version that we built was very scalable, but it didn't provide, you know, kind of replicated fault tolerance guarantees like a database would, right? And so it was applicable for these kind of high-volume activity data like messaging systems might have, but it wasn't really applicable for kind of, you know, core payments transactions, can't lose data. And then, you know, then we did work to add replication to that. So then we had something really right about, right, right, right around the time that that blog post was written that was both fault tolerant and scalable, but it was still relatively low level. We were giving you this kind of commit log you could go do with it as you please. So it was kind of a, you know, a very primitive abstraction, powerful, but it was up to you to figure out how to use it. And of course, that's a lot of work to take advantage of. So a lot of Silicon Valley companies that just had very large engineering teams would, build around Kafka, just build it into all their different applications and data systems, and they would get to really large scale with it. But kind of outside of that world, it was very hard for people to kind of take advantage of the idea. And so then at Confluent, the kind of big investments we've made is kind of continuing to make that foundation better and better, but also adding a layer that would manage kind of pre-built off-the-shelf connectors. So you could just plug in databases. If you have MySQL, there's no reason you should have to build the, you know, 10,001st MySQL integration. You should just be able to get the connector and use it and then build stream processing capabilities that would allow you to build applications that react to these event streams. And so we found that people were just kind of building the same stuff over and over again as they were trying to, you know, work against this. And we realized that in some sense, If you think about this kind of commit log or event stream, that's a foundational abstraction, but it's pretty low level. So it's kind of like building directly on top of that can be a little bit like building an application directly on top of the file system in terms of its data access. And a lot of the work in recent years has been moving up the stack and trying to make that more approachable and try and give a much more like a kind of almost like a relational database way of working with event streams. And that's been this area of stream processing and things like KSQL that allow you to just apply SQL on top of data streams and materialize new data structures and data formats and things that you can act upon.

We'll put some links in the show notes where people can read about some of these features.

Join hundreds of senior developers, engineers, and software architects in New York this February to learn new skills and technologies technologies at the O'Reilly Software Architecture Conference. Network with and learn from experts as they share their knowledge on microservices, cloud computing, and serverless, as well as emerging trends in AI, machine learning, and data analytics. Whether you're a seasoned architect or aspiring to become one, the O'Reilly Software Architecture Conference is designed to help you go next level. Save 20% on your pass with code SER20. Register at o'reillysacon.com. Oneotherarea.com/seradio.

One other area I want to explore is if you're now putting this— if you're making the event log the real core of your architecture where it's becoming the system of record, in order to adopt it, you need to ensure that it has the same degree of disaster recovery and survivability as the enterprise database used to be, which some of it means features built into the system, like the ability to create backups while it's running. But then other investments you need to make outside of the event log, which is what you do with the backups. Where do you put them? How often? Do you have good operational procedures around recovering them? Has Kafka evolved a feature set that makes it worthy of being a system of record?

Yeah, it's gotten better and better in that area. I think what you say is exactly right. There's a set of features and then also a set of operational practices that are really important for anything that's kind of a critical data store. And so it's important for people who are adopting Kafka to know, are they using it just for transient data or are they using it as something you can kind of fall back on or rely upon? and if the latter, then definitely the degree of operational excellence has to be a lot higher. And so it has gotten better and better at storing data, better at scaling the number of partitions. Backing up something like a log is actually really easy, because you can just subscribe to it and copy it elsewhere. So in fact, the backup features in databases are always built on top of the commit log in the database, or typically are. So that part's relatively straightforward to do, but that is a critical thing for people who are trying to rely on this commit log as something that you can actually restore off of without going back to source systems and trying to reconstruct it.

There's some interesting possibilities with this architecture I would like to get into. One would be if we have all our data in, let's say, a relational database and we realize there's this new type of database called a graph database, it's a really good fit to our use case. How do we incorporate these other types of databases into the enterprise.

Yeah, that's, that's a great point that you raise. So one of the nice things about Kafka is it's kind of independent of any of the different representations or query capabilities you might have for data. And so if you have data feeds that are in Kafka, you can easily load them into different databases and get kind of almost like a materialized view of that data. And so you can think of, you know, data warehousing and analytics layers that way. You could think of a search, search index that way. You could think of a graph data store that way. And this is one of the core use cases people adopt something like Kafka for. They typically have some kind of relational database which is serving, you know, live user experience type UI, and that's actually creating the kind of core transactional data stream. So maybe you're a set of users, but then there's other systems like search or analytics that need a representation of that data. And so Kafka kind of acts as this log of those changes that can feed out to all the systems. So oftentimes a data warehouse would feed slower, some kind of real-time analytics layer might be faster, a search index. Usually you want to be in sync, you know, if it's, if it's meant to be a live kind of search experience, something like a graph database that's providing different types of queries could feed off of that as well. And, you know, that's, that's definitely a core motivation for this. Architectures actually as the set of databases, but also even, you know, in the cloud, a lot of the different SaaS services and so on that you could kind of tap into data. As that set of things has grown, you know, the desire to have some kind of consistent way of broadcasting that out gets, you know, more and more. And we're seeing more of that where it's, you know, it's not just new databases. Like people have integrations with things like Salesforce and, you know, with REST services and with all kinds of other things that they want to react when something happens.

In the article, you express the idea that all of these different types of databases are just specialized indexes on the event log. What do you mean by that?

Yeah, what I, what I mean is, you know, in some sense, if you peer inside of a lot of the distributed databases, they are kind of a, you know, distributed log, and then, you know, some kind of materialization of the data that's there for serving queries. And by having this log as a service in its own right, you can run lots of those systems in the way we, we just described, and you can, you could actually compute new things off the data stream. And so, you know, it may be the thing that you want in your search index isn't just the simple record that you have in the database. It may actually involve stitching together lots of bits of information that you have about the customer and then indexing that. And so you may want to do processing on, on top of that data stream. And so, You know, that, that's very much the idea. And then you can think of that, you know, that search cluster is, you know, it's effectively just a kind of materialized view of your data. You know, it's just a representation that's feeding off of that commit log. In database terms, you could think of it as a, you know, in databases they have tables and they have, you know, materialized views that are calculated off the tables. You could think of it very much like that. It's just that this happens to be in a separate system.

That last point, I want to drill a little bit down into this. Up to now, we've been talking about a lot of examples of An event is some new information that came into the system from the outside world, like somebody, customer, bought something. You could have stock price feeds coming in from the market. A vendor updated their inventory. Use one of your systems to get that information into your enterprise. But now you're talking about you could have events that subscribe to your internal event stream, process the events, and let's say classify this customer as now a higher tier of customer or something, information you've been able to extract from data that came in by processing, and then do you publish that back out onto the event log and now other systems can react to that?

Yeah, that's exactly right. So like a really simple example is in, you know, virtually any big company, you have a bunch of bits of information about your customers. Right, so you have maybe some primary account record, but you have interactions with customer service and you have settings and so on and you have payment information. And so usually one project that's really common in companies is creating some kind of unified view of that that's used in a lot of different applications, that's maybe used for analytics. And so you can think of it as really just you need to perform a kind of continuous join across these different systems which have different databases and data stores. And so how can you do that? How can you do a kind of continuous join across a bunch of different systems that aren't even using the same underlying data storage? And I think one answer to that is, yeah, you treat the updates in each of those systems as a kind of stream of events. You use stream processing to join that together and create this stream of kind kind of unified customer profiles, and then that could be materialized in a key-value store, it could be sent off to a data warehouse for analytics, but that feed is basically published back, as you say, as a log in its own right, which is consumable and subscribable.

I wanna get into a little bit more of the programming aspects. So application needs to get its data into the event log. How does it do that?

Yeah, there's two ways. In Kafka, right, you can directly write a record and it's actually not hard. You just say something like send and you provide a kind of key-value pair and that's sent off to some topic that you would specify and it's appended to the commit log for that topic. And that's kind of like the lowest level of interaction is, you know, it's almost like just saying write in a file system. You just write something. And so that's kind of a push mechanism. Now, if you think about existing systems, you may have some, you know, SaaS service you use, you may have a database, those systems don't know about Kafka, and so they're not actively gonna push data into Kafka. You may actually wanna pull data out of those systems and push it into Kafka, and that's where these connectors come in, and Kafka supports something called Kafka Connect, which is this distributed framework for running these connectors and managing them, and so that's the other, That's the other way to get data in. And then, you know, of course the final version of that is kind of what you described, which is you may be taking some of these streams and actually doing something with it and publishing back some derived stream of data back into Kafka using simple application software, using one of these stream processing tools like KSQL. And, you know, maybe that's a third way. All of these are built on the underlying producer and consumer layers in Kafka, which do the reads and writes.

If I was writing an app today, I know my enterprise has Kafka, so there's not this question, how do I retrofit this app that's writing into Postgres? So, and I wanna make life easy for the programmer, every time I do a transaction, so this app has its own database as well, let's say that's also Postgres, Postgres, every time I write a transaction to Postgres, do I have to now include some logic to write to Kafka, or is there a way, kind of an aspect-oriented style, where I could create a cross-cutting concern against every database transaction that will, say, just ship them all off to Kafka?

Yeah, usually the simple— I mean, there's many ways to handle this, and some of the things that, some of the events that come out of applications may have no representation in any database and you may just write them directly. But, but often the simplest way to do this is to use the connector infrastructure. So most databases like Postgres, they actually expose the commit log in some way. And so either by, you know, querying the tables or reading that commit log, you can actually take that and publish it out into Kafka's, you know, data streams for each of the tables, say. And so that's the pattern that's probably most common. And then there may be additional cleanup you want to do on that data, and that can often happen, you know, via stream processing where you take the kind of original thing that was in the database and you stitch it together into something that is really what you want to publish out to the organization that's in the right format for consumption.

Done all my domain modeling. I understand my business abstractions. Then we get the database people involved and they normalize everything into all these tiny little tables. So I commit this one object, it writes rows into 6 different tables. Now you're gonna look at the commit log with your connector. Do you have to reverse engineer that back, or maybe not reverse engineer that, but do almost the inverse operation to transform that back into your event log model?

Yeah, I mean, I guess there's different patterns for doing this, and it depends somewhat what your goals are. If you need to have read-after-write consistency in the database, you do typically wanna kind of commit to that database first and then have the Kafka log sync off of that. And that could mean either having a table that directly represents the event as well as the more normalized representation in the database, or it could mean stitching it back together off of the normalized representation. And that probably depends whether you're writing the application from scratch or you're retrofitting something that already exists, which of those you would do. If on the other hand, the database doesn't have to, you know, have that kind of read-after-write consistency, you can always just, you know, publish the event stream to Kafka and populate the database off of that.

In the first pattern you talked about, then, if I understand it, it's more like a double write, but the writes all go to the database to take advantage of the ACID property of Postgres where you'd have one table and let's say it's called Kafka log and you write something there then Kafka could come and pull that out, but you don't have the problem of updating two different logs and trying to ensure they both get committed.

Yeah, it depends a bit what you mean by double write. I mean, we don't usually advocate that people kind of, you know, write to one and then write to the other because, you know, obviously if your application fails in the middle, then, you know, the question is how do you keep those in sync? So it's usually better to write to the database and have Kafka subscribe to the database, or write to Kafka and have the database subscribe subscribe to that. If you don't care about correctness, then of course you can try and do that double write, but it's a bit like those, you know, keeping caches in sync that way where it never quite works in the presence of failure and multiple machines and so on. So usually picking either the database or Kafka and having, you know, one of those two be in front is a cleaner pattern.

You put a table in Postgres called Kafka events and you write to all your normalized tables, and then you write the event out to that table, and that's all one commit in Postgres. And then Kafka can just look at that one table.

That's right, and that would, that would simplify the downstream cleanup you would need to do. And, you know, that obviously works only if you're able to go in and change the application. If it's something owned by somebody else and you just want to subscribe to what's in the database, then you got to stitch it back together in the stream processing side, which has gotten easier and easier as the stream processing toolset has gotten better.

So one more main area I'd like to cover. We're getting close to the end of our time. With these event logs being immutable, how do you deal with if bad events get written to the event log, which could either be human error or you have some rogue application that has a bug in its writing a bunch of bad events and now they're immutable, are you stuck?

Yeah, that's, that's a great question. So, you know, it is very hard if you have systems all reading off this in real time to handle kind of bad data. You know, it's very similar to a, you know, the situation with the database, except you may have many apps that all feed off the same thing. And so as much as possible, you know, it is good to actually enforce these schemas similar to a database, right? You know, if you ship a version of your application that publishes corrupt data and overwrites good data with it, you're in a bit of a bind. You're gonna have to go kind of rewind and manually truncate and figure out how to get back into a clean state. It's similar with Kafka where, yeah, you can kind of rewind to when it happened, but you're actually better off trying to enforce the schema correctly and making sure that valid data is what is published. And that kind of schema enforcement ends up being really important as this scales in an organization. I mean, we've worked with a bunch of companies that have done this, and the ones that have taken kind of a, best effort approach. It tends to work up to a certain point, and then, you know, there's just somebody breaking everything downstream all the time because there's so many readers and so many writers that at any given point somebody has a, has a bug somewhere.

What about the schema is correct but the values are not correct? It has to be somehow cleaned up ex post. How do you fix it?

It's similar to any contract you might have, right? So you have the same issue if you're trying to, you know, update individual microservices in a microservice architecture. If you have no contract at all, that can be very hard. Even if you have a contract, the contract can't really cover everything, right? So you may say, oh, there's some field which is URL, fully normalized, fully qualified URL, or is it just the base of the URL, right? Like ultimately the person upstream and the person downstream has to agree on that and kind of keep that in sync, if you get something that's wrong, you do have to either update the processing logic to handle that or go back and clean out the log, which is going to be a very kind of manual step that's hard to do.

That covers the prepared material I have. Is there anything else on this theme or from the article we're discussing that you wanted to cover that we haven't talked about?

Yeah, I think maybe one of the interesting things that we've done recently that's very closely aligned to that is around ksqlDB, which really kind of ties together the ideas. You know, we talked about some of the integration stuff, but there was really kind of 3 points, like integration, stream processing, and the role in data systems. And in some sense, that ties together all 3. So I think it might be fun to talk about that a little bit.

Yeah, let's go ahead. Why don't you describe what it does, what problem it solves?

Yeah, so really just last week, we actually, did a release of ksqlDB, and this is kind of the evolution of a system called KSQL that we've built to allow stream processing in SQL on top of Kafka. And so the idea was, you know, these event streams are new, and, you know, people may not know how to use them, but everybody knows SQL. And so if you could just use SQL on top of the event streams, if you could say, hey, you know, take the stream of updates to user profiles and join that onto the stream of activity from users and get, both together and be able to, you know, count or act on that or transform it all in SQL, that would be really cool. And we've evolved that system and it's really actually, I think, quite exciting. And if you've, if you've read this Log article, it's even more exciting because it brings together the ideas about stream processing, you know, acting on real-time streams of events. It brings together the ideas on data systems that are built around logs because it turns out that, You know, if we want to join our customer profiles to the customer activity, you need some kind of stored fault-tolerant representation of customer profiles. And so what this system now allows you to do that's pretty cool is you can take these event streams, you can materialize some dataset off of that, you can use that dataset to combine with other event streams and you can query it. And so it's, it's, it's very much kind of a database built for working with event streams. You know, it's not meant to replace Postgres or Oracle or whatever you would, you know, kind of have as the, you know, the backend for a CRUD application. But it is meant to solve this problem of, hey, I want to take a bunch of these different data streams, I want to materialize it into some dataset and build an application around that. It aims to make that as simple as possible. And so it includes both, you know, SQL for processing streams of data continuously as well as the ability to do lookups off of whatever the result is.

I wonder if that solves the problem we've been talking about where I have an application that's based on Postgres. I'm gonna commit everything to Postgres and then we have the problem of pulling it back out on the backend. So say I want it to be in the log. Why don't I just write it to the log in the first place? The only problem then is you don't have the ability to read it from a SQL-like interface until a little bit of time has elapsed where it propagates down and gets picked up and added to SQL as a subscribe step. So can I then put it in the stream, and if I need to read it, I can query it from SQL out of the stream now and avoid that consistency delay?

Yeah, I mean, the use case that we're targeting initially is not quite that, right? So our theory was people aren't going to replace Postgres for their CRUD applications or REST services. And we're not trying to get them to do that, but if you think about the applications downstream of that, of actually using event streams, those are actually relatively difficult to build and they end up being kind of, you're gluing together some of these connectors and you're gluing together some stream processing that does something with the events and then you're, eventually gluing on some kind of data store to serve results up to some application. We thought if we could consolidate the set of things downstream, that would really make life much simpler in working with event streams. And so, a very simple example of that that I described was that kind of customer 360 where you're joining together data feeds about your customer across 7 systems and then you wanna be able to look up, hey, what's the current, what's the current unified profile for this, for customer 123. And so you could do that basically just in a couple of queries. And I think that kind of materialized derived cache, I think that's a problem we see everywhere. Like people have these older applications, you know, they're running the business, you have new requirements and you need to build a new dataset that cuts across some of these, the ability to tap into those and, you know, create a new dataset off of it and then issue queries against it. To kind of continually materialize a cached view of some part of the business, that's actually really, really common and we think really well-suited to this. And it's much easier to do it in a correct way than if you're trying to, you know, do some dual write strategy that cuts across multiple applications in some, you know, transient cache because this is directly fed off of the changelog of all of those systems. So those event streams populated, it's always kept in sync.

I see what I think you're saying now is instead of writing an application to do that, which is high programming complexity, you've reduced it to I can write a SQL query against Kafka and get the answer.

Yeah, yeah, so the support that this provides is the ability to manage these connectors. So in my example, the first thing I need to do is actually capture the stream from a number of different, uh, databases. And so maybe I would, you know, I, I now have SQL support to kind of declare those almost like external tables or external streams and, and capture that into Kafka. The next thing I need to do is maybe join and clean up those data feeds and create my unified customer profile. I can do that also with this kind of stream processing SQL. And then the final thing I need to do is be able to actually query and say, oh, I, I need the profile for this customer because I need to put it in my UI. And so I could actually issue those real-time queries against that, that data feed. And so, so bringing those together, the ability to, you know, if you think about it in ETL terms, you're kind of extracting streams of data, you're transforming them, you're kind of loading them into some, you know, stored representation, and then you're actually doing live queries against them. You know, it's really that whole end-to-end stack that would have otherwise required you to glue together specialized systems for each part.

And once you have all the data in Kafka, that becomes the obvious place to do at least some of this processing.

Yeah, that's exactly right. So, you know, we felt like the investments with the connectors helped unlock and get a lot of data into Kafka. And then, you know, the complaint people have is, great, I have all these ideas for what I could do off of these streams of events. But it's, you know, in the end, it's pretty hard, right? You have to build, you know, custom Java code or Python code or whatever that does any of these applications, it gets complicated as you think about fault tolerance and so on. And so having data layer that allows you to do that declaratively in SQL that handles the kind of more complex operations like joining or counting or adding, aggregating, that's actually really valuable.

That wraps things up, Jay. Where can people find you on the internet?

Probably the easiest way is on Twitter. I'm @jaykreps, first name, last name all together. And, you know, other than that, I, you know, I've got LinkedIn profiles and dozen other social media things, so usually pretty easy to find.

And Confluent, where can people find that?

Confluent is confluent.io. And so Confluent produces, you know, a whole platform around Kafka that is, you know, community license. You can just go take it and use it. We produce also an enterprise software product that you can go, if you're gonna run this in production, we produce a cloud service so you can actually If you're looking at this Kafka stuff and you're like, "Oh, I just don't wanna run one more thing," you can get it as a service in any of the major public clouds, Google or AWS or Azure, and you kinda just pay for what you use and you get all this infrastructure as a service.

Jay Kreps, thank you very much for speaking to Software Engineering Radio.

Thanks so much for having me.

For Software Engineering Radio, this has been Robert Blumen. Thank you for listening.

Thanks for listening to SE Radio, an educational program brought to you by IEEE Software Magazine. For more about the podcast, including other episodes, visit our website at se-radio.net. To provide feedback, you can comment on each episode on the website or reach us on LinkedIn, Facebook, Twitter, or through our Slack channel at se-radio.slack.com. You can also email us at team@se-radio.net. Se-radio.net. This and all other episodes of SE Radio is licensed under Creative Commons License 2.5. Thanks for listening.