Announcing ksqlDB ft. Jay Kreps

Regular listeners to Streaming Audio have heard us talk plenty in the past about KSQL. Today, we're announcing that KSQL is taking a new direction as KSQLDB. Jay Kreps, Confluent CEO and co-creator of Apache Kafka, joins us to explain what this means. Hear about it right now on Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am, as usual, your host Tim Berglund, and I'm joined in the studio today by one of the co-creators of Apache Kafka, one of the co-founders of Confluent, the CEO of Confluent, Jay Kreps. Jay, welcome to the show.

Thanks so much for having me, Tim.

You bet. I wanna talk to you about a new product announcement of ours, and that is Kafka Streams. KSQLDB. I would like you to tell us what KSQLDB is. It's a simple question.

Yeah, so let me start at the beginning, I guess. I won't assume everybody knows what KSQL is, but for those who do, it's a SQL layer that allows you to do stream processing on top of Kafka. And so one of the realizations we had with Kafka was once people have all these streams of data, they wanna do stuff with it. The easier we can make that, the better. We really started working on KSQL to give a lightweight SQL layer where you could dynamically process streams and do that right on top of Kafka. It merges the idea of streams of data, like streams of events, with tables of data, like materialized tables, and allows you to go back and forth. So you could capture changes to a table of data coming from an external database and you could process those changes using KSQL and you could materialize new datasets and you could join those onto other event streams. So a kind of simple example would be maybe you have information about your customers, like the kind of profile information about each customer, and then maybe you have a stream of events of what your customers are doing, what they're buying. And you could join that together to make some kind of offer of what you want to send to the customer as a potential discount or upgrade or offer. That's a really simple example of event-driven processing using KSQL. That's what KSQL is. KSQLDB is, we basically are adding a set of features in KSQL that we felt were significant enough that we wanted to change the name to KSQLDB, and those two features were the integration of Connect and the connector ecosystem, and the ability to do pull queries, the ability to do lookups.

So let's take those two things one at a time. Tell me about Connect integration. Now, I'm gonna, we'll put up a link in the show notes to a previous episode on Kafka Connect, so if you want to deep dive on that, if you don't know what it is, we've got podcasts that you can listen to that can tell you that. And I'll just frame, Jay, my question with, I will confess that standing up a Connect cluster is a little bit more work than not standing up a Connect cluster. So I think that's really what this gets at. But tell me about the Connect integration in KSQLDB.

Yeah, it's actually really simple. It just gives you a lightweight SQL syntax that allows you to, you know, run connectors directly from KSQL. And so there's still Kafka Connect connectors, so that whole ecosystem of hundreds of connectors that's out there, you can now use that with KSQLDB. And so the advantage is just that you don't need one more thing. We didn't want you to have to set up Connect and have to configure some cluster of Connect workers that was gonna run your connector, and then set up KSQL, and then run your processing there. We wanted to make it a little bit lighter weight, By having them there, you can just say, "Hey, I want to connect to this database and capture the set of changes about the customer, and I want to capture that stream and be able to process it." That's the first query, and the second query could be the thing that does the actual processing, and you can do it right there in the same tool. That's very simple, but we think really powerful as a way of opening up direct access to all this data that's stored in existing systems. And making that available as streams for KSQL to munge on and process and react to.

That's huge. And Connect, like I was saying, Connect's operational story isn't a hugely burdensome one. I mean, it's pretty easy to stand up a cluster and you install connectors through a reasonable process and post a little bit of JSON to an endpoint and everything. But those are all moving pieces. You know, if I have to administer 2 clusters instead of just 1, that is twice as bad. So that integration is a pretty cool thing. It's just, I've seen the syntax and it is this SQL-like command with a bunch of metadata connector configuration you throw in there and it just kind of happens. And now you've got data from outside being consumed and being pulled in from outside and produced into Kafka topics. Or consumed from Kafka topics and written to the outside, so it's pretty great. But you also mentioned pull queries. Now KSQL has always had tables since it has been a thing, so what do pull queries give us that we didn't have in KSQL?

Yeah, if you're new to this world, it can be a little bit confusing. You know, what a lot of people assume KSQL does is allow you to do, you know, lookups off of your Kafka cluster. So you could say, oh, you know, what's the value for user 123? Go find that record. Which actually isn't what it does, right? KSQL does stream processing. So it takes as input Kafka streams, and it processes them, and it produces as output other streams. And one of the big use cases for stream processing is actually to materialize or create continuously some new dataset. So a really simple example of this is, in almost any large company, you end up having lots of bits of information about your customer. One of the projects almost every company needs to do is put that together into some unified customer profile, some kind of 360-degree view of the customer of what's going on in the support organization and what's going on in different parts of the company, all coming together about your customer. And, um, you know, so it's basically a big join. And so, so KSQL would do that. And with KSQLDB, of course, now you can directly control these connectors and actually, um, you know, capture these streams and, and do this big kind of complicated join with whatever logic to, to stitch these records together and come up with that unified profile. What the, the, the pull queries do is now actually let you look up the value for a particular customer. So whereas before you would just be able to take that stream and maybe send it off to an external store if you wanted it to be queryable in kind of a random access way, what this does is actually allow you to do it directly off the dataset that's stored in ksqlDB. And so many people didn't realize this, but of course a stream processing layer like KSQL has had it has to keep a fault-tolerant dataset to be able to do its calculations, and so opening that up for queries, that's something that the underlying Kafka Streams library has had, but it's relatively low-level to be able to do it, 'cause you have to find the right instance to query against, and so KSQL now makes this much simpler by just allowing you to access the calculated data in SQL, and so you can both, do continuous queries that are kind of pushing out results as things happen, as well as this kind of point-in-time lookup query, which, which we're calling a pull query that just kind of gets the result right then. And so, so then if you put all these things together, it's actually really powerful, right? You can, you can now connect to these external systems, right? So maybe I have information in a bunch of different databases about my customer. I can do this kind of continuous stream processing query to actually join it together and materialize a table of all the information about the customer in one place. And then I could do lookups against that. I can say, okay, what's the unified profile for this customer? And I can do all that in one system.

So we have pull and push queries. And if I understand you correctly, a pull query is like what you do on a conventional table. I mean, it sounds like, I think of a database table and I do a select on that table where my key equals some unique value, let's just say, and I get a row. So that's analogous to a pull query, right? Good so far.

That's right, that's right, that's right.

And then push, if I understand you correctly, is where I have a stream, not a table, but a stream, and I would like KSQLDB to just be pushing me, you know, the processed, transformed, events in that stream. They're not rows in a table at all. They're just messages in a stream or events in a stream, and I want those when they arrive.

Yeah, that's the basic idea, right? So the push queries are kind of continuously pushing out updates to the answer all the time as new things happen, whereas the pull query is kind of a, you know, a point-in-time lookup as of, you know, right now. And so you can access the tables in KSQL in either fashion. right? So I can subscribe, you know, to the set of all changes to customers, right? That's a kind of push query where every time I update this customer profile, I'm pushing out the updated profile. Or I can do a lookup at a point in time and say, okay, I want, you know, I want to pull just this one customer.

And I want to give an example of that, um, because these are, these are, they're new terms, um, but they are not complex ideas, or they're not new ideas, but imagine, let's say we have like a connected car system or something, and every car is, it's got wireless data service, and it's reporting back sensor readings every second, let's say it's super aggressive, and all those are getting produced into some topic, and so you have this big network of connected cars, and a push query, would be one where you said, well, you know, let's select from that and filter it to be only a particular model of car. And so you now have a new stream that you've created that's a filtered version of all these events, all these sensor readings coming in from your connected car network. That select on that stream we call a push because it, from KSQL's perspective, it's pushing out new events as they occur. It's just streaming data, they're not, really corresponding to entities, they're just these events that happen and they get pushed to you. If also we had a table that let's say were subscribers or regions or something like that, a table of entities, of things out in the world, and we wanted to know the current state of the number of cars in a certain region or the current state of a subscriber, is he or she driving too fast or gas is too low or whatever. That's a pull query. So that's like a query of a row in a table, is what we call pull. And push is the engine is pushing out streaming events to you. Um, why the new terms?

Yeah, I, you know, I guess, um, there were two things. There just weren't really words to describe them, uh, and we debated the right words. Um, but, but effectively you already had this dichotomy if you compared KSQL to an existing database where queries in KSQL produced continuous answers. They produce infinite answers effectively. As the underlying data changes, they produce an answer that also updates. Traditional databases, of course, just give you an answer as of right then. So we debated streaming queries versus lookups. We debated push versus pull. They're all different ways of expressing the same idea. But the power, of course, is kind of giving both of those capabilities together because, of course, applications need to both, you know, do these kind of continuous processing and response to streams of events. That's becoming increasingly common, but they still need to be able to look up the result as of now to put into some kind of UI. And so having both of those together is really powerful.

Agreed. And those are— we— that's, I think, a great answer. We didn't really have consensus terms for those things. You know, pull queries we just called queries, but then there's this other kind of query that behaves completely differently that you do on a stream. And once you try to standardize this, you know, when you come up with, with something that disrupts the, the previous data model, you might need some new language.

Yeah, of course. Naming things is always the most controversial part of any software project, right?

So ladies and gentlemen listening, I mean, you know, Jay very politely says this was debated. Yeah, this kind of thing's hard.

You have to be careful or you end up with a piece of software named Kafka.

It could happen, and we'll never really know how. That's cool. You kind of hinted at this, but why does anybody need this? What is important here about it?

Yeah, I mean, I guess the underlying cause I think that's there, if you step really far back, is traditionally databases and the simple 3-tier architecture of an application, it's all implicitly built around the idea that an application is a UI, that that's what it is. Ultimately, a software application is something that shows some UI to a user, the user clicks on the UI, and it sends off some request to the database to do some work, to store some new data or look something up to update the display. Of course, many applications are UIs, but increasingly that isn't true of all applications anymore, I think. In a company now, there's so much software that very often what's happening is the software is connecting to and working with other pieces of software. This is the rise of microservices, but just the general level of software control and automation and instrumentation in a company is much more. So if you think about what the need of a piece of software is, it's really different usually than the need of a human, right? A human, you kind of pull out your phone, you open it up, you open up an application, the application needs to go kind of request the data it wants. But if you think about the needs of software, software is kind of there all the time, doing doing things. So it really wants this kind of continuous feed, this continuous processing, this kind of all the time reacting and responding. That's kind of what I think drives this whole event streaming area. Adding that capability to databases, I think, is really important, like bringing these things together, because if you think about the kind of classic three-tier architecture of an application, it's actually Very powerful, right? You have this one database that does everything for you. So you would have your application code, which you write, you have the UI, which the user interacts with, and then the infrastructure layer is the database, which has all these capabilities. If you think about the ecosystem today, if you try and do anything with streams of data, you're suddenly in this world of complexity where you probably need a traditional database, maybe more than one, You need something like Kafka, you need somehow to hook up Kafka to these other systems, you have all these stream processing layers that you have to add on, you have to wire all this together and secure it all and instrument it all and operate it all. It's just very complicated. And so I think understandably we might say, well, I kind of pine for the days of the simple 3-tier application and the relational database. But if we think about the needs now, there are all these event streaming use cases, and we want to make it as easy as possible to address those. I think the way to do that is to kind of broaden our definition of what databases are, and actually add in the new capabilities that are needed, and put those together. That's really our driving motivation, is simplification. We think if something is going to be a really mainstream way of thinking and working like event streams, it has to be really simple. If we can put together the pieces you need to build an event streaming application in one system, then that's really powerful. You can now do that where you can have end-to-end the capture of data, the preprocessing, reacting, responding, the continuous operations on that data, and then the lookups against the results that would that would drive a UI, and that's kind of all brought together in KSQLDB, which I think is pretty cool.

That is. Is it fair, do you think, well, you must think because it's our name, but is it fair to call it a database? Why call it a database even?

Yeah, that's another thing we debated internally. There was really two motivations in this. One, I think databases get something right in their interaction pattern and the way developers use them. You basically have a system that has this kind of nice query interface where you send off queries and it does what you want. The developer experience is actually good, it's dynamic. If you compare that to kind of classically what these stream processing layers are, the stream processing layers are almost like some MapReduce-y thing where you're sending off processing jobs. To go run, and it's really just not as friendly. And so we felt like that was captured well. We also felt like internally, the architecture of a distributed database and a stream processing layer, as you start to add in the concepts of fault-tolerant state, they end up being extremely similar. And so if you step back, what we had was something which stored replicated fault-tolerant data, that you query with SQL. So we were like, well, what would you call that? And we were trying to think of different names, and we were like, you know, it's a stream processing platform. And eventually we were like, yeah, it's a database, right? If you have SQL stream, if you have SQL processing and querying on top of fault-tolerant replicated data, that's a database. The fact that you've added these additional features, which allow you to do continuous processing, which allow you to, connect to external systems so you can capture external streams, that's actually kind of doing more. And so a way to think of it is instead of thinking of stream processing as some new kind of thing, it's almost like we've taken the idea of event streams and we've kind of integrated it with the idea of tables, and then we've given you these both continuous and point-in-time ways of accessing that, the kind of push and pull queries. And I think that's a more kind of general and complete idea of what a database should do. It should do. Now, we're not done, of course. There's lots more to do, but that was our motivation.

Is it the kind of thing now then that we envision you'll throw all your other databases away? Like, I don't need Oracle, I don't need Postgres, I have KSQLDB. It's a new and better— We've had 10 years now of that sort of talk of the new and better database that would cause you to throw away all your other databases. Are we doing that right now?

No, not quite, right? It's the, what is it, one database to rule them all and in the darkness bind them? Yeah, not quite, right? So, you know, look, all of these databases are really good at what they do. And what this is for is actually for stream processing. If you're building event streaming applications, you know, your situation today is a little bit awkward. You have to glue together all these different things. To get a solution. And so we're simplifying that use case. We're not saying that if you have, you know, some UI-centric application that uses Postgres that you should stop using Postgres and use KSQLDB. That makes no sense, right? We're not saying that if you have some kind of indexing analytics application that's kind of, again, you know, relatively UI-driven, that that's using Elasticsearch, say, that you should get rid of that and use KSQLDB. That, that makes no sense. What we're saying is like, look, if you're doing event stream processing, you know, if you have streams of events in Kafka, you want to work with those, we think there should be something that makes that really easy and convenient. And that's kind of the use case that we're targeting. That's where we think this shines potentially versus, you know, gluing together lots of things. And actually, you know, this makes it a lot easier to, to use stream processing even with external databases because of the Connect integration. So you can, you know, avoid the, the pull queries and just, you know, use the connectors to actually do processing, you know, capture the data and, and get it into the form you want and send that off with again another connector to some destination store for, for performing the lookups. So if you need specialized queries, so maybe you need the capabilities of Elasticsearch, which of course KSQLDB doesn't have the kind of rich indexing capabilities. You can just with a, you know, a single command send that stream off to Elastic for storage and query.

Yeah, so I think the most fair answer that I can give to that is if you're just not doing any stream processing at all, you know, there are lots of good databases that can remember things and do interesting kinds of lookups and interesting kinds of analysis and like that's a solved problem and that whole world is still moving forward like anything else. But once stream processing enters the picture, then we find that those are really much less suitable to the problem. There are just different things happening in an event-driven system, and KSQL is, I think, somewhat more optimized for that case.

Yeah, I think that's exactly right.

Ah, thanks. Suppose I wanted to get it, where would I go to get this right now?

Yeah, that's a great question. So, you know, we've just released this, We're doing it as an independent project, so you can go get it, you can use it with Apache Kafka as you would get from the Apache website, you can use it with an existing Confluent Platform installation. Of course, our intention is actually to add this also to our product offerings, to add it as part of Confluent Platform, and to offer it as part of Confluent Cloud over time. We're starting just getting this out to people who want to try it out, and so don't hesitate. What's out there now is compatible with absolutely anything, so you can go download it, you can go grab it off GitHub. It's out there for anybody who wants to give it a spin.

Nice. Subject to all of the #SafeHarbor qualifications we could make about talking about the future of products and specific releases and all kinds of release dates we won't talk about, what just what's the vision? What do you see in the future of ksqlDB?

Yeah, that's a great question. So, you know, we're really trying to make, you know, these two sides of SQL work well. SQL on, you know, streams of data and kind of the traditional SQL we know and love, really integrate those together into one thing that feels natural together. We think that's what you need to solve problems in this space. And, you know, so we're gonna be making the SQL functionality of the pull query is much more complete. We're gonna be working on the performance and availability of those queries, and of course we have a whole roadmap of work on the stream processing side as well to make that faster, more scalable, more elastic to handle bigger and better state and make that SQL more complete also. So there's a ton of work happening on kind of both sides of the equation, but this is kind of our first, release that puts together all the major pieces of what we envision, the connectors, the, you know, the processing, the lookups, all in one place. And that's what we're so excited about.

My guest today has been Jay Kreps. Jay, thanks for being a part of Streaming Audio.

Thank you so much, Tim.

And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay you don't have to pay for anything. If you're a new user and you go through the regular signup process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first 3 months after you sign up. So that's $50 per month of serverless Kafka for 3 months at no cost to you. So go to the signup link in the show notes. I don't wanna read you the URL and sign up now. I think the only thing The only thing I could really do more is write your code for you. And I think we can both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you wanna discuss it or ask a question, you can reach out to us on Twitter @confluentinc or reach out to me @tlberglund. That's T-L-B-E-R-G-L-U-N-D. Or you can hit us up in community Slack. There's a signup link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.