The Evolution of Apache Kafka: From In-House Infrastructure to Managed Cloud Service ft. Jay Kreps

Hello, you're listening to the Streaming Audio Podcast, and my guest today is Jay Kreps, one of the original creators of Apache Kafka and through a long and interesting journey, the CEO of Confluent. We talk about the natural evolution of companies from those Wild West pioneers of internet scale to today's service providers that let you nearly forget about scale entirely and just worry about your business. And we also talk about the natural ups and downs of life, and I'm still not sure how I feel about that part of the conversation, but you'll find out when we get there. Before we start, let me tell you that the Streaming Audio Podcast is brought to you by Confluent Developer, which is our site that teaches you everything you need to know about Kafka, from how to start it running and write your first app, to architectural patterns, performance tuning, maintenance, and more. Check it out at developer.confluent.io, and if you want to take one of our hands-on courses that you'll find there, you can easily get Kafka running using Confluent Cloud. Sign up with the code PODCAST100 and we'll give you an extra $100 of free credit to get you started. And with that, I'm your host Chris Jenkins, this is Streaming Audio, let's get started. My guest today is Jay Kreps, who is one of the original co-creators of Apache Kafka, former cutter of code on that project, and now the CEO of Confluent. I've got to say, there's something a bit strange. I feel like inviting the Confluent CEO into the Confluent podcast is a bit like I've broken into your house and I'm inviting you into your kitchen.

Easy guest to get, right? Easy guest to get.

Yeah, hopefully. But nevertheless, welcome to the show, Jay. Thanks.

Yeah, excited to be talking, Chris. Cool.

So I kind of wanted to frame this by saying, years ago, I had this contract. I used to be a contractor, and I had this contract where the company did social media analytics, right? But that's what they did in theory. In practice, they spent half their time keeping a Hadoop cluster running.

Yeah.

Right? And it's just the care and feeding of that system. So I've always wondered, what was it like back in the day in LinkedIn when you're trying to build this thing and run it and actually build apps on top of it?

What was life like? Yeah, I mean, I think it was a slightly different world. LinkedIn at that time was one of these early social networks which were kind of scaling like crazy. There was a burst of kind of consumer tech companies, and there wasn't really good solutions for how to build that kind of thing. And, you know, the, the examples of success were like Google, where they did everything in-house, um, and the examples of failure were like Yahoo, where they tried to do everything in-house and they didn't do it good enough, right?

Yeah. And no one was quite sure what the magic sauce was, right?

Yeah, that's right. Well, yeah, yeah, you know, so I think for a lot of the Silicon Valley companies, the takeaway was, yeah, you know, really building out kind of excellent general-purpose in-house infrastructure is the key to success. And, you know, I think at the time that may have actually been right. You know, to actually scale and build rich products, you needed to have these capabilities. Like, you had to have them, and there you couldn't go out and buy it. Uh, so that, that really was what you were going to do. And, you know, for LinkedIn, that was a combination of adopting open-source technology than making some open-source technologies, you know, a little bit of both. And so kind of the— some of the database technologies we built, Kafka we built, you know, Hadoop we, we, uh, we borrowed, right? We, we— but, you know, the interesting thing was even the things that we were getting as open source, you know, it was about the same amount of work to really operationalize it at scale as it was to build it. That sounds counterintuitive, But the problem of running big data systems at scale is so hard. We would actually joke internally that getting to write the system was the reward for having to run it.

Yeah, I can believe that.

And it's one of those problems that gets harder the more successful you are, right? The more the system is used, the harder the problem is. And the more usage, the more applications, the more scale, etc. That was a pretty significant focus for LinkedIn. And with the high failure rate, you had to attract these very high-end distributed systems engineers. You had to kind of conceptualize and build the right thing. It kind of has all the risk of startup to get one of those projects to really be successful. But to some extent, The end result is, well, you just have a database. And I think it's a mark of how much the world has changed that now there's all these cloud services, which are amazing, which do a lot of this stuff.

Yeah, we've kind of gone past those Wild West days, haven't we?

Yeah, yeah. I mean, I think if anything, the mindset is still kind of shifting off the, you know, call it the Google mentality. But, you know, but I think that's that's pretty far along now. We work with a lot of Silicon Valley tech companies, and they're kind of going through this shift where they often have these very high-end infrastructure teams, but increasingly, the desire is to get them focused on something unique to that company, which is typically not the Kafka clusters or the analytics infrastructure or running the databases or writing new databases. It's typically what that— something unique, the kind of special sauce for that business.

And yeah, it's like you want to spend 100% of your— sorry, you want to spend 100% of your time on that social media analytics and 0% on maintaining Hadoop, right?

It's possible, right? I mean, there, you know, there's not great cloud services for everything, so, you know, there are, there are things where you just need to— you just need it. Um, but, but I think increasingly that's changing. The cloud providers have done a great job. There's a whole swath of companies like Confluent that are putting huge amounts of resources that are kind of some orders of magnitude more investment than most internal teams can do. And I think it's a big improvement, honestly. Even though I ran a significant infrastructure effort at LinkedIn, to some extent, we were a little bit like you know, this may only make sense to the Americans, but it was a little bit like AT&T, the phone company in the '70s, right? So they had a kind of a monopoly business. And so, you know, you could get any phone you wanted as long as it was a black rotary telephone. And, you know, you didn't like that phone, you could not have a phone, right? And like, right, yeah, customer service was available 10 to 4. And, you know, it wasn't quite like that. Obviously, we were working hard. But to some extent, that, you know, the amount of choice of what you had as an application developer was pretty limited because of the the difficulty of running this stuff at scale. So we were kind of limited to like, you know, two flavors of ice cream. And our ability to really like produce that on demand was always limited just because it was always a struggle to, you know, scale, keep it up, ensure everything was secure, kind of just the very basics were so much work that it was very much kind of not on demand as it were, right? So, you know, I think in that respect, it's a much better world for application development where you're no longer as constrained buy basic infrastructure.

Yeah, I can see that. How did that evolve though? There must have been, at some point, you said at LinkedIn, "We need to spin this out into a company." And then at some point in Confluence life, you said, "It's not enough to just be providing the software." Yeah, yeah, yeah.

Well, yeah, the first one happened. I mean, when we were starting Kafka, we actually were kind of thoughtful about it. We'd done some other open source projects. We felt like, hey, there's a really big gap. Everybody's focused at that time on Hadoop and these key-value stores. And it was kind of these two extreme ends of the spectrum. One was doing kind of very rich things with data like once a day. And the other was like looking up little keys and values in half a millisecond. And— Yeah. A lot of our challenges of data weren't really bad. It wasn't like, "Hey, we need another storage system." It was really about the kind of flow and interconnection. And I felt like that was very different, where a lot of the distributed systems work was kind of taking an idea that already existed, a database, a file system, whatever it is, and trying to make it scalable. This area of connectivity, I didn't feel like it had really gotten first-class attention. There were older message queues and, you know, kind of bits and bobs that move data around. But if you think about just the level of intellectual energy that went into databases, the research efforts in any university with a computer science department, like the amount of commercial investment, a lot of really smart people tried to come up with a kind of really principled, thoughtful abstraction for managing data at rest. And nobody really put that kind of thought into data in motion, into how data flows and how you can kind of connect and tap in. And so, and that was where we were spending a lot of time. And it was one of those things where you're kind of either going to hire a team of people who do these random one-off integrations and spend all their time maintaining it, which is actually a pretty big challenge, especially if you're kind of operating at scale and even a relatively simple integration can be relatively complicated. To make work well. So, you know, you're either going to do that or you're going to try and come up with some better way. And so we wanted to come up with some better way, and we felt like, hey, this is how companies ought to work. And so, you know, from early on, we felt like, hey, this should be very successful. Now, I would say at first it was very unsuccessful. We kind of launched Kafka, and we were expecting, you know, a little more attention for it. Our key-value store had been popular, some of our other open-source stuff had been popular, And I think at first people were like, eh, you know, who cares? Like, what is it? They didn't even know what kind of animal it was, right? Like, is it a—

yeah, it's so much harder to get people excited about something when it's not entirely familiar, right?

Yeah, that's right, that's right. You know, it's not like we were, um, product marketing experts. So, um, yeah, so it took, it took probably about a year of, you know, talking to people to get a better conception of like, hey, why, why are we excited about this internally and why externally. And then it did start to catch on in Silicon Valley. And I think a lot of the other kind of pure digital companies were, you know, working through similar challenges of just, hey, how does it all connect together?

Was this while you were still at LinkedIn? Were you thinking about it? Yeah, that's right.

That's right. So, so then the, you know, I think the question for, you know, the viability of a business was really like, okay, you know, there's a handful of these very large tech companies that are doing this. You know, it's pretty hard for them. Like, they're hiring these big teams to go build this into all their systems, you know. Is this an idea that would work outside of that kind of tech company and become something more mainstream? Well, smaller tech companies and mainstream enterprises, you know, is this applicable to the larger world or not? Yeah. And, you know, just in talking to people, increasingly we became convinced it was. You know, it was interesting, um, like for, you know, I would call it a more traditional enterprise, if anything, they're more complex, they're more diverse, there's more older technologies that have to come together. Like in some, to some extent, the importance of that connectivity layer, it's dependent on how big the organization, how diverse the business is, how many older systems there are. That's kind of what you're abstracting away. And it, you know, to some extent, the Silicon Valley companies that are these very young companies that turn over the technology stack relatively quickly. They're actually the least complex internally. They may be operating at scale, there may be technical challenges, but it's not like they have technology from 30 years ago that are still running big parts of the business. And so, yeah, as we saw that, we were like, actually, this makes sense for everyone. And the competitive pressures on these more mainstream companies are going to be exactly the same. You know, they're, they're all, they all need to have like a significant digital portion of the business. Yeah. So I thought, okay, you know, this could work. And you know, then it was just a question of, hey, what's the, what's the business model? How do we get going? How, how does one even start a company? Um, is there, is there any hope for this? You know, that, that, that became the concern once we felt like, hey, this is gonna turn into something, uh, or this has the potential to turn into something big. And, you know, then maybe we were as much motivated by, you know, fear, which was like, we were like, hey, you know, this could be a big thing. How sad would it be if we kind of started on this and then somebody else came and did it? So, yeah, fear of missing out is a powerful motivator to go try it out.

Yeah. This also explains why Connect came in so early.

Right, that— yeah, you know, it was one of the dilemmas for us, um, when we were thinking about how to do the company. Um, we were late enough in the game that, um, you know, it was clear you could kind of start with just a managed service in the cloud, or you could start with kind of a software product. For us, you know, we felt like for what we did, it was very important to span all these environments, but you can't do both at the same time. It's just not practical.

Yeah.

You have to pick one. So, the kind of big choice for Confluent early on was trying, you know, do we want to try and make operations easier by offering a cloud product, but still having just kind of the bare core of Kafka, which is like this commit log, which is kind of a, you know, pretty low-level abstraction, or do we want to try and make it a little bit easier to use for complete use cases, which was like the connectors and stream processing capabilities. Which makes it a lot— I think that kind of shows a lot more of the power of the model of what you can do, you know, what's actually different from, like, a message queue or something like that. And that was the dilemma, is, you know, do you want to make life better for development or for operations? What's going to help you kind of get going in the world better? And yeah, okay, it was actually a big dilemma because the reality is you need both, but If you have a company, you're starting out with 3 people, you can't— yeah, you can't— if you try to do it all at once, so you try and build everything, you end up building nothing, right? Yeah, that's right, that's right. So, so we were, you know, we were pretty aggressive, I think, through the early days of the company of just trying to get to that more complete offering, really, on both sides, you know. First, on trying to complete what, you know, what, what the technology could do, and then secondly, on trying to get a managed service out there that would, you make this available without people having to build some internal team. And that combination, I think, is what makes it something that can be kind of a no-brainer decision. If it's like, hey, you have something powerful, it's a good abstraction for data, but you have to go hire some team of experts and spend years baking it into all your systems and figuring out how to run it, that's a big trade-off. It's valuable, but it's really hard. And I didn't think that was inherent. I think it's possible to make something that's a better abstraction, is easy to use and consume and get the value out of. And I wouldn't declare mission accomplished on that ever. The goal is always to try and make it easier to consume, make it easier, abstract away more of the operations. But we've certainly come a long way in that respect, I think.

It's one of those things often in, like, systems design. If you start on a better bedrock, you just keep building forever, but at least the tower stays stable as you go up, right?

Yeah, yeah, that's, that's certainly the hope. You know, I think that's— there's always an element of that in these kind of infrastructure systems because you can't— they're not quite as agile as some application layer stuff. So you have to kind of get the basic, you know, bones architecture of the thing sort of right, and then of course you can evolve from there. But, um, you know, it's harder to make big right and left turns.

Yeah, and as it evolves, you find out how right you were on your initial guess.

Yeah, that's right, that's right, that's right.

So I didn't know in the early days of Confluent that you had the gleam of Confluent Cloud in your eye.

Yeah, you know, it was a big dilemma. Like, at the time we were trying to start the company, you know, it was— there were really no examples of successful offerings on top of the cloud, but we had kind of come out of a world of running Kafka, and we knew that that was like a big, you know, challenge that would be a great product offering. And so, you know, in many ways that was very attractive. I would say we actually almost started that way. And then, you know, the concern on our side was like, well, to build a good cloud service, you know, you're going to spend the first couple years just building out like billing and compliance and operational stuff. And like to some extent, the user experience on the other side doesn't really get any better. You're kind of just offering the same thing, but you're, you're taking away the operational burden. And so, you know, that was why we didn't, you know, it was kind of a combination of those two things. It would probably be a little harder to get started on the business side, and, you know, would kind of limit us to just the Silicon Valley crowd that was all in, all in the cloud, and probably the smaller, more risk-tolerant part of that, since there weren't a lot of examples of people using kind of infrastructure from other companies other than the cloud provider. So, you know, as a result, it was, um, you know, we started on-premise, so then we started internally this cloud effort relatively early, um, okay, after that, like, you know, I, it was in the first couple years that we started the development on that first, that first version. And then of course it takes significant time to really make it good.

So what was the tipping point? When did you say, okay, we've got enough now that it's time to not just do this one arm but launch into the cloud?

Yeah, I mean, for us it was, you know, we, we just felt, um, you know, I think there was never any doubt that that was kind of where the puck was going, um, over time, you know, like, uh, just in terms of where IT dollars were spent. Still majority was on-premise at that time, but, you know, if you look at kind of the first derivative, right, the cloud was the place that was more new use cases more quickly. And then just from first principles, we felt very strongly, for all the reasons I described, that that was going to be much more successful. Because I'd run one of these infrastructure teams, one thing I understood was everybody hates their infrastructure team because you're always the bottleneck, you're always slow, it's never good enough. If you do everything perfectly, then nobody knows you exist. And the one time you make a mistake, that's when you get to meet the CEO, right? So it was, uh, you know, I was, I was very familiar with the dynamic and it just made a lot of sense that if, you know, it's a hard problem, it's a problem that's not kind of the core competency of a lot of these companies, and they're not doing it that well internally, and the, the resulting product internally is not that good. So if they— if there's an option, people are going to move to that pretty aggressively over time. And so So yeah, that kind of pushed us to start early. I mean, that, that was a pretty significant challenge in its own right, just because, you know, what they say in any startup is don't try and have two products. You know, it's like you don't want to divide the army up and fight two wars at the same time. You know, in many ways, a cloud infrastructure offering and a software offering are actually much more than just two products. You know, like two software products or two cloud products, it's actually not that bad. You're kind of doing two instances of the same thing. In many ways, it's like a wholly different business model, development methodology, everything.

So yeah, two business models at once is a big, big thing to take on, right?

Yeah, that's right, that's right. So, so we were, you know, we, we felt we were kind of eyes wide open on the difficulty of that, but it was actually probably more difficult than we thought. To make that successful. I still think it's the right approach for our domain. In most domains, if I was starting an infrastructure company, I would probably just do the managed service. But for us, since we're kind of ultimately about connecting stuff together, connect the applications, connect the data systems, connect all the parts of the company, kind of unlock the data and let it flow between things. To do that, you kind of have to be in all the environments, and that's like a significant value proposition for us. And even to an extreme where if there's a region we don't cover, or back when we didn't have good coverage of the cloud providers, you know, there would be significant pressure from customers. They would say like, hey, we want to use your thing, but the whole point was to connect all the stuff and you don't have regions in APAC, we're not going to work with you, right? And so we're like, okay, we better get regions in APAC. And it's the same thing where it's like, okay, you have AWS, but you don't have, you know, you don't have Google or Microsoft or whatever the other cloud provider is. We'd love to use your solution, it doesn't cover it. And so, for us, we had this idea of just kind of make it available everywhere, and make it possible to connect all that up into one fabric for data in motion. If you do that, that'll be difficult, but it'll be really valuable. And most people are probably too smart to take on all that pain. And that's kind of the— You know, for any company, you're trying to figure out, hey, what are these competitive modes that we can do really well, uh, that are going to be hard to copy? And so you want these things that are kind of painful but valuable, uh, that, that you can kind of offer to customers and that are hard for competitors to offer to customers.

Yeah, and you've got to get the right level of pain where it's achievable without killing you.

Yeah, that's right, that's right. Yeah, well, you have to solve the problem. That's the challenge.

Yeah. Yeah. So I've got to pick you up on trying to get rid of the infrastructure team, which leads me to something I want to know. So you get this thing now, there is a good managed service available and companies start moving towards a managed service. What happened to that generation of Kafka experts who are deploying in-house?

What happened to their jobs? Yeah, well, you know, look, the good news is there's no layoffs in software, right? So the actual natural movement you tend to see in companies is movement up the stack. And for people who've been working in software engineering for, you know, call it more than a decade, you've probably already done this. You know, there's a set of things that were much lower level that we used to do, and then the tooling gets a little better and you're up one level in the stack. And, you know, every, every year it's always like, oh hey, you know, the new tools are going to write all the code automatically and all the problems will go away. And yeah, The problems never go away. They actually tend to get harder. They're just at a slightly higher level of abstraction. They're a little closer to what the business is trying to do. I think it's actually really, really valuable. And you see this just kind of in the raw economics of it. If you look at what's happened to software engineering salaries in the last, call it 15 years, it's amazing, right? You know, what, what, what caused that? Well, partially it's scarcity, right? It's a hard skill set. But, you know, why is there so much scarcity? It's because there's so much demand, and because the value one person can create as a software engineer is now actually quite high. And that comes from working with higher-level abstractions. It comes from kind of not being in the weeds. It comes from kind of moving the needle for what, you know, the business is trying to, you know, do. That's the only way you can kind of produce enough value to pay these very large cash and equity compensation packages is if that kind of pencils out in the end, it is creating in aggregate enough value for the business. And so that trend of kind of moving up the stack has actually been immensely valuable. We see it in our customers where just like, yeah, there's some team, they're kind of struggling to get the next security patch rolled out in their own self-managed thing. There's often some kind of internal debate about like, "Hey, do we want to do this and kind of own it ourselves, or do we want to manage service?" I think for those who move, they immediately move on to a set of more interesting problems of how to really use data in motion. Like, how does it plug into all the systems we have? How do we react and trace what's going on with it? How can we feed this into the machine learning algorithms that are going to make decisions. Like, you know, in many ways, um, the more interesting problems that are a little bit closer to what that company is doing, and I, you know, it's usually more interesting. Um, you know, I think a lot of the work we do is very interesting when done at scale. You know, if you're managing thousands of clusters, if you're very focused on efficiency and security, etc., that's great. But, um, you know, to do it in one company over and over again there's not enough time to really fully automate everything, and there's always just this kind of long, long tail of, like, toil and drudgery. And so, yeah, I think if anything, you know, it ends up being very positive for a lot of these infrastructure teams. And you kind of see the same thing, which is those teams never really— they, you know, they always seem to have a lot to do. It's just the, the level of abstraction changes. Um, I had a funny story from LinkedIn that was a little bit like this. So we were, when we were building out the Hadoop infrastructure, we came up with some like pretty successful analytics applications. So we were always getting these larger and larger Hadoop, uh, clusters to be able to like crank more data. And so, you know, the, the progress was very dependent on this, and it was, it would impact like how many people would join the site and the level of connectivity and the social graph, all these kind of things, if we could get our scoring to be better. And so There was one point where we had built out this new Hadoop cluster and it was delayed. And, you know, what, what was delaying it? Well, we had all the computers, but it turned out that the power cord for the hard drive was too close to this, like, kind of raw metal. I don't know what you would call it, guardrails. And it would kind of scrape the rubber on this little wire. And there was a concern that if you scrape off all the protective rubber, someone's gonna get electrocuted. And so, we're in there, like, taping the wire to the side so that when the drives come in and out, it doesn't scrape. And I was thinking about this at the time, and I was like, you know, I'm not operating at the right level of abstraction. We weren't thinking about the wires, and now we're thinking about the wires, and we're taping the wires, and how many drives can a person tape down? And, you know, I thought, you know, we just want to process more data. Like, this is not the right— this is not right. There was kind of early AWS stuff, and I was like, man, we should look at that and see if it's cost-effective. At the time, it was not cost-effective compared to what we were doing, but like you could see the appeal of just having some of these problems be somebody else's problem.

Yeah, yeah. It reminds me, have you ever read The Hitchhiker's Guide to the Galaxy?

No, no. It's one of those science fiction classics I've never gotten around to.

There's this bit where he talks about exactly this. You never run out of problems, but an early culture has the problem, how can we eat? How can we eat to survive? A more advanced culture still has a problem, but it's where should we go for lunch?

Yeah, yeah, that's right. Well, they have this concept of the hedonic treadmill. Have you ever heard of this? No. Okay. So, like, it turns out people kind of are more or less equally happy. And so when, when something good happens to you, you know, maybe you get a nice house and you get more happy for a short period of time. Right. But then you go back to the same amount of happiness. And if something bad happens to you, you're like, oh, this is terrible. You're really upset. And then you kind of go back to the same. And so my friend is like a video gamer, and in these multiplayer video games, they have a similar thing, which is if you win, they just drop you into games with better people. So you're never really progressing. You're just, just playing with better people who kill you the same amount. And, uh, Apparently humans are wired a little bit the same, which is like, you know, kind of your level of happiness is more or less relatively constant. I don't know if that's good news or bad news, but I thought it was a really kind of fascinating psychology observation.

Yeah, I really don't know if that's good news or very depressing. It's like, yeah, it's, uh, we must imagine Sisyphus happy, right?

Yeah, that's right. Well, you know, it's like we're in a pandemic. At first everybody's like, this is terrible. Then people kind of adjust and they're like, yeah, it's It's bad.

And then we're allowed out again and we're like, "Yay!" for about 10 minutes.

That's right.

That's right. That's right. So to get us back on the topic of computing, so I want to hear more of the— you must have some stories from inside customers of how not just the people on the ground, their job changes, but if you're a company that moves from having to worry about all this stuff in-house to work to getting all that off your plate, how does life change for the company?

Does it—

do— does the company get a different focus?

Yeah, I mean, I, I think it's a big deal, right? So, you know, at one point at LinkedIn, we were just trying to figure out resourcing, and, you know, we were discussing, hey, what's the right level of resourcing for internal infrastructure? And what's the right level of resourcing for kind of product features. And the decision at the time, given like all the scaling challenges and, you know, how much we were kind of slowed down by things, was that about 70% of the engineers should be working on internal infrastructure and about 30% should be working on product. Now, if you think about, for a lot of companies that are product-centric and where software is a significant component of that product, you know, that velocity is going to determine, you know, over a longer arc, a lot of the success of the company. And so if, yeah, 70% of your resources go into something that's kind of a means to an end, then, uh, you know, you only got the 30% that's left. And so, um, yeah, you know, I think that's a very clear illustration of it in a really quantitative way. And so you can see that now where Certainly, if you look at this newer generation of smaller startups that kind of was born in the cloud, they're very aggressive about not taking on— they almost think of these operations projects as a kind of debt of like, "Oh, now we're stuck running this thing forever," and they're very aggressive about trying to avoid that.

Yeah.

And so, I think there's some wisdom to that, probably. I mean, it's interesting. It's a different skill set, I think. For tech companies to be good at buying managed services, getting the value out of those products and vendors, and making sure that they're kind of evaluating them appropriately. It's a little different, I think, than operationalizing open source or building stuff in-house, but, you know, still, still important.

Okay, then what do you— what's your completely unbiased guide to choosing a managed service?

Yeah, yeah. I mean, you know, I think the question is how much of the problem can it take on, you know? So when we've thought about what we're trying to do, you know, there, there's kind of been 3 pillars we've tried to focus on. So one is like really try and make something that's cloud native, you know. You shouldn't be picking out like how much disk space there is per machine or how many CPU cores or whatever. That's— it's just not the right level of abstraction. You want to be buying capability to some extent, and you want someone to properly take over the operation of the system and be accountable for that with a strong SLA. You know, if that part doesn't happen, you don't kind of get the, you know, cloud-native capabilities. I think it's a big loss. You know, people from far away, it's easy to miss this, but the difference between, you know, a Teradata, a Redshift, and a Snowflake, say, is really to the extent to which that system is designed to capitalize on the capabilities of the cloud and the degree to which it's priced as a service versus something you kind of stand up and, you know, pay, pay ahead on. And, um, I think that's actually a very big deal in enabling the agility and getting people abstracted away from it and in actually handing over the responsibility. I think these things where it's kind of partially managed, you know, it's a big problem. It's a very split responsibility between the team using it and the team that's kind of partially running it. And so if it doesn't work, whose problem is that at the end of the day? So I think that's critical. You know, for us, the other thing was really trying to get the full ecosystem around Kafka. So if that first pillar is be cloud-native, the second one is, you know, be a complete offering, like really get the ecosystem, the connectors, the SQL layer, how you govern streaming data, like really bring that together because that's ultimately what people need. And so if you manage one component, but they have to go piece together some ecosystem of other things to make it work, you're not really taking on that much of the management.

And so much of any project is integration.

And then you need to do the ops integrations and make it work across all the environments. And so those were kind of the three that, like cloud-native, complete, and everywhere, like really try to nail those three. And then, you know, the other side of this is even more basic, which is just, you know, is this service trustworthy, right? You know, I think one of the amazing things about something like S3 is you can just treat it like a utility. You can depend on it. And that's what you want in an infrastructure layer. If it's a little finicky or it doesn't always work, it's actually much harder to just kind of consume it. That kind of trust is, is very important. Um, and then, you know, the, the cost effectiveness— like, ultimately you need something, again, like a utility that you can kind of have a cost structure you feel, um, you know, is healthy that you want to build around. Um, I think for a lot of these services, it's possible to end up with something that's very cost competitive, um, with the kind of do-it-yourself route, in, in part because that tends to be pretty expensive, like Engineers aren't cheap. You typically end up with a big pile of semi-utilized cloud resources. So, I think if you can kind of check those boxes, that's what makes it compelling. Something that's really a first-class cloud service that's a good deal. And we do that analysis with customers often of like, hey, what are the economics of running the open source Kafka yourself and try to piece this together, hiring a team? What are the economics of our pure software offering? What are the economics of our cloud offering? And it's interesting, typically the cloud offering is a much better deal than the other two, just in part because of the utilization of resources and then the people and what the people can go do.

Yeah, yeah. If you're spending that kind of money on expensive people, you really want them to be focused on just stuff that makes your business unique. I think.

Yeah, I think that's right. Like, ultimately, um, you know, the, the kind of economics for engineers is kind of how much value you can create. And so one way or the other, you want to be part of something that's kind of adding a lot of value to the business. I think it's one of the things that it's often easy to lose track of, you know. If you're kind of early in your career, you often have a limited domain that you're working on. It can be hard to trace that back. But to some extent, like, the ability to spend on engineering is determined by the value the business is creating. And so, in a very direct way, that's kind of what matters in the end.

Yeah. I often think, as programmers, we don't get paid these great salaries to work. We get paid to teach machines to do the work, right?

Yeah, that's right. There's a really funny or interesting or clever riff by the guy who founded O'Reilly, uh, books. The— yeah, yeah. And, you know, that's one of the things he said is that, uh, in these tech companies, it's actually the, you know, the workers are the computers and the managers are the software engineers. And what you're kind of doing is creating some system of work to try and get everybody to do their part. And, you know, um, I thought it was kind of an interesting way of viewing, uh, this next generation of company that to some extent, the execution of the company is actually happening in large part in software. And the, you know, kind of the building of that system and the tuning of that system, that's kind of what is actually being done. It's a slightly more meta problem. And, um, you know, I thought that was an interesting way. And he was kind of contrasting it to maybe a more traditional, um, you know, whatever you want to call it, industrial economy type organization that, you know, is a very kind of command and control people structure where they're, you know, you have X workers at Y dollars per hour and you're trying to utilize them. And so, um, you know, it's a very, it's a very different way of thinking about the, um, the structure of a company. And then of course it has a whole bunch of implications in terms of how people work together and culturally what's important, um, you know, to make that successful.

Yeah, yeah. So, um, That sort of leads into one other thing I wanted to ask you, and that you mentioned stream governance, right? Which seems to me like one of those things that you kind of want, but you can't even think about it when your main concern is keeping the cluster running, right? It's kind of like a second order thing. You can only want it once you're free enough to want it. Is there more stuff like that coming up in the pipeline, these things that only become demands once you've got the bedrock in place.

Yeah, absolutely. So, you know, kind of our goal is really make something that makes it possible to open up a lot of the data across the company, have applications be able to tap into that and respond in real time and do smart stuff with it. And, you know, there's a lot to that problem, you know, beyond just kind of the raw transport or transformation of bytes. From one place to another, one format to another, you obviously need that capability, right? Like that was—

Yeah, that was hard enough to start with, but—

Yeah, but if you look at kind of that wider world of what does that enable or what's required to make that successful, it's actually a really interesting space. And it's, I think, really underdeveloped. A lot of the ecosystem around databases has kind of grown for a long time. So if you think about the ecosystem around data warehousing, There's all kinds of governance and analytics tools and, you know, graphing layers and GUI layers and layers for, you know, mapping out all your tables and, you know, managing all your queries. And a lot of that is just coming into being in the world of streaming. And, you know, for us, we feel like there's a really significant platform emerging in companies around data streaming, right, that, that can really connect all these things together. But a lot of what needs to kind of grow up there to make that as productive and usable is just coming into being now. And so it's an exciting time for us, I think for the whole community of people who think about this and work around it, because the white space is not all filled in. So there's an opportunity to add to that picture.

Yeah, we're gradually building up on that. So last question, I think, on the same topic. Is, you know, to go back to the start, you said you kind of saw the way Confluent would evolve right from the days in LinkedIn, but where is it at now? Do you still— is the future roadmap still set by that early vision, or is it more talking to customers and getting their feedback and them saying, "Hey, we need stream governance now"?

Yeah, you know, it's a little bit of both. I think, The— you often hear this debate, uh, on product teams, you know, from engineers, which is like kind of, hey, are you more customer-driven, um, or are you more vision-driven, right?

Yeah.

And for the, you know, our kind of product vision, it kind of comes in different forms. You know, I think there's an aspect of like, hey, what, um, you know, what's the kind of high-level thing we want to be in the world? I think there's also an aspect that's more kind of bottoms up. What does the technology want to be, right? Which is a funny thing to say, but, you know, to some extent with these infrastructure layers, you're trying to make them kind of more complete, more general. There's kind of a natural way of generalizing or expanding capabilities, which is often good. Um, and it's, it's often not something that customers will ask for. Uh, sometimes they will, but sometimes they won't, right? And yet it often opens up very new use cases. And so to me, you know, it's usually not one or the other. Typically, customers are very good at assessing their own problems, you know, and companies that don't listen to the problems that customers have usually aren't successful for very long. And, you know, really understanding the environments and challenges people are facing is incredibly important. I think customers are often really good at pointing out kind of gaps. You know, typically when a product infrastructure layer company like us, if you make something, you know, it's not perfect. There's gaps, there's holes, there's things you haven't thought of. And those become really clear to folks when they try and use it and put it into practice for something. And so that feedback is incredibly valuable and you have JAGS on it. You know, I think typically, I don't think you can depend on customers for that bigger picture of like, hey, what are we trying to do in the world? You know, they're just not thinking about you that much. Yeah, maybe they are, but, uh, you know, I wouldn't depend on it. And then that, hey, what does the technology want to be, you know, that, that I think needs to come out of an engineering team that has kind of a strong view of, hey, this is what's happening in the space, this is what makes sense, this is how this can be simpler, more powerful, you know, more elegant, better, uh, over time. And, you know, kind of the, the hard problem is trying to marry all those things. Together so that you're solving for all of that over time. That's quite difficult. Seeing those patterns.

Yeah, you're seeing those patterns and juggling them all, right?

Yeah, yeah, that's right.

Well, I better let you get back to it. Jay, thank you very much. This has been a high point on my treadmill.

Yeah, that's right. Well, good news is it comes back down quickly.

Oh, great. I'll look forward to my evening.

All right.

Thanks for joining us, Jay.

See you again. Bye.

And that was Jay Kreps, off to continue the grand balancing act of customer requirements and software needs. That's actually a really complex path to happiness. If you want a simpler path to happiness, you could experience the fast dopamine rush of getting in touch with us here at Streaming Audio. Leave us a review on your podcasting app, or a thumbs up or a comment on YouTube, or just drop me a tweet. My handle's in the show notes, and we'd love to hear from you. Before we go, let me remind you that if you want to learn more about Kafka, we'll teach you everything we know at Confluent Developer. That's developer.confluent.io. If you're a beginner, we have getting started guides, and if you want something more in-depth, there are blog posts, recipes, and in-depth courses. And if you've come away from all that wanting to use Kafka without maintaining it, then sign up for an account at Confluent Cloud and use the code PODCAST100. That will give you $100 of extra free credit so you can run a little faster or a little further. And with that, it just remains for me to thank Jay Kreps for joining us and you for listening. I've been your host, Chris Jenkins, and I'll catch you next time.

Bye-bye.