Lessons from the early days building Kafka and Confluent | Jay Kreps

Engineers say there's a problem when there's a problem, and then at the director level, people say, "Oh, we have challenge." And senior director level, people say, "Oh, we have an opportunity." And then at the C-level, people say, "We have a big problem." So it goes full circle, right? And I don't think you have to be mealy-mouthed. I think you just say there's a problem. But I think being aware of what is the impact I'm trying to have and getting good at that, that's a skill where you, develop over time, but only if you try. And then, you know, that's the only way you can have impact on trying to get a large group of people to do something differently, is by developing that. If you don't do that, you know, you're just kind of shouting into the wind.

Welcome to the Software Misadventures Podcast. We are your hosts, Ronak and Guan. As engineers, we are interested in not just the technologies, but the people and the stories behind them. So on this show, we try to scratch our own itch by sitting down with engineers engineers, founders, and investors to chat about their path, lessons they have learned, and of course, the misadventures along the way. Jay, super excited to have you with us today. Welcome to the show.

Really excited to be here. Thanks for doing it.

So we thought we would start with asking you about a video I saw on Twitter and this person, Eric Vishria. I hope I'm pronouncing his name right. There's a video of you shaving this person's head, you and a couple other people. And Eric is on the board of Confluent, if I know right. Can you tell us what's going on?

Yeah, happy to do that. Eric was our first investor. So just as we were starting the company, we went and talked to him. We actually didn't have a name for the company yet. So I think we pitched it to him as NewCo and nonetheless he invested. And so, yeah, I think I'm not sure what the milestone was. It, it was some revenue target or ARR target or something, I think. That we beat. And so he had offered to shave his head if we actually hit our— I think it was if we hit our plan for the year or something. And so yeah, we made a video of it, which was— it was originally meant to be just inside the company. And yeah, he didn't look bad. Although, you know, if you've got the shaved head, you've gotta— you have to eventually tan the head. Otherwise you look kind of pale. You look like you maybe just came out of the hospital versus you did it intentionally. So in my case, you know, it's just from balding. It's not necessarily from breaking any plans or anything like that.

#Relatable. Yeah, yeah, yeah.

So it's pretty cool actually that he went through with it and actually published a video on Twitter in this case. I'm hoping Eric's family was okay with the specific choice.

I heard it was not that popular and that was why the shaved head look didn't stick around.

Ah, I see. What was the reverse bet, by the way? What if he won the bet? Was there something that you would have to do?

I think it was one-sided. You know, it was the people doing the shaving was me and the head of marketing at the time, both of whom were, you know, pretty bald. So we had nothing to lose in that respect.

Yeah, nice. So you mentioned that initially when you pitched to Eric, you named this company NewCo. So curious, like, how did that evolve to Confluent?

Yeah, well, we, you know, we were really picky about the, the name that we wanted to give, so we didn't want to put down something that wasn't good. And so, um, you know, you got to put something on the slide deck. We just came up with NewCo, although apparently there's a bit of a tradition of that. You know, companies that don't have a name yet are often called themselves NewCo, but we, I guess we independently invented that name for the interim. I, I mean, my tip to people would be, if you're pitching a company You know, you should probably just come up with a name, you can change it later, versus like having not decided. But the specific sequence of events was kind of unusual because, you know, we talked to the people at LinkedIn as we were leaving and they said, oh, you know, we would be interested in investing, but you would need to go get kind of VC that would price the round. And so we went out and pitched a bunch of VCs much sooner than we'd actually expected to. We'd expected to leave LinkedIn and really kind of spend some time and kind of get everything baked. So we were effectively doing everything a couple months ahead of schedule at the time that we were fundraising because we wanted to make sure the LinkedIn folks didn't lose interest, kind of strike while the iron's hot. And so the result was a bit of a rushed initial fundraising process. Nice.

I think it's pretty cool that you guys have a playful, if I may say that, relationship, right, with this VC, I guess, in particular. How has that relationship— like, was it always like this from the beginning, or how has that evolved over the past years?

Yeah, I think we've been lucky to have, you know, really good early investors, and I think that was really helpful. Anybody who's involved or on the board has a stake in the company, you know, they're going to be part of the early team, and, you know, that can be a, you know, that can be a great addition, or it can be a bit of a pain, really depending on who the person is. You know, it's, it's, I think some investors add significant value, and then I think some probably subtract some value, right? So I think the trick is trying to figure it out when you're raising money because, you know, at least for us, we had, you know, very little experience with venture capital, didn't really know the ecosystem of people, you know, so we were coming into it a little bit blind and trying to make sense very quickly of who all these people were. Again, because of this rushed process to try and keep LinkedIn interested as we were leaving.

So in this case, like you went from being an engineer at LinkedIn to starting the company and obviously Kafka existed at that time, it was open source and you wanted to build the company around it. Fundraising is not a skill that engineers typically have and especially when you were doing it, like you said, a little bit rushed in the process. Did you have any guidance at the time in terms of how to go about this and have these conversations?

No, absolutely none. There's some information on the internet, not nearly as much as, you know, now there's so much information on starting companies that you can go listen to like 10,000 hours of podcasts and blog posts and books and so on. Yeah, there wasn't a ton out there, to be honest. There was some at that time. And so yeah, I'd read some stuff online, that was probably the extent of it, you know. So yeah, we, we definitely did not know what we were doing. I don't think that's necessarily a bad thing. I mean, I actually think the skill for early-stage investors is kind of being able to see the diamonds a little bit in the rough, because the reality is the level of polish may not be a perfect predictor of the quality of the team or the idea, you know, in that very early stage. Later on, should be. And so, yeah, you know, I think actually the fundraising process was pretty successful. We had a number of people who were interested. Yeah, despite us having no idea what we were doing, and it also being just 3 very technical people who had not really even worked in an enterprise business, you know, let alone founded a company.

Interesting. How did you guys make that happen? Who wrote all the emails to reach out, or did they just— one person or 2 VCs know that you guys are doing this, and then more just kind of, um, yeah, uh, yeah, we sourced a bunch.

So, you know, we kind of went out to our connections and tried to figure out Who knows these people? Basically, we didn't know any of the VCs, so we just looked at companies we thought were good and then tried to figure out who invested in those companies and then just met with a bunch of people, right? So then tried to get introductions to them. Neha, who was one of my co-founders, you know, I think it was her cousin was the CEO of a company who was extremely helpful in introducing us to people. There was a few other paths that we had. But we just collected a bunch of intros by whatever path we had to the people that we thought might be good. But of course, it was pretty hard because we didn't really know who to talk to. We were just trying to figure out like, oh, who covers this kind of space and might be interested.

I heard there's like a quote— not a quote, uh, someone told me that, oh, for VCs, right, like their product is money and money is like the most commoditized product there is. So how do you differentiate is a big question. Was it clear to you during that process, like, some VCs were just so much better than others, or did it all feel kind of, um, like middle of the road?

Yeah, for us, you know, I, I think board members can definitely be very helpful to a company, so I don't think it's commoditized. You know, I think that can be very valuable, but it could also not be valuable, kind of as I was saying. So the question is, how do you tell? And the, yeah, the answer is, you know, mostly you talk to other people who've worked with the person, probably the same as any hiring decision, and you talk to the person and try and assess it. So yeah, so, so I think probably that for us, I think a lot of the decisions were like, hey, how do you weight the brand of the firm, the kind of seniority and experience of the individual, and like what they've done, plus your conversations with them, like how much they seem to understand the company and what you're trying to do, plus the references that you do, you know, how do you kind of weight all those different factors? I don't know if we came up with the perfect system, but it worked out well. So, um, you know, but yeah, I think we definitely thought a lot more about it in the first round. I think we actually took like a week to pick, which is probably about 6.5 days longer than we really needed.

That's nice, right?

Usually it's the other way around, but once you've gotten enough traction, right?

Then it's like you are picking the VCs. I know this is a long time ago, but do you remember any question that you asked everybody and then you were kind of judging, uh, like their answers to that question?

Yeah, yeah. I don't, I don't think it was so much that. I mean, the conversations are on both sides, right? They're going to ask you questions and you would ask them questions. I actually go, I put a lot of weight in the questions the person would ask. So I, I actually like people who could understand the business we were describing, why it could be valuable, and understood some of the problems that we would probably face. The more that somebody could pick it apart and have like kind of a very critical take on what could go right or wrong, the more I thought they really got what we were doing and would be helpful. And even though in some sense that could come across as negative, at least for me, that kind of shows, oh, okay, they, they really get it. I've always felt that way. I actually feel that way, you know, if we interview candidates to work at Confluent, you know, the— I always, you know, sometimes people don't really have any questions at the end, which is fine, but when people do have questions and they're actually quite insightful and they're in, you're like, yeah, that is our biggest problem, I always feel like that's actually a really good sign that they kind of have gotten right to the heart of the matter. And to be able to do that, you actually have to have a pretty good understanding of the space and the competitive dynamics and the type of business and what really matters. And, you know, it actually shows a lot.

It's like you actually care, right?

Yeah, yeah, yeah. Well, both care but also can put all the parts together. So I, I thought that was important. The other thing for us, you know, we felt that there was a bigger opportunity around data streaming. And so the, you know, the kind of cookie-cutter thing was like, hey, this is the next like message queue. Like it'll be like Tibco. And we felt like, no, there's more here. Do real-time stream processing of the data. You could like build really a whole platform around this and the way databases have happened. So, you know, we wanted to try and find people who were signed up for the bigger thing rather than the smaller thing. And in theory, VCs always want the bigger thing, but then in practice, the cookie-cutter thing is much easier to see, right? And so, you know, so I think that was another factor for us as well, is how much did it seem like people kind of got the bigger opportunity and vision.

I see.

Did that naturally mean that you had a preference for VCs that had a more technical background, or did you also see that people that didn't have that but were able to ask really good questions?

Yeah, yeah, it's interesting. Eric, who we, who we went with from Benchmark, you know, I would say he's technical but not hyper-technical. You know, his background is not necessarily like software engineering. I would describe him as a very smart generalist. And so yeah, I don't know that it was so much, oh, as somebody who knows a lot about the building of software, so much as somebody who was willing to dive in and understand this space. You know, at that time, there was not going to be anybody who had like studied stream processing who was also a venture capitalist. That was like 7 computer science papers and, you know, whatever. It was just not a, it was not a thing that you would expect somebody to already know about. So, so, but yeah, that, that kind of willingness to really dive into an idea, I definitely think that. I think that's important. I think generally if you have somebody invested in the company, you want them to understand what it's about. And the reason for that is that's what will drive conviction that it can be successful. If you don't really understand something, then you may be excited when it's hot, but you will not be excited when it's not because you're going off of other people's judgment. And, you know, so I try to maintain that. In later fundraising, just the people who are willing to dive in and kind of get into it more, not necessarily the most technical people, but sometimes, yeah, you know, I think that's actually really valuable in as kind of a test of how they approach these things. And I think it was useful for us.

Nice, nice. We were talking to Josh Wills, and one thing that he said that stood out to me was that it was like, oh, engineers don't make good angel investors because they're very pessimistic. It's hard to see, right, like so many years down the road that what's the billion-dollar thing that you're trying to build. Did that— did you find that to be true in your experience?

Um, yeah, I actually, I probably had the opposite problem as an engineer. I was always like, I don't know how to describe it. I'm not like a super optimistic person. I think I heard myself do described as kind of grimly determined, but which is not quite the same. But the, um, you know, at least for me, I, I always feel like it's very clear the bigger thing that you could do, right? And especially in technology, you can easily prove to yourself that it could be done. There's nothing fun— I mean, maybe there's some fundamental problem it can't be done, but, but once you see that it can be done, like once you prove to yourself that the core problems are solvable then I feel like there's, you know, the only thing standing between you and that outcome is kind of the execution, you know, like, are you going to go do it? Right.

Right.

And so, yeah, I do think like engineering, operations, some areas of finance, you do tend to have more— call it pessimistic realists. You know, to some extent, these are people who their work is in direct contact with reality. So they see all the ways things go wrong. And but some people, I think, overindex on that. Versus seeing the thing that you would describe, like, for an angel investment, which is kind of, is there any fundamental reason why this couldn't go right? And I think for a lot of big technology projects, that's the thing that actually gets a team through it, is you see that end state. You're like, we can, we can definitely make this happen, right? We might fail, but there's nothing fundamental that will prevent it. It might be hard, but we can definitely do it. And I always felt that about these big systems engineering projects, any kind of big change or transformation in a company you're trying to do internally with technology, externally in terms of creating a startup or a new product. And I think it is a little bit of a test of personality, you know, whether people believe that you can kind of accomplish some larger change. But I think it's also a question of experience. What I've seen with people is people who've had a lot of failures tend to be kind of beaten down around it. But somebody who in their career, even if they're able to accomplish some smaller change, which led to them being, doing some larger change, maybe in an organization, which led to something bigger, you know, they tend to actually feel, no, you, you can do these big things. There's nothing preventing you. Worst, worst case, it doesn't work, but like the, there's nothing, there's no fundamental problem. And so if there's something very valuable that should happen, then, you know, you should go try. And so, you know, I always think for people in organizations, you want to find a way to try and build that experience for them, give them something that they can go do that's kind of within the, the scope of accomplishment that they may be able to achieve. And as they've done a few of those, they tend to get more ambitious in terms of what they want to go fix. And that, I think that's a really powerful thing to build. And to some extent, a company is very much like that, right? So, you know, that was definitely our take on Confluent. From first principles, we were like, hey, this kind of streaming model for data, it makes a lot more sense than batch and batch processing. To get from A to B is really hard. It's— there's a whole ecosystem around data warehousing that's very batch-oriented, right? You move the data in at the end of the day, you have a bunch of jobs that turn over, you know, you have to recreate all of that. And so when you sit down and you try and think of like, oh, how would you do it, it seems like, oh, that's just a this is really a lot of work. But then all you have to know is that it's— it would be much better if it worked that way, and there's no fundamental technical problem that limits it from happening. Once you know that, then you would say, well, okay, there's kind of an inevitability that somebody's going to do it. And so if we try and do this, you know, what would be the steps? Can we come up with a first step that we could do that would be valuable, right, that would actually have some use cases that people would use? And then if we If we had that, could we come up with a second step that would be a little bit further along that would also be valuable? And, you know, as you start down that thought process and you're like, oh yeah, this, this could absolutely be done. It's a long, it's a long journey, right? It's not going to be like a one-year project, um, you know, but it's definitely doable. So, so yeah, I think that, you know, I, I think that ability to do that, I think, is a very powerful thing, especially for senior people in an organization. You want them to have that you know, that, that kind of belief in the possibility of important changes, because most of the big valuable things in companies are that kind of change.

Going along that, like, do you— can you remember like the specific milestones in your career that led you, like, right, like the first few steps that finally led to you, uh, you know, quitting LinkedIn and then starting the company?

Yeah, yeah, I had— when I came to LinkedIn, I actually came to try and work more with data than on data systems. So my background had been in machine learning, and I was really interested in trying to find a place where I could do that in my job. And so, and I thought these, these social networks would be a really good place for that. And so I came to LinkedIn for that reason. So the, the first project I had at LinkedIn was this kind of news matching thing that, that we were trying to build, and it was, it was the most ridiculous, bad, you know, ridiculously bad approach.

Is that the biggest thing initially?

Unranked Yeah, it was worse than that. It was like an unranked keyword search based on your company name for news. So like everybody working on it was like, this is the dumbest idea. Like it's going to produce terrible results. A lot of company names are just words, right? So you can't, you can't do this. But anyhow, so we, we did that, you know. I had some other efforts that were in the, you know, the use of data, but at that time it was very hard to do that at LinkedIn for You know, a couple reasons. I think one, the infrastructure was very immature. Like LinkedIn was basically running on a very old centralized technology stack. So, you know, there was this social graph system that would get called like a zillion times to try and figure out the routes between people. You know, it had the whole social graph on one computer. And so, and there was replicas of that computer that could also be queried, but The number of edges in a graph, you know, grows quadratically with the number of people. Uh, and so there was like this continual engineering effort to try and shrink that data set. So it was like, okay, first let's write it in C++, and then it was like, well, let's try and compress it. So we were like— and the, the urgency of this was very high. So, you know, it was like, if this project is not completed LinkedIn will like go off the internet. And so that— I was on this larger team, that wasn't my project, but, um, you know, it was a kind of peer team. And, you know, there was just a lot of that where we wanted to do these very sophisticated things with data, but the underlying infrastructure was like, you know, just not the right thing. Yeah, not the right thing. And so, you know, you, you would spend like 99% of the time just trying to get or, you know, hook up to systems or come up with any way you could do anything. And then you would end up with these solutions in the end that were just this like dumb keyword search or something, 'cause that was the only thing that was possible. So over time, a lot of my efforts then switched to kind of infrastructure, and I liked it for two reasons. One, there was, you know, there was no product managers. And so you had more degrees of freedom in how you would solve the problem. And I was so frustrated from that keyword search thing where I was like, this is like not— this is not the right way to build a news relevance system. But nobody would listen. I was so frustrated about that that I was like, oh, this is better. You know, in the infrastructure world, you can just— the customers are internal, you can just, you know, come up with the right approach.

Um, you can reason with engineers on that front and not, not the product managers.

Yeah, yeah. And so, um, you know, and at that time, the technology stack at LinkedIn was extremely frustrating. So You know, so I was riding the train back and forth to work from San Francisco, and, you know, to do something on the train, I had a programming project which was to build a database. And I was interested in some of the distributed systems. There'd been a paper that came out of Amazon on DynamoDB, and it was like a key-value store. And so the first The first project was, oh, could I implement that? And if I did that, I thought, oh, maybe that can be a better replacement because we're trying to do everything on top of these Oracle databases that were really, really difficult to scale. And so I thought, hey, if you did that, you could have really fast access to data. You could do a lot of better frontend features. And at the time, everybody thought I was kind of nuts because it wasn't a thing that you would do to go build the database. But this is a very simple database. I thought, oh, there's no reason you couldn't do this.

So I want to pause you there for a second.

Yeah.

Exactly this part. Today, if someone goes and says, hey, I want to go write my database at any company or even friends, it's like friends don't let friends write databases, for example.

Yeah, it's a terrible idea. It's a terrible idea.

The other thing is, I think this, if you place this back in time, this was like somewhere between 2007, 2009, somewhere around that, I think.

Yeah, yeah, that's right.

And the open source technologies weren't as mature as they are today.

So yes, yeah, there was nothing. There was nothing.

Very few options to work with. Where did you get that conviction from to say, I'm going to go write this database?

Oh, well, you know, I'd read this DynamoDB paper. And then the underlying storage engines, there's these things like Berkeley DB, it was like an embeddable. Now everybody would use like RocksDB, but it was the equivalent at the time. And that was already there. And so I was like, you know, look, these systems are not that complicated. A lot of the data access patterns for live systems are pretty simple. And so if we could just scale that one thing, that would be pretty good. And so yeah, it's definitely true that you should not write your own database. I later learned more about why, because if you do that, then you also have to run your own database, which turns out to be the hard part. But the, uh, but you know, directionally it was right. Like LinkedIn needed more scalable data infrastructure, like, you know, buy, build, borrow, whatever. Like there needed to be some solution. Like the running indefinitely off Oracle was not viable. And so yeah, it was a bit of an internal battle to release the thing, but it was used internally at LinkedIn for a long time. It was open-sourced as a project, and you know, for a while there was this, you know, a whole storm of key-value stores and it was quite popular. But I think it was eventually displaced by the ones that had some company that was, you know, investing in it continuously. And this is one I had kind of moved on to other— yeah, that's right. Yeah, it's called Baltimore.

So we're talking—

it also, it also had a A weird name. That was the other trend.

I think a lot of projects at the time at LinkedIn for some reason had references to Harry Potter, like I think Voldemort.

Yeah, yeah, that was my naming scheme. Oh, I see.

So Kafka, I think, diverges from that in how—

Yeah, we ran out of good Harry Potter names. You know, the name has to like sound cool in a ridiculous way. And so, you know, some of the names were good. Like Voldemort.

And Azkaban, too, right?

Yeah, there was a Nimbus, which I think is like the broom or something. I don't know, there was a few of those, but then after a while we kind of ran out of cool Harry Potter names, and so then we switched to writers. So there was Franz Kafka, there was Camus, there was a few others. But, you know, these things turn over time, so I'm sure Kafka's probably the only one that exists now.

On naming itself, like, did you read these authors at the time and knew of— like, I didn't know of these authors. I learned about Franz Kafka through the project Kafka. So curious, like, how did that happen?

Yeah, yeah, I had read Kafka. You know, I'd done like a literature minor, almost. I think I did. It was like maybe one class short of it. So I'd read some Kafka. I thought it was interesting. And it's also sort of synonymous with complexity. So the idea was to unite a lot of the different data flows inside LinkedIn, and it was indeed a very kind of Kafka-esque setup. Like, we had all these different scripts and little hacks to get data out of one system and into another. And so this was meant to kind of unify those. And so it was It was cough gask. The problem was cough gask.

Oh, I see. So when you're talking to Felix, who actually introduced us over the email, he mentioned one legend where he was basically talking about Ubenes, which is the successor of Voldemort. And he was basically saying that legend has it that Jay Kreps wrote Voldemort on a Caltrain. So that part is—

Yeah.

Yeah, that's true. That's true. There's a second one that we heard about. And maybe you can help us confirm if that's true or not. That Kafka— you started writing Kafka over Christmas break. Is that true as well?

Yeah, yeah. So I, I, um, you know, both these projects were a little controversial internally. And, you know, obviously it wasn't just me. Like, I wrote some of the initial code, a bunch of people did the later Voldemort work. Um, the, you know, for Kafka it was tricky because of course the— each one of these data pipelines was owned by a different team. And so You know, my plan was to like replace all their projects with one thing, not in an aggressive way, but just because, you know, for a company like that that was growing very quickly, they were all breaking all the time. And then I owned some of the downstream infrastructure at the time, so I'd gotten the Hadoop setup going. And, you know, the things you can build in that kind of data lake area, they only work if the data is reliable. And both reliable in terms of like shows up and doesn't get lost, but also reliable in terms of like the, you know, what are the fields, the structure of the data, the schemas, if that's dependable. And, you know, that stuff, it was just utter chaos. And so everything downstream would just break every night in some different way. And, you know, my theory— so, so, and the traditional theory on how you do that is you just hire more and more kind of ETL engineers to fix, scrape data out of different systems, raise exceptions. So at one point in the budgeting process, you know, there was a column. So if you wanted to fund a project, it would be like, okay, how many engineers do you need? You know, how many PMs do you need? How many designers do you need? Okay, fine. Then they added like these data warehouse ETL people on the end. I was like, you know, Jesus, this is now like a tax on everything we do, we're gonna have some number of full-time humans just to scrape data. Like, this does not make sense. And so there's got to be some more, like, infrastructure solution that takes some of the pain out of it. Because the, the data lake was like one destination, but there was, you know, a search system and a whole set of analytics applications and other real-time systems, and just each one was its own little team that kind of building integrations. We had kind of one pipeline per data type or up to 5 pipelines for a single data type depending. So that was the idea, was to unify it, but it was controversial I think in part because nobody wanted to work on data pipelines. It just seemed like not a very cool area to work on. So I think for some of the engineering management in and so on. It was kind of like, not it, right, was the attitude. And then, you know, it also cut across all these other technologies that had full-time teams. So it wasn't political, it was just like, you know, I just thought practically we'd be better off if we could unify some of it. So yeah, anyhow, it was somewhat controversial, and people also thought that you couldn't build something that would work for all the different use cases. So we'd, you know, kind of have strong guarantees and be scalable but also be real-time. You know, there was some reason that we had a batch pipeline and we had log aggregation and we had change data capture and we had like 3 other things. So the, um, yeah, so the, the coding on the vacation was the, um, the way through that. I was like, well, if, you know, if I show you we can do it, then you can't say we can't do it, right?

So it's like, let me prove it works.

Yeah, so there was a— so we, we wrote like a very early version of it that was kind of a, you know, it was like a bunch of individual single node instances and but very efficient. And that actually, you know, it ended up getting released. It was part of the newsfeed early on, but it was, it was quite, quite buggy. You know, I think we— and then we, you know, got better after that as other people started working on it.

Like you're saying, this is like a huge undertaking and you mentioned manager. Oh yeah, was DJ Patel your manager at this point?

No, he was on the product side. So yeah, it was a different group of, you know, there was a whole set of different people, you know, involved in different ways. I just think nobody was very excited about data pipelines, understandably so.

How did you get buy-ins, like, from the manager, from management?

Ah, the prototype, you know, prototype solves all problems, right? Like, if you have a prototype, everybody's like, well, it could almost exists, we should finish it. So, and then I think, you know, it got rolled out for some of the, you know, user activity, kind of real-time logging stuff. And then from there, I think it spread over time to some other use cases. But the— but yeah, it was, you know, it was a challenge. It was— anything that cuts across the natural organizational structure in a company is always hard. You know, from a management point of view, that's what you're always trying to do is actually Design the org structure to make the problem you want to be solved easy. Oh yeah, but whenever you do that, yeah, that's right. Whenever you do that, you make some other problem hard. And so, you know, it's a little bit like the, like, you know, sharding or partitioning of data in a distributed system. Like some queries will be fast because they're all within one partition. Some queries are going to be slow because they cut across, right? So it's a little bit the same with the org design.

Stuff.

So like, when you were working on this prototype on the Caltrain, um, yeah, this was a Christmas vacation.

Sorry.

Yeah, this is a, this is a, I, it was a Christmas vacation, plus I think I, at one point I flew out to see my grandma. Uh, no, no, I guess it was a wedding that was in Pittsburgh, and so I was on the plane and staying, I was staying with my grandmother. Um, my cousin was getting married or something, so did some work on it there. Eventually, eventually got to something that, you know, mostly worked.

So I imagine, uh, like a lot of people would try to start something, right, when they are feeling inspired by like a paper or something else, but then they inevitably gave up, right, like after they hit the second or the third roadblock. Like, for you, there must have been some kind of reward function, right, where as you were going through it, you're like, oh man, I'm like learning so much, or what was that reward function for you?

Yeah, you know, I think it was a combination of things. Like, I like computer programming. So, you know, it was just fun to work on. Like, there was interesting problems to try and design the system, and then, you know, once you're kind of working on it, you're in some rhythm, you know, it's interesting to work on. So I enjoyed doing it as kind of a spare time project, and that was my way of kind of learning and keeping the skills sharp at various— like, at the time I was doing Kafka, I think I had more of a— I can't remember if I was an engineering manager or architect at that point, but, you know, basically I wasn't writing as much code. So this is a way of like not becoming totally a pointy-haired, um, you know, paper pusher. Yeah, yeah. So, um, you know, so I enjoyed it from that point of view. And then, you know, I think I'm just extremely stubborn. So then once, you know, once I was convinced that this solved a big problem, you know, I was just going to keep pushing on it till, you know, till I, I hard failed, which, you know, hadn't happened yet. So So I just kept pushing. You know, I think I, in retrospect, at that point in my career, I think I was a little, you know, I think I was a little controversial internally because I was always pushing on these things in different ways. You know, I think internally people thought I was a little bit of a cowboy, but, you know, it was well-intentioned, so hopefully there's no hard feelings.

Looking back, like, what do you change the way you approach some of this?

Um, you know, I got better at communication over time. You know, I think, uh, probably like a lot of engineers early on, you know, I would just try and like say things that were true.

Um, and it rubs people the wrong way. Why? I agree with you.

No, I, I think that's good. Like, you want to say things that are true. That's not wrong. Um, but that was pretty much the extent. So I would just be like, you know, Wrong. And then I realized, you know, I think managing people is a helpful exercise because you're trying to get people to do stuff. And you realize like, okay, you know, what do I need to do? I need to really understand the world from their point of view. And then when I'm speaking, my, you know, it's not sufficient to just say things that are true. That's like necessary, but it's not enough. I need to also say things that are going to change what they think. And so if I want to change what they think, I have to really understand how they think and then explain it in a way that makes sense in their world and is likely to produce that outcome. And you see this a lot where you actually— it's common among engineers. Their communication is actively provoking the opposite of what they want to happen. And so, you know, the solution to that is not to become, you know, totally mealy-mouthed where you just don't say anything, but the solution is to, you know, think a little bit from other people's point of view. You know, what's going to change their mind and what's going to change how they think. And I think that's a very important skill, you know, for communication. I think it's one of the most powerful things broadly. You know, if you want to change, if you want to get people to adopt some new technology, if you want to change how an engineering team works, if you want You want to start a company. You know, ultimately this is some idea that you want to propagate. So you got to think a lot about that. And so I think that was very, it was a valuable exercise for me. You know, it was definitely immaturity on my part, uh, early in my career. Um, and, uh, it was, it was a good evolution. Now I don't think, I don't think what that means is just to become, you know, like I said, mealy-mouthed. My, my joke internally is that, Engineers say there's a problem when there's a problem, and then at the director level, people say, "Oh, we have a challenge." Senior director level, people say, "Oh, we have an opportunity." And then at the C-level, people say, "We have a big problem." So it goes full circle, right? And I don't think you have to be mealy-mouthed. I think you say there's a problem, but the— But I think being aware of what, you know, what is the impact I'm trying to have and getting good at that, that's a skill where you, uh, that you develop over time. And, um, but only if you try. And then, you know, that's the only way you can have impact on trying to get a large group of people to do something differently is by developing that. If you don't do that, you know, you're just kind of shouting into the wind.

Yeah, yeah, uh, totally agree with that. And this is something I noticed in a lot of senior engineers too, where takes a little more intentionality, takes a little more patience than needed, or than people think is necessary. And you see when some people, when they go into, well, so-called conflict where two teams are basically screaming into the wind, someone comes in and can provide that clarity and still achieve the desired result where the stuff that is wrong gets fixed, but there's a way to put it in words. And it's not a common skill set, though I agree it's one of the most important ones for engineers.

Yeah, yeah, yeah. It's, you know, I think that kind of really building empathy is an important skill, I think, for, um, you know, management, leadership, any of those things, um, which is, you know, ultimately just comes down to trying to figure out where, where people are coming from.

So, and, uh, yeah, one aspect of this is like, uh, when you ask people who are really good communicators, um, one thing that people often say is like, it's practice. You do more of it and then you get better at it. And initially no one is perfect. So have there been resources apart from trying it a whole lot that have helped you in improving this?

Yeah, I think that's accurate. I mean, I actually think the problem with communication is people don't like to work on it. And so, you know, there are different people think of different things as work, and people don't like to do things that they don't think are important work. And so you find this a lot in technology. So if you want, you know, if you want a product or an idea to be adopted, you have to really think about how to communicate that idea. And, um, you know, it turns out for these kind of enterprise products, that has a word, it's called product marketing. And And yet a lot of engineers who actually care a great deal about building things that people use, they actually are quite allergic to product marketing. They think it's awful, right? There should be no— and yet it's actually quite hard to convey things. And, you know, if you think— I always think about it like, look, you know, people are very busy, you know, so they don't have a lot of time for new thoughts and ideas. So if you want to get something through, you have to really boil it down. You can think of it almost like a kind of compression. So it's very, you know, compression is very CPU intensive. If you want to compress an idea, you have to really think about how to convey it. What is it that people understand that's like that thing? You know, how can I tell you about it in a way that's, um, brief and compelling, but also accurate, like actually provokes the right understanding on your side. And if I can do that, that's actually quite valuable. And many of the biggest and most successful companies have been built on an idea that's ultimately relatively short. It's like actually easy to convey, like a picture of the world like that. And so if you can find it, you know, it may not be easy. Um, you know, I think it's a really valuable thing. I think it's a valuable thing if you work inside a company and you're trying to accomplish something. If you can boil down why that matters, that idea can spread a lot more effectively. Um, and yet for most people, they would not be willing to spend 2 hours thinking about that, uh, problem, even though they would agree that it's very important. So I think some of it just comes down to what do we think is work? Like, what are we willing to spend time on?

That's super fascinating. Like, um, okay, make it product practical. Like, for Micra Engineers, like, what do you think are like the most underrated, or like the work that adds a lot of value but people like don't perceive it as like real work?

For engineers, I typically think that communication side is, is lacking. You know, I, I typically think senior engineers can have a lot of impact just by explaining what they've done well and, or what they want to do, or what they want other people to do in a way that's actually very compelling. You know, the same with speaking, etc. A lot of people develop this, but, you know, the willingness to do that I think is a big magnifier of impact. You know, it was interesting when I was at LinkedIn, I was a mentor to a number of people, right? So you get paired up with somebody and you try and, and one of the things I realized is like, oh, everybody, you know, they don't actually really want mentoring, they actually just want to get promoted.

Oh yeah.

How do I get promoted? Basically.

Worst ever strategy.

Yeah, I have one question about, you know, that I want out of my mentor, how do I get promoted? I was like, yeah, I actually don't think it's that hard, right? If you, if you just consistently do good work and then you explain to people what you've done in a way that they can understand and why it's valuable, you know, and you try and increase the scope of that value creation over time, eventually they're going to promote you. You know, maybe it's 6 months later than it should have been, but like, it doesn't really matter as long as that line is going up. And it was interesting, like, people don't think about that underlying value nearly as much as they think about the the path to promotion, and so I was like, well, you know, I thought the formula was kind of simple. It was like, do something valuable, communicate effectively around it, that's kind of all you need to do. Let's spend the time on the valuable thing. And yet most people want to spend a lot of time on the mechanics of promotion, which, you know, it seems like is kind of an inevitability if you have somebody in an organization who's just consistently producing a ton of results. You know, whether the process is perfect or not, typically they end up, you know, getting promoted over time just because what they're doing is bigger and bigger.

What you're saying is like promotion is kind of a byproduct of what you do, not the goal in itself.

Yeah, yeah. It's like a lagging, you know, it's a lagging measurement of value, not a leading, you know, and I think many people see it the other way in engineering. That is like, oh, I should get promoted so I can do bigger projects.

That's right. But your earlier point is kind of like the mechanism is the more interesting work, right? Versus the actual value creation.

Yeah, yeah, well, it makes sense. I think what people actually want is just the output data.

So many blog posts around how to get to this level so-and-so, do X, Y, and Z. Anyhow, so talking about Kafka itself, so you had this initial version. By the way, what did this prototype look like at the time? Like, what did you have all the—

Yeah, the early prototype was, you know, it was more like a single server Kafka that you had to have like a load balancer in front of. And then we, by the first kind of multi-use case version inside LinkedIn, we filled it out. And at that time, Jun and Neha had joined the effort, so there was a full team of 3 working on it. And at that point, it was kind of a quasi-distributed system, at least, that would kind of scale out and handle a bunch of use cases. So depending on when you count that first version, it was actually, by that point, it was actually quite similar in terms of the overall style, what it enabled. Not nearly as sophisticated or good as it is now, but the basic idea was there, that kind of core abstraction was the same.

I see. And in this case, did you always intend for it to be open source, or did that just happen by chance?

Yeah, yeah, it was built to be open source from the beginning. Um, the, you know, that was kind of our strategy for those early infrastructure projects. It was, it's really hard to get people to work on internal infrastructure because it kind of, you know, it's some nameless layer inside the company that people only know about when it breaks. And so, you know, this turns the things into more products. And so my, my theory was, oh, you know, we can hire better people and they'll do a better job. Because if you think about something like a product then you're kind of like, well, is my product good, right? Like, do people like it? You know, does it have a good interface? Is there documentation? So like internal infrastructure, nobody will write any documentation, but if they open source it, it's like, oh yeah, they'll document it. So that was the theory, and you know, it was mostly successful. Not all the things that we did got a ton of adoption. You know, Kafka was early on extremely unsuccessful as an open source project until we kind of went and did some talks around it and tried to get some adoption in the rest of tech. But yeah, it was intended to be open source early on because we thought, oh, this is, you know, this is, you know, I guess our feeling at the time was, hey, everything is kind of moving to a world of these, you know, elastic distributed systems. So all the things that we had that were single server, you know, we'll kind of move to distributed, you know, but a lot of the work at that time was these databases, right? So it was different types of key-value stores, data lake, whatever, some kind of storage system of some sort, storage inquiry. And, you know, at least the feeling for me was, you know, hey, a more original and interesting problem is the kind of movement or flow of data. So not like data storage, but, you know, kind of data, data in motion, not data at rest. And, you know, so the idea was really, you know, trying to explore that and do that right. And having built one of, you know, whatever, 50 key-value stores that all became open source at the same time, I thought, oh, you know, it's better to work in some area where nobody else is doing it because, you know, it'll, you'll be more likely to, you know, kind of build something that people use. And, and I think that was true. Unfortunately, it was a little too true, uh, you know, being an area that nobody was interested in at the time. We initially didn't get any interest or adoption. Option, but then, you know, I think as people realized, you know, how this could help with their kind of internal data flow problems, it became much more popular.

Can you speak a little bit more about that in terms of how you got more adoption, both like internally versus externally, like with the talks and things like you mentioned?

Yeah, we, you know, so the challenge for us was, you know, our vision was something that was a lot more than just like a simple message queue, but it was really hard to figure out how to explain it to people, right? So it was kind of back to that product marketing thing. You know, I think we didn't really— we were like, well, what is it at the end of the day? Is it like a log? Is it something else? So the initial effort was we kind of went around and did some talks in Silicon Valley. I think I did a talk at like Airbnb and a few other places, and those went well. People were excited about it. As we were thinking, you know, we're kind of contemplating, oh, maybe there could be some kind of company around this, and I wrote a blog post called The Log. It was this really long thing, and the idea was, well, if we could get people excited about the bigger picture vision, you know, maybe there is some viability to this project, and if not, maybe not. So I spent a lot of time on it. Yeah, I probably spent like 3 full days just writing this long blog post that was like every thought in this area collected in one place. And, and it was really popular. So then we were like, oh, okay, like people are excited about this. We just have to explain it to them. Even if we don't have a pithy one-sentence explanation yet, at least we have a 20-page explanation, uh, you know, that, that people can get interested in. And, um, you know, kind of following that, I think we decided, oh, you know, if we, if we really want to take the project all the way, you know, it probably makes sense to go start a company. Like, at a certain point, LinkedIn only needs an internal infrastructure layer to be so good. You know, like maybe there's 6 people working on this, you know, maybe it goes to 10 people, but there's not, there's not a world where LinkedIn needs to build this the way all the other companies need it to be because it's just not their, it's not their problem. And so if we kind of want to finish the job, you know, we're probably not going to be able to do that, you know, working in the, you know, in the side corner of a professional network software thing. We got to go turn it into something that directly goes out, figures out the problems that people have using this and adopting it, and can start to build out some of the layers around it that kind of complete the product. And so that was the origin, and that was when we kind of decided to go start Complement.

So how did you convince others at this point? Or did you and Neha, for example, all of you co-founded the company? Were you all convinced at the same time, or did it require some—

I think it was, yeah, I think we were all kind of thinking, you know, I definitely been thinking about it. I think Neha actually originally proposed the idea across the, you know, the, the three of us. Um, but I think it was kind of in the air, you know, we were hearing from a lot of companies that they were interested in this, you know, I think at one point I forget what it was, but it was, you know, some media or TV company was like trying to adopt the open source. They were like, okay, look, like, we just need to pay you to do these things. And we were like, well, you can't, you know, that's not really available. Like, we're, you know, like, you can do the things in open source, but like, we're not available to help you. So, but we were like, well, okay, this is interesting. Like, at least You know, it does look like this is applicable outside of tech companies. And so that was a big question for us, you know, was how broadly applicable is this? Is this something where it's like, yeah, okay, all the parts of the tech company that has to happen in real time, but does that make sense elsewhere? Like, would a bank work that way? And it turns out, yeah, like these big, you know, other companies outside of tech are actually bigger, more sprawling, more business units, more systems that have been built over more time, you know, more legacy.

Kafka-esque.

Yeah, yeah. So how it plugs together in a sense is even more valuable. Um, and so, so yeah, in fact the need there was probably, you know, even better. And so, you know, as we saw that, we were like, oh, this is, you know, there's this fundamental transition from thinking of data as, you know, kind of almost static tables, right, where the fundamental thing in a database is how do I lock the data in place, run a query, and then kind of unlock it, or, you know, some mechanism to pretend that things aren't changing. And then, you know, that's most exaggerated in a data warehouse where you're literally just like copying in a snapshot or something. And, you know, that, that, that's a very different model from the kind of flow, you know, flow of data across across the company. And so, okay, there's a, you know, there's a very fundamental change that's going to happen because ultimately, you know, companies are out there in reality doing business in some way, you know, and that's a very continuous activity. Um, you know, so like a LinkedIn, it's available all around the world. There's things happening all the time. It doesn't make sense that it, like, you know, we at 12 12:00 PM Pacific, we like copy a bunch of data into some data lake and run our most sophisticated processing. Like, there's nothing magical about 12:00. That was just a convenient time to schedule it, right? You know, the natural way of doing things would be some kind of more continuous stream processing. And so, you know, it seems like, okay, inevitably that change has to happen because the problem in the world doesn't match the underlying infrastructure layer. And there's some valuable initial step that's applicable to a bunch of companies today, right, that we have. So if you put those two things together, you kind of have a valuable first step with a really valuable next 10 steps, you know, that, that could be a business in some form. I don't, I don't think we knew exactly what form, but the, uh, you know, at least it seemed like a, you know, good direction to start walking.

And, uh, I remember reading that blog was the first time I learned about Kafka, and it was extremely valuable. And that was like, I was just starting out in engineering and just reading, and a bunch of things just went way over my head at the time. It's like, why stuff— is this really a problem? But like, as time went on, I learned the value of it. But when you're starting this company, and let's say you're pitching to investors, as you mentioned, what did the evolution of that pitch look like at that point? Or what was that pathy, uh, one-line or two-line statement that you could share?

Yeah, yeah, that was, um, that was definitely the evolution of my, uh, understanding of how venture capital works. So the, the initial pitch was, um, you know, these are the trends in data and infrastructure. It was like, okay, like, look, there's more and more complicated specialized systems and applications, right? So there's a ton of sprawl and then the volume of data and what we consider data is growing, and then the needs from customers, like they want better experiences, or the digital part of the business is growing. So effectively there's like more data, you have to use it in more sophisticated ways across the company, so all the parts of the company need to— going to need to kind of connect together, and there's going to be a need for infrastructure for that. And then sort of at the end we were like, oh yeah, you know, and we're a team that is done this at very large scale. And there's open source, which is like, you know, very broadly adopted that people are building like big, important production systems about. And so then what I learned was people mostly like tuned out the whole first part of theoretically why this would definitely happen and kind of ignored, you know, the first, you know, whatever it was, 25 minutes of a 30-minute pitch. And then at the very end, they were like, oh, people are using this. And It's recognizable companies doing really important things. So then I was like, aha, take the side about the traction, put that first, then explain the theory. Then they're very interested in the theory. But the first thing is actually just like, yeah, there's a thing happening in the world and I can prove it. And that, you know, requires heavy involvement and investment from companies, and that means there's a good chance they'll buy a product in that space. So, and indeed, it's actually a smart way of thinking. It's very easy to come up with clever ideas. It's very hard to produce actual results. And you see that, I think it's true inside of a company as well. The kind of actual results speak a lot louder than theory. And once you have the results, then you're very interested in the theory. But, you know, as long as it's purely theoretical, you know, we can all remain skeptical or have different ideas. Once you've kind of shown it, then it's a little different. And, you know, one of the results that's the hardest to get is always, you know, something that people actually want. That, you know, it's very hard to take time out of your day to go adopt some new software platform and build around it. Yeah, that's a big undertaking. And so if you're choosing to do that, there must be something that's very valuable to you that you would take that risk, you know, invest that time. And so that's actually a very good signal.

And Kafka in this case, as an idea, was already validated in a way with like open source adoption. And like, again, when you were building Kafka originally, you were trying to solve this problem of like instead of data at rest, putting data into motion. Business is not something that many engineers think about very, uh, at least organically. It doesn't come to many of us naturally. So what did that look like in terms of developing a business around it? Like, how did you go about figuring out, this is how we can charge people for it, this is how we can pitch, this is how much we should raise?

Yeah, yeah, it was an, it was an interesting, um, thing, you know, as we were starting, you know, at that time there were some open source companies, they weren't very successful or very well thought of Cloud Era was a company, but it had kind of grown to size, but it had a number of challenges and kind of skeptics, which actually increased over time. But at the time we were pitching it, that was the state of things. So that was kind of the only model you could point to. There are some much older things like Red Hat, but it didn't— it wasn't really a replicable model. And so, yeah, that was the question, is, hey, is there a product here that can be successful, and we felt like, yeah, there is. The dilemma for us early on was whether we wanted to start with a software product or a cloud offering. Indeed, it was a very big dilemma because at that time, if you looked at where money was spent, it's probably 90% on-premise and maybe 10% in the cloud, right? And if you looked at the size of the data streaming market, It was basically like $0, approximately, rounds to zero. So we were like, well, you know, on one hand, like, you know, a cloud offering is a really great way to deliver this, but on the other hand, it's not really clear that you can take a really early market that's gonna grow and just build in 10%. Like, first of all, somebody else will do the other 90%. Second of all, it's just not enough to get going and start a company. It was a difficult time in that respect because I felt, at least for us, we had to do both early on, and that's very challenging to try and have two models or two products. It ended up being very valuable because a lot of the companies now are now in the same situation where they're actually spread across their own data centers and cloud instances and so on. That ability to kind of span all of that ended up being a really big deal for customers that have that problem. But for the initial part of the company, it was quite challenging to be able to build both products in a way that was good. But yeah, the initial challenge for Confluent was, yeah, go figure out how to sell it, because we were actually kind of starting a little bit ahead. We had at least an open source offering that was pretty successful. Building a very light product around that was not super hard. So yeah, the first challenge for me was like, okay, go figure out how to build an enterprise go-to-market, which I knew nothing about. And, you know, I just had the unluckiness for that task to fall on me because, you know, the other two co-founders had proper technical jobs and, you know, I was the CEO, so I was like, well, okay, you go do it, right? And, you know, and indeed I was quite clueless. Like, even as we were pitching the company, you know, I think some investor asked me me like, oh, what, you know, what's your strategy with services? And, you know, what they meant was consulting, but I actually had no idea. I was like, I thought they meant web services or something. And I was like, yeah, you know, we'll definitely build some web services, but like, I don't think that's the first thing we're going to start with. And the guy looked at me like I was, you know, crazy. And, but I wasn't crazy, I was just completely ignorant. Like, I'd never worked in an enterprise company. I had no idea that, you know, there was a whole consulting services strategy that would go along with adoption. Um, and you know, so that was just the state of things. But you know, the good news was we had people who were using the open source that we could go talk to. And so, you know, the nice thing about enterprise products is you can actually just ask the customer, like, what would you buy? And you know, you got to be thoughtful about, well, you know, everybody tells you nice things, but would they really do it? And so yeah, just at that point, it was kind of a blunder through it. And so we, you know, we hired some early sales reps. You know, I hired somebody who was just kind of a generalist and had been in different roles in enterprise companies and could help build that out and run that part of the organization. And, uh, so yeah, but the, yeah, the initial, um, you know, customers I just kind of went and found, uh, and then as we had sales reps that helped to actually get deals closed, because it turned out I was much better at the sales pitch than the, you know, getting the final money out of them at the end, uh, which turns out to be a skill in its own right. Uh, so the, um, yeah, so, so, you know, I was actually very close to the kind of early sales and marketing effort for that reason, and that's the best way to learn something. You know, I think in a large company, the go-to-market ends up quite complex over time in terms of different segments and strategies and teams and how it all fits together. But in an early startup, I mean, it's just very simple. It's, you know, it's like if you want to learn how, um, you know, transportation works, you want to look at a bicycle, not a, you know, combustion engine automobile, right? Because it's just one is so much simpler than the other. You're just like, oh yeah, you turn the pedal and it just moves, the wheel goes, right? And it's kind of like that. It's like, okay, like, what, what are, you know, what do we need to do? You know, we need to find a way to tell people about the thing we have and why it's good, and then we need to have a way to allow them to give us money to get the good thing and then help them use the good thing. And so it's not, you know, it's not a rocket science problem. You just have to, you know, figure out how you're going to accomplish that with a relatively small group of people in an early company, and then how it will kind of grow and evolve over time. So yeah, you know, I've been relatively close to that side of the business since, but you know, of course we brought in really amazing people who did all the hard stuff once we were past those very early stages.

As you were picking up these like new skills in a very pressurized and like rabid manner, was there any sort of lessons that you've learned from engineering that you were able to apply to these new things that you were learning?

Yeah, absolutely. I, um, you know, it's, it's interesting. I think, um, engineering is probably the least useful skill in a lot of ways, um, for becoming a CEO. But there's some parts of it that are really good. So why is it the least useful? Well, it's like you actually don't need the CEO to write a lot of code. Uh, in fact, it's not that helpful if the CEO is trying to write a lot of code because they're, they're usually not around to like do all the work through to production and support it, right? At best, do they kind of fling something from some night, you know, coding session, uh, and hope somebody like fixes it and makes it work, uh, which nobody really wants to happen. And so, um, you know, so the core skill set is actually not a useful thing. You know what CEOs do? They do a lot of communication. They do, you know, you might incidentally do some of that as an engineer, but of course you do some of that in every role in the company. So, but there are some things which are good. Um, you know, one is engineers have to learn a lot of stuff, so you're kind of just on this treadmill of learning new stuff. And usually engineers have high confidence in their ability to learn something. And so if there's something new, they're just, you know, they'll just read about it. And, you know, there's a belief that you can do that. I think that's really important for a CEO because actually, you know, it's one of a couple of roles that sits across. The CFO is this as well, where you sit across every function, you have to kind of understand how they work, and you probably won't understand it as well as somebody who's run that function through their whole career, but you need to understand it well enough to know, you know, if that's going well, you know, if you've hired the right person to do it, you know, to have an opinion on what's needed there to kind of help it, guide it, and integrate it into the the rest of the company. So you need to understand it pretty well. And so if you're not, you know, if you're not going to spend a lot of your time kind of learning and thinking about things outside of what you already know, it's not going to go well. And then I think as a company gets bigger, you know, I do think that ultimately an organization is a kind of system. And so that kind of system thinking and design, you know, kind of modularity, how, you know, How does this module— is this module overly coupled to that module? You know, do we have a good API? You know, what's our observability story for knowing if this thing is working? You know, what, like, what's my feedback loop to, like, run this part of the business and know it's on track versus know it's not on track? Um, I think that style of thinking is, um, you know, extremely important in software engineering. And actually very much what you do for a larger company is definitely, you know, those same kind of things, which is okay, you know, you peer inside a module and try and get it in good shape and working. And then ideally you want to have, you know, what are the outputs of that that I can reason about? I know kind of what the input is, like what's the investment and what's coming back. If that's good and it seems like there's somebody we trust running it, then I can just kind of let it run. That's, that's the observability story. And if not, then, you know, maybe I need to get in there and do some profiling and see what's going on. So yeah, I think that, I think that's, you know, that's good. And then I think this is true of engineers generally, it may be even particularly true of these distributed systems people. It's actually a relatively gritty profession where you spend most of your time on things that are not working. And I think that's true of management roles as well. You know, you're— you spend most of your time fixing problems and trying to get stuff to work, and then once it works, it's like, okay, right, on to the next thing. And so, uh, you know, there's, there's some analogy there as well. And so, you know, if you look at people who run tech companies, there's actually a good percentage of engineers, even though the, the, the actual core skill set is not, is not that useful And I think maybe it's for those reasons.

That's super fascinating. On the learning part, right, I remember the memes about frontend engineers having to learn a new framework every 2 years.

Yeah.

But now I'm like, oh, maybe that's not a bad thing. I mean, you just keep your skill set of how to learn new things in check. So that's really powerful. Thanks.

So it's not every day we speak with the CEO of a public company and you took Confluent public, started with like Kafka as a project to a startup to now a public company at this point. So great achievements all around. What does a typical day in your life look like? So I'll pose two questions and you can take it in either direction. So like that part. The other part is, as you were describing all of this, where having analogies for distributed systems to basically having well-functioning orgs, have you developed certain practices to stay productive? And kind of get ahead of the problem a little bit and still keep some of your time free to think? Because I'm sure there are a million things wanting your attention.

Yeah, um, you know, the, the job for a CEO is— there's different ways to do it, right? So, um, you know, I've seen kind of two models. One is, you know, people effectively, you know, kind of manage at a relatively high level and just kind of hold accountability across different organizations. And then put, you know, a lot of effort into being the public face, you know, of the company externally, and the face of the company internally, you know, the integration with different stakeholders, which tends to be the employees, customers, and the investors. So, you know, one model is you kind of spread yourself across, and that's the full-time part. The other model, I think, is a little bit more T-shaped, which is, okay, you try and take that horizontal bit and actually get that down to about 40% of the job so that, you know, in steady state, your day would be about half free. And then the other 50% is just very topical. So at any given point, there's some big problems or some big opportunity, right? So it can be something forward-looking. What's the next thing we're going to do? Or what's the horrific tragedy that is unfolding that we need to solve. And, um, and then you dedicate about half the time to that. Usually it's like 1 or 2 things that can slot into that, and you have to just be very disciplined. And so I definitely follow the second model, which is I ideally want to push down, you know, the steady-state commitments to a reasonable amount, and then the, um, you know, the remainder of time is relatively kind of flex. And then it, you know, I'll have a focus area at any given time that I'm kind of more deeply involved in. And that could be, that could be an area where I'm trying to hire a leader for the team. It could be an area that I don't think is on track, or it could be an area where there's some big opportunity or unlock. They're like, oh, if we can get this right, um, you know, that would be great. And then I'll try and dive in, uh, actually at a relatively low level and just really understand what's going on there as best I can. And, you know, try and help get it into good shape.

These like bigger, um, these blocks, like, do they just kind of come up or do you have like a system to kind of track them so that once you do have free time, you're like— sorry, flexible time— you start digging into one?

Um, yeah, yeah, the, you know, I always feel like time management is just a process, you know. I feel like you're out in the jungle with like a machete and the overgrowth that you grows and it just consumes everything, the vines and the weeds and whatever, and then you kind of hack it back, but the jungle just keeps growing. And so, you know, so I think it's a little bit like that where effectively the day-to-day activities just kind of eat up more and more because like things creep and there's always another one-on-one and another check-in and another something that gets proposed, and then you just kind of hack it back and you're like, okay, You know, got that done and then it grows back again. So I don't know if that's a methodology, but yeah, it's, you know, it's just calendar management. Like, where, you know, looking at the calendar, where am I actually spending time? Does that actually match my priorities? And is all my time accounted for? If all my time is accounted for, then there's effectively no flex time for whatever the most important thing is, which seems wrong. It seems like there should be some time for the most important thing. So then, you know, then the machete comes.

Well, speaking of time, we are reaching towards the end of our conversation while we have a bunch of questions to ask, but we'll save them for another time. And thank you so much for taking the time from your busy schedule. But before we let you go, one last question. In terms of putting data in motion, I think you have been successful at that with the open source project, with Confluent as a company where Kafka has become this de facto choice. People don't ask, what should I use if I want to move data around? It's like, oh, use Kafka. Like, that's just the option. What is something that you're looking forward to next?

Yeah, I, I think it's still early in the journey, you know, that, um, in what we wanted to achieve, we always felt like Kafka was kind of like step one, right? So you have to have some stream that people agree on, that everybody will write the data out to and read the data from, but that's not the end of the stack, right? So if you think about what are the needs for that, you wanna be able to do really rich, easy, real-time processing of data, that kind of stream processing and the reprocessing. And so the evolution of the ecosystem around that is at least as important in the product as that kind of core stream itself. To some extent, you want that core bit to just be solved. Like, it's like a utility. It's like the, you know, do the wires in the house conduct the electricity? Like, if they don't, that's the big problem. But if they do, it's not something you spend all day thinking about, right? Um, and so a lot of what the electricity goes to, that's kind of the exciting thing. So yeah, you know, so our goal is continue to build out that, you know, platform around data streaming and fill out those other pieces. Yeah, so we've been putting a lot of work into our offering of Flink. You know, this is a stream processing system. You know, typically for stream processing, it's been pretty ungainly to run these systems yourself. You know, it's relatively hard. The opportunity in the cloud is really take that away, like make something that's just very elastic where you kind of write the code, you write the query, it goes and runs. You know, I think there's really an opportunity to take stream processing and do it in a way that's kind of a generalization of the batch world. So, you know, I said, okay, a batch system, you kind of freeze the data in place and query at some point in time. You know, the opportunity in a stream processing system is, you know, you kind of process at a point in time, but as new data comes, the answer just keeps evolving, right? And that's a very powerful generalization that matches a lot of the activity and applications in a company and makes it vastly easier to build around streaming data. So that kind of growth of the platform that makes it really easy to connect into all the systems, really easy to process the data, really easy to kind of govern it in the large. This is something a lot of the companies that built the most around this have kind of built in-house, but our goal and a lot of the work we're doing now is really bring that out as part of the product. So you can just have something that's kind of very cost-effective, very easy to scale, just kind of solves this problem for you. So every company can have awesome infrastructure that everything plugs into around streaming and make that a really default way of working, a really obvious way that systems would interact and that applications would be built. That's as important as the kind of database world in companies. So that's the journey that's ahead. And to do that, of course, is well beyond Kafka. It's really the how all these parts come together into a unified product.

We look forward to what comes next, and Jay, thank you so much for joining us today. We had a great time speaking with you and learning from you. We have a ton of more questions to ask, so we do hope there is a second time and we do get to speak with you again in the future. But for today, thank you so much.

My pleasure. Thank you, guys.

Hey, thank you so much for listening listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello@softwaremisadventures.com. We would love to hear from you. Until next time, take care.