How to Make Good Moves with Jay Kreps of Confluent

Welcome to Inc.'s The Founders Project with Alexa von Tobel. I'm Alexa, the founder of Learn Best, author of New York Times bestselling book Financially Fearless and second book Financially Forward. I'm also the founder and managing partner of Inspired Capital, a venture firm focused on the entrepreneurs of the future. Each week we sit down with a top founder to share their story of guts, inspiration, and drive. Hi everybody, I'm your host Alexa von Tobel, and this week I'm excited for you to meet Jay Kreps, CEO and co-founder of Confluent, the foundational platform for data in motion. Confluent helps organizations harness the full power of continuously flowing data to innovate and win in the modern digital world. Jay started Confluent in 2014, raised over $450 million in venture capital, and brought the company public in early 2021. Prior to Confluent, Jay was the lead architect for data and infrastructure at LinkedIn. He is the initial developer on several open source projects, including Apache Kafka. With that, let's welcome Jay. Jay, first of all, so excited to have you on, and let's just start with the basics. Let's get to the beginning. What is Confluent in your own words?

Thanks for having me. Confluent is a company that makes a service and piece of software all around processing real-time streams of data. What does that mean? It's data that's not just sitting and stored, it's stuff that's about what's happening out in the world right now. What's selling in a big retailer, or the transactions in a bank, or activity of customers. And this helps companies that are building software, you know, software that's smarter and it's faster, that's more up to date, that has the right information, that can make smarter decisions. And this, this whole movement around, you know, real-time data, around data in motion, has become a big deal. It's kind of become part of the modern software stack.

What were the pain points that you were trying to solve building Kafka, and why did you decide to make it open source? Let's go there first.

Yeah, so I'd worked on a number of different kind of database and analytics areas inside the company, and I felt like what we were really doing was building a lot of ways of storing data. But a lot of our challenges was how do you get the data? How do you actually tap into the right thing at the right time? And that was a problem that hadn't received as much attention. So in the world of data, you know, you would see a huge amount of investment in databases and in storage layers. And this is areas where huge companies have been built and there's a ton of commercial investment. And, you know, really, if you went to any university in the country, if they have a computer science department, there's probably a bunch of people doing research on databases and how to make them smarter and better. But it really is the idea is, hey, you have a bunch of data sitting in one place. So like one of the metaphors is a data warehouse and it kind of is what it sounds like, right? Like you have some warehouse full of data everything's sitting there. But if you think about a company now, there's not just one thing with data, there's not just one application, there's not one place, there's, there's a lot of them. And all of these are kind of interacting and reacting all the time. And that was definitely the case at LinkedIn. So, you know, it's a very kind of rich digital experience with lots of different internal pieces of software that make all that happen. And to do it in a smart way, they all have to be able to key off each other and, and know what's happening. So if you know, a new user joins the site, that would kick off, you know, dozens of different pieces of software that would do their part, right? If something's shown in the newsfeed or there's, you know, some ad is displayed or anything like that, there's a whole set of things that have to happen in reaction to that. And I felt like that was where we were spending, you know, a lot of our struggle was this big kind of intertangled mess of software and how it plugged together. I think the thinking on this had been, well, that's not really an infrastructure problem. Like, that's not something we can build a general solution to. Instead, it's always going to be kind of bubble gum and duct tape. I think the idea we had was to really take a go at that and try and build an infrastructure platform that could be like databases, but not just about the storage of data, really about that kind of that flow and real-time reaction. And that, that was very popular and successful within LinkedIn. And then we released it as an open source project. To the larger world, you know, really at first to resounding silence, I would say. We'd done a number of open source things, and some of them were quite popular at the time. And of them, probably Kafka, which was the name of this software, you know, that was probably the least successful at first. No one had any idea what we were talking about. We were like, real-time data streams, whatever, and everybody kind of gave us a blank look. But, you know, it turned out that this was the same problem that a lot of the companies that were in Silicon Valley were having, which were they're trying to build these very rich integrated systems with lots of pieces of software, and how that all came together was actually a critical challenge. And so then this ended up being, you know, a key component in that next generation of company, the Netflixes and Ubers and Airbnbs, they all really built very heavily around this open source project, Kafka.

What was the aha moment when you realized that you were ready to leave LinkedIn and go pursue and build a company around your open source technology? What tipped?

I think the first one was really bringing together a lot of the ideas in databases and realizing that, hey, this would apply, you know, if you thought about data as something that just kind of was always flowing and going on forever, you could take a lot of the research that had been done and apply it there. And that that would be a really big deal that you could have something that was kind of like the central nervous system in a company that would have all the real-time kind of impulses of what's, what's occurring in the, in the organization. And of course, there's then a big jump from ought to be to will be. You know, we were concerned whether we could even, you know, make it a successful open source project in Silicon Valley. And then, you know, we start to see that succeed. And we started to talk to companies that were outside of Silicon Valley. There was even a funny story where, you know, it was like the CTO of a major media company was like calling us and we were working, you know, at LinkedIn and the social networking and we were saying, hey, we need to have like a more productized version of this. He was kind of giving this list of requirements and we're like, well, look, this is not how open source software works. Like we're working at this company, we can't just switch and do all this stuff that you need. But it was really clear that what they needed was very general and that that was something that was kind of breaking out into the world at large. And so then I think for us, it was a little bit of, hey, this could really happen. This could be a major part of the stack. That the next generation of software and companies is built around. And then, you know, maybe a little bit of FOMO as well, which is like, hey, this probably will happen, but if we don't do it, you know, probably somebody else will.

Can you walk us through how a customer uses Confluent today? Give us a sense of a major customer. I know you have them from Walmart to Expedia to Citigroup. What would an experience with Confluent look like?

Let me give you an idea of, how this would be used maybe in a big retailer. It used to be in retail that, you know, each store kind of had what they had. They would sell their goods. They would kind of periodically go report back to the mothership, like what's sold, order new things. But it was a very kind of slow process. And at any given time, you wouldn't know what was in all the stores all over. But, you know, in the modern world, if you think about how does a retailer work, It's very different, right? You have to have a digital side of the business where you actually know what's in all the stores all over. And you have to also be able to have a web interface, an application that can tell you the same thing. And they have to be able to tell you, hey, is this particular product, is it in the store down the street from your house? And that has to be right. It can't be the, you know, the information from last week. It has to be the information from right now. Otherwise you're gonna go down the street and not find it. Then I think increasingly companies are trying to figure out how to bring together the digital interaction and the real-world part of the business so that maybe you can buy it online and pick it up outside the store or something like that. To do that, you go from a world of very slow shipping of data around at the end of the week to something where you have to have very real-time interaction of all these systems and know exactly what's there and be able to make decisions and optimize. And so in a lot of these retailers, they've done a similar project, which is, you know, coming out of each store is kind of a stream of data about what's selling, all the sales that are taking place. And you could imagine that the kind of inventory process in a company is kind of keeping track of what's selling as it's selling, what's arriving as it's being shipped and unpacked, being able to track all that, and then having this kind of view and being able to optimize and do promotions and pricing and everything off of that. And so you could imagine, you know, in the world of data, there's this idea of batch processing. And a batch process is something that happens like periodically. It's, you know, the days of mainframe computers, you would submit the batch job and it would maybe run and produce some output and then not run again until you run it. And a lot of the world of data kind of worked in that way. And it's moving now to this kind of real-time continuous world where it's always working. So instead of like updating your inventory numbers at the end of the week, you would do this all the time. You would know what's on hand and be able to react to it. And that transition is very fundamental. It's kind of a, you know, a very core paradigm of how do you think about data? Do you think about it as something that's static, like it could be written out in a book, or do you think of it as something that's dynamic where it's always kind of changing and flowing? And adapting to that is kind of the underlying paradigm that we've built around. And so, you know, one of the ways of expressing that was we started to talk about that as data in motion, that it's kind of flowing, it's moving, it's changing versus data at rest, it's kind of sitting, it's being stored. And that's the kind of big underlying paradigm shift that Kafka was built around and Confluent Services are built around.

Can we talk a little bit about your predictions? If we fast forward 5 years, 10 years, you sit in the most unique spot of seeing where the world is heading. Tell us a little bit of what you see that's so obvious to you that probably isn't obvious to everybody that's listening.

There's this idea about how does new technology get deployed and rolled out. And one of the observations is that's a pretty long cycle. You, you know, maybe you get electricity, but it's not like the whole economy is just electrified overnight. That may take some decade to roll out. I think it's a similar thing with software. You know, the early use of software was this very productivity thing. You know, so it was really, you had an organization that was really about people and processes built around paper, where I send you this document, you send me that document, it goes to this person, it gets checked off. And a lot of the early software is just kind of automating that paper process, right? You have some office suite, you have some other applications that look a lot like filling out forms or paper, you have some workflow within the company that's kind of around sending the documents between people. And that's like the early state of digitization, but it's not really what a business in kind of its natural digital form would look like. It's just a way of translating the thing that was there before onto computers. And I think what's happening now is we're starting to see, hey, what do these businesses look like when you're kind of building for the digital world first? And that's actually quite different in many of these cases. And this idea of data in motion that you would actually have all the different applications connected together is kind of a key part. And you can think about it just in terms of the path to adoption in a company. Maybe at some point there's really no computers or digitization, and then you get these little applications bit by bit that are in this part of the org or this part of the org and are just kind of optimizing that bit or this other bit. But over time, it has to come together so that the operations of the company is holistic, so that the, you know, the different things all trigger each other. It has to come together with the interaction with the customer themselves. And so you go from having these little pools of software to having something where the actual process that the business does is actually implemented end-to-end in software. There's a software version of all the stuff. And it doesn't mean there's not people doing really important parts. That's true in every company and remains the most important part of an organization. But it does mean that that whole overall activity has some representation in the digital world. And that kind of coming together, this idea that in some sense, companies are becoming software. I think that's kind of right at the heart of a lot of this transition to data in motion is we're solving that problem of, hey, like just physically, how do you connect up all these parts of software and turn it into one thing that all works together in a holistic way? And that plays out in really different ways in different types of companies. Like I talked a little bit about, hey, what's happening in retail, right? But it's a totally different thing in banks where of course, you know, for kind of retail banking, they are moving to a very hybrid interaction where there's still branches and people to talk to, but a lot of it is now online through different channels. And the way that that works is, is much more personalized and responsive. And it's, you know, it's a very similar kind of underlying technology trend, but of course, in a totally different business. If you look at something like manufacturing, of course, you know, very similar thing where now there's a very real-time view of a lot of the supply chain and what comes together. But of course, totally different business. And so I think this is kind of an overall trend you can see happening.

Can you teach us like, what is your playbook? Or what do you think is the most powerful way to actually build a business with your customers?

A lot of people talk about, you know, a lot of different tactics of selling or marketing. And you can read a lot about this. But in some sense, the more useful way to think about it is, you know, the go-to-market effort is helping customers actually just get the value out of the product. In Silicon Valley, we're very product-centric, so we think about, oh, what's the thing we build? But ultimately, the customers kind of care about what, you know, what's this project I can do? And the go-to-market effort is really helping people realize, you know, what's the valuable thing that they can do to discover that it exists, take advantage of it for a first project, help that spread to other use cases. And this is particularly important for a company like us where it's not a one-stop thing. It's not like, hey, you come and you install the product and then you're done. It's something that spreads kind of application by application and use case by use case. So if we think about that journey, it can easily be like a 10-year thing where they're spending money the first year and they're spending more money the next year and more money the next year. And as long as we're helping them be successful and we're the platform of choice for this, we can kind of continue to grow and expand in that customer and they continue to realize value. The companies that have done this the best are the ones that try to directly align, you know, what's the value the customer gets and what is it that we're incentivized to do, right? And that's actually one of the big transitions that's happening in our space. You know, you've seen kind of an evolution of software models where maybe early on you bought a perpetual license and you kind of throw the product over the fence and hope that they get the value out, but whether they do or not, you've got the money. And then the next evolution of that was these kind of subscription models where, okay, you keep paying each year. And so if you're not getting the value, then you're not gonna keep getting paid. And so it's a little bit more aligned. And what you're seeing in the world of cloud, and now even increasingly in the world of SaaS, is these consumption models where the idea is like, hey, it's not just that you're buying a bunch of seats and it's kind of fixed, it's very much based on your usage. And that makes a lot more sense for this kind of underlying software infrastructure. The user is not really a person, right? It's a computer, right? So it's about the level of usage. And so you're seeing these models that are very much about the kind of direct consumption, how much data is being processed or stored or flows through the system in different ways. And that alignment is even tighter in terms of like, hey, what is it that we're actually delivering? Are they using more? Are they using less? Are they expanding their usage? And I think that's a super healthy thing. And it makes us really think about, hey, what, What makes this technology succeed in our customers? How could we make them succeed faster? And it turns out that, you know, if we apply our brains to that problem and we're successful, you know, it's not just good for us, it's good for them as well. Really thinking from that customer point of view back versus all of our tactics forward, that's kind of the thing I think is most missed when people have go-to-market conversations is, hey, what is the product? And, you know, what's required to for a person to get value out of that product? And how can we accelerate that cycle, whether that's helping them discover it or helping them use it or whatever else it may be?

In the history of the business, you've raised $450 million. You took it public in 2021, which is obviously pretty crazy to take your company public in the heart of a global pandemic. What moment did that signify for you and the company, taking it public through COVID?

It was a big moment. You know, it, um, obviously the company goes on and on. So any of these like individual financings where you raise money or even an IPO, it's kind of not really the, the end of the story. But it was, you know, the IPO was super fun. And it was, you know, it was happening in the middle of COVID So there was just, there just been a, you know, a year of difficulty and hard things. And so it was great to have just kind of a pure celebratory moment that, you know, it was kind of a recognition of where we got to. And I, I know it's something, you know, not just me, but like the whole company full of people just worked so hard for. And continues to work hard for. So it's nice every once in a while to have one of those moments that you get to kind of stand back and look at it. I think, I think it's a really exciting thing.

And we'll be right back after a message from our sponsors. I want to transition just a little bit to you, Jay. I always like to ask the question, especially, you know, you've now done many things that have changed the world of technology for the better for everybody. If you have to go back to your childhood, is there something that your parents or family members did that you think helped make you special? What would it be?

I had very loving parents. I would describe them as being a little bit part of the counterculture people in California, both my mom and my dad in different ways. Neither particularly wealthy. You know, my dad was a kind of carpenter and did construction work, and my mom did a number of different jobs over the years, but either had like a college degree or more traditional corporate profession, but, you know, they cared about me, they cared about learning and thinking about things. And then, you know, I had a lot of freedom in how I kind of structured my own life. So I had a kind of interesting path where I only went to one year of high school. I spent some years teaching myself stuff.

I was just gonna ask you this, so I'm really glad you brought it up. Keep going.

Yeah, I ended up going to college after that and then, you know, getting a job as a computer programmer, but probably had 7 to 12 different ideas of, uh, what my major would be or career at different points, um, through that. So, you know, I don't, I don't know that's actually, you know, the best way to structure things for kids, but it was good for me, you know, let me explore a lot of different things. I do think it's actually somewhat helpful for a CEO. You know, you're not really responsible for one function, you're responsible for a lot of things coming together successfully. So you do need to have a little bit of a kind of overview of, you know, how it all, how it all works together. And the issues kind of range from, you know, things happening out in the world, things happening in different industries that we sell into, what motivates people, and then all of course, the traditional functions inside of this kind of tech company, which would be, you know, product and engineering and marketing and sales and so on.

You went to high school for a year and then decided to teach yourself and then eventually went to college. What did you do to teach yourself all of what we all learned in high school?

Well, you know, the problem is that you're a teenager, you don't know that much about the world. So I was convinced that it was a complete waste of time to be in high school. And so with some of my other friends, we decided, okay, we will leave and we will just teach ourselves stuff from books. And there was some effort to talk our parents into this, but, um, we did, for better or worse. It turns out it's harder to just teach yourself things out of books than it would sound. I mean, maybe in retrospect that's obvious, but at least at the time we were like, you know, honestly, there can't be more than an hour of learning in a day of high school. So like, this should be pretty easy. It mostly worked, although I would say like learning chemistry from a book is not as easy as it sounds. So we mostly ended up going back to junior college or college through some other track after that.

Was there a major selling point to your parents? Like, how do we— how did you get them on board? Was it just like, let me try, and if not, I'll go back? How'd you convince them?

Yeah, it was that. And I had figured out that, you know, there's some mechanics of like, okay, you can still go to college if you want to do that, which I had not known before, even if you don't go to high school. And, you know, the mechanics of doing it, uh, legally right, because you actually have to be in school. So you can actually create your own school, uh, which I did actually have my own school, uh, for some period of time. It's very easy, you just fill out a form in California and then you have a school.

Wait, Jay, that is— I mean, I wish I knew you then. I would have given you all of Inspired's money at that moment when you created your own school. I would have been like, this is something.

My school was not that successful in that we didn't really have that many students after, uh, uh, after me, but Yeah, yeah, so it was definitely an unusual path, but I think there's something to that. Like I always find when I talk to people, it's usually, even if they claim it was a very linear point A to point B thing, it's usually a little more complicated than that.

Of course, of course. But I do thank you for indulging me in giving me more on that point. You obviously went from, on a personal front, you went from being a tech leader at LinkedIn to the CEO of now a publicly traded company, which is a big difference. What would you say have been the biggest surprises along the way? What surprised you? Because you don't seem like a guy who gets surprised very easily. So what were the things that you said, oh wow, I didn't ever think it was going to feel this way?

Yeah, I mean, I would say the most surprising or challenging thing was actually probably right at the start. You know, in my professional life at least, I always felt like I was very undaunted. Like, um, you know, I, I was not worried that we were gonna fail at something. I thought that was a really good characteristic to have as a software engineer because you're willing to try much harder problems than anybody else. And maybe because I had kind of an unusual path, I was like, well, worst case I fail, they fire me, I'll get a job somewhere else. Like, it's not going to be the end of the world. So we might as well take the thing that's like the most important problem the company has and try and solve that. And if it doesn't work, that'll be fine. So that was my attitude when I was working for other companies. I felt like, okay, starting a company would be very similar where I would be like undaunted, but actually it's very daunting to go from being a software engineer to a CEO and especially the CEO of a small company because you really, you have nothing, right? So you have no product, you have no customers, you're trying to get people to join you. They're kind of look at you and they're like, well, you don't really look like you've been doing this for very long. And you're like, yeah, I haven't. And so, and then particularly for us, because it was something very new and kind of deep in the tech stack, it kind of just confused people or frightened them. Right. And so, you know, certainly the early fundraising, there was, there was some definite naysayers in that process, but there was also interest, but it just definitely was a daunting process early on of like, you immediately step into a job where your peers are like Steve Jobs and Elon Musk, but you have nothing, right? You have, you know, and you really have to kind of build it up from there. And yeah, that was actually very different, I thought, from working in a company to adjust to. And the types of decisions you make are very different as well. So, you know, and this is maybe predictable, but engineering decisions, you usually have a lot of the information and it's mostly about being smart about the decision you make off the information. Whereas it's probably true no matter what, as you become more senior in a company, you're just making very important calls with much less assurance that it's the right thing or that it'll work out. And, you know, so for me, I, I don't think it was so much about getting better at making that kind of decision. It was more about, you know, just not having angst about it.

How did you actually get better? How did you teach yourself? Because the angst, again, especially if you like certainty— of just getting comfortable to go and say, listen, I can't change this. We just got to make a call and move. Did you have anything that you did specifically? Did you work with a coach, or was it more just like you wore yourself some calluses in a way where you just got more and more comfortable over time?

Yeah. The biggest thing for me was more— it was a mentality. I've heard it explained more recently in terms of playing poker. The goal for a poker player is to make good moves. Sometimes when you play well, you still actually lose the game because you just got bad cards or your opponent got good cards. There's like a bunch of things out of your control, but you have some things in your control that you want to make good decisions around. I think that mentality is very important just to be very clear on like, hey, what's in my control and what's out of my control? And am I making kind of the best, the best move I can with the information I have? And letting the rest of it go. And I do think that when people worry a lot, they tend to worry about the things that are out of their control rather than the things in their control. And it almost takes away the energy you have to do the things in your control. And so I think that mentality is actually quite important. I think it's kind of freeing in a way, just trying to feel like, hey, have I done my best? Did I do my best today?

Last quick round, I'm going to ask a question. First thing that comes to your mind, we'll do it really fast. What's a favorite book that you'd like to think about or come back to time and time again?

It's called Battle Cry of Freedom, and it is about the Civil War, and I think it's like an amazing story.

Your biggest pinch-me moment to date at Confluent, the one where you said, "I just can't believe we pulled that off," what was it?

We actually pulled the date of our IPO forward by a month. That meant that it landed on my wife and my anniversary. It was a really funny thing. We were doing, you know, we're doing the IPO, we hadn't really told almost anyone that it was our anniversary, she was there and they actually put on the front of the NASDAQ building, "Happy anniversary, Jay and Jamaica." We were like, "What?" Somebody had orchestrated all this and that was definitely a pinch-me moment. It wasn't really about the company, it was more personal, but it was at least tangentially related.

Other than Confluent, one other idea of innovation, it can be anything, it can be a company that's adjacent or something that you've learned about in the last year that you find fascinating.

One of the things I've been most interested in outside of Confluent has just been the kind of like transition to sustainable energy. You know, this is something that's happening in the background. You can follow little parts of it, you know, and it's just happening in an interesting way. You know, there's the electric cars, which everybody pays attention to, but there's electric lawn mowers and electric all kinds of stuff. And so really each part of that problem and what's involved, that's something that's gone from like a, kind of more theoretical political debate to something that's actually like a very significant engineering and technological problem to go orchestrate in all these parts of the economy. And so I think that's an interesting one to follow. It's not really one thing, it's like 101 things, but it's a fascinating one.

Jay, this has been such a delight. You're such a thoughtful, thoughtful human who has just changed the world of data for all of us. And we've admired you from afar for a long time. So we're so happy to have gotten to meet you, and we're rooting for everything that you're doing, and have no doubt that you'll continue to be very successful. Everybody out there, if you haven't already checked it out, or if your company's not using it, you should check out confluent.io. And Jay, thank you so much for joining us here today. Been an honor.

Wonderful to get to talk. Thank you.