Jay Kreps: Confluent and Kafka, SF Scala @Nitro 20150115

Hello everybody, this is SF Scala Developer Series where we talk about interesting technologies and interesting companies behind them. And today our guest is Jay Kreps. He is the founder and CEO of Confluent. It's a new company which is productizing Kafka. AKKA, a well-known open source project originally which started at LinkedIn and used in a lot of companies, especially Scala-centric companies because it's written in Scala itself. A lot of people use AKKA and Kafka and also Kafka to feed Spark streaming jobs and Storm streaming jobs. So Jay is our guest today and we're gonna talk about the origins of this project, use cases and kind of future. So I guess I'll start. Welcome, Jay.

Thank you for having me.

So I remember I'll pick up on one of our previous conversations when I visited LinkedIn a while ago, also asked questions about Kafka. You mentioned that when you joined LinkedIn, you wanted to do all this cool data science, right? So your original interest was to do data science, and then you realized that there is not much or not enough infrastructure behind it. And so then you essentially became a data engineer and you built multiple systems. Maybe you can kind of outline the history. I remember Voldemort's social graph. So I wonder, how does this trajectory happen? And it mirrors what happens in big data now. We have data scientists who want to look at data, but it's necessary to feed them this data. So you actually went the data pipeline route. So how does this transition happen and maybe what challenges people face? If you want to be a data scientist, do you need to know about data pipelines? How does this whole thing work in companies like LinkedIn? How did it work for you?

Does that make sense? Yeah. So I came out of a machine learning background and I'd done some related work in a web company. Prior to going to LinkedIn. And so I came really because I thought LinkedIn would have the most interesting data, or any social network would have the most interesting data because it was about people. It was really a kind of new application for data stuff. And that was totally true. And the challenge was really that we were operating at a much larger scale than any of the stuff I had really faced. And so a lot of the kind of off-the-shelf machine learning techniques were totally inapplicable. Then just the process of actually productionizing a recommendation engine or any of this stuff was really intensive. We really didn't have any infrastructure or support for it. At first, we were trying to solve those problems in the context of building products, but eventually we realized, hey, if we could do this in a more infrastructural way, we could do it across all the products and really have a big difference. That came out in a bunch of different ways. I worked on the— Hadoop adoption, and we kind of moved a lot of our big batch processing that compute the people that you may know on LinkedIn into that. And then we were able to significantly fancy up our algorithms and scoring, right, just by virtue of adding more data and having more computational power. That really kind of gave us a platform to be able to iterate on that. And I built a key-value store, Voldemort, that you described, and that was really used as kind of the serving the live serving and for low latency access. The social graph, right? Yeah, so over time though, what I came to realize was one of the most interesting and unaddressed areas was actually asynchronous processing. So batch processing, of course, is asynchronous. That's very delayed. But maybe there's no reason that things have to wait 24 hours to get processed. Maybe you could do things in real time in a stream processing processing mode. I felt this was really closely connected to actually making the data available to different systems, available in Hadoop, available for serving, all these pipeline questions that were at the core of really making data usable. Yeah, that was how I became much more of a data plumber. I'd always had an interest in distributed systems just because it's an interesting area, but that was really, I think, and maybe still is actually, the most pressing problem I think to be addressed. Once these technologies get a little bit more mature, these platforms get better, I think of course that stuff fades into the background. You can move up the stack and focus more on what's the point of the data. Yeah, yeah. So it's always about where the most pain is at any time. And certainly for these large companies that have these massive data sets, that's really where a lot of the pain is today. And it's just starting to get, I think, a little better with some of these big data technologies. The result of that was we came to focus on building Kafka, which was meant to be kind of a next generation messaging system. It was very stream-oriented, but it was built much more like a distributed database, more than a traditional messaging system. And it was built to be kind of a backbone for stream processing. And so we spent a bunch of time kind of operationalizing that, getting that to run at large scale, and then getting everything that happened in LinkedIn represented as a stream. Which is actually not that intuitive. So most people are aware, of course, that a log, application logs, error logs, can be thought of as a stream. But it turns out even database changes and things like that can be thought of as a stream, and that's actually a really good way to propagate that kind of change and process that type of change. And so we kind of built this facility that ended up being kind of the core data pipeline for the company that fed everything else. It was the platform for a bunch of really cool applications that were built. And we liked it so much that recently we've kind of left LinkedIn to start a company around it.

Yeah, that's great. And so, I mean, we were enjoying various startups where I worked before Kafka for a while. It's exciting to see that kind of become its own separate project and with organization behind it. So I wonder, it's a very happy, really very happy outcome for open source project, right? I think this is the best you can hope for, right? You started within a big organization, it's motivated by use case, and then it kind of gets enough community support and the organization itself kind of helps, helps us spin it off. I wonder, I mean, a lot of open source folks would want this to happen. So what is special about LinkedIn that first it let you incubate and germinate this, uh, uh, infrastructure projects, right? Obviously, like, you were probably you had some other things, right? Probably the process, you know, incoming data. So you were able first to have an infrastructure group where you were thinking systematically about what kind of pipelines you need in the first place. And then, like you said, you actually had clouds to convert the organization to think in terms of that infrastructure. What is it about LinkedIn or about you or about your group which made it happen in the first place?

Yeah, it was something that came around fairly early after I joined was that the engineers, of course, wanted to open source the stuff they were working on. And the company was kind of like, hmm, why would we do that? And we talked about it. And LinkedIn was actually fantastic about it. I mean, the realization is for a lot of these services or websites, their core IP is actually data. It's data and users. It's not actually their code. and obviously they don't want to make all their code public, but there's large portions of the code that are fairly general purpose. And if you want to convince people to make those parts good, you have to treat them like products in their own right. And one of the best ways to do that is to make it something public. And so we started to do it, I think really just because the engineers wanted to do it. And then over time, I think it became clear that the things we did that for ended up much better. We were able to find better people to work on them. The people were motivated to really document and explain them and really treat it like a first-class product, not just some half-baked internal tool. And that kind of led over time to this culture of really wanting to make our best infrastructure open source and planning for it up front. And I think it's very motivational for people who work on this stuff. Software engineers hate doing things twice. I had friends who worked on the search system at LinkedIn, and their experience was they'd gone from company to company and essentially rebuilt the same distributed search stack. And this was their third take or whatever. So obviously for people like that, it's much nicer to be able to kind of build it once, make it good, open it up, and be able to take it with you as you go. And I think it's to LinkedIn's credit that they realized very early on that that was a positive phenomenon and not a negative phenomenon. It's not a threatening phenomenon. And I think that's becoming increasingly common. You see a lot of companies that do that now. So I think it's a wonderful thing for open source.

So now that you have that open source project, how did it go within the organization? How did the adoption happen? I wonder, what's the mechanics of it? Were there any holdouts? Did you have to do a lot of explaining? And how long did it take to convert majority of the infrastructure?

Yeah, so I'd had this idea really early on for this messaging system that would be much more efficient, and would be built like a database log, and all these other things. And I had this idea that we should replace our main activity tracking pipeline with that. And it was just a lot of work, so no one really wanted to do it. But then our main activity tracking pipeline, which was really just like we logged out to log files and Rsynced the log files around. It started to just kind of like fall apart under load, and it was just not reliable, and we were kind of hacking things to make it work. And then it became a much easier project to pitch. And so we started with that. That was a great use case because it was really high volume, so it really made us get the kind of like operability at scale stuff down up front, which I think is very hard to add on later on. Then we moved from that to all the operational data metrics. Finally, the thing we were doing as I left was all the database changes, that capture. A lot of the core systems are now built around Kafka as a log, so some of the storage systems. Yeah, not only that, but actually moving some of the core data systems themselves to use Kafka as their backbone. So the newsfeed system, which is a serving system in its own right, is kind of like basically a big Kafka log. And they were doing a similar project with some of the key-value stores.

So this is when you go to your home and you see all of that.

Yeah, that's right, the front page feed of activity in your network or whatever. So yeah, that's basically a big Kafka-backed serving system, which is pretty cool.

Great, great. And I think everybody saw that, everybody uses LinkedIn, so that's kind of a good thing to visualize. Interesting. So I mean, I have to ask, being a Scala organizer, how come it was written in Scala. What are the kind of experiences you derive from it? And now that it's open source in Scala, what kind of Scala adds? What it makes easier? What it makes harder? How do you see that kind of choice going forward?

Sure, yeah. So it's kind of an interesting choice. Kafka is basically a pretty low-level piece of systems infrastructure. So the reason we did it was just because we felt that We weren't really sure that Java was going anywhere at that time. This was still the Sun days. There hadn't really been a lot of changes or improvements in the language in a long time. We were concerned about our ability to recruit top-end programmers to work in that environment. So kind of the pro of Java was it was pretty fast. So you could basically build a reasonable system in it, which is not true of, I think, this type of system in Python or whatever. It just wouldn't be practical. But it was still reasonably efficient to build things in. But it was just kind of a clunky language. So the attraction for us was just a more modern Java that had a lot of the more modern language features available. That was what we were looking for. I think it mostly paid off. The things it made easier— the Scala code is more concise, I think. There's a fair amount of asynchronous action in any kind of high-end, non-blocking server. So where you need to have tens of thousands of connections alive, and you need to do asynchronous request fulfillment where a request comes in and it won't be satisfied until a bunch of other things happen in this distributed system, and then you send the answer back. So being able to represent that well, I think some of the functional aspects of language come into play nicely. There's been a ton of downsides. The compatibility issues in Scala, I think, have been a huge problem for the clients. So the clients were finally just moving out into Java so that there's no dependencies and we don't have these issues of compiling for different versions. I think it has traditionally been harder to get contributors. I think because now it's like, okay, in addition to this hardcore distributed systems engineering, you also need to know this language. But increasingly, that's not been the case because now it's so prevalent that there's really a good base of people. Who know it. So I think that's less of an issue now. Interestingly, we've never actually run into significant performance issues, which is unusual for a new language. So we've had very little in the way of that. A few issues where, like, yeah, we had some weird implicit that we were making zillions of copies of, but very, very rarely do we hit anything like that, which was nice.

That's great to hear. So there is this metaphor of Scala tribes where there is FP Extreme, there is Scala as Java and Twitter are kind of middle of the road. How would you characterize your approach, your codebase? If somebody wants to pick up Scala Kafka, can they use it as an example codebase? Can they learn good practices from it? Yeah, yeah.

So it's interesting. I don't know if it's good practices or not. These infrastructure projects end up being kind of their own world. We don't really depend on hardly any libraries because we're like the bottom of the stack. So it's not like we have a ton of interaction with other Scala code. So who knows if Scala people may think it's terrible or good. We're definitely more conservative in our Scala feature usage. Typically the motto of any of these types of infrastructure systems is like no surprises. So we're relatively conservative there. We're probably close-ish to Java, although not that close. So, I don't know, it would probably be up to somebody more proficient in the language to judge the ultimate quality.

How in terms of reuse, do you consciously try to reuse as much as possible and kind of structure libraries this way? Are there any Kafka libraries or components which other people can look at as an example?

Yeah, sure. I mean, any of these systems is broken up into core subsystems. So the subsystems for Kafka are the network server, which is just a standalone network server. There's nothing really Kafka-y about it. So it's just an I/O network server. There's the log implementation, which is actually what's writing out data to disk and checksumming it and dealing with—

So you have a lot of abstractions.

Yeah, and some scheduler which runs background stuff, and the whole replica management system which keeps track of leadership and delivery and stuff like that. So there's a number of these kind of core systems that are there. They each kind of stand alone for the most part with some amount of layering between them. Usually the lower-level ones like the network or the log are kind of the easiest to grok independently. You can just kind of look at it. The log looks a lot like a commit log. You can kind of see what it does.

So a lot of folks now implement distributed systems. And I've seen several new databases are coming up. Startups, right? And I've seen a lot of implementations of Raft or Paxos.

Yeah, yeah.

So can somebody, for instance, take your leadership election algorithm and use it in their own database conceptually, right? Is it, is it something— because we know that Kafka is one of the biggest distributed systems working tremendous throughput, so it's a proven, right, it's a proven system, which, you know, is proven in terms of failover, in terms of throughput. So can somebody Kind of piggyback on that and implement it anyway.

Yeah, so that's right, that's right. So it turns out that a log is actually a kind of core abstraction, a distributed log is actually a core abstraction in distributed systems. And so this is an interesting area, it's not very well explored. It's something LinkedIn was doing a bunch of and a few other companies, which is you can actually take Kafka and use it as a log to build a storage system or some kind of newfangled serving system. What you would do is you would couple it with some kind of key-value store like RocksDB or something like that. So you have kind of serving nodes which have the RocksDB, and when they write data to their stuff, they log it out to this fault-tolerant log. And that basically takes all the distributed programming out of the system implementation. So all that correctness and fault tolerance is punted off to this log, and your system now just has to deal with the, how do I fetch stuff off disk? Right. So it's sort of related a little bit to this kind of Command Query Responsibility Separation or event sourcing. Yeah. So, this is maybe the kind of large-scale partitioned version of that, and especially for data systems. So, you can absolutely use Kafka in that way. Nothing internal to Kafka, I think, is being used as a library elsewhere. I think making like a reusable, I don't know, scheduler or network layer is actually much harder than building a good network layer for Kafka. You have to really do a lot to kind of file off all the edges. So I think there's a big difference between Kafka's network layer, which is probably 600 lines of code, and Netty, which is probably like 30,000, right? For what Kafka does, we don't need any of that other than the 600 lines. But to really make something reusable across every possible use case, you actually need the rest of it, right? So, yeah, I try never to expose the internals of the system because it's hard enough to get the few APIs you do expose right. Okay.

But I mean, I think Scala folks are always asking for interesting systems to look at, so Kafka may be an interesting example for somebody to look at and understand and kind of see. Yeah.

So, for people who are interested in distributed systems, I think it's a nice midpoint where it's not so large that it's impossible to dive in. There's probably about 50,000 lines of code. That's amazing. Yeah, it's not huge. A lot of that I think just comes from Scala. If you were to rewrite it in Java, it would probably be bigger. They're not necessarily more meaningful lines of code, but you would just have more of them. So yeah, it's small enough that you could sit down and basically come to understand it if you understood the external way that the system works in a reasonable amount of time without devoting your life to it.

Right. Okay, so I want to kind of shift gears a bit and ask about Confluent. So now we have this interesting evolution and kind of for the point of reference, so we see several related projects. So often Kafka is used to feed Spark. And Spark now is a Databricks-backed company, $50 million or more funding. And Mesosphere used as a foundation. —same kind of scale, right? So I think given the user base for Kafka and the number of use cases, I can easily project that in 2 years you're looking at the same scale. You're basically looking at giant office in San Francisco and hundreds, maybe thousands of people eventually on the road. So I mean, given that, so it will be interesting to see this a couple years later. But I wonder, how do you see, what is your vision to basically take this open source and evolve it to the backbone of lots and lots of companies, big and small? How are you going to grow the company? What's your strategy around open source? What's your vision there?

Yeah, OK, that's a lot of questions there. All right, let me start with the beginning. So yeah, I think it's obviously an exciting time for open source companies. Where open source works really well, I think, is lower down the stack. So platform things, programming languages, data systems, infrastructure stuff. These are the things that I think just are hard to make into proprietary software companies these days. Probably have always been hard to get off the ground in that way. Open source is a way that you can actually get to some level of usage to the point where you validated that this new idea works. The interesting outcome of that is you're actually— I think the first generation of open source work was largely commoditizing things that existed. So Linux is kind of a clone of the proprietary Unix systems. Obviously, it did a ton of innovative things internally, but its external interface and what it does is totally understood. It's a product category that's well defined. MySQL is, I think, in a similar category. It was taking relational databases as kind of a commoditized version of that. In those cases, especially in the early days, you say, oh, not quite as good, but— much cheaper. But what you're actually seeing that's really interesting now is all these open source companies are actually doing really innovative stuff. Like really innovative stuff, like Mesosphere, Databricks. A lot of these companies I think are really kind of trying to define new categories of software. So I think that's amazing. It's an awesome time for it. This is like one of these rare points in the cycle where the stack of things is getting rewritten. I doubt this opportunity will be around in 10 years. Probably things will have calmed down a little bit more. And we'll have a much more stable stack of technology, but much less new creation happening. Yeah, yeah, so how does Confluent fit into that? Basically, our belief and our vision is that this kind of real-time flow of data and representation of data as a stream is actually a really powerful thing. And it's a really powerful thing that solves a really ugly, important problem for companies, which is around how they funnel data between systems, how they process data. There's a lot of kind of nasty batch processing and, you know, R syncing CSV files around, and this kind of stuff is ugly. It's expensive to have people doing it. We think there's a better, you know, much cleaner way to do this at large scale. And, you know, kind of the thing that's new is the ability to really, you know, build a distributed system that can kind of have a single instance at the heart of a company and have everything plug into it. And that's kind of the vision that we're we're trying to make come true. So that's our belief about the world. Obviously, to make that happen, there's a ton of things that you have to do. And that's what the company does, right? So there's the kind of normal things you need with any software, which is support and training and all that, and we're doing that. But there's also just this whole ecosystem of software that's grown up around Kafka that has to be matured and hardened and all the tools you need to run it. And that's what we're really putting together. My belief is when I started on this looking at what was in LinkedIn, I felt really if you were to start from scratch, about a third of the things that happened in a company like LinkedIn would probably be in this asynchronous stream processing domain. Maybe a smaller percentage probably would be fully batched, and the rest, probably the largest percentage, would be real-time.

So all the client interactions with LinkedIn. Yeah.

Yet when we were starting on this project, if we looked at what we did, very little was in that domain. The reason was because there was no infrastructure support for it, right? And I think this is actually probably even more true of many other companies than it is of LinkedIn. It's just that we started on it there earlier. So yeah, that's our big bet. That's our belief about the world. And that's what we're kind of trying to make come true.

Right, so it's interesting, right? Like if you look at companies like Spark, so it has a lot of enthusiasm. Yeah, yeah. So the open source community is actually driving it. Yeah, yeah. And it kind of, you know, the enthusiasm is ahead of a business model, right? Because it's undeniable that people want to use API. Yeah. And I think personally that in my kind of simplified view of the world, Spark is distributed Scala. It took Scala collections and presented this really beautiful API to developers. Now they can do MapReduce in a small and they can do the same thing in the large. And that resonated. And in a way, streaming processing resonates with data pipeline engineers. I think that's kind of why a lot of startups are picking up Kafka and kind of Scala folks understand this. So it's very predominant among the community. So how do you kind of harness this enthusiasm? Do you, for instance, let it grow organically? I mean, obviously Spark is, you know, Spark Spark Meetups is one of the biggest meetups in the world. It was actually spun off of SF Scala 2 years ago. Now it's about twice as big, about 4,000 developers and growing rapidly, right? And Databricks constantly supports it and builds it in different cities. Do you have a strategy, outreach, evangelism like that? What's kind of your thinking there?

Yeah, so far it's just been totally organic for Kafka and we're just starting to actually try and get involved and help that. And it's amazing how many things are actually happening, like different companies getting involved. The people from different web companies are now committing code all the time. And just watching that spin up is super exciting. So we haven't done much yet to really kind of help amplify that. But yeah, we absolutely intend to. And I think the right way to— if you're trying to make some kind of platform change, in the technology stack, it's a very big change to make, right? Right. People don't like to replace the platform very much because that's what everybody stands on. You don't want the platform to break. So it only happens now, again, if you want to do it the way you have to do it, you have to make it open. Nobody's moving towards proprietary platforms at this point. You really have to make a kind of open source standard thing that is truly community-driven in the sense that if any one company, including Confluent, were to die, this thing would would continue and would not go away. That's what people want to adopt. Yes. And second, you have to actually express the idea that you have about the world. How should you be doing it to everybody else, right? And the way to do that is, again, I think through a really rich community. I think what this recognizes is how much of software is actually in people's heads. It's actually human capital, not intellectual property. It's actually human capital, right? It's things that are in people's heads. Know how to do. It's how they conceptualize the problems that they see. And so yeah, I don't know the right answer to your question, but I think the result you want to see is you want to see a diverse community with a bunch of people talking about it. People at different companies, people using it, all those kinds of people. And yeah, the role for these open source companies that are centered around a technology is absolutely just to help amplify that voice and let people speak.

OK. So since there is a Spark Summit and MassAss Convention, I wonder, what's your prediction? When are we going to When can we see the first Kafka conference?

Oh man, as soon as we can get it together. There's a lot right now, we're just trying to get office furniture.

Okay, okay, yeah, I'm looking forward to it, to bring the community together. Okay, and I guess, I mean, the last question I'll ask is, it's, so there's this fine balance, right, between open source and proprietary, and I think in case of a lot of companies doing support and training, right, and some companies are picking pieces which they keep closed source, such as, you know, Databricks Cloud, which is a platform, right? And I wonder what is your plan here? Are there going to be pieces of Confluent Kafka which will be closed source? And if they are, how are they going to interact with the open source parts? What's the vision there? Yeah, that's a good question.

So yeah, the balance with any of these companies is you need to have a working business. In order to fund the company. And so there's lots of ways to go about it. What I like, the model I like the most is companies that take some piece of infrastructure like Kafka and they take a use case and they help to build it out as a product. And so we're taking a couple of the common use cases around data integration and processing and really trying to productize those. What do you need to run it, blah, blah, blah. What we are building and selling to people and supporting is really an open source platform. That's what you build around, that's what you plug into. But there are areas I think on the side for applications that help you run things and so on that can be proprietary. It has to be optional, it can't be your core commitment, otherwise you don't want to adopt it, right? No one wants to be on like a forked version of an open source product. They don't want like some weird proprietary Kafka that they're getting. But there is, I think, a lot of opportunity for a company that wants to adopt it faster and get going and run it easier to basically pay money and get some piece of software that helps them do that. And that's a totally legitimate and good model. And I think you can look at these open source companies and see a number of them have done that successfully, and we will be as well.

Great, great, great. So do you think that it's safe to say that open source, people picking open source, Kafka, GitHub, will always be basically basically able to work with, let's say they take it first and they test drive it and they should put it in production, will they always be able to kind of upgrade to Confluent offering without kind of any major, like it will be like a kind of all benefit and no problems? Will they be getting always the full version of the platform and like will the closed source pieces would be monitored monitoring and debugging or diagnostic. Do you think that some functionality itself, in terms of SLA performance, can change in the paid version?

Yeah, I think one of the important things, you can look at these different open source products and companies around them. Some of them are very open source and some of them are kind of not. There's no real open source community. They publish the source somewhere, but it's under some license where you can never use it. —and the usage of those systems ended up being really different. And there's nothing wrong with closed-source software, and there's nothing wrong with open-source software that has no community. That's right. But for what we're trying to do, where we really want to be kind of at the heart of the company and really be this kind of interchange or hub for data, that's a platform thing. You're not going to want to be stuck on a proprietary version of it. And one of the commitments for that to work has to be that, yeah, not only can you easily upgrade from open source to the closed source, but but even more importantly the other way. You need to be able to downgrade from the Confluent offering to just Apache Kafka that I downloaded off the Apache website. And what we're hoping to do is actually just amplify the work of the community. A lot of the gaps that constrain things today are really the set of clients for Kafka. There's a dozen open source projects that have different clients in different states of maturity. None of them are tested against the main server when it's released. Not all of them are even feature complete. So just being able to package these up and make sure that they're robust and they work with each version. And extending that development, getting people to help out on those projects and advance them along. That doesn't need to be proprietary. I think that adds a lot of value and helps amplify the work that's already happening in the community. I think it's totally the right approach for this stuff.

Great. So, I guess last kind of question I will have, if folks heard about Kafka, they think it may be useful to their business, what advice do you have for all the startups and all the business out there? How can they start? What should they do to check out if this is something which will kind of help them re-architect their systems for the future?

Yeah, that's a great question. So, the documentation on the Kafka site is actually pretty extensive. It'll walk through a little bit of the use cases. We're putting together more material to help walk people through more specifics of like, "Hey, how do I build X?" So we're going to start releasing these things as blog posts and stuff as we start to have that more built out. I did a really long blog post on kind of logs and the theory of it that people can pick up. It's even an O'Reilly book now if you're interested called "I Heart Logs." So all of those are great sources and the mailing list for Kafka is also very good. So if you feel like you have a really weird use case that's not discussed anywhere, just send an email to that mailing list. There's usually a dozen people will spring up and give you a variety of ideas. OK.

And I'll also mention that Jay gave a talk called I Heart Logs at SF Scala earlier in 2014, and that's also available on the same YouTube channel, so you can look it up. We'll probably link to it from this one, and you can use it to get educated on Kafka and get it going. Thank you, all. Bonjour.

All right. Yeah, thank you. Thanks.