Keynote: Jay Kreps, Confluent | Modern Data Flow: Data Pipelines Done Right | RTA Summit '22

Thanks, Tim. I'm really excited to be here today. I'm gonna be talking about modern data flow, and this is real-time data pipelines done right. And if you're interested in real-time analytics, then the first problem you have to solve is, hey, how do I get the data there at the right time to be able to visualize it, to be able to analyze it, to be able to use it? You know, I think this is part of a broader trend around how data is used in organizations. And this broader move to streaming, there's really something happening here. And you can see one illustration of that in just the growth of Kafka usage and the Kafka community. There's hundreds of thousands of organizations using Kafka, tens of thousands of people attending meetups. The ecosystem around this has just grown and grown. And you can see that over time If you look at the usage of Kafka, this is the people kind of actively integrating with the Java library. There's again, hundreds of thousands of people who are now using this, and it's increasingly part of the production stack in virtually every company. And one of the use cases we see at Confluent over and over again is the integration into the real-time data stack, and especially with Pinot, and that's why I'm really excited to be here with StarTree, at this summit to talk about some of the use cases here and a little bit what we see happening in the real-time data world. And, you know, what drives this, I think, is just increasingly a rich array of real-time use cases for data. And this is something that would've been, you know, very hard to do a decade ago, but is increasingly just part of what customers expect, is just part of how you operate a, modern digital business. And this spans from everything to, you know, how you get your prescription filled at the pharmacy, to providing care for elderly people on demand, you know, when a caretaker needs some help and needs to get somebody to come help them, to, you know, the kind of orchestration and delivery of groceries, to how we're modernizing our energy infrastructure and taking advantage of more dynamic supply and demand in a modern electrical grid. And all of this is, you know, part of a major trend which has driven the rise of a new platform around not just data at rest, stored data, but data in motion, data as it moves and as it flows. And this has been our focus at Confluent is, you know, really trying to build something that can act as the central nervous system for this data flow. That can help connect up to all the different data systems and applications that can take these streams and make them accessible to all the different use cases. And we've seen just a vast array of these use cases that has arisen. And it ranges from things like detecting credit card fraud, to bridging across, you know, on-premise and cloud environments, to like rich, real-time customer 360 analysis, dashboards, personalization. You know, we see just a really broad variety of use cases. But if you're trying to categorize these in a simpler way, I think you could cut them into, you know, a set of applications which are carrying out some business logic in real time on these streams, and a set of pipelines, you know, flows of data that are trying to get data from place to place, you know, get the data to, you know, the right systems, the right applications, the right SaaS layers at the right time. And today I'm gonna be talking about this right-hand side, about the pipelines, about the, you know, real-time data flows. And when you think about data pipelines and the technology that supports that, there really is, you know, a vast array of different things, old and new. You know, a bunch of technologies that have been around for a decade or more, a bunch of things that are just coming onto the scene. It can be hard to make sense of all the stuff that's out there. And, you know, part of what I wanna do in this talk is try and lay out a little bit of the framework of where I think this is going. You know, what are the principles for how, you know, modern data flow in an organization that's really taking advantage of real-time data should work? And, you know, Kafka, I think, is a big part of this. It's a necessary part, but it's not sufficient. It's not the only thing in this ecosystem. It, you know, it's actually managing these streams. There's a rich array of new systems that are arising around streaming data to allow customers customers to take advantage of this, to harness it, to visualize it, to use it. And obviously Pinot is one of the most exciting technologies in this space that is really built for real-time ingestion, visualization, and analysis of data. And so, you know, I think that gives a little bit of a motivation for why you wanna be thinking about this more modern data flow, how you wanna take advantage of it. And so I wanna transition and give, you know, a few simple principles that I think are core to what's happening in the world of data flow, data movement, data pipelines. So there's 5 of these principles, I'll go through each of these. So the first is, you know, maybe the most obvious and kind of what you would expect coming from, you know, somebody who's done a lot with Kafka, which is streaming, right? Modern data flow is about streaming. We're moving out of a world of batch data copies into something much more real-time where data flows as it is created and arrives in systems as soon as they can take it, consume it, use it. And we're really building out the technology stack to enable that. And, you know, what's surprising in this is if you actually look at the vast array of things out there, many of them are not built to support streaming. And, you know, that's actually surprising. I think it's really surprising that, you know, in the year 2022, you would be adopting or building out a data movement technology that didn't support real-time data flow, given the fact that Virtually all the data in an organization is created in real-time. And indeed, you know, reality itself is real-time. Businesses do things out in the world and generate data continuously as they do that. And a simple question we've often asked, you know, to help people understand this is, you know, would you be willing to blindly cross the street with traffic information that is 5 minutes old? And I think the answer is no. I think if you have out-of-date information and you're acting on the world, you're doing things, you're making decisions, you have software systems that are controlling the activity in a company, you're very likely to make bad decisions. You're very likely to do the wrong thing. And the only reason you would do this is if you had to. And if you think about those use cases I described, all of them depend on this orchestration of real-world things that are happening continuously. and all of them require an up-to-date view of how the world is right now. And, you know, the other thing that I think has driven this is, you know, back in the day, I think we might have thought of streaming or real-time data as nice to have, but too hard, you know, too difficult to get right. Maybe there were limitations in making it correct, ensuring that you got all the data delivered that you expected, doing the kind of richer transformations and processing you might need. There might have been limitations in efficiency or complexity. And I think increasingly, these problems are either solved or being solved. Increasingly, you can have your cake and eat it too. And you can start to view the world of streaming, the world of continuous processing, as a kind of generalization of batch processing, where instead of just, you know, processing the data up, up to now, it keeps processing into the future as new data arrives. And the technology to make that simpler, to ensure you get the right answer, to ensure that what you build is scalable, increasingly, that's becoming part of the underlying infrastructure and is not something that you have to worry about if your goal is just to get data from place to place to transform it, to take advantage of it. And, you know, this means that increasingly, if, you know, you bet on batch systems, you're gonna end up doing everything twice, right? You're gonna have a set of batch pipelines that deliver to batch systems. You're gonna end up with a set of real-time systems that feed the real-time applications. And these data flows are quite complicated. And so the ability to actually do this across a broad organization and have, you know, different flavors of every single data flow is just not really that feasible to manage. You need to move to something that is real-time and scalable and reusable across an organization. That's what streaming enables. The second big principle and trend is the move towards decentralization. And, you know, the way to understand this, I think, is actually just to think about kind of the most classic data flow, you know, in an organization going back really decades, right? I would say the, you know, origin of software in a company is mostly a bunch of applications that would stand alone. You had an application here, it had its little pile of data. You had an application here, it had its pile of data. They mostly didn't talk to each other. You know, one of the biggest areas where data started to come together was really with the rise of, the classical data warehouse. And, you know, this was, you know, drove a need to actually get data out of all these operational systems, you know, put it in one place and structure it for analysis. And, you know, this meant that you would have a, you know, centralized flow of data, a batch tool that would probably scrape data at the end of the day, and usually a centralized team that was responsible for, you know, collecting that data, managing the integration, trying to get data into the right form. And this worked well enough in its time. You know, everybody probably has their list of complaints about their data warehouse, but on the whole, this actually served the need. But if we think about the modern organizations, the modern software stack, it's no longer that. It's no longer a handful of applications, and there's no longer just one destination for data. Increasingly, you know, we have this rich, complicated suite of different software systems, databases, SaaS layers. And the interchange of data is not just one-directional. It's not just flowing to one place. There's rich interflow of data between all these things. The data warehouse remains incredibly important, but it's not the only place that data goes to. There's more needs for more data across all of these things. So how do you solve this problem? If you try to solve this problem point by point, you know, it's simply unmanageable. You have to move to a model where there's some kind of contract and there, you know, there's some kind of central interchange that people can go to to get the feed of data so that each team can publish what it is that they have to share with the organization. And each team that needs data can tap into the right streams and take what they want. And to, you know, to make this more specific, you know, it's really worth thinking about one example in illustrating this kind of publish-subscribe use of data in the large. And the idea is very simple. You know, a common use case we would see is sharing of customer data. You know, virtually every business has customers. You have often many rich sources of information about customers. You need to get it into some usable format, and you need to be able to share it out to different systems that would take advantage of this in different ways, in different timeframes, for different use cases. And, you know, thinking about this in terms of a contract really makes this much easier so that there is, you know, data flowing out of source systems. It needs to be processed in real time into something usable and supportable as a kind of API, you know, something that people can depend on as your notion of a customer record. And then use cases can tap into this and subscribe to what they need. And, you know, analytical systems can ingest this and index it for, analysis and serving, personalization systems can act off of this. And this ability to connect the producers and subscribers across an organization really reduces the number of connections, the number of custom pipelines that have to be built. And in this world, you know, what's really important is to understand this contract between the producers and the consumers. And increasingly Kafka and the schemas that are associated with these streams that acts as that interface. You can think of it as being, you know, almost like the contract you would establish for a microservice with maybe REST and JSON, where you would say, hey, here's my API, here's my data format, you know, you can call this API and get what you want. Increasingly, Kafka is that interchange. The schemas for these records are what defines compatibility or what enables people to build against these data streams and trust how they will evolve and change over time. And that's what enables this kind of publish-subscribe use of data in the large. And a critical point here is to do this well, you have to be able to, you know, really process the data into the format that you want to publish out. It's not enough to just take, you know, what's sitting around in different weird formats and databases and send it on. You have to have something you can maintain and support, you know, very much like a REST API. You have to have an API you can stand behind over time. Otherwise, other parts of the company can't build against it. And, you know, likewise, these downstream systems that want to consume this have to be able to take that data, stitch it together with other things, get it into the format they want for their use case. And, you know, this is a very different way of thinking about data in terms of this flow. You know, the classic use of data, you know, data flows, data pipelines, data processing, is really about these batch stages where you kind of have some script that maybe extracts data, you have some other scripts or jobs that kind of process into different forms, eventually it dumps it somewhere. And so you have this kind of DAG, like a directed acyclic graph of processing. And, you know, this is the kind of classic model you might have in an ETL-type flow. But what's happening in the world of streaming and in these decentralized models is actually quite different. Instead of having this, you know, centralized batch processing, you now have a decentralized graph of data flow where the data itself is the DAG. The graph of data flow and transformation is actually what defines the exchange of data across the company. And you don't need to worry about what's, you know, upstream of the data streams that you're subscribing to. You just need to worry about the format, of the data that you're ingesting. And what's powerful about this is, you know, it's really the only way of thinking about data exchange that really scales in the large, that allows broad publish-subscribe use of data across many systems. It simply isn't possible to build one big batch workflow for everything. Many systems don't even need batch at all. They need something much faster. This is a model that you can use across virtually every team to exchange data. This is how companies are supporting this move towards decentralization, towards a kind of data mesh where different teams are able to exchange data independently. The next principle is really the move towards declarative processing, towards saying what it is that you mean rather than, you know, all the imperative details of how you're gonna get it. And this is something that has always been important in the data world since the rise of relational databases. Increasingly is an important part of the streaming world as well. And I think this is really important. You know, I think the focus in the infrastructure world should always be, how can we make things more simple? You know, there's a famous and perhaps somewhat overused quote from Albert Einstein that everything should be made as simple as possible, but no simpler. I think that really is the task for infrastructure designers is to try and build systems that abstract away a lot of the details of how it is you get what you need, and instead let you express just what you want. And, you know, this has been what SQL does in the world of data at rest, and it turns out it's equally good at doing this for data in motion. It turns out that transforming real-time data, this kind of data flow, is really well suited to SQL. SQL is, after all, a data flow language, a language about describing the graph of transformations and operators and the flow of data between those things. And, you know, this is something that we've tried to help enable at Confluent with KSQL, you know, building out a SQL language that allows you to do rich, correct, real-time transformations on data. So here's, you know, a very simple model of a script that builds out a data pipeline, you know, using Kafka, This can put together both the connectors, which extract data, the transformations and aggregations that you might have on that data, the connectors that might load that data into destination systems. You can take each of these pieces and connect them together into one data pipeline. The fourth principle is being developer-oriented. This is something that I think some of the older tools in this space got wrong. And I think something that, you know, even some of the newer ones aren't quite nailing, which is, you know, it's actually great to have a UI, to have tools that are higher level, but it's really important to start with an API, to start with the underlying infrastructure platform that allows software developers to build on your platform. This is particularly true in the world of data in motion. You know, data movement, it's not just a tool, it's actually a core part of how a business operates. You need an open infrastructure platform for this. This is something we've tried to take very seriously at Confluent when we think about how do these data pipelines work, right? It's really important to start with just the observation that, you know, code is king. The toolset around code is incredibly rich, how you version control it, how you check it for correctness, how it's deployed. You can't recreate all of that in any kind of siloed tool. You have to support kind of an open coding language. Next, development is about evolution, it's about how things change over time. How can you change your processing as the business itself evolves? You know, how can you do that on the fly without bringing everything down? And finally, how can you do this with an open platform? I think what has made Kafka so successful is really the ecosystem around it, everything that plugs into it. That's very much because it's an open platform that is widely adopted. And so I think this openness, this developer focus is key to the, you know, the next generation of data pipelines and technology around data in motion. And then finally, you know, this is one of these ones you just have to do, which is if you want to use data in the modern world, you need to actually put the thought into how to do this in a way that's correct, that is in compliance with all the rules and regulations around the use of data, that is doing the right thing by your customers, taking good care of their data. This is something that is increasingly a tension, I think, within organizations where there's really kind of two conflicting pressures, right? There's a need to open up the data, to harness it, to take advantage of it, to make the business smarter, to make the interactions with customers better, to make the company operate more efficiently. That's really about how can we unlock our data and make use of it. And there's another pressure, which is, hey, we need to lock up the data. We need to keep it safe. We need to make sure we don't lose it. We need to make sure we understand where it goes to. We need to make sure this data is correct and it's something that we can build on and depend on. If you're trying to solve for both of these things at the same time, you can't just do that in a purely organizational way. You have to have kind of an underlying tool chain that makes it possible to both open up data to the right use cases, but also reason about the correctness and the security of that usage. There's a few areas we've tried to build out at Confluent to try and make this a little bit easier. And many of these ideas are not new in the world of data, but they are new in the world of streaming data. And, you know, this starts with the catalog that allows you to discover what streams are there. You know, it moves into the schema management of, hey, what is the structure of these data streams? How can I build against that and rely on it? And finally, lineage, how do I trace that across different parts of the organization? So, you know, this catalog, this is a very simple tool which kind of knows about everything that's there, allows you to tag what data is, what's PII, you know, where did this come from, be able to search and reason across this. This becomes incredibly important in an organization in the large, right? How do you know what data you should be tapping into? How do you know where something comes from? Who's responsible for it? This is a critical problem that you have to have. You know, the schemas become very much the definition of these data streams. What are the fields? What do they mean? What are the compatibility guarantees around them? How can this data evolve over time? If I build an application, what is it that I can depend on, you know, even in the future as these datasets evolve? And then finally, lineage. This is kind of like the, you know, Google Maps for your data flow. This is what tracks where the data comes from, how much of it there is, where it goes to, how much delay there is. This is something that's really important. You know, one of the advantages I think that batch processing had was it was simple, right? Either the file got there or the file didn't get there. It's really easy to reason about it. Either you processed the file or you did not process the file. It's a little bit more complicated when we move into a streaming world and, you know, the data is always flowing and we're always processing and it becomes much more about how caught up are we? You know, are we there yet? Are we 10 seconds behind? Are we 1 second behind? Are we right up to date? Being able to reason about this and reason about it in the large is really important. And, you know, this allows many teams to view the global data flow graph, not just the part that they create, and reason about what has gotten to them, you know, how they fit into that overall flow, and operationally, whether they're getting what it is that they expect. And if not, you know, where is that getting held up? This is critical for managing this kind of data flow graph in the large. And then all of these have to work together, right? All of these tools have to be designed for a decentralized streaming developer-oriented world. All of these are what come together, and this is what we think the core principles of modern data flow are. And you can see these as a kind of evolution, right? The world of batch processing is moving to streaming. The world of centralized data management, the, you know, kind of one database to rule them all, is moving to something decentralized where there's broad exchange across the organization. You know, an infrastructure-heavy imperative way of thinking about data processing in the streaming world is moving to declarative. And, you know, kind of purely GUI-oriented data movement tools are moving to something that's more developer-oriented, to something that's more programmable. And finally, you know, these kind of complex, ungoverned, dark streaming systems are actually coming into the world of governance and observability so you can understand what's going on there. I think this is what is gonna enable the next generation of data flow across companies. I think it's what's enabling the rich ecosystem of tools around this streaming world, and it's what's enabling amazing technologies like Pinot that allow companies to harness this real-time data, you know, put it into their product, take advantage of it internally, and really get the value out of it. So I couldn't be more excited about where we're going. Thank you very much.