Beyond Speed: Kafka Summit London 2023 Keynote | Jay Kreps, Co-founder & CEO, Confluent

We've all been there, thinking we've done everything just right, and then— but it's okay. Fuelled by coffee and fast-approaching deadlines, you've got this. And thankfully, you're not alone. With thousands of enthusiastic contributors from around the globe, your Apache Kafka community has your back. Together, your passion for solving problems and building great solutions is what makes Kafka strong because of you and answers is often just a click and maybe a paste away. And when it all comes together, well, that's when the magic happens. This is the power of community. This is the power of Kafka. Welcome to Kafka Summit. Kafka Summit London is officially open. Please welcome co-founder and CEO of Confluent, Jay Kreps. Hello everyone. I'm gonna be talking today about the evolution of data streaming and the stack that is coming into being around that. And to move beyond just speed, beyond just fast data, into a rich platform that connects all the parts of a company. But I would be remiss if I didn't start by just beginning with saying welcome. Welcome to Kafka Summit London 2023. I don't know about you, I don't know about you, but I've come to really enjoy these events. I know we had a few years off in the pandemic, and I never thought I would miss tech conferences. You know, I have to say I was a little burnt out on them by the end as we were heading into that. But as these started up again, I started to find I really enjoyed them. I didn't think it would last, but it seems to have. I'm always really excited now to see a big group of people, to get to talk a little bit about what's going on in the community, what people are working on, and I'm looking forward to all of that. I'm looking forward to meeting all of you and talking to as many people as I can, and I think that's where the real conference happens. And it, it really is an awesome time for this community. The growth in this area has continued to accelerate. There's incredible adoption of Kafka in all kinds of industries and use cases that I never could have imagined when I was first working on the early Kafka code first working to get that rolled out at LinkedIn. And it's, it's been really amazing to watch, and I think it's been driven by a whole set of trends and pressures that have pushed this need for real-time data and for streaming. You know, technology trends like IoT that take the reach of data out into the real world, trends like cloud that means suddenly we have You know, a lot more data systems than we used to in a lot more places. Trends like AI and machine learning that are creating these very data-hungry applications that are sucking in data from all over the organization, but also the pressure on businesses, the need to make customer experiences more engaging, smarter, more personalized, the inevitable pressure on efficiency all throughout the organization that's driving software and automation end-to-end through companies. And I, I think this is part of a bigger change that's happening in how we think about software, how it is used in organizations. In, in some ways, the early application architecture was kind of like a single-celled organism. You would have an application that would kind of stand on its own and, you know, it would get all of its input and output through its UI. It didn't really connect to too much else. And that was how we thought of software, was these little bits of, you know, things here and there, kind of like an amoeba, right? Each application is its own entity. But I think increasingly, if you, you know, kind of zoom out a little bit and you think about all the parts of software in a company, they're starting to become connected. And it's, it's kind of going from from software as a single-cell organism to something multicellular, something where it's a sophisticated animal that actually connects all the parts of the company, that drives it where something happening here triggers something else. And I think that's increasingly the role for data streaming, is as that kind of connective tissue that actually plugs together all the applications, the different data systems, that lets companies act off what's happening. And, you know, what's emerging in that area is, I think, one of the most important data platforms in a company, the data streaming platform. And that's what I'm gonna talk about, you know, throughout this talk today. And it's gonna be a little bit about the evolution of this stack from, you know, where Kafka started, a kind of low-level commit log that's scalable and fast, this kind of hub for streams of data, to, you know, where this whole area is going today, which is something I think much richer, where, you know, it's an emerging ecosystem that's all built around processing streams, around connecting different systems, around governing data in the large. And this is the stack that I think is coming into being, and Kafka is the foundation of that. You know, it's the, the hub for all of these streams of data. But I think that there's a really important second layer that I'm gonna drill into today that is the connectors, the governance of streams of data, and the ability to process data, to build applications in, in real time that react and act on all the things that are going on in an organization. And what's exciting is there's, incredible innovation that's happening at each layer in this stack, you know, in the core of Kafka itself, in this layer above that I'm describing, and of course, in the application layer that sits on top of that. And so, you know, over the course of this talk, I'm gonna go through layer by layer in this emerging data streaming platform and talk a little bit about what's happening. And I, I was reflecting on this as I was thinking about the talk, last night. There's probably the most cool open source stuff, new product announcements, new features, uh, new things happening in this space, uh, that I've ever had in a talk. So it's going to be a lot to show you all. And of course, this is only scratching the surface. You know, if you go out to that vendor hall, if you go out through the, the rest of the talks, there's so much happening in this community and so much innovation to take in. It, it's actually quite exciting. But I, I'm gonna start this tour and I will go from the bottom floor up. Uh, I'm gonna start with Kafka. You know, what, what's happening in Kafka? It's, it's really come into its own as the foundation of the streaming world. Um, is there more left to do? And I think the answer is yes. I think there's incredibly exciting things that are going on in this area, and it's really driven by the community around Kafka. You know, this is something that, you know, we have all built together, and there are hundreds of thousands of organizations using this technology. People are attending meetups all over the world, participating in forums, helping each other learn how to take on the next generation of application, and there's really exciting fundamental changes happening in the Kafka project, and I'm gonna talk a little bit about some of the things that we've done recently, I'll talk a little bit about what's coming up next, what's on the horizon, and even some of the future ideas that people are kicking around in Kafka itself. I'll start with what we've done recently. The big kind of internal rearchitecture project has been this removal of ZooKeeper, Kraft, the ability to have Kafka manage its own metadata on its own log. And this has been a pretty significant effort it has a whole range of impacts. It makes Kafka simpler to manage, it makes it more scalable in terms of the number of partitions and metadata operations it needs to, you know, actually handle and produce. And this is coming into a state of production readiness. It's fully supported in Confluent Platform. This will be the default in the future, and it will actually make it much easier to run Kafka yourself. And I think this is something everybody has been very excited about for a long time. And it, you know, really took a big effort across a large community of people to make this real and make it production-ready. We're very excited about what's happening there. So what's next? Well, there's additional core innovation happening. You know, equally big is work that's happening in the storage area, and this is work that really was pioneered by by a team at Uber that wanted to be able to tier out storage of data in Kafka out to S3, and they did incredible work on making that possible. Contributions from folks at AWS, from folks at Confluent, from a larger community that helped work on that design and code base. You know, it's gotten real production usage, and this is coming out into the mainstream of Kafka as well. So what's next? What's coming on the horizon? What do we do after that? Well, it's always hard to talk about a roadmap for an open source project, because of course, innovation comes from everywhere. Anybody has the ability to create a proposal or contribute a feature. But I'll talk about a few of the areas that we're interested in at Confluent, where we're contributing. This is really just scratching the surface of what's possible and what's what, you know, may happen in future years, but I think it gives a sense of some of the things that I think are exciting. So there's incredible work that's just starting out on really trying to improve the ecosystem of clients and the protocol. This is an area that's often a pain point. You know, there's a rich ecosystem of clients for Kafka, but they're relatively complicated and some of 'em are amazing and some of 'em aren't. And really bringing this up up to par, making these kind of equally good across each language, trying to work on the feature set that, that makes these possible to manage more easily. And this has become really important to us as we've started to run a cloud service. It, it's possible for us to make our cloud service, you know, kind of arbitrarily reliable, but we need it to be the case that it's equally easy for customers to, to manage their clients and not have to worry about that. And we found over time that's an area where a lot of the complexity can fall in terms of configuration, in terms of management. How can we simplify these clients? Well, a lot of this comes down to the underlying protocol for Kafka, and this is something that has been evolved steadily, always in a backwards compatible way, and is an incredibly important part of the project. In many ways, this is kind of, you know, the core contract that it provides out to the rest of the world. In some sense, you know, I think Kafka has become this de facto standard for data streaming, and the Kafka protocol is kind of like what HTTP is for REST. It's kind of like the default way of sending events around. And so really making sure that this is kind of set up for the next 10 years, that we can continue to evolve it in a way that's compatible, but actually starts to simplify what's needed out of clients. What, you know, what kind of things are exposed, that are internal to the system, you know, make it a better and better foundation for the rest of this ecosystem to build around. I think this is incredibly important. So, so I think this is really core foundational work. And building on that is something that I think is a lot of fun and is maybe a little bit heretical. I think if there's anything up here that's, you know, likely to get me burned at the stake, it's this. So KIP-932 there is a set of people who are interested at Confluent in adding queues in Kafka. So what does that mean? Isn't Kafka already a queue? Well, not quite, right? Not quite. It captures data streams. Data streams are inherently, you know, one linear ordered stream. You consume them in order, and that's a really powerful pattern. It lets you capture the changes coming out of a database. It lets you have strong semantics for stream processing, but it's not the only pattern for processing data. Sometimes you just wanna farm out tasks to, you know, as many dynamic workers as possible with as many different threads or instances, as much parallelism as you can to get all the processing done as quickly as possible. And you know, this has been the domain of these kind of more traditional message queues, you know, the worker farms, you know, this is something which you can definitely use Kafka for, but it hasn't been fully supported all the way into the depths. And so what this proposal does is actually really clever. I think it's really elegant. It basically takes the fundamental consumer group mechanism in Kafka and it opens it up as a mechanism for queues as well. And so it's not the case that you need to create a new queue instead of a topic. Instead, it'll be the case that all the topics can also be consumed in this way as queues, and you'll get the semantics of that. So it's a very natural generalization of what's there, and I think means it's probably one less set of systems you would need to run parallel to Kafka. So I think this is really exciting work. It's something a lot of people have asked for for a long time, and it's not all, you know. So if we go beyond this, out into what are kind of the ideas people are kicking around. And these are things that, you know, nobody's actively working on yet, but we'd like to. The kind of things that we're thinking about, you know, what should be in the future of this area? And I'll talk about a couple of these that I think are important. The first is trying to figure out how to organize all these topics. When we go and work with customers that are running at scale, we often find a complicated system of naming the topics that, you know, has different components that say which team it is and what group it is and what application set. If only there was some way to organize groups of data. Well, you know, there's, in the world of data at rest, there's been such a thing for a long time. We have directories, we have other mechanisms for kind of grouping or tagging or collecting things together. We haven't had that in Kafka. And you know, that means that people come up with a lot of ad hoc solutions. It makes it harder to reason about how you apply permissions, you know, how you actually structure data, what's public and private, you know, what belongs in one team, what's one domain of information. We need to provide something like this. And, and I, you know, this is not something that anyone has a concrete proposal for yet, but, but it's something that we're interested in trying to drive in the future. And, and finally, the, the last one I'll touch on here is, you know, coming up with a way of simplifying partitioning in Kafka. This is still one of the last internal details you have to kind of really think through publicly is how do you set the right number of partitions? What do you do if you got that wrong? Why can't you dynamically kind of split and merge these partitions? What if we could make it, you know, just easier to kind of auto-scale up those partitions or kind of take that away out of the calculus of the consumer entirely? This would be a nice foundational simplification simplification, and I think ultimately that's what's important in this core layer, is trying to figure out how to make it as simple as possible for the rest of that ecosystem to build around. So that's, I think, a pretty rich and exciting set of things that are happening. The things that are kind of there and out, you know, getting tuned and rolled out into production, the things that are being contributed and worked on as that next generation, some of the work that we're just starting, and some of the ideas that we're kicking around. And whenever we think about where does this come from, well, it's really clear, right? It comes from all the contributors, all the open source work across a number of different companies that help make this possible, that help do this work, that help make sure it's well documented, that help make sure it's well tested, that go through line by line and look at every bit of code that's contributed, make sure it's reliable, make sure that by the time it comes out, it actually works, that it runs, that it's secure, that it's something you can depend on. And so a huge round of applause and thank you to all the people who have helped contribute in this area over a number of years. And this is something that, you know, has really been enabled by broad commercial support. And, you know, as we were starting Confluent, there's really only two companies which were really helping to fund, you know, full-time development in Kafka. There was Confluent and there was LinkedIn. And there was a lot of people who were putting in time on the nights and weekends. And, you know, over time that's really changed. There has emerged a set of, you know, large contributions from a wide array of different companies. Companies that are all trying to make this system and ecosystem successful, and I think that's an incredibly powerful thing. So, you know, thank you to those companies as well. And so, you know, that's a little bit of what's happening out in the world of Apache Kafka. I also want to take a little bit of time and talk a little bit about what we've done in our cloud service, and I think this is incredibly exciting technology. I think it's been an awesome journey for us to go from taking a system that was, you know, built to run and be self-managed, and taking it out and turning it into a fully managed service. You know, really trying to be clear on what, what we wanted that to be, and try and make it that. And this, this has been a journey for Confluent that we started in 2017 with our first kind of production quality release, releases even earlier than that in actually kind of building some of the underlying technology. And the early version of what we had was pretty rudimentary. It was basically, you know, a bunch of Kafka clusters on Kubernetes with a, you know, billing, simplified billing system managed by us. But over the years, we've realized that's not exactly what a cloud service is. That, you know, in some sense, putting Kafka Kafka in the cloud isn't just about putting Kafka on some servers. You know, what people want out of a cloud service is actually a very different experience. It's something which, where you can just kind of get this fundamental protocol and set of capabilities on demand whenever you want it, as elastically as possible, where you can have flexibility in how that scales and how it's built. And this is something I think we intuitively know, that, you know, the right solution in this space is not exactly the thing that you had on-premise. And so, you know, when we think about storage in the cloud, it's not about, you know, having an NFS server stood up for each customer on some EC2 instances. If we think about data warehousing in the cloud, it's not about, you know, how do I just take my on-premise database and run that as a service service, one for each customer. You know, there's systems like S3 in the storage world, there's systems like Snowflake or BigQuery in the data warehousing world that have really taken those set of capabilities and reimagined, hey, what is a cloud service for that? And that, that's what we have been trying to do for Kafka. You know, we really think that building a Kafka service really imagine, you know, really requires rethinking from first principles what that can be. And you know, at Confluent, we've come up with a set of core pillars that we believe for our products are where their differentiation is gonna lie. And that's being, you know, truly cloud native, you know, trying to build something that was thought through from first principles as a cloud service, being complete, bringing together all the parts of this data streaming platform that I described, and making it available everywhere. Making it available across all the different clouds and environments that our customers run. And so, you know, in this section of the talk, what I wanna do is dive a little bit into this first pillar, a little bit into the cloud-native area, and a little bit into our journey in this space. So, you know, what do I mean by cloud-native? Well, it can sound a little bit fuzzy. I'm gonna make it more concrete. You know, one of the things we learned when we were going after this area was, it really is a very different problem. It's a very different problem to, you know, offer Kafka as a service for tens of thousands of clusters and thousands of customers. To do this, you have to design a system that's multi-tenant, really from the ground up at every single layer of what it does. You have to really be able to operate at arm's length from customers where you enforce strong boundaries and deliverables and limits so they always get what has been promised. Promise to them and they know exactly what to expect. Um, you have to do something that works well across the array of cloud networking technologies and most foundationally something that's built to be managed and operated by software and by computers, by machines, not, not just by an army of people. And I think this is fundamental to getting a cloud service right. At the same time, there are a few things that are a little bit easier. So, you know, a cloud service really only needs to run on 3 clouds. You know, it doesn't have all the configuration, it doesn't have all the different hardware profiles, it's kind of only installed once. It really is a very different set of design constraints to solve for. And the work that we did over the years was to really try and build a system from the ground up that was built to do this. And it, you know, it's something quite different, even though it ultimately serves the same protocol as the open source, it does it in a way that is really unique. And for a long time, we didn't really have a name for this system. It was just Confluent Cloud. But as we've grown more services in Confluent, you know, some of which I'm going to talk about, we really felt like, hey, we should give this a name. We should have something we call it. And so we started calling it Cora. And Cora is the core engine underlying Confluent Cloud offering of Kafka, and it really is a an incredible piece of technology. It, you know, was designed from the ground up for this problem. We've taken it and built something that runs in large clusters, that is run across 85 different regions, across 3 cloud providers, that replicates data between availability zones, between regions in the cloud using our cluster linking. that divides the capacity up into underlying cells and provides strong availability guarantees and limitations of faults within those limited cells. And it's done an enormous amount to create feedback loops and automated optimizations that take the actual operational data of how customers are running, what the load looks like in real time, and feed that back into how the system works. Into the placement of data, into the movement of partitions, into the dynamic quotas that actually, you know, hold performance to limits. And, you know, take this really down to the last detail in the cloud environments that we work off of. You know, try and really optimize for the networking technologies, for the machines that this runs on, and try and make something that's as efficient as possible and as resilient as possible. And, you know, this is something that we think in operations has probably about a 1,000x improvement over, you know, just running open source Kafka one cluster at a time. We looked at, hey, what's the resource investment for our customers if they take this and, you know, hire a team and do it? What is our resource investment to run these tens of thousands of clusters and that ratio is about, you know, 1,000x improvement. And in fact, this massive system run across these clouds is, you know, supported by a relatively small on-call team of core Cora engineers that designed and built the system. It really is an incredible piece of technology. And so what are some of the benefits of this for our customers? Well, it does pay off in the experience you get. In a set of ways that I think are core to the cloud. The first is in elasticity. You know, these clusters can scale up and down elastically and do so much faster than virtually anything else out there. The next is in resiliency. You know, we actually stand behind this technology. We offer a much higher SLA than other comparable services, an order of magnitude higher in terms of, you know, the kind of nines of availability that we provide. We compare that to those who are managing Kafka themselves, and it really is actually much better. And you know, that comes out of a lot of the automation work that I described. That stuff is not just about saving ourselves time, it's about making sure that we don't make mistakes because we're doing things accidentally, that we don't have things manually configured, that we're actually built to operate at scale. Underlying the storage layer is really intelligent tiering of data out between memory, between object storage, between local storage, and the migration of that back and forth. And this has enabled our customers to store more data, take advantage of this more elastically, not have to worry about the underlying capacity that's available to them. And you can see this in the adoption. So as we've rolled out these capabilities in each of the three clouds, we've seen the stored set of data that our customers have go up and up. And so all of these have been benefits for our customers, as well as for Confluent itself. And you might say, well, okay, it's more efficient, and it's easier to manage. That sounds really great for Confluent. You guys don't have to work too hard or spend too much money, but what about for me? What do I get? And it turns out that, yeah, that's not just us being greedy. Because we do that, we can actually pass on some of those savings to our customers. You know, because we have significant advantages in the operations, because we've done a great deal of work on tenancy, on efficiency within the cloud, we're actually able to offer very competitive pricing on this. And so one of the things we've started doing is what we would call a cost savings challenge. And so anybody who wants to understand their own costs, of running Kafka in the cloud? You know, if you have a team of people that's doing this, you have some servers, we found it's actually hard for people to estimate. Most people don't really understand what goes into it. They don't actually know that networking is usually the biggest cost in the streaming world. People is probably second, servers is third. There's a lot of other components to that, and you can actually build a very quantitative model of what that cost structure is, and, The exercise that we do with customers now is do a very quick spreadsheet on that. We, we have kind of a template we use, make sure that we're using, you know, your costs, not just the averages, and then give you a proposal on price that beats that. And this can be a substantial savings, especially for those who are operating at large scale and are spending a lot on this platform. And, you know, this comes out of the underlying efficiency of the system, but it's not just cheaper. It's also faster. So we've done incredible work to really drive, you know, better latency across some of the hardest workloads. And so here's a graph of, you know, a very small cluster where we took the exact same machines and with Cora and with open source Kafka. And to some extent, this is an unfair comparison 'cause of course Cora has a whole set of processes that, you know, check the health, that ensure durability, that do kind of miscellaneous correctness and observability work that we didn't have on the Kafka servers. But nonetheless, even in that small setup, you can see how this scales as you start to take it up the slope of higher throughput, of higher number of partitions. You know, overall we're able to keep, you know, very flat and predictable latency through all of this. And this is something that's continually getting better. So we have a team that continually works on Cora's performance. And in this next generation of technology, they're, they're doing work on the replication engine that drives this latency even lower. And so that, you know, this is all part of incredible work in this internal engine that the team has built. You know, it's not something that you can download or even that you could run outside of our control plane, but it's one of the things that allows us allows us to offer this cloud service at scale. And it's something we're gonna continue to invest in as we've, you know, continued to invest in the open source. And we think these things don't compete with each other. At the end of the day, you want both. There's a need for an open source thing that you can download and run, that you can have on your laptop, that you can kind of set up and run in your own company. And there's a need for great cloud services that were built to provide this in a truly cloud-native way. So that is a little bit of some of the innovation happening out in open source, a little bit of what Confluence is doing at this lowest layer in the stack, at the actual, you know, storage of the stream, the reads and the writes, all the actual flow of data. What about this next layer up? Is there anything happening there? Well, it turns out that there is. And so I'm gonna talk through connectors, I'll talk through governance, and I'll talk through stream processing. And I think these are 3 of the most key layers that you need in a data streaming platform. We found in virtually all our customers, as they get to scale with this technology, these are the areas that they build out. These are the areas where they're piecing together, uh, open source things, they're adopting our product features. These are the problems that they're trying to solve. And, and we see this kind of widely across everyone using Kafka. And so I'll walk through each of these areas. I'm gonna start with connectors. Now, you know, connectors are kind of humble but mighty. You know, nobody thinks much of a connector. It just plugs from one thing into another, but it's actually really important. In order to actually do all the fancy stream processing, in order to, you know, get the data that's gonna be, you know, replicated around the world in Kafka, you ultimately have to connect to some. And, you know, that's a problem that many people think is easy to solve. And, you know, at least in proof of concept, it kind of is. You can always get something connected to something without too much work. But to do it right is often a lot of work. To actually get the scalability and the fault tolerance and really get the semantics of the system right, it's kind of ongoing work. And it's not something that should be— have to be done over and over again in every system, you know, for every system in every company. It's something that you wanna share. And fortunately, there's hundreds of open source connectors that have emerged in the ecosystem around Kafka. And I think this is a really powerful thing. I always think of each of these as, you know, code that you don't have to write. And there's a lot of that code that you don't have to write. And this is an ecosystem that's continued to evolve and to grow with, you know, projects like Debezium and folks really in each of these technologies that integrate, helping to contribute and make these connectors better and better. And as part of our product at Confluent, we have taken about 70 of these and offered them as fully managed connectors where we really offer them in a serverless way. You don't have to think about like managing Kafka Connect tuning servers or picking out infrastructure it runs on. You just take them and, and they go. But we've always felt like this is incomplete. I mean, it's nice to have 70+ connectors, but what about all the rest? What about those internal systems in your organization that we don't have a connector for? Well, today we're announcing the, the JI Custom Connectors. This is the ability to take any connector that you have, something that you wrote yourself, something that you downloaded, something that you hacked together, and run it in Confluent Cloud. And this, you know, has the same ability to ensure the availability, to drive high performance, and just take away the operational burden of connecting to systems, but it now opens it up to this much larger ecosystem. And this is a ton of deep work to do well in our cloud to make sure that these connections connectors are secure and well isolated, even though they may have come from anywhere. But this, this is a foundation that we're going to be using for a lot of our custom code operations, and we're excited to open it up here first. So that's a little something happening in the space of connectors. Now I'll talk about something happening in the world of governance. So this is an area that many people are just starting to think about. You know, a lot of people are just starting to think about governance for any data at all, let alone streaming data, but for some reason, it's more important in the world of streaming, and I, I think it's ultimately because these streams go between systems, and if that data is not well governed, if there aren't schemas, if there aren't guarantees, if you can't reason about what it has, if you can't reason about compatibility, things are gonna break, and they're gonna break at scale, they're gonna break across teams, you're gonna have bad data, and ultimately, once organizations have solved some of the infrastructure problems, you know, once they've gotten reliable Kafka that can scale the way they need, that's secure, the next layer up the stack is how do I actually reason about that data? How do I govern that data? Who's allowed to access what? How is the data allowed to evolve? Where does it ultimately go to? And there's a set of features that we've added to our cloud product around stream governance that help companies solve this. This means, you know, managing schemas with our schema registry, so you can say what the structure of data is or should be. It means managing metadata so you can tag these schemas and say, hey, this is, you know, the owner of this field, this is PII, this is where it's going. It means being able to trace the lineage of data, as it flows throughout an organization. Where did it come to— from? Where did it go to? Did it flow into this other region? Is this supposed to be there? And then finally, being able to actually validate the integrity and check this as it comes. So these facilities have been part of our cloud offering for a while, and today I'm announcing some new functionality that expands this: data quality rules. So this is the ability to take your schemas and extend them beyond just simple semantic rules. So, you know, today you can say, hey, I'm expecting a number, or I'm expecting a string of characters. But oftentimes that isn't enough. Oftentimes what it means for data to be correct is a little bit more nuanced than that. You weren't just expecting any string of characters. You were expecting an email address, and an email address has a form, and you want to be able to check that. You weren't expecting just any number. You were expecting a phone number, and that has a particular format that you were expecting. And beyond just being able to check this, you need the ability to act on it. You know, what if something is wrong? Should it be rejected? Should it be sent to some dead letter queue for storage? Should it issue some kind of warning? What about evolving data? What about when you change a schema in some new way you know, what, what's the action that should be taken to kind of map the old data to the new? There's a set of functionality that we're releasing here that starts to broaden this notion of how you work with streams and make it easier for organizations to do this in the large. And this has been something that we've found our customers find essential once this technology moves beyond a single team. So that's a little bit about what's happening in the world of governance. and I'm gonna move on to, you know, last but far from least, in many ways what I think is, you know, the most important developing area in the streaming world, stream processing, and this is really natural. Once you have streams of data, once those streams of data are reliable and you can work with them at scale, the critical thing is how do I build my applications around them? How do I make it easy to get value out of them? That. Of course, you can do this, uh, on your own with the clients. You can just, you know, read and write directly to the streams. But I, I think people have found that there's technology that can make this easier, that if you're just kind of working with the low-level streams, you often find yourself repeating the same kind of patterns over and over again. And, you know, there's been a few different families of technology that have arisen in this space, you know, in And on the left-hand side here, you know, the kind of simple standalone apps, Kafka Streams has really dominated. You know, it's a simple library. You can embed it in any Java application. It takes some of the core stream processing primitives, and it makes it just immediately accessible to that application. And I think rightfully, this has been incredibly successful, but there's been a whole family of standalone frameworks that take over more than that. They actually take over the lifecycle of the application, how do you run it, how do you scale it, how do you handle faults, how do you recover from that, there's definitely power in that. That definitely provides some value. And I think in this family, there's a bunch of interesting technologies. I put a few up here on this slide. You could probably see, you know, half dozen others out in the vendor halls and in some of the other talks. But there really has been a technology that I think has kind of has come to the fore in this space and is kind of an emerging de facto standard, which is Flink. And th— this is a technology which I think has an incredibly important role in the streaming world. And one way to understand why this is so important is kind of by analogy to the world of data at rest. So, you know, if you think about how did stored data come about, how did we start out working with that, You know, at the lowest layer of stored data, you have file systems, right? And for a long time, that was what applications had to work with. They could read and write from the file system. It was a kind of linear stream of bytes. It's a bit low level, but it gets the job done. After all, there's nothing you can do with a database that you couldn't do with a file system if you wrote enough code. It's just hard. But as we built applications for stored data, I think we realized, hey, this is, this is a lot of work. There's a lot of patterns that we're kind of reinventing each time, and this is where taking, you know, some amount of integrated processing, combining it with data storage, building higher-level primitives that makes applications easier. This has been the role of databases and the rise of relational databases, other paradigms of databases. This is what applications tend to build on today, and this is what has made it easy to build around stored data. So that's, that's been the evolution of the stack for data at rest. What about data in motion? Well, I, I think it's kind of the same thing. In some sense, Kafka is the lowest level in that stack. It's the kind of, you know, hub of these streams. It allows you to read and write streams of data. It serves an incredibly important role across an organization as that interchange. But when it comes to the patterns for how you can build on top of that, I think there's a role for technology that that makes that easier. That makes it easier to build a scalable application that is correctly fault tolerant, that works with streaming data. And I think in many ways, Flink can be the kind of database layer for streaming. And if you look at the adoption of these projects, they've actually followed very similar trajectories. So Flink is a bit newer, and so, you know, it's followed the same kind of up into the right curve curve. In this picture, we overlaid some of the Kafka adoption data. You know, Kafka, we don't have the data going all the way back past 2016, but if you start them at a similar scale, you can see they're following a similar growth trajectory in terms of adoption. And I have no doubt that Flink is going to be at that same kind of millions of, of unique active users in just a few years. And you can see that already in the kind of leading edge companies that have adopted it. You know, Flink is used by some of the most technically sophisticated companies in the world, and this is the same set of companies who were the early adopters for Kafka. And I, I think that that tells us something. And the reason it's been so popular, I think, ultimately comes from the power and breadth of the underlying platform. It offers a set of APIs that range from declarative SQL to, uh, programmatic from batch to streaming, um, all the way out to, you know, stateful functions. And it does this across a variety of different programming languages, all on top of a shared runtime. And this allows it to span a whole set of use cases, from analytics to data pipelines to core applications, the kind of thing, you know, you might carefully write in Java or other code, you know, carefully test, carefully deploy, and monitor. And so this I think is the, you know, kind of underlying power of Flink. As we watched this community grow and evolve, and as we watched this technology develop, at Confluent we thought, hey, you know, this is a really important part of this ecosystem, and we would like, you know, to think about how we can fit with this. What is it that, you know, we could bring to our product? Is there anything that we could do that would we make an offering around this that would be worthwhile, that would be valuable, that would solve problems for our customers. And when we thought about what this could be, you know, we actually realized it followed a lot of the same patterns that we had thought about for Kafka. You know, building something that is truly cloud-native, a truly cloud-native offering of Flink, building something that's complete, that takes all these parts of a streaming platform and makes them work together and offering it everywhere across clouds. And so I'll talk a little bit about how we've approached Flink and bringing it into Confluent, into Confluent Cloud, and what we've done in each of these areas. So first, cloud native. You know, what we're announcing is an offering around Flink that we're gonna be releasing in pieces. The first part coming, you know, in an announcement today. And I think already it has some of the core capabilities that we would want in a stream processing layer. It has auto scaling, so you don't have to pre-provision a set of capacity. It auto-balances load and data, uh, across the processing layer. So you, you can start with absolutely nothing. It scales to zero, and you can issue a query and have it scale up with your data, scale down if that data goes down. Um, it's kind of always there for you and running. There's no setting up of a cluster. In fact, there is no cluster, uh, there's just a compute pool that you can manage those resources with. It's fully managed, right? We take accountability for the operations of this system, and it's upgraded automatically. There's no kind of pre-provisioning, and you know, it's on a full usage model. You pay for what you use. It, it comes out of the same kind of unified commit or pay structure of Confluent overall. Next is complete. You know, we want to build something that's not just a collection of parts, but is actually an integrated data streaming platform that brings together all of these parts. That's Kafka plus Flink plus the connectors plus governance that takes the kind of underlying metadata, security primitives, networking primitives, semantics for fault tolerance, billing model, and makes them work across all of these. And this is where I think that there's an opportunity to make this ecosystem much easier, to make it so that you don't have to spend your weeks figuring out how to plug different ingredients together, that you can just have stuff that works natively. And this is already true in the product that we're announcing today. You know, even in this version, all of your Kafka topics, you know, automatically show up in the metadata store for Flink SQL. You can see the full list of things there without having to do any integration work. These products already talk to each other. They're, they're already connected. And finally, everywhere. This is easy. We wanna offer this everywhere you wanna be, in any cloud, in any region. The initial release is in AWS, but we're gonna be rolling that out across the rest of the clouds as this goes into public preview and GA. So today we're announcing early access. If you're interested, sign up and we'll hook you up with access to this product. This is the SQL API. That's the first bit that we're getting out there. That will become GA by the end of this year and we'll be doing programmatic access in Flink for the other APIs in 2024. So I'm incredibly excited about this set of functionality. We actually added some folks to the team and I wanted to welcome to the stage Martijn Visser, product manager and Flink contributor, who's gonna tell us a little bit about some of the work that he's been doing on Flink. Martijn. All right, so maybe we could start with, you know, maybe you could tell me just a little bit about the team that you were a part of at Imrok and, you know, the community around Flink that was there and a little bit about how you ended up at Confluent.

Yeah, definitely. So first off, it's great to be here. Um, yeah, I was part of Immerok, which was acquired in January of this year, and we were a startup with 20 people who had worked in various companies. We originated at Data Artisans as well, and we had 4 of the top 5 Flink contributors. And overall, we have 16 PMC members or committers that are now part of the Confluent organization, and we're working really hard on making being available here. It's super exciting.

Yeah, yeah, I couldn't be more excited. This was a really lucky thing. We started working on a Flink offering, and of course we realized there was a startup company that was doing exactly the same thing and had, you know, incredibly like-minded people. And so it was really exciting to get to join forces. Your journey with Flink began really as one of its earliest users, you know, back when you were at ING. You know, I'd love to hear about that. How did you come to become so involved in the Flink world and what do you think made it so successful? You know, there's been a lot of stream processing technologies and frameworks out there. A lot of them have kind of died on the vine. You know, this is one that with a really small group seemed to, you know, really take the world by storm. Why do you think that is?

Yeah, so definitely when I was still working at ING, we were in the fraud department department and we needed to make a transition, right? We wanted to not notice fraud after the fact, but we wanted to prevent them. And so what happened is, is that we had already moved to a Kafka-based solution for making or having the ability to inspect all the activities that are happening on the website and on our mobile apps. But we needed something to actually— that allowed us to act on that data. And there are a couple of reasons why Flink was so interesting for us. So first of all, it was open source, so that meant for us that we could adapt it and modify it to our own needs. Another really important thing was the scalability of it, because it allowed us to scale as we were growing as a more digital company. The fault tolerance of Flink was super important in there as well. And I think last but not least, we have seen how the community has evolved over time from this really small group of people to this immensely great group of people that is now making Flink better, has made Flink what it is right now, and bringing it to the next level.

Yeah, the Flink community is really amazing. It's an incredibly passionate group of people that's really thought deeply about stream processing and what it means. You know, I think a really complete vision. Well, maybe you could tell us a little bit about what the team has been working on at Confluent and what you're up to now?

Yeah, definitely. So one of the things that I've heard is that the one thing that is harder to operate than Kafka is actually Flink.

Oh.

Now I don't know that much about Kafka, but I do know that operating Flink on a scale can be tricky. And it is something that we have been working on tremendously to make it available for the people out there. And I actually think it's time to show them a little bit about that.

All right, you better show it.

Thank you. All right, so as I said, I am super excited to be here. Nowadays, there are many examples of industries that have been transformed by cloud-native companies. Take new media, for example, going from video streaming to online gaming and social media. All these companies are depending on the cloud, and it allows them to create new digital ecosystems and audiences for all their products and services. In this demo that I will show to you, we will have a deeper look at our fictional video streaming service called Spa TV. Video streaming has become an essential part of everyday lives. If you would open up your favorite streaming service, it would similar— look similar to this. Content is organized by categories. Some of them might be curated by genres or comedy. Like comedy. Other categories could be more dynamic, could be determined by viewing patterns, your personal preferences, and popularity. Customers expect the service to learn about them and present with content that they want to engage with. This categorization is all powered by events and data streaming that I will show you in the next few minutes. So let me kick off with a demo and show how our streaming service is using Flink on Confluent Cloud. Over here we have 3 main environments. There's the integration environment that stores all of our raw events, and there's a quality of service environment and engagement environment that contains all of our business data. It's one per team. The integration environment is shared amongst different teams, and each team has their own cluster. We have a playback cluster which is used for capturing all video activities, and the website cluster is used for measuring everything that's —related to impressions that are being generated in our streaming app. For all business events, each team has their own environments. We have a quality of service environment which has one Kafka cluster running which contains all the quality of service metrics that are being generated. Next to that, we also have our engagement environment which has two Kafka clusters running, one for capturing all the core metrics and one for our— all the personalization-related metrics and business data. Now each environment in Confluent Cloud has a so-called compute pool provisioned. This is on by default. There's no setup needed to get started. A compute pool provides the resources used by Flink SQL queries. These can either be ad hoc exploration or long-running queries. Compute pools are a new concept, and it auto-scales up and down to meet the needs of the workloads using them. There's no need to configure, size, and scale Flink. The platform takes care of that. There's already a Flink query running over here. Now let's talk about data exploration. Data exploration all begins with a query. In Confluent Cloud, all the metadata is integrated to create one frictionless experience. Let's see this in action by demonstrating the integrated metadata model. First off, I'm going to connect to the compute pool. As Flink is serverless, there's no need for sizing workload or infrastructure. It just works. All the metadata is integrated. First off, we have a concept of catalogs in Flink SQL. Catalogs are mapped to the environments that we have in Confluent Cloud. In here, I can see the different environments, and we'll use the integration environment in this case. Next up, we have a concept of show databases in Flink SQL. Databases are mapped to the actual Kafka clusters. As you can see here, we have the playback and the website cluster, and we'll use playback in this example. Next up, in Flink SQL, we have a concept of tables. Tables are mapped to the actual Kafka topics in here, which we can also see in the interface. By doing a show tables, I see that there is one video view events. That's the topic. And it has a schema in there which contains all the information about titles and categories. And I can do a describe statement and immediately see the results of that schema in Flink SQL as well. Now I can also do some descriptive analysis by just looking at all the video acting— actions that are happening right now. That's looking great. So what if I want to do some descriptive analysis like measuring the amount of unique viewers? I do a count distinct and can immediately get the results. Do I want to get them per category? I just add a category and do a group by, and immediately I have the first measure of popularity that I can use to power our home screen. Now monitoring overall time isn't that interesting, so let's use a tumble window and group them over time intervals. As you can see, immediately the results are visible. Now what if I want to materialize this? I can create a table in Flink SQL which will create the underlying topic in Confluent Cloud. When the topic is there, I can do an insert into of my previous query, and it will insert the results into that topic. As you will see in a moment, the job is already running, and messages are going to be produced into the topic. So there's a couple of important things, actually two. Flink compute is separate from Kafka storage. It enables us to read from two Kafka clusters in different environments, compute pools can access all data in the same region. I also didn't specify any resources or parallelism. All Flink jobs are auto-scaling, and the autopilot continuously monitors and rescales so that there's always enough resources to keep up. Now let's take a look at two more advanced scenarios. So complex event processing allows you to derive new events based on patterns in our data. This is done with the match-recognize function, part of the SQL standards and built into Flink SQL. Let me demonstrate how we can detect videos that needed to rebuffer after starting. This query checks if an event occurred where a video was started followed by an event because the video needed to rebuffer. Rebuffering events could be a user problem or service problem, which would require an automated downstream response. You can see the results immediately on screen. Let's check out one more. A common scenario is that you have one data stream and multiple time-based aggregations. Without Flink, you would have to create multiple queries, one for each aggregation, and each query would read from the same Kafka topic. That results in increased complexity and additional costs. With Flink, you can read the data once, aggregate against multiple time windows, and fan it out to different topics. Let me show you how that is done using statement sets. We want to count video attempts across 3 dimensions in 2 time periods. So that means that I have 6 queries in total. They all are reading from the same topic and are applying the same filter. What we are first doing right here is we are creating our 6 output topics that are again immediately visible on Confluent Cloud. And with that, we're going to wrap those 6 queries into one statement set. Set. What happens under the hood is that Flink optimizes this so that it only has to read from that topic once. This saves on resources, reduces egress and costs. Let me summarize what we have seen right now. So I've shown you a brief exploration, data exploration example, some data enrichment and how you could create new data products. I've shown you the always-on compute pools and how you can use standard SQL in combination with the auto-scaling and auto-rebalance workloads to create a simple experience. It's integrated but separate, so you can query across environments and clusters. There's independently scaled storage and compute, and there's the fully integrated metadata, security, governance, and monitoring. And I've shown you some of the advanced language features with the match recognize in the statement set, but there's 150 more Flink SQL functions out there for you. So that you can, that you can use for powering your own business applications. I'm super excited about the Flink journey here at Confluent, and I welcome you all to become a part of the Flink community as well. If you're interested in learning Flink, Confluent now has two Flink courses available for you. Back to you, Jay.

Awesome. Thanks so much for showing that. Thank you. And thanks for all the hard work on building it. Yeah, yeah. Thank you. All right. So for, for those who wanna try it out, we're opening the early access now for Flink SQL on Confluent Cloud. You can take this QR code and we'll hook you up and we're excited to get you all using it. We'll be rolling this out through the rest of the year as it becomes a public offering. So we're really excited about that functionality. This is probably one of the, you know, biggest product initiatives at Confluent since we released Confluent Cloud itself. And it's really exciting to get to tell people a little bit about it. So that, you know, that's a little bit of my tour of the data streaming platform. You know, I think what's exciting about this emerging world is there's really innovation happening at each of these layers. I covered, you know, what's happening in the internals of our cloud with Cora. I talked about some of the cool open source work that is happening in the larger community. I talked about some of the product offerings that we're releasing in the connectors, in governance, in stream processing. But in many ways, the exciting thing isn't just this data streaming platform, it's not just the infrastructure and the technology, it's all the applications, it's all the things that sit one layer up, all the things that this allows you to do and the emerging ecosystem of technology that's coming to be around this. And a lot of these vendors are out there and you can go see them. In many ways, data streaming is becoming kind of the fourth estate in data. When we think about where does our data live, we think about where is it stored, right? Where does it go to sit? And maybe it sits in operational databases, maybe it sits in, you know, some enterprise SaaS applications, maybe it sits in some analytics stores and, you know, lakehouses, data warehouses. But increasingly, you know, when we think about data, it's not just where does it sit, it's about how does it flow. And that's what's so incredible about this emerging ecosystem that there really is, is, you know, a new pattern for data that's emerging and connecting all the parts of the company. This central nervous system that's connecting all the applications and kind of turning it into, you know, one fully integrated company where organizations can share across all the departments, across all the different acquisitions, you know, across all the different systems. And, you know, this is something that we at Confluent have been working with our customers for a long time to make real. And the nice thing about working with customers is they always ask you for one more thing. They always say, well, you know, that's nice, but I just have this one more request. And so we told them, you know, about all this functionality, and they said, you know, this is nice, but I don't just need to share data in my organization. I need to be able to share data with, you know, some partner organizations, with some of my vendors, with some of my suppliers. and when we first heard this, we thought, that's a little bit niche, right? Like, most people don't need to do that, but we kept hearing it, and we kept hearing it in every industry. We would hear it in travel companies about how they would integrate between airports and online booking and airlines, and we would hear it in financial services. We would hear it in retail. We would hear it in manufacturing about suppliers and supply chains, and, you know, it, it was really an interesting thing where organizations would describe to us, they would say, hey, we have this great data streaming platform, all the data can flow really wonderfully, and then we're building all this gunk to try and connect to this other organization. And eventually we talked to them and we said, you know, hey, what's on the other side? And they said, oh, it's Kafka. And we said, oh, so basically we have our Kafka and you have your Kafka, and we've built this giant pound of gunk to try and securely connect the two things together. Maybe there's a better way. And they came to us and they said, you should build a better way. And so one last thing in this talk, we have built a better way and we're announcing today stream sharing. And so, you know, everything that you have in— somebody is trying to share some data there. Everything that you have in Confluent Cloud, any of these topics, you can now opt to open them up and securely share this with partner organizations. This is something that is fully in your control. Um, you can do this in a way that's safe and that is well governed. It will use the same schemas that you apply to your data internally. You can quota and check the load that this will put on your cluster. And we think that this allows, you know, the world of streaming to extend beyond just the the internals of your organization out into the economy at large. And so with that, I promise that was, you know, the very last product announcement and the very end of my talk. And, you know, to actually talk about how some of this stuff is used out in real organizations, I want to introduce Confluence Chief Product Officer Sean Close, who's going to talk with some of our customers about some of the amazing work that they're doing. Done.

Thank you, Jay, for that, uh, compelling view of how Kafka is going to continue to evolve to enable data to flow broad and deep across every organization everywhere. But now Now, let's hear from a couple of pioneers who are bringing that vision to life. First up, we've got Michelin. Now, I'm pretty sure that everybody in the room is going to know Michelin as one of the world's largest manufacturers of tires. But what you might not know is that Michelin is also processing tremendous amounts of data as part of being a sustainable mobility provider to a diverse array of clients all around the world. They're powering really interesting business use cases. For example, predictive analytics on when a tire is going to fail and needs to be replaced before that happens, or route planning for when a truck rolls to use the minimum amount of fuel to traverse the, the route you need them to go across. But let's hear a little bit more about some of those business use cases and how Kafka and real-time data streaming are solving those problems at Michelin from Olivier Jauge.

Welcome Olivier.

Thank you. So Olivier has been at Michelin for about 20 years, started in software engineering, team leadership, enterprise architecture, and now you're the CTO for Michelin Experiences. Welcome. Yes, that's me.

Thank you for having me.

Thank you for coming. So many people in the audience You may know that Michelin is actually the company that invented radial tires about 100 years ago, but now it's obviously about a lot more than just tires. You guys are a service provider and innovator using data-driven approaches. How are you thinking about innovation for the next 100 years?

Well, you know, for a company like Michelin, so we make tires, so we are an industry, so we have 80 plants across the world. Our supply chain span across 170 countries. We have 3 R&D centers. So I mean, one way to help our business to innovate is to capture as much data as we can, process them in a timely and accurate fashion. So that's one way. But we are exploring new business markets. You know, Michelin always wanted to help people moving around the planet. So these days, it's all about sustainable mobility. So in that area, we are harnessing a ton of data from IoT sensors to help our customers be in that space of sustainable mobility. And we are helping them to create insights. When do they have to replace tires? Obviously, buying new Michelin tires. But not only. It's also to find the best route to optimize their fuel consumption. Or if you get in the EV, you know, electrical vehicle, that's new for our fleets customers. So how can we help them? How can they, you know, deal with battery usage? How can we make sure that their drivers go home after their day? So that's all about data and that's what we want to do to become, it's to become a data-driven company.

Yeah, that is a lot of data. So tell me about how Kafka has helped you along that journey.

So that journey started 5 or 6 years ago with a simple but very concrete and core use case. We were struggling to propagate our product reference. And at Michelin, the product is, you know, it's the little black round thing we call tire and that we love. We were struggling to make it available to all the downstream systems that would need it because we were using lots of technologies, so files, ESB, ETL, you name it. And we were lacking data retention. So we looked on the market, and 5 years ago, there was a new kid in town called Apache Kafka. We looked at it, and we did a proof of concept. And it worked. It worked so well that we did not have to make it any publicity. It immediately increased its usage. And every day now we see new use cases with Kafka, IoT, application in our plants, you know, exchanging data with our machines in real-time inventory. But we made a step forward in that journey when we deep dive in our supply chain orchestration engine. So that's a key core piece of our entire business. And we had a huge monolith that was a business process engine. And we decided a couple of years ago to rewrite it using microservices, streaming apps. And we completed that journey 3 weeks ago, actually. So it's live. It's working. Wow. Yes, that was huge, a huge step forward for us.

So it sounds like it basically all went smoothly, easy and smooth, right?

We could say that. But to be completely transparent, we I mean, we started that journey with, by installing our Kafka clusters on-prem, you know, in our data centers. It was easier for us. So we started to create those clusters and we, you know, we dedicated people to operate those clusters. But you know better than me that it's a nice but could be complex technology to operate. And we started to see some scalability issues because, Kafka was so popular that, you know, we had to increase our cluster, not every day but every month. And so we somewhat reached our limit to operate ourselves our clusters.

So I think you actually migrated to Confluent Cloud as part of an Azure initiative about 2 years ago. Tell me, why was it that open source Kafka wasn't, you know, doing what Michelin needed at that time?

I mean, we were reaching a point where we started to have outages. On our clusters that we were operating ourselves. And because Kafka was used in critical place of our business, we could not afford that anymore. So it became obvious to some of us that we had to find, you know, a managed service for that. And we wanted to offload to the cloud a portion of our Kafka workloads. Why I'm saying portion? Because we are, an industry, so we cannot move everything to the cloud. We have to put things in our plants. So we have a Kafka cluster in each and every plant that we have because we can't rely on the network. In case the network goes down, the plant still needs to operate. And we have sensitive data centers that are not in the cloud. But everything else, we decided to move it to the cloud. But where? So Azure was the answer for us because it was one of our partners, but which managed services. So we looked again on the market, and basically there was some competitors, AWS with Kinesis, Azure with Event Hub, but none of them were even close to your platform back in the days. I'm not sure they are close today either, but when we looked at your cloud offering, it was really the managed offer we wanted. With all the features we wanted and operating at scale. So we decided to go 2 years ago, 2021, we completed that migration and luckily, so touching wood, we haven't suffered any outages since then and we have optimized our running costs.

So it was a good move. Makes sense. And so how do you think about managed services and cloud as part of your journey towards a modern data platform? One more, Michel?

So if coming back to that migration, so when couple of us started to say internally how we have to move our workload or Kafka workload to the cloud, some colleagues yelled at us internally, "Oh, why the hell are you telling us after only 2 years you have to move? We don't have time, the roadmaps are full." You know the story. So there was a need for convincing. So this is where you guys came into the game So we decided to ask the guys that created Kafka, how could we migrate? Because that seems a huge task for our people, so everyone was scared. I think I could share an anecdote. I think we talked more about the migration, 9 months, than actually doing it. But it has to be done because people needed to be relieved that that was feasible. So we made that migration, and that's We are on the journey at Michelin to try to leverage the cloud as much as we can. But the real shift for us is to put, you know, data in motion. We don't want to have our data locked in file system or databases anymore because we want our information system to be as agile as possible and be able to react to the unknown. So event-driven architecture and Kafka are two key enablers there and, And that's a huge mindset shift in our company to tell people, okay, open up your data, put them in motion in Kafka, and that would be our first challenge. The second one is I see Kafka and data in motion as a unique opportunity to reunite the transactional world and the AI/BI world. So the transactional world is where the data are, and those guys, data scientists, S&Ts, BI engineers, whatever, they want those data that are locked here. If you put them in motion using Kafka, I mean, you won. So that would be our next challenge.

Absolutely, that is definitely the promise. So hey, we've talked a lot about tires, we've talked a bit about logistics, what about everyone else's favorite Michelin product, the Michelin Guide, restaurant guide?

Yeah, so we make tires and we recommend restaurants. I mean, things are all— Tires and restaurants, for sure. Yeah, tires, restaurants, it goes together always. So you would be surprised that we do make the Michelin Guide, and we use, it's powered by Kafka too, and Kafka really helped us to transform the guide. You know, it used to be a book, so we print a book, so all of our publication systems, it was optimized for printing. How to get into the digital age? I mean, through Kafka streaming, So if you go to a 1-star or 2-star restaurant, that's powered by Kafka.

I love it. Real-time fine dining. Yes, exactly. Thank you very much, Olivier. Thank you, Fabien. So thank you again, Olivier, for betting the future of data streaming at Michelin on Confluent. And speaking of betting, many of you in the room will be familiar with notable European gaming and betting brands such as Paddy Power, uh, Betfair, or Sky Betting and Gaming. And those platforms are all actually owned by Flutter, which is a $30 billion global enterprise. So now let's hear directly from one of Flutter's lead infrastructure engineers on how they're using open source Kafka and Confluent together to deliver on their complex real-time streaming needs across the multitude of platforms that they operate. Please welcome Callum Blanchard.

Thank you.

All right, so, uh, Callum, at Flutter, your team is responsible for all of the data-driven— sorry, the event streaming infrastructure at Flutter. And obviously you're operating in a very dynamic, fast-moving industry. How are you thinking about success within those guardrails?

So within our industry, uh, a lot of our business critical data needs to be as close to real time as it can get. Um, the things that come to mind there are things like bet placements, cash outs, and spins on our applications, because when a customer interacts with our platform, they expect that immediate feedback. Um, and so within Flutter, we use Kafka to facilitate this, uh, and we found it to be really successful because then that initial data gets used all across our platforms without any real additional work.

So it sounds like you've got a lot of inherent kind of real-time use cases in the business that you're operating in. What are some real-life examples of data in motion?

So we can enrich the bet processing stream or the bet placement stream, for example, with external data sources like real-time events from sporting events, such as football matches or horse racing. and we can then use that to update things like our pricing models, our bets, risk, and liabilities that we're exposed to, as well as maybe pausing things like cashouts while we verify an in-play transaction. These kinds of things, generally, we need to use a few different engines for, but Kafka kind of lives under the hood there. And we were spread out between about 30 different Kafka clusters in total. They're spread out between 3 environments, so our production one, our disaster recovery, and our test environment. And we're also kind of working in the cloud, you know. We've got a few Kafka clusters spun up as self-managed Kafka, as well as also MSK. But as Jay mentioned earlier, the difficulty is you can't really just lift and shift things from on-prem into the cloud. It sounds really easy, and it's not always as simple as it, uh, as it can be.

Yeah, for sure, it's always complex. So you're also working in a really regulated industry. How does that play into the considerations you have?

Yeah, the gambling industry is quite heavily regulated in the UK, and there are a lot of safeguards in place to make sure that customers are able to use our platform safely. So things like setting deposit limits or self-exclusion if they, if they feel like they need to, as well as things like affordability checks, and just to make sure that people people are able to gamble and bet in a way that's sustainable and safe for them. But it's also our responsibility to make sure there's no criminal activity happening on the platform as well. So things like fraud detection or anti-money laundering as well. And again, Kafka plays quite a big part in this. One of our kind of first use cases actually for Confluent Cloud was the cross-brand self-exclusion project, which allows us to, allows customers to self-exclude from all of our platforms with a single click. This was quite a big project that we delivered to, first of all, you know, get ahead of some regulation, but also because we really value customer safety.

Makes a ton of sense. So how do you think about managed services and cloud as part of your infrastructure? You already mentioned it's a pretty complex picture.

Yeah, I mean, we're a year-round business, right? But we do have 3 big events in the UK and Ireland that really put pressure on our systems. Those are the Grand National and the Cheltenham Festival, which are 2 horse racing racing events, and also Boxing Day football as well. And because of the way that we've kind of scaled these things, we're scaled to that peak load 365 days of the year, with the majority of that capacity being unused for the rest of the time, right? For example, on the Grand National Day, we had, I think it was 2.5 million bets placed, and we get nowhere near that amount of traffic, you know, the rest of the time. And we have this sort of hybrid cloud model, currently, as mentioned before. But we do have a hosting strategy in infrastructure, and that's to move towards public cloud and containers wherever possible. So when we assessed our, both our existing AWS offering, as well as what we expect to be doing in the future with new projects, we knew there was kind of gonna be a bit of a gap with what we currently manage and our team's capacity as well, right? We're quite a small team, and we want to be able to scale up as much as we can without any additional or much operational overhead. And so for us, being able to scale our clusters down as well as up was really important, and we found that Confluent Cloud was able to facilitate that for us.

Yeah, makes sense. So your team is ultimately providing event streaming infrastructure across all of the Flutter group, and I guess it's to accelerate the progress of the other teams. How do you think about making developers' lives easier?

So as the team that managed Kafka, we're not particularly well-versed in actually using it. So we're very much the administrators for the entire company, and we want to continue to do that. You know, we want to be able to centralize as much as we can, and just make developer experience as easy as possible. So when a developer gets, or needs a Kafka cluster, they don't need to set up the cluster itself, they don't need to create the security, the topics, anything like that. We can provide you the bootstrap server, we can provide you the API keys and the ACLs, And we want to configure that as code as well. Because it's really easy to get carried away with just manually doing, you know, one thing after the other. And losing where you actually are. So if we can use something like Terraform in a more repeatable pattern, we've managed to get to a kind of end-to-end deployment process using, using config as code. So currently with Confluent Cloud, we can create environments and clusters. We can create the private networking into our AWS. Kafka Service Accounts, we can create topics, ACLs, you know, we can create service accounts and API keys, and we can even actually then integrate those API keys with our secrets management. So it really is a full solution that we've managed to create for our customers.

Yeah, so developers can just expect it to be there. I really like that. So obviously you— Flutter's been very successful with open source Kafka over many different years. What types of things do you look to Confluent for?

So the main thing we looked for Confluent for on-premise at least was, we had these 3 projects that we really wanted to deliver on-prem. So they were the improved geo-replication and disaster recovery, as well as self-balancing partitions. And we also wanted to kind of provide those additional services such as Kafka Connect and KSQL DB, maybe now Flink as well. And the first 2 of those are really not trivial to create, right? There are open source solutions, so things things like cruise control or the MirrorMaker v2 for geo-replication. But they would have been quite a lot of work for our team again. And when we looked at Confluent Platform, both of those things were kind of built in natively. So we had self-balancing clusters as well as cluster linking for geo-replication. And the third one, we've had a schema registry that we've, that we've managed for the last few years. But ultimately, the rest of the things that customers rely on, things like Kafka Connect or KSQLDB, they need to manage themselves. And they do a really good job at this, but again, we know that this is time taken away from doing the work that delivers real value. So again, we want to be able to create and provide, like, a managed pattern to be able to allow customers to do this without much overhead on their side. And for us, that's where Confluent for Kubernetes really makes a lot of sense.

Yeah, it makes a ton of sense. So any final advice for those people who are earlier on their streaming journey than you guys are?

I think if you're new to Kafka, don't expect that everything's gonna become, you know, event-driven overnight. It's very much a journey, but if you start small and kind of find a use case that's really well-suited towards Apache Kafka or event streaming in general, every subsequent additional event stream in that becomes much more easy to implement. But start with the baby steps and you'll get to the sort of nirvana of event streaming by the end.

Yeah, I think that makes sense. I don't think setting your data in motion is a destination. I think it's a journey. I think you have to get all that way. But thank you very much, Calum. That was great. Thank you for sharing that.

Thank you.

Thank you. So before we close out today's session, I wanna take a brief moment to highlight some of the key things we've talked about today. First up, you heard us talk about some of the amazing enhancements that are coming to Kafka over the next year and beyond, including tiered storage and queues for Kafka. Second, we unveiled Cora, which is Confluent Cloud's native, elastic, scalable, high-performance Kafka engine that is in Confluent Cloud today and is the product of over 5 million engineers engineering hours. And third, you heard Jay talk about how when you surround Kafka with governance, connectivity, and stream processing, you get something much more powerful. You get a complete data streaming platform, and related to that, we made a couple of announcements. We talked about Custom Connect, which is the ability to load your Kafka Connect connectors into Confluent Cloud, and we talked about data quality rules. Which you can use in governance to ensure that the data meets your standards and is reliable and usable. And fourth, we talked a lot about stream processing and about how having streams is one thing, but the ability to process them, to mix them, remix them, filter them, enhance them, evolve them, and build new applications of all types really takes it to the next level. And we did a deep dive into Flink and a demo of the now EA version of Confluent Cloud's link product, so you can see exactly how it works. But you know, obviously this summit is really about the opportunity to hear and learn from experts, but also from each other and be inspired to take back new learnings, new opportunities to take data into motion across all of our different organizations. And so we'd encourage you to take advantage of some of the sessions that have been programmed for you over the next 2 days. There are over 70 different sessions. There are 2 panels, 15 lightning talks, 53 breakout sessions. There are over 10 sessions just on the Kafka core, and there are over 10 sessions on Kafka streaming as well, including 3 deep dives into Flink for those of you who are interested. Some of the highlights for me would include Apples and Oranges, comparing Kafka Streams and Flink, which is by Bill Bajek, and that's happening later today. And also Apache Flink on Kafka: Reliable Data Pipelines That Everybody Can Code by Ella Demir, which is taking place immediately following this keynote. So with that, thank you so much for joining us this morning, and we hope you enjoy the rest of the summit. Thank you.