Event-Driven Architectures | Jay Kreps, CEO, Confluent (Overview for Technical Leaders & Executives)

Hi, my name's Jay Kreps, and I'm the CEO here at Confluent. Today I'm gonna introduce you to some of the key trends and concepts that are at the heart of a revolution that's happening in data infrastructure and application architectures. I'm gonna tell you about events, stream processing, Apache Kafka, and Confluent, my company. First, let me start by introducing Confluent and the open source project Apache Kafka. I helped create Apache Kafka with my co-founders, and Confluent was the company we created to help the world adopt it as a product. Confluent offers the leading distribution of Apache Kafka and is driving the majority of the contributions to Kafka and much of its surrounding ecosystem. Kafka itself is one of the most popular open source projects in the world. It's a technology that scales from a single application to company-wide usage, and it's used in production in tens of thousands of companies. Confluent was founded by the original creators of Kafka in 2014. We're backed by some of the best venture capitalists in Silicon Valley, and we're operating globally, transacting in over 20 countries worldwide. All right, so what is Kafka? Well, it's what we call a streaming platform, but what's that? What's the motivation for this technology, and what's the problem that it solves? It's a technology that's built to let organization harness streams of data, and to react in real time to the events that drive their business. You can think of it as a database for storing, and reacting, and processing infinite streams of data instead of just finite tables. People come at this area from different points of view, and I'll give 3 lenses for thinking about Kafka, each of which make more or less sense depending on your background. The first is by analogy to Hadoop in the big data area. You can think of Kafka as being like Hadoop, but made fast and continuous. Unlike Hadoop, which stores static files and does big batch processes at the end of the day, Kafka stores continuous streams and does continuous processing with millisecond latency. This enables it to be used not just for analytics, but to support operational workloads that directly run your business, and power customer experiences. A second lens is to view Kafka as the evolution of the category of enterprise messaging. Traditional enterprise messaging systems are being displaced by Kafka at a very rapid rate, as Kafka can operate as a unified, global, elastic, cloud-ready infrastructure that spans your entire company. Kafka does everything these older systems do, but it's built on a modern distributed systems foundation, so it scales horizontally. And it gives you a lot more power and productivity in how you build applications on top of it. A third lens is to see it as the evolution of the ETL and data integration space. Unlike traditional ETL, Kafka allows building around continuous real-time streams, and it can connect not just off-the-shelf systems, but also allow custom applications to tap directly into the same data streams. We're bringing these areas together in a way that's really powerful, and it's all imagined on a platform built around real-time streams of data. Kafka is a technology that was created here in Silicon Valley, and it got its start running at massive scale in companies like Netflix, LinkedIn, and Airbnb. If you were to pick a technology company in Silicon Valley and ask them to draw their architecture, they would probably draw a diagram with an Apache Kafka cluster at the center of it. Since those early days, Kafka has spread well beyond tech companies, and it's been adopted by large enterprises in every industry you could imagine. Companies ranging from Goldman Sachs and Audi to the New York Times are using Kafka as the foundation for essential parts of their business. To get a sense for how these companies use Kafka, let's look in more depth at a few use cases in three very different industries, each of which is going through a significant transition as they focus on reimagining their company in software. This will let us see how events and event streams are at the heart of the modern architecture these companies are evolving towards. The auto industry is one of these industries. It's moving away from a world where car companies shipped a hardware product and their customers were dealerships, to a world where they are now offering an integrated hardware and software product and an ongoing service that powers the interaction with their customers. The combination of connected, self-driving, and electric cars, and even cars that are operated as a service, is a huge transition that's gonna be happening over the next decade. And the largest automotive companies in the world are betting heavily on Kafka as a foundational layer to enable that. One critical part of this is enabling connected cars that continually send streams of data on top of which a rich set of features can be built. To do this, they need a global IoT platform that takes gigabytes of data per car and operationalizes that data to create new features, both in the car and within the firm. This way they can tell you there's an accident coming up ahead, they can gather the mapping data to power self-driving cars, they can enable predictive maintenance that tells you to get off the road before your car breaks down, as well as the analytics on the performance and the parts from different suppliers. In an entirely different industry, we have Royal Bank of Canada, which is the largest bank in Canada with more than 16 million customers. Banking is another industry in transition. Much of the customer's experience of a bank is shifting from a visit to your local branch to a digital experience on a browser or mobile device. In their initial use cases, Kafka enabled RBC to unlock data stored in mainframes and integrate the online experience in real time. As they began adopting Kafka, they quickly realized the value of offering this capability as a service across the multiple lines of business they had. They saw the fastest adoption of a software stack they'd ever seen in their company, with 37 applications adopting it within 6 weeks. They were able to accelerate a series of batch-oriented capabilities from 2 days to seconds, enabling new, better services for their customers across the globe. Meanwhile, the retail industry is going through a sea change of its own caused by online retailers, which are already heavily using data and distributed systems infrastructure to power their businesses. Here, we're able to use a central streaming platform to unite the brick-and-mortar part of the buying experience with the online buying experience. A Kafka-based streaming platform is allowing retailers large and small to respond to point-of-sale transactions in real time, manage and optimize an up-to-the-second picture of inventory, and ensure the customer experience is optimized across all of these. Thank you for listening. I hope this has been a useful introduction to how real-time streams of events are shaping industries and the role that Kafka and Confluent are playing in the transition.