EP 372: Maximize the Power of AI With Data Streaming

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

When it comes to generative AI, sometimes it seems like yesterday's data may as well be last year's data. Right? I think for especially enterprise companies and for all of us interacting with them. Right. If you're chatting with a large language model on a big company's website, you need today's data. No, you need this hour's data. You need this minute's data. And that's where data streaming and the intersection of data streaming and AI become so important. So that's what we're going to be talking about today and doing a deep dive into not just what data streaming is, but kind of how it helps power AI that we all need and we all use. All right, so that's what we're going to be talking about. I'm excited. If that sounds like something you want to know, well, you're definitely in the right place. So if you're new here, thank you for joining us. My name is Jordan Wilson and this is Everyday AI. This is a daily live stream podcast and free daily newsletter helping us all learn and leverage generative AI to grow our companies and careers. So if you haven't already, please go to your everydayai.com Sign up for the free daily newsletter. We will be recapping today's conversation and a whole lot more. Everything you need to know. Speaking of everything you need to know, let's start off as we do every single day, with a quick recap of the most important AI news. So Nvidia has launched an open source AI model called nvlm, challenging industry giants. So Nvidia's release of nvlm, the family of multimodal language models, is a significant development in the AI landscape as Nvidia will now be competing with established proprietary models from companies like OpenAI and Google and, and the open source or open weights leader in Meta. So the NVLM D72B, that's. That's a mouthful. It showcases exceptional performance across both vision and language tasks, enhancing its capabilities in text only tasks, which is a notable achievement in AI development. So Nvidia's decision to make the model weights publicly available, along with plans to release training code, mark a pretty significant shift toward open source practices in a field dominated by closed systems systems. So benchmarking comparisons, early ones for Nvidia's new model show it performing competitively against leading models such as GPT4 Claude 3.5 and Llama 3. All right, speaking of models. Yeah, tons of model news. So Google is working on a reasoning based model similar to OpenAI's 01 or Strawberry. So according to reports from Bloomberg, Google is making significant strides in the development of artificial intelligence software that mimics human reasoning, a move that intensifies, intensifies its rivalry with OpenAI and their recently released O1 preview model, what a lot of people refer to as Strawberry. So multiple teams at Google are reportedly making progress in AI reasoning software, which excels at solving complex multi step problems in areas like math and programming. So the company's efforts come in response to OpenAI's recent advancements, particularly the launch of its O1 model known internally as Strawberry, previously as Q Star, which has raised concerns within Google, Google DeepMind's team about falling behind. All right, last but not Least, speaking of OpenAI, they secured their $6.6 billion funding round while reportedly restricting investment in competitors. So OpenAI has successfully closed their $6.6 billion funding Round from some pretty prominent investors, including Thrive Capital and Tiger Global, marking the largest round of funding ever at 6.6 billion. So OpenAI has requested, though according to reports that investors refrained from funding five of its competitors, including Anthropic XAI from Elon Musk and company Safe Super Intelligence ssi, as well as two AI application firms in Perplexity and Glean. So the exclusivity. The exclusivity agreements could reshape the VC landscape, concentrating funding around fewer larger companies and potentially stifling innovation among smaller startups. However, this ambitious strategy from OpenAI may open it up and attract some increased regulatory scrutiny and could push competitors to accelerate their innovation efforts to remain relevant in the market. All right, that's a lot in a very short amount of time. Don't worry, we're going to have all of that in today's newsletter and a lot more. But, but today we're here to talk about data streaming. Like, what the heck is it and why is it so important for both big enterprise companies and for all of us, right, functioning in our daily lives. Because now it's like we need, you know, all big companies to have data and to have chatbots, just like we need them to, you know, have a website. It's kind of the same thing, right? So enough of me chatting. I'm excited for today's guest, so let me bring on to the stage. There we have him, Will La Forest, the global field CTO of Confluent. Will, thank you so much for joining the everyday AI show.

My pleasure. That's, that's, it's not a trillion dollars, but it's a lot of money.

Yeah. My gosh. Right, yeah. Or, or 7 trillion. Right? Those, those, those numbers.

I knew it was in the trillions.

Yeah, it's wild. So let's, let's start here. Well, first of all, Will, thank you so much for joining and you know, for our live stream audience. Hey, it looks like LinkedIn streaming's back. That, that's great. So if you have questions for Will, please get them in now. But let's start at the top. Will, what the heck is data streaming?

Yeah, yeah, absolutely. Sometimes people call it event streaming as well, but essentially it's a, it, it's a category of data technology that's really created to handle the continuous flow of data, right? So as data is generated, changes occur, it's published, and it can be continuously processed, enriched, filtered and acted on. And so this data, or the derivatives of it then can flow to any number of downstream consumers that are subscribed and at massive scale. So, like, great example of one of those consumers that we'll be talking about today, I'm sure, is things like a vector database to support rag architectures, which are really, really critical. I think the easiest way to explain this, this technology to people who are not like data nerds, not data infrastructure people, is to, is to use an example, right? So if you think about Uber, for instance, if I'm a driver, you know, I, I have this phone and I'm moving around like it's constantly sending these events, this data. It's like I'm Will, I'm in a Mercedes, I'm at this specific location and I may be with the passenger or maybe not, right? A second later I'm a new place, and a second later I'm someplace different. Okay? If I'm a writer, same thing's happening. Like I'm walking around trying to get to the front of my hotel or, you know, walking in a terminal of, of an airport. And so the most obvious thing is, with these constant streams of data is like if I ask for a ride, it needs to match me with someone, you know, Uber. And pretty much every ride share company does the same thing. You need to match riders with drivers, right? If you don't do it fast enough, like, I'm gonna get pissed off and say, okay, let me check lift or let me just take a taxi. I mean, we all know like consumers now are like super temperamental. I said, time is money, okay? The Other thing I'd say is like those streams of data, like if you're a data driven company and you're applying machine learning, like that data needs to flow to all these other systems as well. So like Uber was, did a lot of early work in traditional machine learning. They had this platform called Michelangelo. But like that data's flowing there. They're looking at like how do I affect surge pricing to like squeeze as much money out of us as possible? You know, how do I look at driver behavior ratings, you know, when do I add more drivers like that. All that stuff has to be done on the data and the faster they can do it, the better the business. So same thing with I would say AI, like the fresher the data, the better the results.

Yeah. And Will, I do want to kind of, for our more non technical audience, I want to put this into perspective because you, you know, you mentioned vector databases and embeddings and rag. Right. Retrieval, augmented generation. We why are these things so important, right? When, when people think, oh, you know, I can just slap, you know, use an OpenAI API or I can use a Google API, throw a chatbot on my company's website and we're good to go. Right. But you need that layer of real time data about your company or what your customers need. So can you explain this concept of RAG and vector embeddings and why this is ultimately important when we're talking about generative AI and large language models?

Yeah. Oh, I liked how you started the, the conversation with saying like, you know, a data an hour late is super late. I think that was the, the time increment you were talking about. I would argue seconds late is too late. So I mean a great example is, imagine the best way to think about this is you have these amazing, you know, general purpose large language models that are trained to really effectively understand all these facts and communicate with you, the consumer. But if I am an airline, this is a great example. If I'm an airline industry and I'm providing a chatbot, okay, you, it's not possible at the moment to train a model. So that's constantly up to date to the second. So if I log into some app and say, oh, what's the status of my flight? Well, there's no large language model that was trained a year ago, a month ago, a day ago or even an hour ago that's going to have the relevant information. Right. So you need to actually provide that to these models that you were describing, whether it's GPT or LLAMA or whatever. It doesn't matter. It all works the same way. You need to provide the information so that I can, you know, reason with. So it will like for instance, say, okay, well this person is Willa Forest, he was on this flight. Here are all the bookings and this is the status of the airplane that all has to be provided when you ask the question. Right. So that's, that's sort of the criticality of real time data is that businesses that are looking to use these large language models, they have to bring it to the table, if you will, when you ask the questions because they're not trained on all this up to date information.

And we kind of just got straight to the, to the heart of today's conversation. But maybe will, if you could explain a little bit like what the heck does confluent do? And you know, we all assume it's in the data streaming, but maybe just to describe real quickly what it is that you all do.

Yeah, so I would say Data streaming about 14 years old now, so relatively new. And I would say data infrastructure world compared to like databases and mainframes. They're still out there. People. I know where this is an AI conversation but. And it was created at, by our founders at LinkedIn just to solve this problem. How do we deal with massive scale data and do it in real time? Okay, Confluence was founded about 10 years ago and really the idea is how do we take this technology that the, the, these super giant technical companies are all using the Ubers and Netflixes and LinkedIn's of the world and how do we make it possible for mere mortals to use it for their businesses? And so you know, that's, that's really been our mission to make it as easy as possible to people, for people to do things with real time data. And this has never been, I would say never been more important than before with generative AI for you know, just the everyday person, if you will, the everyday business. So that's really our mission. We've had all these pieces on top of it, but that's the general idea.

Yeah. And so you know, obviously you all work with enterprise companies and most enterprise companies have had their kind of data game strong for many decades right now and many companies have been using artificial intelligence for many decades. But how does this kind of, you know, even you said, quote unquote, newer concept of data streaming, right. Been around for a decade and a half. How does this change what's possible for everyone else, right. Even for, you know, medium sized businesses, mid market companies, right, like can you talk a little bit about just how much data is truly available and you know how even those medium sized companies that are growing can start to leverage this data streaming concept and pair it with AI. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Genai. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and built a straight path to ROI on gen.

Yeah, that's actually a great question because oftentimes when people hear about like these like massive scale projects going on with data streaming, like, well, clearly this is not for me because like my data just doesn't change that fast. I don't have that much data, I'm not uber. Right, but if you're doing generative AI, it all gets back to time, right? Time is, is, is absolutely critical to how quickly you can make use of data that's in these databases. So let me just step back. Traditionally the way data worked is it's really been done the same way for, I don't know, five decades, six decades. Take your data, you stuff it into some place and then you ask a question and it gives you back a batch of data, okay? And then you do something with that data. The and, and really nothing's changed. I mean we've added like new ways to do it faster and do more data and make it cheaper. But it's the same principle. The problem is is that doing things in these batches immediately introduces this time it takes to actually make use of the data. So if I'm a, let's just say I'm a business and I have a customer set of data that lives in a database, right? And you know, I'm constantly updating, oh, I Have a new customer, like here's his address or whatever. Like you want that data as soon as it's changed to be available to these downstream systems. And if you use sort of the traditional approaches, you're basically, you have to do these batches and usually you do them like on an hourly basis. Every hour I'm going to take all my customer data and I'm going to move it to my, to this vector database I'm using to provide context for my generative AI. So now you move to this model that every time I update my customer user base, you know that that is immediately sent to my destinations and is immediately enriched and provide context, etc. So, so that's, that's really where I say data streaming helps mid level. It's not, it's not necessarily about the skill. Yes, it can be used for massive scale so you can grow like, like OpenAI is a customer of ours, they do massive amounts of data. Right. But it's about the time.

So, so, so Will, you kind of gave us some great examples and I love those by the way. Right. Like Uber needs to know whether I'm in the front of my building or the back of my building. Right. You know, we need to know if that Uber has taken a left yet. Right. But so some great external use cases or use cases that the everyday person might, you know, need or rely on data streaming. But I like that example that you just gave there. Right. Like a customer, you know, changes their, their address. Right. But I want to, you know, maybe kind of explain how data streaming is not just, you know, an external thing. Right. Not just for me to know about Uber, but also for internal teams to know. Right. Because I think when we think of large language models, I think unfortunately we just think of, oh, I'm a consumer using this on the company's website. So how does data streaming and AI help internal teams work with more real time data? Because I can see in those instances, yeah, last hours data could be very bad to have right now.

Yeah, well not only that, but you know, generally speaking, if you're using a public model, which almost every, you know, medium sized business, they're not training up their own large language models. They typically aren't even using the open source models and trying to operationalize it. You talked about like llama, etc. So those models don't even have your business information. They know nothing about your customers and you don't want them to know about your customers either. Like you don't want to send that stuff. There's like sensitive Information. So you have to provide that to the model when you ask a question. So you know, actually it's kind of interesting because if you think about businesses like startups, maybe not as much, but medium sized, large businesses, I would say the first use case for Gener is just increasing the efficiency of employees. Right. And, and this means like, and I do this all the time, like internally we have all this information. I want to be able to ask a question rather than like search all these different silos of data. Right. And it is, and the only way that works is if it, if it's that information is provided to the large language model at the time that I asked the question. And yeah, these, these are internal sets of data. This is like customer information, employee information, product information. So that's a very, I would say a very common use case, especially even for large businesses. If you're a bank or you're a healthcare and you know, company, it's actually kind of hard to start using these generative AI stuff for your customer facing stuff. There's just so much risk. But you can use it for internal information relatively risk free. Right. Because there's always going to be a human in the loop. Right?

Yeah. And that's exactly what I was just getting to.

Right.

Because we just talked about all this, this promise of data streaming and marrying it with generative AI and how that can be so impactful both for real front end users and for internal teams. But maybe we'll talk to us a little bit about those risks and challenges because it all sounds great, but what happens when data streaming goes wrong?

Never goes wrong.

Numbers don't lie. Right.

Well, I do like to say, and I don't think again, if you're not a data person, you're not familiar with the category. Like it's not, it's not an exaggeration to say that let's. That if data streaming one day just woke up and had a bad, bad day, like it didn't drink its coffee, that literally the, the global financial server system would just fail. Like because like all these banks, the way money transfer behind all these things, there's data streaming, like it's just everywhere, it's ubiquitous. So like data streaming is super rock solid. Right. So but of course anything can fail. But like I think the bigger concern is especially if you're a big business where it's always a risk reward, like if you're a small business, you don't care that much about potentially what happens if it goes wrong. You're like, okay, well maybe I'LL just shutter my doors and just start up a new company. Right. But like if, if you're a bank and you provide false information to Wall street or you're, or you leak patient information, that's a pretty big deal. And so I think where it could go wrong, I think this is where I. One of the things that confluent does is really important on top of data streaming. Data streaming fundamentally is just about delivering data really, really fast and acting on it. But the question then is if I'm using generative AI and I get some sort of response, how did it achieve that? Response, how do we ensure the wrong data doesn't get to the model and then get turned sent to the wrong people? Right. Those are really important things. And so that's where this, there's this category called data governance which is all about as this day is being produced. I need to look at the data, make sure that like PII information doesn't get sent through. I need to make sure that you know, PCI information that's a banking standard for, for payments, like your personal information doesn't get through. So, so data governance is important. And the second thing there I think is if something does go wrong, and this is where like all the big generative AI players have to look at this medium, the medium guys do too. Like how do I know what went wrong and when? Why? So let's just say I enter in some prompt to get some crazy ass hallucination that leads people to make bad business decisions. My customers, how do I. If all this data is in real time, it's not in the model. How do I know what that was? So you need this ability to trace what's called the provenance, the source of the data that was made in that decision. And I will say where I think we're going to be going in large length like right now, that doesn't really exist very effectively in the large language models themselves. Essentially they're trained on the entire Internet. Right. And it doesn't really track like where the pieces of information are coming from. They kind of don't want to, to be honest, because it opens up a can of worms, which I'm sure you've talked about in podcasts.

Yeah.

So that's another place where it can wrong. Like you need to know how these answers were provided.

Yeah. And speaking of that, a great, great question here from our audience from Cecilia. So she's asking Will, how do you ensure that data is truly coming from diverse sources so that you are getting complete information? How do you discern who is providing the data or maybe what that. Is that data actually telling the full story?

Yeah, yeah. I mean, that is a great question and it is, it is a challenge, that big challenge we have right now in terms of applying generally I, in an ethical manner. So I mean, first of all, I wish, I wish I had the magic incantation to solve that problem. I think I'm going to speak to how do you know you're getting the complete information and who is providing that data? So again, I think the data that's provided to the large language models, these generative AI models, you can actually track what that is. So, so this is basically, I'll just simplify the way these things typically work. Someone enters a prompt, you take that prompt, you get a bunch of other data, you throw it on top of the prompt, then you send it off to the model and you get a response, right? All that extra data that you're, that you're sending to the model, you can track that and you can know what source it's from and you can know when it was created. All that is what's called lineage information. So it, if to have ethical generative AI, we have to do a very good job of tracking that to ensure that you're not getting biased results and that when you do get biased results, they can figure out what the problem is. So hopefully that kind of answers the question, no?

Yeah, it does, absolutely. So, you know, one thing that I, I kind of want to, you know, share, shift toward and I, I never put, guess really in the hot seat or ask you to tell the future, but you know, one thing I'm always talking, talking about and wondering about is just this availability of data. And even for, you know, smaller businesses, which we kind of already referenced, you know, but you know, five years ago, small businesses weren't thinking about using AI until generative AI came along and kind of democratized that, you know, might there be a similar, you know, realization when it comes to even data streaming? Right. So where do you kind of see your industry headed as this generative AI landscape is changing so, so quickly, which brings more, more and more people in using models and wanting data. So where do you see it headed?

I mean, I think what, there's a couple things we're going to see. One is to be honest, like the average company is probably not going to be directly be dealing with data streaming themselves because data streaming has become so, so I would say inextricably linked with how generative AI is being done, that you're going to be using a service that makes it incredibly easy for you to do this without ever having to touch data streaming underneath the covers. Like that service will be doing it for you. I mean there's lots of great examples like take something like a notion AI. I don't know if your listeners, probably some of them are familiar, like underneath the covers they're using data streaming. If I'm using notion for productivity reasons, I don't need to know about how they're using data streaming. So, so that's one thing you're just going to see these services do a much better job at connecting to your data sources and doing this all phrase so you don't have to do this crazy rag vector database stuff. Like even that is too much work. It's a lot easier than it used to be. Like you said, like tremendously, but that's a lot more work. Second thing I think is we're going to see a lot more industry specific integrations, training services take place. So if I'm a healthcare company, there's going to be companies that specialize in how do I deal with patient information. That's a very hard problem right now. You know what I mean? So I think we're going to see a lot of industrialization, energy, healthcare, financial services happening already, but it's going to become mainstream.

All right, so will we've, we've covered so much in a very short amount of time, right? From, from data privacy and governance to how data streaming is, is changing, both internal operations, external expectations. But as we wrap up here, maybe what's your one most important piece of advice for business leaders out there, you know, who are really wanting to maximize the power of AI with data streaming.

I mean this is so cliche now, but it's so true. Let's just, just remember this. Garbage in, garbage out. Like large language models are great, but if you want to use it for your business, you have to have high quality, well governed data that's always up to date. Because as you said to start the show, an hour late is far too late, Right? You're just going to piss off your customers. Right? I don't even want to see a data a second late to be honest. Like I get, I get cracked, I get cranky, right? So making sure you get high quality data that's fresh and up to date is absolutely critical. So like the first thing in any major, if you want to make use of generative I. It's a data problem. It's always a data problem first.

All right, well this was I think whether you're a data geek or just trying to better understand data streaming and how it. How it impacts us all, I think today's conversation was an especially important one. So thank you so much for taking time out of your day to join the Everyday AI show.

My pleasure.

All right. And hey, everyone. Yeah, we just all got, like, some kind of degree and data streaming that was great. If you weren't taking notes as quickly as I was in the background. If you hear, you know, pound, pound, pound on the keyboard, that's me getting our newsletter ready for you. So, yeah, we covered a lot and it's going to be a lot more. Some of the most important takeaways. So if you haven't already, please make sure to go sign up@youreverydayai.com if this show was helpful, tell someone about it. Yeah, I know Everyday AI might be your little secret to make you the smartest person in AI at your company. But share the wealth, please, if this is helpful. If you're listening on the podcast, leave us a rating. Subscribe to the podcast and go to your everydayai.com. sign up for that free daily newsletter and we'll see you back tomorrow and every day for more Everyday AI. Thanks, y'.

All.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.