#222 [Radar Recap] Scaling Data Quality in the Age of Generative AI

Hi, I'm Maggie, VP of Curriculum at datacamp. And I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

All right, all right.

Hello, hello everyone, and welcome to the final session of the day of Data Camp Radar on scaling data quality in the age of Generative AI. So of course, as organizations continue to embrace AI and machine learning, the importance of maintaining high quality data has never been more critical. There are arguably no better people in the data business across the board than Bar Moses, Prakal Pasankar and George Fraser to come talk to us about data quality. So first I'm going to introduce Bar Moses. She is the CEO and co founder of Monte Carlo, a pioneering company in data reliability and the creator of the data observability category. Monte Carlo is backed by top VCs such as XL, GGV, Redpoint, Iconic Growth, Salesforce Venture and IVP. Bar, it's great to see you.

Thanks for having me.

Next up is Prukalpa Sankar. She is the founder of Atlan. Prakalpa is a leading modern data and AI governance company on a mission to enable better collaboration around data between business people, analysts and engineers. She has been awarded the Economic Times Emerging Entrepreneur of the year, Forbes 30 under 30, 40 under 40, and the top 10 CNBC Young Businesswoman of 2016. Prakalpa, it's great to see you.

Thanks for having me.

Then last but not least is George Fraser, CEO at fivetran. George founded fivetran to help data engineers simplify the process of working with disparate data sources. He has grown fivetran to be the de facto standard platform for data movement. In 2023, he was named a Dataname Person to Watch. He also has a PhD in neurobiology.

George Retcio, great to be with you.

You know, it's safe to say that data quality is at the top of mind of many data leaders today, Especially with the generative AI boom that we see. How would you describe the current state of data quality within the industry, within organizations today? What do you think are the common challenges organizations are facing when it comes to maintaining high quality data bar, I'll actually start with you.

I have lots of opinions on this topic. Trying not to hog the entire time. Yes, data quality. Well, frankly, let me start by saying it has been a problem and an issue in the space for the last couple of decades. So nothing is new, right? We've been complaining about data quality for a long time. We shall continue to complain about the quality of our data for a long time. However, I do think a few things have changed. First and foremost, you know, obviously generative AI products being more prevalent, at least in terms of the desire to build them, data teams are put under a lot of pressure. We actually put out a survey that showed that we surveyed sort of a bunch of Data leaders and 100% of data leaders were cited as under pressure to deliver generative AI products. No one said they are not being asked to build something. However, only 70% of them, just under 70, 68% of them actually feel like their data is ready for generative AI. So that means that while there's a ton of pressure from C level and board and others in the market to actually build generative AI, no one or the large majority of people don't think that their data is ready for that. And I think that poses a good question for us as an industry to figure out why that is the case. And my hypothesis is that what I would call the data state has changed a lot in the last five to 10 years. So the way in which we process, transform, store data has changed a ton, but the way in which we manage data hasn't changed at all. And so that means that if you go back to the survey, actually 50% of those data leaders still use manual sort of approaches to data quality. And so while we've become a lot more sophisticated with what we demand from our data and from our data infrastructure, we have not become more sophisticated in how we manage data quality. I think manual rules will and always be important, but they're not the end all be all. In fact, it is just the starting point. And so I think, in short, if I had to respond to what is the state of data quality today? I think it's sort of an old problem with new challenges that we have not cut up yet. I definitely have ideas on how we need to solve that, but I'll pause there for a minute and see if any reactions from my esteemed fellow panelists.

Pukalp, I'll let you react here.

I think I agree with everything that Bar said. But I think one thing to abstract This a little bit over. I think about this concept of data trust more than just data quality or like, and maybe this is the reason you have the three of us in this panel. But like, you know, the way I think about this is like if you think about that final layer of trust and you have a human who says this number on the stashboard is broken. Oh my God. Or it doesn't look right, like what's wrong? It sounds like a very simple question. It's actually a very difficult question to answer because the reason a number could be off could be because the 5, 10 pipeline that they broke and then run, it could be because it never happens.

What are you talking about?

Or it could be because the data quality checks that they failed. It could be because someone changed the way we measured annual recurring revenue and no one remembered to update the data consumer. If you think about this flow, I almost think of it as you have data producers who want to guarantee trustworthy self service data. No data producer wants to spend their time answering the question of why your number is off. And on the other hand you have data consumers who actually want to use data. Like no one actually cares about the quality of the data. Like they actually just want to use the business. Like a data consumer cares about making business decisions. And in the middle we have this gap. And the reason we have this gap is because we have a prolific, it's almost self created problems. We have created significant number of tools that have scaled massively, but we have a proliferation of tools. We also have significant diversity in people. So any single final dashboard probably had five people touch it. This problem just gets worse in the AI era. So at least if I was a human, I look at the number and I'm like, maybe the number doesn't look right and I can do something about it. If I'm AI, I don't do that. And that can actually lead to pretty significant decisions. So I think the way we think about this and we sit on that layer between the producers and the consumers and bringing this stack together is what does it mean to create these data products? Finally, what makes something reusable and trustworthy? And how can you bring context across from the pipeline, from data quality, from all of these layers in the stack, like human context to solve the trust problem or the gap.

Okay, that's really great. And George, I'll let you react to both.

Framework is right that there's a lot of layers to the system and it matters a lot where the problem is arising. Except the part about 510 rates. But I mean you would be. We try very hard to avoid contributing to the data quality problem in our layer. You would not believe the amount of effort that goes on behind the scenes to try to chase down the long tail of replication out of sync bugs that can happen with all the systems we support. We are not perfect. I can only say that we are better than everyone else. I think where it happens is very important in terms of troubleshooting. You, you ask like why despite all the efforts in this, is this still so top of mind? And the answer is because it's impossible to fix. Data quality is not a problem you can ever fix. There are so many layers and there are so many ways for things to go wrong. And sometimes the source of some of the later stage problems that Procalpa was just talking about are things like you can't actually 100% know whether the person who checked out at your point of sale system was actually the same person who created an account on your website. So you can never actually get to 100% with data quality. You just have to manage it and you have to identify what are the highest priority areas. Where is it most important that the numbers be right? Prioritize those, work on those. But I think you've got to start out acknowledging it will never be perfect.

Yeah, no, I agree. I think the thing on that is kind of what you said, right that I think things will always like the reality of running, like especially real time dynamic data ecosystems is that things will always break. Like there's, it's likely that there will always be things that are. Because it's, it's just the nature of the beast. And so that's why I think a lot of our, like when you think about trust, trust doesn't actually break because something went wrong. Trust breaks because someone told you, your stakeholder told you that something went wrong without you telling them actually something went wrong today. Maybe you should like. And I think that's the element of trust, which is it's one layer above something. And I don't think the solution is trying to make sure nothing ever goes wrong. The solution is how do you go one level above and make sure that you solve for trust and then how do you measure and manage it?

Trust is word. It is very hard to win and very an example of this I heard a long time ago. It's funny, I'm in New York right now and I met with somebody earlier today at Bloomberg actually I still had the iced coffee that I got while I was there and long time ago. I don't know if you know what Bloomberg is, but they do data feeds for finance, the kind of data management. But it's, it's data that, you know, like stock prices, commodity prices, gas prices, things like that. Many years ago, when fivetran first started, one of the things I learned is a key element of their business is, is that it is accurate. Even in this obscure case, you know, the price of beans in Korea or whatever it is. Like, even the most obscure data feeds, they are more accurate than anybody else. And that is really important because if one thing is wrong one day out of the year, that is a huge problem. That's something we've always tried to emulate at fivetran in a very different context, replicating a company's own data. But it speaks to how when you're in any kind of data business, you can be the difference between zero errors and one error is bigger than the difference between one error and infinity. Trust is so hard to win and so quickly lost.

Let you react and then I'll ask my next question.

Oh, I was just going to say.

Just reflecting on this, I wouldn't be surprised if we would be sitting in a panel in 10 years from now still having sort of similar discussions, except the words change and definition change. So maybe we call it trust or data quality or whatever now, like hallucinations in the context of generative AI, right? But the problem remains the same. I think one of the sort of interesting questions to answer is like, what are the challenges that our customers today are faced with and how are they dealing with that and how is that different from flow years ago? Or honestly, just like a year ago. And the reality is like these problems are just not going away. And so figuring out how to address those in a way that adapts to where our customers are and meeting them where they are is I think, super important.

I want to pick up on that point because while you alluded to meeting a panel in 10 years, I'd be very excited to have a Data Camp Radar AI Edition 2034 where we discussed this. And what it seems to me is that the goalpost on data quality is shifting every year, right? So like, organizations make strides, they make investments, but then the ambitions of what it means to have like high quality data also shift with that. So the challenges also still remain the same. So you mentioned this bar earlier in the discussion. Is that same problems, different challenges. What are those challenges today?

Yeah, great question. And I mean, I'll start by saying, look, like if you could in some world, like if model output is wrong, or you're prompting with a question and the answer is wrong, and is it better to not have an answer at all? Is no data better than bad data? Maybe? I think so. But also then what's the point of having a Q and A or chatbot if it can't provide you an answer at all? And so to your question, the definition of good, what does good look like? Actually becomes tricky. And how do you define what should we strive for? Changes, I think. But to your particular question, what are the challenges or kind of pinpointing those? I think kind of how I alluded to how the data state has changed over time. I think historically what we've done when it comes to trust, to Prakalpa's point, was really start with can we figure out about data issues before anyone else downstream learns about it, Whether that's in generative AI or not, whether it could be in a dashboard? So the thought is that if we can catch issues before others downstream do, we can either repair that trust or rebuild that trust. I think what we're seeing right now is that is definitely a very important challenge to address. And I think detection capabilities have evolved to a certain degree. I sort of talked about manual solutions for that versus not, I think sort of the big kind of like next leap here for building data quality, data trust, whatever you want to call it, is sort of going beyond detection and taking the next step of sort of understanding how do you actually resolve, how do you actually address these problems? And when you think about the root cause of these challenges, that has changed too. And so in the past, most teams really just focused on sort of cleansing the data. And once you cleanse the data and it was fine, you brought it in, you're all good. 5chan never breaks, so you were okay. And in today's world, 5chan still doesn't break. Sorry, George, I'm picking on you. But if you look at the data landscape, it's become really complex. It's like super, super complex for even for a small team to manage. And I think if you think about the core pillars of what makes up the data state, I would call it, there's three things. The first is the data itself. So actually like the data sources, whatever kind of you're ingesting. The second is the code. So code written by engineers, machine learning engineers, data scientists, analytics engineers, et cetera. And the third component is the systems or the infrastructure, basically the jobs running, all of that. So you have multiple teams building multiple complex webs of all three of those things. The Problem is that data can break as a result of each one of those three. It could be as a result of the data just that you ingested being totally incorrect. It could be the result of bad code. Bad code could be like a bad join or schema change, or it could be system failure. I won't name names, but systems do fail. It could be any elt, general ELT solution they use. And so actually understanding that in order to really build reliable products, you have to look at and understand each of those components. You first of all have to have an overview and visibility in each of these components. And then also understand can you correlate between a particular data issue that you're experiencing and say a code change or an infrastructure change or anything like that. That is really, really hard to do today. What ends up happening is that data teams are inundated with lots of alerts or quality detection, data quality issues. And they're all flying around between 20 to 30 different data teams and 10 different domains. And go figure, who needs to address which problem or which alert you? Actually, down to the brass tacks of how do we handle this? Those are some of the challenges that I think really figuring out how do we both have really strong detection of issues, but then how do we go to the next step and actually figure out what is the root cause? And honestly, oftentimes it's more than just one root cause. So it's typically this storm with like every single thing breaking. Like it'll be both a data code and a system issue. And so when I think about how our systems can get more sophisticated or how we build more reliable data systems, it has to have a more sophisticated view of what's actually, what are the various components of that and what could break.

That's really great. And maybe George, from your perspective, adding on top of what Bar said, what are the challenges that you're seeing today when it comes to scaling data quality or like improving moving the needle on data quality?

Well, I mean, we look at a very particular slice of this. We look at the replication piece. Does the data, the central data warehouse, match the data in the systems of record, be it a database like Postgres or Oracle or a app like Salesforce or Workday. And we've come a long way with just centralization of effort, just the fact that 5chan pipelines are standard, so everyone is running the same code. There's this cumulative thing where over years we fix bugs. That's the main tactic at the end of the day that we use to identify and squash Data integrity issues. We are experimenting with some new ideas to try to get that last little bit. That last 0.1% is very hard. They include exciting idea right now is the idea of doing direct sampling for validation. So when you're in the business of replication, data quality can be seen as you basically just need another sync mechanism that you can use to compare against. We've done a few iterations internally, we've shipped things and these are all running in the background. These are not things you see as a fivetran customer and where basically we pull samples of data from the source or the designation and compare them just to create a totally out of band mechanism to verify. And we've discovered, for example, we discovered a floating point truncation bug when we write CSV files for loading into data warehouses by doing this. And we think there are more things out there that we could discover and fix by doing that. And then the other side of this is at some point we want to make these capabilities customer facing because there's a lot of phantom data integrity issues in our world. We get a lot of reports from customers where they're like, oh my chain is broken. The system doesn't match and sometimes they are right. We do occasionally hit bugs, but a lot of the time there's something wrong with the comparison that they're doing. And it doesn't mean that we just tell them to go away. We have to figure it out. We have to verify that it's false alarms. We get a lot of false alarms at 5 trans. To the extent that we can build tools for quickly proving or disproving the concern where we're thinking about that too.

That's awesome. And then Prakalba, from your side of the equality island, what are the challenges that you're seeing today?

Yeah, so the way I think about it is I think of it as a three step framework. It's actually very similar to health. I think like generally like life health, it's awareness that's the first step. The second step is cure and the third step is prevention. And I think if you think about each of these steps like awareness, for example, what we're seeing with customers is we like we have a ton of customers who use us with FiveTrans Monte Carlo. And I think like for example FiveTran we were like the metadata API that came out. Now we have customers that say let's pull out context on what's happening and send out an announcement directly to my end users, which is red, green, yellow. Is the pipeline run or did it not run? Did it run as I expected? Stuff like that. We have the same thing with anomaly detection on the. So the stuff that the data producers know, can we share awareness to end consumers and end users? And in a way that's easy for them, it's in their BI tool, it's in Slack. It's this like green announcement that says red, green, yellow, right, like stuff like that. That's first step. Can we create awareness of where we are? The one big change we've seen is this move to this concept of a data product where I think some of the most furthest ahead teams are actually taking all these metrics and metadata and converting it almost into a score, which is a data product score. Let's create a measure of. If you don't measure, you can't really improve. What's the measure of reusability and trust? As I think about a data product, I mean I've been super surprised by how quickly that adoption has grown across our customers. The second one, Cure Bar, alluded to this collaboration. I think that's the most broken flow that exists right now because cure is a solution between business and data producers. Both need to come together. So there's a mass like there. I think we have a lot of work to do when we come back. Maybe not in 10 years, like we come back in even like a year. Like we've made significant progress in collaboration. And the third is prevention. I think the biggest piece here, like we're seeing a lot of adoption around data contracts. And so how do you take what you learned in awareness and cure it, but also make it something that's more sustainable over time? And I think that that's actually where there's been a, a bunch of innovation. I think like we launched a module, but there's been a ton of innovation over the last, the last some time. And hopefully all those three things together actually get us to a point where we solve for data trust. And I really hope my vision for this is like in a few years it becomes a really boring problem that we're not talking about. It's that it just, it's there and then we keep improving it. But it's not a problem that we should have a topic of conversation about. It should become table sticks. So the simplest way of this is how do you help a producer and a consumer align on an sla? That's the best way that we're thinking. And so what do you believe are the four rules for data quality? Again, it's a little bit more of a collaboration problem actually more than it's a technical problem, which is what do we agree on is our core layers of this is what we believe. And then how do you translate that into the actual data producer workflow itself? That's the best example of what we're seeing customers turn the center.

And there's one thing that you mentioned, Prakalpa, which is on the collaboration side that I think is very important, which is that data quality is often a cultural issue as much as it is broken pipelines or like something happening on the data collection side. Can you walk us through maybe the main cultural issues that lead to poor data quality and expand on that notion a bit more and like, what can organizations do today to shift their culture, to prioritize data quality? So you lead with that and then I'd love to listen from the remaining panelists.

The cultural thing, right? I actually think of it similarly. Like, first, if you believe in good intent, which I would like to believe that most, like everyone actually is trying to do the right thing for the company to a large extent.

No one's trying to destroy data pipelines.

No one's trying to like, no data producer wants to, like, ship something that like, breaks and then spend time. Like, nobody wants to do that. Everyone wants good intent. So I think the first step is really, I think just shared awareness and shared context. So first, I remember this. I remember once I got like, I used to be a data leader in my previous and I got this call from a stakeholder and they were like, number of this dashboard doesn't look right. I remember, like jumping on my bed, I look at my dashboard and there's a 2x spike overnight. And I'm like, oh, my God, this is crazy. And even I in that moment had a question on like, oh my God, did something break or is my data engineer not doing his job? Even I had that because I couldn't just open up airflow and look at the audit logs and see what happened. Like, I just could. So first step, the reality is that five different people with their own DNAs and they don't fully understand each other's work. Unlike any other team, sales teams are very homogenous. Data is not. It's very heterogeneous. So first, how do you create context, shared understanding, awareness across the ecosystem in a way that everyone can understand. Second, can you measure it? Which is why I'm super excited, because if you create a measure, then it becomes very easy for people to move towards. So I think, as you think about, so how do you measure it? Is the second. And then the third is actually then just the process flows. It's tooling process flows like iterative improvement. That's actually the easy part of the problem in my mind. I think the first two things, for example, even shared context. George said this, right? It's very easy to lose trust, but at that time, nobody says 99.5% of the time it was accurate. You see the one time the number was broken and it breaks trust, right? And so then what's the shared understanding? What are we defining as trust and how do you solve that human problem? I think the best examples of this is we've seen people actually have folks who understand both culture and humans and data drive the charge on building that initial people call it governance, standards people, whatever you decide to call it. But that initial shared context and understanding is I think the first step to good culture.

George, I'll let you react to that like maybe what are some of the levers that you've seen that can improve on cultural level to improve data quality within an organization? Maybe getting inspired from five trend customers here.

Oh my gosh. I'll let you know when I find one. A lot of data quality problems have and have origins in like poor systems configuration and those are really hard to fix. If I have one piece of advice for early stage founders, it is keep an eye on your salesforce configuration, because if that thing gets out of joint, man, it is hard to fix. It is a real grind trying to make progress on this. A lot of it consists of going upstream to the systems of record and improving their configuration so that they're not generating a zillion duplicate accounts and stuff like that.

Yeah, I can attest to that. I can attest to that. And then bar, from your perspective, culturally, maybe how do you move the needle as a data leader?

Agree with what Prakaba and George said. Maybe the perspective that I can add here, I think the companies that we are seeing make progress, it's due to a few reasons. The first is there's both this organizational top down and bottom up agreement that data matters and that the quality and trust of that data matter. If it's just one direction that typically fails. So if there's like a CEO of one of the Fortune 500 banks gets upset every time they get a report with bad data and so they actually made it a C level initiative to make sure their data is ready, clean to the best degree that they can, et cetera. That obviously creates some pressure, creates real initiatives in the business, real metrics to Prakalpa's point earlier, that is not sufficient. It's very important. But the business teams, if you will, business analysts, but also the data governance teams in large enterprises, there's various. It could be the centralized data engineering platform, all those people need, all the different teams are all stakeholders in an initiative and need to care about it just as much. For those teams, oftentimes the motivation is that they're spending most of their days in fire drills on data issues. And by the way, I saw someone sort of asking, can we clarify what is a data issue? I think that's a great question. Bad data can look in various forms and its symptoms are so different. But generally the way that I'm thinking about this is if you look at some, some data product, whatever data product that is, again, it could be pricing recommendation, it could be a dashboard that your CMO is looking at. It could be a chatbot. And if you're looking at it, you look at the data and it's very clear to you that the answer is wrong. Maybe the most, you know, an example from the last few weeks from this was from Google. I think someone searched how do I keep my cheese on my pizza? Or something like that. And Google recommended you can use organic super glue. That's a great way to keep from Reddit. Yeah, exactly, that's. Yeah, exactly, that's right. And so that is a good example of bad data. That is one very public. And I think it went viral. And maybe Google can get away with that, but many other companies can't get away with that. So there was an airline that actually provided the wrong discount on an airline ticket. And so a consumer purchased a ticket at a different price and actually sued that airline and got the money back. And so there are really real repercussions to putting bad data out there. And I think going back to your question about culture, I think both the teams working with data have to care about that. Now, the thing is that they don't always do because they don't always understand where is your data going. So if I'm building a data pipeline, I don't necessarily understand who's using that data and why, which makes sense if I'm way upstream. And so oftentimes I find that the companies who have made the most progress are those are able to bring together those teams under a unified view of where we want to go as a company. Oftentimes that can start looking at just how many data incidents do we have, how quick are we to respond, like, what's our time to detection of Those what's our time to resolution? And then, you know, taking this a step further, oftentimes our teams putting together SLAs between each other, so the SLA for particular data to arrive on time or to arrive in some complete state, et cetera. So I would say kind of the focus on metrics, I agree with Prakalpa that that typically drives the right behavior or drives some behavior, which is better than none.

Okay, I couldn't agree more. And then maybe you mentioned something focusing.

On metrics will drive some behavior. I agree with that.

I'm going to get that tattooed. And then one thing that you mentioned, Bar, is the Google example, right? Because I think this is a perfect segue into the nuances of data quality when it comes to generative AI. You mentioned that survey at the beginning of exactly 100% of data leaders are under pressure to deliver generative AI use cases. That does not sound surprising at all. So if you're a data leader, if you're in an organization trying to build a generative AI use case, what are the data quality considerations you need to have? Are they different from the general data quality considerations you need to have? Like what are the nuances you need to have? Like what are the nuances when it comes to considerations of data quality when it comes to generative AI? So Bar, I'll let you know. Continue on that.

Yeah, I mean, I think, look, if I think about sort of the state of generative AI with enterprises today, you know, I mentioned from the survey, 100% are under pressure, by the way, 91% are actually building something. So we've almost all, almost all of us have succumbed to the pressure for whatever reason. And I think when we say we're building with general AI that can take various definitions. So I'll give you an example. Just last week I spoke to one, one enterprise customer who told me we have the full entire sort of like tech stack for generative AI built with all the best in class. We're fully ready to go. We have no use cases or we don't really know. We don't have anything that's tied to a business outcome that we can point to. But the tech stack is ready, we're ready to go. And then another customer, that said we have 300 or so business use cases laid out. We have some great ideas for how to drive business.com. we have nothing on the tech stack. We don't even know where to get started. And I think that represents a spectrum of where customers are at. It can be anywhere on one versus the other side or in the middle. I think there's more questions than answers at this point. A lot of people are experimenting or sort of in the early days of building things in pilots. Some of it is also in production. But I think early days. I think by and large, across all of these instances, companies understand that they have to make sure that the data that's actually serving those LLMs has to be accurate. And here's why. Today everyone has access to the best models. The models being built by 5,000 PhDs and $1 billion in GPUs, we can all access them. There's no competitive advantage for a company with them. Where does a competitive advantage lie? It's actually with the proprietary data that you can bring that can be via rag or fine tuning, whatever method you choose. But it's your proprietary data that will help differentiate your generative AI product so that you can create personalized experience for your customer, or so that you can automate your own business process. But without that proprietary data, there's not really a moat or competitive advantage. And so companies are realizing that they need to get their proprietary data in strong shape. That means making sure that that is high quality data. So we are seeing more and more companies thinking about how do we get ready so that when the time comes and we actually have the tech stack and the business use case and everything and we can actually deliver on that, we have the right, the right data and we can actually use it.

That's great. I can't agree more. And then George, I'll let you react to that as well.

I like your comment about, you know, everyone has access to the models and the access for differentiation is what data you put into them. I have actually heard consultants advise companies that because everyone has access to the public models, the way you need to differentiate is by making your own model, which is insane advice. Like, yeah, that will differentiate you. It will differentiate you because your model will be much more worse than everybody. But it's so early days trying to speculate about this. All the AI stuff is so embryonic. It's very exciting because it's giving us the ability to do something with the unstructured data that we've been, we've had, we've had it for years. But it's giving us the ability to interact with unstructured text in a meaningful but programmatic way. What that turns into, time will tell. I don't know if rag is going to be the be all end all. I question whether chat is even the right long term interface for a lot of these internal applications. But I don't have like a great alternative on the tip of my tongue. I think it's very early days and everyone should have their eyes and ears open.

And Prakopa, from a data quality, data governance perspective, how does generative AI change the conversation?

Yeah, I think it's very early. But a few patterns we're seeing, I think across all our customers, we're seeing this pattern of people deploying small language models, right? More than which is where flags fine tuning, like some of this comes in. That's one pattern we're seeing. And as they look at that, I think the two nuances outside of just normal data quality, which is anything that we're seeing is one, the importance of business terms and semantic context. So for example, we had a customer who's an investment firm and he was like, when someone searches in our, like someone chats saying tam in our company, TAM means total addressable market, not the thing that it means on the Internet. So one layer is just like, like if you get an accurate output, what is the semantic context that's core to the company and how do we feed that? That's one layer that's becoming very important or more important than before. The second is we are seeing around this is a little bit more around governance, but also relates a little bit of trust, which is depending on who's writing a question, what data actually goes into the answer. For example, if I'm deploying something for my HR team, it's probably okay if payroll data gets used in the answer, it's probably not okay if it is across the rest of the company. I buy data from LinkedIn which has certain terms and conditions associated with it. I can only use it for this purpose, not this purpose. And so as you build that scale democratization, the way I think about this, you alluded to this, right? Like, which is the goal post keeps changing. I think that's a good thing. The reason the goalpost is changing is because people are using data more. And the more people use data, the more they need to trust it, the more there are issues. It's actually a good goalpost. And so if you actually play this out, there's more and more people who are going through maybe the dream of like truly democratized data where everybody actually uses data daily. And like, you know, like that's going to play out. But then how do you feed it with the. The right people should only get the right context at the right time in a way that's safe and secure. Like how do you solve for that? Those problems are proliferating and now need to be done in a very different way than before.

I think those problems of permissions are easier than you're getting at. But the reason people think these problems are hard is because they look at the people who make the base models like OpenAI and Anthropic and Mistral and all of them, and they're doing like web scraping. So they have a whole data pipeline that's like a scraping pipeline that is designed on the assumption that it is public data and so everything has the same permissions domain. But a business that's using its internal data in an AI application is not like that at all. They have data with very complex permissions landscapes to. But that's very normal in a like business intelligence scenario. The way you solve this is by using the same data infrastructure that you use for everything else. For AI, I have disappointing news. A relational database is the right answer for your as the data platform for your AI workload as well. Because yes, there is some unstructured data in there. Relational databases have text columns, they've had them for a long time, they work great. But there is also going to be a forest of other tables that tell you all of the permissions metadata that you need to know in order to manage this problem. So I think if your starting point is like a web scraping pipeline that looks like what the people who train the base models are using. Yes, the permissions problem look very hard. But if your starting point is a relational database that is structured similarly to the one you use for bi, this is no problem. You just need to join all the appropriate things and recapitulate the permissions rules of the systems of record in the SQL queries and you're ready to go. It's not that it's like so easy, you're going to do it in a day. But my point is it's not really new. This idea of I have a database, I have a bunch of data in it, there are rules about who is allowed to see what like that. If you have a complete schema of the system that you're talking about, that is a very solvable problem using traditional, traditional techniques.

I don't think my call is for like, what's the new technology we need? I, I think we actually spoke about this maybe like 9 months ago on maybe a different panel, like, and we were like, like, I don't think it's a technology layer. I do think the nuance is solving the problem. Like it does introduce a Lot of new nuances around just because you're processing this at a speed and at a scale and at like. Like there's actually nuances to this which need to be solved for if we really move towards that AI world. And then second, there's also a human collaboration problem, which is what is the policy? It's not even a technology problem. It's like, how do we collaborate to figure out what the right policies are for, what the right use cases are? And that used to happen on people. I've seen so many examples of people doing this. Like there's documents written and like publish them and nobody ever uses them. And it was okay because it was like a few dashboards here and there, which is just not going to be okay in the future. So how do you solve for that? I think that's why it becomes more important.

Okay, that's awesome. So we do have a couple of minutes left. I want to make sure that we answer some audience qa. There's one which is how do you proceed with management to make them understand the cost of data quality issues? Because we've been talking about putting a metric on it. We've been talking about aligning the organization. I think no metric is better than ROI or lack thereof. How are you able to put a cost on data quality issues? So, Kalpa, I'll start with you.

I feel like Barb might have a better answer to this one because I know you have a framework that you put out at some point, but I think high level, your first is almost accepting that everything in data across cannot directly be measured to business value because data itself is a support function inside an org. It's like management. If an analyst produced a report with say like, our CEO talks about this like he was like, I was hiring a bunch of sales reps in my, in my team. Same thing. Like, I hired one analyst who found this one thing that we could optimize and we actually made an extra million dollars through that one analyst. What's the ROI of that analyst time? It's way more than 10 sales reps. So how do you think about it? It's just harder to do because it's two layers removed because data needs to drive strategy or execution. Both of those things together drive business roi and it's just hard to decouple. So that's true for any data platform tooling across. And I think first, accepting that I think is helpful. And then second, then, okay, then what are those? So if you can't get the outcome metric, what's the Output metric that you can get to is the way I think about it. So we do need a North Star that we can progress towards because, you know that this is important to get to the outcome. And you know, if you're in a company that you need to convince people on that, then I would question whether like data is actually really important to the company. Like, that's pretty straightforward. Like, you know, you should have good data to dive. But if you're convincing people on that, like I think the first question to really have with whoever is asking you for a business case is, is this really important for you? Because if it's not, that's okay. Let's have a conversation about it and then I think, what's output metric? And I'll let Bar talk about that because I know you.

Yeah, I'll let Bar end with us on the framework.

Sure. So the number one thing I'll say depends on who you're asking, who you're talking to. If you talk to executives, the number one thing they'll tell you is I just sleep better at night knowing that the data that's powering my business, whatever it is, data products, dashboard, generative AI, call, whatever you like, I just sleep better at night knowing that the data is accurate, which is very hard to measure. To Procalpa's point, if you talk to a data engineer, machine learning engineer, whatever it is, they oftentimes will talk about how much time they spend and are they spending on sort of cleaning up data or cleaning up finances, drills or actually are they sort of, you know, building new pipelines and doing things, doing other things? So that's various answers that you will get. I will say in general, there's sort of three things that we think about. The first is reputation in brand and trust. So when your data is wrong, again, like think about the Google example. I don't know if I'll trust another Google search again after I saw the superglue example. The second is cost of revenue. And so oftentimes, you know, I gave the airline example, but there's real implications. You know, one data issue can easily cost millions of dollars for an organization. And then the third metric that, you know, I mentioned is sort of team efficiency or team time, your organizational time on that. Those are sort of the three high level metrics.

Okay, that is awesome and I think this is a great time to end. I want to say a huge thank you for Kalpa Bar George for joining us for such an insightful session. I truly, truly appreciate everyone. And I also say a huge thank you for everyone who's joining from across the world. You know, people joining us from different time zones, people even like 2:00am, 3:00am watching this stuff. I really, really appreciate it. So I want to say huge, huge thank you to all panelists today and to our speakers and to our audience. I really appreciate everyone speak.