#213 Building Trust Through Data with Prukalpa Sankar, Co-Founder of Atlan

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

That was kind of what started the journey of what turned into Atlan. We were like, how do we solve this problem for ourselves as a team? And some of the inspirations that we took were actually other teams. Right? Let's look at engineers use GitHub. Your code has a profile on GitHub. Sales teams. Every lead and opportunity has a profile on salesforce. Why can you not do the same thing with your data assets? Why can't you have a simple profile that tells you like, where does it come from? What does it mean? Who owns it? What's the lineage of it like? How has it changed over time?

Welcome to Data Framed. This is Ritchie the Tooling for performing the analysis part of data analysis has gotten very powerful over the last couple of decades. Whether you want to point and click in a business intelligence platform or or write Python code with some help from generative AI, crunching numbers isn't that difficult anymore. On the other hand, there are still some challenges that are widespread. Simply finding which columns in which table of that sketchily documented database that you're supposed to work with is the bane of every data practitioner. When seven of your dashboards mysteriously stopped working one Friday evening, it's often hard to know which change to the data pipeline caused it. In general, data governance feels like the last big hurdle to making data teams productive. Today's guest has spent the last few years working to fix some of these issues with data workflows. Prukalpa Sankar is a co founder of Atlan, a data cataloging platform. She previously founded the data for good company Socialcops, and her work trying to make the data team more productive led to the idea for Atlan. Procalpa's work has been highly celebrated. She was awarded the Economic Times Emerging Entrepreneur of the Year. She made the Forbes 30 under 30 list and the Fortune 40 under 40 and was in the top 10 CNBC Young Businesswomen of Pour 2016. On top of this, she's a TED speaker. Let's hear her take on solving those stubborn data problems.

Hi, Prakalpa. Great to have you on the show.

Hi. Thank you for having me. I'm super excited.

I think this is gonna be fun. But I'd like to start off with one of my biggest frustrations. So as a data scientist, quite often I get to ask to analyze some data, and then it's like, hey, go and have a look at this database. And I no idea where to find the data that I need. So do you have any advice on how to make data more discoverable?

Well, I mean, the easy one is use Atlan, but I mean, we hear this so many times in terms of the frustration that most data people have. Right. If you think about it, what seems like the fun part of the job, which is, let's work on this data and analyze it and find interesting trends for the business and solve big problems. Which is why all of us, like, do what we do, right? I mean, like, that's why we decided to pick up this. It's like almost like techno business role in the companies that we work in. And the sad thing is, most of us don't actually ever get to spend most of our time doing that stuff because you're spending like 80% of your time, firstly finding the data that you need for running an analysis, and then you're trying to actually understand it. You're like, oh, what does this metric mean? Oh, there was this new Salesforce field that was created by the sales ops team, but we have no idea what it actually means. Then you're trying to, like, understand what the actual metrics are and how do we define annual recurring revenue? And, you know, what does a customer really mean? And most of your job is doing that rather than actually solving the business problem. At Athlon, we started as a data team ourselves. We were data practitioners. I was a data leader running a team that was doing this. And on the other side, there was this massive frustration that I had as a data leader, which is I felt that we were so fragile. Like, I remember this one time where we had this really big project that was due, and I had, like, looked into the eyes of my client and I was like, we will make sure this happens and I'm going to deliver it. And exactly the week before the project is due, our lead analyst on the project quit. And I remember spending hours, like, I took him out, I was like, three hours. I was like, I literally begged him to stay back. Like, I was like, please don't leave because he was the only one who had all the context about the data. And if he left, I did not know how I was going to deliver the project to my clients, which is a pretty fragile place to be as a leader. That was kind of what started the journey of what turned into Atlan. We were like, how do we solve this problem for ourselves as a team? And some of the inspirations that we took were actually other teams. Right, let's look at engineers. Use GitHub. And your code has a profile on GitHub. Every lead and opportunity has a profile on Salesforce. Why can you not do the same thing with your data assets? Why can't you have a simple profile that tells you, like, where does it come from? What does it mean? Who owns it? What's the lineage of it? Like, how has it changed over time? And so we started saying, what is a data asset360 look like? And then around that. How do you build an ecosystem for search? I mean, it's so much easier for today. On Google, you and I can search to find what we need. And on Amazon, it's so easy to browse for stuff. So how do you kind of bring that ecosystem around your data assets inside an organization? There's been some form of this in the industry called data discovery, data catalogs. They've traditionally failed, and there's a lot of reasons they failed. We failed three times over implementing a data catalog ourselves as a team. And before four time around, we built Athlone. And so we think of, we call ourselves sort of like third generation of a data catalog to actually make this work inside an organization. And we can talk a little bit more about what it takes to make that happen.

Okay, so that idea of fragility is really interesting because it's all very well, your data team's performing in good times, but when times go bad and something goes wrong, you still need to be able to deliver something and avoid those disasters. So you, you mentioned the idea of a data catalog, and I'd like to dig into that further. Just to begin with, can you explain what is a data catalog?

I mean, the origins of a data catalog kind of sit in this principle of if you treat your data the way an organization would treat any other asset. So if you were to look at the best example that I would think of is on Amazon, when you're going to be able to go browse from shoes, and then you can go and look at the types of shoes that you want and those shoes, all the context that you want so you can Make a decision on, hey, here's the shoe I want to buy. You take that same example, and you bring it to data. The most basic first step for a data catalog inside an organization is if I'm a business user and I'm like, I want to understand why our customers are churning, I should be able to go search for customers churn, and I should be able to find the metrics like churn, rate, and then, hey, this is a certified dashboard that someone's already done analysis on. If I want to dig in a little bit more, I can dig in. I understand what each of those metrics mean. I understand, you know, how I can use it. And then I basically, hopefully make my decision. But when I go to my second order, third order, fourth order, all of those things are available. So the way I think of it is it is the home for data inside an organization. How do you build that one home interface for data people?

I like the idea of shopping for data. It's in the same way that you'd shop for shoes. So just carry on with that analogy. So if I go on a shopping website, I see a pair of shoes, there's like a product page that tells you it's got pictures, it's got description, it's got reviews. What's the equivalent for data? And in fact, who would create all that information?

And that's where the nuance becomes really interesting here. Because if you think about data first, what is a data asset or a data product? And is it just the table? No, not really. Right. If you think about today, your data assets are not just tables. Metrics are data assets. Dashboards are data assets. Code itself is data assets. Pipelines are data assets. Each of these things are data assets. And ideally, they should be reproducible and reusable. And that's where the nuance becomes really interesting. Where depending on who is looking for churn and like, let's pick if I'm a data scientist versus I'm a business user. If I'm a business user, I just want the dashboard. And ideally, my system should just show me the dashboard. And then, you know, when I go into the dashboard, I should be able to see what does this. It's almost a profile page for my dashboard. Right. The first step, this is the dashboard. Each metric, here's what this metric means, here's who owns it. You often have this question, like, when you look at the data, you're like, you know, I've had business leaders tell me this. I look at the data and I'm like, that doesn't look right. And the thing is, what I don't know is, is there something wrong with my business or is there something wrong with my data? And in an ideal case, the profile should answer that question for you. You should be able to say, yeah, you know, there's nothing wrong with my data. There's something off in my business. Let me dig in. Let me understand why there's something that's changed this week versus last week. Right? So that's the proliferation of this for an ideal business user. At Athlon, for example, we actually just delivered this into the experience in a dashboard. So they don't even know they're using a data catalog. They just get this context in like a tableau or a power bi when they're looking at it versus if I'm a data scientist and I look for churn now like I'm trying to do a deep dive churn analysis. So the first thing I need to know is what are all the other projects that have ever been worked on in the organization? What are the tables associated with it? And so when I go into a table profile, I probably want to know every column. What does it mean? Audit logs? I want to know version history. I want to know who owns this. I want to know the lineage. Like, how is this created? Created? I want to know when was this last updated? I want this whole 360 context that I need to be able to figure out, okay, should I use this data for my project or not? And so that's where it's very interesting, where if you take consumer patterns, I think the ideal solution for this in data is actually a mix of Google and Amazon and Netflix, where there's an element of Google and Amazon. But the thing about shoes is that if you and I are looking at shoes and we're making a decision on shoes, we probably want to see the same information. That's not true for data because. And that's where like the Netflix analogy comes in. Because if you're a data scientist and I'm a business user and we're looking at the same thing, we want to see different pieces of information to help us make decisions on it. And that's where, like Netflix kind of personalizes the experiences for you and me. I think that's the last leg for making this really work inside organizations.

Okay, that's really interesting, the idea that different people want to see different things from data. And so you've got to cater to all those different audiences. I'd love to get into that in a Bit more detail later, but you also mentioned a term, data lineage. Can you just talk me through what that involves?

The best way to explain it in my mind is the flow of data inside an organization. Typically when you're thinking about building a dynamic real time data system, largely most organizations are, the data is actually coming from a source system. So it's something like a salesforce or a netsuite or like an internal tool that data typically goes in into a larger scale compute system. So that's typically where a data warehouse like a snowflake comes in. Once you go there, you're typically you have your raw data, you're transforming it and then you're turning it into your final cold state data, which is like business verified, which is then getting pulled into a dashboard or then getting, which is a completely different tool. So if you think about this process, your data is hopping through at least five or six sometimes in like large enterprises, I've seen, you know, hundreds of hops before something makes its way down to a final business user on a dashboard. Which is why when you ask this question, why is this number on a dashboard wrong? It seems like a really simple question. When you're a business user, you look at an Excel sheet because your analogy is Excel and you're like, it should be easy to figure this out. It's actually a very hard question to answer because why this number could be wrong could be because the data ingestion pipeline from the source system to the warehouse failed today and didn't update the data. Or it could be because some human updated the definition of annual recurring revenue and so the way we measure this metric changed. Or it could be because the data quality check in the transformation pipeline failed. And so there could be like hundreds of reasons and like at least 10 people involved in that simple question of why is something broken? And so the foundation for this is the map. If you want to solve this problem, you need a map. Have a map of exactly what I described down to the individual field and column level to be able to say, hey, this is how data flow across the organization. And that's what data lineage is.

There's the common sort of idea that debugging anything is twice as hard as building something in the first place. And does seem like once you've got half a dozen tools or even like you said, hundreds of tools, debugging what went wrong in a dashboard is going to be an absolute nightmare. So understanding the process does seem incredibly important. So what would you need to do to set up a data lineage? Like what goes into building this.

So the way we think about Lineage is Lineage is a very complex problem. And the reason most people don't have lineage is because you kind of have to construct it automatically, otherwise you'll never really construct lineage. So the way we do this is three or four layers. First is read everything that's happening in your data estate. So for example, the first step is like, we'll read through like your SQL query history, anything that's written in SQL, reverse construct and create link. So the first step is whatever transformations are already happening, whatever like is written in code, reverse engineer the code to deduce how data is flowing inside the enterprise. The second layer is integration systems and integration points. This is where we work very closely with BI tools to expose more API endpoints. We've worked very closely with transformation tools like dbt, data ingestion frameworks, you know, fivetran Airflow, to again expose more API endpoints through which you can construct lineage across the ecosystem. So that's what I would think of as the second layer. And the third layer then ends up becoming almost the last leg. It's likely that there are going to be internal systems that don't have either of these. Or, you know, you've written your, your script in like Scala or something. You know, it's just, you can't really reverse engineer this very well. That's where what we see with customers is this idea of like integrating into the way that developers code. So we have a customer that calls this eldp extract, load, transform, publish and publish is the last step where you actually publish the transform, like the change that's happening in your data and you integrate it into the way that developers work. And what's been really exciting, I think now with generative AI, is, and we made one of the earliest investments into AI in the space is your AI can do the first level of deduction directly as a part of the way people are coding. And so then a developer kind of gets a recommendation and then they just kind of have to say yes or no and make some changes and publish. So that's sort of like what I would think of as the third layer of constructing data lineage, which is what then creates the map across the enterprise vis a vis. Just to give you a sense of how this used to happen, five years ago, people used to manually construct this. I have seen people manually build maps at a system level and like they've, they've shown us like these manually drawn maps, which just doesn't scale inside organizations. And so if you want to do this. This has to be automated. This has to be system first, otherwise it's never getting done.

Yeah, I can imagine. That's an incredibly tedious job and you're probably just hiring lots of interns because no one wants to do something. Just like working out what system connects to what.

I have a funny story. So recently we were working with a large Fortune 100. You know, they plugged in the system, they got their lineage out in a couple of weeks. We go in, we show them lineage, and this is like across like complex third generation systems. They see this, people are like, wow, this happened. And they were like, how? What did it take to do this? And then our solutions Engineer basically said, 20 interns over six months. And everybody nodded. And then it was a joke that nobody got. And then like our sales leader, he was like, just to clarify, that was a joke. You just plugged in the software. It happened. But it's. Yeah, that's just the reality of how things have been done over time.

Nice. Yeah. So sad for the people who wanted to be an intern, but at least it's not a terrible job. It's good. This is happening automatically. All right, so I like your idea that you can have several different layers of data, Lynch. You don't have to build everything at once. Maybe it's worth just talking through, like how you get started with this. What you need to do. Do you build the data catalog first and then worry about lineage? What's the order of this?

The way I think about this is you think about the category is all of these problems are very interrelated. And there used to be a world where these were almost like separate software category products. There would be like, you know, this separate data catalog, separate. Separate data lineage, separate data governance, separate data stewardship. Like these were all. But if you fundamentally break this down, the fundamental problem you're trying to solve is you're trying to create a single source of truth across your data ecosystem. And that single source of truth translates into all of these different applications. One use case of that is making it really easy for people to find data. Another use case of IT is like prevent breaking changes or root cause analysis of dashboards. Another use case of IT is compliance and gdpr. And that's just the. Those are. The way I think about it is they're kind of like use cases, more than, you know, categories. And so the way we see this or the pattern that we see is you invest in building your metadata foundation. And we think of this as almost like your control plane, like the data Plane needs a control plane and the control plane needs to be in a metadata layer. And so what you want to do is basically invest in building a foundational metadata layer, which is building this data360 of sorts across your enterprise. And then on top of that you almost enable use cases. So the pattern we see is that we've mapped 14 use cases for this, each of which have business value, etc. So typically what you want to do is you want to invest in platform, but enable the first or the second use case and that's dependent and then you grow over time. And the first or the second use case is a function of what's most important to your business today is the biggest bottleneck that you're having. Root cause analysis or breaking changes in dashboards is the biggest challenge that you're having. Compliance is the biggest challenge that you're having. Democratization and self service, depending on that, you kind of prioritize which use cases you want to enable and then go along.

Okay, so I mean, you mentioned a lot of common business problems there. I'm wondering, do you have advice on how or which bits of data you should get started with? Do you start with the easy, simple data sets or do you start with the business critical data sets? Like where do you begin?

Yeah, that's a great question. And the way I think about this is you kind of break it down into implementation complexity versus implementation value. And in an ideal case you are going after something that is high value, low effort. And the reason for this is just, you know, you want to as quickly as possible get to business value and you want to start proving as much business value as possible. So the best examples of this that I've seen have been actually we have teams that actually have this quadrant and it's basically based on value and complexity and they're basically like there's dots on it and they're prioritizing new use cases that they will soon. So it's not a straightforward answer. Because, for example, if the most business critical problem to solve is something that's going to take you a year to solve, and this is not because it's hard to build the technology systems. Part of this today is actually the easy part of the problem or the easier part of the problem today, if you pick the right tool, I don't think that's the hard part of the problem. The hard part of the problem is the human last mile that's important. For example, things like metric definitions, like what does a metric mean? How do we define customer? No tool in the world is going to define a business like how a business defines customer. Right. If it's a use case like that and it's going to take you six months to drive change management, then you could argue that maybe there's a slightly lesser, slightly lower impact, higher, like lesser effort use case that you want to enable first. I think what people get wrong with this is strategy and they don't invest the amount of time it takes to get the strategy right. And we highly recommend like, in fact like we now in our university have this, we have like a video on why strategy is important and we'd be like, don't buy like unless you see this. Like you have to like align to this. Because at the end of the day when you're thinking about building a single source of truth and this is true for any CRM system like Salesforce Lattice for people management, like if you're the minute, it's not just a tool which is switch on, switch off. For example, data ingestion is a switch on, switch off tool. Anything that sits one layer above that's a little bit more strategic in nature across the organization needs people buying across the organization. And so if you get the strategy wrong, everything else breaks. And so that's for example been a very, very big investment that we've made in like saying how can we be a strategic partner in the journey that we help our customers take.

It's interesting that it's like technology is easy, it's your colleagues the problem. So one thing you mentioned was that no one can agree on definitions of business metrics. This is certainly something I've seen at Data Company. You know, we argue about what is the definition for customer lifetime value or how do you do marketing attribution or something like that. These things change a lot. So can you talk me through what you need to do to get to clear definitions of these critical business metrics?

I think of these as like two step problems. The first is a human collaboration problem which is what is the metric? Define the metric and define the change management process around the metric. And so what's the workflow when the metric changes? And so the way we think about step one is figure that out. And you know, you all, we almost think of this as if you have asana for project management. What is your asana for metrics management? And because it has so many different stakeholders, right? Because it has your business leader, it has your business ops, it has your data person, it has your compliance. Like there's a whole layer of people who need to be involved around that, that process, both in the definition as well as the change management. The interesting thing is actually the definition is maybe the easier part of the problem. Right? Like, you get everyone in the room the first time you get it, right. The challenge is it keeps changing because business changes. And if business changes the metric change like this is a. So I think that's the first step of the problem. And how do you, I think, how do you enable effective tooling to enable the true human collaboration challenge that exists when, when something changes? The second layer is tying this into execution and the biggest challenge, actually. So there used to be, they. Gartner used to have a category called information stewardship. Like there used to be an entire category of tools called information stewardship. And have you ever spoken to a data leader who's, we've solved this problem? No, it's never, you know, I've never heard, I've never literally spoken to a data leader who said that this problem is solved. And I think that's because this doesn't get tied into the way people work. So, so let's think about, like, when something changes, typically you're moving very quickly. So someone in business is telling someone in data. And at that point nobody is actually, you know, you're, you're just fixing something and shipping. That's the reality of what happens. So if you're not integrating the way you've defined your metric to the way you're defining your code and the way your code works, this breaks. And so we think of this as, you know, governance as code. So we call it like shift left governance. Take it back into the way developers work, data engineers work, analytics engineers work. So that when you're shipping as a part of that ship process, it automatically updates your metric. So there's a full flow there, which is what takes the biggest connection between the way your data ecosystem works and your developer workflows work. And if you get both of those things right, then you actually have a relatively seamless system in making this work.

Okay, so just to make sure I've understood this correctly, what you want is a situation where if you make a change to the code, the official business definition of the metric is going to automatically change at the same time.

Exactly. And I mean, in some ways, like the way we've tried to define this as we've also realized the workflow of this is different depending on the organization. Right? Because in some organizations, business defines the metrics and data services it. In some organizations, data defines the metric and business consumes it. And so I think the actual workflow of the way this works, where is the approval could change and where is the definition could change. And so I think it's important to build this in a way that's flexible. The only thing we've realized is that the only reality in data is diversity. No two teams are the same. And so you want to be able to build this almost as a workflow systems. But ideally the end state is the way people are working and making changes in their metrics and code is directly related to the way business is consuming those metrics and those definitions. And your change management integrated into the way the organization maybe let's talk more.

About what needs to happen with the people and the teams in order to make this work. So you mentioned a few job titles like data steward or data governor, but then there's also the business people. So in terms of making sure that all your data set up right. Can you talk me through which teams and roles need to be involved in these decisions?

So the core owner of this has to be the data team. It needs to be a core layer in the data team. The ideal case that we've seen is we almost see these structures that we call sort of like the modern data governance team or the modern. And the way we think about that is like the best analogy that I think of is if you think about a sales team, typically sales teams have two key leads outside of sales leaders. One is sales ops and the other is sales enablement. Sales ops is typically working on your systems and your tech. Sales enablement is typically working on people and human change management. To make this work inside an organization from a data perspective, you need both tooling and engineering and you need human change management to work hand in hand. So the ideal structure of this team that I would think of is like a tag team between a data ops engineer and a data governance lead who are both working together to own the project. From there, actually, I think it's pretty decentralized. The key is how do you work with your data engineering team and change the way like data engineering team needs to go from publishing data to shipping data products. Your business teams need to start owning their data. And you know, so there, there are ways that you want to empower all the other teams and embed into the way that they work. But the best case scenario that we've seen is if you have these two kinds of Personas driving this inside an organization, you almost always win.

So I like that those sort of parallels for the sales ops and sales enablement. Data ops is Certainly think. I don't think I've heard of anyone with a job titled Data Enabler. Do you think that should be a real role?

I wish there was. Like, you know, and in some ways I actually think that's what data governance is. If you really think about what a data governance role inside an organization is, it exists specifically for the purpose of bringing people and process and tools together to drive people to use data better, which is a fundamentally enablement focused role. I'd actually written this article called Data Governance has a serious branding problem. Because you know, when people think governance, they think compliance and risk and process and bureaucracy. Like that's what you think. But actually like some, like there's a reason government exists, there's a reason, like, you know, like we don't live in anarchy and like, you know, like there's a, if you think about the parallel to the word governance, like it's not, it's not all bad and it probably helps us as a society work a little better than we would otherwise. Right. Sometimes I think like maybe we should rebrand this function to call it data enablement. Or sometimes you should, like maybe you should rebrand this function to call it data product management or some version of that just because it has a bad rep. But foundationally, I think if data governance done right, helps people use data better and enables people to use data better.

All right, super. I think we've just solved the data governance branding problem from now on. Data enablement, brilliant. And so related to this, what sort of skills do people need in order to do this data governance better or have a smoother data workflows?

I'd say two skills need to come together. Two important skills need to come together. One is change management and people. And the interesting thing about this is there used to be a time where it was okay to deploy this stuff top down. There was like top down, rigid, you have to do this. But the reality is like nobody works like that anymore. You kind of need something that's like a little bit more bottom up, collaborative. So to give you an example, one of our champions, Emily, you know, when she was working at WeWork, she was the change management data enablement person. When she launched, she launched with a data personality quiz. And it was basically this personality quiz for all these different types of people, business people, to find out what their data personality is. It sounds like a very simple thing, but actually what would have otherwise been a survey became, which no one would take, suddenly became this like exciting thing that people were really excited about being part of and they were sharing their data personalities and like, things like that. So it became a thing, right? Change management is hard. Data people. Well, honestly, we don't like doing it because that's why we chose our careers, right? We're not product marketers and we're not like community people. That's not who we are. But that skill becomes super, super critical in driving helping diverse people come together. The second, though, is that we're doing that without deep data engineering experience means that you're basically doing a bunch of human stuff that doesn't get automated and run at scale, which means that you're not going to focus on the right problem. So the best battle to that I would say is like a data architect or a data engineer or data governance engineer, whatever we want to call them, who is able to scalably automate wherever is possible so that at the end of the day, change management is hard. So you don't want to do change management if you can avoid doing change management. Like I just started, like, if you can, if you can automate it, like, don't do it. And I think both of those skill sets and Personas need to come together to do this effectively.

Okay, so this sounds, it's going to be the tricky part then is that you've got to have data people speaking to business people and they're both going to get along and communicate. I suppose this is especially true once you start, like trying to label your data and say, well, the business person knows what the data thing means, and then the data person, like, wants all the technical details. It's going to be very different who writes what. So how do you make sure that both these teams can work together?

I think in some ways it kind of comes down to what makes a great team. The only reality in data is diversity. It might be maybe the most diverse team ever created. Because if you need to solve a data problem, you need business, you need analytics, you need data engineers, you need science, machine learning researchers. Now there are prompt engineers. There's just a variety of people and skill sets that need to come together to make a data problem work. And none of these people can speak the same language. None of these people understand each other's work, which is very different from any other team. Like, let's think about a sales team, right? Most salespeople kind of, they've done the same job. But like, for example, if you. If I remember when I was a data leader and I'm a little bit more on the analytics business side, I'm not a data engineer and I Remember when, like a number on the dashboard was broken and I had to answer a question to my client, I was like, did the pipeline really break or is my data engineer not doing this job? And that was a very real question. And that breaks trust. And when that breaks trust and people stop believing the data or they stop, the worst case, they stop believing the data team. And then there are these, like, shadow analysis, it's just everything breaks and then you have to restart it all over again. And so then the question is, what does it take to build trust and collaboration across these diverse people? And I think the first step is knowing that there is diversity and embracing that. The second is then, I think, investing in systems that create trust and collaboration. So what creates trust and collaboration? Typically, transparency creates trust and collaboration. When people see things, they believe it more. So, for example, if at that point when the number is broken, I just have a proactive announcement that says, pipeline broke Today, as a business person, I'm going to trust my data team a little bit more. So the best data teams that we know actually measure this. They measure an nps, they measure a score, they measure themselves on proactive versus reactive communication. That becomes the one foundation for trust. And the third is then, you know, you find a way to get. Once business understands what the core challenges of data are, and once they believe and trust that you're doing the right thing, then they become a lot more involved in actually enabling data teams with, hey, like, I'm changing the metric definition. And so I think some of the most empowering stories that we've heard have been when we've had these, you know, data teachers come to us and say that, you know, before we used to be a service organization and people just used to ask us for data, and now we are a partner. They look at us as a partner. They're calling us with their strategic problems. They're not calling us with, I want data. And that transformation, I think, is a function of transparency, trust, and then finally enabling collaboration on top of it.

I do like that idea of just being proactive about saying if there's a problem up front, rather than some business person calling you, saying, hey, I think you screwed up the dashboard. Yeah, tell people up front. And that's going to help with the trust. From a product point of view, how do you design a product that's going to be suitable for very different audiences, like, for a business audience and a technical audience?

Diversity needs to be baked in fundamentally into the way the product is built. So I talked to you a little bit about Third generation or why generations of tooling here have failed over time. It's mainly been because you've had tools that were built for one type of audience. And the reality is that data is diverse, people are diverse. So, for example, we think a lot about these systems of how do you create personalization as a default. So when you think about the ui, you are seeing a different, different view versus me versus somebody else, depending on the context, depending on who I am, what projects I'm working on, what my role is, my entire experience. So, for example, something that data engineers really, really want is they want to know pipeline updates from Airflow. They're like, did the pipeline update or fail or not? They want to know did the data quality test fail or not? A business person doesn't want to know any of these things. They just want to know, can I trust it, can I use it? That's it, just give me an announcement. Red, green, yellow. You're actually confusing me by giving me more information. For example, if you're a data engineer and you log in, you see that stuff. If you're a business user, you don't see that stuff. That's a very simplistic example for like, how do you build systems that work for this diversity? I think the second is how do you interface in points that people already use? So one of the biggest challenges with rolling out systems that touch all different kinds of users inside an organization and the reality of data is it touches all different kinds of users across the organization, integrate into systems they already use. So for example, we have a Chrome extension that sits on Salesforce, it sits on top of BI tools. We sit in Slack, we sit in jira, we sit in Microsoft Teams. So for example, there's typically some Slack or teams channel where people be like, what does customer mean? How are we defining it? Imagine a bot just answers that right there and you have a flow built right there. Or let's say you're in jira, you directly get a ticket there that is connected to the data asset that's broken for that day and you can auto trigger it. You can actually say pipeline in Airflow broke auto trigger JIRA ticket. So when I start my day, I just start in JIRA and I get all the context that I need to work. So one key element is, we call it activating metadata and taking it back into the daily workflows that people are already using and just integrating this context and trust into the way that they're already working, that is enforcing them to move systems.

Yeah, I like the idea of not forcing people to use yet another tool, explicitly building things into what they're using already. I'm kind of a techie nerd, but I get overwhelmed by the number of different tools I have to use and I say, oh, I have to go to this app to see this thing and go to the app to see a new thing.

Yeah, you lose 40% of people at login flow. Like, get them to log into a different system. That's it. 40% is gone.

Nice. Okay, so it seems like a lot of the stuff we've been talking about is dealing with things that go wrong. So this data pipeline's broken, this dashboard's not right. Is the main benefit just around making it easier to deal with problems, or are the benefits for when things go smoothly as well?

The way we think about benefits and the way, you know, people have described this to us, when it works well. So when it works well, your organization is able to make quick decisions confidently across the ranks. And I'll break all of those three points. We live in a world where things are changing extremely fast and you have to respond extremely fast to external stimulus. Like just let's pick the last few years, like war, supply chain crisis, Covid, you're responding extremely fast to external stimulus. So first, how do you move quickly and how do you, how are you agile from a decision making perspective? The second, when you're making the decision, how do you make it confidently? How do you make the right decision? And that's where trust comes in. Can I trust the data? Is something wrong with my business or is something wrong with my data? And the third is, can you do it across the ranks? So right from your board to the last person on the ground, last frontline, can they all make decisions quickly and fast, confidently? I'd say those are the three broad. So when done well, that's what it enables. And there's this very cool. I was looking at this the other day. There's this like top 10 companies by market cap. And you look at the companies in 2010, you look at the companies in 2022, and they're very different companies. And the difference between the companies in 2010, companies in 2022, the companies in 2022 are digital first, tech first, but data first. The way they use data is celebrated across the world. They've invested in these automated systems. They're very good about the way that they do this. Right. I think that's the best case scenario. To get there, you have to be able to get out of reactive mode to proactive Mode. And I think that's the trap that most data leaders fall into, or data teams fall into, which is there's a lot of foundational stuff to fix. And so you're constantly servicing, fixing the problem. I mean, I think the Chief Data officer tenure is the shortest that I've ever heard. I think it's like 18 months or something like that because you jump in and you're constantly fixing stuff and you don't have the time to get to proactive mode. Just to this end, state that I was talking to you about. And so that's why getting the first set of things fixed is important. So you actually have the time to do the second set of things. But ideally done well, that's what it leads to.

Okay. Yeah. So I like the idea that if you trust the data, you don't have to waste time second guessing yourself and sending out that message to the data person saying, is this number right? And that's going to speed up your decision making. So I can see how that's going to be a productivity benefit, even if things aren't actively going wrong.

I mean, productivity, but also like, and this is the hard part, right? Like, I think data teams suck at, like, finding the business value of data. But for example, our CEO Omar Cooper tells the story in his previous company, he was running a sales team and he was like, I brought in an analyst who within the first two months of being there, identified this gap that helped us improve our conversion rate in this process. And we made an extra $10 million that quarter. And there was nothing I could do that could have been more impactful than what this analyst did. And so what is the value of a confident business decision? We're not very good at measuring and quantifying the value of that, especially in short periods of time. Because, you know, even in this case, was it the fact that we identified this gap or was it the fact that we executed well to actually improve the conversion rate? Both of those things actually needed to happen for the business to improve, but both of those things did impact it. Right. To me, I think a lot of data people fall into the trap of thinking about their job from a productivity lens, but that's not the value of a data team. The value of a data team is you're helping make the right decisions confidently and there is a value to business that comes with that. How do you define it? Harder question.

Yeah, definitely. And I think most businesses wouldn't mind too much if they got an extra $10 million a quarter. That's a nice little Bonus there from a change in just finding something with data. Okay, so it seems like this sort of better data governance and better observability of data could help you out for industries that have a lot of compliance requirements and regulations to deal with. Can you talk me through how it's going to help?

So typically, I think most organizations today are dealing with a ton of regulation compliance and it's actually amplifying now in the AI world. And we traditionally been dealing with things like, or like in the last five years, gdpr, ccpa, local regulation, which country residency acts. Right. So first step is like, as you think about this, there's like, what's the policy? And second, how do you again execute the policy across your data estate? So the way we're seeing customers that leverage this is to say, okay, let's define our policy, this is how we use gdpr. But then we have to connect that to across my single source of truth and my data estate, what is pii? And then I use lineage to propagate pii. So if one column is pii, I propagate that everywhere else in the stack and I say any column that's derived from this all the way to my dashboarding layer is also pii. And then the last step of this is enforcement. So you say, hey, let me make sure that wherever in the ecosystem there is PII data, it's masked and it's available. Right? So that's the foundation metadata control plane that can enable that across the ecosystem. And what we're starting to now see is with AI, this is even becoming like, you know, we're seeing customers who want to deploy different bots for different use cases. So I had a customer the other day say, so when I'm deploying a bot for my HR team, it's okay if payroll data feeds into the answer, but when I'm deploying for the rest of my organization, you probably don't want a bot saying, this is my colleague's salary, so how do I enable that? And that's a relatively simplistic use case. But like, you're buying data today and there's terms and conditions that are associated with buying data. We had a customer that said we buy LinkedIn data, but there's terms and conditions that's associated with what I can use this for. So how do you first define policy, the human collaboration problem, and then how do you execute it across the entire stack so that you're actually adhering to policy? You know, when something's broken, you're able to monitor it and how do you build that flow across the ecosystem? And then the use cases become real time enforcement, data access, governance, but also audits. How do you build out audit reports for specific use cases? How do you trust the report that you're sending to a regulatory authority? All of those become use cases associated with this.

Okay, so things seem like fairly grim tasks that I'm sure most humans don't want to do, like saying, hey, is this table or is this column actually PII that I can't use for some use cases? And then I'm sure nobody loves writing reports for auditors, so having all that done automatically does sound incredibly useful. All right, so before we wrap up, what are you most excited about in the world of data management and governance?

I think what I am most excited about is we are in this very interesting moment where we can get it right right now. And a few things have come together this year. Chief Data Officers Gartner did a survey. The highest spend budget that people have is for data governance this year. And that's mainly because as you think about where we are as an ecosystem, many companies have implemented the cloud, they've implemented data warehouse, they've implemented. But there's this last mile, and that last mile is data usability. And we've had customers tell us this where our business users have gone through three generations of data warehouse implementations, but it doesn't actually matter to them because at the end of the day, they can't find or trust or use their data. But with what's happened with the cloud and the data stack migrations, I actually think that we've gotten it right. Like foundation of the modern data stack, as it's called, is there today. And we're now at the last mile problem today. It's possible to solve the last mile problem because it's possible to automate things because of what's happened with the cloud and companies like Snowflake and Databricks and in the last five years, that was never possible before. So, for example, we could never solve this last mile problem if we needed like hundreds of people to be drawing manual lineage graphs. It's never going to happen. We're now at a point where we can solve it. And so that's sort of like the second wave. That's, I think, most important where we can solve this problem. We can do this right. And I kind of see this as the, you know, if we do this right over the next two or three years, we can hit that final, I think the Mecca that we've all tried as data people been trying to get to it. We have, you know, data driven decisions across the organization. Like, everyone's using data, like, we're truly there. And this time, I've seen many, many waves of data and technology over the last few decades. And it feels like this time we are finally a few years away from, as an ecosystem, solving data management challenges once and for all.

I think that's the most optimistic take I've ever heard on this, because normally it's like you speak to chief data office and like, data quality will never be a solved problem. This is a disaster and a few years away from success. That's wonderful. So I think we'll end on that note. End on the happy story. So, yeah. Thank you so much for your time, Prakalpa.

Thank you so much, Richie. It was lovely speaking to you.