#64 Creating Trust in Data with Data Observability (with Barr Moses)

Hello, this is Adele Neme from datacamp, and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. You know, when we say that an organization is data driven, what do we truly mean? While there are many competing definitions that point to skills, infrastructure, culture and a lot more factors, we know that data driven organizations extract value and insights from data at scale. For this to happen, trust in data and data quality are investments organizations need to sustain for the long haul. This is why I'm excited to be speaking with Bar Moses. Bar Moses is the CEO and co founder of Monte Carlo, a data reliability company backed by Axel, ggv, redpoint and other top Silicon Valley investors. Previously, she was VP of Customer Operations at Gainsight, a management consultant at Bain Co. And served in the Israeli Air Force as a commander of an intelligence data analyst unit. Bar graduated from Stanford with a Bachelor's of Science in Mathematical and Computational Science. In this episode, Bar and I talk about her background, the state of data driven organizations and what it means to be data driven, the data maturity of organizations, the importance of data quality, what data observability is and why. We'll hear more about it in the future. We also cover the state of data infrastructure, data meshes, and more. If you enjoyed today's conversation with Barr makers, make sure to also download our Guide to Data Maturity Whitepaper, which discusses a holistic overview of data maturity and the steps organizations can take to scale the value they reap from data. We made sure to include a link in the show Notes. If you want to check out previous episodes of the podcasts and show notes, make sure to go to www.datacamp.com community podcast bar, welcome. It's great to have you on the show.

It's great to be here. Thank you so much for having me.

I'm excited to be speaking to you about the state of data driven organizations, data quality, data reliability, and all the cool things Monte Carlo is working on. Before we get started, I'd love if you can walk us through your background and how you got into the data space.

Yeah, happy to. So I was actually originally born and raised in Israel. I was actually drafted to the Israeli Air Force and so was a commander of a data analyst unit. I moved to the Bay Area about a decade ago. My background after that is in math and stats and actually I thought I was going to go into academia but my dad is actually a physics professor and so I thought that I was going to follow his footsteps but ended up having to. It was my big fail. My dad moment when I didn't continue, maybe sometime later in the future, but instead actually went to work for a consulting company called Bain Co. Where I worked mostly with companies actually on their data strategy and operations and later on joined Gainsight, which is a customer success company where I was very fortunate to be at the company at a time of very fast growth and work with some great people, learn about how to create a new category, Gainsight, help create the customer success category and build a number of different functions. Among them the customer data and analytics team, where we were responsible for our data internally and also for surfacing our data to our customers and helping them see value from it. And so those were some of my experiences with data and actually how I've encountered sort of the problem of data downtime prior to starting Monte Carlo a few years ago.

That's great. And how have these experiences led you to founding Monte Carlo? And can you walk us through that journey, please?

Definitely. So when I was at Gainsight, you sort of mentioned I was responsible for the team, so responsible for customer data and analytics. And we were starting to get really data driven as a company, which meant that basically we had a lot more data that we were analyzing, a lot more data that we're storing and collecting, and a lot more users were actually depending on this data. Among them were our executives, our CEO actually using the data to make decisions on what new products to launch, which customers to focus on, on where we're seeing the most traction. Kind of pretty basic questions about the business that you'd want to have answered with data and kind of the person responsible for this data. My experience was I would wake up basically every Monday morning to this barrage of emails asking me questions about the data. Why is the report here wrong? Why does the data look not fresh here? What happened to this graph? It's suddenly all null values. Lots of people just asking confused questions about the data, which led to a fundament, sort of distrust in the data. Right. Which was really frustrating to me. Right. I was like, what am I doing wrong? Why do we continue to have all these problems? And on top of that, it felt like every time that there was a problem, it took us also a very long time to figure out what was the root cause. So, for example, if a report was wrong, it could be because the report was not refreshed. It could be because one of the tables in the data warehouse didn't get updated. It could be because one of the third party sources that we were relying on made a change in their API. There could be so many different reasons for why the data was wrong. And it took us a really long time to both learn about it and learn about the root cause and fix it. I remember that as being a really frustrating experience for myself. Then in talking to other data organizations and other data people in other companies, I recognized that I wasn't the only one experiencing this. In fact, this was something that felt really ubiquitous in the sense that, you know, if you are in data, you experience something like this and, you know, sort of that kind of got me thinking, right? Like, why are we, why don't we have better ways to manage this problem? Right? And do we even have the right language to describe this problem? And so in starting Monte Carlo, I actually spoke to hundreds of data leaders, from very small startups to large organizations like Netflix and Uber, where data is sort of core to their mission, and learned a few things. Well, one, learned that, as I mentioned, every company runs into this problem. And two, it's a really unsolved problem. The sort of question of like, how do we trust the data? How do we know that data is reliable? And so actually decided to start Monte Carlo with this, with the mission to help organizations become data driven by minimizing what I think is the biggest problem today, which is data downtime, which is sort of the term that we coined for describing incidents when data is wrong or inaccurate or just can't be trusted.

So I'm really excited to discuss with you all the cool things Monte Carlo is working on. But before we do that, you mentioned here the state of data driven organizations. And one thing I'd like to discuss with you is really the state of data science and the march for organizations to become data driven. You know, over the past decade, we've seen tons of investments in tools, infrastructure, and hiring. But that doesn't mean organizations are making the most of their data or are necessarily data driven. This is something I've seen you write and speak about. Can you walk us through how you would define a data driven organization?

That is such a great question. Because I feel like we were throwing around the term data driven for like about a decade, and I'm not sure there's always something concrete behind that. Right. So when a company says, we really want to become data driven, what does that mean? Right. Sometimes it can be really surface level. It can be someone just like, woke up one day and they're like, I want to become data driven and then hire 400 data scientists and invest in tool X and that's it, call it a day. We're data Driven.

Right.

Success. But it's not that simple. Right. It incorporates what I think really requires both a mindset shift and organizational and cultural shift that needs to be supported by technology. But you can't have one or the other. And it certainly is not a surface level initiative when done well. Right. And I'd say today there's probably two main use cases that we see companies using data for. This is really simplifying it, but very, very in a simplified way. One, it could be using data to drive digital products in the product. Second, it could be to actually make decisions, whether that's based on sort of machine learning models or other sort of ways to actually determine what's right for our business or what sort of strategy are we taking or sort of data driven decision making. And I think we have a long way to go on both of those fronts. Right. And so when people typically ask me, like, how do we actually create like a data driven culture, what does that look like? I think that starts with obviously collecting data and storing data and making sure that you have data accessible for everyone in the organization. Right? So even if you have a team of data scientists or data engineers, data analysts, that's where the journey begins. Right? But truly becoming data driven is when marketing and sales and customer success and product are all very strong customers of the data organization and work hand in hand with them to make these decisions and to power the business. And that's really when you see the competitive advantage of becoming data driven. Very often we still see companies and organizations say, okay, we don't really have the data here or we can't really trust the data. Heck, whatever, let's just like resort to gut based decision making, right? You know, or, you know, we'll have a conversation with someone, they'll be like, yeah, you know, we're not really data driven. We sort of like make decisions based on like what we think of or some of the instincts. And there's definitely room for instincts and for gut based decision making in companies and certainly, you know, new categories or new markets, you can definitely have an argument for that. But I really don't think there's any excuse for a company today to not use data and to not, you know, at least start their journey in becoming data driven. Right. And there's definitely different sort of, you can sort of plot people or plot companies on sort of a maturity curve, right? Like how early are you in your, in the maturity of your business in terms of becoming data driven? And we actually worked with organizations to help sort of create this journey. You Know, whether it's like in the early days where you're really just sort of, you know, very reactive, trying to figure out sort of what data to work with all the way to sort of having sort of best in class, very scalable and automated ways to empower people to make decisions in real time with data. So I think we've definitely come a very long way in the last five days, in the last five years. And today when companies say we want to become data driven or we're on the path to become data driven, there's a lot more behind that and there's a lot more that we know how to do, whether it's organizationally, culturally or from a technology perspective.

That's spot on. And you mentioned here the organizational dimension, cultural and technological dimensions to becoming data driven. What do you think are some of the main challenges affecting organizations who truly want to make the most of their data?

Yeah, that's a tough question, right? Because I think for, you know, if you think about sort of the, there's in terms of, you know, what does stop us from becoming data driven? You know, in a way, sort of the last, you know, the last year or so, you know, with COVID 19, that has really redefined and you know, completely changed how we think about work and how we think about data in particular. Right. And so in some ways that has actually accelerated everything around the cloud and data and also sort of fast tracked all of the challenges that we see. Right. And so I think there's probably three main trends that we see that are contributing to these challenges. The first is there's a lot more data. Right. Especially if you sort of think about like the rise of fintech companies recently, right. Like financial technology or financial services companies heavily rely on third party data and there's just a lot of it, right. So it's not uncommon for a company to have thou thousands of data sources that they rely on. And you know, that's sort of, sort of the first trend. I would say the second trend is that there's a stronger, like more reliance on data. So actually, you know, to your previous question, people today understand that data is important, that it's important to become a data driven organization in order to create an advantage for yourself. If honestly if you don't want to be left behind, you need to be thinking about how to become data driven. And so as companies rely more on data to power their products, to power their decision making, that means that they are also, you know, the stakes are higher for data. It's just like you can't afford to make as many mistakes. When there's higher reliance, there's more eyes on data. People care more, a lot care more about it. It's a bigger deal. Right. And then I would say the third trend is we're seeing this, you know, fragmentation of what I would call sort of the modern data stack. Right. Our data infrastructure and our data pipelines today are really complex. There's a lot of different systems that you can be using. People have choices. You might have one team running on Snowflake and Looker, and you have another team running on S3, Athena, Hive and Tableau, and a third team running on something totally different. There's really so many options that you could go with. There isn't quite yet a standard stack. And so as we see sort of these are three trends really kind of happening in the last few years. I think they're going to continue in the next decade, the next five to 10 years, I will say, and will only exacerbate the challenges that we're seeing. Organizations who want to become data driven. And from what I see, the number one problem that people have as soon as they want to become data driven is sort of what I kind of refer to as trust in data. Right? Or data trust. The number thing, the number one thing that people run into is like, okay, we have all this data, we have all these reports, we have all this, like, machine learning models that we can use. Awesome. Like, let's go, let's like, let's start writing, let's start using. And then suddenly someone is like, hey, hold on a second. There's like stale data here that's impacting my model. Or wait just a second, I'm looking at this table and you know what? The values here don't make sense to me. They're all negative. And there shouldn't be negative. Like, that's really weird. And then next question is, people are like, well, I don't even know where this data come from or who's using it, or should I be using it? Should I be using some other table? And all that leads to one big question of, like, can I even trust this data? And I think that until we solve that, that is going to remain the single most important challenge in order to become truly data driven.

Okay, that's awesome. And I want to talk to you here about potential remedies for these challenges. And one thing that is central to Monte Carlo's mission is the importance of data qual. And here you're mentioning it as one of the challenges organizations face which is trust in data. So as data becomes even more critical for decision making and product development, data quality can have massive implications for an organization. Can you outline your thinking around data quality and what you think are key components of a successful data quality strategy?

Definitely. So, you know, the problem of data quality is not new, right? It's been around for 40 something years. But I think the way that we've been thinking about data quality needs to adapt to the new way that we think about data and data infrastructure. So I'll explain what I mean. You know, there's this saying of kind of like garbage in, garbage out, right? Which was really, you know, appropriate for, I would say, what is sort of the traditional way of thinking about data quality, right? Where if you sort of think about the standard pipeline and the standard way, we kind of had like one ingestion point, and then you just needed to make sure that the data that you're ingesting is of high quality. And then you knew that the data that you were using was going to be of high quality, right? So there was one point at which you needed to make sure that the data is accurate. And I think that for the last couple of decades is really what data quality was centered around, right? Profiling the data, making sure that whatever you're ingesting is of high integrity, and then making sure that sort of on the other end of that, you're using that data appropriately. However, the challenge with that in today's world is that the way that we manage data has become a lot more sophisticated. And so you might start by ingesting into the data, but then you have so many transformations, different layers of the data downstream, right? So you might have a data lake, maybe multiple data warehouses, ETL, ELT, BI, machine learning models, etc. And data can actually go wrong at any step of the process, not only just upon ingestion, right? On top of that, there are different people in different organizations in different steps of this process. Sort of imagine this pipeline all the way on the left side from ingestion to all the way on the right side to actually consumption of the data in reports or machine learning models. There are different peoples along that step, right? So in the past there's only like one organization or maybe one, a couple people were really responsible for the data. Today you have engineers upstream, and then maybe you'll have a product, data product manager, and then you might have a data scientist or a data analyst or data engineer, and then you might have an ML engineer and you have all these different titles and people and they can all contribute to the problem of data downtime. I think in order to really think about a strong data quality strategy requires thinking through what are you trying to solve through, right? Are you trying to like, where is the source of the problem typically? And how are you addressing not only the question of like where data is breaking, but also who owns that problem? Right. And who's actually going to fix the data quality issue?

You know, one thing we see data teams struggle with is really trying to gain executive or leadership buy in around data quality initiatives. How would you go about determining return on investment over data quality initiatives so that data teams are better equipped to get this buy in?

That's a great question. You know, I think in terms of sort of getting buy in and getting sort of executive alignment or just alignment in general right across the organization in order to get a data quality initiative. I think there's a few things, right, ROI certainly being one of them. But taking a step back for a second, the first thing is that you actually need buy in that data quality is an important thing to anchor on. Right? And actually, you know, some of our customers will ask me like, is data quality something that, like I just need to invest in as a one time initiative and then, you know, I'll forget about it for the next five years? Or is this something that I need to think about consistently, like, you know, how much, how much calorie input should I consider in this? And you know, I think for a company that's really becoming data driven and a company that's truly putting data at the forefront, data quality or data observability, whatever you call it, it has to be a consistent top line kind of thing that you're thinking about all the time. It has to be part and parcel in terms of the same way that you're thinking about how to make data accessible, how to store the data, how to analyze it, you need to also make sure that you're thinking about how to trust it. When you think specifically about the impact of data quality and how we measure that. And that goes back to your question of roi. There's two main metrics that we think about. One of them is time to detection, which really refers to how long it actually takes the team to identify an issue. Oftentimes we actually work, you know, in sort of speaking with companies, it could take them months to detect a data quality problem. And that's not uncommon, right, because maybe some table, you know, sort of broke down somewhere and just nothing was alerting on that. Right. So how are you supposed to know, right. There might be all these like silent failures impacting your business and you're learning about them weeks or months later. And those silent errors can have like, you know, many millions of dollars in impact on your business. So you can't really afford to prolong the time for detection here. And so time to detection is a very important metric. The second key metric is time to resolution. And that pretty straightforward metrics like how quickly are you able to resolve a data incident once you're alerted on that. Right. And how many people are involved in that process. And so both of these metrics together give you a strong sense of like how strong operationally your company is, is in actually resolving sort of managing data quality. And I think having a very strong lens on these along with sort of what we call data downtime. How are you improving on data downtime overall will give you again a good sense of sort of the ROI and the impact of data quality 100% and.

Zeroing in on the problem of alignment between the data team and executive team. How important do you view the role of data literacy when fostering align and when scaling data science in the organization?

Yeah, I think data literacy is probably one of the most important things that companies can invest in as sort of a really kind of like table stakes investment. Right. And let's define for a second, what does data literacy means? Right. I don't think data literacy means everyone in the entire company needs to learn SQL or R or whatnot. Right. That's not data literacy. Right. That's. I don't think that's necessarily the goal. I think what we need to define is how do we want each team or how do we want our company to engage with data? How, what kind of decisions or what kind of goals are we going to drive based on data? So it always has to go back to the business outcomes. Right. So it starts with saying, what does our company want to achieve this year? What's sort of like the big hairy, audacious goal that we're going after. And based on that, what kind of data does our team need to work with in order to get there? Right. I actually, you know, there was one chief data officer that I spoke with that actually created this matrix, you know, where you can see on sort of one access the different teams, different functions like, you know, marketing, sales, customer success, product R and D, et cetera. And then on another function is sort of of the different kind of skill sets and data and the different languages and basically kind of like provided a score for every company for every Function, sorry, With goals. So the marketing function literacy is obviously very different than product or engineering. But having some way of saying this is how much we think a team should be able to do with data. And this is what we expect folks to be able to work with. Data will basically create sort of a baseline across the company so that people across different functions could use data effectively. So I think there's a little bit, you know, when we say data literacy, there's a little bit of a risk that, you know, we sort of go overboard in one direction. I think it's really important that we, we, we agree it's very important, but we tie it back to what are we trying to achieve with data. It's not data literacy just for the sake of it, it's for the sake of empowering us to, to achieve better outcomes.

Yep. So another thing that is central to Monte Carlo's mission is the concept of data observability. So this is a relatively nascent term and is an emerging category in the data space. So can you define data observability and how it can help so many organizations data quality problems?

Definitely. So I think the concept of observability is a very interesting one. And actually, you know, it originally stems sort of from the world of software engineering and DevOps. Right. So you know, if you think about observability in the concept of software engineering, it's a kind of that has emerged in the last couple of decades as a fast growing area that really supports DevOps teams who manage application and infrastructure downtime. Right. And so these teams track metrics that help understand the health of their systems. Right. So observability really speaks to the ability to determine the health of a system by observing its output. Now companies manage infrastructure and application downtime really diligently. Right. Like you can't imagine a company sort of apps just going down or if their website just going down. Like, you know, today we sort of what we call like five nines, right. Every company needs to have five nines of application uptime or sort of strives to that. Right. And you know that the sort of the tools of DevOps observability has helped software engineers better manage the health of their applications and their infrastructure. Today you can't really imagine any engineering team operating without something like New Relic or AppDynamics or Datadog or Grafana. Some of these tools that help engineers make sure that their apps and infrastructure are up and running. Now if you think about that corollarily, in the world of data, we actually are Trying to do the same thing in terms of, like, running really a complex system and having, like, you know, really great data that we can rely on, but we don't have the tools to manage them. Right? Which is a little bit crazy. Like, we don't have. We don't actually have the corollary of that in data, and yet we are holding ourselves to this very high standard of creating very reliable, fast, speedy and efficient data systems. Right? Which is kind of crazy. And so actually, what I strongly believe in is that we need to take the same concept of observability and apply that to data. Right? So if you think about, like, what does it mean to have really reliable, trusted data? We actually sort of broke down sort of data observability into five core pillars that we believe that if you monitor for and, you know, really sort of think holistically about these five pillars, then you can have a strong sense of the health of your data. And so these five pillars are actually. I'll walk through them quickly. But the first is freshness, which speaks to the timeliness of the data. You know, as is the data arriving on time. Like for a particular table, let's say it gets updated like five times a day and it hasn't been updated at all today. What does that tell me, right? Is there an issue or not? The second is volume. So the volume of the data, you know, if I expect sort of 10 roads and suddenly I'm getting a million rows, you know, what has changed and why? The third is distribution. So this is a whole slew of metrics around the field level and the values of the data. So I gave previously an example. Say I have a table that's fully populated, but it's all kind of like null values or negative values, very different from what I expect. There might be an issue. And then the fifth pillar is lineage, and lineage, this is both the table level and at the field level, really helps us just get a view of the data and for a particular asset, all the sort of upstream dependencies on it upstream, which could give us clues to sort of the root cause of a particular problem, and all the downstream dependencies of table asset, which could give us an understanding of the impact of a particular problem. And so together, these five pillars can actually bring you the confidence and the visibility into the health of your data and makes it easy to understand and sort of quantify the impact of data quality on a particular business. And so what we see best in class data teams do is not only think about sort of great infrastructure and great Pipelines, but also think about great data and high quality data and how do I actually manage the data in a way that I can actually trust it.

That's awesome. And how does Monte Carlo intend to solve these problems? And can you walk us through some example, use cases that you've worked on?

Yeah. So luckily, the solution or the way that I think we need to approach it is also something that we can borrow from software, software engineering and from the best practices of DevOps. Right. So before sort of talking about what, you know, I think the, the best solution is, I'll. I'll just speak to like what people might be doing today. Right. So there's a lot of, you know, most data teams or many data teams have traditionally resorted to sort of manual ways of making sure the data is trusted. And what do I mean by that? You know, there might be a team of sort of tens or hundreds of people who are literally just staring at dashboards and making sure that the numbers are accurate. Right. Like, I remember personally when I was responsible for a particular report that our CEO and our board was relying on me and my team. We would wake up every day and spot check hundreds of data points in that dashboard to make sure that nothing has changed, the data is still accurate and we can still use it. If by chance something was off, we'd basically spend the rest of the day trying to understand what happened and why. Very manual ways of making sure that the data is accurate. It. I think that would have worked fine five or 10 years ago when there was like, you know, maybe just a small handful of people using data. Really not often. Right. And maybe once or twice a year. Today's world, we have thousands of people in a company that are relying on data in real time. Right. It could be, you know, maybe the entire company is actually using data in real time in some organizations. And so you can't possibly think that like manual process is going to be sufficient. Right. We have to be crazy to do that. And so if you take some of the concept of observability and apply that here, I actually think we need to think about the corollary of that. Sort of like a new relic or appdynamics or datadog, but for your data. Right. And what does that mean? It's a solution, a data observability platform that can help with the instrumentation, the monitoring, the alerting and the collaboration and the resolution of these data issues that are sort of encapsulating these five pillars that I talked about. I think a strong data observability Solution needs to connect to your existing stack and to connect to it end to end. Right. Meaning it's not sufficient to just have data quality or data anomaly on a particular part of your stack, like just a couple tables in your data warehouse or just a particular sort of data set, you actually to really sort of address this problem, it has to include wherever your data is. So including your data lake and your data warehouse and your BI or your machine learning model. And you have to have that end to end visibility from ingestion all the way to consumption. You know, I also think that you need to think about how do you provide sort of rich context to each of these sort of, you know, problems that sort of the platform identifies. And so having sort of strong root cause tools in place will help you, will help you actually determine quickly what the problem is and how to resolve that. And actually probably the best kind of way to approach this is to prevent these issues from happening in the first place. And so what we're finding is that by exposing some of these best practice observability in data, where companies are actually able to reduce their data downtime by 90% just by implementing some of these sort of best practices and bringing more awareness around what does our data look like? What is the health of it? And more metadata to help us determine the health of our data.

This is really exciting. And now one thing you mentioned, especially at the end here, is metadata. Can you speak to us on the rise of metadata management tools and data lineage tools and how you see them impacting organizations?

Yeah, so the whole kind of emergence of the metadata space has been super fascinating. Right. I think in the last couple of decades we've become really good at kind of like collecting and tracking data. We've actually gotten to a point where we're like hoarding data. You're like, the more the better. Bring it on just wherever it is, you know, let's just collect it more. And often what we see is that, you know, companies typically have more data than they can ever manage, let alone process or analyze or like make sense of. And they often sort of start drowning in lots of data and asking questions like, okay, what data do I really need? And then starting like, you know, there's like a gold, you know, gold, silver, you know, what have you, and you know, like Tier 1, Tier 2 to 3 data. And there's all these like different acronyms like ARR Revenue 1 or ARR Revenue 2, all these different naming conventions to make sure that you're using the right data set. And it gets really, really complicated. And potentially it's like self inflicted pain in some way. And I think what we're seeing in the metadata space is actually kind of like in the same way there's this rise of metadata where we're like, oh, you know, there's a lot of data that we can collect about our data. How awesome that is. And now we're just like starting to hoard that a little bit. Right. And companies are like, let's collect lots of metadata. I'll just say something really controversial for a second. I think metadata by itself is completely useless. There is nothing that you can do with just metadata on itself that's actually practically helpful to your business. I think metadata is very incredibly valuable when put in the right context with the right business outcome in mind. Right. So I'll take Lineage for example. Lineage is something that people get really excited about. Oh my God, like, show me, you know, the best sort of lineage map of my business. They look at it for five minutes, they're like, okay, let's move on to the next shiny object. Right? Like, I'm done.

Right.

But where does the power of Lineage come from? Right. Lineage is valuable when it's used in a particular context to solve a particular problem. So for example, if there is a particular table that is not. Has not gotten updated, maybe there's a specific freshness problem. But you know what, there's zero dependencies on that table downstream. So who cares about that table? Maybe it's not being used at all. So I shouldn't care, I should worry about that. I don't need to know about that. Maybe I just need to deprecate that table. On the other hand, if there's a particular table that's not getting updated and there's hundreds of thousands of dependencies on it, specifically those dependencies are users that are making decisions about the business, or there's actually customers that are getting access to that data. Or maybe these are like mission critical machine learning model. In all of those instances, you want to know about that freshness problem as soon as possible. Right. And so this is an example where an understanding, a strong understanding of Lineage can help you operate your infrastructure better, can help you operate your business better. Not just Lineage for the sake of Lineage. And so I think we need to get a lot smarter about metadata and particularly apply it in the context of solving real customer problems. That's really where the magic happens.

Definitely agree with you here on business value alignment. So pivoting away slightly from talking about data quality to just data infrastructure in general. Another thing that's often talked about this year is really the rise of the data mesh and how it can potentially solve some of the bottlenecks of the current paradigm and data infrastructure. I would love your thoughts on what these bottlenecks are and what a data mesh is and how would it solve these problems.

So I think the concept of data mesh has really, as you mentioned, become more and more of a hot topic recently. And I think it ties back to really everything that we just discussed. More and more organizations want to become data driven, realize that it's important in order to become competitive in markets. It's important in order to become successful in business. More and more data sources, more and more data consumers, fracture, fragmentation of the data stack, et cetera. All of these actually led to this point in which companies are asking themselves, what is the best way to organize ourselves or to organize our organization in a way that helps us adopt data. And in the current model, the more data that the historical notion has been, the more data you have, the better. But actually, at the point that we're at today, some of the bottlenecks include data that's not being used at all. Teams using different kinds of data. So maybe the finance team will use a particular data set, but the sales team will use a different data set, and the customer success team will use a different data set. All three teams are relying on data, but they might come to totally different conclusions about what needs to happen in the business business, because they've just been using different data sets or looking at different metrics or interpreting data in a different way. Right. And so as every company sort of strives to become a data company, there's different use cases that emerge. And as a data engineering team, you're really trying to serve lots of different use cases and lots of different consumers in a way that can be really, really challenging. And so that one of the most common sort of questions that, that I sort of get from customers is we are responsible for the trust of the data. We're responsible of kind of our analysts or customers wanting to use data, but we can't actually fix the problems in data. Why? Because it's sort of the teams that are upstream from us who are producing the data, potentially an engineering team. So a data team will be somewhere in the middle of the chain where downstream from us, we might have people consuming the data that we're working with, and upstream from us might be an engineering team that's producing the data, for example. And so I might be on the receiving end of lots of questions around when was the data updated, when was last refreshed? But I don't actually own the data to the point that I can fix any of those problems because it's a different team. And so in those cases, oftentimes starts this culture of sort of finger pointing and blaming and saying wait, hold on. It's like this other team that's responsible for that part of the data pipeline. And you know what, I can't really solve that. And so there's lots of questions of ownership and that leads to a situation where it's hard to become data driven. Right. And so then the question is, you know, who actually owns the data? What's the best way to structure this? Is there a centralized model? Should there be one team that sort of owns everything with a center of excellence? Or should there be like distributed teams with embedded data people within organ, within different functions? And we see companies all across the spectrum all the way from centralized to completely decentralized. Right. And so within the context of all this sort of the topic of data mesh, which again actually sort of borrows on concepts from software engineering, it's sort of defined by jamaic digani from ThoughtWorks. And this basically brings kind of the theory of domain driven design to the concept of data infrastructure. And really what sort of the idea here is a proposal for how kind of to solve the question of ownership and accessibility and to solve the problem of silos and basically by bringing the best of both worlds where you can have sort of a universal domain agnostic and automated approach to things that need to be standardized at the company level like data governance, data lineage, data monitoring, sort of best practices. And on top of that have domain specific or kind of like autonomous teams that have ownership across their data pipelines and in those cases empowering those teams with self serve discoverability, self serve observability and others tools in order to really truly adopt data. And so the concept here is how can we organize our people and our organization and our tools in a way that allows us to become data driven? I will say it's sort of the early days of data mesh. And so I think companies mostly sort of ask themselves like, you know, how do I actually adopt this? What would be sort of the, you know, what are the first steps to do that? And you know, I think it also, it's, there's something about this concept and how simple it is that's very appealing in the sense of sort of enabling, enabling organizations to fast track their path to becoming data driven.

And how do you see data mesh Adoption growing in the future?

Well, I actually, you know, in terms of sort of the, if you think about the last couple of years and how quickly it has been adopted, I'm really curious and excited to see sort of the next couple of years and how it's going to accelerate. You know, I think there's definitely, there's actually a Data Mesh learning group that has sort of emerged on Slack where you can sort of, you know, catch great conversations by leaders like Zhamak and others and actually ask them direct directly questions about Data Mesh or others. And more and more data teams are adopting that. So the platform team at Intuit actually recently wrote a series of articles about their experiences sort of on their path towards Data Mesh. So there's definitely more and more, both organizations adopting Data Mesh and also writing about that, which is very helpful for people kind of wanting to, to learn about that and to figure out how to adopt that or use that model for their own organizations.

That's very exciting and we'll definitely make sure to include some of these resources in the show notes as well. Finally, Bar, it was great to have you on the show. Do you have any call to action before we wrap up?

Absolutely. Well, first of all, I always am happy to talk about data science and data analytics best practices and always looking to connect with folks folks. So feel free to reach out. Recommend sort of checking out our blog, Monte Carlo data.com blog and sign up for our newsletter if you'd like to keep up with all things data observability in general. Just really excited about where the data industry is heading and so really looking forward to hearing what's top of mind and exploring exchanging ideas. So feel free to reach out out.

We'll definitely make sure to link to all of these resources in the show notes now with that in mind, thank you so much, Barr for coming on today's episode of Data Framed.

Absolutely. It was fun. Thanks for having me.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed Barr's insights on the data quality challenges organizations face and how Monte Carlo solves them. If you enjoyed this episode, make sure to leave a review on itunes. Our next episode will be with Elad Cohen, VP of Data Science at Riskified on how data science is being used to fight fraud in E commerce. I hope it'll be useful for you and we hope to catch you next time on Data Framed.