Observability with Raj Joseph - AgileData #38

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hey, thanks for having me, Shane. I'm Raj Joseph, CEO of Deque Labs.

Hey Raj, thanks for coming on the show. We've got an exciting one today. Well, they're always exciting, but this one's another exciting one for me. So we're going to talk about observability and all the other abilities and data words that we can bundle into that one and what patterns are useful and what patterns aren't. Before we do that, why don't you just give the audience a bit of a background about yourself and your history of working in this magical thing called data.

Sure. So my background started off from data engineering background. So started as an engineering back in '98 with an engineering degree right after college, and then went through lots of startups and bigger fintech companies and slowly moved into product leadership for a data marketing company. So there I built a big data platform with a focus towards quality, and one of the challenges there we did was trying to consolidate data from different sources and trying to build a consumer profile. And of course, lots of duplicates, lots of bad data. So my life in and out for a long while has been in data quality. And then later on, I also started a services company in 2010, focusing on data, data management and governance. And then in 2020, came into an opportunity to build a modern data quality platform with a focus of bringing in observability and business quality together. And so that's kind of where it ended up. Too. So that's kind of like a brief profile of me, actually. Yeah, good.

And that's one of the reasons I wanted to have you on the show is because you've been there, done that, you know, a bit like me where I've been a practitioner of delivering data to customers, finding all the horrible things that are nightmares for us, and then how do we create products or services that fix that? Your concept around bad data reminds me of one of my favorite t-shirts is Breaking Bad Data t-shirt. Trifecta created it. I'm tempted to make a new version because the one I've got, I wear so often it's got holes. Maybe that's what we'll theme this whole podcast around, right? Breaking bad data. So, To start off, observability, data observability is the new word coming out. The patterns underlying it have been around for ages. There's always a risk with new terminology, it's just buzzwords rather than things that have got meat and potatoes behind it. So, for you, data observability, how would you describe that pattern if somebody asked you for it?

So, for me, I think observability is a capability because it's really when you look into what you are kind of observing at the surface. Shallower, more broader data, but more faster processing. So when you are having a bigger growth in data, it's much more easy capability and a technology that could be adapted for multiple different reasons. There is software observability, as always we know about. And now with the data focus, we have this concept of data observability. Even within data observability, for me, I try to see that observability concept can be used in multiple ways. And for me, it always goes back to the quality. And how can we observe a data to figure out some aspects of quality? Because quality is a bigger, broader way, broader name and category in itself. And for me, observability gives some kind of measure around reliability. So the word that I can perceive and conceive when I hear data observability is, okay, yeah, observability is a capability that I can use to measure the reliability of data delivery. So that's one aspect that immediately comes to it. But there are other dimensions of things that we come in this day and age with modern data cloud architectures and things like that, where architectures are spread across hybrid and also modern cloud warehouses where cost is a factor. So you can also use observability as a capability for cost. And then also if you want to measure some of those business quality in a way, you can also measure if it is, there is any anomalies that you can check, and et cetera. So, but more around reliability is where I kind of tend to focus.

Yeah. And so for me, I agree, it's data observability is a bucket. It's an overarching term, a bit like a cloud analytics warehouse. So it's an overarching term that has multiple parts to it, multiple patterns. Each of those patterns have value. And we kind of saying there's a new category, there's a new group of patterns that we want to put together. One of the ones I want to pick up on that you talked about is this idea of scale. So if we think about if our data platforms, our data systems, the data we work with is tiny. And not tiny number of rows, but tiny number of columns or data sources or users. So we're at a really small scale. A human can observe, right? We go and grab one table from one system. I can observe that. If I'm only running it once a day, I can go in and observe it happen. If I've only got one user, I can observe what they're doing with them to see where the problem is. So that's all good, right? But as we scale, as we add 100 different systems, as we add 1,000 users, as we add 3,000 tables with 60,000 columns, we bring in SAP, we can't observe anymore as humans. We need the machine to help us. And for me, that's what observability is about, is the machine doing as much of the work as it can, because it ain't, even with ChatGPT, it ain't fully magical. I don't have a magical observability hat where I just dump all my data in and I go to sleep and don't worry, and it tells me, but it gives me hints. I like that way that you broke it down. We can look at observability patterns for quality. We can look at observability T-patterns for cost. We can look for reliability and we can look for anomalies. They're really good 4 areas that we can do. So what's your favorite one? Which one do you wanna do first?

I think since my background has been, and most of my life has been in data quality, I would just focus on quality as a place where to start. And it's a, it's a broader category in itself because if I ask you, Shane, what is data quality, you may come up with a definition than what I may come up with. And most of the time, what I have understood in the industry is it's really in the eye of the beholder, meaning the roles and the responsibilities defines what they're looking for. When you're talking to a data scientist and ask them data quality, they are more talking from a fit for purpose, meaning when they're building analytical models, their needs of that particular model may be very different than what they might have done a day or a month back. So their needs are very changing. And is based on the business strategies and the tactical things that they do in terms of solving those strategies. While if I talk to a data engineer, theirs is maybe a little different. Maybe they are more looking from reliability of the data in terms of making sure that the data reaches from one source to another target. And then also their pipelines are processing as it, and then try to prevent as much as they could before it lands. So that's where I would start observability with a focus towards quality. And then I'd say it's a break it by the product topic itself, based on the roles and responsibilities of the user.

I agree. I think if you take a persona view of it, you get different answers. So if you say to a business consumer, why do you care about quality of data? They go, I need to trust it. If you tell me I've got 100,000 customers, I just expect that number to be right. Why would it not be right? So it's really interesting in the data world, right? We have this habit of just giving data to our consumers and getting them to find the problems. It's a car with only 3 wheels and we go, ah, yeah, does the other wheel really, the 4th wheel really necessary? And then the consumer hops in the car, it falls over, scrapes on the ground and we're like, oh yeah, we can fix that. So for me, that's what they say. Whereas, you know, you're right. If you're an engineer or you're in the plumbing, then you care about some other things. One thing I find really a bit of a struggle for me is if you look at DAMA and data management and data quality, it's been around for decades. There's always the abilities. There's always 6 abilities for data quality buckets or categories you're going to look at. And for me, I've always struggled with a pattern or a rule and where it fits. So the one I had the other day is, let's say that we have a table and the table's got a foreign key on it. Yeah. So each of the values in that column are meant to be unique. There's only ever one. And we bring that data in and it fails that test for whatever reason. Is that a uniqueness test that failed? Is that a duplication test? Those categorizations that we often hear about, each of the rules that we could use could go into any of them. What have you found, like, when you say if you had to categorize the quality as being high or low for a certain thing, do you use the standard 6 from Dahmer, or do you think about it differently?

I know the 6 quality dimensions you're talking about. So, I was at some point as a practitioner, and then as a preacher of data quality and all those principles. I was madly following all these dimensions without understanding why and what and etc. to a specific level. But what I really more and more understood is it's more than the measurement. The purpose actually defines the measurement. So measurement without a purpose is messed up, right? So the 6 categories are very useful when you to get into some kind of defining the purpose, or if you don't have a structure, if you don't have a framework of how to measure, that kind of gives it. But what I have seen from working with different clients and et cetera is each one has a different level of expectation and governance process that they kind of manage. So for example, I'll give you one example. This is a bank in Canada. So this particular bank, they don't look into quality the same across all the data. So first, they, what they do is they define data in 3 levels, level 1, level 2, level 3. Level 1 is good, I have the data. That's great that if it is there, it's good. Level 2 is something that is used for reporting and some kind of data science model. Level 3 is, it is used for KPIs, business metrics, because the leadership is gonna be looking into it. Could be example, could be is revenue metrics across products, while your level 2 could be is a product name. And level 1 is, could be like some flags that may be sitting there, ones and zeros of sorts, right? Some bits that needs to be turned on and off when application, a feature sample. So for them, when they see level 1, do I care about quality? No. Do I care about reliability? Make— meaning the data is there or not? Yeah. If for some reason, if there is a big data gap or data leakage, yeah, I want to look into it. Maybe a system is down, maybe something has happened. So the level of detail of how I need to measure and the quality is based on the criticality of the data derived from the business use of that. So, so that's how they do it. So level 1, level 2, 3, and then so as you go up, your 6 dimensions of data is much more higher needed. And it can be even beyond that too. It could be even like a subjective— these 6 dimensions are more quantitative. And if you want to even go subjective, meaning like, yes, the data is there and the quality of the 6 dimensions are saying great, but nobody using it, Then what? Why care? It's kind of like that.

Yeah, it's the boil the ocean problem. It's the same problem we have with data catalogs where we bring every SAP table in and column, and now we've got 60,000 columns. Nobody's going to catalog all that, and they shouldn't. We catalog what we care about. So I think that's good. The way I think about it is blast radius. You know, if we have a problem with the data, whether it's quality, cost, there's anomaly, or it's not reliable. What's the blast radius? What's the impact to the organization with that data being wrong, not being trustworthy? And then we tailor our patterns because of it. And that's why often for me, we can figure out a small set of things we want to focus to begin with because we understand what data is really, really important. It's normally the data we touch first, right? Not the first bit of data we bring in, but the first data we expose to a consumer because you've asked for it. And then the second thing is often, you know, we have a problem and then we have to go and observe and manually figure out what the hell went on. And then we have to go and fix something. Again, that's when we need to bake in some observability. We have to say, okay, well, let's just make sure that if that happens again, we identify it. And ideally we go fix it so it can never happen again, but we definitely want to be able to observe it if it does and be notified.

Exactly.

So again, start small, bake in the simple ones, go figure out what fields shouldn't be null. Real simple one.

Do that first. Exactly. And those kind of checks is what I call as wide and shallow validations, like null checks, blanks, empty, is uniqueness, or even the number of unique or data distribution in terms of statistical evaluation that you do. You're kind of like checking more in a wider, faster, bigger aspect of amounts of data and more shallow validation. So you are still on the more at the surface level. You are not going deeper. You're not having any narrow validation, like what you may do in a business checks. And I think that's okay at that level 1 zone we are talking, the boiling the ocean. But as you go further down, that's where we start defining what is critical. And maybe when it comes to business quality checks and things like, I think I always tended to see, to your point, impact or looking at a business impact. The first thing is what is the business impact? And it's not an easy answer to solve, but the way I kind of create is like, okay, business impact, meaning what are the KPIs? So that's KPIs can be easily translated to the actual data that is being provided towards to make the KPI metrics. And so, so I kind of look into the metric stores or the business metric stores, and from there I kind of connect to the tables and the columns and then kind of drill down into it. Okay, these set of data are much more business impacting data. Versus all these other non-business-impacting data.

Yeah. And for me, usage of the information is something we should be monitoring. We should be measuring, we should be logging, we should be measuring, we should be monitoring how often a certain piece of data is used and who's it used by. And that helps us with our blast radius. So we can say, well, actually, if the data here becomes untrustworthy, we already know who's using it, right? We can communicate with them or we can understand, is that affecting 1 user, 1,000 users? Can't quite tell, is that affecting 1 piece of data that that 1 user happens to do manually that drives the whole profit of the business versus the 1,000 users that are just in the factory part of the process or not. You often don't see that usage logging as part of observability. That tends to sit into another category. So for me, it needs to be observability. If we're saying we need to be able to observe things that are important at scale without a human doing it manually, that's our definition, then that's in there. The other one that's really interesting for me is, and you talked about it a little bit there, is around cost. So, in the old days, you know, when we had on-prem databases, we had an Oracle or SQL Server, we could run a hell of a lot of observability tests if we wanted to, because there was no additional cost of hardware. There was just a cost to the consumer, right? Because we take all the CPUs, their reports would run slow, or our ETL would run slow. In the new cloud world, when we add this new observability capabilities, this new workload, There is a cost. We start paying for more credits if you're using one that's credit-based, or more compute, or more CPU, more slots, or depending on which technology you're using. But you actually have a larger overhead on that platform. And what we've seen is actually running our observability stuff actually costs us more than running our transformation stuff, unless you really think about the pattern. And I'll give you an example. We might say, we've got this table, it's got some categorical variables, it's got some regions, and under each region is a bunch of cities. We know that there's a mapping. We know that in the source system, somebody's gonna go change that mapping. They're gonna go remap a city or a new city turns up or something's gonna happen. And we want to capture that from an observability point of view because we know that for some reason, that piece of data, that mapping's used for our financial forecast that's going out to our shareholders. And so it's a high-risk piece of data for us. For us. A lazy architecture basically says, scan those tables for distincts. But what you're doing is you're scanning that entire table every— and that's an expensive compute. So you need a pattern where actually what you want to do is break out that unique set of values into their own small tables. And every time a new piece of data turns up, compare that piece of data to your reference table and go, have I seen this one before? And now you're only processing new rows. You're reducing your cost the compute. So you have to bake that in. Do you find that? Do you find that actually observability, quality, anomaly detection, reliability, cost monitoring, usage logging, they're all high expensive compute in the new cloud world?

Yeah, you're right, because if you see the adoption of cloud is growing and it will grow, no question about that. But what I have seen, it's very interesting, is the adoption of cloud is also a little bit slowed at some point because of the cost factor. We have folks using all these modern cloud warehouses and cloud lakehouses and things like that, and then they move the data, and then every time now they query, there is a cost. You want to be processed even faster, then you need to pay more. The size of the warehouses or the size of the compute power that goes around it, which makes sense to a degree. And then to your point, you cannot be observing the whole ocean because you are actually computing more and then therefore the cost is more. And so, and in some business transactions we have come across, if your spend for cloud warehouses or lakehouses is $1 million or $2 million, your observability spend needs to be a fraction of that. It cannot be like $300K or $500K or massive, which questions the value of that itself. And the way to do that is rather than spinning up and going across all observability metrics, going into the table level, you kind of used some of the logging mechanisms of that. For example, if you take Snowflake, Snowflake provides your metadata log on in terms of volume freshness. I mean, almost all this modern data warehouses provides time travel mechanisms. So you can see the snapshots of time and when it got updated. So you can easily retrieve some of those information without doing any kind of complex query logics or looking do that, but the same thing when you have some other legacy databases or even on-premises, then it does not scale to that level, which is okay because it's not cost compute based. But however, the point I'm trying to make here is when you start with an observability, you don't necessarily don't do all, just maybe stick with a few dimensions of data. So we always recommend volume, which is row count. And then we look into the schema, which is more the column counts. and then we look into freshness, which is kind of like the timeliness of the data when it got updated. And then if you want another fourth metric, we also look into duplicates. The duplicates could be based on primary keys or it could be a composite key on a particular table or an asset. So we just gonna stick to that as every asset. So you are not doing too much deeper checks, but at the same time, kind of some kind of higher level check and statistics that kind of gives you some information. So that's one. And then the second aspect of that, what we do is we look into usage information. Who's using this table and what roles? Is it a system which is kind of spending for no reason, or is it an actual user from a consumption standpoint? At least that kind of tells you how relevant or how impactful in some ways, just because using doesn't also mean it's business impactful because somebody may be running a query for no reason. So, but at least it gives you another dimension of data to look, and then kind of combine and then go into a methodological process around defining some criticalities using this KPIs and other things, and then go around that.

I agree. I think the technology choices you make as part of your stack actually drive how you manage the observability cost. An example is because we are all based on BigQuery, we have a choice between slot-based and size of table, size of query effectively. We are able to profile the data within the same query. At relatively low cost. What we do is we profile the data and then store the data when we process it and store the profiling information next to it. If we were using something like Snowflake, where you're actually paying for the size of that compute, then actually you wanna be a little bit careful because every time you profile everything, you're gonna do it. Whereas if you're using something like a lakehouse where you're using, say, Iceberg as storage, then you probably wanna profile the data as it comes through again, but this time you're storing it as another object. You're storing it next to the data. So again, those patterns are slightly different. I think one thing that really interests me, if we think about cloud databases, they inherited the on-prem database patterns. They typically have an information schema. We can go and say, what's the name of the column? What's the column type? Is it a numeric, a character? What's interesting is they don't do that with profiling data, right? They don't store this column's got nulls. They don't store the standard deviation of that column or the maximums. And so it's going to be really interesting to see over time, whether they do, whether they actually capture that. Store it and surface it up for free without a cost to compute. Because actually, as data users, that's what we want. They know what the data looks like because they're storing the bloody stuff. So just give it back to us and don't charge us.

You already see, I think, all these modern warehouses and lakehouses has already going into a path of introducing some observability metrics as part of their stack. I mean, it's inevitable. I think they have to do it. And because I think that will make, I mean, more value add one as a platform, as an offering from it. And also it gets a little bit touches into the data management aspect of it because it's another big area of revenue, but also more adds value as a platform offering. So I would see more and more of these vendors trying to build observability metrics into their platform default. The challenge where a business has is, I mean, it would be great if a organization has only one technology, but that's never the case. So they use a combination of technologies. I mean, I've seen like people using both Snowflake and Databricks. They compete with each other and then also use for different reasons. And so, so I think that's kind of the reality of it. Even though these vendors, some vendors may incorporate it, still others wouldn't. And then your ecosystem has multiple technologies and then how do you measure that as the data moves from one place to another place and then also give a bigger whole vision of where the real data stack is from a health standpoint and from a reliability standpoint. I think that's critical.

Yeah, and I think that's a really good point you raise there around about the different technology stacks, but also about the end-to-end process. So again, we're still at a stage where a lot of the logic still gets landed on the last mile, gets in the Tableau or the Power BI or the Looker Studio reports, or they're cemented layer. And so we still have the ability for the quality or the reliability of the data to be broken at that last mile. I liken it to a kitchen. We measure the quality of the goods coming into our storeroom, the good flour. We measure the quality of the process we use to bake it in the kitchen. We measure the quality of how we serve it up to that little window, but we don't measure the quality of the waiting person who picks it up and then spits in the food before they deliver it to you. A lot of the observability capabilities are just up to that serving layer, right? But they don't actually then worry about what's happening in that last mile. And so therefore we can degrade the quality. So are you seeing that or are you seeing that market starting to move that people are starting to worry about what happens to that last mile of delivery before it gets to the consumer?

I think the whole, I mean, still there is a lot of innovation that is still yet to happen in the data world. Even the adoption of cloud is still in the initial early stages. Of it. We haven't even seen the full revolution of everybody in cloud or everybody has adopted to it. We see cost as a factor that this needs to be optimized and contained. The governance around a cloud in terms of privacy classifications, the end-to-end visibility is still a struggle because all businesses today have some on-premises, some legacy technologies, some modern technologies. And that's the reality of it. And in order to get this end-to-end visibility, it's not an easy problem to solve. And it's very challenging too. And then also like each vendor and each technology gives some part of it, not necessarily the same in terms of metadata. So you get some time travel kind of metadata logs from some of these modern stacks. But if you go to the traditional, you have to pass through lots and lots of logs to get to that point of data, right? So I think there is a variation of that. All of this makes it more and more complex. But I think rather than trying to get all of those visibilities, at least if you can define some metrics in each one of those stop points or pit stops, then I think that would help to get closer to that. In reality, I don't think any company can get 100% visibility on this end-to-end, but the more they can get will help them to be more smarter and more faster and in terms of their data-driven strategy.

I think that's a good point. People often underestimate that every time they add a new technology or a new moving part to their data stack, they're actually increasing complexity and therefore they're reducing observability unless they can apply those observability patterns to that new component, which they often don't because it's a last mile one, it's hard. But what's interesting there is you brought in a new one. You talked about privacy and classification of data.

Oh yeah.

And again, that's a form of observability. Am I holding credit card numbers? Am I holding licenses? Am I holding tables and names. If we think about observability as a way of getting told by the machine something happened at scale, because I can't go look at it, I'm not gonna go scan every column visually and go, that looks like a name. I want the machine to do it. So we gotta add privacy and data protection, observing what's happening in that space into our observability bucket. If we are saying we want observability of the whole system, don't we?

I think you're 100% right, because when you look observability, it's more this blanket catch-all term. And then you add data, still it's very generic, very broader, but the more context you put on top of it, and the context could be either your privacy, it's very focused on privacy, or it's focused on cost, or it's focused on usage analytics, or it's focused on health, or it's focused on reliability in terms of as the data moves from pipeline, or it's focused on business. Now you're having lots and different sort branches that goes from that and getting in its own division. So there is a privacy regulation that itself is a bigger, broader industry, and you can use observability in some kind of capability. So the application and the impact is more, but I think where it will become more and more meaningful is taking one subject like quality, data quality, or a business quality, and then trying to see how you can use leverage observability as a capability within the constraints of what you are trying to do. From different personas, from a leader, from a business consumption standpoint, data scientist, data analyst, and then also from a data engineering standpoint, looking at all those three different roles and how they see the data quality and then kind of mapping that observability would be very critical. And that's the same thing for privacy and classification too.

Coming back to the idea of persona, as a persona, difference between consumer and say an engineer, I want to observe some different things. I need different outputs to tell me whether I can trust it. Can I trust that my DAG ran versus can I trust the number you're giving me that tells me how many of those we have? And when we come to that, we start talking about language. If we think about data freshness, is the data up to date? From a technology point of view, we've always talked about SLAs and SLOs, service level objectives and service level agreements. And you talk to a business user, somebody that's not in the technology domain about those terms, and they just look at you blankly and they say, what? But if you talk about freshness, is the data fresh or is it stale? They kind of get that. I always go back to baking and cooking for some reason. My bread's fresh, my bread's stale, one's good, one's bad. And so they have expectations about how fresh it should be. It's about use. Some data doesn't have to be refreshed or fresh every day. Once a week's okay. Some data has to be fresh today. And so if we think about that terminology, that language we use. We've seen a new term come out last year and this year, and it's all hot and maybe buzzwashing, maybe not. And so it's data contracts. Now, I knew it was coming there.

I was just waiting for that, like, bingo.

It was either going to be data mesh or data contracts, right? So data contracts. And I look at them and I go, what's— so whenever I hear a new term come out, I look at it and I go, what's the underlying patterns? What are the patterns that are available? What's the context where I can apply that pattern? What action or outcome does that pattern give me? And so if I think about data contracts, if I think about something simple like schema mutation, I have a table coming from a source system, it has 10 columns, my contract is for 10 columns, an 11th column turns out that contract has been broken. And so a test or a pattern for observability of the number of columns that we expected has changed, is valuable. Again, if I've only got one table coming in, I can go and eyeball it. If I've got 1,000 tables coming in, I can't. So the machine needs to observe that for me. But that's been around for ages. That's not new, schema mutation. Then we go on to some other patterns around it. They go, is the context of the data, is the data types changed? Has the volume of the data changed? Has the relationship between two columns changed? So we start getting to quality tests in my head. And then there's that Nirvana one, which is actually, I actually have a contract. I have a policy that's written in code and that both sides are complying with it. It's actually driving everything we do. And I haven't seen anybody actually do that yet. From your point of view, data contracts, first question, buzzwashing or not? And then second question, where's the value? What are the patterns?

I mean, every innovation or every new principle that has come has been a buzz at some point of time. And And I think it's, again, if you look the hype cycle, it just goes up and then it just goes through this maturity lifecycle and then just kind of comes like, okay, this is hype and this is actually some value to it. So I think there is a lot of value that we can get in terms of data contract. I mean, if you really look what's happening is software observability is now turning into data observability. Service level agreements is becoming data contracts, data level agreements, and the concept of centralized governance is going more towards decentralized from a data mesh principle standpoint. Also, this self-serve data platform as a concept is making— how do you share the knowledge at the level where the SMEs are versus trying to consolidate and have business overtake and share that? Because it's too much responsibility to be asked from a business to own when they are not even the ones who are producing the data. So all of this in a theory and in the way it is constructed makes a lot of sense. But in terms of, so it's, I don't think it's a buzzword, but actually where it really, as an organization or a user of the platform of these concepts struggle is how do we implement it, right? What is right? And because these are all like newer concepts, even observability is still new and emerging because it hasn't reached its maturity in terms of how and well it can be used because you have here alert fatigue. If you turn on Observe across your whole ecosystem, it's gonna just look for anomalies everywhere. Some may be meaningful, some may not be. And as far as I know, when you get like 1,000 emails, most of the time you're not gonna be looking into that. It's just, so there is a maturity component that sits with that. And same thing with the data contracts. I feel there is a definitely a value. View, because now if you look into data quality or data observability or any of those things, both the business and the engineering, which is data engineers and data scientists, or even the business leaders, they need to collaborate together and they have to see it in some kind of common way. And I see data contracts as that. It just kind of gives them to meaningfully commit to both from an engineering standpoint and also from a business telling like, this is my expectation. Otherwise, what happens is the life of a data engineer has to suck because they think today it's great data, and then tomorrow you will hear from the data scientist, this sucks, because their needs change. So that kind of allows them to bring and bridge the gap and promote more collaboration and be more productive and kind of also tie back to the business impact you started on early and drive towards the business outcome. But the bigger, broader question is, what are the categories of data contract? Again, the implications are more because it could be privacy, right? Because I can have SSM, so that means it needs to be masked, it needs to be governed well. And then two, it also from a cost standpoint, it could be, or it could be from a reliability and accuracy standpoint, from a data quality too.

Yeah, that's a good point actually, is that the contract can include many things and it should, and then you have to decide what you're going to do about it. So. You could have a contract with a data provider that they won't send you names because you're storing them in a different way. And so you need to monitor that that contract's not broken, and then you need to decide what to do if it is broken. One of the things I love doing this podcast for is, is I talk to people and then somebody just goes, bing, in my head, and that just happened. People, we've talked about data products for a year or two, and I've worked for the last 10 years around this idea of an information product. And the way I describe it is we sometimes produce data depending on the persona, the consumer. So data scientists typically want data, not information, but our business consumers typically want information. They don't want a dump of data, right? They want to get an answer. And so if we think about data contracts and we think about a data value chain or data value stream where data is produced on the left-hand side by a source system, a machine or a human, and we do some stuff to make it useful and we give it to the person that actually wants to consume that data or that information. The data contracts have all been focused on the left-hand side, right? We focused on our problem. We haven't actually focused on an information contract. We haven't focused on our consumer about what good looks like for them. There's a bunch of tests, but there's no contract. And then if we bring that lens of cost into it, and that's just freaking awesome, because now what we can say is, look, here's the contract, here's the base contract. Data won't have any nulls, and it will turn up once a week. If you want to increase the level, if you want us to do more testing around it, you want us to get it to you faster, then the cost of that contract's going to go up. So there you go. I reckon 2023, we need, we just need to trademark this information contracts as important as data contracts, because we should actually have a contract with our customer, our consumer, because the things we do to make that data better quality, to make it cheaper to produce, to identify the anomalies, to make sure it meets the freshness requirements, to ensure that it it's retaining or complying with the privacy rules that we have. Each one of those has a cost in both human terms for us to build it and implement it and monitor it, and a system cost in terms of executing it and the observability engine. So yeah, for me, that's information contracts. There we go. Heard it here first.

Information contract could also have sharing agreements, right? In terms of who accesses this data and things things like that from a usage and accessibility standpoint. So like, because of so much choices in today's technology, and I mean, if an— I mean, if an organization is starting today as a new business, it's easy for them to adapt to some of these technologies faster. But as they grow, I think I cannot call this— when you're small and you have a data which is created by your own applications, you have a higher degree of quality. But as you grow, your data is going to be coming from outside partners, suppliers, and systems that you don't necessarily manage. And now you have what I call as inorganic data, but it's very essential for your reporting analytics and whatever you are doing in terms of your business outcomes. And I think that's where all this challenges comes in because in an RIA environment, your ecosystem has so much different technology, so much different nuances to it. Data coming internally, data coming from different stakeholders, which you don't necessarily contain. So I always take an aspect of like, don't overcomplicate things. Start simple. Be pragmatic by putting a process in place. And a process needs to be very simpler because I think when it comes to people, process, technology, technology is the easiest piece. Process can be moderately okay. People are the most challenging piece. Humans, we humans are really nasty when it comes to that. And because, and so I always take a notion of try to keep it simple, simple process, something that people can understand relatively easy, and then slow on, do something very simple, like maybe metadata level agreement, and then expand it to your aspect of what you're saying, like information. And even in information, it could be one specific metrics that you're really looking at signing up. And then third, it could be cost. And then fourth, it could be sharing. So if you have a plan like that, then you feel good about some of these options. If you try to do everything all at once, it's a never-ending project, or you will not be able to measure the milestone of success. That's what I say. Yeah.

And again, you're going to boil the ocean. I actually have a different view on first-party data versus third-party data. And my view is that data we acquire outside our organization, that third-party data, often is of better quality because we're typically paying for it. So the provider who's sending it to us actually cares about the quality because we don't pay them if it starts being crap. Our first-party data is typically managed by our people and our people get paid regardless. So the quality is not that important often, but they don't get fired for typing the wrong data in the wrong box. The classic one is you have a system where you have to type in date of birth and the system allows you to put in 1900, even though we know there's nobody alive at that date anymore. And therefore, when people are busy, when they're trying to get through those stupid screens and they've got the customer on the phone, they just whack in 1900 because it gets them to the end of their goal really quickly. And they don't get fired for it. Yes, we might find it and then somebody will go ape about it later, but now we've got a rich respectively go in. And that person who let that field be entered that way and the person who entered the data, there's no consequences to them apart from maybe a telling off. So again, I find that first-party data often is of less quality than stuff we buy. Then we come back to that idea of boiling the ocean, which again, you just talked about really well. And I take it to the next level of signal versus noise. Scenario, we start off, we've got 10 systems coming in, we bring in some observability capability, we build some profiling rules, we build some tests, some quality tests. We do a little bit of cost modeling to make sure we know when things are going to go a bit weird. We do a little bit of anomaly detection, maybe a little regression on our load stats to go, hey, these tables look like they're outside the norm. We bring in some freshness monitoring. We go, oh yeah, well, we'll just tag each of the tables with the day they, how often they should be updated and we'll monitor that. Bring in some data loss protection, automated privacy stuff that flags data as high risk, low risk, medium risk. Been a name. Okay, so now what we've got is we've got 30, 40, 50 things alerting me. Ah, and it's doing an additional thing. I've got 100 high-quality, high-risk tables that we've said, look, that's actually what goes into our board reports. So those 100 tables are the key tables we care. I get up in the morning, I go and look at my little dashboard, and I've been notified on 36,000 alerts. What do you do? How do you deal with that signal versus noise?

You hit the point on alert fatigue. I think that's where I said don't overcomplicate things. You don't need to observe everything as a start. And then as you observe, or as you measure quality, or as you start looking into privacy compliance and things like that, I think even if you can build a percentage of, or in application or some kind of specific context within your bigger ecosystem, And then build a confluence that's much more easier to observe and then train the people to look and make it more sharp. So I would say like it's a more of a strategy, how you deploy it. It's not the limitations of the tools and technologies. It's just like one, the people have to buy in what is being shown in the front of their dashboards. And if that is not scaling even within one application context, screw it. Why do you use it? Right? Like, yeah, I mean, like you're going to be spending hours and hours standing. Or most of the time what happens is they buy all these toolsets, they don't use it, it's just sitting there. This is very common in banks. If you go to a bigger bank, they pretty much have all different types of tools because different lobbies, they want different products, different things. And then you pretty much have 10 different catalogs, 10 different quality tools, each one doing different things. Then the value is always questioned at the end of the day because They don't necessarily start off with a focused approach. So that's what I would say. Like, I don't even want to start with 30 metrics. Start with 5 metrics. If you can measure 5 metrics in an application in a meaningful way and build confidence, now you can replicate that same mechanism across multiple applications and across the whole organization. Now you go and add another one. So that would be the kind of the step in the process that I would follow. Of, of sort.

I think it's not a solved problem yet. If we think about the end-to-end lifecycle, it's always interesting for me that if we have a dashboard or a machine learning model recommendation in Salesforce, it's almost virtually impossible for the consumer of that piece of information to understand the quality of the data that came from, because the quality information, the observatory information sits in another system. They have to go and look somewhere else. They have to go and log in or they have to look somewhere else. We don't embed it right next to it. And as you talked about that organization that had level 1, level 2, level I remember, oh, it must be 20 years ago, we were doing a project and we actually classified the end-user reports in 3 colors. We had bronze, silver, gold, and that was all based around the process we used. So the gold reports were the ones that had gone through, this was back in waterfall days, our extremely long, extremely painful 6 to 12 months development lifecycle. But we inferred that because it went through that process and went through so many hands, the data in that report report was of higher quality. The silver was, it went through some kind of ad hoc process, but there was a peer review. There was a community review by subject matter experts. And then the bronze one was just an analyst hacked some stuff together and pumped it out. And so the end consumer could look at it and have a feeling of quality, but it was all based around the process. It wasn't around observability. It wasn't around the data. I think it's just start off with something and say, how do we actually give confidence to the person consuming that information or that data? Around it, around that observability. And coming back to those key themes that we talked about. So we talked about the quality of the data, or can we trust it? We talked, and observing that, we talked about observing the cost of that data in the platform and what we've done. We talked a little bit about detecting anomalies, how observability can tell us that something looks weird. We talked about freshness, has it been refreshed as I would expect or as I need? And then we talked about privacy. Is the data meeting our privacy things? So again, got a couple extra ones from when we talked at the beginning, so 50 minutes later talking at the end. From your point of view, anything we've missed? Any patterns that you go, oh, that's the one, that's my goal?

Yeah, I think there are certain aspects of observability which it does not cover, right? I mean, example, when you're talking about different applications, you can have a little bit around it, like foreign keys and joins, and then checking the data integrity. I mean, observability cannot do it. You can do the count and you can do an anomaly and you can do some manual thresholds and things like that. But I think it gets into, you need to look into the data in itself. Okay. Like, yeah, I see something is happening, but then I have to go look into the data. And now it's kind of like you have to look across multiple applications and data and then try to do more of a business process checks, which kind of gets into this process things. So I think there was some more developments that could happen. Definitely as the industry matures, I think these are things like root cause analysis and trying to look into observability from a process standpoint using data. There are some things that could happen too. So these are areas where we are looking into it and I'm trying to say like, okay, we can do something at the metadata level. We can do something towards reduce that noise. So the signals are more valid. And then we can also provide some guidance to the businesses in terms of start slow, crawl, walk, run versus try to do everything. Some limited SLAs on a higher level from a data contract standpoint, and empower this federated governance or decentralized ownership from a data mesh principle standpoint. But still, I think I'm super excited because like the cloud hasn't fully hasn't really yet seen its full peak. We are just in the start of it. We are already seeing ChatGPT evolution in terms of AI. We are still struggling to govern and data management. That is a huge— as the data grows, your management needs to be even more sharper and more agile, which means innovation is waiting and more to happen. So I think it's kind of like fascinating to be in this time of the age to just to witness all of this stuff. So more to come, I guess.

Yeah, I think the key for observability right now is there are a large number of patterns that have been shared. These patterns that we've had for 20 years, and there's a whole lot of new patterns that technology and process and people have enabled. So if you're starting your journey, don't worry about the technology yet, right? Worry about the people and the process. Find some patterns you think may have value. Try them out in your organization with your context and say, are they valuable? Are they worth the cost of implementing from a technology point of view? Once you start getting to a scaling problem, the machine needs to do the work. And once you start getting too many of those things, you've gotta deal with the signal versus the noise. And that's the next problem to solve. First thing, just make sure you check for nulls. It's a simple one.

Oh yeah. Yeah. Check for nulls.

Check for nulls.

If the data is not there, why bother, right? You have other problems to worry, like you need to call the engineer or you need to call the pipeline or look into the pipelines.

Nulls always bite you in the bum. I'd love somebody to do an actual research survey using data from from lots of customers around what are the number one observability tests.

Like, when you go into those details, sometimes it's very funny. Engineers, like, as much as they are super smart, sometimes it's also like, it's not that they cannot, it's just the way it is being constructed. So sometimes you may have SSNs or some security numbers which is in a varchar, like, is text, right? Just because it has hyphens or they think it make it hyphens and et cetera. I mean, so I have seen in an organization the same field SSM, sometimes in numeric without hyphens, sometimes in varchar text with hyphens. And then, and sometimes it even has spaces because some application did not trim the spaces and just puts it for whatever reason. And now you have trailing and leading spaces across the data. And so now you're kind of having all these data quality issues. So yeah, nulls, blanks, some basic checks will go a long way, is what I can offer you. Yeah.

And I think it comes back to, you know, at the moment, observability is the first sieve, right? It's the machine telling you to look over there. And then, you know, the second sieve is still a human, right? We haven't automated that yet. Human still has to go and look at it and, you know, observe what's happening at the next level and go, okay, I can see the problem. I know how to fix it. Yeah. Maybe ChatGPT will give us that.

Yeah.

And I think it will.

You can ask one day the question to ChatGPT and say, like, is my data reliable and accurate? And then, like, ChatGPT will give you the answer for you on that.

I think once you ask it for the bio of somebody and it actually doesn't lie to you, it actually, what do they call it? It basically makes stuff up. And there's lots of cases that are coming out now where people go, tell me my bio. And it's like, you went to Harvard. and they're like, no, it doesn't. So once they solve that problem, right, then maybe we can trust ChatGPT to observe our data for us and tell us what's going wrong. But exciting time. We've got some cool technologies that hopefully people in process will catch up to, but people in process first, right?

Yeah, that's funny. I just went and just typed this metadata label and I killed it in ChatGPT.

Ah, see, you know, you used to be, yeah, how to tell you're a geek used to be the t-shirt you wore or the TV show you watched. Now how to tell you're a geek is that your other browser window that's permanently open is ChatGPT.

At least it was honest. It said like, as an AI language model, I don't have access to your data, so I cannot evaluate.

Yeah, well, that's the other vanity metric, isn't it? Is if ChatGPT can actually do a bio for you, even though it's wrong, it still means you're more famous than most of us. So excellent. Hey, look, Yeah. That's been great. I think we've gone into a little bit more into observability into areas that I hadn't thought about. I come back to the idea of an information contract. You and I came up with that here. I've still got to ChatGPT it in a minute to make sure it's not there anyway. So if people wanted to get hold of you, they wanted to find you without asking ChatGPT, what's the best way for people to get in touch with you and what you do?

Email raj@dqlabs.ai, or you can just reach out on LinkedIn. It's again Raj. Joseph, R-A-J, J-O-S-E-P-H, Raj Joseph. That's my profile link, LinkedIn. So one or the other works. I also have my phone number in the LinkedIn profile, so you can text me, whatnot, and etc. Don't spam me.

You gotta get those automated Visa credit card calls in about, yeah, yeah, in about 10 seconds.

Yeah, but yeah, definitely I think LinkedIn and email is the best medium to reach out, and then after that Definitely. I'm happy to always talk because there is so much innovation and so much ideas and things that comes to the surface every moment and every moment. So happy to entertain.

Yeah, that's great. Hey, look, thanks for the time. Thanks for sharing some patterns and thanks for chatting about all things observability.

Will do.

We'll catch you all later.

Thank you so much, Shane. Thanks for having me. I appreciate it. Bye.

And that, data magicians, was another Agile Data Podcast. If you'd like to learn more To learn more on applying an agile way of working to your data and analytics, head over to agiledata.io.