AgileData #38 - Observability - Raj Joseph

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hey, thanks for having me, Shane. I'm Raj Joseph, CEO deqlabs.

Hey, Raj, thanks for coming on the show. We've got an exciting one today. Wow, they're always exciting, but this one's another exciting one for me. So we're going to talk about observability and all the other abilities and data words that we can bundle into that one, and what patterns are useful and what patterns aren't. Before we do that, why don't you just give the audience a bit of a background about yourself and your history of working this magical thing called data.

Sure. So my background started off from data engineering background. So started as an engineering back in 98 with an engineering degree right off the college, and then went through lots of startups and bigger fintech companies and slowly moved into product leadership for a data marketing company. So there I built a big data platform with a focus towards quality. And one of the challenges that we did was trying to consolidate data from different sources and trying to build a consumer profile. And of course, lots of duplicates, lots of bad data. So my life in and out for a long while has been in on data quality. And then later on I also started a services company in 2010 focusing on data, data management and governance. And then in 2020 came into an opportunity to build a modern data quality platform with a focus of bringing in observability and business quality together. And so that's kind of where it ended up too. So that's kind of like a brief profile of me. Yeah. Good.

And that's one of the reasons I wanted to have you on the show, is because you've been there, done that, you know, a bit like me, where I've been a practitioner of delivering data to customers, finding all the horrible things that are nightmares for us, and then how do we create products or services and then fix that? Your concept around bad data reminds me of one of my favorite T shirts is Breaking Bad Data. T shirt Trifecta created it. I'm tempted to make a new version because the one I've got, I wear so often it's got holes. Maybe that's what we'll theme this whole podcast around. Right. Breaking bad data. So to start off, you know, observability data. Observability is a new word coming out, the patterns underlining it. Being around for ages, there's always a risk with new terminology, it's just buzzwords rather than things that have got meat and potatoes behind it. So for you, data observability, how would you describe that pattern if somebody asked you for it?

So for me, I think the observability is a capability because it's really when you look into what you are observing at the surface, shallower, more broader data, but more faster processing. So when you're having a bigger growth in data, it's much more easy capability and a technology that could be adapted for multiple different reasons. There is software observability, as always we know about. And now with the data focus, we have this concept of data observability. Even within data observability. For me, I try to see that observability concept can be used in multiple ways. And for me it always goes back to the quality. And how can we observe a data to figure out some aspects of quality? Because quality is a bigger, broader way border name and category in itself. And for me, observability gives some kind of measure around reliability. So the word that I can perceive and conceive when I hear data observability is okay. Absorbability is a capability that I can use to measure the reliability of data delivery. So that's one aspect that immediately comes to it. But there are other dimensions of things that we come in this day and age with modern data cloud architectures and things like that, where architectures are spread across hybrid and also modern cloud warehouses, where cost is a factor. So you can also use observability as a capability for cost. And then also if you want to measure some of this business quality in a way, you can also measure if it is, there is any anomalies that you can check in, et cetera. So but more around reliability is where I kind of tend to focus.

Yeah. And so for me, I agree it's data observability is a bucket. It's an overarching term, a bit like a cloud analytics warehouse. So it's an overarching term that has multiple parts to it, multiple patterns. Each of those patterns have value. And we kind of saying there's a new category, there's a new group of patterns that we want to put together. One of the ones I want to pick up on that you talked about is this idea of scale. So if we think about if our data platforms, our data systems, the data we work with is tiny and not tiny number of rows, but tiny number of columns or data sources or users. So we're at a really small scale, a Human can observe, right? We go and grab one table from one system. I can observe that if I'm only running it once a day, I can go in and observe it happen. If I've only got one user, I can observe what they're doing with them to see where the problem is. So that's all good, right? But as we scale, as we add 100 different systems, as we add 1,000 users, as we add 3,000 tables with 60,000 columns, we bring in, we can't observe anymore. As humans we need the machine to help us. And for me that's what observability is about. Is the machine doing as much of the work as it can. Because it ain't even with chat GPT, it ain't fully magical. I don't have a magical observability hat where I just dump all my data in and I go to sleep and don't worry and it tells me, but it gives me hints. I like that way that you broke it down. We can look at observability patterns for quality, we can look up observability T patterns for cost, we can look for reliability and we can look for anomalies. They're really good. Four areas that we can do. So what's your favorite one? Which one do you want to do first?

I think since my background has been and most of my life has been in data quality, I would just focus on quality as a place where to start. And it's a broader category in itself because if I ask you Shane, what is data quality, you may come up with a definition than what I may come up with. And most of the time what I have understood in the industry is it's really, really in the eye of the beholder, meaning the roles and the responsibilities defines what they're looking for. When you're talking to a data scientist and ask them data quality, they are more talking from a fit for purpose, meaning when they're building analytical models, their needs of that particular model may be very different than what they might have done a day or a month back. So their needs are very changing and is based on the business strategies and the tactical things that they do in terms of self solving those strategies. While if I talk to a data engineer theirs is a little different. Maybe they are more looking from reliability of the data in terms of making sure that the data reaches from one source to another target and then also their pipelines are processing asset and then try to prevent as much as they could before it lands. So that's where I would start Observability with a focus towards quality. And then I'd say it's a brick and product topic itself based on the roles and responsibilities of the user.

I agree. I think if you take a Persona view of it, you get different answers. So if you say to a business consumer, what do you care about quality of data? They go, I need to trust it. If you tell me I've got 100,000 customers, I just expect that number to be right. Why would it not be right? It's really interesting in the data world, right. We have this habit of just giving data to our consumers and getting them to find the problems. It's a car with only three wheels, and we go, ah, yeah. Does the other wheel really. The fourth wheel really necessary? And then the consumer hops in the car, it falls over, scrapes on the ground, and we're like, oh, yeah, we can fix that. So for me, that's what they say. Whereas, you know, you're right, if you're an engineer or you're in the plumbing, then you care about some other things. One thing I find really bit of a struggle for me is if you look at Dharma and data management and data quality that's been around for decades, there's always the abilities. There's always six abilities for data quality. Buckets or categories you can look at. And for me, I've always struggled with a pattern or a rule and where it fits. So the one I had the other day is, let's say that we have a table and the table's got a foreign key on it. Yeah. So each of the values in that column are meant to be unique. There's only ever one. And we bring that data in and it fails that test for whatever reason. Is that a uniqueness test that failed? Is that a duplication test? Those categorizations that we often hear about, each of the rules that we could use could go into any of them. What have you found? Like when you say if you had to categorize the quality as being high or low for a certain thing, do you use the standard six from Dharma, or do you think about it differently?

I know the six quality dimensions you're talking about. So I was at some point, as a practitioner and then as a preacher of data quality and all those principles, I was madly following all these dimensions without understanding why and what and etc. To a specific level. But what I really more and more understood is it's more than the measurement. The purpose actually defines the measurement. So measurement without a purpose is messed up, right? So the six categories are very useful when you to get into some kind of defining the purpose or if you don't have a structure, if you don't have a framework of how to measure that kind of gives it. But what I have seen from working with different clients and etc is each one has a different level of expectation and governance process that they kind of manage. So for example, I'll give you one example. This is a bank in Canada. So this particular bank they don't look into quality the same across all the data. So first what they do is they define data in three levels. Level one, level two. Level three. Level one is good. I have the data. It's great that if it is there it's good. Level two is something that is used for reporting and some kind of data science model. Level three is it is used for KPIs business metrics because the leadership is going to be looking into it. Could be example could be is revenue metrics across products while your level two could be is a product name and level oneness could be like some flags that may be sitting there, ones and zeros of sorts. Right. Some bits that needs to be turned on and off, an application, a feature sample. So for them when they see level one, do I care about quality? No. Do I care about reliability make meaning the data is there or not? Yeah. For some reason if there is a big data gap or data leakage, yeah, I want to look into it. Maybe a system is down, maybe something has happened. So the level of detail of how I need to measure and the quality is based on the criticality of the data derived from the business use of that. So. So that's how they do it. So level one, level two, three and then so as you go up your six dimensions of data is much more higher needed and it can be even beyond that too. It could be even like a subjective. These six dimensions are more quantitative and if you want to even go subjective, meaning like yes, the data is there and the quality of the six dimensions are saying great, but nobody using it then what? Why care? It's kind of like that.

Yeah, it's the boil of the ocean problem. It's the same problem we have with data catalogs where we bring every SAP table in and column and now we got 60,000 columns. Nobody's going to have a catalog all that and they shouldn't. We catalog what we care about. So I think that's good. The way I think about it is blast radius. If we have a problem with the data, whether it's Quality, cost, there's anomaly or it's not reliable, what's the blast radius? What's the impact to the organization with that data being wrong, not being trustworthy? And then we tailor our patterns because of it. And that's why often for me, we can figure out a small set of things we want to focus to begin with because we understand what data is really, really important. It's normally the data we touch first, right? Not the first bit of data we bring in, but the first data we expose to a consumer because you've asked for it. And then the second thing is often we have a problem and then we have to go and observe and manually figure out what the hell went on. And then we have to go and fix something again. That's when we need to bake in some observability. We have to say, okay, well let's just make sure that if that happens again, we identify it and ideally we go fix it so it can never happen again. But we definitely want to be able to observe it if it does and be notified.

Exactly.

So again, start small fake in the simple ones. Go figure out what fields shouldn't be null. Real simple one, do that first.

Exactly. And those kind of checks is what I call as wide and shallow validations. Like null checks, blanks empty is uniqueness or even the number of unique or data distribution. In terms of statistical evaluation that you do, you're kind of like checking more in a wider, faster, bigger aspect of amounts of data and more shallow validations. You are still on the more at the surface level, you are not going deeper, you are not having any narrow validation like what you may do in a business checks. And I think that's okay at that level One zone we are talking boiling the ocean. But as you go further down, that's where we start defining what is critical. And maybe when it comes to business quality checks and things like. I think I always tended to say to your point, impact or looking at a business impact. The first thing is what is a business impact? And it's not an easy answer to solve. But the way I kind of create is like okay, business impact, meaning what are the KPIs? So that's KPIs can be easily translated to the actual data that is being provided to us to make the KPI metrics. And so I kind of look into the metric stores or the business metric stores and from there I kind of connect to the tables and the columns and then kind of drill down into it. Okay, these set of data are much more Business impacting data versus all these other non business impacting data.

Yeah, and for me, usage of the information is something we should be monitoring, we should be measuring, we should be logging, we should be measuring, we should be monitoring how often a certain piece of data is used and who's it used by. And that helps us with our blast radius. So we can say, well actually if the data here becomes untrustworthy, we already know who's using it. Right. We can communicate with them or we can understand. Is that affecting one user? A thousand users, can't quite tell. Is that affecting one piece of data that that one user happens to do? Man, that drives the whole profit of the business versus the thousand users that are just in the factory part of the process or not? You often don't see that usage logging as part of observability. That tends to sit into another category. So for me it needs to be observability. If we're saying we need to be able to observe things that are important at scale without a human doing it manually, that's our definition, then that's in there. The other one that's really interesting for me is, and you talked about it a little bit, there is around cost. So in the old days when we had on prem databases, we had an Oracle or SQL Server, we could run a hell of a lot of observability tests if we wanted to because there was no additional cost of hardware, there was just a cost to the consumer. Right. Because we take all the CPUs, their reports would run slow or our ETL would run slow. In the new cloud world, when we add this new observability capability, this new workload, there is a cost. We start paying for more credits if you're using one that's credit based or more compute or more cpu, more slots or depending on which technology you're using, but you actually have a larger overhead on that platform. And what we've seen is actually running our observability stuff actually costs us more than running our transformation stuff. Unless you really think about the pattern. And I'll give you an example. We might say we've got this table. It's got some categorical variables, it's got some regions, and under each region is a bunch of cities. We know that there's a mapping, we know that in the source system somebody's going to go change that mapping, they're going to go remap a city or a new city turns up or something's going to happen. And we want to capture that from an Observability point of view, because we know that for some reason that piece of data, that mapping is used for our financial forecast, that's going out to our shareholders. And so it's a high risk piece of data for, for us a lazy architecture basically says scan those tables for distincts, but what you're doing is you're scanning that entire table every and that's an expensive compute. So you need a pattern where actually what you want to do is rake out that unique set of values into their own small tables and every time a new piece of data turns up, compare that piece of data to your reference table and go have I seen this one before? And now you're only processing new rows, you're reducing your cost of. So you have to bake that in. Do you find that? Do you find that actually observability, quality, anomaly detection, reliability, cost monitoring, usage, logging, they're all high expensive compute in the new cloud world.

Yeah, you're right. Because if you see the adoption of cloud is growing and it will grow, no question about that. But what I have seen, it's very interesting is the adoption of cloud is also a little bit slowed at some point because of the cost factor. We have folks using all these modern cloud warehouses and cloud lake houses and things like that and then they move the data and then every time now they query there is a cost, you want to be processed even faster than you need to pay more the size of the warehouses or the size of the compute power that goes around it, which makes sense to a degree. And then to your point, you cannot be observing the whole ocean because you are actually computing more. And then therefore the cost is more. And so in some business transactions we have come across, if your spend for cloud warehouses or lake houses is 1 million or $2 million, your observability spend needs to be a fraction of that. It can be like 300k or 500k or massive, which questions the value of that itself. And the way to do that is rather than spinning up and going across all observability metrics going into the table level, you kind of used some of the logging mechanisms of that. For example, if you take Snowflake, Snowflake provides your metadata log on in terms of volume freshnesses. I mean almost all this modern data warehouses provides time travel mechanisms so you can see the snapshots of time and when it got updated. So you can easily retrieve some of those information without doing any kind of complex query logic. So to that, but the same thing, when you have some Other legacy databases or even on premises, then it does not scale to that level, which is okay because it's not cost compute based. But however, the point I'm trying to make here is when you start with an observability you don't necessarily don't do all, just maybe stick with a few dimensions of data. So we always recommend volume, which is a row count. And then we look into the schema which is more the column counts. And then we look into freshness, which is kind of like the timeliness or the data when it got updated. And then if you want another fourth metric, we also look into duplicates. The duplicates could be based on primary keys or it could be a composite key on a particular table or an asset. So we just kind of stick to that as every asset. So you are not doing too much deeper checks, but at the same time kind of some kind of higher level check and statistics that kind of gives you some information. So that's one. And then the second aspect of that what we do is we look into usage information, who's using this table and what roles is the system which is kind of spending for no reason or is it an actual user from a consumption standpoint at least that kind of tells you how relevant or how impactful in some ways. Just because using doesn't also mean it's business impactful because somebody may be running a query for no reason. So but at least it gives user another dimension of data to look and then kind of combine and then go into a methodological process around defining some criticalities using this KPIs and other things and then go around that.

I agree. I think the technology choices you make a part of your stack actually drive how you manage the observability cost. An example is because we're all based on BigQuery, we have a choice between slot based and size of table, size of query. Effectively we're able to profile the data within the same query at relatively low cost. What we do is we profile the data and then store the data when we process it and store the profiling information next to it. If we were using something like Snowflake, where you're actually paying for the size of that compute, then actually you want to be a little bit careful because every time you profile everything you're going to do it. Whereas if you're using something like a lakehouse where you're using say iceberg as storage, then you probably want to profile the data as it comes through again, but this time you're storing it as Another object, you're storing it next to the data. So again, those patterns are slightly different. I think one thing that really interests me, if we think about cloud databases, they inherited the on prem database patterns. They typically have an information schema. We can go and say what's the name of the column? What's the column type? Is it a numeric, a character? What's interesting is they don't do that with profiling data, right? They don't store this column's got nulls, they don't store the standard deviation of that column or the maximums. And so it's going to be really interesting to see over time whether they do, whether they actually capture that, store it and surface it up for free without a cost of compute. Because actually as data users that's what we want. They know what the data looks like because they're storing the bloody stuff. So just give it back to us and don't charge us.

You already see, I think all these modern warehouses and lake houses has already going into a path of introducing some observability metrics as part of their stack when it's inevitable. I think they have to do it and because I think that will make, I mean more value add one as a platform, as an offering from it. And also it get a little bit touches into the data management aspect of IT because it's another big area of revenue but also more adds value as a platform offering. So I would see more and more these vendors trying to build observability metrics into their platform default. The challenge where a business has is, I mean it would be great if a organization has only one technology, but that's never the case. So they use a combination of technologies. I mean I've seen like people using both Snowflake and Databricks. They compete with each other and then also you see for different reasons. And so, so I think that's kind of the reality of it. Even though these vendors, some vendors may incorporate and still others wouldn't and then your ecosystem has multiple technologies and then how do you measure that as the data moves from one place to another place and then also give a bigger a whole vision of where the real data stack is from a health standpoint and from a reliability standpoint. I think that's critical.

Yeah, and I think that's a really good point you raised there is around about the different technology stacks, but also about the end to end process. So again we're still at a stage where a lot of the logic still gets landed on the last mile gets in the tableau or the Power BI or the looker Studio reports or their semantic layer. And so we still have the ability for the quality or the reliability of the data to be built broken at that last mile. I liken it to a kitchen. We measure the quality of the goods coming into our storeroom, the good flour. We measure the quality of the process we use to bake it in the kitchen. We measure the quality of how we serve it up to that little window. But we don't measure the quality of the waiting person who picks it up and then spits in the food before they deliver it to you. A lot of the observability capabilities are just up to that serving layer, right? But they don't actually then worry about what's happening in that last mile and so therefore we can degrade the quality. So are you seeing that or are you seeing that market starting to move, that people are starting to worry about what happens to that last mile of delivery before it gets to the consumer?

I think the whole still there is a lot of innovation that is still yet to happen in the data world. Even the adoption of cloud is still in the initial early stages of. We haven't even seen the full revolution of everybody in cloud or everybody has adopted to it. We see cost as a factor that this needs to be optimized and contained. The governance around cloud in terms of privacy classifications, the end to end visibility is still a struggle because all businesses today have some in on premises, some in legacy technologies, some in modern technologies and that's the reality of it. And in order to get this end to end visibility, it's not an easy problem to solve and it's very challenging too. And then also like each vendor and each technology gives some part of it not necessarily the same in terms of a metadata. So you get some time travel kind of metadata logs from some of these modern stacks. But if you go to the traditional, you have to pass through lots and lots of logs to get to that point of data, right? So I think there is a variation of that. All of this makes it more and more complex. But I think rather than trying to get all of those visibilities, at least if you can define some metrics in each one of those stop points or pit stops, then I think that would help to get closer to that. In reality, I don't think any company can get 100% visibility on this end to end, but the more they can get will help them to be more smarter and more faster. And in terms of their data driven.

Strategy I think that's a good point. People often underestimate that every time they add a new technology or a new moving part to their data stack, they're actually increasing complexity and therefore they're reducing observability. Unless they can apply those observability patterns to that new component, which they often don't. Because then it's a last mile one. It's hard. But what's interesting there is you brought in a new one. You talked about privacy and classification of data.

Oh yeah.

And again, that's a form of observability. Am I holding credit card numbers? Am I holding licenses? Am I holding people? If we think about observability as a way of getting told by the machine, something happened at scale because I can't go look at it. I'm not going to go scan every column visually and go, that looks like a name. I want the machine to do it. So we've got to add privacy and data protection, observing what's happening in that space into our observability bucket. If we're saying we want observability of the whole system, don't we?

I think you're 100% right. Because when you look observability, it's more this blanket catch all term. And then you add data still, it's very generic, very broader, but the more context you put on top of it, and the context could be either your privacy, it's very focused on privacy, or it's focused on cost, or it's focused on usage analytics, or it's focused on health, or it's focused on reliability in terms of as the data moves from pipeline, or it's focused on business. Now you're having lots and difference of branches that goes from that and getting in its own division. So there is a privacy regulation that itself is a bigger, broader industry and you can use observability in some kind of capability. So the application and the impact is more. But I think where it will become more and more meaningful is taking one subject like quality, data quality, or a business quality, and then trying to see how you can use leverage observability as a capability within the constraints of what you are trying to do from different Personas, from a leader, from a business consumption standpoint, data scientist, data analyst, and then also from a data engineering standpoint, looking at all those three different roles and how they see the data quality and then kind of mapping that observability would be very critical. And that's the same thing for privacy and classification too.

Coming back to the idea of Persona, as a Persona, difference between consumer and, say, an engineer, I want to observe some different things. I need different outputs to tell me whether I can trust it. Can I trust that my dad ran versus can I trust the number you're giving me that tells me how many of those we have? And when we come to that, we start talking about language. If we think about data freshness, is the data up to date From a technology point of view, we've always talked about SLAs and SLOs, service level objectives and service level agreements. And you talk to a business user, somebody that's not in the technology domain, about those terms, and they just look at you blankly and they sell what? But if you talk about freshness, is the data fresh or is it stale? They can only get that. I always go back to baking and cooking. For some reason. My bread's fresh, my bread's stale, one's good, one's bad. And so they have expectations about how fresh it should be. It's about use. Some data doesn't have to be refreshed or fresh every day. Once a week's okay, some data has to be fresh today. And so if we think about that terminology, that language we use, we've seen a new term come out last year and this year, and it's all hot and maybe buzzwashing, maybe not. And so it's data contracts.

Bingo. It was either going to be data mesh or data contracts, right? So data contracts. And I look at them and I go, so whenever I hear a new term come out, I look at it and I go, what's the underlying patterns? What are the patterns that are available? What's the context where I can apply that pattern? What action or outcome does that pattern give me? And so if I think about data contracts, if I think about something simple like schema mutation, I have a table coming from a source system. It has 10 columns. My contract is for 10 columns. An 11th column turns out that contract has been broken. And so a test or a pattern for observability of the number of columns that we are expected has changed is valuable. Again, if I've only got one table coming in, I can go and eyeball it. If I've got a thousand tables coming in, I can't. So the machine needs to observe that for me. But that's been around for ages. That's not new. Schema mutation. Then we go on to some other patterns around it. They go also, context of the data. Has the data types changed? Has the volume of the data changed? Has the relationship between two columns changed? So we start getting quality tests in my head and then there's that nirvana one which is actually I actually have a contract, I have a policy that's written in code and that both sides are complying with it. It's actually driving everything we do. And I haven't seen anybody actually do that yet. From your point of view, data contracts, first question, buzz washing or not. And then second question, where's the value? What are the patterns?

Every innovation or every new principle that has come has been a buzz at some point of time. And and I think if you look the hype cycle, it just goes up and then it just goes through this maturity life cycle and then just kind of comes like okay, this is hype and this is actually some value to it. So I think there is a lot of value that we can get in terms of data contract. I mean if you really look what's happening is software observability is now turning into data observability. Service level agreements is becoming data contracts, data level agreements and the concept of centralized governance is going more towards decentralized from a data mesh principle standpoint. Also this self serve data platform as a concept is making how do you share the knowledge at the level where the SMEs are versus trying to consolidate and have business overtake and share that because it would say too much responsibility to be asked from a business to own when data were not even the ones who are producing the data. So all of this in a theory and in the way it is constructed makes a lot of sense. But in terms of. So it's. I don't think it's a buzzword but actually where it really as a organization or a user of the platform of these concepts struggle is how do we implement it right? What is right? And because these are all like newer concepts, even observability is still new and emerging because it hasn't reached its maturity in terms of how and well it can be used because you have here alert fatigues. If you turn on observe across your whole ecosystem it's going to just look for anomalies everywhere. Some may be meaningful, some may not be. And as far as I know when you get like thousand emails most of the time you're not going to be looking into that.

So there is a maturity component that sits with that and same thing with the data contracts. I feel there is a definitely a value because now if you look into data quality or data observability or any of those things, both the business and the engineering, which is data engineers and data scientists or even the business leaders, they need to collaborate together and they have to see it in some kind of common way. And I see data contracts as that. It just kind of gives them to meaningfully commit to both from an engineering standpoint and also from a business. Telling like this is my expectation. Otherwise what happens is the life of data engineering has to suck because they think today it's great data and then tomorrow you will hear from the data scientist. This sucks because they need changes. So that kind of allows them to bring and bridge the gap and promote more collaboration and be more productive and kind of also tie back to the business impact you started on early and drive towards the business outcome. But the bigger, broader question is what are the categories of data contracts? Again, the implications are more. Because it could be privacy, right? Because I can have a ssm, so that means it needs to be masked, it needs to be governable. And then two, it also from a cost standpoint, it could be. Or it could be from a reliability and accuracy standpoint, from a data quality Stu.

Yeah, that's a good point actually is that the contract can include many things and it should. And then you have to decide what you're going to do about it. So you could have a contract with a data provider that they won't send you names because you're storing them in a different way. And so you need to monitor that contract's not broken. And then you need to decide what to do if it is broken. One of the things I love doing this podcast for is I talk to people and then somebody just goes bang in my head and that just happened. We've talked about data products for a year or two and I've worked For the last 10 years around this idea of an information product. And the way I describe it is we sometimes produce data depending on the Persona of the consumer. Data scientist typically wants data, not information. But our business consumers typically want information. They don't want a dump of data, right? They want to get an answer. And so if we think about data contracts and we think about a data value chain or data value stream, where data is produced on the left hand side by a source system, a machine or a human. We do some stuff to make it useful and we give it to the person that actually wants to consume that data or that information. The data contracts have all been focused on the left hand side, right? We focused on our problem. We haven't actually focused on an information contract. We haven't focused on our consumer about what Good. Looks like for them there's a bunch of tests but there's no contract. And then if we bring that lens of cost into it, and that's just fricking awesome. Because now what we can say is look, here's the contract, here's the base contract. Data won't have any nulls and it will turn up once a week. If you want to increase the level, if you want us to do more testing around it, you want us to get it to you faster, then the cost of that contract is going to go up. So there you go, I reckon 20, 23, we just need to trademark this. Information contracts as important as data contracts because we should actually have a contract with our customer, our consumer, because the things we do to make that data better quality, to make it cheaper to produce, to identify the anomalies, to make sure it meets the freshness requirements, to ensure that it's, it's retaining or complying with the privacy rules that we have. Each one of those has a cost in both human terms for us to build it and implement it and monitor it, and the system costs in terms of executing it and the observability engine. So yeah, for me that's information contracts. There we go. Heard it here first.

Information contract could also have sharing agreements, right in terms of who access this data and things like, like that from a usage and accessibility standpoint. So like because of so much choices in today's technology and I mean if an, I mean if an organization is starting today as a new business, it's easy for them to adopt to some of these technologies faster. But as they grow, I think I can call this. When you're small and you have a data which is created by your own applications, you have a higher degree of quality. But as you grow, your data is going to be coming from outside partners, suppliers and systems that you don't necessarily manage. And now you have what I call as inorganic data. But it's very essential for your reporting, analytics and whatever you are doing in terms of your business outcomes. And I think that's where all this challenges comes in. Because in an area environment, your ecosystem has so much different technology, so much different nuances to it. Data coming internally, data coming from different stakeholders, which you don't necessarily contain. So I always take an aspect of like don't over complicate things, start simple, be pragmatic by putting a process in place. And a process needs to be very simpler because I think when it comes to people process technology, technology is the easiest piece. Process can be moderately okay, People are the most challenging piece humans, we humans are really nasty when it comes to that because. And so I always take a notion of try to keep it simple, simple process something that people can understand relatively easy and then slow on do something very simple like maybe metadata level agreement and then expand it to your aspect of what you're saying like information. And even in information it could be one specific metrics that you're really looking at signing up. And then third it could be cost and then fourth it could be sharing. So if you have a plan like that, then you feel good about some of these are options. If you try to do everything all at once, it's a never ending project or you will not be able to measure the milestone of success is what I say.

Yeah, and again you're going to boil the ocean. I actually have a different view on first party data versus third party data. And my view is that data we acquire outside our organization, that third party data often is of better quality because we're typically paying for it. So the provider who's sending it to us actually cares about the quality because we won't pay them if it starts being crap. Our first party data is typically managed by our people and our people get paid regardless. So the quality is not that important often. But they don't get fired for typing the wrong data in the wrong box. The classic one is you have a system where you have to type in date of birth and the system allows you to put in 1900 even though we know there's nobody alive at that date anymore. And therefore when people are busy, when they're trying to get through those stupid screens and they've got the customer on the phone, they just whack in 1900 because it gets them to the end of their goal really quickly and they don't get fired for it. Yes, we might find it and then somebody will go ape about it later, but now we've got a rich response and that person who led that field be entered that way and the person who entered the data, there's no consequences to them apart from maybe a tilling off. So again, I find that first party data often is of less quality than stuff we buy. Then we come back to that idea of boiling the ocean which again you just talked about really well. And I take it to the next level of signal versus noise scenario. We start off, we've got 10 systems coming in, we bring in some observability capability, we build some profiling rules, we build some tests, some quality tests, we do a little Bit of cost modeling to make sure we know when things are going to go a bit weird. We do a little bit of anomaly detection, maybe a little regression on our load stats to go, hey, these tables look like they're outside the norm. We bring in some freshness monitoring, we go, oh yeah, well we'll just tag each of the tables with the day they, how often they should be updated and we'll monitor that, bring in some data loss protection, automated privacy stuff that flags data is high risk, low risk, medium risk. Been a name. Okay, so now what we've got is we've got 30, 40, 50 things alerting me. Oh, and it's doing it. Each thing. I've got a hundred high quality, high risk tables that we've said, look, that's actually what goes into our board reports. So those hundred tables are the key tables. We care. I get up in the morning, I go and look at my little dashboard and I've been notified on 36,000 alerts. What do you do? How do you deal with that signal versus noise.

You hit the point on alert fatigue. I think that's where I say don't overcomplicate things. You don't need to observe everything as a start. And then as you observe or as you measure quality or as you start looking into privacy compliance and things like that, I think even if you can build a percentage of, or an application or some kind of specific context within your bigger ecosystem and then build a conference that's much more easier to observe and then train the people to look and make it more sharp. So I would say like it's a more of a strategy how you deploy it. It's not the limitations of the tools and technologies, it's just like one, the people have to buy in what is being shown in the front of their dashboards. And if that is not scaling even within one application context, screw it. Where do you use it? Right? Like, I mean like you're going to be spending hours and hours timing or most of the time what happens is they buy all these tool sets, they don't use it, it's just sitting there. This is very common in banks. If you go to a bigger bank, they pretty much have all different types of tools because different lopees, they want different products, different things. And then you pretty much have 10 different catalogs, 10 different quality tools, each one doing different things. Then the value is always questioned at the end of the day because they don't necessarily start off with a focused approach. So that's what I would say Like I don't even want to start with 30 metrics, start with 5 metrics. If you can measure 5 metrics in an application in a meaningful way and build confidence now you can replicate that same mechanism across multiple applications and across the whole organization. Now you go and add another one. So that would be the kind of the step in the process that I would follow.

I think it's not a solved problem yet if we think about the end to end life cycle. It's always interesting for me that if we have a dashboard or a machine learning model recommendation in Salesforce, it's almost virtually impossible for the consumer of that piece of information to understand the quality of the data that came from because the quality information, the observer information, sits in another system. They have to go and look somewhere else, they have to go and log in or they have to look somewhere else. We don't embed it right next to it. And, and as you talked about that organization that had level one, level two, level three, I remember must be 20 years ago, we were doing a project and we actually classified the end user reports in three colors. We had bronze, silver, gold. And that was all based around the process we used. So the gold reports were the ones that had gone through. This was back in waterfall days, our extremely long, extremely painful 6 to 12 months development life cycle. But we inferred that because it went through that process and went through so many hands, the data in that report was of higher quality. The silver was. It went through some kind of ad hoc process. But there was a peer review, there was a community review by subject matter experts. And then the bronze one was just an analyst hacked some stuff together and pumped it out. And so the end consumer could look at it and have a feeling of quality. But it was all based around the process. It wasn't around observability, it wasn't around the data. I think it's to start off with something and say how do we actually give confidence to the person consuming that information or that data around it, around that observability. And coming back to those key things that we talked about. So we talked about the quality of the data or can we trust it? We talked and observing that, we talked about observing the cost of that data in the platform. And what we've done, we do. We talked about a little bit about detecting anomalies, how do the observability can tell us that something looks weird. We talked about freshness, has it been refreshed as I would expect or as I need? And then we talk about Privacy is the data meeting our privacy things. So again, got a couple extra ones from when we talked at the beginning to 50 minutes later talking at the end. From your point of view, anything we've missed? Any patterns that you go oh, that's the one. That's my goal.

Yeah. I think there are certain aspects of observability which does not cover. Right. I mean example when you're talking about different applications, you can a little bit touched on it like foreign keys and joins and then checking the data integrity. I mean observability cannot do it. You can do the count and you can do an anomaly and you can do some manual thresholds and things like that. But I think it can gets into. You need to look into the data in itself. Okay, like yeah, I see something is happening, but then I have to go look into the data. And now it's kind of like you have to look across multiple applications and data and then try to do more of a business process checks which kind of gets into this process things. So I think there was some more developments that could happen definitely as the industry matures. I think these are things like root cause analysis and trying to look into observability from a process standpoint point using data. There are some things that could happen too. So these are areas where we are looking into it and trying to say like okay, we can do something at the metadata level, we can do something towards reduce the noise so the signals are more valid. And then we can also provide some guidance to the businesses in terms of start slow crawl, walk, run business, try to do everything some limited SLAs on a higher level from a data contract standpoint and empower this federated governance or decentralized ownership from a data mesh principle standpoint. But still I think I'm super excited because like the cloud hasn't fully yet seen its full peak. We are just in the start of it. We are already seeing ChatGPT evolution in terms of AI. We are still struggling to govern in data management. That is a huge. As the data grows, your management needs to be even more sharper and more agile, which means innovation is waiting and more to happen. So I think it's kind of like fascinating to be in this time of the age to just to witness all of this stuff. So more to come, I guess.

Yeah. I think the key for observity right now is there are a large number of patterns that have been shared. There's patterns that we've had for 20 years and there's a whole lot of new patterns that technology and process and people have enabled. So if you're starting your journey, don't worry about the technology yet, right? Worry about the people in the process. Find some patterns you think may have value, try them out in your organization with your context and say, are they valuable? Are they worth the cost of implementing? From a technology point of view, once you start getting to a scaling problem, the machine needs to do the work. Once you start getting too many of those things, you've got to deal with the signal versus the noise and that's the next problem to solve. First thing, just make sure you check for nulls.

Simple 1. Oh yeah.

Data is not there, why bother, right? You have other problems to worry, like you need to call the engineer or you need to call the pipeline or look into the pipeline.

Nulls always bite you in the bum. I'd love somebody to do an actual research survey using data from lots of customers around. What are the number one observability tests like?

When you go into those details, sometimes it's very funny. Engineers like, as much as they are super smart, sometimes it's also like, it's not that they cannot, it's just the way it is being constructed. Sometimes you may have SSNs or some security numbers, which is in a watcher, like a text, right. Just because it has hyphens or they think it make it hyphens and et cetera. I mean, so I have seen in an organization, the same field ssn, sometimes in Newmark without hyphens, something in Wireshark, text with hyphens. And then. And sometimes it even has spaces because some application did not trim the spaces and just puts it for whatever reason. And now you have trailing and leading spaces across the data. And so now you're kind of having all these data quality issues. So yeah, nulls playing some basic checks will go a long ways is what I can affirm.

Yeah. And I think it comes back to at the moment, observability is the first sieve, right. Is the machine telling you to look over there and then the second sieve is still a human, right? We haven't automated that yet. Human still has to go and look at it and observe what's happening at the next level and go, okay, I can see the problem, I know how to fix it. Maybe ChatGPT will give us that. I don't think it will.

You can ask one day the question to ChatGPT and say, is my data reliable and accurate?

The answer for you on that, I.

Think once you ask it for the bio of somebody and it actually doesn't lie to you. It actually, what do they call it? It basically makes stuff up. And there's lots of cases that coming out now where people go, tell me my bio. And it's like, you went to Harvard. And they're like, no, I didn't. So once they solved that problem, then maybe we can trust ChatGPT to observe our data for us and, and tell.

Us what's going wrong.

But exciting time. We've got some cool technologies that hopefully people in process will catch up to. But people in process first, right?

Yeah. That's funny. I just went and just typed this Media Daily label and I killed in ChatGPT.

You used to be. Yeah, how to tell you're a geek. Used to be the T shirt you wore or the TV show you watched. Now, how to tell you're a geek is that your other browser window that's permanently open is CHAT GPT?

Unless it was honest. It said, like, as a language model, I don't have access to your data, so I cannot evaluate.

Yeah, well, that's the other vanity metric, isn't it?

Is.

If ChatGPT can actually do a bio for you, even though it's wrong, it still means you're more famous than most of us.

Excellent. Hey, look.

That'S been great. I think we've gone into a little bit more into observability, into areas that I hadn't thought about. I come back to the idea of an information contract. You and I came up with that here. I've still got a chatgpt in a minute to make sure it's not there. Anyway, so if people wanted to get hold of you, they wanted to find you without asking ChatGPT, what's the best way for people to get in touch with you and what you do?

Email rajdqlabs AI or you can just reach out on LinkedIn. It's again, Raj, Joseph R A J Joe J O S SCPH Rajoseph. That's my profile link. LinkedIn. So one of the other works. I also have my phone number in the LinkedIn profile, so you can text me whatnot and etc. Don't spam me.

You gotta get those automated Visa credit card calls in about. Yeah, yeah.

Yeah, but yeah, definitely. I think LinkedIn and email is the best medium to reach out. And then after that, definitely, definitely. I'm happy to always talk because there is so much innovation and so much ideas and things that comes to the surface every moment and every moment. So happy to entertain.

Yep, that's great. Hey look, thanks for the time. Thanks for sharing some patterns, and thanks for chatting about all things observability will do. We'll catch you all later.

Thank you so much Shane. Thanks for having me. I appreciate it. Bye.

And that Data Magicians was another agile data podcast. If you'd like to learn more about on applying an agile way of working to your data and analytics, head over to agiledata IO.