Driving Data Quality with Data Contracts: What Data & AI PMs should know

Hello, I'm very good, thank you. Thanks for having me on.

It's a pleasure. Andrew has been a member of our community for quite a long time, and I, I see you're always on the, on the other webinars, so super excited to kind of flip the table and have you be the one on the spotlight this time. So I guess I'll let you give us a quick introduction, then we can address the data quality-shaped elephant in the room after that. Tell us a bit about, yeah, who you are, what you do, what's your backstory?

Yeah, sure. So I'm Andrew. I'm a principal data engineer. I live in the UK, just outside London. And for the last— well, first half of my career, I was more in software engineering, moved data second half of my career, sort of 10 years ago. Really been focusing on building data platforms for most of that time, leading data platform teams. And then moving into more principal engineer role over the last few years, which is more where I work less with a certain particular team, but looking across The entire company trying to help improve our data and help improve the applications we build with data.

Awesome. And you're so modest, right? Because you've not even mentioned that you're also a published author. You run workshops and have a course about data contracts. But don't worry, I'll make you talk about all those things as we go. And I wanna jump into that. So I'm gonna assume that some people who have dialed in know what a data contract is, others don't. Before we start kind of getting a bit more technical and defining what that's like, maybe tell us a little bit about what led you to implementing data contracts in your previous organization, all the way to writing a book about it, now training other people how to do it.

Yeah, so at the time when I started thinking about what became data contracts, I was the tech lead of a data platform team. And the company I was working for, we had a new strategy. And one of the 3 key pillars of strategy, as we called it, was to use our data, apply some ML to our data, and then create some applications that we could sell to our customers to drive additional revenue streams. So initially, as a data platform team, we were asked to support the data scientists building those ML models, which was things like, you know, helping them do feature engineering, a feature store, things like that, notebooks. That was all fine, but we kind of realized that that's only part of the story because the data we were using to build these applications, which is model— and build these models which drives applications We were already using that data for quite a few internal processes and it wasn't very reliable. It was failing regularly, like sometimes weekly. And that's okay, it's not great for internal applications either if it's failing that often, but at least it wasn't impacting customers. But we were trying to build these models that we were going to drive customer-facing applications that we're trying to sell and make money from. We can't have these things falling over weekly. It just wouldn't be good enough to sell. So we started thinking what we could do as a platform team to help resolve those data quality issues, those data reliability issues, which is one aspect of data quality. And it was really that problem that led us on a journey to what became data contracts to try and solve those reliability issues, to implement more well-defined interfaces between where data is produced and where it's consumed so that we can have more reliable data that ultimately would drive revenue in the end.

I feel like you're describing the classic situation that almost every data team is in, where you're the ones getting yelled at when quality isn't good enough, whether it's for an internal use case or external, it almost doesn't matter though. Like I say, it's more high stakes when it's customer-facing. But you're not the ones causing the quality issues, or at least I would imagine most of the time it's not, oh yeah, we built a pipeline wrong, there was a bug. It's more with the upstream systems where the data is coming from, right?

Yeah, that's right. Well, that's what we assumed. But obviously, the first thing we had to do was to work out exactly where our reliability issue was coming from. So we did this kind of root cause analysis. We looked through recent incidents, tried to work out where the problem was. And you're right, and it wouldn't surprise anyone, that the problems were mainly out of data teams' control. It was the upstream data changing. At the time, the company I was working for was scaling very quickly. We were launching in new countries and, you know, scaling up very quickly. And therefore, the upstream data sources, in particular our upstream databases, They're changing very quickly. And it's those changes that were impacting the downstream applications of data because we're strongly coupled, strongly tied to those changes. And yeah, that was the root cause. And once we understood that, then we could think of a solution to try to reduce the amount of breaking changes that impacted downstream applications without slowing down the software development teams in our case. as these producers of data. But yeah, very familiar story to, to many people, I think.

Oh yeah. And, and so I, I'm guessing before even you did that root cause analysis, there would have been instances where you had to go talk to the people managing the upstream systems. That's probably software teams. How often was it good enough to just go and say, oh hey, there's this breaking change, or there's this other quality issue, Can you deal with it? Like, as a workflow, would that work or would you get responses like, yeah, no, we're not gonna do that, or it's gonna be priority number 57 'cause we have to do other things?

Yeah, that's a good question 'cause a lot of it does come down to like talking to other people to try and incentivize them to understand why this is a problem and to spend some time helping you resolve it. And it depends on who you're talking to. You talk to software engineers, you talk to a product manager, you talk to the CEO or whoever it is, the CTO. So you have to think about— so a lot of this is talking. I was very lucky, I had a very good PM at the time and she really helped with a lot of this and I learned a lot from her in terms of how to try to engage with the stakeholders and talk to them. The other thing we did was We had quite a strong incident culture at this company. So when we had data incidents, we started following the same workflows that software engineers would follow. As part of that, there'd usually be— there'd always be a— we call it a postmortem process, where there'd be a meeting after the incident. There would be— you'll get people involved from the downstream consumers who are impacted to the people who caused the change in the upstream service. So you're getting these people in the same room with software engineers, with data engineers, data scientists, maybe some business people as well. And the good thing about software engineers is they love solving problems, right? That's, that's kind of what they do. So once they understand these problems and they get engaged in problems, they're quite happy to help think about how to solve this and bring their expertise into that, think about how it might impact their workflows and try and come up with solutions that work for them as well as for you. And that's what we did really. We really worked closely with the software engineers as, as we'll talk later, as the ultimate people who would own these state contracts and made sure we implemented them in a way that they would be happy to use them because it was easy to use because we already built them, but also because they understand the problems that are solving downstream, which ultimately aligned to our strategy. So that's about alignment as well going on. In terms of why it was valuable for the company. So yeah, a lot of talking, a lot of influencing, a lot of incentivizing. But it's definitely doable.

I guess it's one of those things that if you're a product manager, you hear, oh, the solution is to go and talk to people. Amazing. I can do that. Maybe if you're an engineer, you're like, oh, that is my least favorite thing to do. But so what I heard is these data quality issues, and they were severe enough to be incidents as well, right? Took place, which meant, 'cause you had a good incident culture, you brought everyone involved into the same room to figure out, hey, what's the problem? What are we gonna do about it? Come up with a solution. Does it stop there? You know, can we finish the webinar now and be like, hey, if you're having data quality issues, as long as you identify the root cause and get people together, That's kind of it. Where, like you said, some of it is about incentives. And I guess that is a totally different beast. And correct me if I'm wrong, but for me, it's probably a totally non-technical beast as well. It's like, you need to make sure that if I'm running a software team that operates a system, some of my, however I'm defining performance and incentives, come down to Are there data quality issues being introduced for our other applications and use cases downstream? But is that all to it? Is it just about incentives and kind of transparent communication to make sure people are aware of those issues? Where, to bring in the topic we've promised everyone to talk about, where do data contracts come into that equation? What do they look like?

Yeah, that's a good question. Just one thing I want to make, one quick point before I answer that question. You mentioned how— I mentioned how we use the incident culture to help us move this forward, and you said that although most— you kind of assumed that that meant the issues were severe enough to be incidents, but in a good incident culture, it's not just the severe incidents that— issues that become an incident. It could be most— it could be anything. It could be a problem with HR systems up to a Cloudflare-style incident that takes off half the internet. But the process is the same, meaning you still try and learn from it and try to bring people together to prevent the same thing happening again and again and again. I think that's important. Just one more thing on that before I answer the question, because it increases the visibility of problems. Often in data teams, things go wrong and, oh, it's the upstream issues that caused it and we just go and work around it. We'll fix the— put workarounds in that ETL and move on. It happens again next week and next week and next week and the rest of the business doesn't know you're dealing with this. Your ETL is getting more complex. and you're not really solving problems, just working around them. So that's very common again for many teams, but I think incidents are a good way of increasing visibility and something you should consider even if you haven't got a strong culture at the moment, it's something you can start introducing. But yeah, to answer your question, once we've started understanding the root causes, once we started talking to people and trying to get some alignment, what's next. And you can get quite far with that, but ultimately it's, it's not very kind of tangible. It's like people can talk, people can agree, but then what happens? How do we solve this problem? And you can maybe document some things in a wiki page or something, but again, you aren't really— you know, maybe this is where a lot of their contracts are. They start documenting things in a spreadsheet or a conference page. But again, that's not really doing anything to prevent these changes happening. It's just documenting conversations. What we wanted to do was to prevent these breaking changes from being rolled out into production. And we did that by looking at how other interfaces, integrations are built between teams. So think about software engineering teams, when they depend on another software engineering team, I'd always have an interface, an API between them, between the 2 services, and that would have change management on it so that you can't make breaking changes to it. If you do, there's a change management process you have to follow. That would have things like SLOs, that would have ownership and governance around it, and that's what you need if you're depending on— you have this dependency but you want to be reliable. you need some sort of interface on it. So you see interfaces everywhere in software engineering— APIs, libraries, everywhere. And we started wondering why, why we didn't have the same sort of interface for our data. Why we didn't have this change management on this interface when it changed. Why we didn't have any guarantees around the SLOs or any expectations being set around the quality of data. And Again, we got lots of engineers involved in this discussion, let them help us come up with solutions. But we really started to think about this API for data, we started calling it this interface for data. And the way that became a data contract was because to create this interface, to create this API, you need to document it. But you also need to use it to create these interfaces, use it in code essentially to create these interfaces. So then we started documenting it, but also using it in coding automation. Then it became more of a documentation because part of the system, part of platform, part of the way we made data available to other teams. And that's when it became something we felt was a bit more, more than just a document. It became a contract. And I can't remember if the word data contract came from.

So to to kind of help understand it and maybe explain it in more plain terms. I'll repeat it in my probably half-broken understanding. You tell me how off I am. A data contract is a black box of code. I say black box purely for my purposes. Like, I don't need to understand necessarily what's in it. But what that does is if I have, as the data platform team, a data contract with the team that owns our transactional system that makes sure that transactions get logged, partly so that we can sell our products, partly so that the data team can have sales data about those sales. What the data contract does is if the software team decides, oh, you know what, this column that, I don't know, says whether VAT was applied to the transaction, we don't need— we don't think we need it anymore, we can just delete it, or we can change the data type, or we can rename the column name. Mm-hmm. That little black box is there to kind of sound the alarm and say, hey, that is not a column you can touch. Because if you rename it from, I don't know, VAT included to VAT underscore included, that's going to break all the data pipelines downstream that needed that column to be named that or to have that data type or even to exist in the first place. Is that— before we talk about maybe what's in that black box, is that right? Is that what it does? Is it a thing you give to the upstream software team?

Yeah, effectively, it's, um, maybe taking a step back. So what goes into this box is essentially the data contract. It's a description of the data, it's just metadata. So it's a very simple idea really, you're describing the data in a human and machine-readable format. So you think the most basic data contract would have is a schema, and it describes yeah, this column is the VAT column, it should be between these numbers, it should never be null, or whatever kind of things you need to have around it and make some documentation around it. But that's kind of a minimal data contract. That's useful for you as a human if you're looking at it, but on its own, it's just documentation. Like, it could just be anywhere, it doesn't really matter. But the key thing that makes it go beyond documentation is you then use it in your platform to enforce those kind of rules you just spoke about. So enforcing, okay, this VAT field is a required field, we can't just remove it without having some sort of migration path to a new version of that contract like you would an API. So you're using that and you're having platform— if the platform itself is doing those checks so that software engineer can't just— it's not just relying on people remembering to do those to follow those rules that they've agreed, because that— those people forget the platform is enforcing those rules. So it's—

Is it a little bit like creating the unit test? You know, when I'm writing software, if I have unit tests, it basically says my code needs to do these things, needs to pass these tests. And so if I make updates to the code and I try to merge it back into the main branch, if any of those tests fail it'll give me the alarm and I'll go and fix it. That's kind of what you're describing, right?

Exactly right.

I'm curious because the word contract, I don't know, it inspires a very harsh, blunt instrument that, you know, if I— what happens if I break the contract? Is it the same as if a unit test fails that I could kind of force the change to go through anyway? Or in your platform, did you have it that physically, like, an engineer could not merge into the main branch, could not push something into production if it was going to break a data contract.

Yeah, in our platform it was the latter, so people couldn't deploy to production if contracts failed. Um, so really enforce there. I think it's, it's better if you can do that, obviously, because then you prevent things going into production, because once they're in production, the impact radius gets much higher. Um, so trying to do it later is difficult, and it's less— it's much more expensive to fix any problems that go wrong. So we made sure the platform implemented these rules and prevented breaking change from being deployed to production.

Were there many objections when you started implementing this and rolling it out? Now, you mentioned that you had some of the software engineers involved in the discussion, but my guess is Maybe not everyone was super on board from the start. Did the software engineers see it as, oh, platform team is trying to kind of add roadblocks to impede our progress?

Yeah, that's a good question. That's where a lot of people worry about day contracts. And we only get objections, but the way we overcome those was we made sure that the experience was as good as we could make it for the software engineers. And software engineers, they understand interfaces, they understand APIs, they understand why we'd have these kind of things. The difficulty before we built the platform features was like, well, to do that, like, how do I do that? It's gonna be a lot of work for me. I'm gonna have to keep remembering. Or in fact, actually, one thing that we tried when it was a much smaller company was every time a software engineer made a change to the application that we thought might impact data, it requested a code review from a data person, and we had to wait for someone to tick it. And clearly that didn't scale, that became a real bottleneck. And also, the data engineer reviewing software engineers' code, they didn't understand the code well enough to know whether it's going to impact data. It was just a waste of time. So that didn't last very long. But we kind of learned from that a bit in that we want to make sure we weren't a bottleneck. But we wanted to make sure that we were running the checks. So we worked really closely with the software engineers to implement these checks within their existing workflows, really meet them where they were, rather than having a human review it or having them go to a data platform or to a data catalogue or somewhere else and come back and back and forth, come out of their workflows. So for example, We, our software engineers, they defined the APIs in this kind of weird language called JSONnet, which is like JSON with extensions, but it doesn't really matter. But the key thing is that we therefore implement data contracts in the same language, in JSONnet, and with the same tooling, the same workflows in BearKit repos, really meeting where they were, giving them a good experience as a customer so that The friction for them wasn't too high in terms of adopting this thing.

I guess in that sense, it's similar to a lot of engineering best practices where, yeah, at the beginning when you ask someone to even, I don't know, use Git, you know, or write more clean and readable code, they might go, yeah, but I can work faster doing things the way I'm doing now. And it's like, yeah, you can work faster This week, but if we zoom out to the whole year, how much of your time are you spending dealing with these issues? Exactly.

Yeah, exactly. And I think in, in other areas of sort of platform engineering, they talk a lot about platform as a product. We obviously talk a lot about data as a product, but same sort of thing, applying that product thinking to what we're building for them, not make them go to where we are in our data tooling, in our data world, but really meet them where they were. And one thing we did before data contracts was nothing we learned from. It didn't go so well. And it was kind of our first attempt at doing this sort of API for data. But this time we, in that first version, we really did try to make them come to where we were. Made them have to define schemas in a different repo. It wasn't in a language they knew. It's in something called Avro, which is a definition language. I did it in Avro. Turns out Avro doesn't work very well with Ruby, which is what they were writing code in. It doesn't really like— Avro was good because it's what we knew, but it's not what they knew. And all these little issues on their own, they're not deal breakers. We add them all up, it's just more reasons for someone to say no to this thing. So really, if you're trying to write update contracts, I talk a lot about the customer, the people who want to own the update contract being software engineers. So we want to meet them where they were. If the soft customer, if the owner of data contracts were the SAP admins, then there's a different conversation there about where to meet them. But really thinking about, look, we're asking you to do some things for us effectively. Let's make it as easy as possible for you. Let's make sure we talk a lot with you. Let's make sure we understand your concerns and really try and meet them where they are, not where we are.

So does that mean that if an organization wants to get, well, to get serious about dealing with data quality and governance issues by adopting this contract way of doing things, is that primarily like an organizational— I don't want to say just a single conversation, but it's mostly about getting to understand, like you said, Where are my upstream data producers at? I said, what languages are they comfortable with? What systems are they using? And then think about, okay, what is the technical solution to that? It's not, oh, data contract means you need to do exactly this, implement this exact technology. But is it— what I want to understand is it Of course, there needs to be the stakeholder engagement part, but does the technical solution differ very much? Like if you were to implement data contracts in a new organization, or I imagine you've spoken to a lot of people that have read your book or worked with you, does the solution look more or less the same at different places? How much does it vary? Yeah.

Yeah, that's a good question. I think the sort of the UI, the user experience varies across companies because you're, you're thinking about your customers and what they want to use. But the actual tooling behind it is largely similar, and it's actually not that hard to build. Like, I'm not talking— we did that POC in a couple of weeks really, where we worked with a software engineering team who who were already quite data literate, understood the problems we were having. Wasn't the most important dataset in the company, but it was one we were willing to work with us. And it only took us a couple of weeks really to do a POC, which had a lot of things I'm already talking about. So defining the schema, having change management on it, creating the kind of interfaces. It's not a lot of work. In my book, it's like one chapter to go through this and build it all. So we're not talking about like months of effort to do this. But, um, I mean, from there you can obviously extend it and add much more, more automation, do a lot of governance in it and those other things we could talk about maybe later. But yeah, I think because it's not actually that hard to build, you shouldn't be too put off from like thinking this is a massive investment. But then there are patterns you can copy from other teams and other companies in terms of implementing their contracts. You're not You'd have to do it all on your own.

Makes a lot of sense. I'm guessing a little bit like with tests, you probably can't foresee everything you want to create a contract around from day zero, but it's more okay, you can think of 50 or 80% of them, start implementing them, and then anytime there's an incident that you've not covered, you're like, let's add this. Is that, is that usually how it works?

Yeah, exactly that. Yeah, for us, like, a lot of incidents were change management related, people breaking the schemas. So once we fixed that, that resolved a lot of incidents. We also had, um, various things around data quality, and, and we've added more and more data quality checks to the data contracts that automatically run, and then the alerts would go back to the data producer and they could fix it. Or potentially preventing data from being written to the data platform until they're fixed, like in their code having alerts, providing libraries to do that. But we also realized that once you start describing your data, having this data contract describes data, you can use that to power a lot more automation as well. So you can start adding other things in there like categorizing data, is it personal data or not, is it related to a customer or an employee, those kind of things. And then you can automate around— automate data retention away and GDPR deletions and backups and those type of things can all be automated away. And this, apart from being great for the data platform team because now you're building very generic tooling that will work for anyone who's using data contracts, whenever you roll out a new feature everyone gets it, it's also great again for the data producers who are owning this data because now they're not becoming experts in data governance and data regulation. Again, reducing friction and the amount of cognitive load they have to have for being data producers, data owners, by automating a lot through a platform. And again, a lot of this tooling, you can plug in some open source tooling, you can build it yourself in days, weeks. It's not— it's not looking loads of investment to do this. You're just applying a lot of tooling you already have around data management, data governance, but you're rolling it out a lot wider using this data contract to give you context to apply it to datasets all across the company. And really, like I said, we didn't think about all this at the start, but incrementally, every time we wanted to add a new capability to our data platform or implement a new governance policy or data management policy. We just found we could just do it by using data contract to drive the tooling. And yeah, everything we added over 5 or 6 years was added this way.

That makes a lot of sense. So what you're saying is the data contract becomes the vehicle through which it's not just, oh, like me as a data engineer, as a data platform owner, I'm going to impose these requirements on upstream software teams and make it easy for them to be compliant with those requirements. But there might be other people in the organization, like someone from legal or from data governance, that also uses the data contract to specify those things that either need to be categorized or create some additional rule about it, like retention, or like, I don't know, maybe we need to hash this column because it's super sensitive. It can't be in the main database, uh, exposed because it's, I don't know, people's phone numbers, that kind of thing.

Yeah, exactly. The data contract is gathering all this context and keeping it up to date because it lives near the code that generates data. It's usually checked into version control, into Git, like the code is. It's— so it's beyond documentation. A lot of companies will have kind of standards for how they handle personal data And they have standards for various other forms of data governance and various other documentation around it. But how do you know it's being applied? Like, you're relying on people to have read it, to understand it, and then apply it themselves. And you've got all these different data owners across the company applying their own understanding of the standards to their data. What we did was we have standards being defined by sort of chief data officer or data governance team or whoever in company would be doing that. And then we would implement those standards in the platform using context from the data contract. So the data owner, they know their data, they can say this data is personal data, and then the platform says, okay, in that case, after this many years we have to anonymize it because it's expired from its data retention policy. But as a data owner, you don't need to know the policy inside out. You don't need to remember to check it every quarter to make sure it hasn't changed. You don't need to write your own code to implement the logic on your own data. It's kind of all done for you. And that's how— that's what you need to do really, I think, if you want data owners to actually take more responsibility for the data. You need to make it much easier for them to do so. And I think you do that through automation rather than through Writing up standards and hoping they follow them.

That's it. I mean, the number of times I've heard people talk about the importance of identifying data owners, but then nothing really happens off the back of that. Or you tell someone, you know, are you going to be the designated owner for this data? And they'll either say yes and have no idea what they've signed up to, or they're not going to say yes exactly because they don't know What does that entail? And do I need to now become a data expert as well on top of doing the rest of my day job?

Exactly.

The other thing, as you were, you were talking about putting everything into that single central kind of repository of context, all your different policies and constraints, I was thinking, oh, that's also super true of everything we're doing now with AI, that we need to have all the context for someone, be it a new starter or an agent, to understand what is going on here. And it's, uh, I, I— Marcin has been very patient in waiting to have his comment, um, kind of relating to this answered. I want to get through the, let's say, the more vanilla data contract stuff first before diving into it. Let's spotlighted on here. So Marcin wrote, in AI-driven ideation, agents rely on unstructured context, strategy, user data to augment participants' work. Standard data contracts focus on schema rigidity, but here the risk is vague content— I get context leading to generic outputs. Do you see data contracts evolving to include semantic checks for unstructured text, like asserting that a source document actually contains actionable goals before the AI is allowed to process it. I like that. Maybe should we start with the, the kind of premise of this question? Like, is it true that that's the focus of data contracts now, like maintaining a schema? Or like I said, have they already evolved to cover more things beyond that?

Yeah, people have used it to cover more things beyond that. Basically, whenever you want to describe the data in some format, then you can put it in a data contract. So it could be you have unstructured documents in a bucket store somewhere, like AWS storage somewhere or something like that. And you have those documents in there and they might relate to identity checks, people uploading their driver's license or their ID cards or whatever it is to this storage. That's unstructured data they could potentially run AI over, I guess. But you still need to track, you know, when was that data uploaded, how long should I keep it for, um, who owns this data, all those kind of things. Still need to track that metadata. So even if data itself isn't structured, metadata would benefit from being structured, and that context will be used for, for humans who want to understand what's in this bucket, but also for tools that want to automate retention policies and stuff and governance policies on that data. But also for the AI, the agent as well, right? They benefit from this context. They want to know, you see a lot of this in files like PDF files. You've got a lot of this metadata within file anyway. But if you're just searching across a whole bucket, then you can use the context from the data contract to help the AI discover the right files to go and look into in the first place before it goes to try to parse them all. So yeah, anything humans and AI both generate from the added context that you can put in a data contract.

Makes a lot of sense. So yes, the, uh, your, your objects might be unstructured data, not structured, but the metadata still is structured and you can still treat it in a similar way. But I think the other thing—

oh, sorry.

Go ahead, I'll ask after you.

I think, give an example project of something we did around that contracts with unstructured data. And, um, we had, um, we were storing contracts, as in the company-to-company style contracts. Um, that was all in PDFs and unstructured data that we— and they weren't particularly standardized. We had to try and extract data from that so we could better build our customers and What we did there was we were going to use AI, but it turns out actually just a good PDF parsing library in Python was fine for us and a cheaper option. So we did that, but then we wanted to pull that data into structured format and we had that as a state contract that then allowed the billing team and various other teams using this data to depend on this contract data. So Even if you don't think you need to apply the data contract to the actual files themselves, when data contracts become much more useful and the best place for data contracts really is the interface between when you're making the data available to others. So in this case, we, this particular team, we owned contracts and contract data. We were producers of that data, owners of that data. We made it available to our consumers and we had a contract there to allow people to consume structured contract data that had proper SLOs on it and had change management on that data to our consumers. So yeah, I think that's really key on where data contracts are. I wouldn't use data contracts to describe and document all our data. I'd probably, I would start with where they add most value on these kind of boundaries of data, where data moves between ownership from one team to another. That's where they add most value.

I guess for everything else, You hopefully are still doing your more normal data quality checks. You don't need a contract.

Yeah, within your system. So say you're, well, in this case, I use the same example. We obviously, we ran a lot of data quality checks on the PDF parser that's parsing these contracts. It could have been AI. It wasn't in this particular case because, like I said, it was just cheaper not to be. Um, and it worked just as well. But you can imagine it was some sort of LLM. And we had lots of checks making sure that the output of parsing was correct before we gave it to our customers. And we didn't need to use state contracts for that. We just, we just used Python testing essentially. Um, it's really— and same if you are a software engineer, you own a database, you would still have tests on your database, have constraints on it, have sort of non-null constraints and, you know, constraints in your ORM and various other places. But that's different. Both of those are different because you can make changes to internal data structure with low friction because no one else is depending on that. The changes you make to your external interface, your API, that has a lot more— that should be more friction on that because you're impacting everyone consuming data. And therefore, that's where you want to have it more control over the change management and things like that.

So help me, help me understand this, and maybe the answer is I just don't need to care about the technical difference too much as a data PM. If my data engineering team or my data science team are using, or the upstream engineering team, something like Great Expectations to again kind of create a JSON and define what are the data quality checks I want to be applying. How, in terms of, of course, there's the talking to people side of it that differs a lot. But as a technical solution, how different is that, like creating that effectively metadata to define the quality checks I'm going to perform, to doing that but with a data contract? What's the— The extra or the different thing, is it just that it's a different, like, way of structuring those specifications, or is there extra information that's not included in the data quality check?

So it kind of depends what you're testing and where you want to do the testing. If you talk about testing you might do at what we're calling, like, the interface layer, like where you have a data contract, just making it available to others. So this— so I, I'm a data producer and I'm making this data available data available to you. And I promise that this field will always be present and it says non-null and always presume a value of 0 and 1. What we had is we had those in the contract and then the platform ran those quality checks on the data as it was ingested into the platform. Oh. And the user didn't have to care or know how those were ran. They just ran and the alerts would go back to them, they'd find the Slack channel and it would go back to them. And we could use Grapefruitations, we could use Soda, we could use OpenAPI libraries, it doesn't really matter. And the user didn't care, we could change it over time, it just worked for them. So that's a case of where again we're trying to provide a platform that allowed them to not have to care about setting up their own Great Expectations framework and their own, um, scheduler and things like that. But if you're also using Great Expectations, maybe you're running dbt pipeline using Great Expectations to test each individual pipeline within your own control, or just using dbt test and stuff like that, I wouldn't say you need to use data contracts there because I think you probably want to be testing with your own testing rules and it doesn't matter so much. But most— yeah, so it depends on who your data scientists are and what tool they're using, how I think that's less important.

So again, rephrasing it in my, my dumbed-down understanding, the data contract is effectively a wrapper that the platform team creates so that you don't have to build it yourself.

Yeah.

Like a wrapped data— let's say data quality check or whatever else might be inside that contract. Yeah, that's a shared language between producer and consumer every time.

Yeah, it's a capability that platform is providing for you. You just have to tell platform what you want to do with, how you want to, what checks you want to run, and platform work out how to run it and send the alerts. And, you know, if it doesn't, make sure it runs it reliably and makes sure it runs at a certain time of day and things like that. You don't have to care about that anymore as a data owner, platform decides for you.

Gotcha. And so going, going back to Marcin's question, can those things that are being checked, or have you seen cases where you actually are looking inside the content of the things like a PDF document, and it's more complex than, I don't know, something that you can defined very clearly and programmatically. Like, I actually using an LLM to interpret what's in the document. Like the example that he gave us, let me put it back on the screen for everyone to see, was, okay, does this document contain actionable goals? You know, which is a very— it's not, does it include, I don't know, is the language English or is it something otherwise straightforward to test for? Is, is that— it sounds like from what you've said that you could build your data contracts to also include that. I'm just not sure how reliable necessarily it would be.

Yeah, I think at some point you need to convert this into something you can programmatically run against this document. So you could have your date contract, it could say actual goals equals true is one of the checks, checks I want to run on every every document in this case. And your platform, or any code, if you want to try and run that test, needs to convert that into something you can run against the document. And it can run it by checking the output of LM, or it can run it by using LM to look for— like asking, does this document contain actionable goals? And hope it works out the right answer. And then if that data quality check fails, the alert will then go to the data producer. And in this case, I'll be questioning whether this is a useful test to run would be how many alerts are being sent to the owner of data and are they useful alerts when I check it to actually, you know, how good is the LM, how good is the agent at running this test if you're going to do it that way. But these are just the sort of things you need to think about whenever you're using LMs and agents to try and define something programmatically because they are not deterministic. So the data contract is just providing context to help it work out what to run. If you're trying to provide a platform capability to do this, you just— it kicks off that capability to kick off that run of the check, runs it against the document. The question is there, it's not anything about the data contract, it's about how do I then programmatically run that check and what would it look like? But yeah, there's no reason why you can't do that.

Makes sense. I guess this reinforces the importance of unambiguous checks as well, right? Because I can imagine that with those kind of examples, maybe the LLM will return has goals no. It'll flag an error, you know, it'll be considered the same as a breaking change. owner gets notified, gets alerted that, hey, you've tried to upload something that's not allowed. But then if they go, no, actually, goals are included, it's just, I don't know, worded in a more subtle way, or it's implied, or whatever, it's basically something that they can dispute. Whereas something like, hey, these files are only meant to get uploaded at 7 AM every morning, or you need to have these columns, That's completely unambiguous. You can't have the data producer come back and say, no, no, that column is definitely there, unless if there's a bug. But is that something that happens when you're running these data contracts, like cases that get disputed in the same way that you would dispute the terms of a contract, and you have to figure out, okay, we need to make this tighter, more specific, change how exactly we've structured this data contract?

Yeah, um, yeah, I think it's kind of when you're writing any sort of check though, any test, it's like, how useful is the test going to be? Is it worth the cost of running it? Is, is the result going to give me a useful alert or one that I'm going to get every day but always ignore it? And then that's always a bad situation to be in because then you start getting alert fatigue and you just ignore the good alerts, the important alerts, and It's a very common problem, I think, for data teams and for many other teams, really, software engineering teams too, where you go a bit wild with your tests. You maybe don't write the best tests you could do, the best checks you could do. And then one day you wake up, there's like 14 fires, and actually none of them matter. Next day, there's also 14, but one of them is different. And actually, that's really important. It does matter, but you're not checking them anymore because you were expecting fires every day. So yeah, you need to be careful when you're designing these tests, make sure they are unambiguous, make sure they're actionable. Otherwise, you're going to drown yourself in errors, which a lot of teams do.

Oh yeah, alert fatigue, unfortunately, is very relatable. Had to deal with a lot of that in previous roles. There's one more question. And thank you, Antti, for bearing with us. that I want to spotlight. So it's split into 2 parts. So Antti wrote that we're going to use OpenAI to deliver data for our stakeholders. The contract is clear in terms of expectations of data scope and format. And the next part is that our concern is that since the model is not ours, it'll be more difficult to monitor changes in the output, may even need semantic checks like Marcin wrote in his comments. Can a data contract catch this or do we need to handle this with monitoring or regular testing?

Yeah, that's a good question. Similar to the other one, I think we can use a data contract, I guess, to— we can use it to define any test, right? So the same sort of thing you could do in an observability platform, like look for anomalies and things like that. You could define the actual rules, your anomaly rules, in your data contract. And benefit of doing there rather than in a platform is that you're keeping those checks next to the code that's suggesting the data from OpenAI. So if you change your code, change your prompt, change your API calls, you can update the data contract in the same commit as a software engineer, in the same, about the same time, and both checks will stay in sync with the code. Whereas if you're going off to an observability tool and you're writing checks run out of sync with the code, and they tend to fall out of sync and you get loads of alerts and get alerts fatigue like we just spoke about. So I think the way I would probably approach that— well, there's 2 things I consider. One is like, is this something that a lot of people want to do? And if so, I'll try and probably create a platform capability for it. If not, then I'll say, you know, to go off to our observability platform and create it and come manage it yourself, because that's very unique for you and it's not worth the investment. Let's say it was worth investment. I want to continue this contract-driven platform that I've been talking about. Then I would be thinking, okay, how can I allow them to express the check they want to check in a human and machine-readable format? So it works good for documentation and for humans to read, but also for me to consume that definition and then be able to programmatically run this test against the data, the output data from the API, OpenAI API. Cool. And I might say, I might, yeah, so I might try and define some anomaly checks like, does something change by this percentage from yesterday or from last week or whatever it might be? If it's so, it's not quite deterministic or it's not, it's not that specific. It's not saying this has gone from 0 to 1, but it's still deterministic, still programmatic. Programmable, and then I can then call the API of my observability platform and get that to run it. Or I could even take myself and have me read every day and do my own checks. So I think yeah, I can't see why you couldn't do a date contract. It's same as the what we're saying in terms of you need to somehow convert this into code. If you do it via observability platform, maybe UI for you to kind of. drag things around and, and try and create this check. If you want to do it via date contract, you need to find some sort of way to, um, to allow them to define it in the right— with the right expressiveness, but still being able to make it too complicated. Um, so yeah, you could definitely do it with date contract. Whether it's worth investment would depend on how many of these you want to do and how much you want to invest in contracts, rather than saying, okay, this is just a specialized use case, go go to the SAP platform for now and let that handle it.

Really sounds like a lot of normal data and analytics work, right? We create new features because we want to do analysis based on those features. As long as you can create those true/false or scale or whatever other columns that can describe what's in your document or row of data or whatever, you can then apply— you can use data contracts for them. But one thing, and you've kind of alluded to it a couple of times now when you've talked about the cost of running a particular test, let's park the labor cost aside, like how much human time is it going to take to actually create the test, check it, make sure it makes sense, implement it as a contract, go talk to stakeholders. What about the computational cost of doing these things? Because in my previous job, we were ingesting a few billion rows of data every day from the telcos that the group owned around the world. We were spending a huge amount of money literally just to ingest the data, do some basic schema checks, and then store it, let alone all the processing we did on top of that data. Afterwards, if I'm gonna start implementing data contracts to deal with my upstream producers sometimes introducing breaking changes, how significant is the, the AWS bill increase gonna be?

That's a really good question because it really depends on where you're implementing these checks, and the later you do it, the more expensive it will be. Not just in terms of computational costs like you're talking about, but in terms of the sort of blast radius, in terms of how far this bad data has gone into your systems. So when we approached this, we wanted to do this as early as possible or as left as possible. People talk about shifting left and, and it's the same sort of thing. So when we implemented these data quality checks, we didn't run them on data warehouse every night. that would be expensive. And that's where a lot of people get stuck when they build— they bring in an observability tool that's got its own licensing costs, which are not insignificant. But then you realize applying these checks to a table also has a lot of cost in terms of your data warehouse costs, and you end up spending more money than the value you're getting from those checks. And it can be tricky. So we didn't want to do that. We, again, we focused on our customer, which was our software engineers. And we wanted to make sure that, you know, what's the earliest point we catch this data that was not valid, that didn't meet the contract, would be in their code when they tried to publish it to their platform. So we started providing libraries that would do checks within their code, and if it failed, it would raise an alert within their code. And they had a system for dealing with that already that would then create an incident channel and various other things, and they look into it. We implemented patterns like if they let a queue for bad data to go down and various other things. And this meant it was a lot cheaper because you're not running this on a data warehouse, you're running this essentially free using the code. And also the data hasn't left their system and gone into the data warehouse or gone into stream applications and then gone to various other downstream applications and suddenly it's gone everywhere. So the actual cost of the bad data being remediation costs are a lot less because it's just it's just remediation within one system. It's all within that team's ownership. They can fix it, they can republish the data. Again, the libraries help them do that easily, and therefore it didn't cost much at all really. And yeah, that's how you avoid the problem with a lot of the normal approach of doing it in data warehouse. That's where it gets expensive. But if you start thinking outside of where you control the warehouse and where your data has been produced in, in code, in Salesforce, in whatever it is, that's the best place to implement checks, and that's the cheapest place too as well.

And that brings us back to one of my favorite notes. I've written it in red instead of blue like, uh, the other notes I've been taking, which is we need to meet the data producers where they are, not Bring them to where we are for our convenience. I love that. I'm conscious of time. It's probably going to have to be a yes/no answer, but we've got one last question from Nigel, which is, have you used evidence of a data contract being broken in terms of commercial impact or SLA within the supplier? So gotten your data producer in real trouble, I guess.

Not personally, but I would I've spoken to a lot of people who've tried to do this or have been thinking about doing this but not actually got to production as yet. But certainly when you're consuming data from a third party and you have this relationship, then if you have a dependency where you're using an API, often there will be SLAs in that. And if they breach that, then you'll get some sort of discount or for some other there's some penalty for them breaching the SLA. For some reason, again, we don't get this a lot with data being passed from one company to another. But again, there's no reason why it couldn't be if the data you're ingesting is important enough and if you've got enough influence over that supplier. So I put in the same contract. You know, if you're a small company trying to get Salesforce to do this, it's probably unlikely. But if you've got a bit more power in that relationship Then there's no reason why you couldn't have this data contract being part of the SLA and part of the commercial contract you have with a supplier.

That's it. Like, as both a data buyer and a data vendor in previous jobs, it wasn't quite a data contract, but in the literal contract there would be an appendix specifying things, and the more robust that appendix was the better we could cover ourselves, or well, the better our buyer in the case when I was the vendor could cover themselves. Now, so often if there's a single breach, you're not necessarily gonna, you know, cancel the contract, issue a massive refund. That's handled by the commercial people that negotiate. But strictly speaking, you can track the breaches in that way.

Yeah, exactly. Yeah, an example, I worked with a company and they, um, They were a big construction company and they had to get data from their subcontractors related to, um, something like carbon tax, I think. So they had to get some data on like, um, yeah, the amount of carbon that's been generated by subcontractors as they built these construction things. Um, and although that's a company-company relationship, obviously a bit closer because you've got the, um, construction company and the subcontractors. But they were looking at date contracts to ensure their subcontractors could provide them data in the right format so they could do their reporting to the government to meet the regulation they had to meet around carbon reporting. And they were going to use date contracts there. I don't know how far it went in the end because I didn't work with them that long, but that seemed like a good solution for date contracts.

I'd love to talk more about this because I actually have a lot of construction data quality related stuff to, to comment on, but we're out of time. Andrew, thank you so much for your time. Thank you for joining us. One thing we didn't get to mention is that you've also created a course for people that want to start implementing data contracts, and I think you, you're going to give us a special discount code for the community. Is that right?

Yes, of course. I course. And really what it is, it's called Implementing Data Contracts, and it's about the patterns I've been talking about of how to implement data contracts in your organization. So no matter how you choose to do it, no matter how your customers are, no matter who your data owners are, you can have patterns to create the data contract, to create the tooling around it, and then apply them in your organization. It's not just technical, let's talk about the stuff we mentioned at the start, really, the stakeholder engagement, how do you get buy-in, how do you get alignment, how do you get incentives aligned, and that as well. So not just the tech side, but also actually how you deploy this and roll this out. So yes, I have a course called Implementing Contracts, and I have a discount code, which I didn't have immediately handy, but maybe we can— Don't

worry. I'll send it around. Everyone that's signed up to watch this webinar through Luma, you'll get an email. If you're watching this and you're like, I don't know what Luma is, but I want the code, I guess drop me or Andrew a message and we'll sort you out.

Yeah, I'll get you— there's a 25% off code you can use. So just message me if you like one. And if you've got any L&D budget expiring at the end of the year, maybe this is the perfect gift to yourself.

Amazing. Well, thanks so much for doing that. Um, especially 25% is no, no small number. And thank you also for giving up all this time of yours to, to explain data contracts to someone asking all the dumb questions about them.

No, you're very welcome. It's a great conversation. And yeah, we could speak for hours, but hopefully it's given a good introduction to people who want to learn more about it.

Oh yeah. Thanks to everyone who joined. Thanks to the people that asked questions. And yeah, we'll see you at the next one. Bye-bye. Bye.