Data Contracts with Andrew Jones - Episode #54

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Andrew Jones.

Hey, Andrew, thank you for coming on the show. The reason I've asked you to come on and you graciously accepted is to talk about this thing called data contracts. At the beginning of this year, I was trying to guess what terms in 2024 are going to end up being buzzwords, uh, and what terms I think are going to survive to 2025 because I think they actually have value. And data contracts is one of the few in my head that I said I think they've got legs, uh, I think they're valuable, I think it's a pattern we should implement, and I think it's something that will survive to 2025 and 2026. But before we rip into what is a data contract and why should we care, why don't you give the audience a bit of a background about yourself and where you other than where you've come from.

Yeah, sure. So I've been in industry for around 20 years now. I started off more as a software engineer and then moved towards building platforms, like engineering platforms, infrastructure platforms, and then after that into data platforms, which is what I've been doing for the last 10 years or so. And it's been around 6 years since I started thinking about data contracts and using that to build data platforms.

Excellent. And it's rumored that you're the person that invented the term data contracts.

Yeah, so yeah, it's around 6 years when I started thinking about what became data contracts. It's the same sort of time that Jean-Marc released data mesh articles, and they were quite good inspiration as well. But I kind of didn't want to call it data mesh internally because that felt like a big thing. I was just leading a data platform team. I wasn't ready to change the organization in that sense. So I started thinking about, and we'll go into what data contracts are later, I started thinking about what became data contracts. So I just needed a name to give it internally. I was thinking it's a bit like an API. An API is like a contract between a provider and consumer. Maybe this is the same thing for data. Maybe I'll call it data contract. And that's kind of supposed to be a temporary name, but it kind of stuck internally. And I started thinking right about it publicly. I started Googling it, seeing if anyone else used the same sort of term or used similar terms for similar things. But at the time, I was just Googling data contract. I was getting, you know, get a 4G mobile data contract. I couldn't find any other literature around it, although I could see people doing similar things, but there wasn't really a name for it. So, I started using the word data contract in my public writing as well, and that sort of has stuck now for the last few years and became, yeah, became a bit of a pattern, a bit of a category now in data.

You've implemented data contracts in the company you work for, and you've written a book about it. So it'd be fair to say that you have quite a bit of experience in this thing that is a data contract.

Been using it internally for about 6 years in production of company I work at, speaking about it publicly for around 4 years. And a book came out last year. So we've been doing it for a long time and we're doing it very successfully on our work and also with other organizations I work with now as well.

Let's start with the basics. How would you describe what a data contract is?

So a data contract at its most simple is a very simple idea. It's something that describes your data, so holds your metadata, and that's all it is. So it's quite a simple idea really, but it's the way you use that, that's what makes it powerful. The way you use it to do whatever you like really, whatever problem you're trying to solve for the organization. So the problem I was trying to solve initially was we had upstream changes breaking downstream data pipelines and data applications. And that happened too often. And we were trying to use our data for more and more important things, things that drive revenue, things that are key processes, shouldn't be breaking every couple of days. That was the problem I was trying to solve initially. And what I realized is that we were basically building on top of the upstream databases. We had to change data capture service, chuck it into a data warehouse, and to build on top of that. We're essentially building on top of a database. The upstream database. So when that changed, and it changed quite often as our software engineers developed features, it just broke things downstream. So that's the main problem I was trying to solve. So I thought, well, again, I spoke about I've got a software engineering background. If I was a software engineer, I wouldn't build on top of someone else's database. They wouldn't let me do that. We'd always have an API. And I started thinking, I want the same thing for data, because data is as important as our software services. Why don't we have the same thing with data? Why not? Starting to get towards the solution there of like what became data contracts. I needed something to describe this interface, something that could then be used to provision an interface, and that became the initial data contract. So essentially it was just a schema and a few other things in there that we then used to create the interfaces to move away from JDataCapture. So that's the problem we try to solve, but it's a very simple idea, data contracts. If you've got other problems you're trying to solve, they can help by— they can solve by describing your data better. For example, maybe you're trying to do access controls, automate that based on roles, and you want to categorize the data, what's personal, what's not, what's confidential or secret, what's public. We do that as well in our data contract, and there's nothing put in there. So it really depends on what problem you're trying to solve for your organization. But the most common problem I'll say is around the change management of upstream data, and that is where we started with data contracts as well.

So I can see a lot of patterns in a data contract as you describe it. So the obvious one is a series of tests. Definition of a schema. Does this data match the schema that we've agreed? If you were doing a minimum viable data contract, would you start with schema definition as the first thing you start with?

Yeah, schema definition is kind of something you need in any data contract, I think. So think about what needs to be in all data contracts. You probably have a schema definition, you probably have an owner, the kind of basics there. Probably every data contract has that. And then you can use that schema definition to do simple checks like Does this data match schema? Is it a string or a number? Maybe you can do more complex checks, like does it match a regular expression or is it an email address? Whatever you want to do there. But you can start off just have schema and types is the bare basic of a data contract.

And then the bit that kind of intrigued me was to go from being a series of data quality tests or schema validation tests to being a contract is invoking this idea of an agreement. If I think about it, I can have a data catalog, I can have metadata management, I can have schema definitions, I can have data quality tests, I can have schema enforcement tests on my side. But that's not a contract because it's a one-sided thing. There is no agreement from the person sending me that stuff that they're actually going to bind themselves to this contract. And for me, that was the slight difference in this pattern was, yes, there's lots of tools and techniques that we've had for years. But actually the push is to then have an agreement with the data provider, the data producer, as a true contract, a true agreement that we both abide by. Is that how you see it? Is that the thing that actually makes it different?

Yeah, exactly that. Because I think you described there, and a bit what we've been talking about so far as well, has been the kind of tech side. So you can have a data contract, you can do tests, you can do schema validation, you can do CI checks, you do all those kind of things. That's kind of tech side. But on its own, that doesn't solve the problems I want it to solve. So, I think it needs to be the people side as well. And that's where you start thinking about things like agreements and collaboration. The day contract is a great place to have that collaboration, to talk around, to have something you agree on, to define who owns it, define who's accountable and responsible for meeting that day contract. It's only partly about tech. In some ways, it's more about the people side and the agreement and what you're promising each other.

So it's that conversation. It's me saying, I think this is the agreement, do you agree? And then we kind of negotiate down to a contract that we both can live with versus me saying, and I'll keep using schema because it's a really simple one I can understand. Here's the schema, make sure you follow it. That's not a contract, that's not an agreement, that's not collaboration. That's me saying, this is what I want and I don't even care whether you think you can provide it, I'm telling you. So again, The idea of using a contract is we actually are then forced to get two-sided agreements between the people providing the data and the people collecting it. Is that how you see it?

Yeah, exactly. And that's a place where people sometimes go wrong with data contracts. They kind of forget about the agreement bit, the people side, and see it as, I'm giving you this contract, I'm going to force it on your data, and if you don't meet it, something will happen. And already you kind of come from a place of friction there, a place of confrontation there, which is not what you want. Important if you are trying to move to a more collaborative model where everyone agrees that it's worth doing this data contract because by putting a bit more effort in how data is generated and a bit more structure around that, a bit of discipline around that, we're producing better outcomes for business because this data is ultimately used to drive revenue or meet some sort of strategic goal or drive some key process, internal process. So, it's about that communication and often Data teams don't get to communicate that much with the people generating data. In a lot of organizations, that would be software engineering teams. There's reasons for that, like, organization structures and Conway's Law and all those kind of things. But data contracts is a way to try and bring them closer together and try and break down those barriers and try to have some communication. And then once you've had agreement, you can codify in a data contract and use that in your tooling and do the enforcement there. But that's not where you start. You start with agreement. You start with conversation first.

As technologists and data people, we love to codify technology first, people second, if at all. So I read somewhere that actually one of the things you should attempt to do is have the data publisher create the contract and then you negotiate with them to make it as fit for purpose for you as you can versus the data team creating it and then trying to have that conversation. Is that what you see? As you see by asking them to create the contract of what they think they're going to provide. Is a better way of starting that conversation than going to them with some predefined definitions?

Yeah, I think so. I strongly believe that the contract has to be owned by the data generator because ultimately they are the only people who can meet that contract. They're the only people who have the ability to change the data, to change the schemas. They're the only people who have all the context around the data they're creating. They're the best placed people to own the data contract. If I go to someone else and say, this is my contract on putting your data, and they say, well, we can't meet that, we can't meet the SLOs you want, we just haven't got the architecture to do that, or we haven't got the time to do that, whatever it might be, then what's going to happen next? Not a lot. Whereas again, it should be more of a conversation. We need this data, these are our requirements, and they say, well, we can meet some of those requirements now, maybe some later, but let's codify what we can meet now. And for me as a consumer of data, at least I know what to expect now and I can build about confidence. I know it's not going to change overnight. I know maybe not as high as I want it to be, but it's high enough and I can work with that. But yeah, I can start building it with confidence. I know what to expect. The only people who can change the data are the data producers. They must own the data contract.

How does that work? If you look at the example that you've got, your company was creating the software themselves. So you had software engineers that were building the core systems that you were getting from. And we know that often people building those software systems, create the data schemas and the data fit for purpose for capturing that data, transacting that data, not for making it fit for purpose for answering business questions. And we end up being data teams that deal with what I call exhaust. They kind of spew out this data or we go and grab it without their permission using CDC. So, we get this kind of exhaustive data and their focus is creating new features, creating value, getting a product, getting product market fit, making their customers happy. and often their focus is not making sure that the data is fit for purpose for the bit of work we do. So, how did that work for you? Was it just the fact that you were in an organization where somebody said, I'm going to bang their heads and make sure the software engineers and the data engineers work together? How did you deal with that political part of the organization and that problem?

Yeah, there's a few things to that. I think one thing we did is we didn't have with a CTO or someone important banging heads, doing that kind of top-down approach from the start. We started more bottom-up. That's partly because of where I was in New York, you know, I was leading the data platform team. I didn't have the kind of authority to start banging heads. But also, I think it worked out better that way because you could start proving value and do that quickly. And then once you've proven value, then you can start thinking about larger rollouts and larger deployments. But you can start small and really focus on solving a particular business problem. And once you do that yourself, next one, next one, and then gradually you're building some momentum there. And then you can start making a more bigger program of work to maybe move over to contracts if that's what you want to do. So I think it's best to start small, start focused on the value you want to deliver for a particular use case. So say for example, the team somewhere building this data application that is going to be revenue generating in some way, you're going to sell it to clients, or being part of some sort of application that makes revenue. You're trying to make sure that's a success. And the problem they're having is they The data's maybe not quite quality, or it's not fair, or it's not got very good SLOs, or it keeps changing, or there's some kind of problem there you want to fix. And the data's being produced by a software engineering team, in this example. So we know the problem we want to solve, and we know it's valuable for the company, and we know that we want the software engineering team to do some work there, and we know ultimately they want to own the data contract. So start off by talking, get everyone in the room, discussing problems, get everyone on board, and then start thinking about solutions. If I want the data contract to be owned by the software engineering team, I need to make it as easy as I can for them to do that. So maybe they can define it in code, maybe in the same repository as their, the other code that's actually generating data, maybe alongside how they also generate APIs and define APIs, or define their infrastructure as code, or whatever it might be. But trying to think about how do you make it work for them, because we want them to use it, and want that to be easy to use. As easy as creating an API, or as easy as spinning up a database in your cloud using your infrastructure as code platform. So really focusing on that experience for them, because we want them to do that work. And we know that they're busy and they've got other things to do. So we want to make it easy, but also they understand the value of doing it. So explain the value as well. That's where we started really. And we kind of got them involved as well in the solution, because software engineers love solving problems as well. If they feel like they're part of producing the solution, they're more likely to use it, more likely to talk about it when you're not in the room, more likely to get other software engineers involved, kind of becoming your champions. For want of a better word. You start there, like kind of MVP, you start building the small amount of tooling you need to enable the software engineers to produce the data they need, solve that problem you had of producing the data you need for that revenue-driving application. And then you prove it there, go on to the next problem, next problem, next problem, and you start gradually rolling out Data Contract to your organization while delivering value along the way.

One of the things I've seen is when you start talking to the software engineers about what you're using that data for and you show them, they then understand the value of helping you get that data in the right way because they actually understand the pain and they're engineers, right? They know how hard it is. So they're like, yeah, I get you. I understand what you're living with now. But they also see the value to the organization that the data is delivering. And so they're much more open now because they see that whole value stream. Let's carry on that theme though of, okay, so you were an organization where the software engineers were building the system that had the data that you had the contract for. If we're working in an organization that's using out-of-the-box software as a service platforms, and they may or may not have an API. So let's just say they do have an API. There's no real way though to negotiate data contracts with that vendor because they've got lots of customers. So if you're walking into an organization that is primarily using out-of-the-box software as a service applications, I'm guessing it's going to be a lot harder to implement a two-sided data contract pattern.

That is a lot harder. In those cases, you really just have to go as left as you can and put the data contract as close as you can to the source, particularly if you're dealing with a big vendor. So say you've got Salesforce data coming in, you're probably not going to get Salesforce to change how they're giving you data. Maybe using a 5-channel stitch or something like that to bring data through their APIs, but that's kind of the best you can hope for. But what you can do is if you have a Salesforce admin, they can own the data contract and they can be the first person who is responsible for the data from Salesforce because they have a bit of relationship with Salesforce. In this example, it could be Salesforce, it could be someone else, but Salesforce example, they have that relationship. They also, as admins, they can change things in Salesforce. Salesforce is quite customizable, so a lot of the problems come from custom fields and changing fields, which is directly caused by them, not by Salesforce changing their API. Salesforce API is quite stable, it's just that it's customizations people put in Salesforce, or B, the API, while stable, it's not really made for extracting data. So there's kind of problems there which we can't really solve. But the best you can do there is have the admin, in this case Salesforce admin, or whoever owns the vendor relationship, have them own the day contract. If it's a smaller vendor, maybe you have got a bit more leverage there and you can have a bit more conversation and say, hey, these are problems we're having. We love using your product, but the data we're getting from it is not good enough for these use cases. How can you provide that as better data? Maybe eventually that could be part of your actual legal contracting. It says there's something about the data quality and SLAs and how you won't breach it and what response time will be if you do. Those kind of things can eventually become part of a contract with a smaller vendor where you have a bit more leverage, a bit of a better relationship. In both cases, really, as left as you can go, ideally to the person that owns that vendor relationship.

You mentioned that problem you struck was you were CDCing off the database, and I've often heard people describe that as taking data without permission, because, you know, I've seen it in especially large enterprises. It's like, okay, nobody's talking to us. We'll just get the permission to turn the CDC on and we'll just grab it all and deal with it. Then you move to API or event-based as a way of getting that data. So does that mean that actually you can't use a data contract pattern if you're using CDC off a database? Do you have to have API and events to make that pattern valuable? Or could you actually use it if you were using a change data capture or a log mining type of pattern to collect that data.

So we have looked quite hard at how we can apply data contracts to change data capture data, but we never really found a solution that works, really meets the goals we have with data contracts, where we have something that's more stable, we have something that's being produced at maybe a bit higher level quality. So not just structuring the database schema, we are saying these are events, or these are bits of structural data that make sense for business, not that makes sense for application upstream, which is a different thing. Often. You never really find anything that works. Because say, for example, you did try to apply data contract to CDC data and they want to change their schema. Now suddenly the software engineering teams can't change their schema with autonomy. They, they're kind of stuck waiting for review by some data team, or their schemas have to stay stuck in stone. And that's no good for them. They need to be able to change schemas to add new features, to improve performance, do whatever they need to do. That schema needs to change quite regularly, particularly in more tech or any companies where they've got good software engineering practices, they should be changing quite frequently. So we don't want to prevent those from changing. So you need some kind of abstraction from that database schema, something else, which is the interface, which is the API. Again, like I said earlier, like, software engineers would never build on top of someone else's database, and they wouldn't let another software engineering team do that. So why is it okay for data teams to do that? The problems are exactly the same, and the solution really is exactly the same, is to use some kind of abstraction, some kind of interface. Which for data is data contracts and for APIs, it's APIs.

What that means though is that there's more design, more planning, more requirements, more understanding upfront. And then also the need to be able to adapt constantly. If I give you an example, if I use a CDC tool against the database and I bring all the data into late persistent staging, bronze, history, whatever layer you want to call it, that first raw layer where we get all that change record. I know I'm getting all the data, which means I can delay a whole lot of decisions around data design, around historization of that data, because I know I've got it all, and I can now grab it from that layer whenever I need to without talking to anybody else. Whereas if we take a data contract approach, I've got to actually define the events I need in conjunction with the people that are generating those events. I've got to agree what those events look like, and when I go, oh, shit, you know, for the next information product, I need data from another event. I don't want to have to wait 3 months for the software engineering team to get around to creating that event stream for me and the contract. In that scenario, it sounds like the perfect organizations for this are ones where their software engineering team are used to rapid releases, small chunks, so that I can say, Hey, we've got this new information product coming up. We've already got the event for customer orders product. This one's all around returns. So now I need an event for customer returns product. And then they can basically implement that, that event, that API and the contract agreement in a day or two rather than 3 months. So that seems to be one of the things that would make the pattern more valuable, easier to implement. Would that be true?

Well, I think it's a trade-off, and what you're saying is we're going to do a bit more work up front, and that means we have something more stable we can build on because we're doing something important. And if you're doing something important, really you want that to be stable. You want to be able to ship data, this process, this application. You want to be able to ship it and for it to be reliable. You don't have to keep fixing it every week. So you are giving up a bit of flexibility, but in exchange you're getting more stability. And maybe It was better to prioritize flexibility when all you're doing is reporting. I don't know if that was true really, because you give those revenue reports to your board and they're wrong and they change from last time. That's not a good situation to be in. And the reasons they're very wrong is because this work you're doing from your Bronze Layer or whatever into something more structured, you think you have all flexibility, but really you're having to do a lot of logic there that's very brittle. You're implementing a lot of logic that's already in the upstream service. When that upstream service changes, you have to redo your logic. They fall out of sync, you don't realize it. All the problems that probably everyone listening here understands, right? We've had these problems for years, decades maybe, in data. You could argue that giving up that flexibility, but I'm not sure that's a good idea anyway. The other thing I would say as well is the assumption you're making there is that data changes regularly and that, I mean, I'm not gonna have to find anything upfront, in a couple weeks' time I'm gonna realize I need something else and I don't have to wait another 3 months to get that. But again, in practice, it doesn't change very often from what I see. So for example, I've been working at a payments company for 7 years. That's changed a lot. It's sort of gone from a scale-up and grown very fast. And we've expanded different markets, we've got different products, it's changed a lot. But our core data models around payments are very similar to what they were when I joined. And it's around what bank accounts they go from and to, and how much was it. It's got a few extra fields around exchange rate used and things like that, because we're taking money across borders, but it hasn't changed that much. It changes when new features are added, and as part of that project, adding a new feature, you say we also need to update the data model for that, and that's just part of the requirements. But otherwise, it doesn't change that much. So I think this assumption that data changes a lot, and that we have to react to it, and we have to be agile, and to— or, not sure agile is the right word, but we have to react to it as best we can. I think that's another assumption that we can challenge there. And it's not that data contracts can't evolve, they can evolve, but you're doing that more deliberately. You're not finding out the next day that something's changed, and that field you were using before is now full of nulls because I'm using new field, those kind of changes you don't need to worry about if you're using some sort of abstraction at a higher level.

From a technology point of view, it seems that there's a bit of a race to create the default de facto standard definition of what should be in the contract definition. I think there's probably, I've never bothered to look, but 2 or 3 open standards now, which look the same, but don't. Is that what you're seeing, that actually there is this move to trying to agree a minimum viable viable contract definition for, from a technology point of view, not, not the technology that's going to validate it, but the definition of what should be in a contract, standard for 10 kind of thing. Is that what you're seeing?

Yeah, I'm kind of involved with the Open Data Contract Standards. I'm part of the technical steering committee there. And that's got quite a few people who have also published open standards and trying to work out how we can just have one that's backed by the Linux Foundation. So it should be open. It's got some good governance around that. I can see the attraction for a standard. It'd be great if we could Define a data contract in a standard format. And then for free, I get some tooling that anonymizes my data based on the categorizations I put in there, or some tooling that populates a data catalog or whatever it might be. It'd be great to just get that for free if you all use a standard that's an interchange format between vendors, between suppliers, between tooling. So I think there's the benefits if one of them becomes popular is it's potentially very valuable. This particular standard, there's quite a few people involved with it. There are people publishing their own standards. In a way, it doesn't matter too much who wins as long as we, as long as we have one. But I think the Open Data Contract Standards, which I'm involved in, got quite good traction. There's a lot of people involved, people from vendors, people from other standards, people from industry like myself. That's got a good chance of succeeding, but it's still, it's still quite early days. It still needs a lot more adoption before it can be really useful.

I'm a great fan of a standard. SQL is a great standard. It's been adopted by lots of people. What tends to happen is vendors then want their version of the standard, right, to differentiate themselves. And as a startup, if we could have a standard that we just use and we know we get interoperability with anybody else in the ecosystem, then it's valuable to our customers and therefore it's valuable to us. I think the main point from my view is if you think you're going to go into the space and look at start implementing the data contract pattern, go and have a look at that open standard that's under Apache and grab it as your starter of 10. Don't reinvent the wheel. And then if you need to change it or augment it, do so, and then try and push it back about why, why the bit you wanted is missing.

In the meantime, what we end up doing is we did end up writing quite a lot of conversions from our, that contract to another structure. And that's not so bad, it's not that hard to write. But for example, define that contract in a certain format, we convert it to Protobuf, so we can then use that in, we use Google Cloud, so have PubSub schemas, because you'll see Kafka, and you can do some schema validation there. We convert it to JSON API to create some software libraries that we can give to our software engineering team to make and do some validations there that match the date contract validations. So that might be things like email addresses or a regular expression or something like that. We do find ourselves quite often converting the date contracts into different formats, and some proprietary formats as well. We use date contracts to provision a BigQuery table, and that's got its own custom JSON structure that defines a BigQuery table, so we convert it to that. And it's not hard to write these things, it's quite simple. Imagine if we didn't have to do that, we could just convert it to one format and get integrated with any of these kinds of tooling we want, that'd be a great situation to be in.

How often does that happen? How often would you see the data team writing the libraries or the API tooling and giving it to the software engineering teams to say, hey, use this, it'll make your life easier, and therefore, you'll create this contract that we can then agree to? Is that a better way of bringing the software engineers on the journey rather than saying to them, I need you to agree to this contract together, and then I need you to create some tooling where you can validate on your side, and then I'm just going to sit back and wait for this beautiful stream of gorgeous data to come and not have to worry. Do you find that most data teams now to be successful are trying to help the software teams with the tooling around this?

I think so. And I think often you have a, or more and more data teams have a data platform team. They may have a bit more of a sort of DevOps or software engineering background, as well as knowing about data. It's kind of my background, and I led the data platform team. And what they're doing is they are providing platform capabilities capabilities to maybe other data teams who then do transformations or whatever else they're doing on the data side. They provide us with orchestration software and orchestration tooling. VMware are also providing tooling for software engineers to produce data through data contracts. And they're providing other tooling around the data contract, for example, the data retention kind of style tooling, backups, access controls, all those kind of things. And all of that tooling we've been building on data contracts. So we really built built a whole data platform around data contracts. I mentioned at the start, data contracts are a simple idea. You just describe your data in a format that's sort of human and machine readable. It just unlocks so much that you can do with it. And we found that any platform capability we wanted to add, we can add on top of data contracts. So simple things like backups, more complicated things like performing GDPR deletions and data retention and role-based access control. All those kind of things, we build on data contracts because the data contract contains the information we need in a machine-readable format. So it says, is it personal data or not? Is it about a customer or about an employee or whatever it might be? How long should we retain it for? What's the backup policy should be? What SLO should it have? It describes those kind of things. And then the tooling doesn't really care what the data itself looks like. It could be very wide schema, it could be very deep, doesn't really matter. It could be one field, it could be a million fields, it doesn't really matter at all. We can easily build tooling that can use the date contract to apply policies, to do something with that data. Over the last 6 years, any platform capability we've wanted to add, we've built on date contracts, and we haven't found anything we can't build on top of date contracts on platform layer.

Okay, but I think you just switched patterns here and used the same term for it. So let me explain why I say that. So in our product, we are what we call config-driven. I used to use the word active metadata, but then I saw how everybody else defined it and went, nah, shit, that's not what we do. So the way I describe it is everything we do is stored in config, and then the config is used to generate anything else we need. So if we need code to transform data, config's read, the code's generated, it's executed, and then we dispose of the code effectively. If I want a security policy, you know, the policy's in config. When something interacts with that data, the policy's applied. So this idea of a, a set of rules is how I think about it, a set of rules that can then be used in various different ways to do various different actions. And I think that's what you just described. You took me from a data contract as being human and machine readable definition, an agreement between the data providers and the data collector or consumer to make sure it's fit for purpose. And you've extended that config pattern out now into various other things where it was highly useful. Is that what you've done? Is that the jump I think I saw you just do?

It's kind of, it's kind of a journey we've been on with data contracts. Our initial thing was, we need something we can use to have people collaborate a lot more, and also to move us away from sort of CDC and becoming a source of data to something where we create an interface. But once you define that, you think, well, how do we make it easy to create this interface?

Interface.

So then you start thinking, well, let's integrate into a platform and have a platform capability. And then you're thinking, well, now we're asking software engineers to own this data, but I don't want to learn about all the privacy regulation, and I don't want to know how best to back up BigQuery data. I just want that to work. It should just happen. Like your governance policies, they should just be automated as part of platform. So we use that as part of platform as well. It's kind of comfy, you know, -driven. I'll say it's more, they describe the intention, it's more intention-driven. For example, this is the data, this is what it looks like, it shouldn't cross borders or shouldn't be visible to people outside of Europe, for example. I don't really care how that happens. They're not saying, okay, spin up a BigQuery table in this particular geo and put these extra controls around it and put these backups. Not deciding all of that, but not defining all of that in a massive infrastructure as code, like maybe Terraform config or something like that. They're telling us the, they're describing what they have, and we are applying policies to that based on what they tell us, and based on their internal policies. So there's a slight difference there, I think, because I think it's, I see this in platform engineering as well, in software platform engineering, moving away from this idea of having software engineers define every single thing about the platform that they need to make happen, by defining all these Kubernetes things they're deploying, and all the cloud stuff they need, to saying, I just want a database. And I want it Postgres, I want version 15. And then you get all the stuff for free. You get all the backups, you get, get the access control, all that just free. You don't really care about that. You don't have to think about that. You're focusing on, I'm using this database to run my application. I don't want to care about all the other stuff. Make that happen for me. And the same for data and data contract in your data warehouse that you're owning. I don't care about all the privacy stuff. I do care maybe, but I don't have to think about how best to apply those policies to my data. I want that just to be taken care of. It's kind of two patterns. They can't go hand in hand though. You can't ask people to take more responsibility for data and then expect them to learn about all the stuff it takes to look after data and be compliant with your governance. You have to give them the tools as well to do that.

Let's just break that pattern down. So let's use an example. So we're extending away from data contracts being agreements between two parties of the structure format quality, SLA, frequency of that data. And we're extending the agreement and code out to a whole lot more use cases. And so one of the use cases that I see massive value for is around this access to PII data. Tell me if I got this wrong, but you know, so we have this policy effectively that says, let's take a simple example. This part of the organization can see people's first names, last names. This part of the organization, really, really simple.

Rule.

How do we do it in the old days? We'd go and write some views, or we'd put some row-level filtering on with a WHERE clause, or holy shit, we'd hold two tables, one with the name and one without the name, depending on what technology we got and how many times we've done it. What do we really want? Well, what we really want is somebody to be able to apply a tag to that first name column and that last name column, and then the policy to be inherited regardless of who's feeding it, who's reading it. That policy is now enforced as code. But as the provider, all I gotta do is flag it PII. That's it. And then everything's taken care of me because that's a simple contract for me to adhere to. I know that that data is PII, so I'm just gonna tell you, and then the system should take care of it. Is that what you're saying? Or is that them just extending that out to something like backups? I can see then how you can, you can take that that policy-based pattern and apply it in many different places over and above providing of the data, the application of security, you could then apply it to many different things. Is that where you got to?

Yeah, exactly that. And it goes back into saying how the data generator has to own the data contract because they know most about the data, not someone centrally, maybe some sort of data governance role or some sort of data stewardy kind of role. They don't have the same context of data as the person generating data. So they are best placed to categorize data and say, well, this is going to be personal data, or this is going to be a bank account, or this is going to be something sensitive. But they are not the best place to decide exactly how the company should be handling that data. That's probably best decided centrally and implemented as rules in the platform so that it's just everyone who is using this platform just get the, the policies applied for free in the right way that matches the company's central policies.

So why wouldn't we just get them to define all the transformations we need for data as a contract, and then have the system automatically generate it? Why would we just push all the work back to the software engineering teams?

Well, that's a really good question, because I've been thinking a little bit the same way. Not completely get rid of data engineering, that's not what I want to do necessarily. But certainly, when you've moved away from building on top of database schemas, the upstream database schemas, and you're moving, you're building on top of this abstraction interface, which can look completely different to the database and can look hopefully more useful to people in business. It's a usable data product, for lack of a better term. It's a usable data product that is well documented. It uses the terms the business uses, not the software, not the product. And it's usable, and it's hopefully, because it's a better quality dataset, you can hopefully do fewer transformations on it, which I think is great because you're saving a lot of time, a lot of money. It's quite expensive to do transformations on data warehouse typically, but also that logic there I mentioned earlier, that logic is quite brittle. You're putting that in data warehouse, you don't really know how the upstream application might change over time and it's going to fall out of sync. You can't maintain that logic in two different places. Now it's in only one place in the upstream application. You should kind of shift that left. So then you start thinking, well, what kind of logic do we need to do in data engineering? And maybe it's where you're joining these data products to create something new. So maybe you're joining a few data contracts together to create something a bit different that's still useful, maybe. I would like to think in a few years' time, we won't be building these complex, expensive data pipelines that have lots of business logic in it, because through this change, we are producing better quality data at source. I think that's going to be a better outcome for business and for data engineering as well.

So this thing takes us down to the struggle that my data warehouse brethren have. So those of us that have done this for a few decades, We know that historization can be automated. We know that immutable data can be done. It's the combining of the data together is the thing that always takes a lot of time. And this is one of the areas that when data mesh came out, it never answered that question. In my view, that was because it was written by somebody from a software background, not a data background, who didn't quite have the war wounds of combining data. Let's use a scenario, I've got got an organization that's merged and now I've got Salesforce with customer and I've got HubSpot with customer. And let's just say both of those were managed by a software engineering team, but I've still got two definitions of customer. And at some stage I've got to put those together and team Salesforce ain't doing it and team HubSpot ain't doing it. So that's where the data team always came in to do that work. Initial definition of a data contract, provider and receiver. Agree that the construct, have an agreement around it, then potentially we can make the, the providers give us the same key, the same ID for the same customer potentially, and our job goes away. 9 times out of 10, that never happens. So apart from that, have you seen any other ways that this pattern of contract or policy can help us in terms of combining that data from disparate systems?

I've seen one other pattern that's quite interesting from another organization building implant date contracts. It's a bit different because in your example you get some more third-party data. So again, you kind of got problems there. There's only so much you can do, and probably in that case the best you can do is have your Salesforce admin make sure that they give you a particular key that you can use and that's not going to change, and your HubSpot admin gives you that key that's not going to change, and then together you can join it. And that's still probably a data engineering task. Probably not an overly complex pipeline, hopefully, but then your data engineering team is providing that as a product for the rest of the business, saying, "This is customers from our CRMs." And the rest of the business doesn't care that we've got 2 CRMs, or however many CRMs you've got. So that's, I think that's still viable. That's why I don't think data engineering is going to disappear at all. But I have seen interesting patterns where, I guess more where you've got a software engineering team, and it's like, well, can we provide libraries that ensure that the keys are always correct at the time the data is generated, so at the source. So what I do is I write a small software library that if I'm producing customer data, it's always going to have a customer ID in this format. If not, it will, it will throw an error and the software engineer needs to fix that error. You could potentially take it further and have that library interact with some sort of central ID service and get IDs from there if you want to have a centralized ID generation service. It's where I think I'm going in future, particularly as I grow. It's quite a small company at the moment, so I haven't got the problems of loads of different systems, but you can imagine one day with mergers and stuff, they might have that, and that's when they might start thinking about centralised services. But you hide that from the data producer. The data producer is saying, I'm providing customer data, it looks like this, and these are the measures and attributes and fields about it. And Phil Libby takes care of making sure the ID is in the right format that we can then use it to join data wherever it is. So I mean, not doing that as a sort of data engineering sort of MFT kind of style process, you're, you're doing that out of source as far upstream as you can. That's quite an interesting pattern, I think. It'd be interesting to see how that develops over time.

In that scenario of Salesforce and HubSpot where we don't control it, at least we know we have no agreement. If we say, ideally we want a single key that's unique and consistent across every system, and the Salesforce admin goes, I can't guarantee that, and the HubSpot admin goes, I can't guarantee that, at least you know where the shit's coming from, you know, to deal with it. If you've got software engineers are building the systems that you need the data from, then it becomes a political conversation about why can't they give you consistent shared keys? Because we're in control of destiny. So why is the organization letting this complexity come in? Why can't they actually have an agreement with us to do something? And then by showing them what we do with it, then they can go, oh, actually we get it. We wouldn't want to do all that moogiematchy stuff that you're doing. Because like you say, software engineers, like data engineers, love to solve problems. Think about data transformations. What do I do? I take something, I change it, and I put it somewhere. I might change the structure. I might change the values. I might create table as. I'm still putting it somewhere, even though it's temporary. I might loopy-loopy that a couple of times in memory before I write it out, but I'm doing a bunch of what I call nodes and links. And on the other podcast, we had a guest that came on this week, and it was all about system design for organizations. So again, this idea of nodes and links, a circle that does something and then a line to the next circle that does something. If you think think of a DAG, think of an Airflow, think of all those nodes and link paradigms. And what they said is when you go into an organization for the first time, ask a person, who do they rely on and when? And who relies on them and when? So this idea of handoffs. If we think about data transformation, same problem, nodes and links. Why wouldn't we implement contracts between every node? So if I write a piece of code, why wouldn't I write a contract so that the next piece of code that reads the output has certainty and agreement of what's in there and will know when it's not being met or when it's been changed?

That's a good question. I mean, you use the word handles there, and I think that's kind of how I see where data contracts have most value, is when you hand it off to a different owner, maybe a different team or different process, to have boundaries of ownership. Is where it's most useful to have DateContracts. With external in a transformation pipeline, you could use DateContracts there. We kind of do this as well. It's kind of interesting because it wasn't how I expected it to be used, but because we have built so much around DateContracts, it's a bit like Describe. Why wouldn't you use it for internal things? You get all this stuff for free, which makes the next bit of code easier to write. We're seeing we're doing that. It's not quite how I imagined it from the start. I always imagined it being at this boundary of ownership, and that's that's where it's most useful. But because we built it as part of our platform capability, it then becomes a nice way to automate even internal transformations. So I think, yeah, when you have built these capabilities and you've built them well and they're easy to use and give a lot of value, without— I had a bit of friction. You are defining a data contract, you are putting some restrictions around how you might have to change that in terms of maybe you won't, maybe you have to make breaking changes on new version, things like that. So you are, you're paying a bit for the contract. In terms of like the friction, you'll put a bit more effort in, in the creation of that. But if your game is software-free, then maybe it does make sense to use it even for internal transformations or moving data from one place to another, not necessarily to another team, but it's still moving data around. It's viable to have some things you get for free with DataContract as part of a transformation. You could see that in future. Who knows how far this can go really in terms of automating transformations to a degree where Maybe you don't have to write loads of SQL in DBTO. Maybe there's another way of doing that in future that's built around data contracts, more describing what you want to happen rather than detailing how you want to do it in thousands of lines of SQL.

And the reason I brought it up is it's something that's annoying the hell out of me right now. Even in our product where we've broken it down into the smallest chunks of nodes and links as possible, you still have to take a series of steps to get to that metric or that value or the data for machine learning model, whatever the valuable information product that you need to produce this, or data product if all you're doing is producing data. And then what happens is you make a change, and even though you can see the lineage, you know where it flows, you can trace it, you can track it, all that good stuff, it's still not telling me exactly when a contract's been broken. Now, if I have tests or what we call trust rules, it tells me when I broke it, but it doesn't tell me before it breaks. And so that's where I keep thinking, actually, this idea of a contract, a policy agreement, to say between this bit and this bit, the agreement's been broken for whatever reason. So let's go back to that and let's go back to that data collection problem, the original use case of a data provider and we're the data receiver. I see in the data quality world, and especially in the event streaming world, this idea of dead letter queues. And in the old days, we used to call them error tables. The problem I see with that pattern is, okay, I get some data coming that doesn't meet the contract, or I get some data coming in that doesn't meet the schema, or I get some data that comes in, doesn't meet the test, which whatever pattern you're using, and then I stick it somewhere else, and I'm going to deal with it later. But that gives me a whole lot of problems. Well, how do I insert it where it was meant to be? How do I tell the person at the end who's consuming the information product that of the million records we got today, 5,000 of them didn't turn up because they were wrong? There's a whole lot of problems we get when we dead letter queue. And so in my head, if we have a data contract, it's immutable. If the agreement's been broken, everything stops. Is that how you see it? Is that how you implemented it?

So we kind of support both. We use the dead letter queue quite heavily. I think it really depends on, on what your users expect, or rather what you define in the data contract. So what they know to expect. So it's about expectations again. If as a user, you'd rather not see any data if it's not all there, you care most about completeness, then maybe you want that, oh, this data is not meeting the contract, let's just pause things and we'll fix it rather than provide partial data rather than provide no data. Or you might say, actually, it's more important if some of this data gets through, we can then use it for a process that might, I don't know, detect some fraud or something like that. It's better to detect some fraud than no fraud just because some of the values are wrong. So actually, we might then let it queue it. We'll fix some, we'll be paged maybe if everyone calls for it, and we'll go fix it. It might be a couple hours, but eventually it'll come through. But in the meantime, all the other data is going through, and then you are Catchments Board, for example. So it really depends on requirements of a user, I think.

What you're saying is I have to understand the information product that's going to be delivered and how that user or that system is going to use that information product. So the action it's going to take and the outcome it's going to achieve before I can actually define the contract, because that would affect the context or what's actually in the contract.

Exactly. You're providing this data for use case, right? To meet some user's requirements. Maybe you've got many users and you have to sort of compromise and trade off some pragmatic. You can't make more, That's fine, but at least you know what they want and you're trying to provide them what they, what they need. Then once you've agreed on that, maybe making trade-offs here and there, but there's some agreement there, you then codify that in a contract and that's what people can expect from it. Because they know what to expect, they can use it with confidence. It might not be exactly what I want. In a perfect world, I want it to be 100% reliable and I want someone on call at the weekends in case it goes wrong, but maybe that's not feasible for whatever reason. But at least they know now and they can build their own processes around that. So yeah, back to that people side again, back to that conversation and that agreement side.

In theory, every one of our dashboards should have a codified data contract to bound to it.

I think so. I think they should. I think if dashboard is important enough, or if your data contract is easy enough that you should use it anyway, then why would you not use it? I see this sometimes. I was using a dashboard a few weeks back and it was about cost information on our cloud. What I was trying to do is to like work out One of my kind of hats I wear at the moment is around sort of FinOps and trying to reduce our cloud costs, which is a lot of people doing that at the moment in various companies for various reasons. I look at the dashboard, I don't know if the data is correct, I'm not sure I trust it, I don't really understand what this field means, and I don't know who to talk to, I don't really know who created this, and I'm sure it looked different last week. I couldn't use that data to make decisions about whether to start investigating a particular service, has that service increased in cost or not? I'm not really sure what's happened here. Maybe it's not the most important dashboard in the company. It's not revenue. It's not some kind of process that's fraud detecting or something like that. It's not that important compared to the, in the grand scheme of things. But also it wasn't very useful to me. It was, I couldn't use it. I ended up wasting a lot of time trying to work out how I can use this. I ended up going to a different source completely to find the data I needed. It actually turned out to be Google Cloud Console and then just less good data, but I knew I had more confidence in it because it's better documented. I knew where it came from. Even for the most simple dashboards, I think it's valuable to have a data contract around it if you want it to be used. Otherwise, why would someone want to use it?

I haven't caught up on where the whole reverse ETL went, but this idea of activating data in other systems where we, we're actually pushing data records into, back to the system of capture or system of control because it's valuable. So rather than give it to humans who can take the action, give it to the system so it can automate that action. Having this idea of a codified agreement between those two. It just makes sense. It's what software engineers would do. But as data people started taking over that integration, we seemed to lose that rigor. We would just treat it as an ETL task, not a systems integration task.

Yeah, I think it's a bit of a maturity curve we're going up, I think. And we're starting to apply more discipline to data engineering as a whole. And that's been going on for a few years now. We're using version control a lot more than we used to. We're doing proper pull requests. We're doing we've got nice patterns for deploying things, maybe to make sure things don't break, and can roll back quickly. We're using instant management processes. A lot of this, and all this comes from software engineering really, which itself is also maturing in the same sort of way, and people are moving towards more strongly typed languages and things like that. So it's a little maturity curve we're going on, I think, and I think it's good. I think we're doing that because what we're producing now is of more value than it was before, and that's that could be various things. I was, it could be AI, it could be, it doesn't really matter what it is, but what we're producing now is more valuable than it was before. And therefore we need to apply more discipline, more rigor to that production of that value.

I think the key takeaway is my perception that data contracts were solely between the data producer and the data receiver. And that always indicated the left-hand side of the value stream, the data factory. It was always the data coming into the platform. That one's not quite true. It's a core pattern of, here's an agreement of what it's going to look like that two sides have agreed to because there is a handoff. It may be a handoff between teams, it may be a handoff between systems, but I can't see any reason why it's not a handoff between two blobs of code. As long as there's an expectation that when it turns up, it's going to look like something and behave like something, and these other things have already going to be done so I can trust them, don't have to do it myself. Then it's valuable. I definitely think it's a good fit where this, for an organization where the software engineering team's in the organization, I think that's, uh, given the immaturity of the data contract patterns at the moment, and the technologies around it, I think those organizations are going to have a better chance of implementing it well. If you've got 55 out-of-the-box systems and no control over them, then adopt the pattern, but it's going to be slightly harder, I think, to lead that path. I still think it's one of the few terms that's come up in the last couple of years that's actually got value and going to survive, in my opinion. Before we close it out and see where people can find you and learn more about this, anything else you want to cover around data contracts? Anything I kind of haven't asked about or forgot to ask, or you go, hey, this is the bit you really need?

No, I think we covered a lot of detail in this, and I think it's been a great discussion. I think what I would say is a lot of times I feel like when I'm talking about day contracts, I'm talking about this perfect world we can get to. And of course it's a journey. It's a journey that involves a bit of tech and a lot of communication, a lot of people side. And that's, that's probably the harder part, but it's the necessary part. And there's always going to be trade-offs. You've got to be pragmatic with this. Don't expect to be able to go to day contracts overnight and go from CTCs to full day contracts in it's a journey going on, but it's fine if the journey takes some time because you're starting to embed that into your culture. And that's what we're seeing. So, I was in a meeting with some software engineers a few weeks back. They were just talking about, like, how do you make data available, and they were— they came up with a term that I didn't even use myself. I described it better than I could. They were like, "Oh, so I see what we're doing here. We are moving from more passive data generation to more active data generation. We're actively providing data to you because you want— you need to use it downstream for these reasons." I was like, yes, It's exactly that. And this is someone I've never spoke to before, never spoke to their team. He's quite a new hire, but it's already part of our culture where he kind of knew that and explained it better than I did in a way, because it's part of our culture now, because we've been doing it for 6 years and we've got tooling there. It's just the way we do data now. And we still have CDC in the background and we're still trying to move things over. I mean, not perfect, but it's a journey. If you put hard work into that journey, you'll find it becomes part of your culture, and then you just carry on that journey and you'll get there eventually.

It's actually a good point. For us, I talk about 4 ways we get data: push, pull, stream, and drop. Our customers sometimes push the data to us. Sometimes we go and pull it, yeah, from in the environment or from their SaaS platform or whatever it is. Every now and again, we stream. The data will be streamed to us, which is a form of push. It's just more recent or more frequent. And then drop is a file drop or a file upload. And what we know is, we know file upload is the worst for us. Highly flexible, really fast, easy to prototype or ideate off, but it's not repeatable. The contract is constantly broken. Somebody changes the CSV, they change the Excel spreadsheet every time. There is no contract and it's never honored. And I hadn't thought about this idea of when they push it to us, one of the benefits is we have an agreed contract to a degree. We know what the schema looks like. We know what the data looks like. We don't really talk about it yet as a contract and we don't agree some of those other things other than the schema, which is what we're starting to do now. When we pull it, we tend to treat it as a CDC problem. Even if we're hitting Salesforce APIs using Data DDO, it's still a pull, but we're still accountable for the contract. And it's a really interesting idea to say to the organization, actually, you need to be in that conversation. Because like you said, they can go and add a field to Salesforce that breaks the API contract that we're expecting without us knowing. And so even though it's well documented and lots of tools know how to use it, it still can break. I'm going to think about which side of the fence does the accountability sit for something to happen as part of that contract conversation, because that's the people part and process part that intrigues me. And then definitely looking at it internally is if I'm moving something and there's a dependency, why didn't it just become a contract? Because once I have the contract pattern, I can write lots of them quickly. They can be automated, they can be honored, they can be validated. I can just keep using that pattern time and time again. Because it has value to me. So if people wanna learn more around data contracts, if they wanna get hold of you, apart from coming to the Agile Data stand at Big Data London, stand Y760, top right when you walk in the door, apart from that, how else can people find you and find out more about this data contract stuff?

I have my book on data contracts I mentioned earlier, which will be given away at your stand at Big Data London. So looking forward to that. To find out more about me, if you go to my website, andrewjoshjones.com, I also have a white paper on date contracts called Date Contracts 101. It doesn't go into much detail as we've been in today, but more of a high level, say these are problems you want to solve and this is how date contracts can help you solve it. And to get that, you can go to dc101, so dc for date contracts, 101 with numbers,.io, and that will take you to that white paper and it's free, a free download from the website. So that's a good overview, I think, of date contracts and problems trying to solve, and not as much details we described here, but if you like more detail my website's a good place. And from there, I've got a daily newsletter about data contracts as well. I go into detail about how to write data contracts at your organizations.

Excellent. And you hang out on LinkedIn as well. Go find you there and sign up to your newsletter because you post every day on the newsletter. So lots of little contracts, which is good. Excellent. Thanks for coming on the show. It's changed my understanding of the data contract pattern quite a bit. Like I said, I was stuck in the land of sitting on the left-hand side of that data factory between the data producer and the data receiver has been the only place it played. So I'm going to go think about this one a bit more, which is all good. I hope everybody has a simply magical day. That, Data Magicians, was another Agile Data podcast. If you'd like to learn more on applying an Agile agile way of working to your data and analytics, head over to agiledata.io.