AgileData #54 - Data Contracts with Andrew Jones

Foreign.

Welcome to the Agile Data Podcast, where we talk about the merging of Agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Andrew Jones.

Hey, Andrew, thank you for coming on the show. The reason I've asked you to come on and graciously accepted is to talk about this thing called data contracts. The beginning of this year, I was trying to guess what terms in 2024 are going to end up being buzzwords and what terms I think are going to survive to 2025, because I think they actually have value. And data contracts is one of the few in my head that I said, I think they've got legs, I think they're valuable, I think it's a pack we should implement, and I think it's something that will survive to 2025 and 2026. But before we rip into what is a data contract and why should we care, why don't you give the audience a bit of a background about your other than where you've come from?

Yeah, sure. So I've been in industry for around 20 years now. I started off more as a software engineer and then moved towards building platforms, engineering platforms, infrastructure platforms, and then after that into data platforms, which is what I've been doing for the last 10 years or so. And it's been around six years since I started thinking about data contracts and using that to build data platforms.

Excellent. And it's rumored that you're the person that invented the term data contracts.

Yeah. So, yeah, it's around six years when I started thinking about what became data contracts. It's the same sort of time that Jamek released Data Mesh articles, and they're quite good inspiration as well. But I kind of didn't want to call it Data Mesh internally because that felt like a big thing. I was just leading a data platform team. I wasn't ready to change the organization in that sense. So I started thinking about. And we're going to what, Date contracts. Later I started thinking about what became Date Contract. So I need a name to give it internally. I was thinking it's a bit like an API. An API is like a contract between a provider and consumer. Maybe this is the same thing for data. May I call it Date Contract? And that's kind of supposed to be a temporary name, but kind of stuck internally. And as I started thinking right about it publicly, I started googling it, seeing if anyone else uses the same sort of term or use similar terms for so many things. But at the time I was just googling date contract. I was getting a 4G mobile date contract. I couldn't find any other literature around it, although I could see people doing similar things, but there wasn't really a name for it. So I started using the word date contract in my public writing as well. And that sort of has stuck now for the last few years and became, yeah, became a bit of pattern, a bit of category.

Now, in data, you've implemented data contracts in the company you work for and you've written a book about it. So it'd be fair to say that you have quite a bit of experience in this thing that is a data contract.

Been using it internally for about 6 years in production of company I work at speaking about it publicly for around four years and a book came out last year. So we've been doing it for a long time and we've done it very successfully also with other organizations I work with now as well.

Let's start with the basics. How would you describe what a data contract is?

So a data contract at its most simple is a very simple idea. It's something that describes your data, so hold your metadata and that's all it is. So it's quite a simple idea really. But it's the way you use that, that's what makes it powerful, where you use it to do whatever you like, really, whatever problem you're trying to solve for organization. So the problem I was trying to solve initially was we had upstream changes breaking downstream data pipelines and data applications. And that happened too often. And we were trying to use our data for more and more important things, things that drive revenue, things that are key processes shouldn't be breaking every couple of days. That was a problem I was trying to solve initially. And what I realized is that we were basically building on top of the upstream databases. We had to change that capture service chucked into a data warehouse. And to build on top of that, we're essentially building on top of a database, the upstream database. So when that changed, and it changed quite often as our software engineers developed features, it just broke things downstream. So that's the main problem I was trying to solve. So I thought, well, again, I spoke about, I've got a software engineering background. If I was a software engineer, I wouldn't build on top of some database. They wouldn't let me do that. We'd always have an API. I started thinking, I want the same thing for data, because data is as important as our software services. Why don't we have the same thing with data? Why not starting to get towards the solution there of like what became data contracts. I needed something to describe this interface, something that could then use to provision an interface. And that became the initial data contract. So essentially it was just a schema and a few other things in there that we then used to create the interfaces to move away from data capture. So that's the problem we try to solve. But it's a very simple idea. Data contractor, if you've got other problems trying to solve, they help by they can solve by describing your data better. For example, maybe you're trying to do access controls, automate that based on roles and you want to categorize the data. What's personal, what's not, what's confidential or secret, what's public. We do that as well in that data contract. And there's nothing put in there. So it really depends on what problem you're trying to solve for your organization. But the most common problem I say is around the change management of upstream data. And that is where we started with data contracts as well.

So I can see a lot of patterns in a data contract as you describe it. So the obvious one is a series of tests. Definition of a schema. Does this data match the schema that we've agreed? If you're doing a minimum viable data contract, would you start with schema definition is the first thing you start with?

Yeah, schema definition is kind of something you need in any data contract. I think. So think about what needs to be in all data contracts. You probably have a scheme definition, you probably have an owner, the kind of basics there. Probably every data contract has that. And then you can use that schema definition to do simple checks like. Like does this data match schema? Is it a string or a number? Maybe you can do more complex checks like does it match regular expression or is it an email address? Whatever you want to do there. But you start off just have schema and types is the basic data contract.

And then the bit that kind of intrigued me was to go from being a series of data quality tests or schema validation tests to being a contract is invoking this idea of an agreement. I think about it, I can have a data catalog, I can have metadata management, I can have schema definitions, I can have data quality tests, I can have schema enforcement tests on my side. But that's not a contract because it's a one sided thing. There is no agreement from the person sending me that stuff that they're actually going to bind themselves to this contract. And for me that was the slight difference in this pattern was yes, there's lots of tools and techniques that we've had for years, but, but actually the push is to then have an agreement with the data provider, the data producer, as a true contract, a true agreement that we both abide by. Is that how you see it? Is that the thing that actually makes it different?

Yeah, exactly. That become things you describe there. And a bit what we've been talking about so far as well has been the term tech side. So you can have a date contract, you can do tests, you can do schema validation, you can do CI checks, you do all those kind of things. That's kind of tech side, but on its own that doesn't solve the problems I want to solve. So I think it needs to be the people side as well. And that's where you start thinking about things like agreements and collaboration. The date contract is a great place to have that collaboration to talk around, to have something you agree on, to define who owns it, define who's accountable and responsible for meeting that date contract. It's only partly about tech. In some ways it's more about the people side and the agreement and what you're promising each other.

So it's that conversation, it's me saying I think this is the agreement. Do you agree? And then we kind of negotiate down to a contract that we both can live with versus me saying, and I'll keep using schema because it's a really simple one I can understand. Here's the schema, make sure you follow it. That's not a contract, that's not an agreement, that's not collaboration. That's me saying this is what I want and I don't even care whether you think you can provide it. I'm telling you. So again, the, the idea of using a contract is we actually are then forced to get two sided agreements between the people providing the data and the people collecting it. Is that how you see it?

Yeah, exactly. And that's a place where people sometimes go wrong with data contracts. They kind of forget about the agreement bit the people side and see it as I'm giving you this contract, I'm going to enforce it on your data and if you don't meet it, something will happen. And already you kind of come in from place of friction there, a place of confrontation there, which is not what you want. If you are trying to move to a more collaborative model where everyone agrees that it's worth doing this data contract because by putting a bit more effort in how data is generated and A bit more structure around that, discipline around that we're producing better outcomes of business, because this data is ultimately used to drive revenue or meet some sort of strategic goal, or try some key process, internal process. So it's about that communication. And often data teams don't get to communicate that much with the people generating data. In a lot of organizations that be software engineering teams, there's reasons for that, like organisation structures and there's Conway's Law and all those kind of things. Data contracts is a way to try and bring them closer together and try and break down those barriers and try to have some communication. And then once you've had agreement, you can codify a data contract and use that in your tooling and do the enforcement there. But that's not where you start. You start with agreement, you start with conversation first.

As technologists and data people, we love to codify technology first, people second, if at all. So I read somewhere that actually one of the things you should attempt to do is have the data publisher create the contract and then you negotiate with them to make it as fit for purpose for you as you can. This is the data team creating it and then try and have that conversation. Is that what you see, as you see by asking them to create the contract of what they think they're going to provide is a better way of starting that conversation than going to them with some predefined definitions?

Yeah, I think so. If I strongly believe that the contract has to be owned by the data generator, because ultimately they are the only people who can meet that contract, they can be the only people who have the ability to change the data, to change the schemas. They own people who have all context around the data they're creating like they have best place people to own the data contract. If I go to someone else and say, this is my contract on putting your data, and they say, well, we can't meet that, we can't meet the SLOs you want, we just haven't got the architecture to do that, or we haven't got the time to do that, whatever it might be. Then what's going to happen next? Not a lot. Whereas again, it should be more conversation. We need to say this is our requirements. And they say, well, we can meet some of those requirements now, maybe some later, but let's codify what we can meet now. And for me, as a consumer of data, at least I know what to expect now and I can build them out of confidence. I know it's not going to change overnight. I know maybe not as high as I want it to be, but it's high enough and I can work with that. But yeah, I can start building with confidence. I know what to expect. The only people who can change the data are the data producers. They must own the day contract.

How does that work? If you look at the example that you've got, your company was creating the software themselves. So you had software engineers that were building the core systems that you were getting from. And we know that often people building those software systems, systems create the data schemas and the data fit for purpose, for capturing that data, transacting that data, not for making it fit for purpose, for answering business questions. And we end up being data teams that deal with what I call exhaust. They kind of spew out this data or we go and grab it without their permission using cdc. So we get this kind of exhaustive data and their focus is creating new features, creating value, getting a product, getting product market fit, making their customers happy. And often their focus is not making sure that the data is fit for purpose for the bit of work we do. So how did that work for you? Was it just the fact that you were in an organization where somebody said, I'm going to bang the heads and make sure the software engineers and the data engineers work together? How did you deal with that political part of the organization and that problem?

Yeah, there's a few things to that. I think one thing we did is we didn't have a CTO or someone important banging heads, doing that kind of top down approach from the start. We started more bottom up. That's partly because of where I was in New York. You know, I was leading data platform team. I didn't have a kind of authority to start banging heads. But also I think it worked out better that way because you could start proving value and do that quickly. And then once you got proven value, you then can start thinking about larger rollouts and larger deployments. But start small and really focus on solving a particular business problem. And once you do that yourself, next one, next one. And then gradually you're building some momentum there and then you can start making more bigger program of work to maybe move over to contracts if that's what you want to do. So I think it's best to start small, start focused on the value you want to deliver for a particular use case. So say for example, the team somewhere building this data application that is going to be review generating somewhere you're going to sell it to clients. Being part of some sort of application that makes me new you're trying to make sure that's a success. And the problem they're having is they the data's maybe not quite quality or it's not fair, or it's not got slos or it keeps changing or some kind of problem that you want to fix. And the data's been produced by a software engineering team.

So we know the problem we want to solve and we know it's valuable for the company. And we know that we want software engineering team to do some work there. And we know ultimately they want to own the data contract. So start off by talking, get everyone in the room discussing problems, getting on board, and then start thinking about solutions. And if I want the data contract to be owned by the software engineering team, I need to make it as easy as I can for them to do that. So maybe they can define it in code, maybe in the same repository as the other code that's actually generating data, maybe alongside how they also generate APIs and define APIs or define the infrastructure's code or whatever it might be. But trying to think about how do you make it work for them? Because we want them to use it and want that to be easy to use. As easy as creating an API or as easy as spinning up a database in your cloud using your infrastructure's code platform. So really focusing on that experience for them, because we want them to do that work and we know that they're busy and they've got other things to do. So we want to make it easy. But also they understand the value of doing it. So explain the value as well. That's where we start, really. We haven't gotten them involved as well in the solution because software engineers love solving problems as well. If they feel like they're part of producing the solution, they're more likely to use it, more likely to talk about it when you're not in the room, more likely to get other software engineers involved, kind of becoming your champions, for want of a better word. You start there, like kind of mvp. You start building this small amount of tooling you need to enable for software engineers. Produce data they need to solve that problem. You had of producing the data you need for that review driving application, and then you prove it there, go on to the next problem, next problem, next problem, and you start gradually roll out that contract to your organization while delivering value along the way.

One of the things I've seen is when you start talking to the software engineers about what you're using that data for and you show them, they then understand the value of helping you get that data in the right way because they actually understand the pain and their engineers, right, they know how hard it is. So they're like, yeah, I get you, I understand what you're living with now. But they also see the value to the organization that the data is delivering. And so they're much more open now because they see that whole value stream. Let's carry on that thing though. Of okay, so you were an organization where the software engineers were building the system that had the data data that you had the contract for. If we're working in an organization that's using out of the box software as a service platforms and they may or may not have an API, so let's just say they do have an API. There's no real way though to negotiate data contracts with that vendor because they've got lots of customers. So if you're walking into an organization that is primarily using out of the box offer as a service applications, I'm guessing it's going to be a lot harder to implement a two sided data contract pattern.

That is a lot harder. As cases we just have to go as left as you can and put data contract as close as you can to the source, particularly if you're dealing with a big vendor. So say you've got Salesforce data coming in, you're probably not going to get Salesforce to change how I've given you data, maybe using 5 channel stitch or something like that to bring data through the APIs, but that's kind of best you can hope for. But what you can do is if you have a Salesforce admin, it can own the data contract and they can be the first person who is responsible for the data from Salesforce because they have a bit of relationship with Salesforce. It could be Salesforce, could be someone else. But Salesforce for example, they have that relationship. They also as admins they can change things. In Salesforce. In Salesforce quite customizable. So a lot of the problems come from custom fields and changing those which is directly caused by them, not by Salesforce changing the API. Sales API is quite stable. It's just it's A custom customizations people in Salesforce or B the API, while stable, it's not really made for extracting data. So it's kind of problems there which we can't really solve. But the best you can do there is have the admin in this case sales admin or the whoever owns the vendor relationship, have them own the day contract. If it's a smaller vendor, maybe you have got a bit more leverage there and you can have a bit more conversation and say, hey, these are problems we're having. We love using your product, but the data we're getting from it is not good enough for these use cases. How can you provide us better data? Maybe eventually that could be part of your actual legal contract and as I said, something about the data quality, investor loads and how you won't breach it and what response time will be if you do those kind of things can eventually become part of a contract with a smaller vendor where you have a bit more leverage, a bit of a better relationship in both cases really as left as you can go, ideally to a person who owns that vendor relationship.

You mentioned that problem you struck was you were cdcing off the database. And I've often hear people describe that as taking data without permission because you know, I've seen it in especially large enterprises, it's like, okay, nobody's talking to us, we'll just get the permission to turn the CDC on and we'll just grab it all and deal with it. Then you move to API or event based as a way of getting that data. So does that mean that actually you can't use a data contract pattern? If you're using CDC off a database, do you have to have API and events to make that pattern valuable? Or could you actually use it if you were using a change data capture or a log mining type of pattern to collect that data?

So we have looked quite hardly at how we can apply data contracts to change data capture data, but we never really found a solution that works really meets the goals. We had data contracts where we have something that's more stable, we have something that's being produced at maybe a bit higher level quality. So we're not just structuring the database schema, we are saying these are events or these are basically structural data that makes sense for business, not that makes sense for application upstream, which is different thing. Often you've never really found anything that works because say for example, you did try to apply date contractor CGC data and they want to change their schema. Now suddenly the software engineer teams can't change their schema of autonomy. They're kind of stuck waiting for review by some data team or their schemas have to stay stuck in stone and that's no good for them. They need to be able to change the schemas to add new features, to improve performance, do what they need to do. That schema needs to change quite regularly, particularly in more tech oriented companies where They've got good software engineering practices, they should be changing quite frequently, so we don't want to prevent those from changing. So you need some kind of abstraction from that database schema, something else, which is the interface, which is the API. Again, like I said earlier, like software engineers would never build on top of Sonos database and they wouldn't let another software engineering team do that. So why is it okay for data teams to do that? The problems are exactly the same and the solution really is actually the same, is to use some kind of abstraction, some kind of interface, which for data state contracts and for APIs, it's APIs.

What that means though is that there's more design, more planning, more requirements, more understanding upfront, and then also the need to be able to adapt constantly. If I give you an example, if I use a CDC tool against the database and I bring all the data into late persistent staging, bronze history, whatever layer you want to call it, that first raw layer where we get all that change record, I know I'm getting all the data, which means I can delay a whole lot of decisions around data design, around historization of that data, because I know I've got it all and I can now grab it from that layer whenever I need to without talking to anybody else. Whereas if we take a data contract approach, I've got to actually define the events I need in conjunction with the people that are generating those events. I've got to agree what those events look like. And when I go, oh shit, you know, for the next information product, I need data from another event. I don't want to have to wait three months for the software engineering team to get around to creating that event stream for me and the contract. In that scenario, it sounds like the perfect organizations for this are ones where their software engineering team are used to rapid releases, small chunks, so that I can say, hey, we've got this new information product coming up, We've already got the event for custom orders product, this one's all around returns. So now I need an event for customer returns product and then they can basically implement that, that event, that API and the contract agreement in a day or two rather than three months. So that seems to be one of the things that would make the pattern more valuable, easier to implement. Would that be true?

Well, I think it's a trade off. And what you're saying is we're going to do a bit more work up front and that means we have something more stable we can build on because we're doing something important. And if you're doing something important, really. You want that to be stable, you want to be able to ship data. This process, this application, you want to be able to ship it and for it to be reliable, you don't have to keep fixing it every week. So you are giving up a bit of flexibility, but in exchange you're getting more stability. Maybe it was better to prioritize flexibility when all you was doing with reporting. I don't know if that was true really, because you give those revenue reports to your board and they're wrong and they change from last time. That's not a good situation to be in. And the reasons they're very wrong is because this work you're doing, from your bronze layer or whatever into something more structured, you think you have all flexibility, but really you're having to do a lot of logic there that's very brittle. You're implementing a lot of logic that's already in the upstream service. And the upstream service changes. You have to redo your logic, they fall out of sync. You don't realize it. All the problems that probably everyone listening here understands, we've had these problems for years, decades maybe, in data. It could argue that giving up that flexibility, but I'm not sure that's a good idea anyway. The other thing I'll say as well is the assumption you're making there is that data changes regularly and that I'm not going to have to find anything up front. In a couple weeks time I'm going to realise I need something else and I don't have to wait another three months to get that. But again, in practice, doesn't change very often from what I see. So, for example, I've been working at a payments company for seven years, that's changed a lot. It's sort of gone from a scale up and growing very fast and we've expanded different markets, we've got different products. It's changed a lot. But our core data models around payments are very similar to what they were when I joined and it's around what bank accounts they go from to and how much was it. It's got a few extra fields around exchange rate use and things like that because we've taken money across borders, but it hasn't changed that much. It changes when new features are added and as part of that project add new feature. You say we also need to update the data model for that, and that's just part of the requirements, but otherwise it doesn't change that much. I think this assumption that data changes a lot and that we have to react to it and we have to be agile.

Or not sure agile is the right word, but we have to react to it as best we can. I think that's another assumption that we can challenge there. And it's not that day. Contracts can't evolve. They can evolve, but you're doing that more deliberately. You're not finding out the next day that something's changed. And that field you're using before is now full of nulls because I'm using new field. Those kind of changes you don't need to worry about if you're using some sort of abstraction at a higher level.

From a technology point of view, it seems that there's a bit of a race to create the default de facto standard definition of what should be in the contract definition. I think there's probably. I've never bothered to look, but two or three open standards now which look the same but don't. Is that what you're seeing? That actually there is this move to trying to agree a minimum, viable, reliable contract definition from a technology point of view, not the technology that's going to validate it, but the definition of what should be in a contract standard for 10 kind of thing? Is that what you're seeing?

Yeah, I'm kind of involved with the open date contract standards. I'm part of the technical steering committee there. And that's got quite a few people who have also published open standards and trying to work out how we can just have one that's backed by the Linux foundation, so it should be open. It's got some good governance around that. I can see the traction for a standard. It'd be great if we could define a date contract in a standard format. And then for free I get some tooling that anonymizes my data based on the categorizations I put in there, or some tooling that populates that catalog or whatever it might be. Be great to just get that for free. If you use a standard that's an interchange format between vendors, between suppliers, between tooling. So I think there's the benefits if one of them becomes popular is it's potentially very valuable. This particular standard, there's quite a few people involved with it. There are people publishing their own standards. In a way it doesn't matter too much who wins as long as we. As long as we have one. But I think the open date contract standards which I'm involved in got quite good traction. There's a lot of people involved, people from vendors, people from other standards, people from industry like myself, that's got a good chance of succeeding. But it's still quite early days. It still needs a lot more adoption before it can be really useful.

I'm a great fan of a standard. SQL is a great standard. It's been adopted by lots of people. What tends to happen is vendors then want their version of the standard, right, to differentiate themselves. And as a startup, if we could have a standard that we just use and we know we get interoperability with anybody else in the ecosystem, then it's valuable to our customers and therefore it's valuable to us. I think the main point from my view is if you think you're going to go into the space and look at, start implementing the data contract pattern, go and have a look at that open standard that's under, under Apache and grab it as your starter of 10. Don't reinvent the wheel and then if you need to change it or augment it, do so and then try and push it back about why the bit you wanted is missing.

In the meantime, what we end up doing is we do end up writing quite a lot of conversions from our debt contract to another structure. And that's not so bad. It's not that hard to write. But for example, define that contract in a certain format. We convert it to protobuf, so we can then use that in bootguard Cloud, so in pub sub schemas. But you also use Kafka and you can do some schema validation there. We convert it to JSON API to create some software libraries that we can give to our software engineering teams and they can do some validations there that match for data contract validations like things like email addresses or a regular expression or something like that. We do find ourselves quite often converting the data contracts into different formats and some proprietary formats as well. We use date contracts to provision a bigquery table and that's got its own custom JSON structure that defines a bigquery table. So we convert it to that. It's not hard to write these things. It's quite simple. Imagine if we don't have to do that. We could just convert it to one format and integrate with any of these kind of tooling we want. That'd be a great situation to be in.

How often does that happen? How often would you see the data team writing the libraries or the API tooling and giving it to the software engineering teams to say, hey, use this, it'll make your life easier and therefore you'll create this contract that we can then agree to Is that a better way of bringing the software engineers on the journey rather than saying to them, I need you to agree to this contract together and then I need you to create some tooling where you can validate on your side. And then I'm just going to sit back and wait for this beautiful stream of gorgeous data to come and not have to worry. Do you find that most data teams now to be successful, are trying to help the software teams with the tooling around this?

I think so. And I think often you have a or more and more data teams have a data platform team. They may have a bit more of a sort of devopsy or software engineering background as well as knowing about data. It's kind of my background and I led the data platform team and what they're doing is they are providing platform capabilities to maybe other data teams who then do transformations or whatever else they're doing on the data side. They provide some orchestration software and orchestration tooling. Venmo we're also providing tooling for software engineers to produce data through data contracts and they're providing other tooling available around the data contract. For example, the data retention kind of style tooling, backups, access controls, all those kind of things and all of that tooling we've been building on data contracts. So we really built a whole data platform around data contracts. I mentioned at the startup, data contracts are a simple idea. You just grab your data in a format that's sort of human and machine readable. It just unlocks so much that you can do with it. And we found that any platform capability we wanted to add we can add on top of date contracts. So simple things like backups is more complicated. Things like performing GDPR deletions and data retention, role based access control, all those kind of things we built on data contracts because the data contract contains the information we need in a machine readable format. So it says it's a personal data or not. Is it about a customer or about an employee or whatever it might be? How long should we retain it for? What's backup policy should be, what slo should it have? It describes those kind of things and then the tooling doesn't really care what the data itself looks like. It could be very wide schema, it could be very deep, doesn't really matter. It could be one field, could be a minion field, it doesn't really matter at all. We can easily build tooling that can use the date contract to apply policies to do something with that data. Over the last six years Any platform capability we've wanted to add, we've built on date contracts and we haven't found anything. We can't build top of date contracts from platform platform layer.

Okay, but I think you just switch patterns here and use the same term for it. So let me explain why I say that. So in our product we are what we call config driven. I used to use the word activemetadata, but then I saw how everybody else defined it and went, nah, shit, that's not what we do. So the way I describe it is everything we do is stored in config and then the config is used to generate anything else we need. So if we need code to transform data, config's read, the code is generated, it's executed, and then we dispose of the code. Effectively. If I want a security policy, you know, the policy's in config. When something interacts with that data, the policy's applied. So this idea of a set of rules is how I think about it, a set of rules that can then be used in various different ways to do various different actions. And I think that's what you just described. You took me from a data contract as being human and machine readable, definition and agreement between the data providers and the data collector or consumer to make sure it's fit for purpose. And you've extended that config pattern out now into various other things where it was highly useful. Is that what you've done? Is that the jump I think I saw you just do?

It's kind of, it's kind of a journey. Runtime with data contracts. Our initial thing was we need something we can use to have people collaborate a lot more and also to move us away from CDC and become a source of data to something where we create an interface. But once you define that, you think, well, how do we make it easy to create this interface? Interface? So then you start thinking, well, let's integrate into a platform and have it a platform capability. And then you're thinking, well, now we ask software engineers to own this data, but they don't want to learn about all the privacy regulation. And I don't want to know how best to back up BigQuery data. I just want that to work. It should just happen. Like your governance policies, they should just be automated as part of platform. So when you use that as part of platform as well, it's kind of comfy tonight. Driven, I say it's more. They describe the intention, it's more intention driven. For example, this is the data, this is what it looks like. It shouldn't cross borders or shouldn't be visible to people outside of Europe, for example. I don't really care how that happens. They're not saying, okay, spin up a bigquery table in this particular Geo and put these controllers around it and put these backups. Not deciding all of that, but not defining all of that in a massive infrastructure's code. Maybe turn from conflict or something like that. They tell us the, they're describing what they have and we are applying for policies to that based on what they tell us and based on their internal policies. There's a slight difference there, I think, because I think it's. I receive this in platform engineering as well in software platform engineering. Moving away from this idea of having software engineers to find every single thing about the platform that they need to make happen. And by defining all these kubernetes things are deploying and all the cloud stuff they need to saying I just want a database and I want it postgres, I want version 15 and then you get all the stuff for free. You get all backups, you get the access control, all that just free. You don't really care about that. You don't have to think about that. You're focusing on I'm using this database to my application. I don't want to care about all the other stuff. Make that happen for me. I'm saying for data and date contract in your data warehouse that you, you're owning. I don't care about all the privacy stuff. I do care maybe, but I don't have to think about how best to apply those policies to my data. I want that just to be taken care of. It's kind of two patterns. They can't go hand in hand though. You can't ask people to take more responsibility for data and then expect them to learn about all the stuff it takes to look after data and be compliant with your governance. You have to give them a tool as well to do that.

Let's just break that pattern down. So let's use an example. So we're extending away from data contracts being agreements between two parties of the structure, format, quality, sla, frequency of that data. And we're extending the agreement in code out to a whole lot more use cases. And so one of the use cases that I see massive value for is around this access to PII data. Tell me if I got this wrong. But you know, so we have this policy effectively that says, let's take a simple example, this part of the organization can see people's first names, last names. This part of the organization, really, really simple rule. How do we do it? In the old days we'd go and write some views. It would put some row level filtering on with a where clause or holy shit, we'd hold two tables, one with the name and one without the name. Depending on what technology we got and how many times we've done it. What do we really want? Well, what we really want is somebody to be able to apply a tag to that first name column and that last name column and then the policy to be inherited. Regardless of who's feeding it, who's reading it. That policy is now enforced as code, but as the provider. All I got to do is flag it pii, that's it. And then everything's taken care of me because that's a simple contract for me to adhere to. I know that that data is pii, so I'm just going to tell you and then the system should take care of it. Is that what you're saying, is that just extending that out to something like backups? I can see then how you can, you can take that policy based pattern and apply it in many different places over and above providing of the data, the application of security, you could then apply it to many different things. Is that where you got to?

Yeah, exactly that. And it goes back into saying how the data generator has to own the data contract because they know most about the data, not someone centrally maybe sort of data governance role or some sort of data security kind of role. They don't have the same context of data to personally generate data. So they are best placed to categorize data and say, well, this is going to be personal data or this is going to be a bank account or this is going to be something sensitive. But they are not best placed to decide exactly how the company should be handling that data. That's probably best decided centrally and implemented as rules in the platform. So that it's just everyone who is using this platform just get the policies applied for free in the right way that matches the company's central policies.

So why wouldn't we just get them to define all the transformations we need for data as a contract and then have the system automatically generate it? Why would we just push all the work back to the software engineering teams?

Well, that's a really good question because I've been thinking a little bit the same way. Not completely getting rid of data engineering, that's not what I want to do necessarily, but certainly when you've moved away from building on top of database schemas, upstream database schemas, and you move on. You're building on top of this abstracted interface, which can look completely different to a database and can look hopefully more useful to people in business. It's a usable data product, for lack of better term. It's a usable data product that is well documented. It uses returns the business uses, not the software, not the product. And it's usable. And it's hopefully because it's a better quality data set, you can hopefully do fewer transformations on it, which I think is great because you're saving a lot of time, a lot of money. It's quite expensive to do transformations on data warehouse typically. But also that logic there, I mentioned earlier, that logic is quite brittle. You're putting back in data warehouse. You don't really know how the upstream application might change over time and it's going to fall out of sync. You can't maintain that logic in two different places. Now it's only one place in the upstream application. You should shift to that left. So then you're starting thinking, well, what kind of logic do we need to do in data engineering? And maybe it's where you're joining these data products to create something new. So maybe you're joining a few data contracts together to create something a bit different that's still useful. Maybe. I would like to think in a few years time we won't be building these complex, expensive data pipelines that have lots of business logic in it, because through this change we are producing better quality data at source. I think that's going to be a better outcome for the business and product engineering as well.

So this thing takes us down to the struggle that my data warehouse brewers are in have. So those of us that have done this for a few decades, we know that historization can be automated, we know that immutable data can be done. It's the combining of the data together is the thing that always takes a lot of time. And this is one of the areas that when Data Mesh came out, it never answered that question. In my view, that was because it was written by somebody from a software background, not a data background, who didn't quite have the war wounds of combining data. Let's use a scenario. I've got an organization that's merged and now I've got Salesforce with customer and I've got HubSpot with Customer and let's just say both of those were managed by a software engineering team. But I've still got two definitions of customer and at some stage I've got to put those together and team Salesforce ain't doing it and team Hubspots ain't doing it. So that's where the data team always came in to do that work. Initial definition of a data contract Provider and receiver agree the construct, have an agreement around it, then potentially we can make the providers give us the same key, the same ID for the same customer potentially and our job goes away. Nine times out of ten that never happens. So apart from that, have you seen any other ways that this pattern of contract or policy can help us in terms of combining that data from disparate systems?

I've seen one other pattern that's quite interesting from another organization building imprint date contracts. It's a bit different because for example you gave some more third party data so again you can't go to problems there. There's only so much you can do and probably in that case the best you can do is have your Salesforce admin make sure that they give you a particular key that you can use and that's not going to Change and your HubSpot admin gives you that key, that's not going to change and then together you can join it. And that's still probably a data engineering task, probably not overly complex pipeline hopefully. But then your data engineering team is providing that as a product for rest of business saying this is customers from our CRMs and the rest business doesn't care that we've got two CRMs or however many CRMs you've got. So that's, I think that's still viable. That's why I don't think Data Engine is going to disappear at all. But I've seen interesting patterns where I guess more where you've got a software engineering team and it's like well, can we provide libraries that ensure that the keys are always correct at the time the data is generated at the source? So what I do is I provide a small software library that if they're producing customer data, it's always gonna have a customer ID in this format. If not it will, it will throw an error and the software engineering team have to fix that error. You could potentially take it further and have that library interact with some sort of central ID service and get IDs from there. If you want to have a centralized ID generation service is where I think you're going in future, particularly as I grow. It's quite small company at the moment, so I haven't got the problems of loads of systems, but you can imagine One day with mergers and stuff, they might have that and that's when they might start thinking about central site services. But you hide that from the data producer. The data producer saying, I'm finding customer data. It looks like this, and these are the measures and attributes and fields about it. And the library takes care of make sure the ID is in right format that we can then use its join data wherever it is. So they're not doing that as a sort of data engineering, sort of MFT kind of style process. You're, you're doing that of a source as far stream as you can. That's quite interesting pattern. I think it'd be interesting to see how that develops over time.

In that scenario of Salesforce and HubSpot where we don't control it, at least we know we have no agreement. If we say, ideally we want a single key that's unique and consistent across every system. And the Salesforce admin go, I can't guarantee that. And the HubSpot admin goes, I can't guarantee that. At least you know where the shit's coming from, you know, to deal with it. If you've got software engineers that of building the systems that you need the data from, then it becomes a political conversation about why can't they give you consistent shared keys. Because we're in control of destiny. So why is the organization letting this complexity come in? Why can't they actually have an agreement with us to do something and then by showing them what we do with it, then they can go, oh, actually we get it. We wouldn't want them to do all that mujy matchy stuff that you're doing. Because like you say, software engineers, like data engineers, love to solve problems. Think about data transformations. What do I do? I take something, I change it and I put it somewhere. I might change the structure, I might change the values, I might create. Table as I'm still putting it somewhere, even though it's temporary, I might loopy, loopy that a couple of times in memory before I write it out. But I'm doing a bunch of what I call nodes and links. And on the other podcast we had a guest that came on this week and it was all about system design for organizations. So again, this idea of nodes and links, a circle that does something and then a line to the next circle that does something. Think of a dag, think of an airflow, think of all those nodes and link paradigms. And what they said is when you go into an organization for the first time, ask a person who do they rely on and when and who relies on them and when. So this idea of handoffs, if we think about data transformation, same problem nodes and links, why wouldn't we implement contracts between every node? So if I write a piece of code, why wouldn't I write a contract? So that the next piece of code that reads the output has certainty and agreement of what's in there and will know when it's not been met or when it's been changed.

That's a good question. I think you use words handles there and I think that's kind of how I see red date contracts have most value is when you hand it off to a different owner, maybe a different team or different process to have a boundaries of ownership is where it's most useful of date contracts with its panel in a transformation pipeline. You could use date contracts there. We can't do this as well. It's kind of interesting because it wasn't how I expected it to be used, but because we have built so much around date contracts, it's a bit like describe why wouldn't you use it for internal things? You get all this stuff for free, which makes the next bit of code easier to write. We're seeing, we're doing that. It's not quite how I imagined it from the start. I always imagined it being at this boundary of ownership and thus that's where it's most useful. But because we built it as part of our platform capability, it then becomes a nice way to automate even internal transformations. So I think yeah, when you have built these capabilities and you built them well and they're easy to use and give a lot of value without add a bit of friction. You are defined date contract, you are putting some restrictions around. Hey, you might be able to change that in terms of maybe you won't maybe have to make breaking changes that new version, things like that. So you're paying a bit for the contract in terms of like the friction, you'll put a bit more effort in the creation of that. But if you're getting this stuff for free, then maybe it does make sense to use it even for internal transformations or moving data from one place to another, not necessarily to another team, but still moving data around, it's valuable to have some of the things you get for free with data contract as part of a transformation. You could see that in future. Who knows how far this can go really in terms of automating transformations to a degree where maybe you don't have to write loads of SQL and dbt, maybe there's another way of doing that in future that's built on mandate contracts, more describing what you want to happen rather than detailing how you want to do it in thousands lines of SQL.

And the reason I brought it up is it's something that's annoying the hell out of me right now. Even in our product where we've broken it down into the smallest chunks of node and links as possible, you still have to take a series of steps to get to that metric or that value or the data for machine learning model, whatever the valuable information products that you need to produce this or data product. If all you're doing is producing data and then what happens is you make a change and even though you can see the lineage, you know where it flows, you can trace it, you can track it, all that good stuff. It's still not telling me exactly when a contract's been broken. Now if I have tests or what we call trust rules, it tells me when I broke it, but it doesn't tell me before it breaks. And so that's where I keep thinking actually this idea of a contract to policy agreement to say between this bit and this bit, the agreement's been broken for whatever reason. So let's go back to that and let's go back to that data collection problem. The original use case of data provider and we're the data receiver. I see in the data quality world, and especially in the event streaming world, this idea of dead letter queues, and in the old days we used to call them error tables. The problem I see with that pattern is, okay, I get some data come in that doesn't meet the contract, or I get some data coming in that doesn't meet the schema, or I get some data that comes in doesn't meet the test, whichever pattern you're using, and then I stick it somewhere else and I'm going to deal with it later. But that gives me a whole lot of problems. Well, how do I insert it where it was meant to be? How do I tell the person at the end who's consuming the information product that of the million records we got today, 5,000 of them didn't turn up because they were wrong. There's a whole lot of problems we get when we dead letter queue. And so in my head, if we have a data contract that's immutable, if the agreement's been broken, everything stops. Is that how you see it? Is that how you implemented it?

So we can't support both. We use the dead letter queue quite heavily. I think it really depends on what your users expect or rather what you define in their contract so what they know to expect. So about expectations again, if at the user you'd rather not see any data, if not all there you care most about completeness, then maybe you want that oh, the data is not meeting the contract, let's just pause things and will fix it rather than provide partial data, rather provide no data. Or you might say actually it's more important if some of this data gets through, we can then use it for a process that might detect some fraud or something like that. It's better to detect some fraud than no fraud just because some of the values are wrong. So actually we might dedicate, we'll fix some will be paged maybe if everyone call for it and we'll go fix it. It might be a couple hours but eventually it'll come through. But in the meantime all the other data is going through and then you are catching his Fordham for example. So it really depends on requirements of your user.

I think what you're saying is I have to understand the information product that's going to be delivered and how that user or that system is going to use that information product. So the action it's going to take and the outcome it's going to achieve before I can actually define the contract. Because that would affect the con, what context or the. Or what's actually in the contract.

Exactly. You're providing this data for use case, right? To meet some users requirements. Maybe you've got many users and you have to sort of compromise and trade offs and pragmatic you can't mean more maybe that fine, but at least you know what they want and you try and provide them what they what they need. Then once you've agreed on that maybe making trade offs here and there, but there's some sort of agreement there, you then codify that in date contract and that's what people can expect from it because they know what to expect. They can use it with confidence. It might not be exactly what I want. In perfect world they might want it to be 100 liable or something on call at the weekend in case it goes wrong. But maybe that's not feasible for whatever reason, but at least they know now and they can build their own processes around that. So yeah, back to that people side again, back to that conversation and that agreement side.

In theory every one of our dashboards should have a codified data contract bound to it.

I think so I think they should. I think if the dashboard is important enough or if your data Contract only is easy enough that you should use it anyway. Then why would you not use it? I see this sometimes. I was using a dashboard a few weeks back and it was about cost information on our cloud. What I was trying to do is to work out one where I've kind of hats over at the moment is around sort of finops and trying to reduce our cloud costs, which is a lot of people doing that at the moment in various companies for various reasons. I look at the dashboard, I don't know if it's correct. I'm not sure I trust it. I don't really understand what this field means and I don't know who to talk to. I don't really know who created this and I'm sure it looked different last week. I couldn't use that data to make decisions about whether to start investigating a particular service. Has that service increasing costs or not. I'm not really sure what's happened here. Maybe it's not very supporting dashboard in the company. It's not revenue, it's not some kind of process that's fraud detecting or something like that. It's not that important compared to the. In the grand scheme of things. But also it wasn't very useful to me. It was. I couldn't use it. I ended up wasting a lot of time trying to work out I can use this. I end up going different source completely to find data. I needed actually to be a Google Cloud console and then just less good data. But I knew I had more confidence in it because it's been documented and I knew where it came from. Even for most simple dashboards, I think it's valuable to have a date contract around it if you want it to be used. Otherwise why would someone want to use it?

I haven't caught up on where the whole reverse ETL went, but this idea of activating data in other systems where we were actually pushing data records back to the system of capture or system of control because it's valuable. So rather than give it to humans, going to take the action, give it to the system so it can automate that action. Having this idea of a codified agreement between those two, it just makes sense. It's what software engineers would do. But as data people started taking over that integration, we seemed to lost that rigor. We always treat it as an ETL task, not a systems integration task.

Yeah, I think it's a bit of a maturity curve. We're going up, I think and we're starting to apply more discipline to the engine as a whole. And that's been going on for a few years now. But using version control a lot more than we used to, we're doing proper pull requests, we're doing. We've got nice patterns for deploying things, maybe to make sure things don't break and can roll back quickly. We're using instant management processes. A lot of this and all this comes from software engineering really, which itself is also maturing in the same sort of way. And people are moving towards more sort of strongly typed languages and things like that. So it's a maturity curve we're going on, I think, and I think it's good. I think we're doing that because what we're producing now is of more value than it was before and that that could be various things, it could be AI, it doesn't really matter what it is. But what we're producing now is more valuable than it was before and therefore we need to apply more discipline, more rigor to that production of that value.

I think the key takeaway is my perception that data contracts were solely between the data producer and the data receiver. And that always indicated the left hand side of the value stream. The data factory was always the data coming into the platform. That one's not quite true. It's a core pattern of here's an agreement of what it's going to look like that two sides have agreed to because there is a handoff. It may be a handoff between teams, it may be a handoff between systems, but I can't see any reason why it's not a handoff between two blobs of code. As long as there's an expectation that when it turns up it's going to look like something and behave like something and these other things are already going to be done so I can trust them, don't have to do it myself, then it's valuable. I definitely think it's a good fit where this for an organization where the software engineering teams in the organization, I think that given the immaturity of the data contract patterns at the moment and the technologies around it, I think those organizations are going to have a better chance of implementing it. Well, if you've got 55 out of the box systems and no control over them, then adopt the pattern. But it's going to be slightly harder, I think, to lead that path. I still think it's one of the few terms that's come up in the last couple years that's actually got value and going to survive, in my opinion, before we close it out and see where people can find you and learn more about this. Anything else you want to cover around data contracts? Anything I kind of haven't asked about or forgot to ask or you go, hey, this is the bit you really need.

No, I think we covered a lot of detail in this and I think it's been a great discussion. I think what I would say is a lot of times I feel like when I'm talking about date contracts, I'm talking about this perfect world we can get to. And of course it's a journey. It's a journey that involves a bit of tech and a lot of communication of people side and that's, that's probably the harder part, but it's a necessary part and there's always going to be trade offs. You've got to be pragmatic with this. Don't expect to be able to go date contracts overnight and go from CDC to fully date contracts in.

The journey going on. But it's fine if it only takes some time because you start to embed that into your culture and that's what we're seeing. So I was in a meeting with some software engineers a few weeks back. They were just talking about like, how do you make data available? And they were, they come up with a term but I didn't even use myself and described a bit of an I could be like, oh, so I see what we're doing here. We are moving from more passive data generation to more active data generation. We're actively providing data to you because you want, you need to use it downstream for these reasons. I thought, yes, it's exactly that. And this one I've never spoke to before, never spoke to their team. He's quite a new hire, but it's already part of our culture where he kind of knew that and explained it better than I did in a way because it's part of our culture now because we've been doing it for, for six years and we've got tooling there and it's just where we do data now and we still have CDC in the background. We're still trying to move things over. I mean not perfect, but it's a journey. If you put hard work into that journey, you'll find kind of part of your culture and then you just carry on that journey and you'll get there eventually.

That's actually a good point for us. I talk about four ways we get data. Push, pull, streaming and drop. Our customers sometimes push the data to us, sometimes we go and pull it from in the environment or from the SaaS platform or whatever it is. Every now and again we stream, the data will be streamed to us, which is a form of push. It's just more recent or more frequent. And then drop is a file drop or a file upload. And what we know is we know file upload is the worst for us. Highly flexible, really fast, easy to prototype or ideate off, but it's not repeatable. The contract's constantly broken. Somebody changes csp, they change the Excel spreadsheet every time. There is no contract and it's never honored. And I hadn't thought about this idea of when they push it to us. One of the benefits is we have an agreed contract to a degree. We know what the schema looks like, we know what the data looks like. We don't really talk about it yet as a contract and we don't agree some of those other things other than the schema, which is what we're starting to do now. When we pull it, we tend to treat it as a CDC problem. Even if we're hitting Salesforce APIs using data DDO, it's still a pull, but we still are accountable for the contract. And it's really interesting idea to say to the organization, actually you need to be in that conversation because like you said, they can go and add a field to Salesforce that breaks the API contract that we're expecting without us knowing. And so even though it's well documented and lots of tools know how to use it, it still can break. I'm going to think about which side of the fence does the accountability sit for something to happen as part of that contract conversation. Because that's the people part and process part that intrigues me. And then definitely looking at it internally is if I'm moving something and there's a dependency, why didn't it just become a contract? Because once I have the contract patent, I can write lots of them quickly. They can be automated, they can be honored, they can be validated. I can just keep using that pattern time time again because it has value to me. So if people want to learn more around data contracts, if they want to get hold of you, apart from coming to the Agile data stand at Big data London stand Y760, top right when you walk in the door. Apart from that, how else can people find you? And find out more about this data contract stuff.

I have my book on data contracts I mentioned earlier, which will be giving away your stand at Big Data London. So look forward to that to find out more about me. If you go to my website andrewjustjones.com I also have a white paper on debt contracts called debt contracts 101. It doesn't go into much detail as we've been in today, but more of a high level say these are problems you want to solve and this kind of date contracts can help you solve it. And to get that you can go to DC101. So DC with DateContracts101 numbers IO and that will take you to that white paper and it's free, a free download from the website. So that's a good overview I think of date contracts and problems to try and solve and not as much details to be described here, but if you'd like more details my website's a good place and from there I've got a daily newsletter about date contracts as well. I go into detail about how to write date contracts at your organizations.

Excellent. And you hang out on LinkedIn as well. Go find you there and sign up to your newsletter because you post every day on the newsletter. So lots of little contracts which is good. Excellent. Thanks for coming on the show. That's changed my understanding of the data contract pattern quite a bit. Like I said, I was stuck in the land of sitting on the left hand side of that data factory. Between the data producer and the data receiver has been the only place it played. So I'm going to have to go think about this one a bit more, which is all good. I hope everybody has a simply magical day.

Data Magicians was another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.