AgileData #30 - data products - Eric Broda

Welcome to the Agile Data Podcast, where we talk about the merging of agile and data ways of working in a simply magical way. Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hi Shane, my name is Eric Broda. How are you doing?

I am good, thank you. Thank you, Eric, for coming on the show. And today we got one that I'm pretty excited about. We're gonna talk about this thing called data products. But before we rip into that, why don't you tell the audience a little bit about yourself and your background?

Sure, perfect. First off, thank you very much for having me. So again, my name is Eric Brode. I run a small boutique consulting firm and we have deep skills and experience in data mesh. And as you can imagine, data mesh really at its highest level is just an ecosystem of data products, the things that we actually want to talk about today. I've written about my experiences on medium.com. There's a probably about 13 articles, give or take, around data mesh, data products, done a variety of podcasts, conferences, and I'm actually one of the privileged few who have actually implemented an enterprise data mesh at a global financial services firm. So I have a fair amount of experience, practical hands-on experience that I'd like to share today.

Excellent, and yeah, data mesh is definitely the hot buzzword of the 2020s. I tend to talk to people and say I have a black hat and a white hat on it, you know, they're, I think the fundamental principles of data mesh are ones we've been striving for for 30-odd years in data, if not longer, and they're just hard, right? And then, so that's the good. And then I think there's a bunch of principles and very few patterns out there for implementing it. So lots of vendor washing, lots of mesh washing happening. But one of the areas that has been around for a while, and this concept of a data product, And it would be fair to say that the majority of the people I talk to struggle with defining what a data product is, let alone how to implement one. So, you know, starter for 10, how would you describe a data product to somebody who didn't know what it was?

Sure, data product is kind of almost a combination of two topics, as you can imagine from the actual term. So first off, it's a merging together of this notion of data domains and product thinking. So, I think data domains are pretty self-explanatory, but it's a related set of data. And the product thinking really, you know, if I contrast it to project thinking, projects are short-term. They start, they stop, but products have a long-term time horizon. And I think a lot of folks actually understand products almost as well as projects in the sense that Most of the clients that I work with, they sell products. So, they know all about products. So, if you're Nike, you have shoes. If you're a bank, you have checking products. So, product thinking, believe it or not, is actually embedded into the DNA of almost every company that I work with. I think probably fair to say most companies out there. So, it's actually bringing those things together, data domains and products, and being able to be able to provide boundaries around data, have clear owners like products, uh, and make them actually discoverable so that folks can actually do something with them. So, so that's really the, the core concept. It is packaging data, uh, by, by putting a boundary on it and an owner to it, uh, and making it discoverable so that people can actually consume data just like a product. So that's kind of the simplest explanation that I would suggest is out there, but I'm sure it's going to create a few questions.

It definitely is, and so I agree on the idea of boundary. I think, you know, I want to deep dive into the difference between data products and data as a product in a minute because I think that's actually an interesting distinction we should make. But, you know, I've worked with data analytics teams kind of helping them adopt an agile way of working over the last few years. And one of the things that we were iterating with each of those teams was this idea of an information product. And the way I described it, it was a while ago that we came up with that term and we've iterated on the templates, but it was all based around boundaries. And at the time, it was a boundary of some data. We didn't tend to use the word domain, but I would now. It was a boundary of some code and it was a boundary of some output that somebody used. So, could be a dashboard, could be a report, but could have been an API, could have been a data service, could have been a file. So being very clear that an information product wasn't always a dashboard, right? Sometimes it was a different delivery mechanism depending on the persona. And the reason we created that boundary was we wanted to break the work down from being a year into a small iteration or a series of iterations. And so we had to create some boundaries to timebox that work. And so that started having success. We found this term information product people could understand. And then we found 101 other uses where that definition of a boundary were useful, things like prioritization. And so effectively creating a roadmap of information products, what's the next most valuable product. But our goal was to stop, you know, 6 months of data design, get something out in front of the customer's hands, you know, internal stakeholder or external customer that had value and we got feedback on early. We didn't go into the whole product thinking consciously, but when we started prioritizing and road mapping the information products over a time horizon of, you know, 1 to 5 years, that was a form of product thinking. We didn't start exploring the idea of data as a product, right? We didn't really explore the idea of observability, those kind of features that made products easy to find and easy to buy. And so, we should be adopting that. So, with that, I always kind of say at the moment, for me, a data product is a subset of an information product. Where the persona that's gonna consume it is a data literate person, it's an analyst or another engineer. Because in most of the descriptions I see, that's the definition of a data product, right? We're gonna get a domain of data, we're gonna create a thing that somebody else is gonna access via code and use. And I differentiate data as a product, as a way of working, right? That product thinking, that discoverability, that treating it as a thing that's gonna survive for a long time, or we're gonna kill it. It's not a one-off project, right? It's not just a piece of code that goes to die. So, what do you see as the difference between data products and data as a product, or do you not see a difference?

At one level, I would say it might be semantics, but I think the whole idea around data products enforces the product thinking. Now, let me kind of give you an analogy around how I think of data products. And the analogy is going to be something that everybody kind of knows and loves in many respects. It's Airbnb. So let me tell you why I think Airbnb is a perfect analog for a data product. So first off, Airbnb brings together producers and consumers. Those producers are property owners, okay, and the consumers are renters. So, so the first thing in a product is you have to have somebody who creates the thing and somebody who consumes it. And presumably you have some kind of financial arrangement where it's beneficial for all concerned. Okay, now Airbnb has an absolutely clear boundary. Okay, it's the legal structure, uh, and it's the service offerings that delineate its boundary. The interesting thing about it, it also has accountable owners, the shareholders who own Airbnb, and the legal, uh, in regulatory frameworks that determine how it runs. That's the accountability of the owner. Okay, so Airbnb has clear boundaries, it has an accountable owner, it has an empowered team. Those shareholders have empowered the CEO and the full complement of staff to actually run Airbnb. Now, Airbnb is also a platform, and, and here's the distinction that I make. It's the data product, which is the clear boundary and the packaging of it, but the ability to surface it into a self-serve type capability. So how does Airbnb, to continue the analogy, do that? Well, first off, Airbnb is known for ease of use, not just for consumers to find and rent properties, but for producers. There's an equally sophisticated, uh, user interface and capability and a whole set of aftermarket capability that makes the product, you know, makes the data product easy to use. Okay, you know, in data's case, it would be easy to find, easy to consume. Next thing is a data product or a product in general has clear contracts and expectations. Airbnb provides absolutely clear guarantees of service, payment, and safety backed in fact by public contracts and insurance policies. So, so now the interesting thing about this, and this is where I think the, the byproduct, the value of the data product actually comes in play, is like Airbnb, bringing producers and consumers together creates that virtuous circle of market growth. More property listings bring more renters, more renters bring more property listings. So Airbnb is a fantastic analog for data products. Now the interesting thing about it, now to extend this into a data mesh, Airbnb actually is an ecosystem of Airbnb sites and locations geographically, each geographically bound by its own set of legal restrictions, regulatory constraints, privacy, etc. So Airbnb is a product, okay, operating in an ecosystem of products. So now coming back to Airbnb as a data product, if you will, or the analog for a data product, it has all of the characteristics that I just mentioned, okay, when we move into the enterprise. So the key here is everything that I said about Airbnb, you could direct, you know, you could say, uh, are the characteristics of a data product in an enterprise. So now again, semantics, data as a product or data product. I think data as a product suggests it downplays the, the intertwining between the data and the whole notion of product thinking. So again, with Airbnb, Mapping to a data product inside an enterprise, clear boundaries. Data product has that clear boundary, has accountable owners. That's the data product owner, and, and they are accountable and responsible for delivering the service and expectations, the clear contracts and expectations that are out there. They have an empowered team. They, they create a platform, hopefully, that is— it makes data easy to find, share, consume, and govern. Okay, and they have producers and consumers. Producers, typically when you think of a data product in most enterprises, we think about the pipeline that feeds the database, that's the producers. But data products bring together producers, the folks who create the data, whether they're source systems or otherwise, that move the data into the data product, but it also addresses the consumers. And it actually has a— like any product, when you think about whether it's a banking product or Nike shoes, you're out there to address the consumers. Interestingly enough, data products or data as a product, I think, suggests and emphasizes much more the ingestion of data into the, the domain, whereas data products, in my terminology, it's about the, the ability to bring producers and consumers together to safely interact and in some cases transact. So, so, so again, at one level it's semantic difference, at another level, uh, data products, um, provide, uh, the direct analog to things that we see out there today that can be directly applicable into the enterprise today. So, so again, I, I come back to analogies that make it really easy to understand. So again, data product is Airbnb, the principles of Airbnb inside the enterprise.

Yeah, so I think for me, semantics count, right? Because when we mix our toasties, when we use the same word for different things or people have a different perspective, then we get confusion. And we get that within data domains, you know, how do we create a boundary around the data domain? So I'll come back to that. But I take your point. I mean, your definition of data as a product was nowhere near my definition, right? It was like, Yeah, that's not what I meant when I said those words. So, for me, data as a product is our way of working. So I think I will change my terminology, right? I'll talk about either data product thinking or data product way of working because it is those processes. It's the idea of a data contract, right? It's the idea of enabling the producers as much as the consumers. It's all those things we do when we create this data product. And what I see in the market is lots of confusion because data product sounds like a thing. Right? It sounds like a product on the shelf, you know, it sounds like your bottle of Coca-Cola or Pepsi, uh, whereas a lot of the thinking we have is around how we produce that in, in a certain way and how we make it available for consumption. And I agree with you, I think, you know, Airbnb is a great analogy for the shift in where we're going because that's why we're seeing such a big move towards data marketplaces, a place where you can present your products. You produce your products and you put them up for sale, uh, even internally or externally, right? It doesn't have to be for money, it can to a bunch of internal stakeholders, and then people consume them, right? It's effectively the Amazon for data, and we're gonna see that a lot, I think. But let's go back to that idea of producers. So what I see is a large amount of conversation around the consumers of the data products. How do we make it so they can consume that data? How do we make it so they know what it is? And one of the core principles of data mesh for me is idea of decentralizing the data work back into that software engineering team, back into the teams that are producing the data. You know, 'cause one of the major problems we've had in the data world for many, many years is people create data as a result of their application and it becomes exhaust. And the data teams are kind of capturing that exhaust and trying to make it useful. And so that massive disconnect is where a lot of the effort and the problems actually happen. So if we push all that work back into the producers, right, where they can actually produce data that's for purpose, data that I can buy, data that I can use as if it was a product, then we solve a lot of those problems. But I see very little conversation about how we do that. Again, it's all focused about data catalogs or consumption or quality of the data, right? All those consumption-based things, not those producers. Are you seeing that, or are you seeing some good work happening and enabling people who don't normally produce it to produce it in a better way?

Yeah, I would actually— I'm gonna suggest a slightly different terminology. So the notion of the data product has that boundary, okay? And the data product team and the owners and the data product platform manages that data. So here's my slight disagreement with what you're saying is it is absolutely not about the producers, okay? And it is absolutely not only about the consumers, okay? And in fact, I actually see much more emphasis on the producers. In fact, the terminology used is the data product focuses on the producers, and I think they're actually separate entities, and the data product brings consumers and producers together. So here's what I mean, is to your point, who actually does the work? Well, typically there's a source system, okay, that— let's just paint this scenario. I have a source system going into a data— multiple source systems going into a data product, and I have multiple consumers. Okay, the, the key thing here is the producing systems, whether the source systems or other data products, are separate, unique, and have their own owner and their own boundary. Okay, and if it's done well, whether it's through pipelines or through APIs or queries, there's a way to actually get that data, push it into a pipeline, and ingest it. The responsibility for ingesting, okay, i.e., consuming that other source system or other data products capability, is actually the data product team. Okay, so they own end-to-end getting the data in. Now obviously they collaborate with a whole bunch of other folks, the source system team or the other data products, but their responsibility, the responsibility of the data product team, is to get the data, transform it as necessary, store it as necessary, and make the interfaces available to expose that data. So that includes, you know, the discovery, conceptually a /discover endpoint. It includes, um, /consume endpoint, or more likely in an analytic the federated queries that are available to actually access that data. So the only distinction that I make is the data product owner and their team is self-sufficient and they interact with other source systems and other data products to do their work. And they are accountable for making the consumption patterns, consumption services available to a consuming capability. So what I'm seeing out there is, a very significant delineation in terms of responsibility. So most organizations do emphasize the producing pipeline, the data engineering team, and far too often they build that as a shared service. They build all the pipelines that connect all the various data products. And what I find is the moment you go into the shared service capability, the centralized capability, what you end up doing is you introduce unnecessary bureaucracy, you slow things down, and your agility diminishes as it gets more complex. What we're saying as a data product is, I'm saying, all that responsibility is the responsibility of the data product owner and the team. In other words, they don't have to, you know, it'd be silly if they didn't use, you know, vetted enterprise capability, but they don't have to. It's the prerogative of the data product team. They may say, you know what, I don't need data pipelines in, I'm gonna go real-time and I'm gonna listen to a topic "now I'm going to get those things, trickle feed them in real time into my data product." Okay, in other words, the data product owner has the wherewithal, decision-making rights, and the capability to ingest the data any way they want. In other words, they don't have to rely on a shared services team, okay, but they may if it makes sense. They— but they own the capability and they determine how it actually works. The same corollary on the consumption side. Sometimes, well, most times actually, folks will say, I just wanna access via SQL. Give me my SQL query, leave me alone. And today, actually, I'm gonna say 5 years ago, that probably was the only way that people went. Today, what I see is this notion of the real-time enterprise. Consumers are saying queries are interesting and useful, okay, but that's not the way I wanna do it because I need real-time data and I'm not gonna issue this query especially if it's relatively complex. If I'm going out, you know, doing a 100 million by 100 million join, row join, you know, you're going to want to do that very, very carefully. What they're saying is, I want to, and I don't want to ping that thing and execute that too often, I want to actually just listen for when the data in the data product changes. So, the paradigm that I'm talking about is the accountability, the ownership, and responsibility is in the data product team. Not in a shared services pipeline team or a shared services engineering team. Although again, it would be foolish of them not to use capability that makes sense and is standardized, but it's the prerogative of the data product owner and team to set up their data product any way they think is equitable, financially sound, and makes sense.

Yeah, so I think you've, you've covered a fairly large number perspectives or lenses in that definition. So I'm with you on a lot of it. So, you know, when Shopify published out, you know, quite a while ago, their ways of working, you know, doing the finger quotes here, the Shopify model, one of the core tenets of that, one of the core patterns was self-organizing teams, and the teams were able to use whatever technology they wanted. And what they said was, over time, teams found that rather than standing up their own technology and maintaining it, where they saw critical mass in the organization for a technology, they would adopt because it became easier. Yeah, the, the community was there to understand how to use it, the support was there internally, the bodies of knowledge were there, uh, and then over time, uh, teams broke off to become shared services teams effectively for that technology to support those other teams. But that was a natural thing. As a boundary, as a domain team, you could decide not to follow that path and do whatever you want because you were empowered to produce effectively a product within the application. And for me, that is a very successful pattern. If you start off with your teams as you scale and give them that right to be in charge of their own destiny, they will 9 times out of 10 do the right thing. And they will adopt shared patterns because it makes sense to them. Humans are inherently lazy, right? We don't want to do work we don't need to do. So that's a good pattern. I think the other thing that I want to pick up on that is we need to differentiate team topology, our operating model, the way our organizational team hierarchy works, and data products. So I'm with you still that data products is a boundary. So it's a boundary of who owns it, Who produces it? Who consumes it? How do they want to consume it? What's the data that's in there? What's the data that's not in there? What's the code? What's the observability? There's a whole lot of patterns that we would— lenses we should put across it that give us that boundary. But we can have a single centralized team that's producing data products if we wish. We just have to deal with the consequences of we now have a backlog that's massive because there's only one small team. We can have a bunch of centralized teams, pods, squads, whatever you want to call them, who each own a set of data products or a domain. And that's okay, right? And we can have them using a shared platform or not using a shared platform. It's all okay. So I think with data mesh, one of the things is it talks about the four principles, which is great, and then it brings in a fully decentralized topology of pushing all the producing work out to the software engineering teams. And I think that is where we're go to, right? It's nirvana, but we've never managed it. But I think that's also confusing the market, the difference between the principles of a data product and that team topology. I just want to jump onto the idea of data contracts. So I did do a shout-out on LinkedIn a while ago and said, could somebody give me a template for a data contract, right? Because I didn't have one, and I actually needed to put one in place for some stuff that we were doing as part of our startup. And yeah, I got a couple of examples, but not a lot. So that idea of a data contract, you know, I take your analogy, right? The Airbnb has a contract for its producers and a contract for its consumers. And, you know, you sign up to it, it's well-formed. You might not like it, but it's still there, right? But we don't have that, right? We don't have patterns for data contracts widely available in the data world. Or have you seen it differently? I mean, have you seen some really good data contracts out there?

I think it is a gap today, but let's kind of unpack that a little bit, I suppose. When I think of a data contract, there's kind of minimally 3 things that you need to think about. One is obviously, you know, there's a tech set of expectations, contracts. So, so for example, you know, SQL would be, you know, and that's, that's one way to— and it has its own set of contracts and expectations. You may use APIs, use OpenAPI specification. So the tech side is one element of it. There's expectations, uh, around, you know, change, for example, that there's an expectation that, you know, what works today will work tomorrow and that there's some level of backward compatibility. There's also hard things like SLAs, service level agreements, or expectations where, you know, uh, you know, 99.8% of the queries will happen within 2 or whatever that may be. So, I have not seen anything that spans all of those, to be honest with you, but what I would say is people are starting to understand that when you look at a data product, it is the vehicle to bring all those things together. So, I'll give you a simple example. At some of— several of my clients, we've implemented this notion of a data product registry. And what the registry does is— and it's a prototype at this point in time, um, you know, undergoing some, some industrialization, may one day be a product, who knows. But ultimately what the registry tries to do is— and it's different from a catalog, and you'll see why, like it's not a Calibra, it's something very different— but what it tries to do is say, you know, if I want to go and find the data Okay, I should be able to have some way of actually searching for all the data products that are out there, finding the one I'm looking for, you know, double-clicking on it. And I should be able to see much more than just the database schema and the tables and the columns and all that. Even there, you should be able to have, you know, which columns are sensitive or whatever. But it should say, you know, who's the owner? So who do I call if I actually want to get something? What are the SLAs? What are the— now, here's the real thing. What are the actual access mechanisms? Now, here's when you boil down data mesh and kind of what I do for a living, data mesh is a set of principles, data products being the architectural quantum. What I try and do with my clients is I turn the practices in the principles into practices. So, this notion of a data product registry actually crystallizes it for all. It's like when you think of Airbnb and you go and search for a rental property, you know, you can put any level of filtering, geographic location, price, etc., and hit go, and it gets you this beautiful thing, and you can find properties anywhere in the globe. Same idea with a data product registry. I should be able to have that simple an interface, and I should be able to navigate from the forest, find the tree I'm looking for, drill down, find the, you know, of the, the forest of data products, I should be able to find the data product that I want Okay. And I should be able to now look at the service level agreements, the hard, hard agreement. I should be able to look at the service level expectations, backward compatibility, etc. We promise not to break things or whatever. And I should be able to look at the tech expectations. What are— again, if I look at the analytical space, if I'm trying to do a join of a 100 million row table with another 100 million row table, which, you know, you think is not that frequent, it's actually more frequent than people think about it, especially in financial services. You don't want to have people doing that just, you know, at the drop of a dime. You want to have those queries vetted, performance tested, all the data has been scrubbed for all the regulatory and privacy concerns. In other words, you want to have vetted queries. So that's another set of capabilities that is exposed in the data product. In other words, The data product is the container that houses the data, but it also houses all of the formal and informal expectations, service level agreements, and access mechanisms all available. That, in effect, is what I think of a data product contract. There's nothing out there that has that today, although some of the work that I'm doing at my clients, we're pretty darn close to instantiating that. But that's what I think of the contracts. There's no vendor product out there today that does it. There's no formal spec that does it. But if you're able to think through it a little bit, you can actually piece the puzzles, you know, you can actually put the puzzle piece together so you can actually see something that makes sense.

So, I really like that. I haven't heard that lens before. So, one of the things we know we do badly in the data world, right, is we confuse people who aren't data literate. Yeah, we talk about, you know, in the old days, EC2, but, you know, now we talk about mesh and, you know, fabric and APIs and all these words that consumer of a product doesn't care about, right? They just want the product. So I heard— so I like that idea that we can use consumer-based terminology to describe what they get. And as I was thinking of those words you were using, you know, that's effectively some of— one of the things that you were talking about is the warranty. When I buy a product, I get a 1, 2, 3-year warranty. I get a promise of how long this thing's going to survive. So, I heard that, right? I heard that form of a warranty. One of the other things, whenever I'm coaching a team, I get them to start off first is what I call definition of done. And it's a little bit different for the data team than what typically you do in Scrum. So, for me, definition of done is not what the product owner is asking for. It's the set of criteria that as data professionals we expect to do when we build data, data products. So, you know, our consumers, our product owners expect us to test our code. They don't expect to have to tell us to test our code. They expect us to validate the data. They expect us to make sure that we haven't lost any. There's a whole lot of expectations that you would expect a data professional to do. And that for me is that definition of done. And I can, I can take those definitions and I can then think about, okay, if we had this registry of data products, those should be things that we have green ticks on, right? We have done these things to produce this product. It's almost like a QA process in a factory, right? To say, it's passed all the tests for that product to go out and be boxed and shipped. So I like that, right? I like that analogy. I think we're going to see a lot of confusion between registry and catalogs. Right? Because they are similar but they're not the same.

Um, yeah, well, let me, let me just— I want to kind of touch on that real quickly because you did mention it. First off, there is, there is confusion in the market. That's why I purposely called it a registry. And, and, you know, point well taken. Maybe I'm not adding— maybe I'm not adding clarity as I do that. But let me, let me explain why I did that. Um, so, so first off, you know, let's start at basic principles. What is a domain? So typically, you know, the domains are defined by, you know, the Chief Data Office in a typical large organization, but they're macro-level domains and they're quite granular, okay, coarsely grained rather. The practitioner wants to have something that is finely grained, something that I can use. Well, lo and behold, the data product is exactly that. It's designed for consumption. And the production ingestion of the data should follow the consumption of that. So, so first off, domains are different governance domains, which is their traditional data domains in an organization, which is different than the data product, which is practitioner-focused. Again, the audience of the data product is the developer, the data scientist, and the analyst who has a practical job to do. The, the, the job of the data domains at the CDO level are to ensure that we're doing the right thing with our data, setting the policies that the data product owner and their team actually have to implement. So by its very nature, data products have a finer, much finer-grained domain. So now if I think about applying that to data product catalog, the confusion comes in when people think of a data catalog. You know, there's, there's products out there like Calibra Okay, and Collibra is a great product for what it does, but let's make no mistake about it, it's a governance tool. Independent of how people want to, you know, modify it, change it, use or abuse it, it's a governance tool. Okay, and it's designed that way. The data product needs something that is a practitioner tool. So for example, let's come back to the things I mentioned earlier. Collibra may be— it It definitely can, can provide the schema of the data product. Absolutely. You probably can actually identify the owner. Sometimes you may be able to define the service level expectations, but it's a little bit ad hoc. Okay. It has— it largely is, is agnostic to this notion of expectations, versions, backward compatibility, because it doesn't talk about the practical implementation. Once you go into the implementation, you know, what are the APIs, what are the queries, okay, what are the events that I'm listening for, you know, that's nowhere to be found in the governance tools. It's a different, it's a different problem you're solving, okay. We're solving the— Calibra is, is something for the CDO and the governance team, and it's a super valuable product for them, great product for them, but we shouldn't confuse that with what the practitioner needs. The practitioner needs, you know, very practical. How do I actually consume this darn thing? What are the vetted SQL queries? What are the APIs? What are the events that I can listen to in addition to the expectations and the SLA stuff that I mentioned earlier? So I call this thing that exposes that information, I call that the registry. Now, we too often think of the registry as just a user interface. The registry is two things. It's a user interface, and it's, for example, the /discover endpoint, the /observe endpoint, the /usages endpoint, the /logs endpoint, it's the /alerts endpoint, each of which, for example, alerts what happened last night, what are the bumps in the road, and can I actually see, you know, what actually occurred? So I can look at the alerts. I can also look at the logs to maybe do some diagnosis. I can look at the usage to say maybe there, you know, the, we had, a huge amount of new users that came on board, and that's why the performance was poor. So, so, you know, and I can do a /observe to see, you know, what else happened in my data product. All those things are wrapped around this data, this notion of a data product. And I, I call the registry not only the user interface, but that set of /discover, /observe endpoints that make it machine-readable, if you will. So that's kind of the, the very the distinctions that I draw.

Yeah, so I won't go off on a rant about governance by committee and how unvaluable it is. So, again, I can take a whole lot of other words and use them to describe what you're talking about, right? And they all have value. So, you talked about a lot of DataOps behavior, that idea of logging, that idea of audibility, that idea of monitoring and exposing performance. Anomaly alerting, all those things that we should do for a data product. They're all valuable, right? I mean, we have a massive problem at the moment that we're, you know, the market's unbundled everything except for the database. I'm always confused that we say we're in an unbundled world, but everybody uses Snowflake, which is highly bundled. So we're going to see that. I think the word registry, for some reason, whenever I've heard the word registry, I've read the word Protobuf. Which I don't actually understand what it is and I've never bothered to go and look, but it just sounded like a technical word. But I do remember a data, a really light data catalog tool that was out there for a while. I think it was the one that Qlik bought and embedded into their product. And I think they lost this feature, but what it did, it had this feature that I thought was quite interesting. And the idea was it treated, there were a bunch of tiles and it looked like Amazon effectively. right? And so each one of those tiles was effectively a dashboard, you know, it would be a form of, light form of data product, right? It had a boundary of data, it had a boundary of code that transformed that data, and it had a way of delivering that data, typically in a dashboard. And then what you could do if you wanted access to it, you dragged it to a shopping cart, and when you hit buy, it actually sent off a request to the owner of that data, the person that was recognized as the owner or the steward, and then they approved your access, right? And did the RBAC, change for you automatically. And I like that, right? I like that analogy because it was like me buying something from Amazon. Yeah, it was, I could treat that piece of data as a product to get access. So, I'm with you. I think this idea of extending the paradigm or the pattern of a catalog from just being governance or just being seeing the columns or just being, you know, identifying the PII data to having all those lenses, right? All those things that a good product should have.

Yeah, so just to build on that, like, like I'll give you an example at some of my clients. What we've done is we've looked at it and said, not only do that— does the data product registry have all the capability we said, but suppose lo and behold somebody actually finds the data and they, they want to actually access it. What do they do? Well, in the old world, you look through your Outlook organization chart to figure out who in security may be able to call. Hopefully you get the right person who will refer you to another right person who may be the right person at some Anyway, we try and avoid that telephone tag by allowing folks to actually be able to create an access request, attach it to the registry, facilitates that. So I can now— so I'm just giving you a few of the highlights, but there's a lot of process capability that you can layer into this data product registry, okay, that makes it— again, the mission I have is to make data easy to find, consume, share, and govern. Okay, and getting, getting access to data is fundamental to actually achieving that. Now, what the registry also can do is— our, our goal was recognizing, again, our data products would be relatively finely grained. We wanted to, to be able to spin up, just like if you want a cloud instance, you know, a VM or whatever in Amazon or AWS, you know, it's all user interface, a few clicks, and lo and behold So, so part of what we wanted to do is we wanted to say we should be able to have, you know, a step-by-step user interface guide that walked them through spinning up a brand new data product. So you'd sit there and say, here's the owner, here's some of the— here it's a Postgres database, here's the URL for the Postgres, or is the configuration for Postgres, and hit go. And it's— it created automatically the /discover, /observe, /usage endpoints for you, and you filled in a little bit of data, and literally within 5 minutes you could have a data product. So, so where I'm going with it is that the registry is, is, again, the mission that I have is, is make data easy to find. So that's the search capability. Make it easy to consume, as I offer the list of APIs, etc. Uh, make it easy to share the APIs, the SQL and stuff does that. But it also make it easy to govern and operate. So, so being able to, you know, find the data, and create an access request. Being able to spin up a brand new data product in 5 minutes or less, that's all achievable today. And we bundle that capability, right or wrong, with the data product because it comes with all the benefits of having clear boundary, an owner, and a team associated with it. So it just made eminent sense that that's how we actually structured it. And I think that's where I'm, you know, if I were to look, my crystal ball gets kind of foggy. After, I think, 6 months ahead. But I think that's where data products are actually going. You're gonna have, they're gonna focus on the developer, data scientist, analyst user experience, and be able to make it easy to, again, find, consume, share, and govern data. And that's what the Data Product Registry's mission in life is.

Yeah, and so I'm gonna go back to my definition of data product versus information product, right? So you gave a very good description of a data product, right? You talked about the persona, it's the analyst, it's the data scientist. You taught about the data experience that they wanted, right? Find some data, stand up the endpoints, get access in that. If I talked about Chief Marketing Officer, their experience has to be different, right? They want an information product. They don't want to know, sorry, maybe some do, but most of them don't want to know about an endpoint, right? They just want to know that they have some data to consume. They want a view of it, they want to consume it slightly different. So their experience has to be different, and that's okay. We just got to be clear that that changes the boundary, right? The boundary for a data product that's focused on an analyst or a data scientist or an engineer is a different boundary that's focusing on a consumer that is more at the business level, and that's okay. They are two different products. They may use the same data, right? They may reuse the same platform. They may reuse a whole lot of the moving parts, the Lego blocks I call them, but they're a different product because we have a different customer and the experience we give them should be different. So for me, that's, you know, we've got to bring in the persona, the consumer of that product, as one of the tenets of the boundary we create when we define that product.

Absolutely, 100% agree. This is, again, it's pivoting from a producer— how do I find the data and get the data into the database— from, you know, pivoting and saying, what does the consumer need, and I'll go get the data and make sure I can transform it to address consuming need. It's a slightly different perspective, but I agree 100% with where your head's at.

Yep, excellent. And so let's go back to data domains, right? Because this again is a tricky one. So I was intrigued that I'd never quite thought or seen the data domains coming down from a CDO. I think in New Zealand, we don't have a lot of CDOs, right? It's still relatively new to us. And if we do, they don't really act like CDOs we see over So what I tend to see is there's a couple of lenses with which data domains get defined. So the first one is, I think it's Conway's Law, right? In terms of the, you know, everything will behave the way the organization does. So the organizational hierarchy will define the domains. You know, there's marketing, there's sales, there's finance, there's HR, or people. I see another lens, which is core business process. If we have a supply chain, if we have an HR leave process, those kind of boundaries where there's a beginning and an end, and data moves through it to achieve some core business processes, I can see those often become domains. Another way to do it is to think of it as concepts. We have the concept of a customer and the concept of a supplier and the concept of an employee and their domains, right? We're gonna get lots of core business events, it's going to span across multiple organizational hierarchies or silos, and that's our definition of a domain. Those are the ones that I typically see. I'm sorry, and source-specific silos. We have Salesforce as a domain, we have HubSpot, that's a domain, we have SAP, Oracle Finance, that's a domain. So those are the lenses I've seen. What do you see? What do you see and what do you use to get a good definition of a data domain.

Yeah, sure. The, the very first one you mentioned, I really— I'm very glad that you mentioned Conway's Law. Um, it, it, it is, uh, it is prescient in terms of how, uh, how, how true it actually is. And it was created by Mel Conway, what, in 1960s or something like that? But it's been so spot on. But here's the lesson learned: if you don't follow Conway's Law, okay, you're going to be swimming upstream, okay? And like anybody swimming against the current, you're going to get fatigued. And in terms of projects or the longer-term funding, you get funding fatigue, okay? So here's— therein lies the problem with shared services, okay, and centralized organizations. Who's the owner? Don't know, but it's definitely not the folks that have the money, and the, the, the— and it definitely doesn't follow the organizational decision-making tree, which is really the org chart kind of shows that. So, so going against Conway's Law is, is you really gotta think through why you're doing that. And I think today, you know, I mean, if you think about why people created shared services is because the skill set was very hard to find or extremely expensive and you wanted to share it. Most of the technology that we're talking about today is not in that camp anymore. Okay, now, you know, maybe it's not too easy to find an AWS engineer, but it's, you know, there's no need to have shared services built around that anymore today. So first off, Conway's Law, absolute fantastic hint for where your data products are. Second thing is business processes. I tend not to find those because what they do is they actually swim across organizational lanes. So they kind of violate Conway's Law in some respects. That's not to say it doesn't work. But it has to be a super important process for it to actually work well. And you have to have an extremely strong owner of the actual business process to make it fly. More often, what I find is something that most organizations, or many of them do, at least the larger ones, is a business architecture. Okay. So, it's kind of the, you know, I've seen hundreds of these things, but they're PowerPointware. But they're very useful in the sense that they identify the domains. Now, they happen to be usually one level below the governance-level domains. So, for example, client is a governance domain. Client is so abstract, it can mean anything to anybody. But, you know, a business architecture would probably go down and say, in this geography, here's what a client actually means, and here's all the other capabilities. So, business architecture actually is the next level of granularity where I look to to define my domains. Fortunately, most of them line up to the organizational structure. Again, Conway's Law coming in. Now, there's the interesting thing I find is depending on what industry you're in— I'm in financial services, so there's a lot of pre-canned models that give you tons of hints. So BIAN, B-I-A-N, is one that I've seen many times before. It's financial, it tells you the business architecture for an internet or global banking organization. I've seen things like Teradata's financial services data logical data model. The data model isn't as interesting as the fact that it has this huge glossary of, of all of the business entities, which again are organized typically around the organizations that you typically find within an organization. And again, the last one, and perhaps the— hate to say it, but sometimes the least useful— is the domains that the Chief Data Officer comes in and defines. And again, it's not that they're not appropriate, it's just that they're addressing a different need. They're addressing a governance need, which is absolutely crucial. Don't get me wrong, it's needed, but it's too far removed from the practitioner that, you know, it offers some useful hints, but you have to go several levels, you know, deep down into the organizational structure before you can get some something useful. So, that is like, to be honest with you, it's one of the very first questions that always get asked at my clients, which is, "I love this data mesh thing, and I think these data products are fantastic. How do I find one?" So, there's no— I'll be honest with you, it's a little bit of my secret sauce, I suppose, my consulting business, you know, but it's not necessarily rocket science. There are some pretty solid hints that if you're aware of simple things like Conway's Law, you can find your data domains, your data product domains, or data products rather, a lot easier than you may think.

Yeah, it's interesting, isn't it? I'm now worried that we're gonna see the enterprise data mesh model. So I remember the days of Oracle, IBM, Teradata, where you could buy the banking model and everybody was proud of how many walls of paper you could put up there for that model with typically the horrible party is a party of a party. And, you know, it was a very expensive piece of paper that was very hard to implement. So, I think for me, I agree with you that the first thing you should do is observe the organization. So, typically, if I observe an organization, I hear the term value streams or customer journey. And then I keep observing and I see the organization actually behave that way, it's not just service, then that idea of having a domain based on core business processes has possibilities. Because what they're saying is they want to work in that end-to-end way. But if they don't, right, if they have organizational hierarchies, then yeah, yeah, you need to bound your domains to that. I think the other thing is, and it's something I haven't really been thinking about, but I will now. So it's probably one of my big takeaways for this is, as I said in the beginning, when I was working with organizations and we defined an information product, right? We wanted— our goal was to break the work down into smaller chunks, right? That was, how do we take this lifetime value thing that we know is 12 to 24 months' worth of work? How do we break it down into smaller parts, right? How do we break it down to say, well, actually, there's a definition of revenue that we need to do and there's a definition of expenditure and there's a definition of margin and a definition of product— profit, There's a definition of churn, right? There's a whole lot of these like Lego blocks that we need to put together to do lifetime value. And each one of those takes time and effort, right? They're not easy. So how do we break that work down? And that's the kind of one of the first rules of the boundary was decompose it down into small iterations. As a result of that, we found some value, right? We found that we could define a canvas, right, which took us 15 minutes to fill out. And we found that actually the canvas was so simple because it was based on the business canvas. which is a thing of beauty, that actually the product owners would be able to pick it up and do it themselves. So, they'd bring the canvas pre-filled out to the team and start the conversation with that. Then we found out that we could use that canvas, that boundary to define some form of value that could be prioritized. So, we started talking about, okay, if this product was delivered, what value is there? And everybody went, oh, blah, blah, blah, right? It all became very high-level, you know, increased revenue, better customer experience. Okay, well, that's, that's not working for us, right? Uh, so we, we ask questions of what was the action you're going to take. Uh, so then it was like, oh, okay, well, this is a churn product. What we're going to do is we're going to make an offer to the customers that are expected to churn in the next month, uh, and we expect to retain, you know, a certain amount of revenue and increase, you know, uh, retain a certain amount of margin as a result of that. So we're like, cool, there's the value, right? The action you're going to take informs the So, if we think about actually creating a checklist almost, a lens of what the drivers of the boundary are, then that would be valuable, right? So, you know, you talked about binding, you know, the warranty, the definition of done. Well, that's not the word you used, but the words I'm using, right? But that SLA, that warranty, right? That definition of done, that expectations and the technology. You know, how we can access that. You're binding that to the boundary of that product. What else could we bind to it?

Yeah, so first off, I think you're hitting on a key point. The question obviously is, what is the boundary of a data product? And here's the answer. It's not the tables, but the tables are important. I can point to them. But the key thing that defines the data product is what you send back when you hit the /discover endpoint. Point. Okay, whatever you send back, that is the data product. So if I send back the technical contracts, that's how I consume it. If I send back the expectations, the version— I promise not to break over, you know, the backward compatibility, etc.— that's something that defines the behavior of the data product. If I have the SLAs, that sets, you know, sets the expectations around the operating characteristics and performance of my data product. So, so the key thing here is, is the definition of a data product and the boundary around it. You know, the key thing is, is that's manifested and realized the moment you actually expose the /discover endpoint. Because here's how the /discover endpoint gets used. I just— I use, I use the REST terminology, but I think we all know when you hit that endpoint, you're going to get all that data back. and you're going to surface it either to a machine that eventually surfaces it to a, a person, or you're going to provide that, that UX, the, the data product registry UX that I mentioned, um, or it surfaces into something like it sends it to a Calibra or otherwise. But that's the boundary. That actually is the definition of the boundary. And here's, here's the thing about it, is you're formalizing it in black and white. You're formalizing it, and that's the key. So, so here's the, the diff— here's the difference between the data product and the data domain is even if they're granular enough, they're abstract. It's, it's raw data, but it doesn't define any of the SLAs, the expectations, nor the technical consumption patterns. Okay, so in other words, the data domain is not sufficient to, to turn into a product. You have to have those other things— the behaviors, the service levels, the expectations wrapped into it. That's the thing that defines the boundary of a data product. And that's why this, this whole notion, you know, far too often we think only of a data product of, you know, what are the— how do I get data in and how do I get data out? And those are critical, absolutely. But I would argue the single most important capability that a data product has is it can tell you about itself. It's discoverable. The second most important characteristic observable, it can tell you what's happened dynamically in it. That's the things that practitioners need. That's the thing that actually makes a data product real.

Yeah, so when I've been doing it, I've been coming from the beginning, right? I'm at the requirement stage, right? Like I said, I'm trying to do that. And if I think about it on the canvas, there is a box typically of SLA, right? Because we want, from a requirement point of view, we wanna know, do you want this refreshed every second, every hour, every day, every week, every month? Because it's changes the way we build that product. It has an owner, right? Who's the person that's going to make the trade-off decisions, right? When we say, well, you can define, we could do it this way, we could do it this way, here's the cost and consequences, who's going to decide the trade-off decisions for that? So they were all on there. What I hadn't done is made the jump to say, actually, if we then codified that at the end, made all those boundary decisions quantifiable and discoverable, there's massive value there. And so I'm with you, I think there is a bunch of ways we can define what a data product or an information product boundary is. And then as you're talking, I'm thinking, okay, so actually we can get to well-formed boundaries on those products. Now we've got a problem with domains, right? Because they're bigger, right? They're more macro. And why do we want a domain? Well, we want a domain so we know who's going to work on it. Who's going to work in that domain? Who's going to be the subject matter expert? When a product gets requested, that fits in that domain, who's going to do that work? It's an organizational boundary, it's a team topology. So for me now, I'm thinking, well, what we do is we define boundaries for the product, right? The lenses we're going to use to describe that this thing is unique and it's different from another product, and we can describe that. We'll describe those early so we know what to deliver. We'll then codify the delivery of those boundary parameters back via the registry, right? So I can discover and see what how they are different. And then we use team topology to figure out where the domains are, and they may be the organizational hierarchy, they may be the value streams, they may be core concepts, they may be something else, right? Yeah, maybe HIPPO, right? The highest-paid person in the room actually decides where the boundary is because they get all the cool stuff. But those domains really are just a mapping of when a product turns up to be built, which team, Which group of people are going to work on it? How do we know which domain it fits? And that's a good model in my head, right? Because it answers some questions. But it comes back to, you have to define the data product boundaries really clearly at the beginning, right? And you have to present them at the end. That's the key, right? And that's actually easier than defining this big, nefarious boundary of domain in my head. All right, that's been enlightening. So if we have to think about product thinking, Like I said, the idea of an information product and a bunch of canvases actually helped prioritization in the organizations I worked with. So that was kind of an undiscovered, unexpected win, right? It was like, oh, we can actually use this, and it's been very, very successful. What other product thinking behaviors have you seen that this idea of a data and information product allows us to adopt that has value?

Maybe I'm going to repeat myself here, my apologies, but the single most important concept around, again, data boundary and data owner. But the next one in the hierarchy of importance is it actually brings together, you know, consumers and producers. And that pivot from focusing on the consumer, what they need to identify the data and the domain that we will put into this data product, and then using that to find the producers that create that data is a very, very different pivot than what we have today. So if you think about how does a typical enterprise data warehouse get filled up, what happens is, is it's a big landing zone, you know, it's a big honking set of data, and I have, you know, links probably to from hundreds of different source systems, and I have these pipelines coming in And fundamentally what ends up happening is, is I, I lose context. Um, uh, I, I focus on the producers, okay, but the consumers are so far down the pipe that, that I can't actually draw the line necessarily, at least not easily, between what this consumer needs to meet this particular regulatory thing or enter into this type of specific market. I can't actually draw that line to the producer. So what we have is a very disjointed experience. The thing that the data product forces is it starts with the consumer, okay, which is served by a data product and a data product owner and their team, okay. And then it derives the, the sources of the data and ingests those, transforms them as necessary to serve the needs of the consumer. So it's a very bit different pivot from what your typical analytics organization is structured today. And I think that's kind of the organizing principle around a data product is consumer first, producer, you know, form follows function. Based on the consumer, the producers will be self-evident instead of the other way around. Create the producers, put them into a data lake or a data mart or a data warehouse, and then hope that, you know, the theory is if I put enough data in there, somebody will find use in it. That's typically not— you can do that for a short period of time until the cloud costs start to add up. That's not the typical way that you want to do things. Build it and they will come works great in a movie but not in real life. We really need to get to the point where we pivot from what the consumer needs to drive, you know, particular going into a particular market, addressing a particular regulatory need, driving revenue, optimizing, whatever the case may be. The consumer is the driver for organizing the data product, the owner, and the team, and what the what they do. That's the pivot that I think anybody who's come from an analytics organization, shared services, enterprise data warehouse type background, that is completely foreign in many respects to what they're doing today. That's the big aha that I think a lot of folks have as they move in.

Yeah, I agree with the traditional data warehousing way we used to do it, which was build it and they will come. Kept us very busy, right? Team was always busy working on data, but we had very few customers. So I think, you know, that idea of bringing product thinking and bringing customer first, I think we've got to be a little bit careful that we don't move to the one report, one customer paradigm. Yeah. So I think it's that horrible balance, which we've got to be good at. And that's where the idea of a data and information product gives us that boundary, that it's reusable. It's not just a single report for a single person, right? It's something that should be reusable, but picking up that product thinking of engaging with our customers early, So prototype those products and get them in front of a customer for feedback before you go through another iteration on them. Don't build the whole thing and then put it on the shelf and wonder why nobody bought it. I think the other thing is, we've seen a couple of waves in the market over the 30-odd years I've been doing it. And we've seen the analytics engineer wave right now, which smells very much like the self-service reporting wave we had a few years ago. And the way I articulate it, you know, in the days of Tableau and Qlik and when they all came out, we empowered the analysts to do the work without the engineering teams, right? And they had massive, that self-service had massive value to the organization. But what we saw was we saw lots of littles, right? We saw lots of siloed little bits of work being done and it became chaos, right? You ended up with, you know, 3,000 Power BI reports all coming off spaghetti. And my view is we're seeing that with DBT and analytics engineers now, right? We're seeing a whole lot of what they call models, but I'd call code to create a single table, no shared reuse. And if I put my product thinking hat on, you know, we've now basically built 1,001 products that we've put in the warehouse or on the shelf somewhere, and we can't see them anymore. We don't know when they've gone degraded. We don't know when they've gone rotten.

Gotten.

We don't know which ones are being used and bought. We don't know which ones aren't. We're still trying to maintain all these supply chains for these products, whether somebody's using it or not. And so we've got to bring that thinking back in, that a product has value, right? It has value when it's used by a customer or a consumer. If they're not using it, then it has no value. And you typically, you know, if you take product thinking from another, another domain, you cancel that product line. You don't keep producing something that nobody else is consuming because it's not viable, it's not valuable anymore. So I think, you know, in terms of registry, that discoverability, all those metrics about who's using it when, tells us when a product's gone stale and we have to be willing to kill the product, right? It has no value anymore. It's been replaced by a better product, ideally from us, but maybe from somebody else. We're going to see internal data product marketplaces and external ones. And so, you know, there's a big chance now that another company or another vendor will sell a data product to your consumers who are your internal consumers and they find better value value in it. And that's okay. And that's actually one of the things I wanted to go back to. So you talked about shared services. I still see centralized teams and shared services as being valuable depending on your organization. But the key change we have to make is we have to decide that that team is no longer funded based on how many people are on the team. They're funded by the products they sell. And the teams that I've seen really successful are the teams that have a natural salesperson in their team that's out talking to all their consumers in the business. Business and selling them something, going, hey, you know, it's like a little crack, you know, do you want a little sniff of this data? How about I give you some counter customers? Oh, that was good, wasn't it? You know, how about I give you a counter customer by store? Whoa, you know, and, um, you need that, you know, we're selling products, right? We have to excite our customers and make them want to buy them off us effectively.

Yeah, yeah, absolutely. I think like when I think of the shared service organization, I think of of, uh, you know, if you want to have anything done, it goes through a, quote, intake process. And, uh, that intake process, you may actually— if you're lucky, you have visibility into it, but more often than not, you don't. And, you know, magically, 2 weeks later, somebody may call you. You know, 2 weeks is, you know, far too long. So, so I think the, the, the, the problem with shared services is, um, they, they tend towards tend towards bureaucracy. They tend towards this notion of intakes and such. And they remove the engineers from the visibility of what the business is actually trying to do. So, I agree strongly with your notion. I mean, there are certain things where skills are extremely expensive or rare, you have to have a shared service. But when you do it, the ideal way to do it is remove as many obstacles to talking to your customers as you possibly can. And then you have a chance of, first off, understanding what the business is trying to do. And you actually have an opportunity now to prioritize your work to address what the business actually wants, as opposed to, you know, what the shared services organization may do, which largely is, you know, to be honest with their focus is in some respects cost optimization. Okay, the business will And this day, today, every single business that I see in all the industries, it's actually about speed and agility, time to market. Getting stuff out there is what it's all about. So, you know, that is countercultural to the shared services organization. You have to figure out— every organization has to figure out, are you optimizing for cost or am I optimizing for speed and agility? Hands down, if you were to talk to 100 business executives 99 to 100 of them will say, I'm optimizing for speed and agility, you know, and if you can get me time to, you know, to market, I'll pay you just about anything you need to do that. So, so I think, you know, the shared organization thrives, you know, when it does that, and it continues to have to justify its existence when it doesn't do that. And as you know, Conway's Law again comes into play, you know, if the businesses are, or if the shared services several steps removed from the business, the funding and the decision makers don't understand why they're getting, you know, paying as much as they do for what they're getting. You have to know that if you're in shared service and you have to do it, you have to know your business, you have to reach out there. Every single person, like you said, Shane, is a salesperson.

Yeah, and you know, if you think the first thing a shared service group does is put a ticketing system in, and you think in, in the consumer world, what organizations make us use a ticket? Oh, the telcos, right? We log a call to get our phone turned on and it takes 2 weeks. Or, you know, maybe takeaways where we go in and it's busy and we get a ticket, but we get the product pretty quickly, right? It's tickets, okay, because we actually, we can see the queue, right? We know where we are. They yell out, you know, number 99, and we know we're 103. We can estimate how long it is, and we know we're next, right? There's no magic. 104 never gets yelled out before 103, otherwise, you know, bad things happen. So yes, I think you're right. It's, they're not focused on the customer. They think of it as not even a service, it's optimization of the service. Like you said, not a product, right? They don't use that product thinking. So yes, shared services or groups of people working together for other groups is okay, but you still have to treat the other group as a customer and properly as a customer, not as a series of tickets. And yeah, that's the pattern that we should adopt. Look, we've covered a lot in there and that's been great. Just to finish off, you know, if people wanted to get hold of you and talk to you further, what's the best way for them to do it?

Sure, I have email, so it's eric.broda@brodagroupsoftware.com. And feel free to reach out on LinkedIn. It's Eric Broda, all one word. And if you're inclined and want to see some of my articles, it's medium.com/@ericbroda. Again, E-R-I-C-B-R-O-D-A. And I'd be happy to respond to any emails and LinkedIn questions as a result of this. So let me— I just want to close with kind of one thing, Shane, is we talked about data products, and I'm absolutely convinced that this is the way that we're going to be structuring our data organizations going forward. And here's kind of why is is, you know, I actually like the, the name of your podcast, Agile Data, um, uh, and here's why. Um, the, the thing that I pitch is data agility, right? Data agility. Now, now, at one level it sounds kind of trivial, uh, and trite, and in some respects it may be, but here's the thing is that we've been here before, okay? Agile turned into DevOps, turned into DevSecOps, and it changed the way we deliver software. Data was left behind. Okay, now we're pivoting from shared services data organizations into data products. Easy to deploy, easy to build, you know, all that kind of stuff. And they deliver exactly what the business is looking for. It's agility. So we've been here before, and we should definitely learn from the agile practices and the approach they took. Second thing, businesses are pivoting from cost to speed and agility. The shared services team, except for rare circumstances, their days, I think, are numbered. It's not gonna happen immediately. The last thing I would say is, as you go about your data mesh or data product journey, Architecture 101 still matters, and you can get yourself in a lot of trouble by architecting things poorly. And Data Placement 101, still matters. You need to think that through. And the way that I recommend folks to do is this data mesh and building the ecosystem, it literally happens one step at a time, one data product at a time, which really means you gotta frame your journey in terms of incrementalism and test and learn philosophy. If you do those things, you know, I think your odds of success are are very, very high.

Yeah, I agree. And yeah, we've seen these waves before and we've seen some work and we've seen some fail. So, you know, one of my key sayings is agile is not ad hoc. Yeah, we don't just make shit up. So there's a lot of patterns out there, right? There's lean thinking, there's Scrum, right? There's flow, there's product thinking. They're all slightly different. There's DevOps, right? And that's different to all those. So we've got to figure out which pattern we want to adopt. And it's a paradigm shift we're trying to go for, right? It's not a technology change. So we need to share patterns. If I think about why is Scrum being so successful, well, the fact that there's 5 ceremonies that everybody who does Scrum can list off the top of their head what they are, right? It's a known pattern that most people who do Scrum adopt those 5 ceremonies. It's a proven set of patterns. And that's one of the reasons it's been so successful. So, you know, whether it's data mesh or data fabric or, data lakes or data warehouses or data, whatever terms we're going to use, it's the patterns under the covers that people want. It's the things that say, if you do it this way with this context, then it has value, right? So, you know, lots of patterns you picked up, right? The idea of treating a set of things as a warranty against your data product and exposing that to somebody so they know what they can trust and what they can't, right? That's a really good pattern. That we can adopt from another domain, another profession. So yeah, exciting times. Let's hope we don't go down the big data problem where everybody thinks it's just a bunch of Hadoop buckets that you put crap in and it's going to magically do it for yourself. Let's hope we do the hard work to make this one stick and actually have value. And I hope we do. So look, thank you for sharing those patterns. It's been really good. And we'll catch you all later.

Excellent. Well, thank you very much, Shane, once again for having me on the podcast, and I hope we have a chance to talk again soon.

And that, data magicians, was another Agile Data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata.org. Ayo!