AgileData #30 - data products - Eric Broda

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hey Shane, my name is Eric Broder. How are you doing?

I'm good, thank you. Thank you, Eric for coming on the show. And today we got one that I'm pretty excited about. We're going to talk about this thing called data products. But before we rip into that, why don't you tell the audience a little bit about yourself and your background. Sure.

Perfect. First off, thank you very much for having me. So again, my name is Eric Brode. I run a small boutique consulting firm and we have deep skills and experience in data mesh. And as you can imagine, data mesh really at its highest level is just an ecosystem of data products. The things that we actually want to talk about today. I've written about my experiences on medium.com there's probably about 13 articles, give or take around data mesh data products, done a variety of podcasts, conferences, and I'm actually one of the privileged few to who have actually implemented a enterprise data mesh at a global financial services firm. So I have a fair amount of experience, practical hands on experience that I'd like to share today.

Excellent. And yeah, data mesh is definitely the hot buzzword of the 2020s. I tend to talk to people and say I have a black hat and a white hat on it. You know, think the fundamental principles of data mesh are ones we've been striving for for 30 odd years in data, if not longer, and they're just hard, right? And then so that's the good. And then I think there's a bunch of principles and very few patterns out there for implementing it. So lots of vendor washing, lots of mesh washing happening, but one of the areas that has been around for a while and this concept of a data product, and it would be fair to say that the majority of the people I talk to struggle with defining what a data product is, let alone how to implement one. So starter for 10, how would you describe a data product to somebody who didn't know what it was?

Sure. Data product is kind of almost a combination of two topics as you can imagine from the actual term. First off, it's a merging together of this notion of data domains and product thinking. So I think data domains are pretty self explanatory, but it's a related set of data.

And the product thinking really contrasted to project thinking. Projects are short term, they start, they stop, but products have a long term Time horizon. And I think a lot of folks actually understand products almost as well as projects in the sense that most of the clients that I work with, they sell products, so they know all about a product. So if you're at Nike, you have shoes, if you're a bank, you have checking products. So product thinking, believe it or not, is actually embedded into the DNA of almost every company that I work with. I think probably fair to say most companies out there. So it's actually bringing those things together, data domains and products, and being able to be able to provide boundaries around data, have clear owners like products, and make them actually discoverable so that folks can actually do something with them. So that's really the core concept. It is packaging data.

By putting a boundary on it and an owner to it and making it discoverable so that people can actually consume data just like a product. So that's kind of the simplest explanation that I would suggest is out there. But I'm sure it's going to create a few questions.

It definitely is. And so I agree on the idea of boundary. I think, you know, I want to deep dive into the difference between data products and data as a product in a minute because I think that's actually an interesting distinction we should make. But, you know, I've worked with data analytics teams, kind of helping them adopt an agile way of working over the last few years. And one of the things that we were iterating with with each of those teams was this idea of an information product. And the way I described it, you know, it was a while ago that we, we came up with that term and we've iterated in the templates, but it was all based around the boundaries. And at the time it was a boundary of some data.

We didn't tend to use the word domain, but, but I would now it was a boundary of some code and it was a boundary of some output that somebody used. So could be a dashboard, could be a report, but could have been an API, could have been a data service, could have been a file. So being very clear that an information product wasn't always a dashboard, right? Sometimes it was a different delivery mechanism depending on the Persona. And the reason we created that boundary was we wanted to break the work down from being a year into a small iteration or a series of iterations. And so we had to create some boundaries to time box that work. And so that started having successes. We found this term information product people could understand. And then we found 101 other uses where that definition of a boundary were useful. Things like prioritization and so effectively creating a roadmap of information products. What's the next most valuable product?

But our goal was to stop six months of data design, get something out in front of the customer's hands, internal stakeholder or external customer that had value. And we got feedback on early. We didn't go into the whole product thinking consciously, but when we started prioritizing and roadmapping the information products over a time horizon of one to five years, that was a form of product thinking. We didn't start exploring the idea of data as a product. We didn't really explore the idea of observability.

Those kind of features that made products easy to find and easy to buy. And so we should be adopting that. So with that, I always kind of say at the moment, for me, a data product is a subset of an information product where the Persona that's gonna consume it as a data literate person, as an analyst or another engineer. Because in most of the descriptions I see, that's the definition of a data product, right? We're gonna get a domain of data, we're gonna create a thing that somebody else is gonna access via code and use. And I differentiate data as a product as a way of working, right? That product thinking, that discoverability, that treating it as a thing that's gonna survive for a long time or we're gonna kill it.

It's not a one off project, right? It's not just a piece of code that goes to die.

What do you see is the difference between data products and data as a products? Or do you not see a difference.

At one level? I would say it might be semantics, but I think the whole idea around data products enforces the product thinking. Now let me kind of give you, I'm going to give you an analogy around how I think of data products. And the analogy is going to be something that everybody kind of knows and loves. In many respects, it's Airbnb. So let me tell you why I think Airbnb is a perfect analog for a data product. So first off, Airbnb brings together producers and consumers. Those producers are property owners and the consumers are renters. So the first thing in a product is you have to have somebody who creates the thing and somebody who consumes it. And presumably you have some kind of financial arrangement where it's beneficial for all concerned. Okay? Now Airbnb has an absolutely clear boundary, okay? It's the legal structure and it's a service offerings that delineate its boundary. The interesting thing about it, it Also has accountable owners, the shareholders who own Airbnb and the legal, you know, regulatory frameworks that determine how it runs. That's the accountability of the owner. Okay, so Airbnb has clear boundaries. It has an accountable owner, it has an empowered team. Those shareholders have empowered the CEO and the full complement of staff to actually run Airbnb. Now, Airbnb is also a platform, and here's the distinction that I make. It's the data product, which is the clear boundary and the packaging of it, but the ability to surface it into to a self serve type capability. So how does Airbnb, to continue the analogy, do that? Well, first off, Airbnb is known for ease of use, not just for consumers to find and rent properties, but for producers. There's an equally sophisticated user interface and capability and a whole set of aftermarket capability that makes the product, makes the data product easy to use. Okay, In Data's case, it would be easy to find, easy to consume. Next thing is a data product, or product in general, has clear contracts and expectations. Airbnb provides absolutely clear guarantees of service, payment and safety, backed in fact by public contracts and insurance policies. So now the interesting thing about this, and this is where I think the byproduct, the value of the data product actually comes in play, is like Airbnb. Bringing producers and consumers together creates that virtuous circle of market growth. More property listings bring more renters, more renters bring more property listings. So Airbnb is a fantastic analog for data products. Now, the interesting thing about it, now to extend this into a data mesh, Airbnb actually is a ecosystem of.

Airbnb sites and locations geographically, each geographic geography bound by its own set of legal restrictions, regulatory constraints, privacy, etc. So Airbnb is a product.

Operating in a ecosystem of products. So now coming back to Airbnb as a data product, if you will, or the analog for a data product, it has all of the characteristics that I just mentioned when we move into the enterprise. So the key here is everything that I said about Airbnb be.

You could say, are the characteristics of a data product in an enterprise. So now again, semantics. Data as a product or data product? I think data as a product suggests it downplays the intertwining between the data and the whole notion of product thinking. So again with Airbnb mapping to a data product inside an enterprise, clear boundaries. Data product has that clear boundary. It has accountable owners, that's the data product owner, and they are accountable and responsible for delivering the service and expectations. The clear Contracts and expectations that they're out there. They have an empowered team. They create a platform, hopefully, that makes data easy to find, share, consume, and govern. Okay? And they have producers and consumers. Producers. Typically, when you think of a data product in most enterprises, we think about the pipeline that feeds the database. That's the producers. But data products bring together producers, the folks who create the data, whether they're source systems or otherwise, that move the data into the data product. But it also addresses the consumers, and it actually has a. Like any product, when you think about whether it's a banking product or Nike shoes, you're out there to address the consumers. Interestingly enough, data products or data as a product, I think, suggests and emphasizes much more the ingestion of data into the domain. Whereas data products, in my terminology, it's about the ability to bring producers and consumers together to safely interact and in some cases, transact. So again, at one level, it's semantic difference. At another level, data products provide the direct analog to things that we see out there today that can be directly applicable into the enterprise today. Again, I come back to analogies that make it really easy to understand. So, again, data product is Airbnb, the principles of Airbnb inside the enterprise.

Yeah. So I think for me, semantics count, right? Because when we mix our toasties, when we use the same word for different things or people have a different perspective, then we get confusion, and we get that within data domains, you know, how do we create a boundary around the data domain? So I'll come back to that, but I take your point. I mean, your. Your definition of data as a product was nowhere near my definition, right? It was like, yeah, that's not what I meant when I said those words. So, you know, for me, data as a product is our way of working. So I think I will change my terminology, right? I'll talk about either data product thinking or data product way of working, because it is. It is those processes. It's the idea of a data contract, right? It's the idea of enabling the producers as much as the consumers. It's all those things we do when we create this data product. And what I see in the market is lots of confusion because data product sounds like a thing, right? It sounds like a product on the shelf. You know, it sounds like your bottle of Coca Cola or Pepsi. Whereas a lot of the thinking we have is around how we produce that in a certain way and how we make it available for consumption. And I agree with you. I think, you know, Airbnb is a great analogy the shift in where we're going, because that's why we're seeing such a big move towards data marketplaces. A place where you can present your products, you produce your products and you put them up for sale, even internally or externally. Right. It doesn't have to be for money. It can be to a bunch of internal stakeholders and then people consume them. Right. You know, it's effectively the Amazon forward data. And we're going to see that a lot, I think.

But let's go back to that idea of producers. What I see is a large amount of conversation around the consumers of the data products. How do we make it so they can consume that data? How do we make it so they know what it is? One of the core principles of data mesh for me is the idea of decentralizing the data work back into that software engineering team, back into the teams that are producing the data. Because one of the major problems we've had in the data world for many, many years is people create data as a result of their application and it becomes exhaust. And the data teams are kind of capturing that exhaust and trying to make it useful. And so that massive disconnect is where a lot of the effort and the problems actually happen. So if we push all that work back into the producers where they can actually produce data that's fit for purpose, data that I can buy, data that I can use as if it was a product, then we solve a lot of those problems. But I see very little conversation about how we do that. Again, it's all focused about data catalogs or consumption or.

Quality of the data. All those consumption based things, not those producers. Are you seeing that or are you seeing some good work happening and enabling people who don't normally produce it to produce it in a better way?

Yeah, I would actually, I'm going to.

Suggest a slightly different terminology. So the notion of the data product has that boundary. Okay. And the data product team and the owners and the data product platform manages that data. So here's my slight disagreement with what you're saying is it is absolutely not about the producers and it is absolutely not only about the consumers. In fact, I actually see much more emphasis on the producers. In fact, the terminology used is the data product focuses on the producers and I think they're actually separate entities. And the data product brings consumers and producers together. So here's what I mean is, to your point, who actually does the work? Well, typically there's a source system, okay, that, let's just paint this scenario. I have a source system going into a multiple source system, going into a data product and have multiple consumers, okay? The key thing here is the producing systems, whether source systems or other data products, are separate, unique and have their own owner and their own boundary, okay? And if it's done well, whether it's through pipelines or through APIs or queries, there's a way to actually get that data, push it into a pipeline and ingest it. The responsibility for ingesting, okay, that is consuming that other source system or other data products capability is actually the data product team, okay? So they own, end to end, getting the data in. Now obviously they collaborate with a whole bunch of other folks, the source system team or other data products. But their responsibility, the responsibility of the data product team is to get the data, transform it as necessary, store it as necessary, and make the interfaces available to expose that data. So that includes, you know, the discovery, conceptually, a slash discover endpoint, it includes slash consume endpoint, or more likely, in an analytical world, the federated queries that are available to actually access that data. So the only distinction that I make is the data product owner and their team is self sufficient and they interact with other source systems and other data products to do their work and they are accountable for making the consumption patterns, consumption services available to a consuming capability. So what I'm seeing out there is a very significant delineation in terms of responsibility. So most organizations do emphasize the producing pipeline that the data engineering team, and far too often they build that as a shared service. They build all the pipelines that connect all the various data products. And what I find is the moment you go into the shared service capability, the centralized capability, what you end up doing is you introduce unnecessary bureaucracy, you slow things down and your agility diminishes. You know, as it gets more complex. What we're saying as a data product is, I'm saying all that responsibility is the responsibility of data product owner and the team. In other words, they don't have, you know, it'd be silly if they didn't use, you know, vetted enterprise capability. But they don't have to. It's the prerogative of the data product team. They may say, you know what, I don't need data pipelines in. I'm going to go real time and I'm going to listen to a topic now. I'm going to get those things, trickle, feed them in real time into my data product, okay? In other words, the data product owner has the wherewithal, decision making rights and the capability to ingest the data Any way they want. In other words, they don't have to rely on a shared services team. Okay, but they may, if it makes sense, they, but they own the capability and they determine how it actually works. The same corollary on the consumption side sometimes, well, most times actually folks will say I just want to access via SQL, give me my SQL query, leave me alone. And today actually I'm going to say five years ago that probably was the only way that people went. Today what I see is this notion of the real time enterprise. Consumers are saying queries are interesting and useful, but that's not the way I want to do it because I need real time data and I'm not going to issue this query, especially if it's relatively complex. If I'm going out, you know, doing a hundred million by 100 million row join, you know, you're going to be want to do that very, very carefully. What they're saying is I want to and I don't want to ping that thing and execute that too often. I want to actually just listen for when the data in the data product changes. So the paradigm that I'm talking about is the accountability, the ownership and responsibility is in the data product team, not in a shared services pipeline team or a shared services engineering team. Although again, it would be foolish of them not to use capability that makes sense and is standardized. But it's the prerogative of the data product owner and team to set up their data product any way they think is equitable, financially sound and makes sense.

Yeah, so I think you've covered a fairly large number of people, perspectives or lenses in that definition. So I'm with you on a lot of it. So when Shopify published out quite a while ago, their ways of working, doing the finger quotes here, the Shopify model, one of the core tenants of that, one of the core patterns was self organizing teams. And the teams were able to use whatever technology they wanted. And what they said was over time, teams found that rather than standing up their own technology and maintaining it where they saw critical mass in the organization for a technology, they would adopt it because it became easier. The community was there to understand how to use it. The support was there internally, the bodies of knowledge were there. And then over time teams broke off to become shared services teams effectively for that technology to support those other teams. But that was a natural thing as a boundary, as a domain team, you could decide not to follow that path and do whatever you want because you were empowered to produce effectively a product within the application. And for me that is a very successful pattern.

If you start off with your teams as you scale and give them that right to be in charge of their own destiny, they will, nine times out of 10, do the right thing and they will adopt shared patterns because it makes sense to them. You know, humans are inherently lazy, right? We, we don't want to do work we don't need to do. So that's good pattern. I think the other thing that I want to pick up on that is we need to differentiate team topology, our operating model, the way our organizational team hierarchy works, and data products. So I'm, I'm with you still. That data products is a boundary. Yeah. So it's a boundary of who owns it, who produces it, who consumes it, how do they want to consume it, what's the data that's in there, what's the data that's not in there, what's the code, what's the observability? There's a whole lot of patterns, lenses we should put across it that give us that boundary. But we can have a single centralized team that's producing data products if we wish. We just have to deal with the consequences of we now have a backlog that's massive because there's only one small team. We can have a bunch of centralized teams, pods, squads, whatever you want to call them, who each own a set of data products or a domain, and that's okay. Right? And we can have them using a shared platform or not using a shared platform. It's all okay. So I think with Data Mesh, one of the things is it talks about the four principles, which is great, and then it brings in a fully decentralized topology of pushing all the producing work out to the software engineering teams. And I think that is where we should go to. Right? That's nirvana, but we've never managed it. But I think that's also confusing the market, the difference between the principles of a data product and that team topology. I just want to jump onto the idea of data contracts. So I did do a shout out on LinkedIn a while ago and said, could somebody give me a template for a data contract? Because I didn't have one and I actually needed to put one in place for some stuff that we were doing as part of our startup, and I got a couple of examples, but not a lot. So that idea of a data contract, you know, I take your, your analogy, right? The, the Airbnb has a contract for its producers and a contract for its consumers, and, you know, you sign up to it, it's well formed, you might not like it, but it's still there. Right, but we don't have that right. We don't have patents for data contracts widely available in the, in the data world. Or have you seen it differently? I mean, have you seen some really good data contracts out there?

So I think it is a gap today, but let's kind of unpack that a little bit. I suppose when I think of a data contract, there's kind of minimally three things that you need to think about. One is obviously there's a tech set of expectations. Contracts, for example.

SQL would be.

That'S one way to. And it has its own set of contracts and expectations. You may use APIs, use open APIs specification. So the tech side is one element of it. There's expectations around, you know, change. For example, that there's an expectation that, you know, what works today will work tomorrow and that there's some level of backward compatibility. There's also hard things like SLA service level agreements or expectations where, you know, you know, 99.8% of the queries will happen within two seconds or whatever that may be. So I have not seen anything that spans all of those, to be honest with you. But what I would say is people are starting to understand that when you look at a data product, it is the vehicle to bring all those things together. So I'll give you a simple example. At several of my clients we've implemented this notion of a data product registry. And what the registry does is, and it's a prototype at this point in time, you know, undergoing some, some industrialization, may one day be product, who knows. But ultimately what the registry tries to do is, and it's different from a catalog and you'll see why. Like it's not a calibra, it's something very different. But what it tries to do is say, you know, if I want to go and find the data, okay, I should be able to have some way of actually searching for all the data products that are out there, finding the one I'm looking for, double clicking on it. And I should be able to see much more than just the database schema and the tables and the columns and all that. Even there. You should be able to have which columns are sensitive or whatever, but it should say who's the owner? So who do I call if I actually want to get something? What are the SLAs, what are the. Now here's the real thing. What are the actual access mechanisms? Now when you boil down data mesh and kind of What I do for a living, data mesh is a set of principles, data products being the architectural quantum, what I try and do with my clients is I turn the practices in the principles into practices. So this notion of a data product registry actually crystallizes it for all. It's like when you think of Airbnb and you go and search for a rental property. You can put any level of filtering, geographic location, price, etc. And hit go and it gets you this beautiful thing and you can find properties anywhere in the globe. Same idea. With a data product registry. I should be able to have that simple an interface. And I should be able to navigate from the forest, find the tree I'm looking for, drill down, find the of the forest of data products. I should be able to find the data product that I want. Okay. And I should be able to now look at the service level agreements, the hard, hard agreement. I should be able to look at the service level expectations, backward compatibility, etc. We promise not to break things or whatever. And I should be able to look at the tech expectations, what are. Again, if I look at the, the analytical space, if I'm trying to do a join of a 100 million row table with another 100 million row table, which, you know, you think is not that frequent. It's actually more frequent than people think about it, especially in financial services. You don't want to have people doing that just, you know, at the drop of a dime. You want to have those queries vetted, performance tested. All the data has been scrubbed for all the regulatory and privacy concerns. In other words, you want to have vetted queries. So that's another set of capabilities that is exposed in the data product. In other words, the data product is the container that houses the data, but it also houses all of the formal and informal expectations, service level agreements and access mechanisms all available. That in effect, is what I think of a data product contract. There's nothing out there that has that today, although some of the work that I'm doing at my clients, we're pretty darn close to instantiating that. But that's what I think of the contracts. There's no vendor product out there today that does it. There's no formal spec that does it. But if you're able to think through it a little bit, you can actually piece the puzzles. You can actually put the puzzle piece together so you can actually see something that makes sense.

So I really like that. I haven't had that lens before.

One of the things we know we do badly in the Data world is we confuse people who aren't data literate. Yeah, we talk about, you know, in the old days, CD2, but you know, now we talk about mesh and you know, fabric and APIs and all these words that consumer of a product doesn't care about. Right. They just want the product. So I heard. So I like that idea that we can use consumer based terminology to describe what they get. And as I was thinking of those words you were using, you know, that's effectively some of one of the things that you were talking about is the warranty. You know, when I buy a product, I get a 1, 2, 3 year warranty. I get a promise of how long this thing's going to survive. So I heard that.

Right?

I heard that form of a warranty. One of the other things, whenever I'm coaching a team, I get them to start off first is what I call definition of done. And it's a little bit different for the data team than what we would, what typically you do in scrum. So for me, definition of done is not what the product owner is asking for. It's the set of criteria that as data professionals, we expect to do when we build data data products. So, you know, our consumers, our product owners expect us to test our code. They don't expect, you know, to have to tell us to test our code. They expect us to validate the data. They expect us to make sure that we haven't lost any. There's a whole lot of expectations that you would expect a data professional to do. And that for me is that definition of done. And I can take those definitions and I can then think about, okay, if we had this registry of data products, those should be things that we have green ticks on. Right? You know, we have done these things to produce this product. It's almost like a QA process on a factory, right. To say, you know, it's passed all the tests for that product to go out and be boxed and shipped. So I like that. Right. I like that analogy. I think we're going to see a lot of confusion between registry and catalogs. Right. Because they are similar, but they're not the same.

Yeah, well, let me, let me just, I want to kind of touch on that real quickly because you did mention it. First off, there is, there is confusion in the market. That's why I purposely called it a registry. And you know, point well taken. Maybe I'm not adding, maybe I'm not adding clarity as I do that. But let me, let me explain why I did that. So first off, you know, let's start at, you know, basic principles. What is a domain? So typically, you know, the domains are defined by, you know, the chief data office in a typical large organization, but they're macro level domains and they're, they're quite granular, okay, coarsely grained. Rather, the practitioner wants to have something that is finely grained, something that I can use. Well, lo and behold, the data product is exactly that. It's, it's designed for consumption and the production ingestion of the data should follow the consumption of that. So first off, domains are different governance domains, which is the traditional data domains in an organization, which is different than the data product, which is a practitioner focused. Again, the audience of the data product is the developer, the data scientist and the analyst who has a practical job to do. The job of the data domains at the CDO level are to ensure that we're doing the right thing with our data, setting the policies that the data product owner and their team actually have to implement. So by its very nature, data products have a finer, much finer grained domain. So now if I think about applying that to data product catalog, the confusion comes in when people think of a data catalog. You know, there's products out there like calibrated, okay? And Collibra is a great product for what it does, but let's make no mistake about it. It's a governance tool independent of how people want to, you know, modify, change it, use or abuse it. It's a governance tool, okay? And it's designed that way. The data product needs something that is a practitioner tool. So for example, let's come back to the things I mentioned earlier. Calibra, maybe it's definitely can, can provide the schema of the data product. Absolutely. You probably can actually identify the owner sometimes you may be able to define the service level expectations, but it's a little bit ad hoc.

It largely is, is agnostic to this notion of expectations, versions, backward compatibility because it doesn't talk about the practical implementation. Once you go into the implementation, what are the APIs, what are the queries, what are the events that I'm listening for? That's nowhere to be found in the governance tools. It's a different problem you're solving. We're solving Collibra is something for the CDO and the governance team and it's a super valuable product for them. Great product for them, but we shouldn't confuse that with what the practitioner needs. The practitioner needs very practical. How do I actually consume this darn thing? What are the vetted SQL queries, what are the APIs, what are the events that I can listen to in addition to the expectations and the SLA stuff that I mentioned earlier. So I call this thing that exposes that information. I call that the Registry. Now, we too often think of the Registry as just a user interface. The Registry is two things. It's a user interface and it's, for example, the slash Discoverer endpoint, the slash Observe endpoint, the slash usages endpoint, the slash logs endpoint. It's the slash alerts endpoint, each of which, for example, alerts what happened last night, what are the bumps in the road and can I actually see, you know, what actually occurred? So I can look at the alerts. I can also look at the logs to maybe do some diagnosis. I can look at the usage to say maybe they're, you know, the. We had a huge amount of new users that came on board and that's why the performance was poor. And I can do a slash observe to see what else happened in my data product. All those things are wrapped around this notion of a data product. I call the Registry not only the user interface, but that set of Discover Observe endpoints that make it machine readable, if you will. So that's kind of the very, you know, the distinctions that I draw.

Yeah. So I won't go off on a rant about governors by committee and how unvaluable it is. So again, I can take a whole lot of other words and use them to describe what you're talking about. Right. And they all have value. So you talked about a lot of data ops behavior. That idea of logging, that idea of audibility, that idea of monitoring and exposing performance, anomaly alerting.

Right.

All those things that we should do for a data product, they're all valuable. Right. I mean, we have a massive problem at the moment that we're, you know, the market's unbundled everything except for the database. I always confused that we say we're in an unbundled world, but everybody uses Snowflake, which is highly bundled. So. So we're going to see that. I think the word Registry, for some reason, whenever I've heard the word registry, I've read the word proto buff, which I don't actually understand what it is and I've never bothered to go and look, but it just sounded like a technical word. But I do remember a data, a really light data catalog tool that was out there for a while. I think it was the one that click, bought and embedded into their product. And I Think they lost this feature. But what it did, it had this feature that I thought was quite interesting. And the idea was it treated. There were a bunch of tiles and it looked like Amazon effectively. Right. And so each one of those tiles was effectively a dashboard. You know, it would be a form of light, form of data product. Right. It had a boundary of data, it had a boundary of code that transformed that data, and it had a way of delivering that data, typically in a dashboard. And then what you could do if you wanted access to it, you dragged it to a shopping cart, and when you hit buy, it actually sent off a request to the owner of that data, the person that was recognized as the owner or the steward, and then they approved your access. Right. And did the RBC change for you automatically. And I like that. Right. I like that analogy because it was like me buying something from Amazon. Yeah, it was. I could treat that piece of data as a product to get access. So. So I'm with you. I think this idea of extending the paradigm or the pattern of a catalog from just being governance or just being seeing the columns or just being, you know, identifying the PII data to having all those lenses. Right. All those things that a good product should have.

Yeah. So just to build on that, like, like I'll give you an example. At some of my clients, what we've done is we've looked at it and said, not only do that does the data product registry have all the capability. We said, but suppose, lo and behold, somebody actually finds the data and they. They want to actually access it. What do they do? Well, in the old world, you look through your Outlook organization chart to figure out who in security may be able to call. Hopefully you get the right person who will refer you to another right person who may be the right person at some point. Anyway, we try and avoid that telephone tag by allowing folks to actually be able to create an access request, attach it to the registry, facilitates that. So I can now, so I'm just giving you a few of the highlights, but there's a lot of process capability that you can layer into this data product registry that makes it. Again, the mission I have is to make data easy to find, consume, share and govern. Okay. And.

Getting access to the data is fundamental to actually achieving that. Now, what the registry also can do is our goal was recognizing again, our data products would be relatively finely grained. We wanted to be able to spin up just like if you want a cloud instance, you know, a VM or whatever in Amazon or aws, you know, it's all user interface. A few clicks and lo and behold. So part of what we wanted to do is we wanted to say we should be able to have a step by step user interface guide that walked them through spinning up a brand new data product. So you'd sit there and say, here's the owner, it's a postgres database. Here's the URL for the postgres or the configuration for postgres and hit go and it created automatically the Discover Observe usage endpoints for, for you and you filled in a little bit of data and literally within five minutes you could have a data product. So, so where I'm going with it is that the registry is, is again. The mission that I have is, is make data easy to find. So that's the search capability, make it easy to consume as I offer the list of APIs, et cetera, make it easy to share the APIs, the SQL and stuff does that. But it also make it easy to govern and operate. So, so being able to, you know, find the data and create an access request, being able to spin up a brand new data product in five minutes or less, that's all achievable today. And we, we bundle that capability, right or wrong with the data product because it comes with all the benefits of having clear boundary, an owner and a team associated with it. So, so it just made eminent sense that that's how we actually structured it. And I think that's where I'm, you know, if I were to look, my crystal ball gets kind of foggy after I think six months ahead. But I think that's where data products are actually going. You're going to have, they're going to focus on the developer, data scientist, analyst, user experience and be able to make it easy to again find, consume, share and govern data. And that's what the data product registry's mission in life is.

Yeah. And so I'm going to go back to my definition of data product versus information product. So you gave a very good description of a data product. Right. You talked about the Persona, you know, it's the analyst, it's a data scientist. You taught about the data experience that they wanted. Right. Find some data, stand up the endpoints, get access in that. If I talked about, you know, chief marketing officer, their experience has to be different, right. They want an information product, they don't want to know, Sorry, maybe some do, but most of them don't want to know about an endpoint. Right. They, they just want to know that they have some data to consume. You Know, they want a view of it, they want to consume it slight different. So their experience has to be different. And that's okay. We just got to be clear that that changes the boundary, right? The, the boundary for a data product that's focused on an analyst or a data scientist or an engineer is a different boundary that's focusing on a consumer that is more at the business level. And that's okay. They are two different products. They may use the same data, right? They may use, reuse the same platform, they may reuse a whole lot of the moving parts, the Lego blocks, I call them. But they're a different product because we have a different customer and the experience them should be different. So for me, that's, you know, we've got to bring in the Persona, the consumer of that product as one of the tenants of the boundary we create when we define that product.

Absolutely, 100% agree. This is, again, it's pivoting from a producer. How do I find the data and get the data into the database from, you know, pivoting and saying what does the consumer need? And I'll go get the data and make sure I can transform it to address that. The consuming need. A slightly different perspective, but I, I agree 100% with where your head's at.

Yep, excellent. And so let's go back to data domains, right. Because this again is a, is a tricky one. So I was intrigued that I, I'd never quite thought or seen the data domains coming down from a cdo. I think in New Zealand we don't have a lot of cdo, right. It's, it's still relatively new to us and if we do, they don't really act like CDOs we see over.

So what I tend to see is there's a couple of lenses with which data domains get defined. So the first one is, I think it's Conway's law, right. In terms of everything will behave the way the organization does. So the organizational hierarchy will define the domains. There's marketing, there's sales, there's finance, there's HR or people.

I see another lens, which is core business process. If we have a supply chain, if we have an HR leave process, those kind of boundaries where there's a beginning and an end and data moves through it to achieve some core business processes, I can see those often become domains. Another way to do it is to think of it as concepts. We have the concept of a customer and the concept of a supplier and the concept of an employee and their domains. We're Going to get lots of core business events. It's going to span across multiple organizational hierarchies or silos. And that's our definition of a domain.

Those are the ones that I typically see. And source specific silos. You know, we have Salesforce, that's a domain. We have HubSpot, that's a domain. We have, you know, SAP, Oracle Finance, that's a domain. So those are the lenses I've seen. What do you see, what do you see and what do you use to get a good definition of a data domain?

Yeah, sure. The, the very first one you mentioned, I really, I'm very glad that you mentioned Conway's Law.

It is, it is prescient in terms of how, how, how true it actually is. And it was created by Mel Conway, what in 1960s or something like that. But it's been so spot on. But here, here's the lesson learned. If you don't follow Conway's Law, okay, you're going to be swimming upstream, okay? And like anybody swimming against the current, you're going to get fatigue. And in terms of projects or the longer term funding, you get funding fatigue, okay? So here's, therein lies the problem with shared services and centralized organizations. Who's the owner? Don't know. But it's definitely not the folks that have the money and the, the, the. And it definitely doesn't follow the organizational decision making tree, which is really the org chart kind of shows that. So, so going against Conway's law is, is you really gotta think through why you're doing that. And I think today, you know, I mean, if you think about why people created shared services is because the skill set was very hard to find or extremely expensive and you wanted to share it. Most of the technology that we're talking about today is not in that camp anymore. Okay. Now, you know, maybe it's not too easy to find an AWS engineer, but it's, it's, you know, there's no need to have a shared services built around that anymore today. So first off, Conway's Law, absolute fantastic hint for where your data products are. Second thing is business processes. I tend not to find those because what they do is they actually swim across organizational lanes. So they kind of violate Conway's Law in some respects. That's not to say it doesn't work, but it has to be a super important process for it to actually work well. And you have to have an extremely strong owner of the actual business process to make it fly more often. What I, what I find is something that most organizations or many of them do, at least the larger ones is a business architecture. Okay, so, so it's kind of the, the, you know, I've seen hundreds of these things but they're, they're PowerPoint where. But they're very useful in the sense that they identify the domains. Now they happen to be usually one level below the governance level domains. So for example, client is a governance domain. Client is so abstract can mean anything to anybody. But a business architecture would probably go down and say in this geography here's what a client actually means and here's all the other capabilities. So business architecture actually is the next level of granularity where I look to, to define my domains. Fortunately, most of them line up to the organizational structure. Again, Conway's law coming in. Now there's the interesting thing I find is depending on what industry you're and I'm in financial services. So there's a lot of pre canned models that give you tons of hints. So Bian B I A N is one that I've seen many times before. It's, it's financial, it's, it tells you the business architecture for an Internet or global banking organization. I've seen things like Teradata's financial services data logical data model. The data model isn't as interesting as the fact that it has this huge glossary of, of all of the business entities which again are organized typically around the organizations that you typically find within an organization. And again the last one, and perhaps the hate to say it, but sometimes the least useful is the domains that the chief data officer comes in defines. And again it's not that they're not appropriate, it's just they're addressing a different need. They're addressing a governance which is absolutely crucial. Don't get me wrong, it's needed but it's too far removed from the practitioner that it offers some useful hints but you have to go several levels deep down into the organizational structure before you can get some, something useful. So that, that, that, that is like to be honest with you, it's, it's one of the very first questions that always get asked at my clients which is I love this data mesh thing and I think these data products are fantastic. How do I find one? So, so there's, there's no. I'll be honest with you, it's a little bit of my secret sauce, I suppose my consulting business there, you know. But it's not necessarily rocket science. There are some pretty solid hints that if you're aware of simple things Like Conway's Law, you can get to, you can find your data domains, your data product domains or data products rather a lot easier than you may think.

Yeah, it's interesting, isn't it? I'm now worried that we're going to see the enterprise data mesh model. So I remember the days of Oracle, IBM Teradata where you could buy the banking model and everybody was proud of how many walls of paper you could put up there for that model. With typically the horrible party is a party of a party and you know, it was a very expensive piece of paper that was very hard to implement.

So I think for me, I agree with you that first thing you should do is observe the organization. So typically if I observe an organization, I hear the term value streams or customer journey and then I keep observing and I see the organization actually behave that way. It's not just look service then that idea of having a domain based on core business processes has possibilities because what they're saying is they want to work in that end to end way, but if they don't, right. If they have organizational hierarchies, then yeah, yeah, you need to bound your domains to that. I think the other thing is, and it's something I haven't really been thinking about, but I will now, so it's probably one of my big takeaways for this is as I said in the beginning when I was working with organizations and we defined an information product, right, we, we wanted, our goal was to break the work down into smaller chunks, right? That was how do we take this lifetime value thing that we know is 12 to 24 months worth of work? How do we break it down into smaller parts, right? How do we break it down to say, well actually there's a definition of revenue that we need to do and there's a definition of expenditure and there's a definition of margin and a definition of product profit. There's a definition of churn, there's a whole lot of these Lego blocks that we need to put together to do lifetime value. And each one of those takes time and effort. They're not easy. So how do we break that work down? And that's one of the first rules of the boundary was decompose it down into small iterations. As a result of that, we found some value. We found that we could define a canvas which took us 15 minutes to fill out. And we found that actually the canvas was so simple because it was based on the business canvas, which is a thing of beauty, that actually the product owners would be able to pick it up and do it themselves. So they'd bring the canvas pre filled out to the team and start the conversation with that. Then we found out that we could use that canvas, that, that boundary to define some form of value that could be prioritized. So we started talking about, okay, if this, if this product was delivered, what value is there? And everybody went, oh, blah, blah, blah, right. It was all became very high level, you know, increased revenue, better customer experience. We, okay, well that's, that's not working for us. Right. So we, we asked questions of what was the action you're going to take? So then it was like, oh, okay, well this is a churn product. What we're going to do is we're going to make a offer to the customers that are expected to tune in the next month and we expect to retain, you know, a certain amount of revenue and increase, you know, retain a certain amount of margin as a result of that. So we're like, cool, there's the value, right? The action you're going to take informs the value value. So if we think about actually creating a, a checklist, almost a lens of what the drivers of the boundary are, that, that would be valuable. Right. So you know, you talked about binding, you know, the warranty, the definition of done. Well, that's not what you use, but the words I'm using. Right, but that sla, that warranty, right, that definition of done, that expectations and the technology, how we can access that, you're binding that to the boundary of that product. What else could we bind to it?

Yeah. So first off, I think you're hitting on a key point.

The question obviously is what is the boundary of a data product? And here's the answer. It's not the tables. The tables are important. I can point to them. But the key thing that defines the data product is what you send back when you hit the discover endpoint. Okay. Whatever you send back, that is the data product. So if I send back the technical contracts, that's how I consume it. If I send back the expectations, the ver, I promise not to break over, you know, with backward compatibility, etc. That's something that defines the behavior of the data product. If I have the SLAs that, you know, sets the expectations around the operating characteristics and performance performance of my data product. So, so the key thing here is, is the definition of a data product and the boundary around it. You know, the key thing is, is that's manifested and realized the moment you actually expose the slash discover endpoint. Because here's how the stash Discover endpoint gets used. I just, I use, I use the rest terminology. But I think we all know when you hit that endpoint you're going to get all that data back and you're going to surface it either to a machine that eventually surfaces it to a person or you're going to provide that ux, the data product registry UX that I mentioned or it surfaces into something like it sends it to a calibra or otherwise. But that's the boundary. That actually is the definition of the boundary. And here's the thing about it is you're formalizing it in black and white. You're formalizing it and that's the key. So here's the difference between the data product and the data domain is even if they're granular enough, they're abstract, it's raw data. But it doesn't define any of the SLAs, the expectations nor the technical consumption patterns. So in other words, the data domain is not sufficient to turn into a product. You have to have those other things, the behaviors, the service levels, the economic expectations wrapped into it. That's the thing that defines the boundary of a data product. And that's why this, this whole notion, you know, far too often we think only a data product of, you know, what are the, how do I get data in and how do I get data out? Those are critical. Absolutely. But I would argue the single most important capability that a data product has is it can tell you about itself, it's discoverable, the second most important characteristic, it's observable. It can tell you what's happened dynamically in it. That's the things that practitioners need, that's the thing that actually makes a data product real.

Yeah. So when I've been doing it, I've been coming from the beginning, right. I'm at the requirement stage. Like I said, I'm trying to do that. And if I think about it on the canvas, there is a box typically of sla. Because we want, from a requirement point of view, we want to know do you want this refreshed every second, every hour, every day, every week, every month? Because it's changes the way we build that product. It has an owner, right. Who's the person that's going to make the trade off decisions, right. When we say, well you can define, we could do it this way, we could do it this way. Here's the cost and consequences, who's going to decide the trade off decisions for that? So they were all on there What I hadn't done is made the jump to say, actually if we then codified that at the end, made all those, those boundary decisions quantifiable and discoverable, there's massive value there. And so I'm with you. I think there is a bunch of ways we can define what a data product or an information product boundary is. And then as you're talking, I'm thinking, okay, so actually we can get to, well, form boundaries on those products. Now we've got a problem with domains, right, because they're bigger, right? They're more macro. And why do we want a domain? Well, we want a domain so we know who's going to work on it, who's going to work in that domain. He's going to be the subject matter expert. When a product gets requested that fits in that domain, who's going to do that work. It's an organizational boundary, it's a team topology. So for me now, I'm thinking, well, what we do is we define boundaries for the product. The lenses we're going to use to describe that this thing is unique and it's different from another product. And we can describe that, we'll describe those early so we know what to deliver. We'll then codify the delivery of those boundary parameters back via the registry, right? So I can discover and see what, what, how they are different. And then we use team topology to figure out where the domains are. And they may be the organizational hierarchy, they may be the value streams, they may be cool concepts, they may be something else, right? Yeah, it may be Hippo, right? The highest paid person in the room actually decides where the boundary is because they get all the cool stuff. But those domains really are just a mapping of when a product turns up to be delivered, to be built, which team, which group of people are going to work on it, how do we know which domain it fits? And that's a good model in my head, right? Because it answers some questions, but it comes back to, you have to define the data product boundaries really clearly at the beginning and you have to present them at the end. That's the key, right? And that's actually easier than defining this big nefarious boundary of domain in my head. All right, that's been enlightening. So if we have to think about product thinking, like I said, the idea of an information product and a bunch of canvases actually helped prioritization and the organizations I worked with. So that was kind of.

An undiscovered, unexpected win, right? It was like, oh, we can actually Use this. And it's been very, very successful. What other product thinking behaviors have you seen that this idea of a data and information product allows us to adopt that has value?

Maybe I'm going to repeat myself here, my apologies, but the single most important concep around again, data boundary and data owner. But the next one that in the hierarchy of importance is it actually brings together, you know, consumers and producers. And that pivot from.

Focusing on the consumer what they need to identify the data and the domain that we will put into this data product and then using that to find the producers that create that data is a very, very different pivot than what we have today. So if you think about how does a typical enterprise data warehouse get filled up.

What happens is it's a big landing zone, it's a big honking set of data. And I have links probably from hundreds of different source systems and have these pipelines coming in. And fundamentally what ends up happening is I lose context.

I focus on the producers, but the consumers are so far down the pipe that I can't actually draw the line necessarily, at least not easily, between what this consumer needs to meet this particular regulatory thing or enter into this type of specific market. I can't actually draw that line to the producer. So what we have is a very disjointed experience. The thing that the data product forces is it starts with the consumer, okay, which is served by a data product and a data product owner in their team, okay. And then it derives the the sources of the data and ingest those, transform them as necessary to serve the needs of the consumer. So it's a very different pivot from what your typical analytics organization is structured today. And I think that's kind of the organizing principle around a data product is consumer first producer.

Form follows function based on the consumer. The producers will be self evident instead of the other way around. Create the producers, put them into a data lake or a data mart or a data warehouse and then hope that theory is if I put enough data in there, somebody will find use in it. That's typically not. You can do that for a short period of time until the cloud costs start to add up. That's not the typical way that you want to do things. Build it and they will come. Works great in a movie, but not in real life. We really need to get to the point where we pivot from what the consumer needs to drive particular going into a particular market, addressing a particular regulatory need, driving revenue, optimizing. Whatever the case may be, the consumer is the driver for Organizing the data product, the owner and the team and what they, they do, that's, that's the pivot that I think anybody who's come from an analytics organization, shared services, you know, enterprise data warehouse type background, that is. So that is completely foreign in many respects to what they're doing today. That's the big aha that I think a lot of folks have as they move.

Yeah, yeah, I agree with the traditional data warehousing way we used to do it, which was build it when and they will come. You know, kept us very busy. Right. You know, team was always busy working on data, but we had very few customers. So I think, you know, that idea of bringing product thinking and bring customer first, I think we've got to be a little bit careful that we don't move to the one report, one customer paradigm. Yeah. So I think it's that horrible balance which we've got to get good at. And that's where the idea of a data and information product gives us that boundary that it's reusable. It's not just a single report for a single person. Right. It's something that should be reusable. But picking up that product, thinking of engaging with our customers early. So prototype those products and get them in front of a customer for feedback before you, you know, you go through another iteration on them. Don't build the whole thing and then put it on the shelf and wonder why nobody bought it. I think the other thing is, you know, we've seen a couple of waves in the market over the 30 odd years I've been doing it. And you know, we've seen the analytics engineer wave right now, which smells very much like the self service reporting wave we had a few years ago. And the way I articulate it, you know, in the, in the days of Tableau and Qlik and when they all came out, we empowered the analysts to do the work without the engineering teams.

Right.

And they had massive, that self service had massive value to the organization. But what we saw was we saw lots of littles, right? We saw lots of siloed little bits of work being done and it became chaos. Right. You ended up with, you know, 3,000 Power BI reports all coming off spaghetti. And my view is we're seeing that with DBT and analytics engineers now. Right. We're seeing a whole lot of what they call models, but I'd call code to create a single table, no shared reuse. And if I put my product thinking head on, you know, we've now basically built 1001 products that we've put in the warehouse or in the shelf somewhere and we can't see them anymore. We don't know when they've gone degraded, we don't know when they've gone rotten. We don't know which ones are being used and bought. We don't know which ones aren't. We're still trying to maintain all these supply chains for these products, whether somebody's using it or not. And so we've got to bring that thinking back in that a product has value, right? It has value when it's used by a customer or a consumer. If they're not using it, then it has no value. And you typically, you know, if you take product thinking from another, another domain, you cancel that product line. You don't keep producing something that nobody else is consuming because it's not viable, it's not valuable anymore. So I think, you know, in terms that registry, that discoverability, all those metrics about who's using it, when tells us when a product's gone stale and we have to be willing to kill the product, right? It has no value anymore. It's been replaced by a better product, ideally from us, but maybe from somebody else. Maybe we're going to see.

Internal data product marketplaces and external ones. And so there's a big chance now that another company or another vendor will sell a data product to your consumers, who are your internal consumers, and they find better value in it, and that's okay. And that's actually one of the things I, I wanted to go back to. So you talked about shared services. I still see centralized teams and shared services as being valuable, depending on your organization. But the key change we have to make is we have to decide that that team is no longer funded based on how many people are on the team. They're funded by the products they sell. And the teams that I've seen really successful are the teams that have a natural salesperson in their team that's out talking to all their consumers in the business business and selling them something going, hey, you know, it's like a little crack, you know, do you want a little sniff of this data? How about I give you some counter customers? Oh, that was good, wasn't it? You know, how about I give you a counter customer by store, you know, and you need that. You know, we're selling products, right? We, we have to excite our customers and make them want to buy them off us effectively.

Yeah, yeah, absolutely. I think like when I think of the shared service organization, I think of, of you Know, if you want to have anything done, it goes through a, quote, intake process. And that intake process, you may actually, if you're lucky, you have visibility into it. But more often than not, you don't. And you know, magically, two weeks later, somebody may call you. You know, two weeks a is, is, you know, far too long. So, so I think the, the, the problem with shared services is they, they tend towards bureaucracy, they tend towards this, this notion of intakes and such, and they, they remove the engineers from the visit, you know, they, from the visibility, what the business is actually trying to do. So I agree strongly with your notion. I mean, there's certain things where skills are, are extremely expensive or rare. You have to have a shared service. But when you do it, the, the ideal way to do it is, is remove as many obstacles to talking to your customers as you possibly can. And then you have a chance of first off understanding what the business is trying to do. And you actually have an opportunity now to prioritize your work to address what the business actually wants as opposed to, you know, what the shared services organization may do, which, which largely is, you know, to be honest with their focus is in some respects, cost optimization. Okay? The business will, and this day, today, every single business that I see in all the industries, it's actually about speed and agility. Time to market, getting stuff out there is what it's all about. So, you know, that is countercultural to the shared services organization. You have to figure out, Every organization has to figure out, are you optimizing for cost or am I optimizing for speed and agility? Hands down, if you were to talk to 100 business executives, 99 to 100 of them will say, I'm optimizing for speed and agility, you know, and if you can get me time to, you know, to market, I'll pay you just about anything you need to do that. So, so I think, you know, the shared organization thrives, you know, when it does that, and it continues to have to justify its existence when it doesn't do that. And as you know, Conway's law again comes into play. You know, if the businesses are or if the shared services several steps removed from the business, the funding and the decision makers don't understand why they're getting, you know, paying as much as they do for what they're getting. You have to know that if you're in shared service and you have to do it, you have to know your business. You have to reach out there. Every single person, like you said, Shane is a Salesperson.

Yeah. And, you know, if you think the first thing a shared service group does is put a ticketing system in and you think in the consumer world, what organizations make us use a ticket? The telcos. Right. We log a call to get our phone turned on and it takes two weeks or, you know, maybe takeaways, where we go in and it's busy and we get a ticket, but we get the product pretty quickly. Right? It's. It's tickets. Okay, because we actually, we can see the queue, right? We, we know where we. They yell out, you know, number 99. And we know we're 103. We can estimate how long it is and we know we're next. Right. There's no magic. 104 never gets yelled out before 103. Otherwise, you know, bad things happen. So, yes, I think you're right. It's, it's. They are not focused on the customer. They think of it as not even a service. It's optimization of the service, like you said, not a product. Right. They don't use that product thinking. So, yes, shared services or groups of people working together for other groups is okay, but you still have to treat the other group as a customer and properly as a customer, not as a series of tickets. And that's the pattern that we should adopt. Look, we've covered a lot in there and that's been great. Just to finish off, if people wanted to get hold of you and talk to you further, what's the best way for them to do it?

Sure. I have email, so it's Eric. So e r I c.brodabro d a brodagroupsoftware.com.

And feel free to reach out on LinkedIn. It's Eric Broda, all one word. And if you're inclined and want to see some of my articles, it's medium.com ericbroda Again, Eric B R O D A. And I'd be happy to respond to any emails and LinkedIn questions as a result of this. So let me. I just want to close with kind of one thing, Shane, is we've talked about data products and I'm absolutely convinced that this is the way that we're going to be structuring our data organizations going forward. And here's kind of why is.

I actually like the name of your podcast, Agile Data.

And here's why. The thing that I pitch is data agility.

Data agility. Now, at one level it sounds kind of trivial and trite, and in some respects it may be, but here's the thing is We've been here before. Agile turned into DevOps, turned into DevSecOps, and it changed the way we deliver software. Data was left behind. Okay? Now we're pivoting from shared services, data organizations, into data products. Easy to deploy, easy to build, you know, all that kind of stuff. And they deliver exactly what the business is looking for. It's agility. So we've been here before and we should definitely learn from the Agile practices and the approach they took. Second thing, businesses are pivoting from cost to speed and agility. The shared services team, except for rare circumstances, their days, I think, are numbered. It's not going to happen immediately. The last thing I would say is, as you go about your data mesh or data product journey, Architecture 101 still matters. And you can get yourself in a lot of trouble by architecting things poorly. And Data Placement 101 still matters. You need to think that through. And the way that, that I recommend folks to do is, is this data mesh and building the ecosystem. It literally happens one step at a time, one data product at a time, which really means you gotta, you gotta frame your journey in terms of incrementalism and test and learn philosophy. If you do those things, you know, I think your, your odds of success are, are very, very high.

Yeah, I agree. And yeah, we've seen these waves before and we've seen some work and we've seen some fail. So yeah, one of my key sayings is agile is not ad hoc. We don't just make shit up. So there's a lot of patterns out there, right? There's lean thinking, there's Scrum, right? There's flow, there's product thinking. They're all slightly different. There's DevOps, right? And that's different to all those. So we got to figure out which pattern we want to adopt. And it's a paradigm shift we're trying to go for, right? It's not a technology change. So we need to share patterns. You know, if I think about why is Scrum being so successful? Well, the fact that there's five ceremonies that everybody who does Scrum can list off the top of their head what they are, right? It's a, it's a known pattern that most people who do Scrum adopt those five ceremonies. It's a proven set of patterns and that's one of the reasons it's been so successful. So, you know, whether it's data mesh or data fabric or data lakes or data lake warehouses or data whatever terms we're going to use, it's the patterns under the covers that people want. It's the things that say if you do it this way, with this context, then it has value. Right? So, you know, lots of patterns you picked up, right? The idea of treating a set of things as a warranty against your data product and exposing that to somebody so they know what they can trust and what they can't, that's a really good pattern that we can adopt from. From another domain, you know, another profession. So, yeah, exciting times. Let's hope we don't go down the the big data problem. Everybody thinks it's just a bunch of hadoop buckets that you put crap in and it's going to magically do it for yourself. Let's hope we do the hard work to make this one stick and actually have value.

And.

And I hope we do. So look, thank you for sharing those patterns. It's been really good and we'll catch you all later.

Well, thank you very much, Shane, once again for having me on the podcast and I hope we have a chance to talk again soon.

Data Magicians was another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.