The Right Culture To Create Data Products | Better Teamwork Tips

Yesterday we were talking about, like, what is a data product? And Srujan, since you gave your answer to this, it'd be nice to hear your point of view, John. So, John, do you want to sell, like, what do you think of as being a data product?

Sure. Thanks. A data product is an organizational asset that enables insight, whether it be basic reporting or supporting predictive or prescriptive analytics. It's that capability to take the corporate data and provide it in a reusable and highly usable fashion. When I think about things, when we call it a data product, the products need to exhibit certain characteristics that make them usable and reusable, and they're easily found, they're trustworthy, they're secure. That's what really differentiates a data product from. From something that's merely a query or a table or a CSV.

Okay. I like the idea that it's easily discoverable and it's secure as well. So you've got extra sort of levels of service beyond just, hey, here's some data set. Could be anything. Nice. Okay, so I'd like to know some examples of successful data products. So just make this a bit more concrete. Strujan, do you want to go for this?

Sure. One of the typical data products I've seen most of our clients implement is a customer 360 that drives oftentimes personalization on their web properties, and it also helps them understand their customer journeys better. That's a product that I see pretty much most of the companies implementing. And it often requires data coming from four or five different types of source systems. And all of that gets packaged into the entire knowledge about your customers, which essentially is what the data product is giving you in a nice curated manner.

Okay, I can certainly see how that's useful. So, yeah, very often it's like, oh, you got like some bits of data about your customer in Salesforce, maybe some like, marketing stuff in Marketo or whatever. And then there's like 11 different systems have just little bits of information about what you know about the customer. So bringing that all together has got to be very useful. John, do you have any examples of successful data products?

Yeah, I'll give an example that's very, very close, but in a different domain. The longitudinal health record. So when you take an individual and you look at their experience over time in their episodes of care and how, what treatments, what medications they're on, the reactions to those medications, there's that ability to create like the customer 360, that holistic view. And also highly, highly important that Temporal view of that patient. We've seen our, the healthcare providers and the payer organizations that we've worked with build those types of data products. And as Sreen pointed out too, they of course come from a host of different systems or solutions to provide that holistic or that full picture.

I can certainly see how that's incredibly useful. So whenever I go to the doctor there's always like, oh hey, just fill out this like 7 page form on your medical history. I'm like, I've visited you like many times before, how can you not know this about me? So yeah, having that all like automated and like easily discoverable for the doctor has got to be useful for the doctor but also useful for their patients as well. All right, so I'd like to get into how you actually go about building these data products and let's talk about who's involved in this. So I'd like to know which teams and which roles are typically involved in creating a data product.

Yeah.

John, do you want to go first this time?

Sure. Thank you. So we actually, when we work with our clients on adopting data as a product and data mesh practices, we introduce the concept of a product data product team and that's a cross functional team and it contains capabilities that from, from understanding the business or the client need to the actual engineering. What we look to do is separate the platform, some of the more technical aspects of, you know, how do I provision services when we try or how do I launch metadata repository from those product creation. So we try and enable the teams to have a set of tools, those product teams have a set of tools to use those more technical capabilities and focus on what is the data the client is looking for, how does it need, where is it sourced and how does it need to be transformed and cleansed to be useful in that data product. So those product teams have a data product manager, data product owner love to talk about that at length. It will have some analysts, engineers, it may have a couple developers representing it in different formats and in some cases clients also take them out to the point of presenting the end deliverable, whether it be a report or a model. So they might have a combination of analysts and scientists in the team as well.

Okay, so there's that split then. So the product people, the product managers can decide what's going to go into the data product and then you've got analysts and engineers who are deciding like how that data actually gets into the product depending on whatever the product manager wants. Is that right?

Right. Very passionate about this role. This new Unicorn in our world is the data product manager, because that role is that orchestrator in the team and they're working closely. The expectation is that they work closely with the client and they understand the data and the organizational model sufficiently to say this is how this needs to be represented. Here are the concerns we have in terms of quality and presentation of the data. So they cross business technology and the realm of data modeling.

Okay, I like that idea of an orchestrator. It's like human kubernetes. Cool. So, Srijan, is that your sort of understanding of how the data product teams work or do you have a different opinion on who goes into it?

No. So I totally agree with John. I think that role of a data product manager or a data product owner, I think that is a role that is going to be essential here. And some of the implementations that we have done, that role has become extremely critical. That techno functional role that bridges the IT and business teams and creates that buffer in a way where they understand enough about the source systems and have a very deep understanding of the business and what the business is trying to do and be able to bring that together. So that's, I think, a very crucial role. And everything else John said about you need data engineers, you need data analysts, you need app developers to be able to present the results that come out of the data products in an efficient manner. All of these roles are important. One thing I've noticed in some of our implementations is the primary or the role that's using the data products the most is being the data analyst or the business analyst that is helping out the business teams. So, and that's a focus area for us to make sure that that business analyst is able to get complete understanding and easily able to access and use the data products.

Okay, so I'd like to talk about what skills each of these sort of team members needs. So maybe we'll start with the data analysts. What do data analysts need to know in order to be part of data product creation? All right, go on, Strujan, you go first this time.

From a data product creation perspective, Rishi, it's usually the data product manager or the data product owner working with the data engineering team or the IT teams and bringing them together to define the data product. The building of the data product is often what I've seen done by a data engineering role. The data analyst is a consumer of the data product. Oftentimes they are like instead of going into your dimension, cubes, etc. Cubes are going into data products to do some of that exploration and provide the analysis on top. So that's ok.

So the data round is more about like, well, I need to be able to test this, actually.

I need to be able to use this. Yeah. I need to be able to provide the answers that my business leaders are asking me and be able to do that quickly and efficiently.

Okay, can you talk me through what skills the data engineers need then?

So it's your typical data engineering skill sets to be able to profile and understand the data, be able to pipeline, apply the necessary transforms that are needed to be able to get the source system data mapped into your data product layer. I would say it's your typical data engineering role that has an understanding of pipelines, that has an understanding of how to do efficient transforms, and has enough deep SQL skills to be able to, you know, start mapping your source systems into the data product constructs.

Okay, so really it's a lot of creating data pipelines then. Okay, that's good to know. It's like no sort of extra skills beyond the standard sort of data engineering role.

Yes. And what for some of the way that we do data products on our side, you know, an understanding of YAML, which is, you know, not that complex, you know, for a typical data engineer, but those things to capture the service level objectives, capturing the security and governance requirements into the data products and make sure that those get implemented as well.

All right, nice. Then the other big role involved in this is the data product manager. John, since you seem very excited about this role, do you want to talk me through what skills does a data product manager need?

Yes. So first, first off, to be effective, a data product manager needs to have a good solid understanding of the domain in which they operate. And I highlight this because what I've seen often as a failure pattern are strong technical, strong modeling individuals come in and try and take this role on, but they really don't understand the domain. So they become more order takers to the business than being able to provide an informed opinion on how those products should be shaped so that they become reusable and more usable through the organization. So having that understanding of the building of the business, what are the metrics, what are the entities that exist with the business, how do they interact with one another? I think that's, that's a core starting point for them. But then they also have to have a grasp of data not to the level, as Sujan talked about, of a data engineer who's going to get down into pipelines and, and in depth knowledge of maybe Python as well as SQL. There Have a more generalized understanding of data modeling how objects interact and what are the requirements for putting together a high quality data product. There's that business experience, there's that experience on the data and understanding it, and there is an understanding of the platform and solutions that come together. As a product manager and you're thinking about, well, how am I going to represent this in a catalog? Or how will, how might those data analysts or other consumers use things like Lineage? That product owner has to have some understanding of the various components that are going to come together and enable those products to be effective.

Okay, so it just seems like these are pretty different skill set. So understanding the business side of things to be able to say, okay, this is a sensible feature that needs to, to be there, understanding the data so you can have an intelligent conversation with the data engineer and then sort of those management skills just to be able to make sure that the right features are delivered on time. Yeah, okay, I can see why you said this is a sort of a unicorn position. There's a lot of different things going on there.

It absolutely is. It's one of the biggest challenges we see for our clients is identifying the right individuals with all, all that, that, that right mix of capabilities, the right mix of characteristics to be successful in the role. It is a challenge.

Okay, so I mean, where do these people come from then? Is it like you go from, I'm a data analyst and I've been using these products to now I want to start building them, or do you go from like, I was a data engineer, so I know some of the technical side and then I'm going to get into it, or do you come from the business side? Who are these people?

You know, it's interesting, throughout my career, and I've been doing this for a few decades now, I've seen those individuals come from all of those directions. So I've seen data engineers, technologists who are, who become really passionate about the subject they work in. And so they start to build up that expertise as an individual who's one of the managers at one of our large financial services organizations. And he started out in it, but he, because of his knowledge of the systems, he began to know the business processes better than the other people around him. So there's, you can definitely see that individual move up into that role. Likewise, you can see people who join, you know, might join a finance department or a sales department as an analyst and start to become really proficient both in the, you know, performing the role of building this analytics, but they can also start to Understand the systems, how the data comes together. I think the key ingredient that any of them have is just a passion about this is if you have that passion for data, you can start in any one of those three corners and then come into it. But it's got to have that sort of excitement about the work that you're doing.

Absolutely. So that's good that you can get into it from sort of any direction. Srujan, is this something that you've also seen, like, if you want to be a data product manager, you can sort of go from like there's a lot of different ways to get into the role.

Absolutely. You know that 100% agree with John. You know, one thing that I've seen for, for this role to be really successful is a deep empathy for, you know, the IT team, some of the decisions that they make, how they run their, you know, organizations and be able to work with that empathy towards the technology teams and at the same time have the deep understanding of your domain, of your business in a more holistic manner. Because one thing you don't want to do is be very narrowly focused and not have the larger picture in mind. Because if you want this data product layer to be the foundational layer for your business, you need to have that holistic understanding to be able to design the data products the right way, you know, for all the future use cases. So that empathy towards both the business and the technology teams, I think is, that's crucial for you because you are really going to have to get both sides, you know, working well together to make this role happen.

Okay, so this actually it's a little bit similar to one of the things we talk about a lot in these, in these webinars, is that if you want to be a good data scientist or a good data analyst, you need to have an understanding of the business side of things as well as the tech, absolute things. So it seems like this is a very common like combination of skill sets or is becoming more popular. Anyway. All right, so I'd like to talk about where these teams fit into the larger organization because it sounds like it's kind of a cross functional thing. So, yeah, what sort of department would a data product team be in and who would they report to? Go, John, you look like.

Yeah, this is a great question and we're talking to a lot of our clients about this right now. As advocates of leaders in Data Mesh, we do espouse a decentralized approach and pushing those product teams out closer to the business units, closer to the consumers and the customers that they're working with. That being said, there is also the need for any kind of technology or technology oriented team also live with like kind. So you need to strike that right balance. If you push them all out into the business units and we had the situation with a, a large oil and gas company, they're saying, listen, that business unit is made up of about 17 people in this one country. I put a couple data product folks out there with them. They're going to get no leadership, they're going to get no support, they're just going to be order takers and get hammered. So you need a mix of centralized support from a technology, maybe a chief data officer, centralized data office that supports the platform on which they operate and provides guidance and structure. But there has to be that close alignment to the business. So it's some kind of matrix organization that's highly dependent on the organizational structure you use in your company you have espoused.

Okay, so basically the trade off is like you're either part of the data team, so I guess reporting the chief data officer, or you may be like if you're doing some kind of customer stuff, then you're inside the sales team or the sales organization and then you're going to get better, better on the business side in one case and better on the data side in the other case. And there's a bit of a trade off there, right?

I think that you're, you need to, you need to be aligned to the business. The one thing that we've seen, and again going back into, even into the 90s of seeing us create these large centralized data organizations supporting large central centralized data repositories. And for many organizations that has become a blocker to progress because everything funnels into these organizations. It's that challenge of everybody wants something from data. Of course, nobody wants to invest in it either. And so they become these very, can become very bureaucratic in nature, very much managed by tickets, very much managed by utilization of resources. And what results in is those organizations very slow at output, they're focused on you often financial considerations. And so we do strongly espouse the idea of pushing those product teams out as close to the business as you can, but recognizing if you don't give them some sort of common support from a technology organization, some sort of matrix reporting relationship, then if they go out to the business, the business doesn't know how to manage them. And like we've seen with many customers in the past, you know, as we started scrum and thinking about product owners, the customers can be, just give me more features, just give me more features. Rather than thinking about how do I build a reusable architecture, how to build up these data products that are going to actually make life better in the future and accelerate my abilities going forward. We need that balance. So it's if you did create sort of two points on the line here between sitting in the business and sitting in centralized, it, it's somewhere in the middle. And that really does start to depend on your organization.

Okay, that's really interesting, Srujan. I mean there's a pretty comprehensive answer, but Srujan, is there anything to add to that? Like do you have a. And depending on whether it should be one side or the other, I think.

The ideal way to run this would be have these teams more closer to the business, that decentralized approach that John's talking about because you have that deep understanding of the domains when you are that closely embedded closer to the business side. But I see a lot of organizations, they're going to go through some sort of a transitionary phase, I feel where the initial phases of the data product roll out. You might be doing this from a CDO office as a centralized team, but as you start building out the skills and I think organizations are also learning, just like when you started adopting Agile, take some time for the organizations to understand this new way of working with data. I see it as a step function where it might start off at a centralized level and start getting decentralized as these roles and processes mature over time.

Okay, start in the data world and then you're probably going to get shunted somewhere towards one of the commercial teams. Okay, so I'd like to talk a little bit about getting started and I think one of the themes from both of you has been that data products really need to align with business strategy in order to work. So first of all, how do you begin by getting that alignment? Srujan, do you want to take this?

Yeah. So the first thing I'd say is, you know, the ideal way to do this and then I'll talk about, you know, what's happening and you know, I would say, you know, do a thorough assessment of all of your source systems. Understand where you have systems that generate similar types of data from multiple products that needs to be standardized. Understanding and assessing your source systems to come up with what are the right source aligned data products that are going to create these sources of truth. If I have 10 order management systems, I ideally like to have a source aligned data product that's going to standardize Data coming out of those 10 systems. The second thing is understand your consumption archetypes, see how different businesses are going to use the data, what sort of needs do they have, what sort of interfaces do they need on the data. Once you get that holistic understanding and start mapping some of the business domains into these consumer aligned data products. I feel as you start creating for example a marketing 360 or a sales 360 or a commerce 360 where you're mapping your business domain into some of these data products, that now gives you a great foundation for any new business use case that's coming in. To be able to quickly build on top of these foundational data products, that would be the ideal way. But again, since this is such a new concept, oftentimes what we do is pick a very specific business problem, get that deep understanding of what is the problem that you're trying to solve and create a thin slice end to end. And we are incrementally building that value today. But as organizations start seeing value over the first 1012 data products that you've created, then we are trying to encourage our customers to say don't just constantly go find a business problem and try to build a data product. Take a more holistic approach now that you get a better sense of how to build and run these.

That's interesting. If I've under the suggestion correctly, you start off with what is a really simple business problem that I can solve without having vast amounts of data all over the place. Just do something simple, get that working and then once you've got a dozen of these, you go okay, well I'm starting to see patterns here. And you can build these up into a small number of larger products rather than having lots of tiny individual data products. Is that right?

Yeah. Yes. Because this is a different way. Right. You're thinking more outcome first. You're not thinking where is my data? How do I ETL it into centralized repository and then figure out what to do. This is a new approach. So it's going to take some time for, you know, like a few use cases and a few successes early for the organization to start aligning around this.

Okay, all right, so in terms of like what makes a good first data product? So Srujan, you sort of said it's something simple, something like close to the sort of revenue or commercial side of the business. John, do you have any opinions on this? Like what, what can you give some examples of like what a good first data product might.

Yeah, and those, those factors are important. There is really, when we talk about adopting, moving towards this, this way of operating, it is an organizational shift. And we say, you know, you've got to start out by proving the value to the stakeholders. Somebody's going to have to sign a bigger check in a while. And so to get them to sign that bigger check, you're going to need to demonstrate why adopting this approach is better than anything we've done before. And so picking that use case that is, that is meaningful to the business, to the stakeholders, the people who will be paying and is demonstrating that a capability that they haven't been able to enable before. So with a life sciences company, one of the outputs from, from adopting a product based approach was producing a trial diversity report. All the information about the participants in trials are captured in clinical trial management systems. And large life sciences companies are often the result of many acquisitions. So you've got a whole bunch of different systems out there. And there wasn't the ability to look across the various trials and say how well do the candidates do the participants, I should say in the trial align to the needs of that specific treatment. So something like sickle cell anemia predominantly affects the African American population. Let's make sure our trials reflect that. And so the ability to pull that information from across those trial management systems and present an accurate and up to date report was really critical to that business and it was something they weren't able to do. And so picking that use case, demonstrating that value and demonstrating, hey, using this approach, we can tackle problems that you struggle to tackle before is valuable in opening up the organization, not just pay for the change, but to also drive the change that's required.

Okay, I like that. So it's not just about we can do this thing more efficiently. We can solve a problem, problem that we couldn't solve before. And that seems to be, I'm sure it's quite a powerful motivator there.

Okay, the point I wanted to make here, you know, I would encourage if you are building data products, start trying to measure the roi, start putting some measurement metrics, you know, around your data products. That is going to really help you drive adoption. If you're able to clearly measure what are you spending on creating and managing a data product and capturing the ROI based on the use cases that you're powering. That I see actually is very crucial in being able to do what John was saying and driving that adoption internally.

Okay, in that case, maybe let's skip to what constitutes the happy ending. What is a good return on investment, what sort of metrics might you use? How do you measure this? Since you started on this subject, do you want to continue?

Yeah. So I mean number one, what sort of compute resources are you spending to actually provision and keep a data product up and running is the number one thing that you would like to capture. You need to get as you're building these data products, make sure that your platform has that kind of capability and that metadata from an operational perspective so you can start capturing this at that granular level. How many data products did I build from the marketing domain perspective? How much am I spending and constantly managing these data products? And then what sort of use cases am I driving? If I'm driving personalization on the website, what was the lift that I got in my conversions as an example? Those again are things that you capture in other tools at the edge from your analytics tools. If you're trying to measure the conversion rates, but bringing that sort of information back from the usage side and having enough of the metadata from an operational perspective to understand what are you spending on this, this also becomes important because if you don't do this, you'll end up with a sprawl of data products that you're constantly managing, spending money on and not retiring. And that is something that won't be very efficient, especially from an OPEX perspective.

Okay, so this is definitely something I've experienced like from sort of creating software. It's like building new stuff, really fun. Maintaining stuff is less fun. And so you're probably going to want to retire stuff at some point. Yeah. So, okay, so you need some measure of like how much money is this actually bring you or how is it actually working towards your business goals?

Absolutely, yeah. You know, if you have, you know, any software product and let's say you know, you have all your click analytics from a mobile app or a website that you capture, you do constantly try to optimize, measure and understand how is this being used? Where is this being used? What are my journeys? What are my drop off rates? Think about data products in a similar way. These are real products, so measure them and use the measurement to be able to manage them better.

Okay, John, do you want to expand on this? Have you seen these specific examples of hey, this is a good metric for a particular product?

Yeah, I think the what, what Srujan has talked about, the utilization, the cost per product, the usage of the individual products is really important to track because one of the challenges we face is the what have you done for me lately Challenge. So we talked about, you know, I talked about that use case and say, all right, we invested some money to come up and now we have this trial diversity report, and that's great. Or another client worked with in manufacturing and we reduced the amount of time to produce their, their scrap report, how quickly they could turn around that information. Once you do it, then it's done. And from their mindset, it's like, okay, it's returned its value and nobody remembers that it used to be 600 hours a quarter to put this together. And so you have to constantly reinforce the value because once these systems are in place, people are consuming them. They become utilities, they become perceived as well. This is just cost of doing business. And I've met, you know, I've worked with many clients saying, well, when does this cost in data drop down? When do I see those initial investments drop into just keeping the lights on? And our problem is our challenges. Well, there are always new questions you're asking. So it's not quite just keeping the lights. So very important for you to constantly align the investments, understand the investments and how you're managing them. So to Srujan's point before, how am I retiring obsolescent or redundant data products? How am I keeping my data quality high so it's having a right impact on my users having those metrics in place. When you go in and start having discussions about, well, could we reduce this headcount? Can we, can we pull back? Is really important to constantly reinforce the value that each piece of data is providing, but then tying it back to those use cases. So figuring out a way for you to represent residual value on, you know, if it's a case around the scrap report and what we're providing, or continuing to find new use cases and enable new research or new insights in your business, that's an important part to keep that business up and running.

Okay, so that's a really interesting idea that if you're doing it right, then people will sort of forget about how good this thing is. I suppose. Like when you turn on a tap in your house, it's like you get water, but you don't think about how much productivity you're saving over. Like having to go to the nearest well or something to get a bucket of water. Yeah. So keeping people excited, I'm sure that's going to be a challenge. All right, so I'd like to talk a little bit about the cultural aspects of creating data products. So what aspects of your corporate culture are going to make you successful at creating data products? Srujan, do you want to take this.

Culture Interesting, because I think John would have, he has a much broader experience. But from my point of view, there are organizations that really treat data as an asset and there are organizations that feel I just need data to do a few things within my business. So that mindset shift, when John started this talk and he said data product is an organizational asset, are you treating it like that? Do you see these data itself as that central to your organization? I think it starts from the top. Where data do you see these as very critical organizational assets like your other tier one systems? If you don't start with that mindset, then you won't see the kind of value I think that is, I'd say the most important thing. And that needs to come top down where you do see data as one of your most important assets in your organization. People talk about that a lot that we invest in data, but that intent of, you know, how do I treat data? How am I funding this? Where does it sit? What is the importance of your data team in the decision making process? You know, that is really where it really shows, I feel.

Yeah. So leadership, caring about data seems important. Yeah. John, do you want to expand? And also there was a bit of jargon you mentioned. So Srujan, you were talking about tier one systems. So I know John, Molly talking. Do you want to explain what those are?

Oh yeah. Those critical business systems that run or that operate your business, whether it's your CRM solutions or ERP or MRP for manufacturing that recognizing ahead of sales. A VP of sales recognizes how important that CRM solution is to tracking their funnel and ensuring that they're converting effectively. So recognizing that, but also recognizing the data that comes out of it. I completely agree with what Srujan said. In fact, it's one of the sort of heartbreaking experiences I've had over the past few years is there have been people who've reached out to us when we work with us, say hey, we're very interested in data as a product and data mesh and we want to partner with you. And we go in and we talk to the technology team and they're very excited about this. Okay, what about Business Sponsor? Well, we were thinking that if we build really great products and then go to the business, they'll finally get in interested in it. And you're just like, oh, I'm sorry, I got some bad news for you. You're not going to get the investments you need. You're not going to. If they're not, as Trujan said, if they're not pushing you for data. If they don't see how knowledge and insight can improve their business, they are going to look at this as a commodity and they're not going to make, they're going to look to solve it and it's cheap fashion as possible. And I, I saw this contrast in talking to some, again, some folks in the oil and gas industry and the upstream parts of the business and discovery versus talking to folks in life sciences. I gave a talk on that actually last year in Norway and the challenges in the, in the, in the drug business. It takes a billion dollars in 10 years to take a drug to market and only like 16% of them at most make it through. So there's a very clear opportunity to drive innovation through finding out what drugs are going to fail as quickly as possible, accelerating that discovery process. People see millions and billions of dollars associated with that. Other industries don't have that driver. And so yeah, I need to have reports so the regulators don't put me in handcuffs. But that's as much as I'm going to invest. That's as much as I'm going to do. I'm not going to drive a transformation in the business business. So again, just agreeing very much with Srijan.

Yeah. Just one more point I'll add here, Richie is, you know, when you one of the places where data products are going to help, you know, when organizations, especially leadership looks at data today. Oh, it's like that technical thing, you know, where I need all of this, you know, investments in platforms, data engineers, analysts and I really don't understand, you know, how it looks very complex data products, the whole approach. Why I was so excited about starting this company is you can finally make data look simple, make data look like a product. And I think that also I'm hoping will help make that shift happen from a leadership perspective that this is more tangible, it is a product, it is easy to use, approachable versus looking at it as a very complex technical ecosystem.

Okay. So in general, like it seems culturally, before you get started, you're going to have to get that leadership buy in from the technical side of things. What do you need before you can create data products? Like what state does your data need to be in before you can just hit, hit go. And who wants to answer this?

John, do you want to go first? And I'll add in.

Sure.

So the good news is I would say you can start building your data products from day one. You don't have to wait until you clean up your data to do that. That being said, you've got to be very careful with how you present those products if you haven't cleaned up that data yet. The idea of presenting low quality data to a consumer is an extremely risky and often failure method of failing. Right. Is a failure mode because once the users look at the data and they don't trust it, then you lost them. So it is important that you do instill that quality. But by building the product, you will often understand the challenges that are associated with bad data quality. Most of the time or a significant portion of the time, data quality is about understanding what the data actually means. So somebody might ask a question and a simple example I give is with a manufacturer saying, hey, how much does it cost to build an excavator? Like, well, wait a minute, how much did it cost in 1979 when that was first built? How much it would cost if we tried to build that today? Did you plan for that? Are you buying steel from the spot market? Tell me what you mean. And it's those meanings, that nuance that people don't understand. So when they look at numbers like these numbers are wrong, well, they're wrong with your filter. And so getting everybody a common understanding what the truth is, understanding those data quality problems, building those products and trying to solve those problems accelerates that process, accelerates learning. And so going out and building these data products and figuring out where the problems are and then driving that back into your, into your policies about governance around data quality helps create that virtuous cycle.

Okay, I like that you can get started with day one, but you're probably going to learn a lot and that's going to have that feedback loop and then improve your data quality. Srujan, you want to add something there?

Yeah. So that what John said about the data quality, I think it's absolutely essential. But as you think about organizations, when your operational system systems are generating this raw data, there are going to be quality gaps, there are going to be issues that you have to address. That requires a flexibility in your platform, in the underlying platform that you build. That will allow your engineers to be able to understand the data to the extent that John said, or be able to convert that data, transform it easily into the data products you need, requires the flexibility in your data infrastructure. One of the things that we've been talking a lot about under cdo, written many blogs around this is start looking at data as software. Bring those software engineering best practices into data and how you have these internal developer platforms that make it very easy and very declarative for developers to start working on building value, I think that is where the platforms need to go, is abstract the complexity, make it very declarative, make the developers focus on the value rather than the process, which is where I feel they spend most of the time today. And having that kind of an underlying infrastructure is going to be essential if you want to be able to really roll out data products at scale.

Okay, so I like this idea of like declarative product creation. So the idea is you say, hey, this is the business problem I'm trying to solve. And the sort of, the how you do it is sort of automatically handled. So I guess this is like the promise of like fancier developer tools and AI, generative AI and all this kind of stuff is like you just worry about like, what are you trying to solve, not how you do it. So since data quality has come a few times the last few minutes, can you talk me through what are the most common data quality issues that you're going to deal with when you're creating these data products? John, do you want to go?

Yeah, gosh, I. It is all over the board. It really is dependent on the source systems. As Sir John was saying before, you know, you have, you have your six. Well, first off, you have your six dimensions of data quality. And so you're looking at things like completeness or timeliness or accuracy. And all of those factors come into play. It can be that you're getting data from systems that are, you know, we're looking to do a form of analytics on our sales funnel, but we haven't broken down our processes sufficiently to track, oh, we moved into solution development. At this point, you're not capturing the data. So the complex completeness, accuracy, you know, when you think about things like emails and addresses and all of that and fat fingering content, these are all issues too. The, the hidden one, I think that is interesting is timeliness is, you know, do people know exactly when the data that they're looking at. Right now I'm trying to track a package that's coming from across the country and I know that it's in transit, but the United Postal Service doesn't actually update their systems frequently enough to let me know that. So timeliness is a huge factor. So all of these things come into play we found with an organization, one company we worked with, and pulling together a lot of acquisitions and trying to get down to a consolidated representation of their P and L was such an endeavor because the systems all captured data a little bit differently. And so it's that reconciliation process. So it's all over the board, in my experience.

Yeah, I guess there are a lot of different ways data quality can grow. And that package example is very relatable. Certainly after I've used a package tracker before, I was like, oh, three days ago, your package was in Kansas. I'm like, is that good? I don't know. Is it coming my way? Nice. All right, so I don't know. Have you seen the patterns in this? Are there any particular common data quality issues that you've seen?

I think pretty much what, John, they are really a wide spectrum. The rate at which, if I have streaming endpoints where I'm pulling data in, the rate at which the events come in, sometimes you just see the drop and how fast the events are coming in. That might indicate a problem in your source system. If you have email fields, do you have the completeness? Do you have all the fields represented properly? Sometimes organizations think, I have 100 million customers in my customer database. And when you really profile the data, you see there's only 30% completeness in the email field. And that is really the reach of what you can get, not really the 100 million. These are the things that you often see.

Okay, so forever in the audience. I've got a few more questions, but we've got time for our guests to answer your question. So please do write in the chat any questions you've got for Srujan and John. All right, so next question is, what can go wrong? So are there any common mistakes that organizations make when they start trying to build data products? Go, John, do you want. Do you want to go for this?

Yeah, I think, you know, it's interesting. Before Srujan said, hey, in a perfect world, we would have the opportunity to build out the data and present it and then, you know, and work on it from a. From an approach of laying a foundation and building up. What we've seen is that. And the reason he said that doesn't work is the reason we've seen is the. The old adage, build it and they will come is not accurate. And so one of the biggest mistakes that I've seen consistently over almost 30 years in this is organizations going out and trying to build out a solution and then, you know, some sort of big tada moment to the business. And the problems are that the delay, you know, everything gets bigger and messier and uglier than you expected, and the projects get delayed because of the delay. That question that a few people asked two years ago and you weren't able to answer, and you've gone off in the corner and you built this data warehouse or data lake or data products when you come back. Well, they figured out how to answer that question. They figured out how to live without that answer if they can't get it. So they're asking different questions now. And so what you've built may not be relevant. And it was interesting. In the Data Mesh radio podcast, a person from Abbott was talking about their experience of trying to anticipate their consumers and try and build ahead of the use cases. They found it just, they missed the mark so many times. So I think the biggest failure mode that we've seen in the industry is trying to jump ahead too far ahead of your consumers and trying to anticipate their needs rather than building on specifically what they're asking for. I think Srujan brought up a great point before too is as a great data product owner, you gotta have a little bit of foresight about, okay, I know what you're asking for now, what might you be asking for next? Or what are the next set of use cases we're going to tackle and think about your product strategy. But you shouldn't get too far ahead.

Okay. So yeah, scope creep is a definite problem in a lot of like software drums as well. So I can certainly say it would be a problem with data products. Srujan, have you seen examples of mistakes that organizations make?

Yeah, so I mean I think you need the mindset shift, right. When you think about data products, this is not your left to right traditional way of thinking about data. So you know, don't, you know, don't like don't take the same old way of trying to do things and just map it. It's not like I'm just converting every view into a data product, but I'm really thinking differently about understanding the business need and creating the right data product that might serve multiple needs, you know, versus just, you know, I have all these views and golden data sets that are powering all my use cases. Just let me just go replicate. You know, that's like buying the most modern lake house and dumping your third normal form tables into that and saying, I just have a modern system. I think that mindset shift needs to happen. I think people need to really embrace organizations that this is a much better way to deal with data. But it requires a mindset shift in the organization.

Okay. Yeah. So you can't just buy tools and be done. You've got to think about all your processes and the people who are involved and getting that sort of buy in and you know, how the teams communicate with each other. All right, we've got a few moments left. So tell me, like, what are you most excited about in the world of data products? Srujan, do you want to go?

I feel like, you know, data, the. What I am excited, and I kind of mentioned that a little earlier, is, you know, I think data is finally going to be a lot more approachable. It doesn't look like it's such a technical thing. I need all of this deep technical skill skills just to use data to drive my business decisions. And that's always been my passion, is how do I think about abstracting out all of the complexity and how do I make something like data be more approachable and more simple? That's what I'm excited about. You see this as any other product that you're using in your house, and I think that is when you really start seeing the democratization that we talk about. Embedding data in all of your business decisions and making it a critical part of your business.

Okay. Yeah. Bringing data to the masses, that's brilliant. Very much aligns with what we believe in here at Data Gap. And, John, what are you most excited about.

On that and building on that? Bring it to the masses. I think one of the things that can be very gratifying if you've been an analyst or data engineer is actually delivering something that delights your clients, that delights the consumers. And so the ability to enable those product teams to answer questions faster, to turn it into a conversation, rather than your business asks you for a question and you come back three to six months later. You know, that sort of pace, it's. No, we're turning things around quickly. We're becoming a vital solution, a vital asset to the organization and the company. You're. The enterprise you're in is feeling that way. That, to me, is the most exciting thing about the product approach is enabling that ability to delight your consumers and respond rapidly to their requests.

Yeah, delighting consumers. That's a very cool thing. I'm sure everyone wants that. All right, do you have any final advice for any organizations wanting to get started with data products? John, do you want to go first this time?

Sure. Thank you. Yeah. I do think the concepts that we talked about before, starting small, identifying opportunities to deliver value and building up the right, the gravity, the support to do more so that incremental, hey, we're going to solve a problem, and then we're going to expand. And so constantly thinking about how you're going to, in each step, build forward and Move faster that, that over. Well, we're going to go off in the corner and build a whole lot of something is always going to be the best way for you to be successful. Okay.

I mean, yeah, I can say it could sometimes be a pair and say I'm just going to go and build stuff, not talk to anyone. But yeah, I suspect that's not going to be very successful. Srujan, do you have any final advice?

Yeah, I'd say, you know.

I think.

This is going to be the future. You know, if you see any other field, the moment you start simplifying the complexity, abstracting, abstracting it out is where the adoption kind of, you know, skyrockets.

Right.

I feel we are at getting to that place where data truly becomes embedded into the business decisioning process. So don't be afraid of this kind of a change. It's not, you know, like we've all embraced agile and see what that has done for our productivity. I think this is going to do the same thing. So be very iterative, quick. Don't try to, you know, solve the world problems in one go. Start small, keep iterating on it and I think you'll get there. I think this is the right, right approach. I feel for the future of data.

Wonderful. And I like the idea. It's like no fear because that's often the thing, new things, very terrifying. But yeah, I'm sure this is going to help. All right, we got one question from the audience. So this comes from. Excuse me, this comes from Ravinder. So we've gone through some examples of a successful data products but I don't know whether you want to come with some new examples. So yeah, and just asking for more examples of what are some cool data products that work.

Like. I started off my chat with Richie, a customer360 as one of the foundational data products. There are so many different use cases that it delivers for you from a marketing perspective, from understanding the customers to be able to better target them. From the customer care perspective, being able to better serve your customers. So this a lot of use cases and what we have seen where we have driven a lot of value initially with our customer engagements is on the sales and marketing side where we are able to really, really transform the sales and marketing teams to be very data driven in their approach. In really understanding and the fact that these data products give you real time analysis of what has happened and how your business metrics are changing and being able to react to it, especially in a marketing context, becomes very, very useful. And that's something I've seen our customers leverage quite a bit.

Okay, so we've talked about sales and marketing quite a lot. How about some other teams in the business? Like what might use cases be for, like a finance team or if you're involved in supply chain or logistics.

So. Oh, I'm sorry, go ahead. I talking about, you know, move over, say, manufacturing. So one manufacturer that we were working, we've been working with one of the products that they have is called shop visit hours. So what they're looking to do is understand the, the, the item that they're, they're, they're build. How long does it sit in the shop? Why is it sitting in the shop? What are the things that are causing it to sit longer in the shop? And so the analysis of the, you know, this, this device came in, it sat in the facility for seven days. And the reasons, primary reasons, and the breakdown of that were, you know, shipping of the part to the right location, finding out there was the right location in the wrong part, shipping the part again, that sort of thing. So, so giving people an understanding of, you know, when you look at something and just say, hey, listen, you know what, these devices sitting idle 7% of the time is costing our company $10 million a month or something and saying, well, okay, let's take a step back and say, why? And what are the real factors? So being able to pull that apart, it was more about process and pulling different systems together than it was about any really technically difficult process.

Okay, that seems very cool. Oh yeah, the part stuck in Kansas somewhere maybe. But yeah, I like the idea that that's a case where it's going to just make things run a little bit more efficiently. All right, so with that, we are coming up to time. So for all of you who are interested, we have some more webinars coming up. So let me see, the next one is on the first of May. So that's a session on Ultrix Cloud. We've got a session on there 2nd of May on how to do data storytelling. Better. Lots more webinars coming up. Hopefully have a QR code on the. Yeah, QR code on the top left. Click on that. You'll be able to send for future events. So with that, thank you, Sirjan. Thank you, John. Like, really informative stuff. I, yeah, I very much enjoyed this discussion. I felt like I learned a lot. Thank you. Yeah, so thank you both. Thank you to Reese for moderating. Thank you to everyone. Ask question. Thank you to everyone who showed up today. I hope to see you all again in future.