Patterns to define the ROI of a data product with Nick Zervoudis - Episode #74

Foreign.

Welcome to the Agile Data Podcast, where we talk about the merging of agile Twitter products and data patterns in a simply magical way. My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, hey, that's not what I wanted, and then you hear your team reply with but that's what you ask for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now. Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Nick Servotis.

Hey Nick, thanks for coming on the show. Today we're going to talk about the thing that I never actually see, which is quantifying return on investments from data teams. Before we rip into that though, why don't you give the audience a bit of background about yourself?

Yeah, it's great to be here. So I'm Nick. I've worked my whole career in data, but always in that kind of squishy role that somewhere between the purely technical people and the people that haven't looked at an equation since about 25 years ago. @ first that was in consulting, first for a boutique consultancy specialized in data science, then for Capgemini and Vent. Then I made the leap to product management. I worked for PepsiCo and then for CK Delta, that's a subsidiary of a large conglomerate owning a lot of infrastructure businesses around the world. And now I'm on my own. I made the leap to be an independent consultant and trainer specialized around data product management because I see so many organizations still make the same really basic mistakes around how should we get value out of our data investments? Which is also why I've named my company Value from Data and AI. That's the short story about me. I'm someone that studied politics, but somehow has been in data his whole career. And even though I never considered myself technical, I've now gotten to the point where I'm often introduced, at least by the business stakeholders, as oh, Nick's the technical guy. He understands the data side. Except when I then talk to the data scientists, then I don't think they think I'm a clueless person that doesn't understand anything they do. But that's how it feels compared to them. So here we are.

Yeah, I'm a big fan of Hitchhiker's Guide to the Galaxy, so I talk about people like that as Babelfish, depending On which part of the organization you're talking to, you're interpreting for the other part that's not in the room. Normally when you're in product management for those companies, was it physical products or digital products? What part of the product management were you in?

Always data products. And here when I say data product, I'm mishmashing a few different types. Like you've got data platforms that are enabling other use cases. They're not necessarily delivering value end to end themselves. Data and analytics products, things like a dashboard or shipping out CSVs to end customers all the way to using machine learning and LLMs. And that's been the types of products I've managed my whole career, including when I was in consulting and was an undercover data pm. And we didn't call it data PM because that title didn't exist and we didn't even know that product management was a thing. And then the other thing that's worth calling out is that I've also dabbled in both internal facing products where your customers are colleagues of yours in different departments, and external data products where we're literally selling data sets or selling dashboard access to customers that are interested in buying that data.

I think it's interesting that there's this move in the last couple of years to bring product management thinking or data as a product, that way of working from the product domain into the data domain. And I think it's been great. I think we've seen some real changes to the way we work with data and the processes we use and everything we do still quite lacking. There's still a lot to learn and a lot to bring to the table. And one of the things that I wanted to talk about is this idea of return on investment because you know, when I'm working with organizations, it's hard enough to get them to get to the stage where they'll determine the potential action that will be taken from the data or information that's going to be delivered and what the outcome to the organization might be from that action. But it's very rare if ever that I can see them make the next jump to actually quantify the return on investment for that outcome. They may say, I need some data. I want to understand all the customers that haven't bought something in the last six months. And you say great, if I answer that question with data and information, what are you going to do? We know that they're likely to leave, so we're going to go and do a churn offer, send out some emails, give them a discount okay, if that Churn campaign is successful, what's the outcome? We'll retain some customers, we'll increase our margin, get some more sales, and you go, great, if you had to quantify that as a number, what would it be? And then you get crickets. And maybe it's because they don't actually have the data to input to help them make that decision and understand what that would be, but there just seems to stop. So how have you dealt with that, this idea of return on investment? How do you actually approach it?

Yeah, it's a great question and I want to answer it by going on a bit of a tangent first because I want to comment on even what you said, that, oh, in the last couple of years, companies are waking up to the need for product thinking. And it's true, like, a lot more companies are doing it and then there's other companies or other data teams that have been doing it for the last 25 years. So I feel like there's a very skewed distribution in terms of which companies are doing things in that value centric and customer centric way versus all the laggards. In the same way that if you look at how many people are using maybe LLMs in production today, it's a very small number of companies around the world. If you looked at it two years ago, it was an even smaller number. And we're in that stage where the laggards and even the people in the middle of the adoption curve are slowly getting into it. And I think that's relevant because for me, a lot of these, let's say, best practices now, how do we figure out the roi? It's not something that's super complicated or cutting edge or, oh, you need to have all your ducks in a row, all your data in a beautiful metrics tree before you can start asking those questions. Because let's jump into your example. Right. I thought you'd give me a harder case to start with. Right. From what you've said, it sounds pretty straightforward.

Let's start with the simple ones. If we see how you can apply the patterns to a simple scenario, then it gives you the sense of the pattern and then from there we can go into the horrible edge cases where we know it's a little bit harder. But again, I still find people struggle with the simple cases.

So here's why, and I'm not just saying it to be smug or anything, but why I think the example you gave me is straightforward because the link between the data request, let's say, and the outcome the business outcome that the customer is looking to achieve is pretty straightforward. We want to reduce our customer churn. That means that even roughly, I'm sure, even if their BI is not very good and there's some data quality issues, they'll have some idea of, okay, how many customers are churning today? What is our average order value? What is our transaction frequency? And therefore, based off those numbers, I know my customer lifetime value roughly. I know my number of churning customers roughly. And I can go, okay, what if I can reduce that by 10%? What is that worth? Is that worth $500? Is it $5,000? Is it $500,000? For me, that's the starting point. And I think the mistake we make in terms of these kind of ROI and value estimations as data professionals is we think it has to be something that we're going to calculate with our Python notebook, that we need the exact numbers, that it needs to be precise and it needs to have the same rigor as a lot of the other data work that we do, when actually a lot of the time all you need is a back of the envelope calculation. I like using the term Fermi estimation, named after Enrico Fermi, who allegedly, just before the Trinity nuclear test, whipped out on a piece of paper his estimation of what's the TNT equivalent of the blast that they were about to witness. And he got it. I think it was, he estimated 10,000, it was 24,000. I've probably missed the exact numbers. The point is he got the order of magnitude right. He was just off by a factor of two. And for these ROI estimations, it's the same thing. It's, I'm not looking to understand, is this going to make us 600k or 650k or 700k? It's more, okay, this opportunity, roughly speaking, is in the 500k to 1 million and this other one is in the 10k to 20k. And this optimization one of my engineers wants to do to bring down costs is going to save us $500 a year. And so if I look at those three examples, it becomes very easy for me to go, okay, roughly speaking, which of these is more valuable or maybe more valuable per unit effort we're going to spend? Because maybe the optimization will take one day and, and the churn modeling thing will take, I don't know, one year. And so that is a very good starting point. And then the other thing I wanted to comment because you say, oh, I ask my stakeholders, how do I go about doing this? Or give us a number and they're like, oh, I don't know. Do you know what makes it so much easier for them to give you a number? If you make that back of the envelope calculation super basic, you sketch it out on a slide on an Excel sheet, whatever, and then you show it to them, you're sharing your screen or you show the piece of paper and then they go oh no, that's not right because they this is not our lifetime value. Or oh no, that's not right because whatever other assumption you've made that's wrong. I found that it's so much easier for both technical and non technical stakeholders to basically critique something or provide me with an input I'm looking for. If I give them the scaffolding, I'm like, here's the shitty first draft, now you tell me what's wrong instead of hey, here's a blank sheet of paper, please fill it in. Even if the blank sheet of paper is like a template for them to fill in, it's still harder to get an answer from them compared to here's my wrong assumptions, now correct them. So much easier if I go, hey, here's the rough Fermi estimation I've done where I've assumed your lifetime value is this order times this order frequency and then that your churn rate is this much and that your growth is going to be like that. And then it becomes really easy for them to start giving me an answer because you said oh, I asked them and how do we quantify that? And they don't know. And a lot of the time it's that maybe when we ask, especially a non data savvy stakeholder that kind of question, they might think that we're asking for something much bigger. Oh, these data guys, they're here to do smart analytics and statistics stuff with their fancy PhDs and Computer Science degrees and whatnot. It's no, actually it's some of these models are super simple and for me what I love about this approach is that it also means that it's so easy to do really quickly for a large number of opportunities. Right. I consider it, let's say we've got 15 ideas or 15 requests or even 150 and if you can spend just a few seconds for each one to figure out, okay, roughly speaking, this is going to save three hours a week from this employee who on average gets paid $50 an hour. So the value of this automation is this much versus this is a cost saving that'll save this much. And sometimes you might be missing one of the Variables, right, Like in the churn model, it's okay, will this reduce churn rate by 5%, 10%, 1%? And you can just put a number that feels reasonable. And what I found is usually that rough 30 second estimation is almost exactly the same as the two week version of that estimation where we build the prototype churn model and we estimate what it's going to be when it's in production and we test a bunch of different data sets. And then all those times this happened in that order, I felt a bit silly when my manager may have been, look, just plug in 10% uplift. And I was like, no, but where did you get that number from? What if it's not 10%? And then I come back two weeks later because I was insistent that this is not a case that we can just do a Fermi estimation and come back and actually it was pretty much the same number. I was like, damn, okay, not necessarily wasted two weeks, but wasted two weeks.

I agree with that. I mean, one of the things I say is when we work on the canvases, we always get asked for how long is it going to take to deliver that product? And yeah, typically we're doing this discovery really early. So any detailed estimation is waste because let's face it, humans are really bad at estimating anyway. But even if we weren't, we're at the discovery stage, we're still ideating. Like you said, there could be a hundred ideas going and spending all that time estimating how long each one of those is going to take is waste at this level. We haven't prioritized it, we haven't said this is the top five. So let's not worry about it, right? Let's just do a quick T shirt sizing, pull a number out your bum, the number actually be quite good because we're good at guessing and then move on. And we can do more detailed estimates if we have to at a later stage. So I'm with you on that. Do it light, do it quick and use it where the value is. But to do that is quite a skill. So if I think about it, you have to understand how data works, how effectively metrics works and potentially metrics trees like you mentioned. And we have to understand the organizational processes and we have to be able to combine both of those to be able to quickly articulate how life lifetime value works or how a churn number will work and what the impact of that is. And that is a skill and it's not a skill that I see a lot. What's your thoughts? Do you see it a lot?

I think for sure, being comfortable doing it, it takes some practice like anything else, and it's going to feel harder, even at least just mentally. But for me, it comes naturally because I spend most of my career either in consulting or in data teams where we acted like internal consultants for our different business units. So I'm very used to not really knowing very much about a domain and asking a lot of dumb questions to my stakeholders about, hey, how does this operation actually work? I draw the flowchart as we go. Maybe I'm even sharing my screen as I'm drawing out a process that a stakeholder is describing. And they go, oh, no, you forgot about this, or no, I forgot to mention about that step. Or actually, this part is more complicated because when A happens, we do B, and when C happens, we do D, whatever else. So the point I'm trying to make here is this is not an exercise that a data professional needs to do on their own. It is a collaborative exercise we need to do with our stakeholders for a couple of reasons. Number one, because like I alluded to, we don't have the full picture, and even worse, we might think we have it based on the explanation we were given. And actually, it turns out our stakeholders didn't mention a whole bunch of other details that were really important. But also secondly, for me, it's also about building better relationships and trust with our stakeholders. I've seen this happen so many times when, and it's in data or tech more generally, where a kind of tech team comes, they have some kind of discovery workshop, and then they disappear for six months and then they show up with the thing they've built and they go, here you go, please test it for us. And one, what happens then is you often end up with having built the wrong thing because again, the picture you got during that initial requirements gathering exercise was incomplete. But also too, let's say you actually nailed it, right? You built exactly what was needed, you estimated the value potential perfectly. But then that stakeholder turns around and goes, who are you? Who are you to tell me that I should be using this dashboard? Now, I've been doing this job the way I've been doing it with Excel for the last 20 years. So it's also about bringing our stakeholders with us on the journey. And that's just as true about the financial estimation part. That's maybe we need to bring our finance colleagues along the journey or the client's finance colleagues, so that when the business case shows up on their desk. They don't go, who is this? What is this thing that these consultants or that the data team wants to do again? They go, oh yeah, this the thing that Shane and I worked on together when I gave them the numbers from the budget to plug into the business case. And then I learned what a, I don't know, random forest algorithm is because I was curious and it becomes so much easier to work together. And the same is true on the user facing side, which I know is a slightly different topic, but for me it's actually conceptually the same thing. It's so much better to build together with our customers, be they external or internal, than to do something on our own and then show up and say, here's my homework. And then you find out that the homework is wrong or that they just don't like the font you've used because that's not the font they're used to.

I agree. I think that constant feedback helps us iterate and figure out where we've heard wrong or where they forgot to tell us something. Or as you said, if you wait six months, something's going to change. Anyway, that may be the most burning problem six months ago, but there's a real big chance that when we go back with the answer six months later, they've moved on, they've fixed it with Excel and there's something else far more important so it's no longer top of mind. So that idea of constant engagement and collaboration is so much value. And one of the things I think you were just talking about is this back to old school almost business process mapping, this idea of nodes and links and saying to the organization, how does the process work? Tell me something happens and then what's next? And let me draw a circle and say, this is the thing happening and here's a line to the next thing happening. And from there we can start identifying those measures that can form that return on investment. And one of the things I think you talked about was looking at it from cost saving, revenue improvement, and there's always a risk one as well. That's the trifecta that you can use is are we going to save money, are we going to make money or are we going to reduce risk? That's the three that they always tend to come back to in my view. So I think, yeah, combining all those patterns together is really valuable, 100%.

And for me, exactly that trifecta is the starting point of what is the benefit we're trying to influence at the end of the day. Because when someone says Hey, I need a data set showing weekly sales. I'm like, no, that that's not what we're here, that that's just a solution you have in mind. You're asking the doctor to prescribe you specific medication and then just like the doctor goes okay, I understand you went on WebMD and you've self diagnosed that you have this, but let's just check your symptoms to be safe. For me it's the same thing. We go, okay, let's understand the business problem and this is for me super simple thing, but that so many data professionals get wrong about it. They see any request that comes from stakeholders as an order, a command and a lot of the time one, it absolutely is not like that. And the person making the request is clueless about what they need and they're coming to you for help. But maybe they've also not learned the right way to do that to say, hey, our sales are down, I need to figure out why. I have a hypothesis that maybe it's because one of the regions is underperforming. That's why my request on the JIRA ticket said give me a sales breakdown by region. But then when they ask that you can start asking more probing questions to figure out, okay, what is this hypothesis exactly? How can we test it? Maybe actually there's an element of statistical know how that this test is going to require that a dashboard with a line chart is not going to solve for the stakeholder. Maybe we can test alternative hypotheses in parallel. That if you know how to write code and to use different models is maybe trivial versus you make the dashboard, it spits out the line chart the stakeholder looks at it doesn't see an obvious pattern or thinks this is going to be too hard to make sense of and they give up on it and you've ended up with dashboard number 952that no one in the business is ever using.

There's a couple of things embedded in there. So one is if we look at product teams in the software industry or software domain, often they talk about feature factories where somebody comes and demands a feature, doesn't tell you what it's going.

To be used for.

Our equivalent is a data request. Here's a data request, give me the data, don't ask me what I want it for. I still blame JIRA for both because basically both of those problems are managed in stupid JIRA ticketing systems. And I think the other thing you mentioned is a doctor saying you've gone and googled WebMD or whatever I think we're hitting that new world right where actually our stakeholders are going to LLM the answer and come to us with a data request that's based on an AI bot telling them what the answer is. So it's going to get worse before it gets better. But one thing you mentioned was this idea of metric trees and I'm old people mention now that I bring up ghosts of data past. And I remember a while ago we were doing balance scorecards, cause and effects. This idea of saying if we have a bunch of things we measure and we understand the business processes, then effectively we can see some causation or relationships between those metrics and where we see those causation or those relationships has some value. We can infer some things and help us make better decisions. What's your view on that? Is metric trees another good thing but just a reincarnation of things we've done before or is it something different?

I guess because one I've not been around this game for as long and two, I wouldn't call myself someone that's gone super deep into metrics trees and understands them deeply. So I'm probably going to give you a very layperson's answer. But it's that yeah, a lot of the core concepts are not new because for me fundamentally it's just about saying I need to understand the relationship between the different inputs in my business and how these translate into outputs and to basically figure out what is my business's sensitivity for those different things. So for example, let's say I've got three potential initiatives I'm considering. One is I want to grow my mailing list. The other is I want to improve my checkout rate. And the other is I want to optimize my pricing to make more profit per sale. If you just tell me those three things, unless if I have a deeply intuitive metrics tree in my head, I'm not as the business owner going to automatically know which one of these is going to make the biggest difference for my business. Whereas if I have a forget about the fancy terms, if I just know that flow of okay, how many people do we have in our mailing list? What is our click through rate every time we send an email? And then of that how many people that land on our website actually go and convert and then what is the average profitability of our products? If I have this information somewhere and if I know those relationships, then I can plug in different assumptions and go okay, what if I were to increase my mailing list by 10%? How would that cause a cascade in all the other numbers? Okay, if you get 10% more people in the mailing list because you actually have a super low click through rate and most of your traffic comes from other sources, your revenue would only go up by 0.1%. Whereas if you were to improve checkout rate because you're getting traffic for all these other sources, but your basket abandonment rate is quite high actually that would translate into quite a bit more revenue. And then lastly, if you know the average margin you're making per product, if you were able to increase prices for certain strategic items, actually that on its own would lead to more profit than the other two things combined times 10 made up example. But for me it's more about the high school economics. Not even high school, more like middle school mathematics that you might have to do. When we did the few exercises around Fermi estimation with my students as they were doing it, I was like, oh my God, this feels like I'm an elementary school teacher. Because I'm just asking these guys to literally they're doing multiplication. There's nothing more to it. I've given them all the assumptions and all they have to do is pick which numbers to multiply where. And that's how I want the data team to show it to their business stakeholders. Even if there's a lot of complexity behind how we calculate basket abandonment rate or margin or whatever else, at the end of the day, the output, especially the output that a business stakeholder sees, has to be super simple so that they can also understand this whole concept of making data driven decisions. That's ultimately what we're trying to do. And I think metrics trees are great because even just visually they help us understand that relationship between different metrics. Instead of here's our metrics report, it has multiple tabs and lots of charts. And you basically can't really figure out how to mentally connect the dots between all these things unless if you have that metric tree in your head.

But I can see as data people, we love the detail. So as soon as you say card abandonment rate, I can imagine a data person going, we can't just use an estimate for that because we know the data's there. So we'll just go and spend three months grabbing it, modeling it, getting the actual abandonment rate, and then that'll make our Fermi model so much more accurate when we try and determine the roi. And I can see that logic. But then it's about time to market. It's about again, we don't know that is the most important information or data product to build next, we're trying to use this technique in the discovery ideation phase, not further down. And therefore light touches have value. But do you find that, do you find that data people naturally want to go and grab the data, do a whole lot of work and make it as accurate as possible?

For sure. And I fall into that trap too sometimes because I've been among data people for long enough that now I do the same. But look, at the end of the day, there's going to be some things where actually knowing the real value is quite important. Either because your estimate might be super off or because actually if we're doing these small optimization things where we're just trying to increase one thing by 1% because that 1% is still worth a big number, actually being off by a couple percentage points could make the difference between something being a super profitable investment and something setting money on fire. But for me, the key thing here about both product thinking and data and building data products instead of data projects and also metrics, trees, which in a way it's just a type of and collection of data products, is that we don't go, oh, there's a new project we're doing now, the team needs to go and figure out the basket abandonment rate and figure out all these other metrics because we're trying to do this one off kind of project. And even if it's not a one off project, but it's a specific use case, we shouldn't have to build all these data models. And then you have 10,000 versions of what lifetime value looks like in your business, you go, we will invest a lot of our time as a data team. Exactly. Because it is complicated and a few of these things will take many weeks to do into building out these key metrics. And so we have our collection of metrics that form into a tree and basically you then go, anytime I want to test a new hypothesis, I want to explore something I can rely on these core standard reusable data assets instead of going, everything is a new project and everything is a new DBT model and everything is a new tableau dashboard and whatever else. For me, that's where the power of it lies. It's a slightly separate topic to the ROI estimation part. I still think a lot of the time you're better off starting with your Fermi estimates. And then of course, if you have the real data to plug into one of those assumptions, go, great. But if you don't wait until you do, unless if it's a really high stakes investment. If you're going to be committing to a seven figure investment in your business, then yeah, maybe back of the envelope isn't good enough to get fully started. But if, as you said earlier, if we're just trying to figure out what are the top four opportunities right now out of the 15 we have, and then once we've zeroed in on the top four based on our rough and ready Fermi estimates, then we can do more homework, then we can send the data team to go and do some more calculations if needed.

I can actually imagine that as you do the Fermi calculations, you're going to find that there are certain inputs that you use on a regular basis, especially if you're working in a specific domain. So let's say the domain boundaries are all based on organizational hierarchy or maybe business process. So we have a sales organization that deals with the sales side of things and they're the highest value part of the organization that we've been asked to help right now because we're bound within that domain. I can imagine that the inputs we use for the estimate that ROI calculation are sometimes going to be reusable. We're going to go, oh, maybe it's lifetime value, maybe it's the funnel and how many people we're converting from suspect to prospect. There's going to be this thing where we're constantly using that as part of our calculation. So we can say because we're using it so often, it seems to have value. Therefore doing a little bit more work on what the actual number looks like is going to be really useful for us going forward. So again, we can use it as a way of figuring out what's the next most valuable thing to build effectively for sure.

And I want to make two comments about this. The first is it's not like a lot of these metrics we are calculating specifically to work out roi. These are also going to be the metrics that are part of the data product or related to adjacent data products. If, for example, we don't know our cost to fulfillment, but we're doing a use case that relates to reducing our cost of fulfillment, getting to the precise number of what is our fulfillment cost is also part of the business case. So we start with our rough estimate because we have an average provided by finance. And then when we want to break it down by product as part of the optimization project, that means that next time around we actually have a data product that is our cost of fulfillment per sku. As an example.

The other thing I can imagine People wanting to do then is go into the detail to say if this is the return on investment that we've estimated for that information product, actually we should measure it at the end to see whether we actually did deliver it. And that makes sense. But then we come to a whole lot of complexity because there's a whole lot of other factors that will influence whether we are getting a reduction in churn or whether we are reducing the cost to serve or the cost to produce. And it's really hard to isolate those other factors to say this one thing we did. Now, my view is I don't care if we do some effort and it looks like it's moving the lever, as long as we keep doing effort and the lever keeps moving, we are always getting to a better place. But how do you find it? Do you find that some organizations or some teams want to then prove the ROI six months down the track?

For sure. And I agree that sometimes the rigor needs to be more than a simple pre post analysis, especially when you've got dozens of confounding factors or if you're at a place where actually you're carrying out many experiments in parallel. For me, that's one of those things. Let's cross that bridge when we get to it. We don't need to assume that by default we need to carry out randomized control trials and super rigorous A B tests in order to figure out what's moving the needle. But then the other thing I want to call out is that everything you've just mentioned is something that typically does not get talked about at the start of one of these initiatives. And so the data team, because they've received a request in jira, they go and fulfill it. There's probably a million back and forths because the request initially that was made, actually that's not exactly what the stakeholder wanted. But you built it. Now that they've seen it, they realize they wanted something different. Anyway, eventually you get to your done state and then you go, oh man, it would be great to know the ROI of this thing we've built for the stakeholder. And then you realize that it's unclear what the definition of success is. You've not done that exercise to go, okay, what is the exact decision that's going to be made by which person and how will we know that they're making this decision because of this data. For me, it's a little bit like how I'd have classmates in high school who would write their essay first and then go look for citations to Prove it. And you know what, in high school, that work, but a little bit later down the line it can't because you need to build up the essay, so to speak, off the back of citations, unless if you're just making things up. And similarly, I find it's so much easier to actually figure out the ROI of something if we've done this exercise that goes what is the business outcome we're going to be influencing here? Because when you ask that question, you very often realize that, okay, the thing we thought we needed to build actually is a little bit different. Or how do we make sure that we embed this decision support information into the decision making process of a stakeholder? How do we then measure or have some even approximate way of knowing when someone took a decision using that data that we gave them as opposed to they just went with their gut, same as they have been, but they also opened the dashboard just to see what's there.

I was going to say our best measure at the moment is that somebody actually used it, that the dashboard's not sitting there being unused for six months. The best we typically have is it got used lots. Not that it was used for anything, but it was opened and run.

Oh yeah, and for me, even though I think it's useful, especially it's useful to know that something has not been opened. Actually the usefulness of that telemetry drops off very rapidly after that because you very often have situations where someone is opening the dashboard but then you don't know exactly what they're doing with it, if it's even useful for them and especially if there's some kind of top down pressure of guys, we spent all this money on this new set of mi. You all need to be accessing it or it'll affect your performance score. Well then easily someone can just open the thing, not even have their eyeballs at it and then close it a little bit later. Or they open it and maybe they actually viewing time has increased because your dashboard is more complicated and the user's not able to actually get things out of it. And for me the reason I think it's a mistake is because in most businesses the number of users you have is just not that big. Which means you don't need to rely on quantitative cold hearted information. When you can just ask people, you can have one to one catch ups, you can interview people, you can even have a survey form where you ask a few open ended questions like hey, what do you like about it? What do you not like about it? Do you have any Ideas for how we can improve, improve it and use that qualitative data because it's so much richer than okay, average viewing time has gone up by three seconds. What does that mean? How does it relate to the business outcome? But also really importantly, because I feel like whenever people are telling me they're struggling to connect the work they're doing to business outcomes and to roi, it tends to be bi. Because the way a lot of BI is built and structured in most organizations is in a way that I just think is wrong. Because it's not actually helping enable better decisions. It's just, hey, let's build a dashboard about this. It's not okay. As part of my workflow as an operations manager, we're going to change it. So actually now every Monday morning I open this dashboard which has these specific KPIs and I make 1, 2, 3 actions off the back of it. Every week that we've built instrumentation to measure how those actions play out and so we can have an idea of how those actions improve in quality over time. Most of the time you don't have that kind of mapping. You don't have any measurement of either the action itself or the thing the action is looking to influence. I'm taking an action to improve our copy so that next week's newsletter gets a higher click through rate. Which means then, yeah, it's literally impossible to measure retrospectively the impact of that dashboard that you built that someone maybe looks at. But it's very unclear what the benefit is. It's either unclear because you as the data professional just don't know exactly what they do with it and it's a black box, or it's unclear because actually there is no real benefit, or at least it's super, super fluffy.

What would you recommend to an organization? Let's say the organization's got to the stage where the data team are engaging with stakeholders. They're thinking they're terms of products, they're asking those questions of with this data, what action are you going to take? And if that action is successful, what outcome do you think you're going to deliver? And then they've led them and done some whiteboard numbers to say, okay, let's quantify this in a really rough way. We think the value to the organization is this. And then they go and build it and they build it quickly and lots of feedback and it's actually what was needed. How do they deal with that last bit? How do they deal with now looping back and saying, we actually want to know whether it was used, whether it helped that action, whether it helped it deliver that outcome.

For me, generally speaking, unless we're talking about a super small trivial requests that we're just going to turn around and it's not a big project, I would basically insist that there needs to be a measurement and evaluation work stream as part of the project. To use the analogy I used earlier, we need to have a kind of research and citation work stream happening in parallel as part of the project. In the same way that there's going to be a sort of scoping phase design, we're going to get approval for the wireframe. Then in parallel, we're building the data model at the same time as building the dashboard. At some point in that parallel stage, we're also defining what are the metrics for success. And if those metrics are not based on numbers that are readily available, because we've got our metrics tree, because it's part of another report, an MI dashboard, then we go, okay, now we need to set up some kind of measurement for this new thing that is going to be part of the success metric. And then as part of that, we're probably going to build a second dashboard that's going to be measuring that effectiveness. I'll give you an example from a project we were doing a few years ago with a container terminal really complex operation where actually anything you try to change is going to have knock on consequences on many other parts of the operation. But after doing a combination of qualitative and quantitative analysis, we're like, okay, one of the things that we can dramatically improve for the overall productivity of the port and the North Star metric of the port, which is the productivity of the big cranes loading and unloading vessels as they come in, is actually if we can optimize the way we allocate drivers in the trucks, that can have a big impact on the overall productivity of the port. So in that case, there were some metrics that we had already, like the big crane productivity, that was a metric that was firmly established in the organization because they reported on it constantly. But then there were some truck driver productivity related metrics that basically didn't exist in the existing MI suite. So it was like, okay, if we're going to build this product to optimize driver allocations, then we also need to add something to the management information that we produce so that we can be measuring the effectiveness of this model so we can see is it actually delivering the value we were expecting it to deliver. And for me, this points to a really important problem. That a lot of organizations face, which is, it's not just that, oh, if you have more data, that's better and you can make more decisions, you need to have the right data. And it's why I really hate the kind of big data data lake approach of oh, let's just dump all the data we have and then the data scientists or the data miners will find value out of it. Because then usually what happens is when you start a project and you look at what's in the data lake, you realize that actually the super important column for the model you're trying to build actually doesn't make its way into the data lake or it never existed in the source system in the first place. And so when we're deliberate about the business problem we're trying to solve, we can also be deliberate about what are the data points we will need and do they exist. And if they don't exist as part of this project, we need to create those data points. It's not a nice to have. It's a must have condition for the success of the project.

Then I'm going to jump naturally to the next step, which is, okay, so let's say that it's optimized for the big crane and therefore we're going to affect the optimization or the workflow of the drivers. I'm naturally then going to want to build out that metric first so that I can benchmark the current state. So then when we make the changes, we can then see have we had a positive or negative impact. And then I can see how that would then potentially delay the production of the information product that has value. Because now I'm spending time getting ready for the benchmarking data before I then go and build the thing that actually is going to make the change. Is that a natural trap to fall into or is that something that if you can, you should do?

Obviously we're talking about it hypothetically and it's going to be a little bit different in each organization. But even in this hypothetical, the two objections I'd have to that criticism would be, one, okay, if this is going to be a project that'll last several weeks, let's just sequence it so that we start collecting this data from the very start of the project. We make it the first thing we do, not the last thing we do. Actually, we've done this a bunch of times where we said, you know what, you guys aren't collecting this data because it gets deleted at the end of the day. But you know what, if we start collecting it now by the time we will need to use it for the model we're building, we will have just enough data. That's one thing. And in some cases maybe this will mean, yeah, the project will take longer, but if that's what's needed, then that's what's needed. It's the kind of objection that if it was a, let's say, a technical objection to do with the core of what's being built, it would be so much easier mentally to say, look, this is a must have, right? I cannot build a model without this data. Whereas it's very tempting to say, okay, fine, I guess we can build the model without also collecting success metrics. Yeah, maybe you can get away with it once or twice or thrice, but actually if you're systematically doing that, yeah, you're systematically not going to know the benefit of what you're doing. Which for me it's so weird because the whole point of having a data team is to make an organization more data driven. And what being data driven means to me is basically being evidence driven. We're not making decisions because of just gut feel or copying our competition blindly, but we're using evidence. But then we are so bad at basically following our own advice and using data to inform what on earth should we be doing and is it worth doing? So that's the kind of first objection. The second objection is you probably should be collecting that data anyway. Same as what we were talking about with metrics trees, the things that are going to be your success factors. Not always. Sometimes they will be super specific to a project, but other times they're going to be core metrics that you should start tracking anyway because there's going to be other initiatives later down the line that we should be doing. But I would rather, let's say, collect that metric about the truck driver productivity as part of this project. That is a specific optimization we're trying to do instead of do the other thing data teams often say, which is, oh, we need to invest in data foundations. Oh, let's build out all the metrics first so that then we can track success later. For me, that's bad for two reasons. Number one, maybe you will track the right success metrics or the right metrics, or maybe when you get to your new use case, you realize this wasn't the right thing anyway. But also secondly, you are proposing building something that delivers no inherent value on its own, which is all too common. All too common that we spent two years doing a big migration promising the business that after the big migration, we'll finally be able to deliver value. And what do you know, two years later, CDO gets fired, new CDO comes in. First thing they do, they want to rebuild the platform and promise again that the results will come two years later. So for me it's also how can we bundle any kind of, let's call it technical debt or platform investment or foundational investment together with something that's going to deliver value to the business and deliver in minimum increments of value, not oh, let's do the platform stuff first and the value making stuff later when we've left the company and it's someone.

Else'S problem and got the new tools on our CV and got promoted in the next job because that two years was really great fun. Got paid a fortune and got a better job. We used to blame waterfall for big requirements up front and foundational builds. And then lots of organizations went down the agile path and we started seeing six months sprint zeros where again it was just pure foundational build. No engagement with stakeholders, no value. We do have to balance it out between ad hoc behavior where we don't do any foundational work. So there is that horrible balance of building the airplane while you're flying it. That is the balance. But it is balance. It's about the context.

I agree. I'm not advocating for basically being the clueless business person that does not understand what the engineers are on about when they talk about technical debt and just wants ship me these features because they're going to make us money. But we need to be intentional about it. Any piece of technical debt for me is not inherently bad. It's debt I have borrowed from our future selves, from our future data team in order to usually test something. There's no point building something super robust and scalable if we don't know it's worth scaling in the first place. But also maybe it's because tactically, actually it's going to really help if we can use this to provide some results for the business before the end of the quarter, either because there's a financial upside or just because it's going to help us get more trust with them so then they can spend more time with us, involve us more early on in decisions around what we should do and all that good stuff.

One of the things that naturally people want to do is take those outcomes and those ROI statements and link it back to the corporate strategy. There's a strategy PowerPoint somewhere with four boxes or there's the top 11, 15, 25 initiatives that are the most important in the organization. And it's easy to gamify it. It's really easy to say, these metrics support this strategy. It's actually damn hard to find metrics that don't support your strategy in some way using creative language. Do you ever worry about it? Do you ever bother saying that these ROI statements, these metrics, line up with the corporate strategy and the initiatives? Or do you just ignore it?

Because I'm not advocating for just, oh, we should link the initiatives we're doing to some other broader business initiative called, I don't know, be loved by our customers or get trusted by our partners, whatever. For me, that's fluff. Sometimes it's very useful to link it. Maybe so it falls under the right okr or that it gets the attention of the right people. But at the end of the day, most things, you should be able to express their value. In financial terms, it's either money in the bank, we have generated incremental revenue, or we have saved costs that definitely would have accrued otherwise. In other cases, it's a little bit fluffier, but still has a financial number next to it. If we build an AI chatbot that helps our employees get questions about HR policies faster, it's not necessarily that, okay, we've saved 10 hours a week from our HR employee, and that's money in the bank because we're still paying that HR employee full time. Now, if you have a massive organization actually go, no, we will downsize that team. That will be money in the bank. But in other cases, you can just quantify the productivity saving. You go, we have saved the legal department 15 hours a week times 20 employees, paid $100 an hour. And you go, okay, that's not money in the bank, but that is the value of the productivity that the legal team now can redeploy elsewhere. And for me, it's not as good as money in the bank, but sometimes it's necessary. That's where our estimate stops. But that estimation is still so much better than just going, we are making the company more productive and AI ready.

Then. Do you ever bring in the cost side of the data team? One of the things people often say to me is, the data team are trying to do the right thing, but nobody from the organization will engage. The request gets handed across. They want somebody from that part of the organization to work with them as subject matter, expert, whatever. And I often say to them, have you worked out what the data team costs every week? And have you gone back and said, hey, this is four weeks worth of build, that's $100,000 of time. Is it worth $100,000 to the organization? And often data teams don't do that. They don't quantify their own costs. Do you do that as part of understanding the ROI or do you not worry about it as much?

Of course, because ROI is not just what is the value, it's return on the investment. So the I part is basically the cost. And here it can get complicated because what are the fixed costs that we're just assuming are sunk in Very basic example, will I consider the Microsoft Teams license of each of my data scientists as part of the investment or do I just go, look, this is a fixed cost the business has made and we're talking about incremental return on investment, meaning how many additional hours of a data scientist time are we going to spend? Or maybe if we need to hire contractors, it's about the cost of those contractors or it's about the incremental compute that this will cause because we're now going to have to pay databricks a bunch more money because we're doing all these building all these new models. So that takes some, I think finessing and basically it requires you to understand what, I don't want to say what you can get away with in your organization. But more, how does your organization think, are there some costs that ultimately they're subsumed in some centralized budget. And so one, you'd be doing yourself a disservice if you included those because the rest of the business isn't. And two, would maybe skew the picture of whether this is a worthwhile activity for me. I also think the thing that doesn't really work as effectively is to just turn around to the business and go this request is going to cost 100k because usually it's not like they will pay for that if you have some kind of cross charge model or you're like look, we would have to hire new people to support this new initiative. And so we need to work together to make the business case for finance to give us this additional budget, then you can definitely do it and it makes sense and it potentially will come out of the marketing team's budget, not out of yours. Again, depends how your organization does budgeting. But for me I think it more in terms of if I'm running the data team and I know that cost, I know how much it's costing us to keep the lights on every week. First of all, I want to make sure that at the very least we are delivering more than that, like a multiple. If we're costing 100k a week, then any given week we should be delivering at least 110, if not 200k back to the business. And to be reporting that upwards on a regular basis because it's sometimes also easy to forget. And if it's not about that, if it's not about the kind of holistic picture of what is the data team doing at the very least using the even without the cost part, the value part, to figure out, okay, my team could work on any of these three things for the next two months, which one is worth doing. And then using that language, not just to decide internally amongst ourselves, which use case are we going to push for, but also if that involves telling two stakeholders that their baby is going to get deprioritized because we need to work on the other use case, point them to those financial numbers, be like, look, this will cost. Your thing is going to cost the business 200k, but based on our projections, it's only going to make us an extra 100k. And the way I usually frame it is I go, look, if you can find a way to get acceptance for that kind of thing, maybe we can do it. But it's really hard for me to either deprioritize someone else's 10x ROI initiative that they've put in front of us, or just to justify it to my boss that hey, we're going to be costing the business money because Bob really wants us to to build churn prediction model version 57 because version 56 wasn't good enough.

That comes back to that really interesting question of who does the prioritization. Because often in an organization it's a committee, it's a group of stakeholders, whoever's got the biggest voice or whoever's the hippo gets to choose and it gets given to the data team. Whereas if we have this idea of a data product manager or data product leader or whatever you want to call them, then often they are the ones that should be making the final decision. They should be saying, okay, based on the value, then these are the ones that are the most valuable to the organization. So this is the one we're going to build. But often data teams don't work that way. Again, it comes back, they get given a data request and somebody outside the team effectively prioritizes them, tells them what's the next most valuable thing is, but that person is not held to account for the value.

I think my answer as to which model works best Also depends on the organization and the politics, who controls the, the budget, the power, what happens when. But in general I'm against either extreme. I'm definitely against the business has decided the priorities and they just chuck them to the data team who have data product owners whose job is actually to just decide maybe the exact sequence of delivering these things and there's no product work being done there. We can rant about safe and scale dagile another day. But then secondly, I think it's also not good if, especially if it's framed as the data PM is the final decision maker. And for me it's fine if maybe me as the data pm, I've got all these requests from different departments and I do that prioritization but then I want to involve my stakeholders in that. And here it gets a bit murky. It's not I'm going to give everyone a vote, but I'll be like, look guys, we've got these five initiatives and they all have a different number of zeros attached to them in terms of what they're going to bring to the business. And so maybe if Bob says look, I know mine is only a six digit opportunity compared to everyone else's seven digit opportunity, but actually for whatever, let's say strategic reason or because it might cause bigger problems later down the line, it's more important and there it's going to be a balance between sometimes we're going to be more consensus driven depending on how the organization works and other times I go, look Bob, if this is so important, basically you need to escalate upwards because my hands are tied. I cannot put your 100k thing in front of someone else's 9 million opportunity, assuming they're going to take a similar amount of time anyway.

I think we see that in the software product domain. We see that product managers often are reliant on influence to get the right things built and that's effectively a job. I find it interesting that if I look at other shared service type parts of an organization, so if I look at HR and I look at finance, they are a cost and yet they very rarely have to do ROI statements for the value they deliver. And for some reason data does. And my hypothesis is because data typically came out of the IT side of the org and that was always seemed to be somehow having to justify what they're spending their time on more than those shared service organizations. What's your view on that? Is that what you not see that?

Great question. The kind of thing that comes to mind first is actually the fact that we see for example HR as a cost center and often it also as a cost center, especially the more base it functions, is a problem. It's a problem because it means that a lot of things become invisible to the business. The fact that we see our expense tracking software, that's just a fixed cost, it's a cost center, it's the cost of doing business. And that's why it has selected the most competitive offer in terms of which horrible expense software we're going to use. And what doesn't happen then because it doesn't have to prove the value or the return of using that software is there's no explanation of opportunity cost. We're not looking at, for example, okay, on average, because we're using our shitty home built software from 20 years ago that our employees need to use Internet Explorer to access the security, submit their expenses instead of just giving concur some money and using that instead or something even newer than concur, that wastes on average five hours a month from 20% of our employees that do lots of business travel. And when we add up the total cost of that, actually it's a multiple of using the fancier software as a service expense software that it doesn't want to use. So my first challenge is actually I think a lot of teams and departments are seen as cost centers when actually that's not right either because we're not evaluating opportunity costs, we're just evaluating cost in a vacuum. But then secondly, I'd say I think it's fine and good for data to have this slightly unfair expectation of proving our value. And the reason is maybe you tell me, look, by and large we don't need to think about the ROI of the laptops we have and of using Microsoft Office because it's well established, it's commoditized, we know that we just need to have it in our business. We just need to have some HR information software. We just need to have a payroll provider. And for some of those, actually, yeah, cheapest is best. Doesn't make a big difference if we're using the new or fancier payroll provider. Its job is to just get money from A to B in data that's less the case in data. Partly because we're a more nascent profession. We haven't figured out how to professionalize our industry, if that's even the right thing to do. There's a lot more question marks. But also secondly, we're not just providing BAU services to the business, we're helping innovate right. Or at least for me, the more value additive things a data team can do. It's not we're going to churn out BI dashboards showing metrics for the sake of metrics. It's we will help the business use data to improve the quality of decisions to potentially unlock new streams of revenue to build new products. Now with AI and the fact that you basically need your data function to build a lot of the infrastructure if you want to have an AI feature in your product, that's becoming more obvious. But it was true before that as well. It was true before that machine learning could be used not just to recommend a decision in a BI dashboard that someone can ignore, but actually automate a certain part of the process to automatically spit out recommended content for someone looking to watch a movie or looking to buy something from an E commerce store or whatever it might be. I don't want the data team to act like a cost center because then we're just going to default to doing bare minimum low value adding tasks that can be commoditized and expressed in the form of ServiceNow or Jira Analytics. And maybe we need to do a little bit of that. But if that's our main focus, I'm going to change careers.

Yeah, you made me giggle on the expense one. I remember there's probably two examples actually. One was somebody working in the defence forces that could go out and spend billions of dollars on new tanks, but then had to go through seven layers of approval in the expense system to get their parking ticket, the cost to park their car paid, and then somebody that was working for a software company and used to do seven digit sales, and again, same thing would spend four hours of their own personal time on the weekend going through the expense system to claim back the coffees. It's just amazing, right? When you think about the cost of that. All right, just looking at time, I just want to close it out with a question around what makes a good product manager? Let me frame it in a certain way. Back in the day when I was working with teams that were trying to adopt Agile and we're primarily going down the Scrum path to picking up the patterns from Scrum and yeah, good point on safe. We probably don't have time for me to rant enough about that one. But in Scrum, one of the roles that was common was the idea of a Scrum Master. And what I found was business analysts for some reason naturally did the Scrum Master role. If I was looking for somebody to come in as a new person into that role and somebody had a business analyst background, I found they approached that role in a certain way that made them and the team successful. Whereas in my experience, and this is my experience, if somebody came in from a fairly long project management background, they didn't. After a while, I saw the pattern and the pattern really was a project manager wanted to stand in front of the team and a business analyst was happy to stand at the back. But what have you found? If you were looking and saying there is a natural set of skills or a natural role that lends itself to adopting the role of a data product manager, what have you seen? Where would that be?

I've thought about this quite a bit. I'd say it's two things that we can call aptitudes, but I think they're learnable as well. It's not, oh, inherently you need to be like this. It's not about, are you technical, are you not? It's not, have you worked in data? It's not, have you worked in product before? To be a good data product manager and also a product manager in general, I think two things make the most difference. Ownership is number one and basically means are you invested in the outcomes you're trying to enable, or do you just go, my job description said I needed to do this task and I've done it, so it's not my problem. The product is still not making money for the business. That sense of ownership has a huge impact in how you go about executing in your role. And product managers often have to basically fill in the gaps and in one team that gap. My fear, more on the technical leadership side. On another one, it might be to do with helping marketing, or maybe you're doing the marketing because there's no one to support you there. And another one, you need to do financial analysis to figure out the right pricing or to work out the cost. So you need to be fluid. And the best way for someone to embrace that is to have that sense of ownership because they're invested in the outcome, not just thinking of their role in terms of processes and outputs similar to those project management types that you mentioned. And then the second thing is a sense of curiosity, because again, every product is different, every team is different. You need to have and cultivate a sense of curiosity to learn more about your users, about your business, about the technical underpinnings of your product, about what the data actually shows and means, because otherwise you're going to end up being either the kind of PM Persona that doesn't understand the tech at all. Doesn't join the engineers, delegates it fully and they just become an information sifter. They go to their stakeholders and they're like, here's the copy paste answer the engineering team gave me about why the product doesn't work as you liked, instead of, hey, because I understand where your feedback is coming from. I've been working with the engineering team for us to figure out how we need to rebuild this, for it to make sense. And it's also the same in terms of understanding your stakeholders. If you don't have a sense of curiosity to be like, tell me more and let me understand this problem, you're never going to build up that picture of the business and the pain points that you need to connect the dots and do more than just turnover requests that are coming reactively to you. In order for you to be proactive, in order for you to be able to actually propose initiatives to the business that are going to make a strategic impact, you need to have that curiosity to understand what is it that we're doing both on the business side and the tech side. And for me, those are muscles that you can exercise or that you can let atrophy. And so I always recommend to people that are either in the role or thinking about the role is how can you start demonstrating those things? Not just to say in a job interview that, hey, I've done them, but to be well placed to actually do a good job when you get the job.

And I think one of the things you said there was for me one of the really important ones. Be inquisitive around what the problem is. Understand the problem itself before you worry about the solution, because otherwise the solution may not solve the problem. Maybe a great solution, but it may not solve the problem, which again comes from that product thinking is understanding the problem. The customer has maybe jobs to be done. I do like the jobs to be done stuff helps articulate things in a way that makes me think different. And therefore I've got to keep challenging myself to, well, actually, what is the job to be done? What is the problem to be solved? So again, just keep asking yourself that. And if you can't articulate it, that means you don't understand it, which means if you're stuck, which you are, between organizational stakeholders and the data team, you've got to understand both sides. And it's a good way of thinking about it. Excellent. If people want to get hold of you, what's the best way to find you see what you're doing, what you're writing, get in touch.

Best place is probably LinkedIn. You can look me up. There's only two Nixervotus out there and the other one does not do anything related to data. I also write some longer form articles on Substack, but all that stuff is linked on my LinkedIn where I spend most of my time kind of writing shorter form things.

Did you start out on Substack or did you start medium and jump to Substack?

No, I started out on Substack. Before that I had a personal blog that was not data related at all.

It's interesting that kind of LinkedIn and Substack seems to be the default now for people that are sharing their knowledge in the data space. And I don't know about you, we find both of them lacking in certain areas. So it's going to be interesting to see if a new product turns up that takes over both the networking capability and the content sharing. We'll see.

I just keep getting Chrome extension ideas for how to make both of them suck less, and I might start building out some of them just to help my own problems. But yeah, I'm a big fan of.

Both if you can solve that one for me, something that sucks less on both of them was something I'd buy as well. So look forward to that one.

Excellent.

All right, thanks for your time. That's been good in terms of articulating some clear patterns around roi, things that are relatively simple. As Juan Cicada says, don't bore the ocean. I think that's one of the key messages that I got was don't overthink it, do enough that it's useful, and then just keep getting better and better at it.

Thanks for having me, Shane. Thanks for listening to all my ramblings and yeah, 100% keep it simple. It's easy to do complicated. The hard thing is to simplify.

It's been a great chat, thank you for that and I hope everybody has a simply magical day. And that Data Magicians was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata IO.