The pattern of Metric Trees with Timo Dechau - Episode #77

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of Agile Twitter products and data patterns in a simply magical way.

My first book, An Agile Data Guide to Information Product Canvas, is published on Amazon. If you hear your stakeholders say, hey, that's not what I wanted, and then you hear your team reply with but that's what you ask for, then you probably have a problem capturing data requirements. The Information Product Canvas is a pattern template that is a quick and simple way to solve that problem. Head over to agiledataguides.com and click on the Green Book link to buy a copy now.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Timo Deshau.

Hey Timo, thank you for coming on the show today. I would like to talk about metric Trees, something that's intrigued me for a little while. So it's great to have somebody who knows a lot about them on the show. But before we rip into that, why don't you give the audience a bit of background about yourself.

I didn't start out in data. I don't know. The interesting question would be if anyone actually ever started out in data. I started out in product. I will give some connections to Metric Trees later. Why? It basically was really important for me to connect it with product. I started out on product, was very quickly annoyed by people doing features just by gut feeling. I wanted to have a different kind of layer. So therefore I started to introduce analytics data to it. So this was really the early days. It was not even called product analytics at that point of time. The term came later. So I spent a lot of time while walking through different kind of product roles to always setting up data setups for that. I also did the first connect there with data warehouses just a slightly little bit. And then after eight years in product, I decided to go all in data so it was more focusing on the analytics side of things. So I say classic marketing analytics and product analytics at that point of time. It was the high time in Berlin where a lot of E commerce startups were really growing a lot. And so I was doing first data warehouse setups mostly in the marketing analytics space. People wanted to have their own attribution models and these things. I did this. I spent then a lot of time in this kind of area. And then at some point I thought, okay, I want to go back to the roots and want to do more product analytics again. But I wanted to do product analytics in the data warehouse, which required some things that were not so common at the time. So one is like an event data model, which some people had some experience with, but not potentially for product use cases. And then the second thing is there was no approach like amplitude. Postdoc were the classic tools that you do product analytics. They then were possible to put them on top of a warehouse. But then I figured out actually they are most of the time they are too complicated for just doing product work. So in the end you have to have a metric first product analytics approach. And this is where I came back to metrics, where I never really had a good relationship with them. So we might talk about this in this session, but I had to basically revisit my relationship to metrics. And this is where I came across metric trees again. I had them already at university and starting to dive deeper into this.

I've followed your stuff for a long time, so thank you for sharing so much great content. And what's always intrigued me is in the data domain, I could almost see a data warehouse cohort, a product analytics cohort and a data science cohort. And although each of those cohorts are using very similar patterns, very similar techniques, they always seem to use different technology stacks. And then we had the whole, what was it? Composable? Combustible, whatever it is composable. Yeah, yeah. So that idea that you don't need a dedicated product analytics tool, you can just do it off at your cloud data warehouse or cloud data warehouse database. So it's really interesting that there was almost a separate track, wasn't there for product analytics and the techniques you used and the technologies compared to some of the other cohorts or tracks in that data domain.

It's super weird because like sometimes you talk to people in the data space. So let's say who do classic bi. And this was my problem when I was starting out because I was looking for specific way how I can model product data. It's the same as marketing data. Marketing data is a bit less, but like it's a lot behavioral data, so a lot of event data. And you do sequence analysis, so funnels or cohorts and something like that. And it was really a hard time to talk to people who are in the classic BI space and tell them most of the data models don't really work for me because I have to do sequence analysis so I cannot create 200 fact tables. It would be bit crazy. And I really had a hard time to explain. But it was also interesting because I learned a lot from them. I think some learned a little bit from me. But it's interesting because everything calls Data, but I always call it they live on two different planets and they rarely visit each other. I think it's getting more, it's getting better. So I think there's more conversion now and something which I for example, work a lot on. So my last, let's say my last mission, if it's my last, I don't know. But my biggest one is really crossing the whole marketing and product behavioral data with revenue data which then makes it really interesting if you do this.

We had a customer and they were a scale up and so we needed to do some work around standard metrics for a startup, the pirate metrics, ARR or all those kind of good things. And when you first go into those areas you go, how hard can it be? And then you find out, damn, this should be a solved problem right now. The core metrics for startups should be well defined. They all use standard type of software as a service products for capturing subscriptions and signups and all those kind of things. There's three or four of them that everybody tends to use. It shouldn't be hard, it should be patterns out there that you could just implement and make it really easy. And again, whenever you say those words, you know it's going to bite you in the bum. And then you start doing it and you're like, actually for some reason there's a whole lot of complexity just hidden under the top of that layer that bites.

You just do MRR calculation. So it's, let's say technically these are all the same metrics. So there's, let's say a whole metric set around mrr. But the tricky thing is how do you define it under the hood? Because everyone comes to you and in every case company tells me like, no, we have a very standard subscription model, there's nothing crazy in it. And then I always know, okay, yeah, this will hold for two weeks until we do the first investigation into all the edge cases and then we will discover all the edge. Oh yeah, there we had to do something different. Oh yeah. And then yeah, we had to switch from one subscription to the end. So it's always messy. And that's the same with product stuff. I think the pirate metrics are still a really good framework to understand how product and let's say this business around is evolving but let's just go for something like retention or active user or activated. So activated by default is in the end one metric but it can take you weeks and sometimes months to redefine how you basically calculate this because it really comes down how the product works. What kind of use cases are you actually solving? So it's a fascinating, interesting thing, but it's also good because in the end it's one metric that you just see, but it's an iceberg. So on the top you just see this one metric, but it's quite complex underneath. But this is where the fun is. If you really do a really good job there and the definition and so on, it can really change the track for a company.

Even then, though, there's still outside of the metrics, there's some core patterns you've got to decide from an engineering point of view. So again, when I look at product analytics, when I look at event tracking in a software product, typically you have a choice of poll every event, so effectively pull event logs to say lots of things happened and then apply the events you care about after the fact. So more of a Lakey type pattern. Or only define the events you want in the product when at the beginning and then only bring those in and use those. And if you want to look at a new event, you've got to go and define it again. And there's a trade off there right between this. If you get every event, you get this wash of noise and it's really hard to figure out where the signal is. But when I worked with a lot of product teams, they seemed really reticent to define the events that were important. It was easy for them to say, I'll just take everything and we'll look at it later. Versus this is the event that will tell us that this feature was successful. That's actually quite a hard metric to define.

It is, this is, this is another, this is my second passion topic and this is the one where I wrote a book about. I think the problem is when you go into this, you have to distinguish between tracking interactions and tracking actually product usage. And so most of the setups are tracking interactions where people click. And it's quite understandable why you do this because you have to define something. So it's not everyone's job to define what you want to track. So it's usually a side job that you have to do in between. So let's say the most approachable way to do this is you open up your own application. Then you see, okay, where can people actually do something? And so they go, okay, they can click here, they can click here and they can do this. And this is what I mean, you track interactions. If, let's say, if this whole output is just for machines, then it might be okay, we are still not there. That, let's say you can collect all these, let's say very noisy, very granular data and then you have something running over it, finding some patterns. So far I didn't really see things that go in this direction, but it might come at some point. But when a human has to analyze it, this doesn't work. It's far, too far away from what actually defines product success or what actually defines business success. So it has too many noise, as you said. And so therefore what I ask product teams is define the use cases and define the jobs to be done, define the entities that make your product. So in the end, it's classic domain driven design. What I do with them is, okay, let's define the different kind of entities that build up your products, usually five or six. And then we define how does the lifecycle look into this. So let's say we have an account can be created, can be updated, can be deleted. So three events updated. Usually you don't need because there's not often a business value in it if someone updates an account. So who would optimize on account updating? Usually yeah, they're edge cases. And you can do this for everything else. You can do it for subscriptions, you can do it for whatever your product is built on of. And then you can just end up with. This is usually my take. You end up with 15 events and the same you end up with 10 metrics. It's sometimes really quite crazy if you really have a good setup. So right now I was just working on a project where we had one event that was defining six core metrics that could explain if the business is running or not. Because there's always, this is always my interesting take when I work with the client is in every setup you have something which I call a heartbeat event. And this is, this is the central event. If you just need to track one event, this is the one that tells you is your product still alive or not. So for Slack messages for Miro, if, let's say an asset, it's added to a board or so you have always this one event where you know, okay, when this is still coming in, things are good, we don't know about it, but they're still there, looking good. And you have to get to this level, then you can basically tame the chaos that product analytics can cause.

Yeah, it sounds a bit like a North Star metric. This idea that yes, all the other things are important, but actually if you can define the one thing that really is the core focus on that and the other things are useful, but if you don't focus on that one thing.

I always have to fight a little bit for the North Star metric because there are so many definitions out there. I think the worst case is when someone comes to you and says, yeah, new mrrs or North Star metric, where you say, no, it's actually not. Let's say the concept of a North Star metrics is it's always bound to successful customer experience. It has nothing to do with revenue. Obviously. We hope when people have a good customer experience that let's say it has a causal connection to more revenue. And then the North Star metrics is usually also built up by two or three input metrics that lead to it. But in the end it's the same. What you said you have a set of three metrics that could explain how is your product actually delivering value at the moment.

I'm with you. Is that the North Star metric is success for the customer, not success for the business because the customers get the success that you promised them, then your business will grow. But yeah, time and time again I see North Star metrics being internal metrics and maybe we should just go read the definition again or call it something else. South Star Metric or East Star, just give it a different name. That's okay.

Exactly. Yeah, you can always call this. Like this is. Yeah, I don't know. I think North Star Metrics is just a great name. So it's a great label so everyone understands what it is. It sounds great, like adventure. So this is our North Star. We have to go there.

That's my standing joke is that as data people, we laugh about the fact that our stakeholders can't define active customer. Active subscription. Yeah, as in the data domain, we argue about the definition of a semantic layer, a data product, a North Star metric, day in, day out, as bad, if not worse than the end. So on that note, let's go and talk about metric trees. So if somebody said to you, what the hell is a metric tree? How would you describe it?

First of all, the same. I think it might be also a definition problem. So some people call it driver tree. There's also like the term of KPI ts. I'm not really good in definitions. I know that they have slightly different variations of it. So I give you my explanation and my definition of it. So one problem, what you often have is metrics is when you take a metric alone. So let's say you have a software as a service, so you're not really sure what you should Measure. You're on LinkedIn, one of these posts where someone is posting, hey, I just compiled the greatest collection of software as a service metrics like comment metrics, and you get my PDF with these 100 metrics. These things never really work for someone. And the problem is why they don't work is because everyone picks a metric and looks at it in isolation. So I don't know. Yeah, for example, mrr, a monthly recurring revenue. So what is have a standard alone there, and then you don't really have an idea what you should do with it. Because two things are missing. And what defines the metric tree mostly is it's missing a relationship. So let's say every metric has a relationship to another metric. I cannot even think about one which is really standing alone. So usually you have a relationship. And why do you need this relationship? Because not every metric is very actionable. So we talked about this North Star, when you say, hey, revenues are North Star. So the problem with mrr, months of recurring revenue is it's an output metric, so it's something that comes out at the end. So when the CEO goes into a meeting and, and let's say assembles all the heads of the different teams and says, we have to increase mrr, everyone would say, yeah, sure, sure, it's our business model, so it makes sense. But no one would immediately come up with an idea, oh, yeah, sure, we should do that to increase more mrr. No, you usually break it down. So you break it down into, oh, okay, when we need MRR and maybe we need more accounts because they could end up in a subscription and then this would be new. Mr. But then this is interesting because this is already the first step to build automatic tree. Because then you have mrr and then you say, okay, what is actually making up mrr? New subscribers multiplied with an average plan price or average price. And then new subscribers you can build up from new accounts with a conversion rate. And then you get into your first version of a metric tree. And I think the nice thing about a metric tree is that it explains how you can get from something that you will end up with to something that you can directly influence. So if you go to a marketing team, tell them, hey, we need more new accounts, they usually have an idea what to do. So they were like, oh, yeah, okay, maybe we have to run more webinars or we have to invest in the podcast or whatever. And so there you make it actionable. And at least this is my driving force where I Metric trees are interesting because they help you to break down something that you want to achieve to something that you can actually do. And this is the tricky thing. So if a metric is not actionable, we'll have a hard life. It lives lonely on this dashboard and no one has an idea what to do with it.

So if I think about it, what you're really saying is there's a bunch of metrics and they have a relationship to each other and we treat it like a tree. So if we think about an hr, human resources org chart where we typically see a tree of people and people report up this tree, what we're saying is if we can find the relationship metrics that behave that way, so these metrics support this other metric which supports this other metric, then that relationship helps us work out what we need to change in our business to move one or many of those metrics.

Yes.

Is that kind of the concept?

Yeah, that's the concept. You can also use it for that. Where I like to use it is to really try to explain how the business works on a, let's say, common concept that at least the data team understands and that the business team understands. And this is something which I would say the metric tree so far worked best. So I tried different kind of formats to bring both teams together. We did it. Okay. What kind of events do we have to measure? Usually it doesn't really work well because it's too abstract and you sometimes find common ground, but it's too far away from what people really care about. If you really build up a metric tree, you have really interesting conversations with, let's say, when I do these workshops where usually with startups, where it's easier to get relevant people into one room. So then we have people from all the different kind of disciplines and we have really good conversations because for a lot of people, it's for the first time that they see how actually the revenue is coming together in the company. So this is, let's say, often unfortunately for data people, it's the first time that they see, oh, this is how we calculate revenue. Because it's usually one person who does something this part, but not everyone. And then for products, the same, they are like, oh, okay, I was not aware that we also do this to get revenue. So it's a really interesting exercise that can open up a lot of black boxes and understands how the whole system is actually building up. And then obviously if you go a little bit more down, then also all the different teams might find a place in this tree where they say, oh, this is actually where I work hard for the data team, they usually don't find a place there, but they're the ones who can build the metric tree and can provide the metrics. But let's say you can have a part where you can see what product can do. You will have a part where you can see what sales do. Then you will have a part where you can see, oh, this is actually the influence area of marketing. And then in theory, you can analyze how these different kind of levers that you have, let's say on a lower level, how they're actually attributing to the whole final output metrics. Every time I did this kind of format in a workshop, it was always some kind of eye opening in there where someone said, oh, I never looked into this. Or someone even said, actually right now, I think with some initiatives, we are starting to hurt our revenue. And so no one has recognized that, because now we see it for the first time that when we do this, let's say, this stream of revenue, we'll have a problem which no one really thought about. And so this is something where metrics 3 definitely can help to bring transparency into, let's say, the mechanics, how the business actually making money. That often is completely overlooked because a dashboard in theory does it, but it doesn't have the structure. The tree structure is nice because someone can look at it and immediately understands what it does. Yeah. So go from the bottom up to the top or the other way down.

I think that's part of the core value of it, is that simple map. Because what we know is we know there's a lot of complexity. And when we draw a simple map, a simple diagram, two things happen. People can visually ingest it and understand it. There's something about a human looking at a map that just naturally sees the pattern. And the second part is you can point to part of the map and everybody knows they're talking about the same thing. So that idea of different parts of the organization, different silos, understanding how they affect another metric. So your example, how do we increase monthly recurring revenue? Well, there's a bunch of metrics that the marketing team can affect. Number of ads placed, number of prospects found, optimizing the funnel from prospect to sign up. And they can look at that part of their map and go, I can do some work in there. And that, in theory, will increase our monthly recurring revenue because that's up the tree. So that idea of a map gives us understanding and also gives us a clarity of a shared language. I'm focusing on this part of the tree.

Yeah, I'm very happy that you come up with that you say the word map because this is actually what it is. It's one map that, that can be very useful. So at least when I create one for the projects, it's not that we constantly work with it. So we might get to this point, like how operational are metric trees? But where I always use it is when we come together and discuss big things, okay, what kind of metrics we should focus on, let's say what kind of initiatives we should look into. So this can always help to have it then sitting around. And when we say, okay, look, now we want to focus on this kind of area, you can always look then in the, let's say in the tree. And the nice thing is when you have the tree with some data, then you can see, okay, we focus on this kind of area because we have the feeling we have a lot of volume here. But we can see, for example, the conversion rates in this kind of areas are not really high. So there's, let's say there's quite nice potential for us to improve things just slightly, a little bit, but see a big impact because we see that a lot of push comes in there. And so then we can analyze how this works for that it's really nice to have it because again, everyone immediately knows where we are. So it's not that you have to to long explain how this might have an impact on revenue because immediately everyone sees that.

And the good thing for data teams is it means they don't have to go and look at yet Another stupid strategy, PowerPoint with four boxes that have no context and no understanding and no data, you might have to. But at least they can say that that part of the pyramid or what are we doing this week, the circle, whatever the latest consulting picture is for, for selling a story. How does that map to the tree? One of the things you said though was data teams would struggle to figure out which metrics are theirs. And that kind of gave me an idea. So one of the things I work on a lot is this idea of an information product. And one of the key things I teach teams is focus on the action and the outcome. If you don't understand what action the stakeholder is going to take and what outcome is going to be delivered from the action and the value of that outcome, then really there's a risk. You're doing data work that has no value. Why we say stakeholders are accountable for that. Actually, as data teams we should be as well, we should be holding our stakeholders to account. That actually they can describe the value, at least the outcome that's been taken. So what would be interesting is if there's a metric tree in place, a data team, which should be able to point to the metrics that they're doing data work for or with to improve. I'm working with the marketing team to use data to reduce the time between a prospect being identified and signing up for an account.

Yeah.

And so they should actually be able to use the metric tree to show where the data work is, adding value to the organization in conjunction with those stakeholders, those different business operating groups.

Yeah, exactly. I think this is where it can really play a really nice role because you can sit in this, let's say you work with the marketing team. Marketing team does some planning for the next two or three months. So they have some ideas. Let's say they come up with three initiatives, what they want to do, and so you can take every initiative and you can then say, let's say, okay, here on the big company metric tree, these initiatives, they're trying to improve these kind of areas. And then we can use this metric there as always as our control instance, to really see, okay, do we see some impact? Because that's a tricky thing. You can run a lot of initiatives and locally looks really great. But when you look at it on the big picture, everyone is, yeah, I don't know, a B test look great, but I don't see any kind of uptick somewhere. So it's definitely nice to have, okay, this is the one big metric that it should, let's say, influence, and let's see if we can fire it up and enough that we can see some influence. Then what you can do, the second thing is let's say you can create different views of metric trees. So you can do this very high level for a company, but then you can also take every initiative and just build a metric tree for every initiative. This is something which I often like to do because again, it helps you to understand, okay, how do we measure the success of this initiative? So you might have this one big metric that we identify in the company metric tree that let's say is at the top end. And then we break it down for this kind of initiative. And so then it really comes, let's say you want to improve conversion rate. So then we can build the whole surrounding around this conversion rate and maybe even break it down by three different channels, because this is what we're trying to achieve. We're trying to get more people from this kind of channel. So then you basically Bake strategy into the metric tree and the strategy connects directly to the initiative that marketing is driving. And then everyone has a very clear idea what you do. And then also you will build something that in the end can marketing and you can use to explain if this initiative was successful or not. And you can validate it before you can ask people, okay, look, when we deliver this, does this make sense to you? When let's say when the initiative is over in four weeks and we report on this metrics and we define okay, success looks like when we move this part, I don't know, by 10%. So does everyone agree with that? Because I have the feeling it makes it for a lot of people easier to think in that way so that they don't say, yeah, I don't have really idea if I should agree or not to this. Because at least, yeah, I think metric, at least they understand, they might ask, okay, how do we define this kind of metric? Well that's fine. So then we can spend some time to say okay, how we define it. But I like this approach a lot. Also like to not always see the metric tree as this one big okay, we explain the whole business model, but really to use it to explain how I run a specific initiative. It's also nice for me, let's say personally when I work on these, let's say as a supporting part for data for these initiatives. It's a great brainstorming tool for me. So let's say someone comes up with this initiative, I do a first version of a metric tree. Then I say, does it really represent what they're actually doing there? And then I say yeah to a very generic part, but maybe not customized enough. So I'm always trying to tweak the metric tree that the people who run the initiative immediately find it in there. So let's say I do something for E Commerce and the E commerce is really pushing and getting high loyal customers. Let's say they really want to improve the segment of high loyal customers who buy all the time. There's this let's say high buyers that you sometimes have. So when I just report on let's say new customers or returning customers, then I don't really have it covered because yes, we are going for returning customers, but for a special segment within returning customers. So therefore what I can do is I can say, okay, I break it down in three groups. A new returning and let's say VIP customers. And then with the VIP customers for the first time I make it visible what kind of impact the initiative right now has, let's say I can see, okay, how is the share of these users is growing or not growing or whatever. And so this gives me a lot of playroom to build something that can really support what the business is trying to do, but do it in a language or in a way that at least most of the people understand.

All righty. So I've got a shit ton of notes right now, so let's, let's go through them one by one because there's so much gold in there. So the first thing you called out is this idea that actually metric trees is a shared language. Yeah. So you can, you'll define a metric tree for what you think you heard from the organization of how their business operates. And by putting that map, by putting that tree in front of them, they're going to identify where you've got it wrong. They're going to look at it and go, yeah, oh, no, hold on. We don't do that or we're different, or I don't understand. It becomes that visual shared language of an entire business and their operating model.

It maps the process to some degree, and that makes it easy for people to see, okay, is it actually what we do?

And then the second thing is you said when you want to go and do something new, you want to do an initiative or some investment or change a process or go into a new market, you can pick the metrics you think you're going to impact, and you can guess how much you're going to impact them by. I remember many years ago, probably late 90s, early 2000s, I got into the whole balance scorecard thing. I was working for a vendor. Balance scorecards were hot. A couple of them were trying to build software. And one of the things was you had this metric and effectively you could put a budget on it. We're going to increase that by 10% over the next quarter. Now, the software itself was pretty hokey, but it was the conversations around, what are we doing and how's it going to influence that metric? So that's one of the key things you called out, is by saying that if we do these things, let's guess which metrics are going to change. Is that what you're saying?

Yes, exactly. And then also really make sure that you measure this metric in the right kind of way so that, for example, let's say you have a corridor that you can see. Okay, is this a normal movement of the metric or not? Or that the initiative is big enough that it can move the metric. This is a tricky thing. But often, at least when I work in product or marketing is that often you see a lot of initiatives that are just, let's say they're very tiny bits because, I don't know, they're not really bold. And then.

You will not see anything. And you can even tell this before, okay, we try to optimize 5% of the typical audiences that we get in there. So we are working on this. There might be still a strategic reason for that, but then you can still make it clear, okay, so we work on that. But because let's say the sample or let's say the audience is so small, we will not see an impact when we look at the completely blended global metric. So then, for example, you have to break down this metric, maybe in different kind of parts that you can still see something. But this check in really helps to see, okay, how do we actually want to see if it makes an impact or not? Because sometimes, just because of the setup, you won't see anything. You can already know this, that it might not happen.

And that was one of the problems with the balanced scorecard. So we had this idea of cause and effect. We said that I think you call them input and output metrics, but this idea that if I improve this metric on the bottom or the left, then it has a cause and effect with a metric above it or on the right. And back then we really wanted to find correlation. We really wanted to say, well, actually, if we just threw all the data at the machine, the machine should be able to find causality or correlation between the metrics. I should be able to codify this. It should be accurate. And we never got there. We still don't have the technology or the patterns to do that at the moment.

No, I don't think so. This is definitely not my area of expertise. So I know some people do some things like that. So they try to find out. Because you have some relationship when you build a metric tree that are not deterministic. So let's say a classic metric tree you can write as an equation because it's basically also quite nice that you can do it. But sometimes, for example, let's say you have a metric which is called active users. And so you have a very well defined active user definition where you say, okay, the people don't just show up, they also do valuable stuff within your product. And so once they do it, and they do it within the last 30 days, we basically flag them as active users. So there's definitely a correlation between active users. And at some point, starting a Subscription. Let's say you have a free plan and so you have an active user and a free plan. So. So there's a correlation between the both because you might assume, okay, someone has to be active to end up in a subscription. But let's say it's not a direct connection. So you cannot really say okay, whenever we get someone as an active user, we have a probability of, I don't know, 57% that they will end up in a subscription. I guess this is more straightforward to calculate. But even there I never really came across, let's say super good models that can predict this. Very good. If someone knows this, please let me know. Ping me on LinkedIn. I'm always interested.

Now the answer is just AI slop, right? You can just AI and maybe you can. Maybe it's good that non deterministic pattern matching would be really good at this. I don't know, I haven't tried lately.

So if we go to this what you had before, like we create a lot of noise and we might throw it in and we might find something. Yes, that's. Yeah. But I think for correlation analyze is potentially not.

But the key is the visualization of a map. The conversations of what on the map, the conversations of what you're doing to improve the numbers on the map, the relationships that actually is where the value is. So yes, if we could get programmatic deterministic correlation across the metrics that would be awesome. But actually the conversations. So the next one is our natural reaction is to boil the ocean. If I think about an information product that's lifetime value. Yeah. I tend to say to organizations, why don't we break it down? We know that to do a lifetime value model we actually have to have revenue cost to serve churn. There's a whole sub. Sub model sub products that have value. Why don't we build the revenue model first and give you a revenue information product? Why don't we do the cost to serve second and then over time we'll get to that lifetime value. I can imagine with metric trees the natural reaction for some data people is to define every metric. So draw the map end to end, define every metric before we even start doing anything. Where for me, I like the ability to change fast. So for me I would be going maybe sketch out a map really quickly as kind of like a blueprint of what we think it looks like and then focus on some metrics and define them and build them and deliver them and monitor them and use that information and then kind of color in the map. Over time, as you learn more, how do you do it?

So I had a phase where, let's say I was in the rabbit hole and I did one exercise where I was trying to map my whole content production as the most extensive metric tree possible. And it was, I still have it somewhere in Miro. It was really a monster.

It was like no one could ever implement this. I think it had in the end, I don't know, 90 nodes and this doesn't work. So I could easily see that it was a nice exercise, was a nice thing to do over the weekend, but it's non operational so it doesn't provide any operational value to do this. And by the way, it would not even possible to track the whole thing because it would include a lot of attribution stuff which is not possible. So the same what you say. I think the model that I use now is try to really keep a simple, I don't know, let's say also try more to work with subtrees and don't be too, let's say don't be too deterministic with it. So give yourself the liberty to say, hey, I create, let's say, a specific tree for this kind of new product feature that gets introduced and have no idea how it connected to the other tree. That's fine, I live with that. Because local optimization is still better than no optimization. So therefore be nice to yourself. But no, you definitely have to keep it quite simple and then you have to know what you can do with it. So I know that some people use it for root cause analysis and I think for that it's really quite nice because you can say, okay look, we lost so much new revenue, let's say compared to last month. So can we follow up with the tree to really see where do we see where stuff got off? So it can help you to, let's say, get a starting point. But then still the deep dive is something different. The metric tree will not tell you where it happened. And then another thing that I discovered is often overlooked, especially when you talk about metrics and the metric tree, it's really big risk is you have time is completely out of this thing. So the metric tree lives in a timeless space. So the problem is, for example, you increase new accounts in the example that I had before and then you wonder, okay, why don't I see any kind of new revenue? Because it happens in 60, 90 days. So whatever your model is, or whatever the average time is that people usually take to upgrade into subscriptions. So the metric tree always does A very bad job where things, let's say down in the branches are already impacting and you see the impact three months later at the top and you have no idea where it's coming from. So if you would do it properly, you would do a cohort analysis where you cohort everything that happens, something. But this doesn't work on a tree. I think the important thing about working with trees is really to know the limitations where it's great or not. As I said, for me, mostly for planning, for brainstorming, for communication. I guess some people use it also for these check in meetings, let's say have a weekly meeting. You check quickly, okay, how's the business performing in specific areas? I think that can work as well. But it's not like that. Some people thought, oh, get rid of the dashboards and we just do metric trees. So no, that doesn't really work. You still need specific, let's say explorative dashboards to figure out why a metric is actually looking a little bit wonky this month.

I think back in the balanced scorecard days there was this concept of simulation. It was almost like a digital twin, although that term wasn't around then from memory. So you could actually go in and say, if I change this input metric by a certain amount, what's the flow through? And from memory we were inferring a delay, like you said, number of prospects coming into the funnel, how long will it be before they create an account and go into a paid plan? How long would it be before that money turns up? If it's a 30 day window for the bill? And so you bake that into the model to a degree so you could simulate some changes. But that was quite technical.

What I often do is I take a metric tree and I just break it down in a spreadsheet and then create something which we can call a growth model. Where I then have the metric tree on the left defining all the different kind of rows and then I can put a timeline on the right in there and then I can just model what you just described. I can do some forecasting and for me the interesting part is I see the mechanics of the business and then look, I can change some conversion values and can see, okay, what impact does it have when we actually increase this by 10% and so on. And then I see it, how it plays down. So it's a very basic and amateur way to do forecast simulation, but often quite enough for most of the stuff.

And again, that relationship's an interesting one and just made me think about something. We're doing a whole lot of work in what we call the context planes. It's this idea of taking the context of our data and bringing it back into. For us, it's centralized. And we think about it as four types of context. We think about as business context, actions, outcomes, glossaries, descriptions, those kinds of things. We think about it as structural context. So physical schema, data types, and all those kind of things we know about our data, we think about it as operational context. So when was data loaded, when was it refreshed, what's the quality score? Those kind of things. And the last one we think about is agent context, the prompting reinforcement, those kind of things. And so there's a bunch of object types we have. So we have actions as an object type, outcomes and auction type, metrics as an object type, a factor, a value as an object type. And then the last part of it is the relationships across those object types. And the reason we're working on this is this context map. Effectively, if you give it to an LLM, it's really good at using it to answer questions. So you can do what we call blast radius. If I change this bit of code, if I change this object, if I change this metric, what's impacted? And so thinking about metric trees, they're giving us a relationship across those metrics, which is effectively describing the business model for the organization. And that relationship would be really valuable to an LLM if you're using an AI agent, because you can say, if I touch this metric, you know what the relationships are, infer what's going to happen. And that actually might be a non deterministic way of getting a simulation quicker than having to build really complex data models.

Yes. So I was experimenting a little bit with that. Let's say in the easiest way, you just do a copy of your metric tree and put it in and then say, okay, can you please run this in specific ways? I also did some experimentation with let's say descriptive YAML format for metric trees, was not really happy with that. So definitely so far I have abandoned the idea to do this. But no, you're right also, because when you put it in lmm, then for example, you can ask, okay, can we run a cohort simulation on top of that. So let's assume we run this initiative in the next months where we think, okay, we will increase this kind of metric and then we want to see how the impact will look like over the next months. And what I often see, for example, when I have, let's say a full Data set up for let's say a startup. And then the metric tree is always. You can use it in an LLM and then you can say, okay, let's brainstorm on this. Right now we have this problem here or we want to dive deeper into retention and this is the whole picture. So can you help us to break it down? So can you give us different kind of versions of what. Let's say what would the level underneath look like? And, and still at the moment this is my favorite LMM workflow. Or let's say use case is really to use it as this massive brainstorming machine where you can say look, okay, let's look at this angle and I can say, oh, okay, let's slightly change this kind of angle if we bring this. And then because you have so many things then mapped out to you so you can immediately say, oh yeah, this is the direction we should follow on. Let's go deeper there. So it definitely helps to not do crazy stuff because you give it, let's say a form or structure.

Yeah. And then my natural reaction is oh no, because LLMs are non deterministic and so the simulations it's going to run is wrong. But we've just already said that correlation of cause and effect across the metrics is hard to do if not impossible. So a non deterministic engine is just behaving like a human going, this is what the patterns we see. So actually using it to do those, what if analysis, those simulations, those how are those things related is just as valuable as humans doing it. I hadn't thought about it that way.

And they usually nowadays would take the other approach and would write a Python program. Often when you ask these things they say, yeah, let me write a quick script for you. Usually does that and then you can.

Just check interesting space. I think we've seen metric layers come out as bi semantic layers and they're not, from my view, they're not making it right. They're not really getting traction in the market. Maybe we'll see a reinvigoration of the Balanced scorecard products as metric 3 products next because there's probably some value there. All right, so let's just go back to basics. I'm rocking into an organization or I'm in organization and I want to start off this journey. We've talked about the map is the most important thing, the shared conversation. So we need to build that out over time and we've talked about doing it step by step. Don't bore the ocean. We've talked about the fact that actually defining metrics are hard. So each one of those you're going to think is, oh, it's only mrr, how hard can that be? You'll find out. So we know that each metric to define it and implement it actually is a lot harder than we normally think and takes time. Where do you start? So if we think about this idea of input and output metrics and metrics in the middle, that we've got this idea of a heartbeat metric, the core metric for your organization that actually is the one you want to look at the most and that's going to be made up of a whole lot of input metrics. We know that. Where do you start? You tend to start it the input side, you tend to start at the output side, you tend to start in the middle. How do you decide which metric to do first?

That's a good question. I start on two areas. Area number one is I usually bring, let's say a set of people of this company into one room that can explain, let's say the whole customer journey or the whole customer flows in this company to me. I usually work with, let's say startups which are between seed and series A. So therefore the whole business model or the whole business processes are not super complex yet. So it's definitely possible in three hours to map the whole thing out. So I do a classic event storming session. So not classic event storming is a little bit broader and it does more things. So like I do a reduced version because I'm just interested in specific things. But when you ask a company to explain how the customer is basically going through the whole process, then you will identify these things. So you will identify, okay, what are the actually important points that they have to come to that in the end, like it means success for you. So we will find this one place usually pretty quickly where we say, okay, this is a sweet spot. So that we definitely have to have a metric for that. We just have to see how we define it. For example, an active user is a North Star metrics in general, it's a definition process that takes longer because you come up with a hypothesis where you say, okay, people have to, okay, I'm just making this up. Let's say when you're a Miro, so they have to create one new dashboard every month, they have to share at least two and then they have to add at least 20 cards on a board or so. So then you would say they're active. So this definition has to be fine tuned over time to see okay, does it actually stick? This comes later. So I start with understanding the whole process. This is part number one, part number two. And this was something that I learned later. So I started very early on with these event storming maps because I use it already in other projects for other things. Things. They're really great to understand a process very quickly. The part that was always missing for me, that came later for me, was I sit down with the leadership team to understand strategy. I really want to understand where they want to go in the next six months. So what is the thing they want to improve, which kind of direction they want to go. Because from there I get the priorities. What you just asked, okay, where do we get started? So what do we have to build out first? I always try to bring it as close as possible to the strategy that the company right now is doing. Because if I do it, I usually get a lot of more interest by management teams, by everyone, I mean, than by everyone else, because they have to report towards this. If I won't do this, people would like, yeah, it's interesting, but it doesn't really, let's say, count into our current initiative. So if I fail to support a current push, strategic push into something, then usually I get a really low adoption rate. So. So this is like the second, I would say, most essential part that I really try to understand. Okay, what they're trying to achieve in the next six months.

So once you've blueprinted out or event stormed the kind of. I think of it as a blueprint, you're creating a hypothesis of what the metric tree might look like when you're finished. Then what you're saying is you then look and talk to them about their strategy to figure out where you should zoom in on. You're basically saying, as part of that event map, that metric. Metric map. And based on your strategy, I think we should do these ones first because they seem the most valuable, the ones that will get the most traction or have the largest conversation and adoption internally, which is the goal is to get more people to understand it and use it. And go, yeah, these are valuable. Let's do the rest of them.

It's even slightly different. It's not like just highlighting the areas. It's often there's a different version of a metric tree that I use. So you can take a metric and you can break it down by specific things. So this is the classic thing that you do in BI as well. So you have sales, you can break it down by something. You can do the same thing with the Metric tree. So you can take a total metric and then you can break it down by some kind of segmentation. Whatever it is in this segmentation, I can try to incorporate the different kind of strategy. So I want to have let's say someone, okay, let's say Miro. I'm always using Miro because most of the people have used it before. So let's say they push into AI supported workspaces. So then I'm trying to get up metrics that will highlight these, let's say the adoption of AI supported workspaces versus non AI supported workspaces. So that everyone on the first glance can immediately see are we making progress with that or not? And so it's really like, like creating a variant of the generic metric tree to say, okay, this is how it would be useful for the current strategy.

Okay, so let me play that one back because I just, I want to make sure I get it. And it's this idea of the buy statements of these metrics. I want this metric by channel. Before we started, we talked about the tools we use to create our content and we talked about the fact that I use zencastr for recording, I use descript for editing, and I use podbean for publishing. And now what I'm seeing is a convergence. Yeah, I'm seeing each of those three products add, add the other features that I do need into their product. And I was ranting about how that actually degrades the value of the one thing I use them for. So let's say that we're zencastr and we want to move into the real time streaming and video editing. So what I'd be looking for is the metric of active, whatever, maybe active user, maybe active sessions. And then I want to be saying, okay, how many people are moving from audio recording only to video recording? Because that's input metric. They're not doing that. They're never going to use our video editing feature. And then how many of those move from video recording to video editing and then segmenting it maybe by region. And we say, actually the US is the target market, so how many people? And by doing that we're effectively going, there's a really small metric. Effectively, we've broken it down to its smallest part, but it's there to support our strategy of getting more people using our video editing editing. And so that's okay, that makes sense for me.

No, it's a perfect example. So exactly like that. And usually it makes it a lot easier than to talk to the people and people find it Very interesting. They want to see the results from that.

And you can get instant feedback because somebody can say, actually, yes, we're going after the video editing market. That's one of our initiatives. Because we all know that Most organizations have 52 initiatives every quarter, but they can say, that's not our most valuable initiative. This one over here actually is our most valuable and that's what we should work on. So when you're talking, you're bringing a whole lot of patterns to the table to support the metric trees. So you're talking about event storming, you've talked about jobs to be done, you talked about domain driven design, you've talked about journey mapping. There's a whole lot of really valuable product and data and agile patterns you've just talked about.

You can see where I'm coming from. So there's a big product influence in this whole thing.

Yeah, they're all valuable patterns that support what seems to be a simple pattern of metric trees, which is define a metric and say how they relate. Who does the work then? Because if I talk about a data team often, I don't see data teams applying product thinking or product patterns. They're data teams. And so you get this idea of a data product manager now who kind of bridges it, but who is the type of person that would do this work, given the level of patent understanding and experience that you actually need to make metric trees work to their best ability, not just defining a metric on a dashboard. Who do you see doing it?

That's an excellent question. I don't really see data product managers, obviously, because I don't really see them happening so much. So from my experience, when I talk to some people who use metric trees, I would say most of the times these are ahead of data leading the data team. Why is it good fit? Because they have a strategic role. And some of these people struggle a little bit to have a strategic role because they love data stuff. So it's a tricky process to go from something which was very operational to something which becomes more strategic. But let's say the one or two people that I had long talks about were all heading data teams and they were completely concerned about is, okay, how can we do a better job to connect what actually the business is doing. They also had a high frustration level, to be fair. So they define North Star metrics without us. And so they do all these crazy things where we actually, as a data team should be involved. But there's a reason why they don't talk to us. So maybe we are not really well prepared for that. And so this is where they invest their time. And I think for someone who's leading a data team, I think all these things are good things to learn because they have a very strategic aspect. Aspect to it and you have the possibility to understand the business, which helps a lot. Let's say as the head of data, you are the chief sales officer for all the data work. So therefore, to have a good idea where you work makes most of the impact and gets you, let's say, gets your team a lot of fame. It's definitely not bad to learn. So I would place it most likely there to not also create a different new role for that.

And it comes back to that value of the data team, value of the things they're doing. And in my head. So within the information product canvas, we have this area called business questions and the questions that we want to answer with that product. And from there I can infer metrics, relatively easy. And so therefore, if I think about it as a map, it goes business question is supported by a metric and an action is supported by a business question. If I answer that business question, what action are you going to take? Take now when I think about metrics in a metric tree as more a business model, I've got to think about where it fits in my. In the model in my head. Because now I'm going, oh, actually, hold on. They're not really a subset of a business question. They're sitting in this pattern map in a different place. I'm going to need to think about that one because that's changed my thinking around metric trees because I just started off this conversation around it's just a metric. And you know, I need a metrics like rather than. It is actually a simulation or a visualization of a business model.

Yeah, yeah, business process. So like both. It's more in this direction.

Yeah, that's perfect. All right, so if people want to find you and find your writing and you run a course that teaches some or all of this stuff.

Yeah, not yet. Okay, so a guest and I. So if, let's say if you look for metric tree content, so you will come across a small group of people who wrote about it. And so I guess this one, I'm another one. There are some others. ABHI has written a lot of stuff about it, Ollie as well. So we are working on a course, but we want to do it right. So therefore this takes a bit of time. So we did already some iterations. We have some ideas now in general, if people want to Read what I'm writing. So you can go to timo-au.com where I write my newsletter where I sometimes write about metric trees. These metrics, product analytics, event data models, is usually like what I write about. And sometimes I do this as well on LinkedIn. You can also just follow me there if you have a direct question. Or you can also just write me directly on LinkedIn. I usually try to answer them.

Excellent. I read your content. It's great. You can tell when somebody knows your craft and then they spend the time to figure out how to write something that teaches it. Because I don't know about you, I find I can write content quickly if I don't worry about simplicity. But when I try and write something to explain a complex thing with simplicity, it takes me a long time to get it where I'm like, yep, I think I'm happy. And then testing it, give it to somebody and they can tell me what it actually means. That is a lot harder. And the way you write, I can read it and go, ah, actually I think I get it.

Yeah, that's good. No, like I just wrote a. I think this is the longest that I ever wrote like an 8,000 word piece about the history and future of digital analytics. And I think this thing was brewing in my head for four months, back and forth with different kind of variations. So like some sketches on a paper and then I'm not really happy. No, this is the wrong direction. And so yeah, these things take long until they.

So on that one, what we're seeing now is we're seeing generation of AI apps incredibly fast. And, and 30, 40 years ago, we used to struggle with one system sitting on mainframe or whatever and getting the data. And then we moved to enterprise resource planning and CRM. So we always ended up with 5 to 10. And then we moved to software as a service. So we ended up with 50 to 100 to 1000 systems. Now with the ability to spawn up an app in a heartbeat, we're going to end up with a problem of 1000, 10,000 systems. Systems that capture data. For an organization that allows that to happen, that's going to change the way we do product analytics, isn't it? Or is it not? Do you think that the techniques and patterns and technologies we have today are going to be able to handle this idea that there are 20,000 one shot apps in an organization that are being built for a very small use case, for a very small set of Personas?

That's a very interesting question. I would say in general not so, for example, what really took me some time to get to this point, others came to this point as well. Unfortunately, it's not so much teach in product analytics that in the end, the tricky thing in product analytics, where do you find a high abstraction layer that still explains the product enough, but it's simple enough that you can basically do some calculation and good stuff around it? So what really works well is the user state model, which is in the end a gross model. So you say, okay, we have a new user, new user can become activated, it become active, it can become at risk when it stopped being active, then it can be dormant when basically nothing happens anymore, and then it can be resurrected and whatever. So you can basically create a loop where people can move through different kind of states. If you take this kind of model, then you can have 400 things under it as long as you can map the things that happen in this to one account. This is always the tricky thing in product analytics. So as long as you can do this, you can still then abstract it on this high level. You can still say, okay, people do different things, or let's say we capture different things, what the people are doing, and let's say these 100 internal tools that we are using, as long as we can bubble them up into one place where we can say, okay, look, when people show these kind of activities, we flag them as an active user. I would say the system still would work. I would say this is the only escape. If you go the classic product analytics approach where you try to track everything and then try to figure something out, obviously this is something you cannot win. You really have to go on a very high abstraction layer. I think the tricky thing is really how you do identity stitching. So let's say the basic function of making marketing and product analytics work is you have to combine all these different kind of things. So if you cannot combine all the different kind of signals happening somewhere, then obviously you cannot analyze them for this one account. So then you basically have 20 phantom accounts that are actually one, but you have no idea that they are. And this, I think can be really an issue, especially when these tools can pop up everywhere and the company, you don't have a concept to make sure, oh, at least we should identify the people in similar ways everywhere.

Actually, you just give me a visual map in my head of how you put all this together potentially so effectively. What I think you're saying is when we have 10,000 apps that are one shot, apparently the first thing we have to do is make sure we've defined the core concepts of our organization. So concept of a user. Yeah, and that's important. We know that anything to do with a user is important to our business. So that concept needs to be defined and held. And then if your application is touching a user, touching that concept, then you need to be aware of that. So you're touching the user concept. And that's an important concept for us. And then we have the metric. So that are a form of statement around active user, but also state of user. So there's a state flow. So your application actually has to be able to do whatever it needs to do to tell us when there's a state change for that. So we can define active or inactive.

The interesting thing is usually I like to model the state changes so I don't want to have the applications to define it because we will play around with this. So we will have different definitions over time. We might even break it down. We might have 6 definition of an active user. This is a great stuff that you can do in a data model in a data warehouse. I usually tend not to have it already on the application layer. I just need to track all the things that are happening. This is why events are nice, because events I can use to derive a state change.

But to do that from a govern point of view, when you've got 10,000 apps is you're going to have to say to them in your app, when a user changes to this state, you need to push it out as an event. That I can say, yes, that has to be a governed thing. You have to actually do these events because these are the core events. If you don't do this, your app is not valuable. And then how do we know it's not valuable? What we can do is show them the metric tree and say if you're not telling us that you're changing the state of a user, then this metric won't move. And if this metric doesn't move, then these other metrics don't move. And they are our core North Star metrics internal metrics. Sorry, our eastar metrics. See, I got it wrong. Then as like our internal metrics are North Star.

No, they're not.

Not North Star metrics are about customer. So they're our internal metrics, but they're the important ones. So if you can't tell us you're improving that metric, then why the hell are you building that app and that it's a combination of all those patterns and defining them early so that all those AI slop applications actually fit into this governance or I definitely have to.

Make a case for AI slop applications. So I privately love to build them because now for the first time, I'm a former product person. I'm still a product person at heart. And my biggest problem was always I need to see things that I can make decisions. So you can create a wireframe, sure. But it's not really to have an app in front of you. So now you really have the possibility you can go down three ways. I would say, okay, I would do it like this. You can compare it and you can say, yeah, no, this makes totally more sense. So I think it's really great what's happening, but you're completely right, so it will create an interesting complexity for us.

Oh, we've seen this before and democratization is brilliant. The ability to put the tools in the hand of people that aren't professionals in their art is great. That is massively variable. We've seen it time and time again. But we've also seen the impact when we don't bring in the principles, policies and patterns that are useful. DBT allowed a lot of people to write transformation code, which is great. It removed the bottleneck of centralized data engineers who were never allowed to do it Fast, fast enough. But what we ended up with is 5000DBT and I'm using air quotes here, models. And we lost the definition of a data model, of a conceptual model of those kind of things because we didn't apply the policies, patterns and principles that were valuable. So I can just see with ability for democratization of building apps, which is great, we're going to have the same problem. And for me, this idea of events defining them, conceptual model and a metrics tree could be the things that we use to create those principles, policies and patterns.

That's true. I also never thought about this in this way. I think it totally makes sense. No, you're completely right. I'm a big believer in Democrats on one side, but on the other side have a really good foundational concept that will tell you where you define specific metrics, that will tell you if the things still work. So for dbt, my, my classic metric is how long does it take a new member of your team to understand the model and make a production commit. If you have 5,000 models, that's what will take eight.

Whereas my metric is what was the original time from a stakeholder saying they had a problem and being something that solved it with data. And you've moved to a team of 10 new analytics engineers and DBT has that time come down? Because if it hasn't, that's great. Busy work. Thank you for hiring more people and being really busy. And the other one that I often talk about when I run my course is the clock starts when the stakeholder says they have a problem, not when that problem hits the JIRA Q Prioritize for the data team because if that takes three months, actually the stakeholders already said it's three months late and it's not the data team's fault because they're not allowed to work into that team. But if we think about nodes and links and metric trees and system maps, now we just want to focus on the prioritization process. Process, because that's where it's broken. That's three months. And if the team take a day, it's still three months in a day. As far as the stakeholders, same kind of patterns as you. Effectively some form of storming, some form of jobs to be done, some form of domain driven bucketing, some form of journey mapping. We can apply that to the way teams work as well. It's the same set of patterns and they have value. Yeah, end of rent. But I can see that metrics tree and that event definitions of those core events has been really valuable in the democratization we're moving to. Excellent. All right, so at the beginning you said, oh, I'm not sure we can talk about the metric trees for an hour. And I said, we'll talk about lots of stuff, as we did. Hey, look, thank you for coming on the show. It's been awesome. I've learned lots and I hope everybody has a simply magical day.

And that Data Magicians was another Agile Data podcast episode. If you want to learn new patterns that you can apply to craft your own Agile Data way of working, head over to agiledata IO.