AgileData #36 - Agile and Product - Justin Bauer

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a simply magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

I'm Justin Bauer.

Hey Justin, thank you for coming on the show. Really looking forward to this one. Before we kick off, why don't we have a bit of a background about you and where you've been, what you've done in the world of agile and data.

Yeah, for sure.

Well, thank you for having me, Shane. Definitely appreciate it. So, yeah, currently I'm chief Product officer at Amplitude. We help companies build better products through understanding user behavior. We've got a product analytics solution, digital analytics platform. So I get to spend a lot of time working with a number of our customers, actually helping them be agile and ensure they're building products that customers love. And I've been doing this myself for over a decade. I've been Amplitude for seven and a half years prior to actually ran a mobile gaming company where Velocity actually was one of our competitive advantages differentiator for us. The fact that we're constantly learning with customers, building new capabilities, I think that was really critical. And prior to that I was actually at a digital analytics company where we're helping a number of Fortune 500 companies actually think about what their mobile strategy should be in this new way of building. And so I think something that, yeah, Agile velocity adaptiveness has been a key part of actually my entire product journey. I think something I've been really focused on helping myself and other companies accomplish.

I think when I first started out my journey, I was focused on combining the agile practices and patterns with the way we work with data. In the last couple of years, what I've seen is actually a lot of the things we need to adopt as well come out of the product world. So it's really a combination of product patterns, data patterns and agile patterns together where we get the really sweet spot in that. Everybody's got a Venn diagram. That's my new Venn diagram. How are you seeing that? Because obviously from a product point of view, your product's well known for powering product companies and powering them with data and analytics. So see what success looks like. How does that work? How do you see companies adopt that data and that metrics to figure out what to build next, what's working, what's not working, Getting rid of stuff, if they ever actually do get rid of features, because I've never convinced people to get rid of them.

That is the hope.

That is the hope. More is not always better. I think it's a really exciting time to be a product leader because we have the ability now to truly understand what users want. They're telling us through their interactions, through their data, if they love a feature, love a new functionality, love your product, you'll see it, they'll come back time and time again. And if they don't, if it's confusing, you'll see them struggling, you'll see them not coming back. That data is an important signal to you that you can use to improve that. And this really wasn't possible in the old days of building product. When you shipped software in a box and delivered to someone you actually didn't know, you could take surveys and you could ask them. But I'll tell you what people say is definitely different than what they do. What they do truly is the barometer of actual product adoption and success. And so I feel like the best companies realize there's a treasure trove of information there for them they can take advantage of to actually build and iterate and improve their product. But that also comes with a bunch of change in terms of how people do it. It's a new thing. It's no longer about this vision that you deliver from on high and say, go deliver. It's now this world where it's actually a lot of things aren't working. They're not necessarily the right, but you're iterating quickly, you're getting that feedback super fast in real time for many companies. And that allows you to get something a lot better out there. So to me, it's just a really exciting movement and I think it's the right way to build product because you're building because the data is telling you what's working and what's not.

So one of the challenges obviously is that the volume of data is now so much bigger. We've been through the big data Hadoop bullet points wave, but now we actually got to a stage where volume of data is a problem. Not from a storage and compute, because we've got cloud capacities now to just eat that stuff, but our ability to look at that data, to understand what's important, to get that signal out of that noise. And with your background from gaming companies, where again, there's a hell of a lot of user interaction data coming. And what you're seeing people do in the product metric space, how are people dealing with that? How are they dealing with. We're no longer just looking at custom orders product, we are looking at user interacted with this feature five times. Quickly or slowly or on a Monday, but not a Sunday. Or like their friends. Or not like their friends. Have you seen good patterns that people can use to really figure out what's important to look at and what they can ignore for.

For sure. The first thing is you're gonna drown in this sea of data if you try to measure everything and necessarily understand everything from the beginning. So you gotta be focused. What we recommend for people is figure out what your Northstar metric is, is that you're really trying to drive. People are probably familiar with the concept of a North Star. That's like the direction you're going, the guiding light. What's the metric that actually reflects that you're actually going in that direction. I found that when you actually ask leaders that question at an organization, what is your North Star metric? You don't always get the same answer back. And so you should actually start there and drive that level of alignment. What is the most important thing that you should be driving for your customers today? And how are you actually measuring that? Get that aligned? Because then what happens is now once you've driven alignment around that, then you can start to dig into the data to understand what are the drivers of that metric. So an example, like for amplitude. So for amplitude, our North Star metric is what we call a weekly learning user. So that is an individual who's using our product. They get to some level of insight where they save a chart, a dashboard, a report, and then they share that with multiple people and at least two other people consume it. So to us, that is like a reflection of they've gotten to an insight, they've delivered value. That is our North Star. And it took a lot of actual iteration conversation to align around that being the most important thing that we want to measure. But once we have that, we can then start to break that down into what are the drivers of a weekly learning user. Okay, we got to have people in the first two weeks actually activate, and then we had to come up with what's our aha moment going to be? So then we had a whole bunch of debate as to what that should be. Once you get someone to aha, what does it mean to actually get them to the point where they maybe become a power user actually starting to share. Okay, we create a definition of a weekly distributing user, which is a sharing user. And then so you have a whole bunch of these different drivers, and then each of those have a whole bunch of questions. Is that the right way to measure it? What are the drivers of those? And then we can finally, map our product, work against that. Okay, so you're working on building out the Slack integration. That's a key part. Your hypothesis is that will increase distribution and people sharing within the product. Let's actually see if that's the case. We can actually measure ourselves to see if that happens or doesn't happen. And what we've seen is in the instance there with Slack, it definitely did. We had a couple other integrations where it didn't.

That's great.

We just decided not to push forward on that. Those aren't things that we're going to pursue. But now we're aligned around metric. We're using metrics to actually guide us versus just falling prey to, oh my God. There's so many different metrics out there, so many different things we can measure.

I don't know where to get started.

Have a hypothesis, have an opinion, and then help. The data will help you actually see if that opinion is right or wrong.

How long do you reckon it would have taken you to go from the initial idea that you got to figure out what your North Star metric is and then to something where you've iterated enough time, gone through enough hypothesis that you've got to something you're comfortable actually as an indicator of value for your customers and value for your organization?

The other thing is it's constantly changing because what you're delivering to customers are always changing. If I think about an amplitude, our first North Star metric, which is probably was maybe three months for us to get to this. We were a much smaller organization at the time, was a weekly querying user, kind of aligned with our mental models. Oh, analyst, product. Someone queries. Makes sense. Okay, check against that.

Great.

Okay, that seems like valuable. And I think at the right. At that moment in time was probably the right thing. But then as our strategy evolved, as our user base evolved, initially it was all about power users. Yeah, they should be querying. But then we got to no, actually we're more about product teams and collaboration. So now we care more about sharing. That was then like, oh, maybe we should actually measure this differently. And I think it probably took us maybe another three to six months of iteration. We tried out a bunch of different ways of measuring it. We looked at how those different North Star metrics mapped to our customer retention because we actually wanted something that was both customer value but also business value. Did some regression analysis, like a bunch of different things to actually uncover that. Like I said, there was a bunch of iteration and we're constantly rethinking that. Now we're a multi product company, maybe actually should be something different for us we're exploring, should actually be at a team level versus a user level. And so I think that. But the point is, you're constantly having this conversation and making sure you're confident that what, you know, the key thing that you're measuring actually is pointing you in the right direction.

Yeah, it sounds a bit like Sprint planning in that the value of Sprint planning is the conversation by the team on what they're going to do. It's as valuable as actually locking in what they think they're going to do, this iteration and what they're going to deliver. It sounds like the conversations you were having around what your North Star metric could be, how you could break it down, how you could measure it, were as valuable as actually coming up with the metric itself.

Because it's all about alignment. Right. That's the thing where I came back to like, if it was just. It's not like a product leader alone makes that decision, it's about actually alignment with the company. And that's where I found that actually when we go into one of our customers and talk to them about how do you come up with the North Star metric? We actually find it's a strategy conversation, not a metrics conversation, because they're actually not aligned on a strategy. And then of course you're never going to be able to align around a metric if you're actually not aligned on a strategy.

The other podcast I do, the no nonsense one we had Jason Yip from Spotify on, and we were really keen to drill down around the what used to be called the Spotify model and what they've done. And he constantly came back to, it's not a conversation about team structure, it's a conversation about strategy. It always goes back to that. One of the mistakes I see people make with North Star metrics, though, is they think about it as an internal metric. And the key thing from my understanding of it, is it's about value to the customer. If I think about our startup, very similar problem, which we haven't solved to what you articulated in that we're a data platform, so we do that middle bit, but we don't do the last mile, we don't do the presentation of the data. Our internal metrics could be volume of data collected, our internal metric could be number of rules or, you know, if you use the term rules for the transformation code. A number of rules have been created and executed, but none of those actually have value for our customers. The customer Value is actually using that data and actually using it to make a business decision. And even though we don't do that last mile, we empower the last mile. So that's where we got to, is we put our customer head on, what's the value they get. Everything else is indicators, but they're not core measures. They don't actually determine the North Star value. So have you seen that? Have you seen that lots of people focus on vanity measures or easy to measure or internal measures because. Yeah, and they're a good start. Right? They're a good start if you've got nothing else. But you really have to get to that understanding of value to your customer.

Yeah, exactly. I think the way you frame it is good start. It's not that those things don't matter. And honestly they're probably going to be drivers or check metrics and we have both how we think about our broader metric framework. We have things that we understand are drivers that we're really investing on. Then we have some things that are check, which is we're not currently investing in this, but if that fell significantly, you'd probably want to understand why. So you should definitely be monitoring them. So many of those things are going to be part of that constellation of metrics that you'll have. But to your point, they're not going to be the North Star because the North Star has to be leading indicator of customer value and business value and it really needs to be both. It has to be very well aligned there. And a lot of times those vanity metrics actually aren't reflective of the customer value. And in those situations, it actually is worth spending the cycles to think about. Can you get to that? I remember for one of our products is an experimentation product that we have and we're really thinking like the whole point of this is not necessarily to run experiments. Like, yes, obviously running experiments matters, but really it's about making a decision. At the end of the day, whether you decide to roll it out or not, it's like you've made a decision. But we didn't have anything that really measured. We didn't know if customers were making a decision. That actually started to get us thinking about, well, why not? That is part of the workflow here. And so we actually started to extend our workflow into that and actually help people. See, you should make a decision here. Don't just let this sit forever. It's going to sit in your code base like now's the time to make decision and make a decision faster. And then we started Instrumenting that. And now we actually see how many times our customers are making decisions positively or negatively. And that's a metric. And so now we actually have that as our North Star. And so we actually built our product in a direction that actually reflected that value and then we can measure it. You maybe can't do that with every time, but I think it's an important conversation to have it aligns back to.

What your customers are looking for as well. They're probably saying if they can actually measure how many hypotheses they had, how many experiments they ran, how many decisions they made, you're helping them measure their value while measuring the value that you've offered. I'm old enough to been through a few waves or errors of data technologies and ways of working. There's been a lot of noise on LinkedIn. I rem in the old days we used to have old tools like Business Objects and Cognos, Oracle Discoverer and we had a semantic layer metrics layer. We had a piece of capability between the visualization tools and the data itself to create a measure once and share it many times. And we're starting to see that in the modern data Jenga stack. The idea of semantic layer metrics layer coming back in as a core capability, what are you seeing in that space? Are you seeing that capability of easy ways of defining metrics and sharing them as being where the market's going, or are you seeing something different?

No, I do. I def. I think it makes sense. I think the mental model makes sense for folks. And the reason for it is because one of the benefits of modern tools like enabletude and others is that it's increasing accessibility. You think back to the old days of cognizant, it was really only accessed by analysts, whereas now we have a world where rightfully leaders, PMs, marketers, like they actually have access to this data themselves, so they can actually make decisions fast. And that enables speed and velocity and all the great things that you want with democracy. But the democracy concept is an interesting one because then you got to put guardrails on it because what ends up happening, and I've definitely seen this myself, is you'll have five different definitions of a metric. And so everyone thinks when we're talking about number of video views, that we're talking about the same. But actually what does it mean to view a video, take a media example, is that they watched it for two.

Seconds, they watched it for five seconds.

For a whole minute. There could be lots of different definitions of that. And so you do need a line around those definitions so you're all speaking the same language. It also makes it a lot easier for people, people to be able to then build on top of that. That's the thing I'm actually like the alignment and then the building blocks. I don't want to just understand how many video views like that might be some vanity metric, but I want to know if people do five video views in their first seven days, are they more likely to retain. Okay, now I'm starting to understand drivers of my business. But then if I can just pull, okay, this is our definition of what a video view is. And I can start to build my analysis on top of that and not be so reliant on a data scientist or analysts do that. But I can just do that myself as a product leader now, I'm able to get to that hypothesis validation a lot faster. And so I think it's an important direction because of those concepts, not necessarily because of computational speed, which maybe was some of the reasons why those things existed in the early days. It's actually an alignment and collaboration are the reasons why it's coming back.

Yeah, I agree. I think we got the metrics layer in the old days because we effectively put a in memory olap cube behind it to give us performance because we couldn't handle it in the on prem databases we had. It'd be interesting to see if what we get now really is just through a true semantic semantic definition or whether we still have some caching technology that adds value. You raise an interesting point there, which we often forget about, is that like the North Star metric, the defining of those metrics starts off simple and then we answer the first question and then we'll get given the next question. So how many customers logged on? Okay, what feature did they use? And then okay, time between features or clustering of cohorts of users by feature. When teams are starting off, they tend to want to bore the ocean. They tend to want to go and get all the questions that may need to be answered and then try and build data that supports every question. And I'm a great fan of just starting small using Agile iterative approaches, which is, yeah, just go and say how many people logged on. It's a good metric, not your North Star, but it's a good thing to monitor a good thing to be able to answer that question and wait for the next question and then iterate it out. Still see people getting a big requirements catalog of everything they need to build and then still spending too much before they get any valuable feedback from their stakeholders.

Unfortunately, we do still see that. I also think that we can probably say quantitatively, that approach tends to fail more often than it succeeds when you start from this huge waterfall. These are all the things I want to be able to measure because you're not delivering value soon. So it's all the same Agile concepts that we think about with software development. It's honestly true for any project as well, which is git. A couple wins early. So start with a key business problem that you're trying to better understand and solve for, and then think about based on that, how would you measure success? And then lead with some hypotheses that you have on what the drivers of that are, and then use that to come up with a very basic tracking plan and then put that into practice. And what we found is that if you do that, then you can probably get up and running in a couple weeks and actually literally have real data validating your hypothesis. Whereas if you take the approach of let me figure out every single metric that anyone cares about and what's the perfect thing, and then we go to instrument every single one of our features, that can take years. And unfortunately, in the period of years, you've now gotten further behind your competitors who do understand these things. You're not delivering customer value. And so we very much preach, start smaller, deliver value, thinly sliced, deliver value much faster to your business and your customers by figuring out a couple key things. Prove that you can do it, and then think about how to scale it.

I think one of the challenges is actually on the technology side these days. In the old days, we'd have change data capture technology. We'd go and hook into the production database and we'd get every change of every record from every table that would come across into the data warehouse. And that enabled us not to worry. We worry about storage, we worry about changes to schema on the source system side. But we never had to worry about actually getting the data because we knew if the data was updated, it got sent to us. Now, in terms of product instrumentation, actually there is effort to capture those events. The app developer has to actually say, these are the events that are important. We want to capture them and we want to move them to the analytics platform so we can query them. So how do you see people dealing with that? How do you see them deciding what events are important right now and the fact that when they add a new event in the future, they don't automatically get the history over all time of how that feature or event was used. Has it actually changed the way people think about instrumenting the collection of that data or not?

I think it's encouraged people to think more about the business value first, because it isn't this world of just collect everything and then we'll deal with it later. Like the concept of the old data lake, I actually think it's a healthy direction because honestly, in a world where you collect everything, it ends up being a big mess and you honestly don't get a lot of value out of that. It's a number of transformations. Those things aren't production quality. I don't know if we actually got what the promise was there, versus in this world where you're much more explicit about what you want to track, you're much more likely to think ahead of time of what is the business reason for this and then you'll actually get that out of it. And so, yeah, there's a little bit of effort, I think. It's not really that much though in the grand scheme of things, if you actually are thinking ahead of time as to what is it that you really want to understand because and to me that should just be integrated into the software development lifecycle. Like everything that we're shipping is a hypothesis. It's a hypothesis that this is going to deliver a better experience to the customer. Most things actually don't. That's the sad truth of it. And so if you think of everything as a hypothesis, what we call them bets, everything is a bet, then how would you not have a measurement of understanding that just be embedded into how you build it? And so to us, it's like a simple, just next step of the process. It's okay, great, so why are we building this feature? Every engineer understands that while they're building it, we want to understand this, we think this is the business impact. Here's how we'll measure it. So once I get to the stage of that, I can put in the simple tracking code to see that. And then once it's released, that's the first thing we're looking at, right, to actually see that impact. And so when you integrate it into the development life cycle and it's expected part of the process, it actually isn't really that much extra layer.

I think one of the differences between say, enterprise applications is that the only time you ever see a field come off a screen in the enterprise application, erp, whatever, is when you do one of those horrible upgrades that take you six months and you lose half your features. Whereas in the product world, in theory, we're losing things that don't have value on a regular basis. So therefore events that we're getting just disappear because that feature has disappeared from the product. How often does that happen? Do you get any visibility on how often people actually remove features out of their products?

That's a good question. So we probably could. We actually have a monitoring feature. We let our customers know that we monitor an alert to them when data drops off. And so that is actually a really great idea for our next product report. We might actually see if we could pull a global stat on how often that's happening across our customers. I might chat with the team about that.

Yeah, because we. Yeah, we don't want to do it. Oh, somebody must be using it. Oh, it's got to happen. We made a bet. We put effort in.

That's why we call them Betsy. The mental model of that, of just recognizing that, yeah, there are hypotheses. Not everything's going to work, that's okay. Part of it, I think that organizations have to get comfortable with is that's okay. That's an expected part of it, that not everything is going to work. Because I think a lot of it comes back to the ego of the person who built it and they don't want to let go. But at the end of the day, it's not about serving our egos, it's about serving our customers and the customers. It's complicating the user experience and that is not serving your customers.

I suppose it relates also to the cost of the ability to add a feature. So if I can add a feature really quickly and easily and I don't have to go through a whole lot of governance forums and fight for that feature and get it prioritized at the top of the list, if I have the ability to add it fairly easily, then those bits are cheaper. I'm spending less money on that bit. Whereas if I'm invested and it's taken me six months to get it to the top of the roadmap and then finally that thing turns up, it's going to become more precious to me the amount of time.

And if it's fallacy, for sure, it's literally sun cost fallacy. Exactly.

So that cost of change, right, is important. But coming back to that, so then it must be important that when you're defining these features that you want to add to your product, that actually you also define measure of success from a data point of view is how you actually going to measure. And that's hard. We know that even corporate Metrics are hard and we have simple ones like profitability and revenue and margin and those ones. But it must be hard to actually for people to think about, okay, if I add this feature to the product, how do I measure success? How do I measure some data that's going to tell me keep it or kill it?

So it's hard, but it's important. I think that goes back to having that constellation and ensuring that everyone at every level within your organization understands what's the broader strategy and then how you'll measure it. And so these things kind of work in parallel. You can at amplitude. We actually do this, what's our overall product strategy, which obviously layers up to the company strategy and then within each product, what is their strategy? And then so overall Northstar metric drivers products have theirs and then also North Star metric drivers. And then it just continues to layer down from product to pillar to pod. That's our organizational structure. And then within a pod, if you're working on a bet, like I said, that bet is a hypothesis. So it has a clear, this is what the impact. And then anyone working on that can actually see, why am I working on this? There's a hypothesis that if I build this thing or change this behavior, it will ladder up to my pillar level strategy, which ladders up to product level strategy, ladders up all to the overall company strategies. You can see that and you can see all the cascading metrics that go with that. If you have that framework in place, then it actually becomes a lot easier because then you're like, okay, I know that I'm on this team whose job it is to try to get people to consume more. Okay, that's because I'm part of this group that's focused on how many people are consuming in the product. Great. I have a hypothesis that if I build this thing, it will get people to consume more. What's the first thing I want to measure? Do people use it? Because if they're not using it, then it doesn't matter. But then do people who use it end up consuming more? So now because I have this broader framework, I actually understand what my role and purpose is within it. It makes it a lot easier than figuring out what are the right things to be measuring there versus if you don't have that, they're not aligned and they don't actually understand how they fit.

Within the broader system, then the challenge.

Is making sure that the metrics for each of the pods is enough to find that they can see their success. But then you do know it goes up into the higher level metrics because there's always that cause and effect. It's everybody else is making bets. So who actually made the dial move that, that little bit to the right is always a hard one.

Yep, that's hard. And then the other hard part of it too is this is a living and breathing system. They're all hypotheses at every layer. The fact that consumption matters is a hypothesis. So you have a team trying to drive consumption, but it may be that actually consumption doesn't matter. So you as the leader needs to be testing that hypothesis. And actually you can use metrics to do that. I've definitely had examples where we've seen one team just do an amazing job driving growth within their area. Their goal was to drive maybe 100% growth and they more than doubled it, tripled it. Okay, that's awesome. They did an amazing job. Did tripling that actually lead to a higher level impact above that? And if it didn't, that's not on that team because they did their job. That's on you because. Oh, interesting. I've now demonstrated that maybe this isn't as important as I thought. Maybe I should shift the strategy here. And so metrics can help give you that feedback mechanism. Like I said, it's a whole learning system.

It doesn't have to be 100% quantitative. So we know that if we want to get a good estimate, what we do is we get a bunch of people to estimate it together. Because we know as a group, based on the conversations and the way we estimate a group of people, always going to come out with a better estimate than one person, even if that person's an expert. I'm assuming from a metric point of view it's the same thing. If every team has a set of metrics that are aligned and they're moving their metric into a better space, then by default the whole organization should be moving forward. Because we've got that alignment of every pod moving forward slowly or quickly. And so therefore, in theory, the organization's moving forward. It doesn't have to be 100% data driven and aligned. It'd be great if it could. And we get the whole cause and cause and effect and it's quantitative and statistically proven. But often that's hard.

It is. And I think sometimes people fall trapped to wanting to prove out the causal relationship of everything when it is in many places not possible because there just are certain things you can't capture and represent within the data. And so that alignment is really important. There's a classic example from Facebook of the early days, which is their goal was to get seven friends within the first 10 days and that was a big driver of growth for them. Now, Was it exactly seven friends in exactly 10 days? Could it have been eight friends and 11 days? That level of precision is not the most important thing, is the fact they were all aligned on that because there was a lot of confusion as to what we should really be focused in on at that day. And that gave them that alignment and that was good enough to be okay, great. We can actually see if we're driving this right. We can see how many friends someone has and how quickly they get to that. That alignment piece is the really critical one. And then later on you can measure to see if actually. Yeah. Do you want to shift it? You can, but the fact you get a whole org aligned around driving something is pretty powerful actually.

Just coming back to that, you know that team topology you talked about, organization topology of going down to pillars and pods. So is a pod self sufficient? Do they have all the skills they need to be to ship something?

That's right, yeah. So our pod involves, it's the triad, PM design, engineering. There might be probably like three to five engineers within it. It's the unit that allows us to actually deliver. And so that's our base level unit.

One of the things I'm doing is I'm doing a reforge program around product because it's a skill that I've dabbled with but never done in anger when I've been going through this. One of the things that's been really interesting for me is the team topology product because the way it's described in the. The course material is very much around a product manager and a product analyst. That the analysis piece is actually embedded into the development team. It's not at the end of the supply. Is that common? Is that common that actually you would typically have data savvy analysts in the team that's actually building up those features.

Yeah. So I think teams decide at what level they want to have that type of analyst support. And I think that actually is a reflection of how empowered and enabled the PMs are with data At a company like Amplitude, where we obviously use our own product analytics solution. As you can imagine, our PMs are highly enabled with data and so we don't have to have an analyst on every single pot. But it does make sense to have analyst support because of some of the things that you talked about. It's not about doing the analysis it's actually about the upfront. How are we going to measure these things? So the conversation around what a North Star metric is, an analyst plays a critical role in that because that's actually something they're really good at. They're really good at translating what the business strategy is to actual metric. They'll understand what can actually be measured within the product. Product analyst amplitude will help. Then put in practice the processes, the best practices we have to ensure that's happening. They're part of the governance count to ensure that we're actually managing and maintaining data quality over time. You want to serve those jobs and then I think you have to think about where you're at within your maturity as to then how much do you need an analyst to do that versus how much can you put on a product manager? I'd say we're fairly mature in this. And so that means we've built enough systems, processes and software so that we don't have to have so many analysts. But if you're pretty early in that journey, you might actually want to have more just to make sure it gets going. But hopefully you can actually systematize that stuff and use software so you don't have to put like a data scientist in every single team. That can be pretty expensive.

And is that part of the. Now you're at the level of maturity, you're typically now hiring in product managers or product leaders who have a data background as well, or do you have a literacy program? So when they come in, you're then upskilling them in some of those areas that they're not getting in their previous roles.

Because there's so many different types of product people out there. I actually think it's really important to think about what's the right product fit to your type of product. And so we're in a unique position that given that we're an analytics company, like by definition, I am going to hire people who are familiar with analytics because it's what we build. But at a gaming company, I'm going to hire PMs who are very familiar with analytics because it just drives so many of the decisions that may not be true at every company. There's a level of sufficiency that I think every product person needs to have. And that's true for every skill set. Design and customer research, analytics, business technology, strategy. Like all those things are different capabilities of a product manager. Some companies you need to be super spiky in one areas, but less in others. At amplitude. People need to understand analytics because it's what we sell. We then do still have sufficiency training, like how do you actually get embedded and use amplitude? Stuff like that still exists. And I think that's important because it is, it is a critical part of the skill set. And maybe that wasn't the case like 10 or 15 years ago, but it definitely is today. It's an important part of how people make decisions.

We talk about maturity of people's T skills, I talk about novice, practitioner, expert and coach. I think it's important to actually understand maturity of the organization both in terms of product and data and agile. Right. So in terms of they start off and things are raw, things are funky because you're all learning, but after a while you get to a level of maturity and then you need to change some things. Everything always needs to change and needs to adapt for that.

I actually find one of the ways you can actually change your own organization's maturity is actually hiring someone in who is a level higher of that. So if you feel like you need to improve on how you actually embed analytics into how you make decisions, go find one team that moves fast or you think there's an opportunity to do that and go hire a product manager who's really good at that and you'll actually find that they'll be successful and they'll start, other people will see it and you'll start to get a more bottoms up movement there. Actually I think it's a great way to actually improve maturity Institute like that.

We see that a lot with organizations. We see team and organization start to experiment with Agile and get really good at it and then it becomes a bit like a virus. Everybody else is going, well, they look like they're delivering, they look like they're having more fun. How can we do that too? Unfortunately, in some organizations that's when they then stop it and bring in a safe consultant and transform it. Whereas it's growing, just let it grow as a good virus. Right? Good mould, not bad mould. So coming back to that idea of self service, as we get more and more self service, more and more distributed work, we end up with that governance problem. And in the past we talk about centralized governance and it's been a nightmare. It's been a bunch of committees, a bunch of people who don't have time to do the work, getting a bunch of decisions with bugger all context and then making some decisions that everybody else is meant to follow. What do you see now in terms of there's a theory coming out of federated governance but very little patterns. It's just like words. So what do you see in terms of the idea of governing around self service? Have you seen anything useful out there?

I'm definitely a fan of that movement. I think we're seeing that within our customers for ourselves as well. I think it's about really lightweight governance and there are just some high level best practices like naming standards. Let's align around that and start small. I think the failure point as you described I've seen a lot of companies is they have these very strict governance. It's like we have this global schema and one needs to confine to that. And what will happen is that if something doesn't confine to it, they'll go through the process once of trying to get in front of the agenda of the schema council and then the council will be like yes or no and it'll take forever. And then everyone else will see that kind of to your point, this is the bad part. Everyone else will see how painful that was and they'll just be like F that I'm gonna go and do it myself. And so then you end up with a whole bunch of different standards and you've created even more of them. And so instead if you can keep it super lightweight like just start at the naming convention and we'll agree a simple thing like video played should have one way to name it. Start there really lightweight and see if you can drive alignment. Second piece is identify a couple of the key things that drive a business strategy forward that are a reason why you need alignment. Okay. So we're trying to better understand how certain products as part of a broader portfolio are driving network retention for us because we're now a multi product company and we need to see like this is a starter product for us. We want to see how people navigate between them. Okay. Everyone agrees that's a company level strategy. Use that to define. Okay, great. So we're going to have a golden set of tables that we really want that enable that to happen. Everyone needs to have understand, okay attach it to what matters from a business, help people understand the why so that in these certain areas yeah fit within the framework. I think the other thing that's really valuable is this comes with golden table and then there's just a place where you just experiment run and just know it's going to be messy and you just let that be, recognize that's the case and allow people the opportunity then to clean it up. But that's just their, that's their like playground that's cool. It's okay. Let them move fast, let them instrument however they want. And then if you find there's something in there that actually is really important, that's needed from a business perspective, you move that into then the concept of the golden table. And I think teams will get that. They're like, okay, yeah, I understand other teams are making decisions on this, and so we'll make sure we fit to the standard there. But you're still giving people the ability to instrument quickly measure the things that they care about. I think if you start like that, lightweight, only the most important things for the business you can start.

I think the idea of focus is really important. I remember in a previous life, Sean McGurr was. Was working with us and he did a PhD around procurement fraud. So he got a whole lot of open data from a fairly large country and his Ph.D. and again, I'm paraphrasing, sorry, Sean, if you're listening, and I've misunderstood what it was, but it's such a cool story. The government actually published all the procurement data. He bought what and when the procurement tender was put out, who won and that kind of stuff. And what he found when he looked at the data was in areas that were large tanks, ships, planes. There was very little fraudulent behavior because all the different government agencies were really focused on each other. They kept each other to account from memory, photocopiers and printers and small things like that.

It was right.

And so again, it comes back to the idea of governance, that if there are things that are important and you make them visible, then actually teams will hold themselves to account. Your answer around naming standards. If you actually say these naming standards are important and then you publish metrics that say the schema over here is not complying with the naming standard, then if it is truly important, then other teams will have that conversation because they're dependent on that naming standard. They've written code or interfaces that say, we understand how the naming standard works and we're going to rely on it as a contract versus going back to your parents. Your brothers and sisters held you to account. They tattled on you. Your parents always got way too busy with everybody and could never figure out what the real truth was. And I liken that to government. As teams, we can hold each other to account for the things that are important and things we've agreed as long as it doesn't slow us down too much.

It's the agreement that's the critical part.

Because everyone can understand.

Everyone's lived through the pain and so collectively you've made a contract with each other to follow through on that. And that's where it starts. Small get those right, maybe that will get to the top 10 things. But if you start from 10 things that people don't understand why that matters, they're not going to follow it because it's going to slow them down and therefore they're going to find the shortest path to actually delivering what they need to.

And again, it comes back to that cost of change. Sometimes if it's too hard to change in the beginning, then you won't change. And then that's how you fail because you're not able to make those bets. You can't make 100 small bets. You're going to make one big bet and then you double down on that bet and it becomes an expensive and dangerous bet because you can't fail anymore because all your eggs are in that basket. So coming back to this idea of product versus agile, and I'm using the old finger quotes there, when you talk to product teams, you see a lot of agile behav, see them inspecting their work, adapting the way they work, making experiments or bets, getting value out to their customers as early as possible. You see all the things that we can use to describe agile iterations, where there's time boxing of stuff. What you don't tend to see though is all the agile words you don't tend to hear. Scrum and Scrum Coach hear all the things that we hear from an enterprise point of view. Is that what you found is that the behavior is there but the terminology and the words aren't? Or am I just in the wrong place?

No, I, I think you're right. So I, I'm based in San Francisco, so definitely part of the Silicon Valley group. And we definitely do not use those words. I think in many ways because any concept, people take it to an extreme and it becomes a trained way. And there's only one approach. And so I think we've gotten away from, from that a bit because if you use the word Scrum master, people have this strict assumption of what that actually means versus the principles behind it. And I think one of the keys to being successful is be principle oriented, but be flexible on what the process is because the process might change and so many ways for many companies. I think in the Valley we believe in the principles, but actually how we, how that might get executed will look differently because there are different people, we're building different products, we're building for different customers or in different industries. I think that has been one of the challenges with Agile. And I think this is true of any big movement. It becomes like, this is the only way to do it. This is the one way.

And it's like, no, there's actually lots.

Of ways to do it. And so it's much more around the principles and then figuring out what that means for your organization.

You're right. We get dogma, we get a gospel that we have to follow now. And don't go outside the boundaries, even if it looks like going outside them adds value to you. And we get terminology bleeding. The one that I still struggle with is the definition or the difference between a scrum product owner and product manager from a product company because they look close enough to the being the same, but they're not. They do fundamentally work in a different way. Have you seen blurring of terminology?

We definitely have.

I think in many ways it's moved back to there's a product manager in terms of we've got product management, we have product operations, which is interesting. I think that's a new kind of new field that's growing and there's a key partnership between those folks.

And so that has been a blurring.

Of traditional roles that were owned by product owner versus product manager. And kind of in traditional sense like what that means. And then the other thing that what I have found is that it can differ depending on company. The most important thing is there's clarity within the organization as to what the role and responsibilities are. But they can actually shift. In some organizations you might have an outbound pm, which is actually much more like a product marketer and an inbound pm. And you might have. In other organizations we might have a PM technical PM, some you might have PM product operations. Maybe what we'll find 10 years from now or 15 years from now is actually we do align around all of these definitions and come to a standard. But at this stage at least, the most important thing is that you're very clear on what the roles do within your organization. So you're hiring people that match that.

I think in 10 or 15 years we'll have a new buzzword. Hopefully it's one that has value and the good patterns and it's not just a buzzword. But speaking of buzzword, so product ops is really interesting to me. We ops everything nowadays. SecOps, FinOps, RevOps. I think we'll get restaurant ops at some stage. But product ops is quite interesting for me because I can't tell if it's more of a DevOps thing around automation of Platforms and processes, or if it's more a way of working around how you determine what features the highest value and what you should develop next. So from your point of view, how do you differentiate between product management and product and your organization?

Our mission for our product operations team is to empower all teams to execute across the product life cycle with clarity through processes, tools, insights and communication. And so what that means is that one product operations and service of the broader organizations, not just in service and product management, that's really important, it's in service of the broader organization. It cuts across the entire product life cycle. So everything from the research phases to post launch and focused on improving velocity and alignment. And then it's how is it is the owner of what are the processes, what are the tools, what are the ways that we capture insights and how we communicate between these different parts of the organization? Probably the best analogy to this is if you think of the product team as building the ship. ProductOps builds the shipyard.

Okay. So again, I'm a great fan of patents. I try and determine which pattern something falls into. And so there's two patterns I've just heard within there. One is this idea of a coaching circle. So this idea that we have agile coaches who look at delivery teams and help them iterate their way of working systems that go where's the blockers? Where's the things we should change, what you should change and how you're going to do it? So product ops, I heard some of that standing outside of the workflow, outside of that way of working, observing and then helping everybody go, okay, this is not working. What are we going to do to iterate that way of working? And then I did actually still hear some platforming ops, DevOps stuff in terms of some automation and some removing the drossi work from those teams because they should be focused on the valuable stuff and we should automate what we can with the machine. So it's a combination of those two from your point of view, to get that value. Okay, that makes sense. Excellent. We've covered a lot in just under an hour, so that's been great. I love the way we've just bounced around in so many different areas with a core set of patents coming out that seem to be common across both the product world and the non product world, shall we call it? So if people wanted to get hold of you and hear more about what you're doing and find out more about your space, where's the best way for them to get hold of you?

Yeah, for sure. Twitter, Justinj Bauer as my Twitter handle. I'm on LinkedIn quite a bit too, so people can really just find me Justin Bauer and follow me on LinkedIn.

And then my company as well.

So amplitude we are obviously we write a lot about these types of topics as well, so that's a really great resource for people to follow too.

That's one of the things I love about the new modern world of product and data, is that large companies are much more open to sharing now. Their teams are publishing great content with stuff that is actually usable and making it freely available these days. So I think that's one of the benefits we've got out of this kind of modern way of working.

We believe it's important because in many ways our mission is to help companies build better products through data. That's a transformational change. We understand that. So it's not just about us building software. We're trying to help people understand what are the processes that people take to do that, get those stories out there from companies who've successfully done that. So we're big believers in pushing that thought leadership as an organization, as always.

Say sharing is caring as well. It's good to have that information out there for people who need it. Excellent. Hey, look, thank you for your time. It's been awesome and we'll catch everybody later.

Great, thank you.

Data Magicians was another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata IO.