Why High Leverage Analytics is a Cross-Functional Team Sport

Sam.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to check join. In the meanwhile, though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments, depending on what platform you're watching from. And yeah, tell us something that you'd like to get out of today's session. A few bits of housekeeping before we kick off. The session is being recorded and the recording will be emailed to everyone that's registered for the event. If you haven't registered already, then you can find the link that I've put in the chat, you can scan the QR code that's on screen now, and you can also head to datacamp.comwebinars as well. Well, we are going to be having a Q and A session, an audience Q and A at the end of the session. So if you have any questions at any point throughout the session, then please let us know and I will save them for Q and A. So yeah, just let me know in the comments what your question is and I'll save it for the Q and A. Aside from that, keep your eyes peeled in the chat. We're going to be sharing resources that are relevant to what you're seeing in the session. And yeah, brilliant. I think that's everything from me for the moment. So I'll be back to repeat this one more time for any new joiners shortly. But until then, enjoy the background music.

It. Foreign.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. Going to kick off today's session in about 20 seconds or so. We've just been waiting so everyone has had a chance to join. Let us know where you're joining from using the chat or the comments. And yeah, tell us something that you'd like to get out of the session today. A few bits of housekeeping for anyone that's just joined. The session is being recorded and the recording will be emailed to everyone that's registered for the event. If you haven't registered already, then scan the QR code on screen, find the link that I put in the chat, or you can head over to datacamp.comwebinars. we are going to be having an audience Q and A at the end of the day session today. So if you have any questions at any point, please let us know in the comments and I will save them for the Q and A for you. And also keep your eyes peeled in the chat for any resources that we send through. Today we're going to be sending through a few key pieces of information that relate to what we're going to talk about today, but I think that's everything from me. So now I'll hand you over to your host for today's session, Richie. Richie, please take it away.

Welcome to the Webinar. This is Richie. Now, when data gets shown in movies, it's always some lone brainiac with glasses solving problems. I mean, you've got to wear glasses to make yourself look smart. But in reality, pretty much every analytics project is going to involve multiple people and often multiple teams. That means that half the trick to analytics success is being able to manage these different stakeholders and participants and get everyone communicating with each other. So if you want to improve the data maturity across your organization, have that more consistent success, you need to stop being scrappy, improve your processes, improve your tooling, things like that. So today we're going to figure out all the devils in the details to make that work. And today's guest is Steph Olafsdottir. So she's the co founder and CEO at avo, which is a platform for data quality in product analytics. She's a mathematician and philosopher turned genetics researcher turned data scientist turned founder. And Steph has a decade of experience managing data and product teams, helping them improve their processes to ship results faster. Steph was previously the founding data person and became head of data science and growth lead at Quizup. And that is to say that Steph knows an awful lot about how to make analytics projects successful. So without further ado, over to you, Steph.

Hello Richie and the entire group of people that's here. Great to be here with you all and thank you for this lovely introduction. I guess a quick intro and then just directly into the slides or what do you say, Reese? Right. Great, we already have enough here. Thank you for introducing the talk so well, Richie, and thank you so much for inviting me to be here with you. I am honored to be here at Data Camp to talk about this, which is a heartfelt thing for me. I am going to start with a brief intro of myself and Avvo and then dive right into it. As Richie said, I am Steph. I am the CEO and co founder of avo. AVO fixes data quality for product analytics. We exist to inspire better data cultures. So the teams can build better user experiences. But what does it mean that in practice that we fix data quality for product analytics. So we specifically help companies like Fender, Walt, IKEA and loads of incredible teams from startups to scale ups make sure they have high quality data about user experiences so they don't fly blind when they're trying to build great user experiences. Specifically we facilitate cross functional collaboration at the source of where the data is created. That's how we differ from other data quality tools which might help you react to bad data, which also helps you react to bad data, but specifically helps facilitate a culture of making analytics a team sport. My talk today is going to be a little bit about the journey from winging it when you have no data really through the wild west, when you have a lot of messy data, through some sort of a governance which is typically sort of centralized governance, and into utopian state where you have like a self serve governance where every team can really run for themselves and creating great data that helps them make good decisions about building great user experiences while also being able to leverage the data from the other teams in the company. So that's sort of the talk of the day. And I will be referencing the AVO toolsuite a little bit, but I want you to know that it's based and inspired on for example what we built at Quizup. And so ultimately it's about finding the right tool suite that helps you on that journey. And I am sort of a big believer that of course tools cannot change cultures, but tools can facilitate culture change. Just like e.g. gitHub can support peer review culture. With that. Let's dive into it. I'm going to intro this by telling you a little bit about my backstory. Like Richie gave a sneak preview of. I used to run data and was the founding analyst of a mobile game called Co is Up and we had a lot of of sort of data quality issues. I joined and I was tasked with understanding user experience and our opportunities for improving it. So both from a go to market perspective, but also from product strategy perspective. And that means, for example, I had to understand why some people would sort of download the app and then maybe start creating an account, maybe get through creating an account, maybe start playing one game, maybe finishing that game, but then just never play again. But there were other groups of people that went through that entire journey and then finished playing seven games in that same first session and then kept coming every day for months and they even ended up finding their life partners there, getting to know people and getting married to people that they met through Quizup. And that was of course, like, very rewarding to work on something that, that help people actually get through a journey like that. So it was a. It was sort of. It's a, it's a game, but it was, but it was really. It was actually changing people's life. People were meeting their life partner. They were literally. Our CEO got invites to more than one and more than two weddings that occurred because people met through Quizza. And so, you know, that's. That's the thing that makes us, people that work in software tick. We want to build great user experiences for people ultimately. And then we are building all of these tools and processes to facilitate that. And analytics and data is a part of that tool suite, building and understanding user experiences so that we can build them and make them better. And I was tasked with that. But endlessly we struggled with just looking at data, thinking that the reality of some users or sort of a set of users or a user journey was a certain way, then shipping an update to the product because we thought something was true, and then realizing later that it actually wasn't true, the data was wrong. This happened to us. We ended up building tools internally to fix this problem. Tools and processes. Then fast forward. A couple of years later, I and a couple of friends from Quiz Apps started another company and had the exact same problem again. At that point, we thought, thought, well, now there must be a solution somewhere in the market that we can buy for this problem. We're not going to build it again, right? But turned out after we talked to hundreds of product managers and developers and data scientists, that the solution wasn't there yet. And most people were just completely miserable about this. It was impacting their success at work and the amount of time that they had to spend on fixing data reactively or even just not be able to make decisions and then have the CEO breathing down their neck on like, but how is it going? And it's incredibly frustrating. And then there was a small percentage of people that had converted to building something internally. So it's a huge problem. And I want to bring you to the story of what happened actually at that second company. So we were shipping a. I actually have a slide here. I forgot this slide of the overview of the analytics events that I came into at Quizup when I joined Quizzic. This is how beautiful the data looked. And this is just a small percentage of the data. This is the way to understand when people start a game, you join and you see all of these different versions of the same user action. Most people that are looking at the data, they don't even know. They don't even know how many variations of the analytics events there are. And then for each version of this analytics event, there's also permutations of the metadata attached to it, something like game ID or type of game or something like that. That sort of explains the situation in the app or the situation for the user when the analytics event occurred. And so that's sort of how messy that data looked like. And you can imagine, I think everyone here is sort of on their journey to sort of build a data career. And I think you can all relate to how frustrating it is when you have to do these huge SQL Join queries to be able to look at this data holistically. And so what happened then at that sort of second company that we built, we noticed that we had the account created analytics event and we had a property on that that explained what type of method people used to sign up. And only 2% of our users were actually using the phone number method. The rest were using the email method. And we decided to use that opportunity to simplify the onboarding journey and sort of reduce steps that people would have to take and actually just ship an updated version where you didn't have to choose and you just went straight to signing up via email. And what happened was that our signup conversion funnel actually plummeted it down from 81% of people that downloaded the app to actually creating an account to 46% after we removed the phone number sign up method. And it took us, of course, like a while to figure out that this was actually happening. Because, you know, like at the beginning you just see the data trickling in and you're like, well, this is weird, isn't it? Well, it's probably just some weird thing. So you collect data for a few days and then you see, well, it's really bad, it's looking really bad. And then you realize this. And then I started taking a look at the data again and it turned out that this is not the actual distribution of the data. We had one more field or field value for the signup method, which was the beautiful phone murmur. So this is all to show and tell that a tiny little seemingly minor typo where you miss a B from phone number can actually make a huge impact on just product development and the success of a company. And this problem is happening all over the world for small and big organizations. We were a small team. It took us a few weeks to implement the new sort of onboarding experience. Then it took us maybe a week to realize that this problem was happening and the drop and then maybe a week or so to actually make the change. So we made an incorrect product decision. Weeks of work of implementing originally the work of reversing the change and in the meantime a severe hit to our signup conversion. So it's a terrible thing and it's just, it seems so small, but it's happening all over. And this is the kind of thing that sort of is a. It's a problem that's very hidden because it's so down in the weeds and so nitty gritty, you know, um, but the scale of how this is impacting them bigger companies is like weeks turn into months and severe hit to sign up conversion ha turns into severe hit to, you know, hundreds of millions of revenue potentially. And so it's actually a huge problem boiled down to this tiny typo. And then, you know, this was just one example of a property field and then like that's happening for everything that's happening in the product. So. But best case scenario version of this is that you realize this data bug before you make a decision, but it took you a while to figure out and dig into the data. So ideally what we want to do is prevent these issues from happening. But sort of before we go into how we prevent it, I just want to paint a picture of why something like this happens. Why do we get releases that have these data bugs that have these phone numbers spelled in more than one way and different versions of analytics events, all of the different versions of the game started event. This is what the typical analytics workflow looks like for a company that has at least maybe a few developers, if not multiple teams working on a product. And then you basically are making product changes and you use your, like maybe your Figma and a Google Doc to design some product specs. And this is happening by a product manager. Then an analyst gets pulled in to collaborate with the product manager on defining what are the metrics that we want to measure the success of this release with and what analytics event structures do. We need to be able to actually visualize those metrics and sort of compute them in our SQL or in our mix panel or whatever tool we use to analyze our data. And that work happens in a different tool. It happens in a spreadsheet or just like a Slack ping or an airtable or even in certain maturity. It happens in a structured JSON schema that's hosted on a GitHub file or something like that. Then the next step for that is Getting feedback. There's a little bit of back and forth potentially between the product manager and the analyst and then also pulling in the developers to confirm that we can actually make implementation structures in the way that we've designed here. Potentially at that stage, the developers come back and are like, actually it doesn't make sense for us to track this this way in the app at that point because we don't have access to the metadata that we need for that. And so there's a little bit of a back and forth in the data design happening right here. And that feedback and the implementation is often happening in a different tool. Imagine you design something here in a spreadsheet and then typically you won't communicate that on a spreadsheet. You could, but commenting on cells on spreadsheet, it isn't really a nice experience. So this conversation typically gets moved to an Asana or a Slack or something like that. And some decisions get made here that render often this spreadsheet already outdated even before the developer starts implementing the analytics events in the product. So we have a source of truth for the way that the metrics and the event structures should be defined that's already outdated even before the developer starts implementing it. That's crazy. Anyway, and then we move into the step of the implementation there what we have is we have maybe a complex and pretty diverse code base that covers multiple paths, code paths towards a user action that to us the user maybe appears to be the same action, like the same experience. So maybe for example, if we think about Spotify, we think about the ability to add a song to a playlist at Spotify. To us that just feels like, like an experience that's available wherever we are in the app. However, behind the scenes to actually make that work, there are like these, there, there are like probably 90 or I don't know how many code paths allow you to, to do that. Because you, you can do it from like the song full screen view, you can do it from the cube view, you can do it from the home view, you can do it from the album view. Like there are so many ways or paths from where you can add a song to a playlist and then you can do it by swiping and you can do it by tapping the three buttons. And this is just the mobile app. And then you have like Android and you have iOS and you have web and you have desktop and you have Apple TV and you like, you have all of these code bases that have this experience to add a song to a playlist and all of These code bases will end up tracking this action with these different names. That's how that happens. This happens so easily because this, this immense amount of code base is maintained by a lot of different developers, obviously. So that's sort of like how this problem happens so easily. It's just like to us, it feels like this simple user experience and it should be easy to just understand, you know, and sort of just track the experience to do it. But it's happening behind the scenes in so many different code paths and maintained by so many different people. And that's how these bugs come about. Then typically what happens is someone goes ahead, maybe the analyst, maybe there's a QA person, maybe also the developer, maybe all three go and sort of confirm that the analytics events actually look the way that we expected them to do. And they typically just use their human eyes to do that. So they go and open the app and they trigger the analytics events a few times and then they go into the tool that they expect their data in and confirm that the data actually looks the way that they expect it. And we all know that human eyes are very error prone. So you can easily imagine why you won't catch all things this way, but you will catch some things. And what happens then is you have to go back and you have to request, you have to sort of beg the developer like, oh my God, please fix this before the release goes out. Otherwise we'll have a data bug live in our production data. And it's difficult to fix that after the fact. And so it's like. And it's frustrating for the entire group of people that I've mentioned, the product manager, the analyst and the developer, because the analyst has to do a lot of work to sort of try to catch the bugs before they go out. And typically they won't catch all of it. The product manager has to sort of. The developer has to go back and fix code that they already sort of let go and started maybe working on something else. And they might have to do that a couple of times or a few times before they're actually able to release their product. And the product manager thinks also this, this is frustrating because they have to wait while this entire thing happens and they just want to get the product update out. They don't want to have to think about all of this stuff, but all of them go through this painfully because they know that analytics is one of the most important things that we have to build better user experience. Yeah. So it's a fundamental tool. And so anyway, so then eventually we have some data in our analytics tools and we go ahead and try to sort of analyze whatever what the user experience looked like and then from there we make decisions about what we're going to change next in the product or in our go to market strategy. And like I said, typically we still have some releases of bad data going through here because this is still very error prone. So this is just a paint picture of the problem that we as data professionals have an opportunity to talk, tackle, let's put it that way, let's put it that way in a glass half full perspective opportunity to tackle. So I just want to quickly sort of talk about sort of the evolution of this journey. I mentioned this in the beginning when we had this sort of title slide. The journey that each organization typically goes through in this is we see of course like there's no data, you know, you're just building the product for the first time. So you're not basing it on data about the user experience. So far it's based on your user research and your intuition. So there's no data. And then you go into the bad data where you have the Wild west situation going on. You are, there's a, there are a few teams working on the product. They are all working on their own analytics events for every single release that they make. But they can't really use the data from the other teams because you know, they don't know what exists. They don't know the intricacies and the details of it. They don't know that there are seven different variations of the events there. They don't know that they maybe are creating a duplicate version of the event that they, that another team also has in place. And then, and this is of course slowing a lot of people down. And this is the situation also where you have like all of these, you know, versions of the analytics events in place. And then from there you go into slow data because you try to adopt some sort of a process like this. You try to be like, hey, can we, can we add more structure to this? So when I talked about this earlier, I told you about like, maybe it's just a slack ping. Maybe it's just a slack ping. Maybe it's just a slack ping. That's the Wild west situation. And then companies mature and then they start pulling in tools like a JSON file and like an Asana ticket for doing this. And so you try to build a structure for it. And so basically you're going into the, so you're going from the Wild west and into some Sort of a centralized governance. And so that's why I call that slow data. You know, you get a little bit of better data, but every product release is a little bit slower. And maybe you might even need an analyst to actually pull the data together before anyone is able to use it because it's just so chaotic still. And the data team is sort of, sort of tasked with making sure people aren't using wrong data to make terrible decisions like we did with the phone number story. And so that's sort of the centralized governance state that people graduate to because this was sort of slowing the company down in a different way, helping people making bad decisions, as we're making sure we're not making bad decisions, but instead we have to move a little bit slower. And then the ultimate state, and I want to talk about what that looks like and how we get to the self serve governance state is this self serve governance. There you have good data and you have that fast. And this is the journey that we went through at Quizup and this is a journey that I've seen tons of companies go through and, and I would even say that we see the industry going through a similar thing in general. So the social governance, what actually has to happen through that? And I just want to sort of call out these things for you to think about if you are taking your company through these stages. So first of all, you should treat your data as a product. I remember this was a revolutionary thing for me to think about as a data professional and reading books like Marty Kagan's Inspired, where you think about your stakeholders and your product and you think about the usability of your product, the desirability, what does the market look like, who are the users and who wants what you are providing? The feasibility, the engineering feasibility, is it, you know, can we actually provide this data? How might we. How, how much of an effort is it? What is the, what is the best way to do it in a business viable way? So these are the pillars of, you know, building good products, usability, desirability, feasible engineering feasibility and business viability. And thinking about the data that you're creating in this way, I think is like a really important journey towards better quality data and sort of a better empowered analytics culture. And then I want to call out like you want to include your stakeholders. When you are thinking about that, you won't be able to figure out what the usability and the desirability of your product is or the feasibility if you don't include your product managers or the data professionals will be using the data or the product managers will be digging around in dashboards and the engineers who will actually be creating the analytics of that at that source. So we really want to include those peoples in the process of when you're designing the data and when you're creating the data from scratch. And then think about this balancing local and global requirements. So you want teams to be able to run forward fast with what they need. Local meaning that like domain specific, like this is the team that handles the playlists. So let that team create a bunch of events that are related to the playlists experience. But then also let's commit to some sort of global standards that allows us to prevent. We all create a lot of duplicates and allows us to really easily understand what an analytics event specifically mean. So for example, if I go back to this slide here, we have both multiple naming conventions going on, but also casing conventions. So for example, if we have Song added with a camel case, no Snake case here, and then we also have Song added as a camel case event, that means that these will get created as two different entries in tools like Mixpanel and Amplitude, but they will also get created as two different tables. If you create an event per table in like an SQL database and obviously like if you, if you, if you swap the words around, of course that will also create the same problem. But this is why casing also really matters. And so committing to something like casing can really help prevent the creation of duplicate entries and event types like this. And then committing to naming conventions such as Object action is one that I personally like a lot of. And this one would be an example of Action Object, like how do you act on a specific object? Why I like Object Action and I prefer actually Title case or Snake case. Those are my favorites because it makes it really readable in a dashboarding tool or things like that. I like Object Action because that will sort of often lead to the situation that you have all of the events that are related to the song object bulk together. Another example of a thing that you might want to consider a global governance for is just what is the word we use for a specific object in our product? For example, at Quiz App we decided we would always use Game and never use Match. That's an example and that really also creates an environment that helps prevent duplicate versions of analytics events. Just these things really help you fulfill balancing global and local requirements with there's some governance that needs to be in place to help people remember that these standards are in place. But of course then you can also then Automate some of that. And then this is my most important sort of thing I think that I can recommend in sort of creating a holistic culture of people who care about this, and I'll talk more about this later in the talk, is making analytics a part of the product release cycle. So don't make it an afterthought. Don't create a culture where analytics is the last thing you do before you ship. Like someone's just about to press the release button and then an analyst or someone jumps in and like did you add tracking for it? And then you sort of try to hack the tracking in afterwards and everyone's like rushed and everyone's frustrated that it's not going out yet. Just commit to designing the data and doing like a data kickoff with the product, Sprint kickoff or like the feature kickoff. And I'll talk about an example of that later in the talk and why I think that's really important. This is one of the things that I think is like a key habit in helping with all of the other things. Sort of like if you will, a flywheel, you know, do it, start doing that with one team and then that'll sort of change and spiral and change the rest of the culture. So I want to put this also then in context with data mesh principles, which is sort of like a set of principles that have been in sort of the zeitgeist, I guess in the last couple of years. And of course like none of those principles is really new, but fantastic. Data professional Zamag sort of put it in context as like these data mesh principles. Here are the things that you should keep in mind if you want to build a good data culture where you have, well, I guess just success in empowering teams to leverage data at scale. And so these are sort of things to keep in mind that you can aspire towards when you want to build high leverage analytics cultures. These are the principles you want to build towards domain oriented, decentralized data ownership and architecture. This is what I meant by the, you know, the playlist team can design their own data, you know, and then the, you know, the, the onboarding team can design their own data as well. Then you want to treat your data as a product. I already covered this a little bit. You know, think about the usability, the feasibility, the desirability, the viability and do research based on their product for that. And then build towards a shared self serve data infrastructure with the ability for people to actually pull in information or pull information that they need to make decisions in their day to Day. I love this subject. I am a huge proponent of self serve data culture, but it's a hot topic and I think that's not because people don't want self serve data culture, it's because they've not seen it successfully executed and I've seen it successfully executed. But of course there are layers to what you want to be able to offer people. I'm not saying that anyone should be able to do anything with any data, but I think that you should always build towards people being able to use the data that they need for their day to day decisions without for example going through a centralized BI team or something like that, or like waiting for an analyst and then they will pull the data and then you can do some analysis analysis on top of that. So build towards that aspiration and then some federated computational governance. And so you'll see how all of this rhymes a lot with what I just talked about in here. What does it take to reach self serve governance? You did a product, include your stakeholders, build global and local requirements and then making analytics a part of the product release cycle is sort of like my recommendations for how to get started really. So let's put these data mesh principles in context also with the maturity journey as I see it. So ultimately what we're trying to do is we're trying to create high leverage analytics cultures, right? And here's the argument also for why or why it is a cross functional team sport. So you are in Winginit, you start there, then you don't have any data. So obviously you're not fulfilling anything here. Then you go into the wild west and then you have the domain oriented ownership probably like every team for themselves really is running forward. They don't necessarily even have standards for their themselves though. So it's not guaranteed that they're happy with their own data. Each team and they probably are not able to use other people's data. That's pretty likely. And so I've checked the box here of self serve data platform because they're able to use their own data. But probably this is like a stretch to put it here. It's like are they though at least they probably, like I said, won't be able to use other people's data. You're probably not treating data as a product because you're not thinking about it holistically as who are my stakeholders who might be using this data going forward? What is the business viability of actually getting this data? What's the engineering feasibility of getting the data in the way that I need it. You're probably not treating your data that way at that point in this point in time and certainly not as cross functional team sport. And you most likely don't have any federated governance. And then you move away from this because it's starting to slow you down, like I said, because you have bad data prompting people to make bad decisions. So you introduce the federated governance and with that you sort of slow down the feature releases a little bit often and you remove a little bit of the domain oriented ownership at least for a while until you get to the next stage of the self serve governance. So you've started to think a little bit more of your data as a product, right? You've started to include the stakeholders, make sure people are conversing like the analysts are, the data professionals are trying to get the needs from the rest of the organization and that's why they're sort of adding some federated governance and making sure that the CEO dashboard doesn't break and all that stuff. And you're trying to build towards a self serve governance platform but at the expense of data ownership, domain specific data ownership so they don't have as much control over what they do anymore. And then ultimately I really believe that you can build towards this if you take your federated governance to a state that people can follow on protocol and if you introduce good tools that sort of prompt people to actually follow those guidelines, then you can help people actually own their own data as long as they fulfill some federated ownership. So with that I want to talk a little bit about sort of this making analytics a cross functional team sport. The reason why I believe this is one of the sort of the flywheels of building good culture is that. So I've done this in a few organizations that did this at Quizza. I have been a consultant a few companies before starting avo. And then we've done this of course, like tons of AVO customers have started using this. This is a process more than a tool, right? So how we recommend this, how we recommend you get started in making sure you have better quality data, high leverage analytics is kickstart every single feature release, cross functionally with a data expert, with a product expert and with the engineers that actually will implement things in code, align on the goals of the release. What does the success look like for you? It looks like if we introduce self serve trials then we think people will get better opportunities to experience the aha moments of the product and thus they will be faster in converting to sort of understanding and getting the value from the product that they need. And business wise they will also convert to becoming paying customers. That's our goal. That's what success looks like for us. So we will measure that as the conversion to becoming a paid customer segmented by whether they actually prompted the trial and what they did in the trial as well. And so we will design the data by okay, we already have a subscription event, we already have a trial, a trial started event because internally we were able to create trials manually before. And we already have a, yeah, we already have some events, but we need on the trial started event, we need a property that explains that the, the user prompted the trial themselves. And so that's how we're measuring now that the person started the trial on their own. And then we'll introduce a new event on what are the sort of paywalls that they click that actually prompts them to going into trial so that we can understand what are the things that actually help people get excited about the product and the aha moments. And so this is an example, like a super quickly run through example of, of a what what can happen in this meeting, how you will design your goals, your metrics and your data. But doing this type of a sit down will result in a few key changes in your organization. Number one, you will design better data if you do top down like this. If you start by designing the analytics event structures and think about it as like I want to cover everything in my product and make sure I'm tracking everything, then you won't design as good data because it's not designed to answer the questions that you need. So hopefully you'll be able to answer your questions if you do it just randomly. But likely you won't necessarily be able to. So it'll help you design better data. Number two, it will empower the people that aren't necessarily data native to actually be able to use that data data. So you're bringing together the, the product manager that understands like what is the goal of the release and how. What would be a good way to measure that. You're bringing the product person that understands that and then you're bringing that data expert that sort of understands what are the tools that you'll use your data in and what do the data structures need to look like for you to be able to use the data in those tools. How will you actually visualize a metric like this? And then number three, you're bringing in an iOS developer, an Android developer, a backend developer and a web developer to make sure that you actually have access to adding the analytics events and the structures that you need them in the product. And so all of these people will feel so much more empowered to look at the data as soon as it's out. If you do this in advance. And you will also design better data. So design better data, people will feel more empowered to use the data and you'll be ready to look at it as soon as the release is out. And the third thing, one of my favorite side effects of doing this, making analytics a cross functional team sport that happens in the beginning, in the beginning of the release, sort of feature release, is I've seen a lot of developers who generally are a really smart group of people. I've seen them have a perspective of thinking about analytics as a chore that they have to do for an analyst somewhere else that they don't really necessarily work directly with and has nothing to do with them or the end user. I've seen the perspective of a developer change from that to oh, actually analytics is the most fundamental tool that I have to build better user experiences. And so I've seen the conversion. For example, at Quizup we had like a stray developer, one or two developers that would occasionally look at charts and mixpanel for example, two for something like 60, 70% of our developers would just come to us and be like, hey, can we do a purpose meeting? Which is what we call this purpose meeting before they started implementing the product and then as soon as the product was out they'd be looking at the dashboards and seeing like how was the release going? What did the users do with the thing that I just built? And so yeah, so doing this making analytics across functional team sport that happens along with the product kickoff has these really key three things that impact your culture. You design better data. People are more empowered to use the data because they understand why it's there and what it means. And then developers get converted to actually wanting to implement analytics and that's why it's a reinforcing cycle. Better breeds better. You have cross functional collaboration where product engineering and data all care about data that breeds better data. They want to implement high quality analytics. They don't want to see that sort of all of the versions of the Ad Song event because they know how terrible that's going to be when they start looking at the charts. So teams are invested in shaping quality data and then they make better decisions because they have better data and then they keep wanting to cross functional collaborate and then this get gets contagious throughout the entire organization. You'll start seeing like so this is maybe so if you're thinking how do I get started? My recommendation is always start small. Start with a data curious developer and a data curious product manager and like a some a data person that really wants to change the culture. Start a trio like that or a team that thinks like that, a cross functional team that thinks like that. And then sort of try doing this with the next feature release for them and seeing how sort of how better empowered you feel to actually understand the release and the impact on the release. And then you will start seeing this sort of spread to the rest of the teams. The other teams want to do this the same thing like use the first thing as a case study. And again don't only start small in the sense that you start with a single team. Also start small in the sense that you should not start with trying to figure this out for your company wide North Star metric. Because your North Star metric will also will typically like a big like what is the ultimate business goal of the organization that will typically require you to impact the quality of the data for like so many parts of the code base. So you want to start just small to show what can be done with a single conversion funnel if it's done well and show everyone how pleasant this journey is if you do it this way. So that's my recommendation. And so just a quick recap on what the tools might look like. Again I mentioned this at the beginning. Right now we have sort of like an AVO local here but I just just explaining the tools that you could be using and then you could use whatever tool like that sort of or build something. Developers love building tools so maybe you'll end up building things but ultimately this is what we want to strive for in a self serve governance workflow. You basically plan an analytics update and you do you maintain the single source of truth. Somewhere you have a JSON schema or YAML file or something and then you have some sort of a process to sort of review and approve and understand what the changes are like what do the changes look like and something to catch when someone has designed data that doesn't match the global standards. So for example here someone added a property name that doesn't match the casing that the company has agreed to. So you have some way of like a single source of truth and you have some governance to catch design because the data bugs don't only happen in the code base, they also happen in the design and then ideally this source of truth is somehow linked to the actual code Base. So for example, at quizup and at AVO we offered code generation, so we would generate SDKs for iOS and Android and web and backend and all of the microservices to implement the analytics according to spac, without really having to, for example, spell out the name of the event, you would just call a function called AVO songed. And at Quizo we had something similar that was home built and that will both shorten the time of implementation for the developers, but also make sure that this documentation is actually up to date because it's directly linked to the code base. Then ideally you also have some sort of a validation in place, like a dashboard that shows you that an analytics event has in fact been implemented and that you're seeing some amount of volume on it, for example, on your development environment before you release. And maybe even what we built at Quizup and what we offer at AVO is like an in app debugger that shows you when you trigger the events so you can confirm that the event is actually triggered at the right time. So all of these tools, like try to link this entire experience together. Instead of actually having these disjointed conversations that are happening on Slack and a spreadsheet and a figma every, try to connect them a little bit together. Connect the schema of your analytics events directly to your code base. And then the last part and not least part, make sure that the schema that you have defined here, upstream that, that actually is available also in your downstream tools. So for example, if you're working in mix panel or amplitude or segment or something, then you will be using your analytics events there. And you want to publish the schemas of your analytics events into here as well. Because your analytics events, as you define them here, they will have a description for what they mean. So playlist created, that will be an analytics event that has a description that explains when the event exactly is triggered and where in the app. And is it triggered when someone presses okay, after they've named it, or is it triggered when someone initially presses the plus icon to create a playlist? These details matter when you're looking at minor conversion rates. And so you want the description of the analytics events and the definitions of them, like what are the allowed properties, the property types, you want those available in the tool that you'll actually be using your data in. And that way you will have a faster, more holistic journey with validation along the way, while having to have done less thinking along the way, basically. And then you'll continue your cycle. You Know, analyze and learn and do more product releases. So that's it for me. Thank you.

All right, super. Thank you so much, Steph. That was really informative. I have to say. I did love your quote about how data is the most fundamental tool that developers have to improve the customer experience. I might have to steal that line. All right, so for everyone in the audience, I've got about 10 minutes left for your questions, so please do ask your questions in the chat. In the meantime, I've got a couple of questions for you. So I think one thing about project analytics is the people that the data folk have to speak to are normally a little bit technical. So speaking to product managers, they're speaking to developers, so they've got technical understanding. They just don't have that data language.

I'm.

What can you do to make communication easier between these different groups? Like between your engineers and your data people and your product managers?

Right, so you're thinking about in the analytics release process, how might you make sure these people are sort of speaking the same language, basically?

Exactly.

Yeah. And that's exactly. That's sort of the. The purpose meeting. This is the slide, the make analytics across functional Team sport. I should probably have a something like the purpose meeting here because this is what we called it at Quizza. People would literally call out for purpose meetings. It's something that originally we just called for the sit down. We were like, hey, engineers. And a product designer or product manager, can you please meet us? Because, you know, we are tasked with designing the data for this release and we don't really know what this release is about, so please meet us and talk about that. And so originally it was just like a casual conversation where we were just trying to talk to our stakeholders and understand what are they trying to build. And then from there we would use our knowledge of the data and the structures of the data to help facilitate the result. Where everyone was speaking the same language. Like, what does a goal look like? What does a goal mean? Well, we're gonna. We're gonna make the onboarding experience sort of more efficient. Okay, well, how will we measure the success of that? Well, we might measure, like, day one retention, the impact of day one retention. Well, what is the day one retention defined as? And then we literally define the day one retention. And then from there we decide, like, what are the literal event structures that we will need for that? And just doing enough of these meetings actually ended up resulting in more people understand, like, speaking the shared language across sort of from engineering to product to data, and the Other thing I would say is, so that's one like the purpose meetings and sort of pulling in the stakeholders. The other really impactful thing that we did is doing general data marketing, internal marketing of data. So the two tools that come to mind for that from me was we had a Slack channel called, I don't remember what it was called at Quizzit, but at Avvo internally we have a Slack channel called Data and User Research. And so whenever we find something interesting, we post it there as a mixpanel chart or something that people can just go and like, oh, that's interesting. And then they sort of start learning like, okay, so you know, here's something that someone found. What might I find if I do something compatible, you know, and how might I dig into this thing that, the chart that they have already built to sort of find something in the next layer? So that's that and then the, the other thing, and maybe, maybe it's the third thing I was going to say is it was a part of the, the second thing. But the third thing is also training sessions in the analytics tools and I think that's high leverage. So we did these regular training sessions and just how to use Mixpanel, how to use amplitude, how to use SQL to query this, how to do this. The reason why we did that is it, it sort of. Because when I started that quizup, you know, it was, we were doing product analytics before it was called product analytics. And I didn't really understand what an event was even. And like what is an event? And there are a lot of people that don't really have a conceptual idea of what event event is, let alone what is a good structure for an event or when it should get triggered and what is a good name for it. And so just doing these training and enablement sessions and sharing tidbits from the data, I think that sort of creates an awareness of what these concepts are that we're talking about. When you go into the nitty gritty of like, what does a data point mean?

Yeah, that's absolutely fantastic. I do like your idea of just sharing insights and dashboards in a public place so people can see that stuff is happening with data and then go on and just have a play around with things themselves. We've got a few questions from the audience now. This first question comes from Paul. Paul asks where does the data engineering team get involved? So who should be responsible for event quality analysis or is QA quality analysis? Probably something like that. Analytics devs, data engineering or all of the above.

Yeah, this is a great question, Paul. Thank you so much for asking that question. So I actually did a talk at DBT's coalesce a couple of years ago. It was called don't hire a data engineer. And but it was just to make it absolutely clear, it was not because I don't believe in the importance of the role of the data engineer. It was because often people and businesses hire data engineers and think that's going to be the solution to their problems. And it was like a hot take talk on understanding what you are looking for in the different data roles at your organization, organizations. And I think that's sort of the essence of the question. Where does data engineering fit in with this process? Can you bring the question up again? I've lost it here for a minute. Yeah, exactly. When you see data engineering team getting involved. So yeah, the reason why I was sort of touching on this like don't hire engineer talk, which had this message that it's like having someone that manages your data pipelines will not be a solution to all of your, your problems around how the organization is leveraging data. You really need someone who is obsessing over sort of like the actual analysis. And that's not necessarily what sparks joy for data engineers. Data engineers often like what sparks joy for them is like building something that scales and like, you know, stitching together data streams and doing stuff like that. And I think how I think about that is you have the analyst that sort of like is working on the insights. You have the analytics engineer maybe that's working on like taking raw data and modeling it. How, what should, like how do we take raw data from there from Stripe and sort of attach that to like the user experience and Mixpanel to create some sort of a metric. And they use tools like DPT and things like that. And then we have data engineers who manage sort of more stressful and raw data pipelines. If you have to port data from one source to another and you have to maybe do some enrichments of data and then it's a complicated process. From my perspective, the ideal state is if the data design of what the event looks like when you actually send it from the code and what it looks like when you use it, eventually the shortest distance, the shorter you can make the distance between what these two objects look like, the better because that just increases the understanding of your data and decreases the risk of sort of things that can go wrong along the way. And that means ideally data engineers, they should always be working towards making themselves obsolete. That's my Opinion like obviously they never will be because there will always be a big problem to face. But and how I like to think about that. And instead it's like try to make yourself obsolete from this so that, so that you can focus on a bigger problem somewhere else.

Yeah, that does seem to be a perennial problem in that defining like the right metrics and what good data quality means is sort of the data engineers have to do some of that, but it has to be have input from the analysts and then the business people as well and get them all to agree on like what the right metric is. It's kind of tricky.

Maybe I can answer this one more quickly. It depends on your stack, your data infrastructure. The role of the data engineering team depends on the infrastructure. So if you have a really complicated infrastructure where what the event looks like when it gets triggered from the code base, the product code base, and what it looks like when it gets used by an analyst or product manager somewhere, if that has to go through a lot of loops that data engineers manage, then the data engineers are fundamental piece in the purpose meeting and in this sit down that we're talking about. But ideally data engineers get to focus on other things and the events sort of get to flow fairly straight in a fairly straightforward way downstream to where you actually use the data. That's my maybe answer to that question.

Okay, yeah, that sounds incredibly sensible. So we're sort of at time. There is one more question from the audience I want to get to. Before that, I was going to say next week we've got three different webinars. So Tuesday we've got a code along on retrieval, augmented generation with Llama index. So if you're into generative AI, got a little bit of Python, then that's gonna be for one to come along to. On Wednesday there's a session on the secret to successful data literacy programs. So if you're trying to train your colleagues on more data skills, then please do come along on Wednesday. And then Friday we've got a session on fine tuning GPT with the OpenAI API. So again that's more generative AI and a bit of Python. So please do come back for those. All right, so let's go for one more question from the audience. This is from Patrick. So Patrick says between working between product engineering and data teams, do you typically see an aha moment? How many iterations would you send takes on average?

Okay, let me see. Working between product engineering and data teams, do you typically see an aha moment in how many others?

See, I think that's do you ever get the three teams to stop fighting each other?

Yeah, exactly. Yeah, I do. Yeah. I thought I, I think this was like, this was the ultimate, ultimate thing that I really wanted to do at Quiz App. I mean, my. If I just take it to like, the, like, the core emotions of the journey is like, this was such a frustrating experience. I was literally, you know, I had this meme on LinkedIn the other day when you remember the spider man. Like, you know this, this spider man is pointing at that spider man. And that. It was like the CEO would ask the product team or me, typically me, like, how did that release go? And I'd be like, what release are you talking about? I don't even know that there was a release. And then the. I would ask the product team, like, did you add any tracking? And they'd be like, you know, why didn't you ask us to? Or, I don't know, you know, and, and then I go back to the CEO and I'm like, I don't, I, I can't. You have to wait for another release. And, and so this is a terribly frustrating experience for everyone. And so ultimately, when we started doing these purpose meetings, I think everyone, it worked out so well that developers started asking for them. And I think that says so much about how much of an aha. This took. It probably took like maybe three or four. It wasn't two or three iterations. Like, do one and then see the data go out and then do the. Not do, do another one. And again, good data goes out and everyone's like, wow, I'm gonna ask for it next time. You know.

It'S not often we have a webinar where like, the solution to everything appears to be like more meetings, but.

Yeah, well, with a disclaimer. Oh, my God, I don't want it to sound like that, but it's like, I mean, what you're trying to do is a really short sit down, 30 minute sit down to just align on this. And you can do that asynchronously. We do that at our asynchronously to do. And as you evolve as an organization, you can do these things asynchronously. But having a conversation is really helpful. And how I like to frame this is, it's way less expensive and way less frustrating for an organization to do a little bit of proactive data management than to be in reactive damage control after the fact and having to take a ton of meetings where no one's able to make any decisions and everyone just points pointing fingers at each other. So ultimately we're not talking about more meetings. We're talking about a simple one in the beginning at least some sort of an alignment session, whether it's asynchronous or as a meeting rather than doing a lot of crazy stuff after the fact, basically.

Okay, yeah. Understanding why you're doing stuff before you do it is often a brilliant, brilliant thing. So definitely in favor of that. All right, super. We're over time now, so we're gonna have to wrap up. Thank you so much, Steph. That was like really, really fantastic stuff. I feel like I learned a lot there. Thank you to Reese Moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. I want to see you all again in the webinars next week. Go to data camp.comwebinars to register. All right, see you again soon.

Thank you so much for having me. I look forward to.