Five Bad Data Habits that Lead to Costly Decisions with Sisu Data | SaaStr Software Community

Build.

Thank you for joining today's session five Bad Data Habits that Lead to Costly Decisions. I'm going to go over a few quick housekeeping items before I pass it over to our speakers today. As many of you know, you've already probably been to a few sessions. Our sessions. You can view our sessions a few different ways. Many of you are in our event platform, which we're really hoping you're enjoying in all the networking opportunities there. To submit a question in that event platform, you can simply go into the question feature on the right rail of your screen and drop that question in. The team will be taking questions towards the end of their presentation. Now, if you'd like to join in the room with us and be able to actually ask the speakers directly your questions, you can click the button that says Join us on Stage button below the player and you can actually unmute yourself, turn on your camera and be part of the session and ask the questions at the end. This will be an interactive 20 minute session and we encourage you again to dive in and be interactive with us. Lastly, as many of you know at Saster, our sessions are recorded and will be available post event. So without further ado, I'd like to introduce the CCU Data team. Peter and Barrett, you'll be taking it over from here. Thanks so much for being with us.

Awesome. Thanks so much for having us. We're really excited to be here today to talk a little about some common pitfalls we find in terms of building data driven companies, especially on the path to growth and opportunities to do better and what to do about them. So just by way of introduction, my name is Peter, I'm the CEO and founder of cc. Before CC was on the faculty at Stanford in Computer Science where I worked on making large amounts of data more useful, making ML more usable, and after shipping a bunch of the work that we're doing at fairly large scale and at web services like Facebook and Google and Microsoft realized having more fun shipping products than writing papers. Here I am today and really excited to share some of the lessons we've learned along this journey, both from the academic perspective and from the perspective of a fast growing analytics startup and founder seat. And I'm joined today by Barrett Hoffman who leads product at cc.

Great, great to be here. Thanks Peter. And as Peter said, I lead product at CCU and I'm personally really excited for this conversation for a couple reasons. First, a lot of what we're going to be talking about today in terms of some of the challenges and common pitfalls are things that I've personally experienced when growing and launching and leading products at both really early stage, higher growth companies as well as at scale while I was at Google. So these are challenges I personally felt and that was a big motivator for me to join ccu. And the second reason I'm really excited is over the last year or so during my time at ccu, we've spent a lot of time, I've probably at this point had hundreds of conversations with customers and through user research, really digging in to understand some of the patterns that we're seeing. And I'm personally excited about the opportunity to share some of the insights from those hundreds of conversations.

Awesome. So what are we doing here at a conference around growth, talking about data? Not a surprise, hopefully to anyone in the audience or listening to this talk that using data is critical to making the right decisions along the path to scaling up and especially when there are difficult decisions and gut, gut feel isn't necessarily serving you. Being able to look at all the data you've got about your operations, your products, your services, your users is critical. And as Barrett mentioned, you know, we were based in this talk based on experiences both in prior lives and also working with companies both very large like Samsung, and also getting very large like Gusto. And the problem today is that making decisions using data is actually pretty hard. Right? If you take it from from Gartner, one of their recent surveys, only 54% of leaders say that their decisions are actually influenced by data. That's barely more than half. And maybe more importantly and for many of the audience, you might also find yourself in the camp of 73% of leaders saying that data hasn't had the expected influence. This is all kind of funny given that if you think about the rise of Data Stack and companies like Snowflake, we have more data, but somehow we're not putting it into action. Question is, what's going on? So our job today, and if we do our job correctly, you'll be able to leave this session with an understanding of five key pitfalls we find leaders falling into. And these aren't just one silver bullet. It's a blend of different behaviors ranging from cultural to processes to tools, where we think you can have immediate impact on your ability to operationalize your data.

Great. And I'm excited to kick it off with this first bad habit, which is probably one of the most common ones that we see. And it's the mistake of analyzing only a portion of your data. And the way we typically refer to this is you're Only looking at the common slices or the common slices and dices that you already probably know have an impact on your business. The challenge is this actually leads to really big missed opportunities in terms of how you could be growing customers and growing revenue. And as Peter said, in particular, given the additional data and the scale of cloud data infrastructure, we have more data. And so these missed opportunities, they're at your fingertips with this additional data. Now, to give a little bit more tangible example of what we mean when we say only looking at a portion of your data, this is an example from one of our CC customers who's a B2B SaaS provider, and they were experiencing or noticing significant churn during a very critical renewal month for their business. And the typical approach for them was to look at a factor or an attribute of customers that they already knew had an important influence on churn rate, and that was what insurance provider the customer was using. That would have been the typical way. And unfortunately, that would have led to actually missing some really key insights that were hiding just a little bit deeper in the data that they already had. What we're really excited to see was that with cc, they were able to actually go beyond just that common slice, that common dimension that they already knew mattered. And they started looking at the other attributes, the other behaviors that customers were taking, and not even just those additional dimensions on their own, but actually combinations of those dimensions. So looking at things like, you know, and actually uncovering that things like the number of employees or specific features that customers were engaging in. When you started to bring in these dimensions and actually look at combinations of them, there were really interesting, more nuanced patterns that were associated with higher or lower churn. By getting into those more nuanced insights, by going a little deeper and using all of their data, they were actually able to create a much more targeted and as a result, much more effective engagement strategy, which for them translated to seven figures of additional retained ARR. So the takeaway here is a common bad habit is only looking at the pieces of your data that you already know matter and what we really encourage and what the best data teams are able to do and the best companies are able to do is actually go beyond those common slices and start looking at all of the actionable dimensions in your data and even the combinations so you get these more nuanced insights.

Another bad habit we often see is treating your data teams like a report factory. So no surprise to anyone here, there's going to be a lot of questions you have about your data. They're Often coming up in urgent times. And the problem we often find is that sending over requests to an analytics team without giving the shared business context or strategic reason behind why a question's being asked can lead to a lot of wasted cycles making your job harder and your data team's job harder. So, common scenario, just to give you a concrete example of this, worked with a product leader who just launched a new product, very excited to see what was driving user engagement, and because of the complexity of both the data involved and the user behavioral patterns, waited weeks as opposed to days for their analyst team to come back with data about what was actually driving engagement. Problem is, once they got the report back from the data team, they found that the report contained what features within the product were leading to engagement. But the question that the product leader actually wanted to know was at what level of engagement these features actually drove adoption. So they knew what features were important and interesting. They wanted to know in their ga, when they moved this over to general availability and open this up for a self service trial, what behaviors they wanted that they should incentivize people to perform. Should they invite three friends? Should they invite five friends? Should they sign up for three subscriptions? Right. They didn't get the right granularity and as a result, although a lot of this analysis had already been done, took another round trip to go back to the data team and go get the actual right answers, which is what thresholds of activity led to the appropriate user behavior. And by failing to align this business goal up front, specifically tell me what thresholds are needed in order to drive good trial activity for the new product experience, this led to a lot of wasted time and ultimately delayed decisions in terms of what campaigns they want to run in market. So the opportunity here is really much more around processing culture, which is to the extent to which you start seeing your data team as business partners and involving them early, you can ensure you're getting these faster and more actionable insights. And by aligning up front about what these insights look like, what specific problem you're trying to solve, and and really kicking off projects together versus treating the data team as essentially only digging in when there's a red alert or something that looks really wrong, you can get the right answer the first time as opposed to having to do multiple iteration loops with them. So in summary, stop treating your data team like a report factory. Instead involve the data team early and align up front on what results are actually actionable to you and your business.

Peter just talked about the importance of not treating data teams like a report factory. At the same time, we recognize that reports are a critical part of doing analysis and actually making the insights from your data actionable. And so this next bad habit, we'll talk a little bit about how to actually approach those reports, acknowledging that they're a critical part of the process. And one of the key insights that we see very commonly is so often the questions that come up that you're trying to dig into your data to understand they're not addressed by the pre built dashboards that already exist. And so what happens is you end up spinning up pretty frequently these kind of ad hoc or situational analyses. And when that happens, people jump into Excel or into writing SQL or writing Python to be able to answer these situational questions. And when they do that, they're jumping into different tools and dispersing the work in a number of places and a number of individuals doing it. And each one is coming up with kind of their own format of how to report the results. And that makes it really hard to, to interpret and orient yourself to the results when the format is changing all the time. And so to dig in a little bit more, what we see here was, what's been really interesting is we hear this on kind of both sides of business. Leaders, people leading functions. Like many of you all listening in today, people leading functions like product or marketing or sales or customer success, they're frustrated by the fact that every time they get a report it's a slightly different format and they're really wishing there was a more standardized way. On the flip side, you've got data leaders and analysts who are actually saying that the generation of those reports and having to do a lot of manual ad hoc work to translate into all these different formats is actually really, really frustrating and oftentimes one of the most painful parts. So how can we address this challenge? Our recommendation here is as early as possible, really invest in and be intentional about what reporting formats make sense for your business and your leadership team. And then try to standardize and encourage people to use those common reports so that you can become familiar with them. And as an example, one common format that we see are these waterfall charts. They're a really effective way to actually communicate in a concise, consumable fashion what subgroups in the data. For instance, actually what was the kind of incremental impact of each of those key drivers in going from one value, maybe one period to the next period in the metrics. So that's an example of the type of format that you might want to standardize on. In summary, we know that analyses have to happen in a situational fashion. You have to respond as the business is changing, as everything in the market is changing. When that happens, it's really easy to fall into this habit of proliferating a number of different reporting formats and not really standardizing. What we encourage you to do instead is notice that that's happening and be intentional about establishing those common formats early.

Once you've got a report output. There's another common concern we see related to another bad habit, which is making it impossible to actually ask follow up questions when you do get a report, whether that's standardized or not. So we all know from getting a report back that follow up questions are inevitable. But many times it's very hard for business leaders to go and answer questions and get a follow up, especially if they have to involve their data team. So with data teams as a critical bottleneck, many of these questions go unanswered and this is a pain point on both sides of the spectrum. So for many of you who are business leaders, you may empathize with these with these scenarios where if something isn't immediately on the dashboard report that I see, it takes too much time to go drill. And I don't know SQL, I don't want to spend my time writing SQL and if I have to get someone on the data team to go do it, it'll take too long. So I'm just not even going to ask that follow up question. And if you're a data leader or if you work with your data team, they're probably thinking something very, very similar but slightly different. Where data leaders feel like these follow up questions could take up 100% of their time if they went and indulged each and every one of them and they have to draw a line because there's other things to go and do. So the opportunity here is really one around process and tooling. So if you can enable your business team to self service based on the results that are provided, you can allow many of these follow up questions to go and get answers without having to wait for an overloaded data team. It's not realistic that everyone on the business side is going to become a data analyst or a data team member. But we found that many of the follow ups that data or that business leaders actually want to ask of their data are relatively simple variants of the results that have already been presented. For example, if your data team is presenting a report on 14 day retention, making that report easily parameterizable, so you can go look at 28 day retention and 60 day retention or go look at other, say other channels and how their 14 day retention performs can be really critical to contextualize a given result and decide if there's further analysis needed or if you're ready to take an action. And the design pattern we found here is really this idea of having a low threshold and a lower barrier for entry to do common manipulations and changes to reporting, but then also keep a high ceiling so those data experts can go in, reconfigure, go ask those follow up questions. And if you can manage to do this in the same environment, you can enable self service for many of the questions and only kick over the most complex ones to your data team. So in summary here, stop making follow up questions impossible for the business and stop overloading your analyst team with those follow up questions as well and start investing in tools that empower your business leaders to go and answer their own questions.

The last bad habit we're going to talk about is probably the hardest to solve, but it's also arguably the most important and impactful to solve. And gesundheit there, Peter. The challenge that we see here is so much time is spent analyzing and trying to get to the answer that by the time you actually get to the answer, the insight, it's too late to act and you're actually missing opportunities. The window closes on the chance to go and proactively retain customers or execute, for example, a compelling go to market motion that might be an additional revenue growth opportunity. What's interesting about this habit is it's something that we see both at the earlier stage higher growth, earlier stage companies, as well as at the common largest tech giants, the typical thing companies that we're all familiar with. And so the two quotes here, the first one is actually from an analytics leader, a data leader at one of these earlier stage higher growth companies who is recognizing sometimes we need the answer that day and it just takes too long. We needed that day because customers are going to turn or we're going to miss that chance if we don't get to it right away. And recognizing the analysis becomes worthless if it takes too long. On the flip side, the second quote was a product leader at a large, very, very large technology company who by the time they received the results of the analysis and identified a specific very strategic customer that was based on their usage patterns, was at risk of churning, took a few days, they called up the customer and that person said, hey, sorry, already made the decision, it was too late. What do we do about this this is where we really believe technology needs to come in and play the role of accelerating and even automating the manual pieces of analysis so that you get there faster and can spend the time on things that are uniquely human. To be a little more concrete, what do we mean about accelerating and automating Here an example is we talked in the first bad habit about looking at all of your data, not just a portion of it. That's a great example where technology can play a really critical role in automating. Looking at all of those different tens or hundreds of millions or even billions of different combinations that exist, you can leverage technology to automate that and help you get down to what's the narrower, smaller set of things I can really focus on. And then you as the expert from a business perspective and really understanding your business, you spend time on identifying which of those things are most actionable and then also executing on those, right? Doing the things that are uniquely human, picking up the phone, talking to customers, executing a cross sell campaign and actually getting into the the action. Right? That's actually going to have an impact on the business. So again, summary here. Don't spend all of your time analyzing, right? We have to run businesses, we need to act on the insights that we have, otherwise the analysis is worthless. And this is where technology can play a really critical role in accelerating and automating those rote manual pieces.

So there you have it. Five bad habits in data and how to break them. These are informed by our experiences both as product leaders and and as folks helping other product leaders with analytics on their path to growth. And the key thing I want to call out here is as in many things with scaling startups, there's no silver bullet here. There's no one weird trick to go and create a data driven culture. But there are a lot of, to borrow a Ben Horowitz phrase, lead bullets that are very, very useful along the way. Some of these are cultural, some of these are process oriented, some of these are tooling related. But with a combination of these, we found it's really possible to move beyond aspiring to do more with data and actually doing more with data. And in the spirit of this event, our really belief from seeing this over the past several years is that recognizing and breaking these habits early can be the difference between a successful outcome and an iconic one that puts you on the stage at Saster in two, three, four years from now. So thank you so much for your time. Barrett and I would love to take questions.

Awesome. Thanks so much. We already have quite a few Questions in the queue. So I'm going to get going. First question is, what do you think the most important data habit to start or break is for the early stage startup?

Peter, do you want to jump in with the first pass here?

Yeah. My first one I would prioritize is just treating that analyst team or that data team as a report factory. I think that we've seen data and analyst teams that are embedded in or have a lot of context in the business being more effective at knowing where to make investments and how to give, not just give the right answers, but understand where to make additional investments, what other data to pull in, what other products and functionality should they be bringing into their internal tech stack? And especially the early in growth stage, being able to cement those good data practices early and get those wins around data which can rally people to the cause of additional investment in data could be really critical. So that report factories. Yeah, go ahead.

Yeah, I'll add just one additional thing which I think so I actually agree with you, Peter. I think that's the most important and one of the other reasons is that it sets a really important cultural norm. That data is a strategic. Doing analysis involving data early when we're thinking about business decisions is a strategic imperative at our company.

Great. Next question. What do you do when the data doesn't support the anecdotal things your business teams are hearing on the front lines? What have you seen be a good way to handle this?

Yeah, so I'll add, I think that's a really challenging one for sure and it's something that we know people run into a lot of. One thing I've seen be effective here is making sure that you're not skipping over the insights and the data that supports maybe the anecdotal or obvious thing. So it can be really easy to jump and almost gloss over or ignore the thing that the business leader maybe already knows is important and that causes them to then say, well, I already know this other thing is important. You're not telling me that thing, so how do I trust anything you're saying? So that's probably my biggest piece of advice there is. Don't gloss over the obvious. Ground yourself in the obvious, build that trust and credibility and then move on to the less obvious. As an additional point, once you've established that credibility.

Great. I think we've got time for maybe two more. So we've got. Let me grab one here. If you could only start with one, what habit would you break from first?

That's good. So I think this is in contrast to the first question, which I think was, what do you focus on as kind of a proactive initiative? First, if you've already fell into some of these habits, what do you break first? You know, I think probably the, and this is, you know, my gut here would be really getting beyond only looking at the portion of of your data. I think that is something that is, like I said, probably one of the most common pitfalls. And it's something that is really important to break because you're investing in getting a lot of data. You're investing in getting that data into probably a cloud data warehouse that you can in theory, do analysis on it. And so I think the biggest impact you're going to have is if you can go beyond, hey, we're only looking at a portion of all this data that we're investing in getting ready for analysis. We're only looking at a portion at it. If you can really start to look at all of it, that's probably the first thing and I think the most actionable thing that we talked about today.

Excellent. Okay, last one and then we'll have to wrap this up and we'll send you the other questions so you have them. What's one tactical, practical way to show the power of being data driven, to get the naysayers to change their mind?