Panel discussion (uploaded March 20, 2025 by Jason M. Lemkin) examining which early user metrics actually predict 12-month retention and expansion. Speakers report a regression analysis showing that certain metrics occurring within the first 90 days are highly correlated with 12-month outcomes; other intuitively appealing behaviors (e.g., commenting/interaction) did not predict growth and were removed from their onboarding scorecard, which now includes six validated measures. The speakers also share cautionary anecdotes that headline growth metrics can mask serious underlying problems ("high growth hides all sins"), stressing the need for careful data-driven validation of KPIs.
Chapters
These metrics are not at all correlated with retention and expansion 12 months later. But we've done a massive regression analysis, we see the user base and we actually know now what are the top metrics, if they happen in the first 90 days, are highly, highly correlated with metrics 12 months later. So we had to check ourselves. We were kind of surprised. As Erica mentioned, number of integrations was an important one. I thought that like commenting and interacting through the product was important. They're like, not at all important. Doesn't predict any future growth. Throw it out. So we really had to be challenged to throw it out. We since redid that onboarding scorecard. It's now made up of six different measures that are proven to be related to retention and expansion. But it really questioned our initial thinking. Even as CSMs, we thought we had a good sense of that and we were seeing maybe false positives. And so having that go through like a data science team that can actually measure that against revenue outcomes or ultimately predictability is what we're hoping for is kind of what really allowed us to redo the metrics.
Yeah. So I'll give sir a very specific example here. So I A couple of companies ago, I was at segment and when you looked at from the outside, the growth story was incredible. Right? Like huge numbers. And growth numbers are the least trustworthy numbers you can look at as a business leader because it masked like insane leaky bucket things that were going on in the business. You were like, holy crap, what is going on in this business? In fact, want to hear firsthand war stories? I see Katrina right here. Katrina, raise your hand real quick. Raise a. Hi, we got Sarah Burns. Where's Matt Smitterbush? Somewhere in the back there talking. We got Rob Schmelzer. Where's Rob? He's probably around here as well. Find any of them? Ask them like the war stories of what it's like to be in the system where you're like, everyone on the outside is, this is great. 3 billion acquisition. Amazing. You're like leaky bucket from hell. Right. Like, that is basically a reality of how much numbers can sort of mask the reality of what's going on in the business.
I'll say, Matt, you had a quote. I don't know if you remember saying this to me, but when we're talking about like, oh, we're hiring really quickly, but we didn't have the process. High growth hides all sins.
