Selecting the Right Performance Indicators

I advised this founder and she started this company and she said, I'm really trying to land one or two of these deals. And I said, show me how many people you're contacting, show me your funnel, show me. And she said, well, I'm talking to these three people. And I said, the list should be 20 because you need to have all these threads. And part of her product actually requires going into partnering with these large organizations. And I'm like, you should be having 20 active conversations because your yield is probably less than 10%. And at 3, your probability of success in 6 months or a year, you're burning faster. And suddenly she's like, oh, I had no idea. And so this is not, she had an amazing product, but at the same time, this was her first kind of founding of a company like this. And suddenly, once she started pulling the cadence into 20, and she's like, I'm having all these conversations, suddenly there's like 5 hot leads. And it went from 3 kind of lukewarm leads. And so a lot of it is really for your industry, for what it is, what is your KPI? For her, I just gave her an arbitrary number, and for that. When I was talking about the growth team, The number that we were really focused on is how fast can you actually show that you can ship? Because part of the issue was we had shipped really slow before. And we needed to prove that a team could ship, first of all. And that while we were building the A/B testing system, we wanted to show that we could ship and we couldn't measure everything. And that once you get that machine rolling, then you can actually improve it over time. I think it really depends on what you— I said for App Install Ads, we were testing— we actually tested 5 or 6 products before we landed on that one. And we cut a bunch of them. First is, do you believe that you have the right TAM and the right market? That is really critical. And if you do believe, then the question then becomes, you know that you're a farmer, now you gotta find the right soil. You already know what seed it is, but you have to pick. Either you pick the soil or you pick the seed. The challenge is some people are like, okay, I know what seed and what soil, and therefore I just keep planting and it's failing. What should I do? I'm like, well, you could buy fertilizer. It could be really expensive. Or you could actually find better soil for you. And so really thinking about for your industry. One thing we did in platform when I was at Facebook was we had this rule. We would just ship things. And when I started leading platform, like, no, no, no. We're going to alpha customers, and we're just going to offer this product to some partners. And then in beta, if we tell them we shut it off and they beg us to keep it, then we know we're going into beta.

And was there a metric that you were looking for there? I know Sean Ellis has—

Yeah, so the metric we had was we actually would go back to the alpha customers and say, we're going to cut this off. And if no one protested, they didn't want it. We kind of just they did it as a favor to us. But if they were like, how can we get to be on the launch list? Then you knew that they had something that they wanted. And we knew that when we started building things, we would shut things off, and then if no one complained, then nobody cared. But if it was actually accretive to their business and they were starting to depend on it, suddenly it was something that they were begging for. And it turned out that the thing they ended up begging for was an ads product, not an organic product, because they could now predict they could raise money on this. It was just a very different way of looking at the world. And so really kind of figuring out what that key metric is for you, And the metric isn't profit, the metric is what are early indicators. For Marketplace, actually the first thing we looked for was not transactions, which is our ultimate metric previously. It was actually weekly active users. Could we get people even interested enough to actually come back? And what is our weekly active retention? It had nothing to do with transactions. This is can we get even people to come to this place? And then we got to are people actually clicking on it? Then we looked at are people messaging each other? Then we looked at measuring transactions. But it was a long journey to get to the actual ultimate transaction.

OK, and so almost as a precursor to predicting willingness to engage or pay, you were looking at a certain level of usage, and then you were taking it away once you had that certain level of usage to look for conviction that they were going to actually champion. But I love this idea of these meta things that you're looking at, like the pace of release versus the actual thing that you're releasing, and just getting that as a predictor of ultimate success.

Well, I think especially in a subscription business, some of those downstream metrics take months to mature, right? And so you don't know if you're going in the right direction. So we look for upstream, what are the precursors? Metrics, and the first was can we release? Second was what is our win rate for those releases? But also what are the interim metrics along the way, like hints accepted or users bill-through rate? So we actually would look, but ultimately a subscription product, people subscribe on monthly or 6 months, and so it might actually take you 6 months to mature any cohort. And so you have to be really careful understanding that.

And was there a philosophical view? You said you did 5 or 6 at once. Do you have a bias about whether or not you should do multiple experiments in parallel? Or just do one each at a time serially and get to a kill or victory and then move on to the next one?

So always do, depending on the size of your team, one of the things, even in a small team when we were testing the app install ads, we were testing 5 or 6 things at once. Because we wanted to see what is, again, the sower sowed all of it at once, and then you see where do you want to double down. It is really hard, some teams aren't great at doing that, but it is important because you might actually find good soil in a place that you didn't expect. And you might find failure in the place that you would have spent 100% of your resources. And the problem with serializing it is you could end up in a place where if it doesn't work, you've wasted a bunch of time. And the most precious resource as an entrepreneur and as a startup is time, because it's opportunity cost. Every day you're burning, it's not just burning dollars, but you're burning your engineering resources, you're burning your customers' time, you're burning your marketing time. All of that is actually your most precious resource.