Jonathan Hsu, lead data scientist at Social Capital, discusses how teams choose key metrics, the trade-offs between design and features, and how to run A/B tests. He explains that required sample size depends on effect size, and covers differences between consumer and enterprise metrics.
Chapters
You're mentioning that Facebook is currently monitoring around 100 numbers. I'm wondering if you can only monitor 2 or 3 metrics, what would you monitor? Is there any difference between the concentration?
It's good. This actually comes down to organizational dynamics. When I say Facebook measures 100 numbers, I don't actually mean that. If you take the sum of all numbers that people are watching at Facebook, it's tens of thousands.
Yes.
If you take any one team, they're all in your position. Well, what are the 2 or 3? And that's kind of for them to decide. For that team, it's going to be, I want to optimize people putting information in through the profile, like the description I just gave. For this team over here, it might be new user acquisition in Japan. It could be very specific. And then when you add together all the specific goals and you sort of go through some layer of filtering, the result is— the 100 number that I'm using there is sort of the one that sort of top level that everything is subject to in some sense, right? But it could be, you know, anything. It could be anything. But actually, I think that there's an important point in here, which is that I think that it's very important to think hard about what it is, right? What it is that you're trying to achieve. There is this phenomenon that I see sometimes where companies, they build something like a consumer app, and users use it, and it's cool. Users are using it. And then they hire a designer. And the designer's like, this thing looks like garbage. We need to redesign the whole thing. And that's not necessarily bad, but the thing is that they would be better off couching that in the frame of some sort of measurable and quantitative goal. We want to redesign this because we want to achieve an NPS increase of x, and we have reason to believe— you better have some reason other than your own anecdote— that changing the design will actually lead to an increase in NPS. So if you have some framing about what the goals are, and you have a product team that decides on some stuff, but the stuff that they're doing does not relate to the goals, you're going to be in a problematic situation. Now, you might say, that's never going to happen to me. That never happens. But it does happen. And it happens almost always the way I just described, usually at the behest of the designers. It's not to say that design work isn't bad. It's just a question of prioritization. Like, should you prioritize the design work over building out a feature that adds this core functionality that you need now, which will add definite incremental value as opposed to some fuzzier NPS value? So when you're trying to prioritize between projects, it's really good to have concrete numbers.
And any difference between the consumer app versus the enterprise app?
Say again?
Any difference between, like, for the metrics, If it's a consumer app, what would you monitor? And if it's an enterprise app?
Sure, sure, sure. I mean, of course there's differences. The most obvious difference is that enterprise apps, people are paying you for them now. And consumer apps, people are probably not paying you for them now. Or if they are, very few of them are. So since people are paying you now, you can create that link between if they like it now, they will pay me now. Whereas in consumer apps, it's like, well, if they like it— well, if I get enough of them, I'll somehow make money in the The main difference is that you hopefully have money in the equation.
Right? Yeah. Yeah.
I have no other slides, by the way.
So in statistics, it's always important to use a testing group, which is a good representation of the real target group. How do you go about that when you have a limited amount of information about your users? Or is this not important when you're testing new functions?
When you have a limited amount of information about your users?
Yeah.
Oh, randomly assign them. You'll be okay. If you have enough of them, right? Presumably you gave them a user ID. Mod it by 2. That works. Hash their email address. Mod it by 2. Random. As long as you're random, you should be okay, right? And as long as you have enough users, you should be okay. Isn't it n equals 30 or greater? No, it actually depends on what you're trying to measure, right? So the thing is this. Let's say you're going to do something that's going to potentially change the user, the object that you're trying to measure, by 30%. Let's say you might be changing retention by 30%, and you have some test that's ginormous. If you believe it's a 30% change, then you really only need 10 or 20 users. But if the change that you're trying to measure is tiny, 0.5%, which we were doing a lot at Facebook, we were trying to measure this— OK, I want to raise conversion by basis points. If you're going to do that, you need a lot more users than 50. You're going to need thousands and tens of thousands. So it's actually a function of the disruption that you're going to be introducing to the system. A big disruption, you need fewer users to measure it. A small disruption, you need more users to measure it.
Why is that?
That seems counterintuitive. Why is that? No, that's right. If it's a big difference, If it's a big difference, right? Does that make sense? If it's a big difference, I only need to, like— think of it this way. Here's a proposition. If I cut off somebody's arm, they might bleed to death. How many people's arms do I need to cut off to test that? Probably like one. Right? I could test that very easily. So big changes, like losing your arm, small n. Good.
Hi. So in terms of analytics, what do you think determines It's the stuff I showed you.
In some sense, from our point of view, measuring these things and weighing these things, if you can demonstrate strength at all these levels, demonstrate strength in the lifetime of the user, the full lifetime of the user, demonstrate strength in terms of balancing off getting new users and losing users, if you can demonstrate strength in terms of the depth of engagement of those users, that's hard to trade. It's hard to fake that, right? So if you can line all these up, we can be more and more sure that you have established product-market fit.
Okay.
And most importantly, if you merely show us the first graph, 5% month-over-month growth, that is not product-market fit. That is a graph that just grows 5% month-over-month.
