Jonathan Hsu explains why Bayesian reasoning matters more than p-values when analyzing product metrics. He uses Facebook's A/B testing framework to show how correlation signals like "10 friends in 14 days" drive decisions without causal proof, and why false positives are inevitable when measuring many metrics at once.
Chapters
So what are the drivers? OK. You guys know this cartoon?
Yeah.
OK. So this cartoon is from XKCD. He says, OK, this is a neutrino detector. The question we're trying to answer is if the sun exploded or not. The way the neutrino detector works is it measures whether the sun has exploded, and then it rolls 2 dice. If the 2 dice both come up 6, it just lies. Otherwise, it tells the truth. Okay, so the machine goes off. Ding. It says it blew up. And then the frequentist statistician says, did the sun blow up? And he says, well, the p-value is something, 0.027, which is less than 0.05, so I conclude that the sun did indeed explode. The Bayesian says, I bet you it did not. Okay, so when you're trying to identify sort of the drivers of your company's success, when I say be Bayesian, what I'm saying here is, Incorporate prior knowledge. So what's going on here is the Bayesian is using prior knowledge, namely the knowledge that the sun has never exploded in his lifetime, right? And he's incorporating that prior knowledge. This person over here is acting as if there is no prior knowledge, completely in a vacuum, and is arriving at some absurd conclusion. Okay, so when you are trying to figure out what are the drivers of your company, of your business, you want to be Bayesian. Do not take the route of just be like, oh well, just do statistics or hire a PhD, because that's not what it's about, right? It's about incorporating your intuition and taking data and folding your preexisting knowledge and the data and coming up to a result. And that's actually what Bayes' theorem is. This thing over here says, okay, well, the probability of something being true, given the evidence, is equal to simply the probability of it being true times some stuff that's dependent on the evidence. Okay, so it's like, This is what you thought it was originally, and now that you've seen some stuff, some evidence, you adjust it, and now you believe the thing is true with some different amount. That's how to interpret this thing. Okay, so example. So for Facebook, they measure like 100 important numbers or so when they do testing. They test a feature, they're measuring a ton of stuff, and they all have confidence intervals of them, like 95% confidence intervals, but if you have 95% confidence intervals, Like, 5 of those 100 are going to be outside of the confidence intervals. So does that mean that you stop working? You're like, oh, well, this feature didn't work. 5 of these were outside of the confidence interval. No, because you have prior knowledge. If you don't have any reasonable explanation for why these things are connected, it's likely that the thing you've seen is due to statistical noise.
Right.
So prior probability is your prior belief. It's not empty. It's not nothing. So in the case of the XKCD cartoon, what this is is the probability that the sun exploded, given that the detector went off. Well, it's equal to the probability that the sun exploded, right, times some stuff over here. If you do the math— you can ask me, I'll do the math for you later on— it's 35 times epsilon, where epsilon is the probability that the sun exploded, your prior belief. If you believe that the sun exploded that the probability the sun exploded is truly 0, this is always truly 0. But if you believe, oh, maybe the sun might have exploded, like 50/50, then it's going to be quite large. Questions on that?
No?
Yeah. Are you guys familiar with Bayes' theorem? A little bit.
A little bit.
So when we are doing, for example, financial projections or user projections, before having any data to base it on, What do you do? Right. Well, you do have some data, right? You've lived in the world. You have observed companies in the past, so you can make reasonable assumptions, and that's what you're doing. I do think it's important to add bounds. This is something you'll notice also if you talk to investors. You give them a model. Typically what the investors will have is they have some associates who come from investment banking, and they will basically add confidence intervals to your projections, or they'll throw out your projection altogether, They think it's garbage. But in some sense, that's what they're doing. This is like some abstract theoretical statement about what they're doing.
Good.
OK. Cool? Don't worry about this. This is a statement that correlation is not valueless. If you really get into data science, people worry about correlation versus causation. They start saying, oh, you observed that, but you don't know if you caused that with your feature. They freak out. They say something like, oh, using the feed on Facebook is correlated with long-term retention, but it's not causal. Did it cause long-term retention? Well, you don't actually know that. Maybe long-term retention caused using feed. Maybe it works the other way. I ask you here, what would it actually take to establish causation in that case? How would you know that feed causes long-term retention? You would have to do a randomized controlled trial. long-term retention of Facebook? Any ideas? What would the experiment be? Say again? Right, you'd have to literally not show them feed. You'd have to be like, okay, these users get Facebook with no feed. These users get Facebook with feed. That's not feasible, right, because people just think it's broken. It's not a useful controlled experiment. That's kind of what I mean. At some level, it doesn't actually matter. What does this have to do with actual things? At Facebook, we had this saying, 10 friends in 14 days. We had this notion that if we can get a user to 10 friends in 14 days, that user's going to be stuck. We didn't actually know that having 10 friends causes retention. It's not that we knew anything about causality, but we observed some correlational behavior.
Some correlation.
Correlation in the data and use that to create a rallying point that the company could circle around.
Question? About—
oops, wrong one. This one? Oh, the quick ratio. Unrelated. Good question. Unrelated. This was brought up to us by one of our—
Okay, cool. Second question was— Obviously, you're giving us a lot of information about increasing stickiness for using the service. Would you be able to outline a specific instance where you rolled out a feature and what steps you took in order to make sure that people used it and that they found value in it in such a way that it was more than just, oh, they're using Facebook more?
In the case of Facebook?
Yeah.
Oh, okay. Yeah, yeah, sure. Well, I mean, in the case of Facebook, right, like, That's one of these examples where the top-level thing, which might be use Facebook more, when you have a big company, individual teams within that have their own sub-goals. This team over here might be like, OK, my team is the profile team, and I'm going to work on the profile. My goal is to get them to use profile more. And we will not necessarily worry whether or not that is connected to making the whole thing used more. Does that make sense?
OK. And was there a specific feature within the profile team? Something like that.
The funny thing about this stuff is that— well, OK, no, I shouldn't say funny thing about it. The thing about my framing here, the way that you do this from a measurable point of view, you can actually do this with every single feature. So any time you saw anything added, it was tested after 2012. So for instance, here's an example. So when you're on the Facebook profile, when you're looking at it, you guys may or may not see this. On the left side, you might see a thing that's like, what cities have you visited? You didn't fill this out yet, right? It's like a little unit. So we tested the unit. Okay, the unit does— and the goal in this case was getting you to tell us stuff. We tested the unit.
Cool.
But we even tested variations of the unit. Okay, what if we make the unit— and then what if we make the unit with, like, a little progress bar? And we could isolate, like, the progress bar drives this much more incremental engagement, because you can measure it.
Yeah.
And so using this framework, you could conceivably measure just about anything. The impact of just about anything. Now, just because you can measure anything doesn't mean you should. It has to be some judgment on you. It's like, well, I got an engineer. I'm going to spend him his entire bandwidth building out my A/B test for this? You have to decide that. But really, you can conceptually do this with just about anything. Any time you wish to make a change.
