Understanding Cohort Level | Lead Data Scientist at Social Capital Jonathan Hsu

So we understand money coming in, money coming out. The thing you don't know from looking at it this way is whether or not when this money falls out, did that money fall out from these guys or from these guys? Is it like these customers who came from a long time ago and then you lost them late in their life, or is it from users who just appeared now and you lost them right away? Big difference, right? So here's one way of thinking about it. So, a visualization we typically use, we really need to understand it sort of at the cohort level. So, on the left, what this is here, the way to read this is the x-axis is the month, or I think this is actually week of joining. So, let's say here, this is July something, 2014. This many users joined. Okay? Now, when you go up, the y-coordinate is how many weeks in age those users are. So these guys from July have been around for this many weeks. The guys who started further back, they've been around more weeks, so this goes up. The guys who just joined, they haven't been around that long, so you don't go up as far. Now the color is the retention percentage, so how many of that cohort stuck around. So what this is saying here is that OK, so if you take a fixed slice here, this is 8 weeks. So at 8 weeks, what's their retention at 8 weeks? Well, back in the past, it used to be red, which is pretty good. And here, it's not as good recently. Yellow, worse. So your ability to retain the 4-week or 8-week user is worse now. So the idea of laying it out like this is now you've broken down all your users in terms of where they are in their lifecycle. As time passes, it adds diagonal lines, right? Calendar time. This thing here is Christmas. You see Christmas, right? Everybody kind of falls out. Nobody opens their phones. This is Thanksgiving right before it. Other interesting things, you can tell that something different happened here, right? Because these cohorts are different than these cohorts. And you can see down here the reason why is because the number of new users spiked. So you sort of increased the denominator. You threw all these new users into the system. And invariably, since you added all these new users, they didn't retain as good as ones you got naturally. Their retention is worse. So these cohorts are performing worse. This is very typical. So in this case, what's going on is that the company is trying out some paid acquisition. They go out and buy users. They buy a bunch of Facebook ads. They're like, OK, we're going to spend 5 Gs or something, 10 Gs a week. And they buy a bunch of users. They get a bunch of users, but they don't retain as well. So the aggregate value of what they've gotten is actually not necessarily better than if they hadn't spent that money. And this kind of gives you a way to frame that. This is a similar visualization. This is actually in Mixpanel. Do you guys use Mixpanel? People still use that? Nobody's using Mixpanel. OK, so Mixpanel is an analytics company that provides this as a service. So if you have your app or your business and you just plug into them, they'll provide you this visualization, which is basically the same thing. So when a company comes, once again, when you guys are coming to pitch, when you guys are coming to look, coming around and trying to get people to look at your companies, this is the kind of stuff we're going to look at. With a combination of this and the ones I showed you before, that basically completely disaggregates your growth and breaks it up in terms of how much of it is coming in versus going out, and how much of it is sticking around at the cohort level. And this way we can sort of understand all the components of it. So, So it's one of these things where you can fake the first graph, kind of. You can just buy a lot of users and make the first graph go up. You can't fake these ones. It's really hard to trick these guys to stick around after 40 weeks. There's not a way to mask that. You can actually use this for things other than retention. So that graph I showed you was retention before. In this case, this is actually a SaaS business. On the x-axis is the month of acquiring the SaaS customer. On the y-axis is the cumulative dollars they've spent per customer in the cohort. Here, this says February 2014, there were— what is this? Some number, 165 customers that you acquired, and then they spent money with you each month. So your cumulative dollars per customer went up each month, which is good. So this is cumulative. It's a little different. You can tell this one is very different. This cohort has some crazy whale in it. This cohort had only 80 customers, but one of those customers must have been spending so much money that it drove the cumulative value of that cohort to be very different. So the notion here, this is just another way of understanding the cohort-level behavior of your application. In this case, this would be like for a— Subscription business. You can do this for anything. Retention, you can do it for cumulative revenue like this. I recommend you do this once you start getting customers. Cool. Here's another concept that we use. The concept of L28. So if you have— so remember we were looking at MAU before, right? If you're in a situation where you're like, Trying to gather users, and you have them in MAU, and you're like, oh, I got like 5 million MAU today. So how do you get a sense of who's the power user and who's not the power user? It's actually not entirely obvious. If everybody was paying you money, you could identify the whale. The whale is the guy who spent the most money with you. Like if you had a game and you were collecting all this money, you'd say, oh, those 5 guys there, they spent a lot of money, way more than everybody else.

Yeah.

But if you're in a position where you're collecting eyeballs, that doesn't quite work the same way. But in some sense, what users are actually contributing to you is their daily active usage. So you can take a whole MAU, and you can say, well, if a user came back 28 days that month, they're like 28 times more of a whale than somebody who just came once that month. So at Facebook, we call this L28, which is the active number of days The number of days active in the last 28. So it's roughly their contribution to DAU, to daily active users. You have all these MAU, but a lot of them don't necessarily contribute to DAU because they're not there most of the time. But some of them contribute a lot to DAU because they keep coming back. This is very typical of a distribution. You'll be in a situation where L28, most of your users probably don't come back much. But some of them do. And these guys here, that's basically— these are the guys for whom the service works. This is product market fit. You achieved product market fit with those guys. Even though you may have, whatever, 1 million users, you've really achieved product market fit with 1/10 of them, which is not bad. What this helps you do is this also helps you identify what the deltas are. So whatever it is that's different between these guys and those guys, that's something you can study. You can go dig into it, figure out what makes them different. How do we get users to be more like this and less like that. I've seen this for a lot of companies. It's interesting. At Facebook, this is very dramatic. There's a whole bunch up over here. And it's more balanced. But for the vast majority of companies, this is not the case. Even if it's a company that has a very large MAU, it's very hard— just because you have a large MAU does not mean that you have a lot of users out here who are deeply engaged in the product. Questions on that? No questions from this crowd.

Yeah.

Oh, yeah. What do I mean by that?

So if you have, like, a feature you're introducing, you do the same graph for that specific feature?

That's right. Yeah. So, I mean, like, so for instance, at something like Facebook where we have so many features glommed on together, every single feature can be analyzed. like this. We would do something like, say, just photo sharing. Even though you might be on Facebook every day, you maybe only upload a photo once a month, twice a month. From the point of view of your photos engagement, you may not be actually that deeply engaged because you're not doing much uploading. You can use this type of analysis to understand your attachment to that feature and how deeply engaged you are in that feature. You can once again create cohorts Like the first time you uploaded your photo, and did that trigger a whole bunch of photo uploading? Does that make sense?

Yeah, and can you do the same thing for the next graph? So you can tell these guys used it 28 times a month or whatever.

That's right. Yeah, exactly. It's one of these things where it applies to all the subcomponents of your product also. For you guys, it's like the thing you're going to want most is just growth, top-line growth. So most likely you'll just want to know overall. But you're absolutely right. Once you hit scale, and you have people working on this aspect of your product, that one aspect on its own should be able to stand up on its own. It should show strength in all of these metrics. It should show growth that's low churn. It should show good cohort behavior, good lifetime behavior. And it should show good deep engagement for a meaningful chunk of the users who use it. That's how you know it's really creating value. If it's just something that floats in and floats out, out not super valuable, right? People aren't using it. Yeah.