[500DISTRO] The Only Metric That Matters

So thanks for coming in. I hope everybody had enough beers that you won't pay that much attention. All of today you guys have been hearing about the psychology of growth, how to think about how to grow your products. We're all thinking about how we get something from very small to very big. And now that I'm a VC, I kind of meet a lot of people sort of in early stages. They're trying to think about, how do I talk about my product? How do I position it? And we all talk a lot about metrics and how do I understand Is my product growing and working? And I'm gonna try to sort of demystify this a little bit and sort of give you a sense of what metrics you should really focus on to get from here to there. So the first thing I like to remind everybody when you're starting a new company is literally everybody starts at zero. There is no secret sauce, there is no super advantage that everybody has. Google started with a webpage with a couple of guys at Stanford. They were like, come and search on my search engine. and they had to ask people to do it. And I even have some of the original emails. My brother was in a different group at that time and had the original emails of like, please come try our search engine. Whole Foods started as just a little store in the middle of Austin. It's now like the massive conglomerate. I wanna point out that like growth matters not just for tech businesses, but for physical businesses, for every establishment. They put together a whole market, renamed it Whole Foods, and it, you know, look at what it's become today. And then obviously even Apple. You know, we say, oh, tech companies don't advertise. These guys were advertising. This ad is from the 1970s where they're trying to get the first couple people just to buy an Apple II computer. So you gotta remember, everybody starts at zero, and yet we all have this dream. We all wanna become one of these unicorns, and we wanna become one of these great companies, but even more, like, it's not just a unicorn. We wanna become a company that makes an impact, that gives people a new way to experience the world, a new way to interact. A new way for them to behave and operate in the world. When Facebook started, it was this little site for college kids to connect. Now it's a place for everybody in the world to connect with the people who matter most to them and to stay connected. And that impact is much more than being a unicorn business with a market cap or, or even revenue, but we're really building all these companies to make an impact. So as you're thinking about all this, you really want to figure out, how do I know if what I'm doing is working? How is it making an impact? And this is where it starts to get really noisy, and you can start to get 30 different pieces of advice from 30 different people, and sometimes you'll have charts that look like the one on the left. They'll be like, everything's going great, it's up and to the right. And some days you'll look at your charts and you'll think, oh my God, something's not working, it's going down, and it's gonna crash through the floor and we'll be out of business overnight, and sometimes it never even gets started. But the reality is, only you can really figure out if it's working if you're starting to deliver meaningful value for people, and you have to remember, this comes from Fred Wilson, Every single startup and every single idea that's trying to go from zero to this massive impact on the world goes through these phases: up, down, sideways, bumpy, troughs of sorrow, wiggles of false hope. I really love this post and this thing because it really captures the journey that we're all on, and they don't always end up, up and to the right. That's— even the very, very best ones go through this horrible journey that is just ups and downs all the time. The real focus is how you figure out yourself if your own company, your own product, the thing you're trying to deliver for the world is starting to add value. The second part of all this metric stuff is you start getting dashboards and you start getting metrics and you go get Kissmetrics, which is a great product, or you get Google Analytics or Mixpanel, and you start looking at these dashboards and trying to figure out what's working. It can get so overwhelming so quickly. There's so many numbers. and so many charts, these start to get lost. And so you say, okay, instead of just looking at these dashboards, I don't know what's going on, I'll benchmark against everybody else. And everybody you talk to will have a different data point to try to sell their business. And if you sit there reading TechCrunch all the time, or talking to all your friends with companies, you're going to hear all these different numbers. Oh, I had 250 million page views in 30 days. We have a million downloads. We just topped 200,000 signups. We have 10,000 people who've tried our product. We have a Dow-Mao ratio of 50%, our D7 retention. These are all things I've heard in like the past couple of weeks from various entrepreneurs I've met. And even me, I have no idea how to place these in any context. Like, are these good numbers? Are these bad numbers? Are these numbers where it was something much smaller yesterday, it's getting bigger, and it's gonna get a lot bigger? Was this a false peak because you got featured in the App Store and a bunch of people downloaded it and nobody would ever come back? And so all of these numbers can be so much noise that I really want to encourage you to think about the one thing that matters, which is fundamentally, are people using your product? Are they coming at some frequency to touch your product, to interact with it, to add meaning to their lives? So that's such a really important kind of fundamental concept for using a product. I'm going to spend a little bit of time talking about the different ways you really start to understand what it means to use a product. So the first thing is just using your product. Like, how do you expect someone to use your product? And I'm gonna go through a couple of examples of various products that we all know and use and love today, but talk about the early days, how they figured out what it meant to really use a various product. The second is the cycle. We often forget that not every product is meant to be used every day. Everybody hears the Facebook stats of, you know, 800 million daily active users. They're like, how do I be Facebook? Not every product needs to be used every day, and it's really important when you're building something not to think about, can I be somebody else, but can I have people use it at the cadence and the cycle in their lives that really matters for me? And then the third piece is retention. If they're using it a little bit, they're starting to use it at that cadence, are they really sticky? Are they embedded in my product? Are they people that I expect to come back not just today, not just next week, but next month, 6 months from now, 3 years from now? Because fundamentally, as we're trying to go from these things that start at zero to this magical unicorn place of having this massive impact on the world, we need to get from having a few people using it today to hundreds of thousands, to millions, to tens of millions of people using it in the future. And that's fundamentally what matters. We're not connecting this growth curve that we expect to keep continuing. We're expecting this compounding behavior to get to a point where people are using your product to add value to their lives at such scale that you have a really healthy business on top of it. And that's what the whole art of building a startup is. So let's talk about kind of this sort of what does this mean for usage? So this is LinkedIn from 2005, back when I worked there. Isn't that look like 2005? The colors were awesome. But what's important to say is not many people actually visited LinkedIn back in 2005. Like even if you signed up, you hardly came. But it's important to know that we were okay with that. We didn't spend a lot of time going, how do we get more people using LinkedIn every day? Because LinkedIn had 2 fundamental value propositions: find and be found. There was actually a very limited number of people who were what I'd call finders on LinkedIn. They were people who were looking for someone to hire. They were looking for a business development lead. They were looking for a connection into a certain company. Man, those people would come a couple times a week. Recruiters were starting to come every single day. to LinkedIn, but that was a pretty small number even relative to how many people we had signed up. The second metric though that we really cared about was people who want to be found. And what we realized with LinkedIn was we needed to get really, really, really big so that everybody who wanted to be found was on LinkedIn so that they could be found. And what we cared about wasn't that they were active on LinkedIn, but that they would actually respond when they got a message. And so we built our entire growth strategy, our entire way of thinking about how to make LinkedIn bigger to get more people into the network so that they were willing to be found. We wanted to find out where they worked, what their title was at that company, wanted to get them connected to a few colleagues that they knew and had worked with in the past because that would help the network grow denser. And then if they never came back to the site except maybe to accept a couple connections, that was totally fine. And then we wanted the searchers to come a little bit more often to actually search. So we started doing these analyses to figure out how many people were coming to search, how many people were coming to be, who were sort of available to be found, and that was really our fundamental metric that we tracked every day versus daily active users. There was this controversy even more recently when LinkedIn announced 200 and then 300 million users on the network, and everybody said that's a vanity metric. And I pushed back and I said, it's not a vanity metric that LinkedIn has 300 million people. It has a database of 300 million people who can be found even if they're not on the site every day. You can't compare it to Facebook, And if you look at the company, it makes so much money off the people who are doing the finding and doing the searching and recruiting and other things that it's a very, very healthy business even with totally different metrics. So then I like to think about Yelp. And Yelp is something that we start to think about as a more active behavior service. The fundamental behavior for Yelp is either write reviews or find local businesses and services. And so if you think about Yelp, is this a daily active user type of behavior? How many people in the room look for a business on Yelp every single day? A couple people raised their hands. How many people look for a business, whether they use Yelp or Foursquare or something else, a couple times a week? That's like most people in the room. Again, it's not a daily active user metric, but if you're sitting there at Yelp, you start to care not do people land on my review pages from Google, because that means they're searching on Google and just ending up on Yelp, but are they actually coming to Yelp either my website or my mobile app now to search a couple times a week. And I used to have conversations with them back kind of in those early days of how they were thinking about their metrics, and it was really interesting that they weren't so focused, again, on daily active users, but the number of searches per day, the number of users who were searching per day, who were starting to come direct to Yelp, even versus total page views, because total page views isn't that valuable for them. And then you think about reviewers, they actually cared about content. They wanted to know how many businesses were reviewed, and how many people were actually reviewing businesses because they knew they needed to amass enough content to make it interesting for everybody, but they also certainly didn't expect every searcher or every visitor to Yelp to be somebody who left content. So again, the metrics don't have to be the same as everybody else, and they're very different than what we talked about for LinkedIn. Now Facebook is obviously the big behemoth in sort of daily active user metrics, and they've kind of redefined the category. Before Facebook, we never even used to talk about daily active users. Websites generally talked about page views and unique visitors and they were all these kind of abstract terms and they were often uncookie people who weren't logged in who you didn't even know. Facebook came out and said we're about staying connected with friends and we expect people who care about their friends to want to stay connected pretty much every day, maybe even more than once a day because if you actually care about your friends you're probably like that unless you're really lonely or want to go live in a cabin somewhere. So what Facebook started realizing was that people cared about their friends so much they were coming every day and they were fully logged in. And we were able to really track who logged in and saw their newsfeed every single day. And that became the fundamental way that Facebook was going to make money, was people coming to look at their newsfeed and then seeing ads either in their newsfeed or in the other pages that they browsed on the site. So they're an advertising business. And in advertising business, daily traffic, sheer volume of traffic is where that really starts to matter. which is very different than for Yelp or LinkedIn. And then lastly, think about a company like Expedia. Travel is a completely different category than all of these. I mean, how many people book and plan trips every day or even every week or often even every month? Not that many. But Expedia, they want to be the best site to plan a great trip and have people come and use them for that. So what they think about is, how do we make sure that every time you want to plan a trip, You think Expedia, you come to Expedia, you have a good experience, and they want you to be able to do that at least a couple times a year. The trick there is it's much harder to stay top of mind when you're something that you don't use every day or every week, so they actually end up, in the early days, they were spending a lot more on marketing and building their brand and building up their base so that they, and building up partnerships and integrations everywhere into places that people did use more frequently. So you think about the metric that matters for Expedia, it's how many people search per day and transact, And how likely is it that we're the first place they go when they do want to search, even if it's not till 3 or 4 months from now? So if you think about all these metrics, and you then start to translate, how should I think about my own product? How are people actually using my product? What are they really doing? Are they using it often enough? You start to realize there's a bunch of people who try your product, who might download it, install it, use it a little bit. But how can you really figure out if they are really, really using your product? Are they coming every day, week, or month to use your product? Are they remembering it in such a way that you really expect them to come back 3 months later, 6 months later, 2 years later? Because until you can start to believe that, it's hard to believe that you have a compounding business that will scale. And so I like to call this really, really using. And at Twitter, we came up with a name for this group of users, and we called them core users. And for the people who were core users, We started to consider them as they had Twitter as top of mind. These are people who remembered us, and whenever they wanted to know what's going on in the world, they came directly to Twitter. And so you wanna think about, are there people who are using your product who actually remember just to come use your product and engage with it? The second thing is, are they recurring? Are they coming back over and over again on the cycle that you expect? If you're a movie site like Flixster, I expect people to come back kinda every week, maybe every 2 weeks to check what movies are out and what movies I should watch this weekend. But don't again expect them to come every day. And then the last thing is, are they referring users? Do they feel so engaged and sort of so embedded in this product in their lives that they start sharing this with others? And Dave talks a lot about this in the Metrics for Pirates R. And I think it's really important to think about these are your fundamental, most important users, but you need to know who they are, you need to know why they're like that, you need to figure out how to get more people like that. So I'm gonna walk through a little bit of how you can take all this knowledge and all this learning and all this looking at your metrics and start to figure out who are your core users. And so this is a really hard thing, is to separate from all the users of your products and all the page views and DAU metrics and MAU metrics who's really sticky. And so this is an exercise we did at Twitter that turned out to be really, really meaningful for us to figure out who was really using Twitter and then how we could get more people to look like them. And what we did is we did this massive matrix. It was a 28 by 28 matrix. There's a trick in metrics is we tend to say month over month, but tend to normalize for 28 days so you eliminate the variance of weekends for months that start on Fridays and months that start on Tuesdays and everything else. On the bottom axis, we basically counted all of our users who had visited Twitter in the past 28 days, how many different days they visited. And we did it by days visited saying if you used a product 4 times in a day, then you went to sleep and then never used it for the rest of the month, That actually counts less than the guy who woke up, used it one day, went to sleep, used it a couple days later, and kind of kept remembering to use it on multiple different days. Because the natural human cycle, day and week, you kind of want to make sure you're actually a pattern in their lives versus lots of bursty usage. So on the bottom axis, we counted everybody who had used it n days in one month, or one 28-day period. And then we looked at the percentage of people who'd used it that many days in the first month, How many of those days did they use it in the second month? And this matrix was fairly complex for us to run at first, and we were doing a bunch of different analyses, and so we actually got it to this, and then this kind of helped us see the light. And the light that we saw was how many times do you use the product in the first month in order to use it at least that many times in the second month? And for Twitter, it surfaced to us that if you visited Twitter at least 7 times in one month, you were over 90% likely to visit it 7 or more times in the second month. And that turned out to be really important for us. That meant basically twice a week you were remembering to log into Twitter or open up Twitter. And that meant now it's a habit in your life. And of course we want you to do it every day, we want you to do it 10 times a day, and like what's going on in the world, and you're always checking Twitter, which is like how I am right now. But at the beginning we thought if you even get to 2 days a week or sort of the 7 days in a month, you're now pretty sticky. So if we know you're pretty sticky, then we can start to figure out what's different about these people. Why are they getting sticky where everybody else who signs up doesn't really get it and doesn't really use it? And so we started doing this analysis. I called that group core users. I called the group who came less than 7 times casual users. I called the group who came 0 times cold users, or comatose is what we used inside the company, but we thought cold was nicer. And we started to build— I called this the kite diagram. which is how do people activate? So when someone signs up brand new, how likely is it in their first month they now, in the second month, they start to look like a core user? How many of them look like a casual user? How many of them immediately go to cold, which means we've pretty much lost them? And this analysis helped us really understand where we needed to focus on the product. Did we need to get more people in the top of the funnel, or do we just need to fix our activation so we got more people? Only 20% of our users were getting into a sticky place where we expected them to come back month over month over month. And that was a real problem for us. That's over 80% of the people we were basically sort of losing out and they weren't getting hooked on Twitter. And so we spent a lot of time trying to understand what's special about that 20% and how can we get everybody else to look at them. So the next analysis we did was deeply on our data. We went through our data and we started peeling it apart and we were like, what do those people do that other people don't do? And we found 2 big traits that sort of emerged. It was kind of a massive correlation analysis I think there's much simpler ways to do it than what we did, and probably on much smaller datasets it might be easier to parse as well. But the first thing we found was if you followed more than 30 accounts on Twitter, the likelihood that you were sticky was great. Now, it's easy to say, well, the people who followed 30 were more committed, so of course they were more likely to get to that point, and that's partially true. And in every company I've been at, this sort of correlation versus causation conversation comes up. And it becomes really, really messy and a big debate and very, very heated. And I say throw it out. If you can tell that those people are much more likely, then I'll bet you you can do things in your product to help more people get to that state. And if they do, they're more likely to be sticky. And if it turns out your theory's totally wrong, that following more than 30 accounts doesn't make you more likely to use Twitter, then you've got to go redo the analysis and find something else. The second thing we found was very unique to Twitter. And it was very specific for our network. And so you're going to find different analyses for different networks. But at Twitter, it was if your account is such that 1/3 of who you follow follows you back and 2/3 of who you follow doesn't follow you back, that was actually the healthiest state you could be in. When you had more people follow you back than you followed, like over 50%, then it was sort of a social product and you weren't necessarily seeing the difference between Twitter and Facebook, and you were sometimes overwhelmed by your feed or you were doing lots of random followbacks and your feed wasn't very good. When it was way under 1/3, like 10% or less of the people you followed followed you back, then unless you were like a celebrity and had lots and lots of followers, then you were in this state where Twitter wasn't very social. None of your friends were there. Sure, you could get a bunch of interesting news, but you didn't have any social interaction or experience. So for us, we found this was a very important point, and so we started building a lot of features to help people figure out who to follow, help them understand why to follow, and help them get their friends enough connected and a few of their friends following them back that it would actually be sticky and interesting. But again, it's very different for every single product, but that's what we learned for Twitter. But the most important thing is we didn't just look at our data, we started calling people. And we learned in people's own words what mattered. There was this great research done at Twitter where we ended up calling up users who had a very specific pattern. They'd signed up, over a month later they had gone totally dormant and weren't using the product at all, and then some period after that they picked it up again and got going on Twitter, and they started going every— started using it actively again. And so we found those users who had signed up, gone dormant, and come back, and we called them and said, what happened? Can you tell us why you signed up in the first place? Then what didn't work for you? Why'd you bail? And what caused you to come back and try it again? and what worked this time. And again, for Twitter, it was a very interesting set of information. People originally signed up 'cause they'd heard of it, and they thought it was this great place. This is, again, data from 2008, 2009 era. They'd heard of Twitter, and they thought it was a place to go broadcast and shout to the world and tell everybody what was going on. And their friend was doing that and making success for their business and said, oh my gosh, you need to go sign up for Twitter. So they'd come and sign up. They'd start following random people, 'cause that's what the user flow did at the time off the suggested user list. And then they would be like, I have no idea what's going on here, I have nothing to say, I'm out. And then some period later, everybody had a different story that was something like, my reverend said he's using Twitter and I can get updates on the church and updates from kind of biblical phrases and everything that he's doing. Or, I heard that Shaquille O'Neal was on Twitter and he's super funny and he tweets about where he is and if I follow him on Twitter, I'll get a bunch of interesting information. My favorite band was on Twitter or something else. They all heard there was someone to go follow there that was useful. When they came back in that mindset, they got really, really hooked on Twitter. So combining the data analysis plus what we learned from everybody, we went back and said, why don't we rebuild the product so it's about following? So we tell everybody Twitter's the place where you find out what's going on in the world and understand what's happening. And if we get people into that mindset first, they're more likely to have a much longer and more successful experience on Twitter. So for whatever you're doing with your own product, we always forget how important it is just to pick up the phone and call these people and cherish them. These users who got your product at the beginning, they signed up, they read about it in an article, they heard about it from a friend, they heard about it from one of your friends. They've already figured out your product in a way that everybody else hasn't. You need to treat these people like gold. You need to figure out how to really learn from them, and figure out how to make everybody else like them. Forget the overall daily active users, monthly active users, page views, all the other metrics. Find these people who religiously use your product and are putting it in their lives, and then your next goal is to figure out how to make everybody else look and work like them. Okay, so you got some users, you got some core users, you figured out some people who get your product and love it. Now what? So the first thing I say is you probably have a couple different problems. One is, Not enough people sign up. Look, we all want more people to try our products. That would be great. But if you actually have these core users, you found them originally— by the way, how do you find them in the first place? You hustle. You're out there talking to the press. You're talking to people. You're asking them to try your product. You're doing everything you possibly can to get the first couple people to try it and see if it sticks. But once you've got a few of them on it, you've got to clone them. And you have to let these core users help you clone them. And the way to clone users is what we call this magical virality. But if you think about virality, it's not always the same, and it actually works very, very differently in very different constructs. The fundamental core of virality is what I call word-of-mouth virality, where I basically am so excited about a product that I tell somebody else about it, and I say, you should use this. I say, I had this amazing experience on PayPal, or I had this amazing experience with my new Galaxy Samsung phone. I'm trying to read all the sponsors back here. I had this great experience with Amazon Web Services. My product works so much better now. And once someone's told somebody else about it, all of a sudden they get it, and they want to come use it, and they understand why, and understand why it's valuable to them. And often you can actually incentivize this too. You can say, hey, please go tell people about this, and if you tweet about it, we'll give you extra money or extra discount or whatever. The second one is what is really important. I think a lot of people don't think about it this way. It's what I call demonstration virality. And this is really where, how do you take what somebody's doing in your product and help them share it with other people? Not force them, but like Instagram, the reason that most people I know signed up for Instagram is 'cause they kept seeing these amazing pictures off somebody's phone on Facebook or Twitter. It wasn't that anybody ever said, you should use Instagram, or it wasn't that anybody said, please come join me on Instagram. It was just that I kept seeing my friends take these amazing pictures on Instagram, and I was like, how do I do that? And so you really want these core users who already get your product and use it, help them share it out in the world in just very natural ways. If they're making something great, help them surface it to their friends through email, Facebook, Twitter. If there's content happening on your site, make sure it goes out through whatever channel you can. So you really have to help everybody demonstrate that. And then the third way you wanna think about it is what I call infectious virality, and this is what we're most used to when it comes to virality, which is sort of invitations. Please come join me on this product. I'd like to message you, but I can't message you on Snapchat if you don't have Snapchat, and otherwise you'll never get these pictures. And so that's kind of this infectious thing, and often you'll incentivize this too. You'll be like, hey, please come on Dropbox. I'll save money and you'll save money, and we'll both be on this network together. But you really wanna think about which one applies to you. Instagram would've never worked through sort of this infectious come join me on Instagram, but it completely worked through demonstration. Google spread almost all through word of mouth back in the early days. It was like the best search engine, it's Google. What, you're not using Google yet? It's incredible. And so build a great product and then figure out these users who already love it to spread it. Then the second thing that I like to talk about a lot is now you might be somewhat successful at generating press or even getting the Apple App Store to feature you or having some people start to tell other people about your product. So they do come and check it out and they just have this big leaky bucket where nobody sticks around. And here's where you really want to think about this moment of time when somebody is on your product for the very first time where they're actually willing to learn. They're willing to trust you with a few minutes of their attention to figure out if it's useful. I think most people completely skip over this when they're building a great product, skip over this path from not knowing to actually knowing your product. And so there's kind of, I say you really need to onboard your users from the very beginning all the way to becoming core users. And a couple things I like to think about. Everybody thinks about onboarding as get them set up and signed up as fast as possible. I like to think of it as a learn flow. Spend your time teaching people step by step the product. Explain to them the key things that will matter. Make them go through steps that help them cognitively start to understand it. Instead of saying, hey, we'll send you daily emails about the news, say, what time would you like to receive a daily email about the news? And the person actually has to pick what time. The moment they've had to make a little decision, they're gonna understand at some time I'm gonna get daily news from this site. And then if the next person says, what was that product? They'll be like, it's this thing that sends you a daily email. Versus if you just tell them, Hey, check this box and we'll send you a daily newsletter. No one will actually remember or pay attention. Onboarding flows can be long. Don't rush them. I'm a big believer that the longer flows actually take the users who are interested much further into actually being sticky. The second thing I like to think about is what I call welcome week. The first moment is this really, really important time where you can get them to sign up, but if they don't get it on the first day, you still have a couple more days to nudge them, remind them to come back and keep going in the product. You can send them emails, you can send them push notifications. If after a few days the person's really not using your product, you're just gonna get annoying if you keep sending them push notifications. They're gonna unsubscribe, they're gonna do a bunch of other stuff. But don't think of onboarding as just the first moment. Do extend the time period a little bit further. And then the last thing I like to think about is social pressure. So the moment somebody signed up is this great moment for all their friends who are actually on the product too and are your core sticky engaged users to start interacting with them. Frontback did this feature, did this notification that said, Dave just posted his first Frontback. Check it out and give him a heart. That was like this great idea of like, oh, I'll actually go and heart his thing because I was heavily using the app. And he would then go, oh my God, I post this thing and everybody loves it. What a great community. And it really helps onboard them. Don't underestimate the power of social pressure when somebody new signs up. Anyway, as you're all thinking about building your own independent startups, the one thing I hope you'll take away is don't get confused by the metrics and the conversion rates of everybody else. Think really about the product, the meaning your product specifically adds to their lives, and focus all your metrics on how many people are really using your product. And then when you go pitch investors, you go pitch the press, everybody else, you get to tell these great stories of here's how I understand how everybody's using my product, here's what I want more people to do, here's where I want to go, and that gets everybody believing that this is actually gonna compound and get much further than if you get into page views, DAUMAO, everything else. So thanks a lot, I hope that was— Thanks.