Spenser Skates: Using Analytics to Build Better Products

We are going to start with the CEO of Amplitude, Spenser Skates. Now, if you haven't had a chance to meet Spenser, he's probably one of the most promising young CEOs in Silicon Valley today. He has a very enviable resume already. He's gone from like winning MIT Battlecode not once, but twice, starting a voice recognition company, working on Wall Street, and then starting analytics company, first leading sales for what turns out to be one of the fastest growing analytics startups in the valley in the last few years, and now just focusing on the vision of our company as the CEO. When I first met Spenser, I was really blown away by all that he had accomplished and his humility. And I was digging around to understand what his secret was. And turns out it's a couple of things. It's a growth mindset, and it's a diet solely limited to chicken tikka masala. So without further ado, please welcome Spenser Skates.

Thank you. Thank you, Sandhya. I hear that here in— so this is actually my first time in Seattle. I hear that you all are known more for teriyaki chicken than Indian food. So if you have any good recommendations on that, please let me know after. I'm definitely interested in taking a look. So anyway, welcome again to the Product Analytics Summit. I'm super excited to be here today. So as Sandhya said, I founded Amplitude 5 years ago. We wanted to help companies build better products, and we didn't see any analytics company that was thinking about that at that time. Over the course of that, of the last 5 years, we've gotten the great opportunity to work with over 15,000 different products. And through that, we've been tracking 1, or I think it's 1.5 trillion user actions. And what makes me really excited today is this is the first this is the first time that we are talking about product analytics publicly. We've always talked about, you know, hey, we're doing analytics, we're doing, you know, growth, but we've never really crystallized it in this form, and so this is the first time that we're doing this today, and that's what makes me excited to share this all with you. And I wanted to kick us off today just reminding us why we're all here as product folks. Actually, just a quick poll of the room. How many folks here are in product? Okay, wonderful. How many folks are here from engineering? Okay, a few more folks. And then how many folks here are from marketing or something else outside of those? Okay, wonderful. For us, product analytics comes back at the end of the day to building a better product. You know, we're not doing product analytics just for the sake of it. It comes back to this. And so I wanted to share 3 stories that really helped, that really resonated with me about this point that, you know, hey, here's how you can actually use product analytics to build better products. So let's dive in. So the first story starts actually way before Amplitude. So as Sonia mentioned, before Amplitude, I co-founded a startup called Sonolight Text by Voice. It wasn't ultimately very successful. How many folks here have actually heard of this? Alright, Sundia. Sundia's heard of it. So not a very successful company. Definitely on to better things now. But for those of you who don't know, so Sonolight Text by Voice was a voice recognition app that allowed you to safely send and receive text messages while driving through talking to your phone. And this was the first startup that me and my co-founder started coming right out of school. And we were really excited because it was like, wow, okay, there's this cool new interaction technology, voice. We think we could make a better interface and a better product. It was kind of like a version of Siri except for Android phones. And so, as part of doing that, we went through the startup program called Y Combinator. Y Combinator is a 3-month program, and at the end of it, we had this demo day where you'd pitch to investors, try to raise money, and kind of get to the next stage as a company. Now, at demo day, we had one of, like, the most awesome demos. I had— I did this thing where I had my phone in my jacket pocket, and I, like, had a conversation with it on stage. And without touching any buttons or without pressing anything, and I successfully used it to send and receive a couple text messages, and people were like, "Wow, this is the coolest technology ever." We actually were the most covered startup coming out of that Demo Day out of 60 different startups. This is some of the press that we got at the time. So from, you know, Forbes, TechCrunch, CNET. —so from pretty noticeable outlets. So we were really excited. We were on top of the world. We had a lot of users who had heard about us for the first time downloading and wanting to try out the technology. Here's a graph of our weekly active users. You know, thousands of folks who were coming onto the platform. And so we were really, really excited. We were like, man, we're killing it. Now, what was super interesting is a few months after that, Here is what our weekly active user graph looked like. Not, right, not the up and to the right that you saw before. And so, we asked around our user base. We're like, hey, you know, what's going on? Why are you not using the product anymore? And what we discovered is we just had a really, really big retention problem. So, a lot of people, when they first heard about the company, when they first heard about the product, they were excited. They wanted to try it out. They wanted to use the cool new technology, but it wasn't a compelling enough experience for them to continue returning. So even though we had more and more people signing up, retaining your users is kind of step number one to building a good product, and we hadn't successfully done that. And so we wanted to dig into that. And we asked ourselves, OK, well, why are users not retaining? So we looked at some of the tools that were out there at the time. And a lot of them were really good at answering, OK, how much are users retaining? Let's say you have 100 users starting. After, you know, 7 or 30 days, how many of them will stick around? But none of them were really good at answering the why behind it. Why are users retaining, or why are users not retaining? And that's really what we wanted to get at as part of Sonalight. And so, like any two engineers— this is me and my co-founder way back then with a good deal of hubris— we ended up just deciding to build it ourselves. How hard could it be to build a product analytics infrastructure stack? Turns out, actually pretty hard. We ended up spending more time building the infrastructure stack than we did actually building the application itself. But we really, really wanted to answer this question around retention, so we're like, okay, maybe this is worth it. And after a few months of building it, one of the first questions that we wanted to answer was, how does the voice recognition, the accuracy predict long-term retention? Because That to us, that was kind of the number one complaint that we heard from users. We knew it kind of anecdotally. Maybe the technology is not good enough. But we wanted to actually conclusively figure out, is this the actual thing blocking users from using it? So what we did was we looked at users who signed up for the application. And we looked at whether they had a successful voice recognition event the first time they used the app. So we grouped them into two buckets, users that had the successful event users that didn't have the successful event. Now, as it turns out, that mattered a ton. Users who had a successful event the first time they used it were twice as likely to retain long-term than users who didn't. And so it was like, wow, OK, that is super obvious to— that's like, now it's really still clear to us that this is the biggest blocker to us being successful as a product. If we can solve this, we can double our retention. That's awesome. Now, unfortunately, being just 2 engineers at the time, the rate at which we could improve existing voice recognition algorithms was kind of a losing battle. If you ever, if you ever all decide to start a company one day, my advice, never ever go into voice recognition. It's way too hard. And that was, you know, that was my big— we spent 9 months in this and that was one of my big takeaways. And so, you know, this was our, this was our graph over that time and we were like, okay, That was a failure of, we didn't end up succeeding in building what we wanted to, but what we did succeed in was understanding the why behind it. I can't tell you how many product folks or teams I talk to where their answer to, hey, a retention problem is like, oh, we're just gonna send push notifications to users, or we're gonna email them. And it's like, no, the reason your users are not using the product is 'cause it's not a good product, not because you're not spamming them enough. And so understanding that, For us, that was a big win in understanding product and product analytics and how you could actually use that to get to the why behind why users are having a good or bad experience within your product. And so as a result of that experience, we actually started Amplitude. And over that time, we've gotten to work with a lot of products, much, much more success— all of them much more successful than Sonalight. So I wanted— so that brings me to my second story that I wanted to share. With all of you today, which is the story of one of those products, Calm. So how many folks here have heard of the Calm application before? All right, so for those of you who don't know, Calm is a meditation application. So it helps you— the idea is it will bring you through a series of meditation— it's a lifestyle app that will bring you through a series of meditation exercises in order for you to help achieve more mindfulness. So do things like stretching and different breathing exercises over a 10 to 15 minute period to kind of help you meditate and walk you through that. They have nice background music. And so they wanted to answer actually the same questions that we did with Sonolight. They wanted to understand why users retained and why they didn't. And so they ended up looking at the actions that their users took in the application.. And looking at those, they discovered something super interesting. They discovered there was a group of users that retained 3 times higher than the average user base. So, if you look at this here, so this is a graph of their retention. So the blue line on the bottom, that's their average retention across their entire user base. So, you know, the weekly retention you're looking at less than 10, around 10, maybe a little less, 10%. The red line is a group of users who did one particular action in the application. And if they did that action, they have 3 times higher retention. So that's even bigger than the insight that we discovered around voice recognition, where it was like 2 times higher. This is 3 times higher. Wow, that's pretty impactful. Now the interesting thing is digging further into this event, they found that only 1% of their users actually did this action. So the blue line on top is all their users, and the red line is users who did this particular action. So while this action was super engaging to the small group of users that did it, most of their user base, 99% of the user base, was not actually doing this action. So before I show you what the action is, does anyone have a guess? Raise your hand. Does anyone have a guess as to what this— so it's a meditation app. Does anyone have a guess to what this action that causes you to be 3 times more likely to retain is? Yes? [Speaker:AUDIENCE] Like one meditation within the first 3 to 7 days. [Speaker:JONAH BERGER] One meditation within the first 3 to 7 days. Good guess. OK, anyone else? Yes? [Speaker:AUDIENCE] Completing a full meditation. [Speaker:JONAH BERGER] Completing a full meditation. Good. [Speaker:AUDIENCE] Daily reminder. [Speaker:JONAH BERGER] Daily reminder. Good. [Speaker:AUDIENCE] Headphones. [Speaker:JONAH BERGER] What's that? [Speaker:AUDIENCE] Headphones. [Speaker:JONAH BERGER] Using headphones to do a meditation. OK, good guess. So here, let me— I was going to tell you, but let me just show you what the action is. So this is the home screen of the Calm application. This is the first thing you see when you log in. And here is how you do this one particular action. So you first click on the profile button on the bottom left. That brings you to this menu. Scroll down. You hit the settings. You scroll up. You hit mindfulness reminder, and you set a reminder. Who said, was it you said that setting a reminder? Exactly, it was setting a reminder to meditate at a later time. Now as you saw, most users, it's not, that's not gonna be obvious how to get there, to that screen. And so only 1% of users were getting there. But if they did get there, they were much more likely to be retained. Okay, so what did that mean? So Calm wanted to try an experiment. So how can we make this more prominent? How can we make more users engage with this reminder? And so what they did was they set a prompt when you first open the application to set a reminder for yourself at a later date. So this is now kind of the new opening screen for Calm. It just pops you straight into the mindfulness reminder section and says, do you want to set a reminder? Now, the great thing was that they went from 1% of their user base setting a mindfulness reminder to 40% of their user base, so a 40x difference overnight. And on top of that, the number of, those users also retained 3 times higher. They were much, much more engaged with the product. And the crazy thing for them is they, com, had never really thought about this as an important component of their product. And the takeaway for me was that as a product, Calm, like, they're spending 99% of their time focused on, okay, let's make meditation sessions good, but a huge part of their product market fit and their value that they deliver to customers was just a simple thing about reminding you to meditate at the right time. And for me, that, I mean, it made sense in retrospect. You're like, yeah, well, maybe someone when they first download the app doesn't wanna meditate right away. Maybe they wanna do it later. Maybe they're thinking about it and they wanna set a reminder for themselves to, go through these exercises later. But it wasn't obvious to them. They had spent 3 years in development before this focused on just making the best meditation experience possible, and they hadn't even— it wasn't even in their Android app. It hadn't even occurred to them. And so as a result of making that change, their retention doubled pretty much overnight for the set of new users who used the application. So that was massive for them. And that enabled them to be much more successful because they understood what a core part of their product's value was, and they used that to help them build a better product. And so that really resonated with me is that a lot of times your product's value may not be, and as product folks, it may not be where you first think it is. All right, so that brings me to the third story that I wanted to share with all of you today. So this is the story of Yelp. So, this is the story of Yelp from the perspective of an end user. So we didn't actually work with the product analytics team at Yelp to do this, but this is my, just my personal experience as an end user of the Yelp application. So, a super common, like, or the most common use case for me within Yelp is finding restaurants that are nearby me that are open now so I can go find a place to eat. Pretty simple. That's like, you know, just one of the basic things that you do within the Yelp application. So when I first started using Yelp, here is how you did that. So this was, I think, 3 or 4 years back. So this is kind of the home screen of Yelp. So what you'd do to find a restaurant that was open, that was nearby you and open now, you'd hit that restaurant button. You'd hit that filter box. You'd scroll down this menu, you'd hit the "Open Now" checkbox, and then you'd redo a search. So that's something like 4 or 5 clicks. You have to do 2 searches, go through multiple screens to get there. Now, the really cool thing for me as an end user was Yelp made a very big— or a small change, but one that had a very big impact on me a year or so back. And here is what the change was, and here is how that experience changes. So you now hit the restaurant's menu, you hit the restaurant search bar, and instead of having to go to the filter menu, you just hit open now at the top. And that was a, now, the crazy thing is, that was a big difference for me as an end user of the product. I now could do the same thing much, much more, it took less than half the effort. You know, you're only making 2 clicks. You don't have to go through multiple screens. You don't have to do multiple searches. You're just doing it once. And that made the application much more engaging and sticky to me. I would now, end up using it more to search. It sounds crazy, 'cause you're like, well, if you wanna eat, aren't you gonna open up the app and search no matter how easy or hard it is? But for someone like me who was very lazy about it, it was like, yeah, it matters, 'cause otherwise I would have defaulted to just going to where I already knew or asking a friend or something else. That little change was actually really impactful for me as an end user. Looking at, thinking about, OK, well, how did they come up with that change? My guess at the time, after seeing that, was that they had some product manager who spent a bunch of time looking at the most common paths and flows through the application. So looking at what the most common use cases through Yelp are. And probably what they found is that hitting filter and that going down to open now was a super common one. You also see price up there in that screen. That was probably a very common one as well. And then they took that, and then they just made a shortcut for that to make the experience much easier. Now, the really cool thing about this story was very recently, we actually got a chance to talk with the Yelp team. And the interesting takeaway when I talked with them was they said that this change was the single most impactful change they had made to the engagement and retention of their application from a product perspective in the last few years. And that was kind of mind-blowing for me, because it's like, OK, not only is this impactful to me as an end user, I thought the app was just kind of evolving to suit my needs. But no, this was true across their entire customer base. Their entire customer base ended up being more engaged and more retained because they made this simple, what looks like a pretty simple change, but was actually super impactful. The other cool thing in talking to them was just hearing about how painful it was for them to actually make the decision to do something. Like, it looks like a pretty simple change. It looks like, oh yeah, just look at the most common paths. That analysis is actually, you know, it doesn't matter if you're Yelp or you're Facebook, that analysis is like not trivial to do. That kind of brings me to where we are today and what we're here to talk about today, which is How do you use product analytics to make a better product? 'Cause ultimately we want any team out there, all of you here, all product teams everywhere to be able to use product analytics and actually leverage it, not just if you have a massive team at Yelp or Facebook, Airbnb, and Uber. We wanna make no matter who you are, give you the ability to actually take product analytics and to use that to build a better product. That's a big— that's the— that's literally the reason why we started Amplitude, and that's the reason why we're here today to share best practices around all these things. So hopefully those 3 stories kind of can help all of you understand what the potential for using product analytics— right, we're not here to use product analytics just for the sake of having good data and to be data-driven, this, that, and the other thing. Ultimately, it really only matters if we can translate that into building a better product. So hopefully the 3 stories that I shared with you today gave you insight as to how that actually can happen in practice. Cool. So hopefully that was helpful. Thank you all. I really appreciate it.

We have a few minutes for Q&A. I can hand you this mic. Just raise your hand if you have a question.

So with the example you gave, which I thought was great, in a way— and I wanted to get your opinion on this— it's almost like people just opted in to get spam, right?

I mean, that's the— On the reminder.

On the reminder, right. So I'm opting in to actually get a reminder. What I was just wondering is, in your experience and having seen probably lots of applications at this point in time, what percentage of cases that drive strong retention fall into that category of opting into a reminder versus a true, I'd say, piece of functionality in the application that is driving the retention?

[Speaker:JOSHUA MARANTZ] So that's a really good question. It all comes— and this is, it depends. So it all depends on the context of the application. What we've found is for lifestyle applications, so things like meditation, exercise, fitness applications, reminders are actually super important because —people are actively trying to develop a habit to use those things. They want the application to push themselves to be better. Whereas for something like, say, a voice recognition application or like Yelp, those are not things that you as a user are actively trying to build a habit around. Those are things that like where the idea to use it comes up at a particular point in time. So for Sonalight, it's like, hey, when you want to send and receive a text through voice. Or for Yelp, it's when you want to look at restaurants nearby. What I'd say is what makes— reminders are specifically, like, really, really good for lifestyle applications where the user is actively trying to build a habit. The other key piece of it is the user opted in. That's a very, very big difference from, you know, just sending them stuff organically or just sending them stuff without any context. That context makes all the difference. We've seen apps that actually, if you're not careful about the channels, the net, the overall long-term retention can actually go down as a result of using them. Like any individual notification, you'll see the retention go up, but if you actually look at it in aggregate, they can tend to go down if you're not careful about how you deploy them. So I'd say the key thing is matching it to the value proposition of your application and running experiments like this, seeing, okay, well, Are users that have a notification, are they actually more long-term engaged and retained than kind of the default user? So competing products, they usually have some kind of notification tool. Do you plan to do that, or do you plan to just focus on building a better product? Absolutely. Really good question. For us, it really comes back to using product analytics to build a better product. That's what we're like 100% dialed in on and focused as a company. One of my frustrations with using some of the earlier tools that I had up on the screen was that they were trying to be like kind of all things to all people. And as a result, they lost focus on making, helping you use product analytics to build a better product. And so they try to be this suite of products that was like, oh, we do push, we do A/B testing, we do, you know, We do, you know, yadda yadda yadda. Like, all these set of things. And they ended up, the analytics experience ended up suffering and really made it difficult. Like, for me, it's like, we're in the Stone Age of analytics. It's like, analytics is so freaking hard to use. And I say this as like a CEO of an analytics company. It's like, no one, like, it's rare you find a person that's like, oh man, I'm super excited to get up in the morning and like, use my analytics product. It's like people are excited to build better products, not to use tools like analytics. And so we want to get to the point where I'd probably— the best analogy I'd have is probably comparing it to search engines. When you want to look up something about the world, you just go on Google or Bing, or for the Microsoft folks here, and just use that. And you don't even think of it as a step in your process. It's invisible to you. It's not like you have, oh, use search engine. As a thing. And we ultimately want to get product analytics to that point, but we have a long way to go. And so that's why we're exclusively focused on product analytics.

Hi.

I was curious if you can share for your own company, Amplitude, how you guys continue to foster experimentation to retain your own users and grow your user base. Now that's a really good question. So what I'd say, So there's, like, a bunch of things that we do. So there's kind of a few things wrapped up in there. So one is, like, how do we foster experimentation? And then two is, like, how do we think of growing the user base? On the first one, that's something that is, like, super core to Arc. Just engineer— that's just an engineering culture thing. It's like, OK, we're always pushing stuff out. We have this thing, for example, called Amplitude Labs, where you can write a feature that you have to explicitly enable as an end user. And that doesn't get pushed to all users automatically. That allows us to kind of experiment with features really fast. And then in terms of how we think about it for growth, I think we do two things that we invest, like, just tons of time on. Like, the product team, like Sandhya, you know, you'll hear from Justin in a little bit, what they spend the most time on is understanding how users engage with Amplitude. Like, that's like 70% of their job in my mind. It's like not to come up with solutions, not to write specs, not to— it's just understanding how users are using Amplitude. And so they do that in two ways. So one, One is product analytics. We look and we use Amplitude a ton internally to understand, okay, what is it that causes— what are the accounts that have success in Amplitude have in common? What are the accounts that don't have success in Amplitude have in common? We're looking at individual user profiles to see, okay, well, what are the flows through the app that they're trying to do? What are they trying to accomplish? And all of that. We look, for example, our metric that we try to drive is what we call queries. So we're trying to just get people to query on the platform, and we're looking at that as a metric. The other thing we do, and I will say product analytics is not a substitute for this, is just talking to the users. So we have something like 300 enterprise customers, and we're just always on the road. A lot of the product team is here today specifically to talk to folks here in Seattle about their usage of the product. So I'd say it's a combination. The product team at Amplitude spends like 60% or 70% of their time between those two things, just really, really understanding the user. Because a good understanding of that will allow you to kind of fill in everything. Like, once you understand the user, their problem, the constraints around that, that will allow you to fill in everything else and kind of be like, oh yeah, well, it's kind of obvious we should do this as the next step. So what's the story around user streams that you took out in v0? Did you see any analytics impacts, product impact out of it? I actually got to defer that one to Sundya.

Is that it? We've actually seen a massive increase in engagement when we moved from our previous design to the Amplitude 2.0 design, mostly because it's much more easier to discover other people's contents and collaborate on product analytics, which was our big goal pushing 2.0. There are individual things that our loyal users would love to have back, and we are understanding and figuring out, like, what it is about the new product that we can improve further.

Were you a big user of streams before? Every single person I know was a big follower.

Like, I moved, I pushed for Amplitude because of that.

Wow. Okay. That's awesome. Well, in that case, I'd love to talk to you right after this and, you know, we'll talk about how to get Amplitude to a better place. All right. Why don't we do one more? Hello. So I have a project coming up in the health industry, but we really want to understand our users before we launch the product. How do you deal with customers in sensitive industries and anonymizing personal data but still making insights out of that data? Yeah, that's a really good question. I think it's like that, whether it's for medicine or whether it's for finance, that there's definitely a lot of And even just generally, a lot of folks are concerned about, OK, well, do we want personally identifiable information here? Is it compliant with whatever set of regulations that we care about? The short answer is we encourage folks to just not send us any personally identifiable information. All of this stuff, like all the stuff I was talking about, we're not looking at any individual users. And so we're really looking at aggregates in this case. And so the key thing— The short answer is, yeah, if you don't send us any, then it's not an issue. And you're still able to get all these insights. Because what you care about is, all right, what are the behaviors? What are the things, the patterns in common of users that have better engagement and better usage of your product? Like what their name is, or what their IP address is, or whatever personally identifying information within the application. Those are not actually factors that go into being a more engaged or retained user within the product. So the short answer is that it's definitely common. We work with a ton of customers that have the exact same pains around that. And impact on product analytics and the ability to use it is, yeah, it's not a big deal. Thank you all.