A/B experiments | Yash Nelapati

I think around 2 years into Pinterest, the team size was around 20, 25 people. And everybody has an idea like, oh, I'm a hardcore pinner. I think Pinterest should be about this and not about this. So everybody had a vision for Pinterest, which kind of becomes a very tricky situation because as a founder, you have a vision. But once you create your team and acquire people, your vision is going to change slightly, a little bit, because that's one perspective. But when you put 20 people in a room and ask them to think about the team, the product, and the vision, it's going to slightly change. So that's when things get a little tricky, like, hey, should I do this or should I do that? It's hard, man. Business is all about making decisions at the right moment and the right second, down to second. It's all about timing. I mean, if you were to build a mobile phone today that has If somebody were to build an iPhone today, it wouldn't sell like the first iPhone, right? The thing is, it's about the timing. They brought the first real web page down to phone. Back in the day, there was only GPRS pages. There was no real web pages on the phone. So that's the thing they solved. But today, if you were to build a phone, that's not enough. Because that's been solved. You need to create something bigger. better. So when your team is growing and you're facing that problem, we faced a lot of problems. Everybody had a vision. I've been using Pinterest right from the first year. And I feel like it's not closing the loop. Pinterest shows this stuff, beautiful stuff. And then what do I do? I pin this, and there's a huge offline experience that Pinterest has no idea about. We have no idea. How do you— gone, did this project? Have you actually went and bought that item? Like, have you got actually married? Like, there's all these questions because we want to know, like, the number one thing I hear is like a lot of women like collect stuff on Pinterest for wedding, and we just want to know when to shut up because we are like, hey, here's more wedding stuff, here's more wedding stuff, and they got married over the weekend and they're like, shut up, Pinterest, I'm out. So there's like all sorts of questions like this that we're trying to solve and like, hey, what's the best way to ask the user, nudge the user, what's their intention. So this is when, like, in 2013 we invested a big effort in data team. So all they do is collect a shit ton of data, measure every interaction. Not just like, oh, they visited the website, they pinned this stuff, they did a search, but down to like, hey, they scrolled, they stopped, they clicked on this pin. So that must be interesting in a feed of 100 things, that one pin caught their attention. So like, what's this pin about? Like, can we just annotate this pin? Like, this pin is about photography or outdoor or REI, or it could be about anything. It's like a myriad of things. So figuring out all that stuff requires a lot of experiments. So as a team, like, we built something called an A/B experiment. So every employee can create an experiment. Today there's thousands of experiments in flight. So instead of just pitching to Ben or Evan or product manager, like, hey, I want to do this for Pinterest. I think this is a perfect opportune moment and we need to cater the user's experience to blah. So instead of making a top-driven approach, like that's right, this is wrong, every employee has an opportunity to prove, I think X is going to work because here's the data. Yeah. There's nothing that's gonna beat data, right? Like, hey, I ran this experiment, the engagement grew by 6%, people are pinning 10% more, they're coming back every day, here's some data. How about data? Nobody can say no to you, right? It's just like, hey, I'm gonna tweak it from 0.1% to 1% of core users, and let's look at if these numbers are gonna stand. Yep, the numbers are standing, so sounds like a good idea, we should push it. Yeah? Did you guys build an in-house tool for the experiments? Totally, yep. That's awesome. It was awesome for a long time, but I just have my own perspective, which is like when you have thousands of things in flight, how can you actually measure one experiment's effect without seeing the noise of others? I mean, it's great. I ran thousands of experiments, but initially it was amazing. Here's a new landing screen and signup flow. Tell me how it is doing. Like, here's a conversion funnel, here's how it's doing. So we built like a ton of dashboards. Like, we got super valuable data. Like, every English line, like copy, we are like testing A and B. Like, every email we send, we test A and B and like see like what's performing better and like keep tweaking the knobs. So I mean, like, I could say like that's how we kind of grew for the later half, which is more scalable version because we have no idea like how people in Silicon Valley react to a product compared to how people in Iowa, like super slow, laid-back countryside, are gonna react to an English template email. So we tried a lot of things, and we like, we kind of made some great decisions. I think like it's all about A/B experiments initially because you just don't know what's gonna work. But I'm just telling maybe a company of 20, 25 people, that's the way to go. But I would say if the company is super small and you still have not validated your idea, I think you should build what you think is valuable and hear the feedback from users, which is what we did. We went to the users, collected feedback, came back, put it into the loop, made the product better again, shipped it, got more feedback. But pretty much Pinterest today is all about A/B experiments. Every feature, everything we build, has like a knob. We just like push the update, but like we turn it off, and then like one day we turn it to like 0.1%. Oh, the app's crashing. OK, switch it off. Collect the crash logs, make it better, push another update, again tweak the knob. So it's all about the knobs. So we ran a growth experiment where we have something called GIF Wrap. Like instead of just throwing people on these pages and like, hey, browse around, Like, we kind of guide them, nudge them, like, hey, there's more to see. Do you want to sign up? You don't want to sign up, you don't want to be hassled. I'm just here because a friend of mine shared this board on Facebook, and I just want to see the board, and I'm out of here. OK, we respect that. You could just keep browsing, X off this thing. So experiments like this gave tremendous growth, because we are always at every stage holding the user's hand and guiding them through our experience. Like how we want them to experience Pinterest. And we built a new tab called Network News. Like we always believed Pinterest is all about following the best pinners in any interest. Like photography, I like photography. Like I just give a lot about like the best photographers. What are they doing today? What kind of cameras? What techniques they're using? So if I could just collect all the data in one tab and here's the summary, I just don't have time today to just go through feed and look at a lot of stuff and like, oh, that's one pin I like. If you could just provide me a summary, like a small curated version of what happened, that would be amazing, which is what this tab was about. This grew engagement by a lot. This was once again also an experiment by the growth team, like, hey, I think people want to know what their Facebook friends are doing on Pinterest, and we just don't have a way to do that right now because they have to go through tons and tons of home feed pins to figure out that's my friend. Oh, she pinned about babies. Huh, maybe she's having a baby or she's expecting a baby or something like that. So kind of made the simple version where you can click and see what every user is up to. People love stalking. That's why Facebook is big. Right? Another thing that I never expected was email deliverability. We just ship a ton of emails and hope that it reached the inbox. But the truth is there's only 75% or 80% of these emails actually making their way to inbox. So we have another 2-member team that only works on email and the deliverability. So you're not sending a lot of emails? So that's another valuable lesson. And like also we need to respect like the spam and all, like how fast are we sending emails, and like people get pissed, like, hey, you just like emailed me in the last 10 minutes, again you're emailing me about some recommendations. So you should like kind of listen to your users and like build all those features. So after a while, like as I said, like we need to guide every user very carefully into your experience, the experience that you want to deliver users. So we brought in education, like, hey, you just signed up, here's what it is, you can discover stuff, and they keep swiping and we nudge them and guide them to every step. I think this is how Slack basically cracked it. Slack is nothing but glorified IRC, right? Every step they just have these glowing things, like click on this, you can go to this room, so it's a beautiful experience. I enjoy using Slack just because everything is so well explained and well documented and great support. So during signup, this performed better, but over time, over the next 1 year, there's a lot of churn that showed up. So we were like, okay, I think we need to redesign or rethink the entire signup loop. Why are people just falling off? Again, a ton of research, brought a bunch of pinners and Hey, why don't you get it? They're like, I don't know. I don't think I found the stuff that I like. Or none of my interests are being showed here. I'm into a specific kind of cooking, but I can't seem to find that kind of cooking on Pinterest. And why is that? Again, a bunch of whiteboarding, prototyping, A/B experimentation. And so we had to come up with new there, like guiding. We completely take over your screen and grab your attention. Hey there. Since you just signed up, click here. Just one button. Anything else is unclickable. And guide through them. Like, hey, what are you interested in? Blah, blah, blah. And then follow. We force follow, or you could skip any one of it. So definitely educations like this actually make your product better, enhance it. And we retain the users, which is most important. more glorified versions of like education. Like click on it and like the entire UI is like paused until you click on it, so.