#289 How I Nearly Got Fired For Running An A/B Test with Vanessa Larco, Former Partner at New Enterprise Associates

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power Your data and AI transformation journey on datacamp.com AB testing is really helpful in some cases and not helpful in others. And it can be game changing in some ways and detrimental in others. And the art is in picking when a B test is applicable and is a strength and when it's actually not. We are running probably like 30 tests a week just to determine the direction of the game and the game mechanics. You can't run AB tests in a vacuum. You can't just say I know nothing about your users, nothing about the product, I don't know the value prop. I'm just going to run this, should it be a blue button or red button and I will build an amazing product off of these optimizations. I have yet to see where that works. When you're changing a button in a game, when you're changing the character, when you're changing the landscape, the color palette, the quest system, users will pick it up or not use it. No big deal. When you're designing software that people rely on to get their jobs done, that people rely on extensively to meet their deadlines, you don't want to mess with that.

Welcome to Data Framed. This is Richie A B testing is an incredibly powerful data science technique. The idea is that you compare two versions of something like a website or a product feature to see which one works better. Used well, it can help you create amazing products that your users love. However, it turns out that it's also possible to cause disasters with a B testing and today's guest is an a B testing veteran who's seen both the highs and the lows. I love success stories and disaster stories in equal measure. So today we're going to learn from her mistakes and figure out how to replicate her wins. Vanessa Larco is an early stage investor running a stealth mode startup. Until a couple of weeks ago she was a partner at NEA leading Series A and Series B investment rounds in enterprise SaaS companies. Before she became an investor, Vanessa ran product teams for the Facebook gaming company Playdom SaaS, communication platform Twilio and enterprise data storage platform Box. It was through these product management experiences that she honed her A B testing skills. So let's hear her story. Hi, Vanessa, welcome to the show.

Hi, thanks for having me. Brilliant.

So I'd like to start with your first experiences of a B testing. How did you get into it?

Yeah, so I, when I left Xbox up in Seattle, moved to the Bay Area, I joined Platum and this was in the like peak social gaming when everyone was using Facebook and everyone was getting spammed to water their crops in these virtual games. That's when I joined Platum and the culture from the very beginning was AB testing. I think Zynga was one of the, probably the pioneers of that idea and the whole ecosystem just adopted it. As we're going to build games not off of gut, we're going to build it off of data. And coming from Xbox console gaming, which is very creative driven, GUT driven, I was fascinated by that. And so I bought into the ethos that everything should be a B tested and all decisions should be backed with data. And it was a bit extreme. The right thing is probably something in the middle, but it was a really fascinating time to be a data driven product manager.

Yeah, that's kind of wild how the rise of AB testing in games came from farming games. It's not something you would imagine would be the driver for like technological innovations.

Gaming surprisingly, is an early adopter of tech innovations. There's actually a lot of money in gaming, but it's hard to have a hit. So you'll take any edge you can get, which is why you adopt the latest thinking on literally everything.

So do you want to sort of spell it out, like how might doing all this experimentation provide some sort of competitive edge then?

There were some really clever things we did because we knew that getting users in the door was the key to making money. You can't make money if you don't have any users. How much you paid to acquire a user was very important. Now that seems very trivial today, but back in the day there wasn't a whole lot of attribution. You couldn't figure out where you were getting your customers from. Right. You put up a billboard and hope someone saw it. And you never knew where your signups came from. The Internet made that a little better. But when you acquired a user on Facebook, you knew exactly what ad they saw, how many times they saw it, how much it cost to click, and then they installed your game. So you saw the whole Funnel. And then you knew exactly who they were, you knew their age, you knew their gender, you knew where they lived, city, state, and you just had all this information you don't really have before. And so the insight was like, man, if we could figure out how to target, that makes it so we can really optimize how much we paid to acquire which types of users. We don't acquire anyone we don't need. So we don't. That's not wasteful. Right. So the user acquisition cost control was a huge unlock and we backtracked that. We're like, well, would it cost less if you had a game that the title and the art and the characters appealed more to the demographic you were targeting? Logic would say, yes, but how do you know what they want besides a focus group and whatnot? But like, you know what, we're going to build a game, we're going to know it's going to have farming mechanics, but we don't know what theme and we don't know what to call it. So we would brainstorm 12 different themes. Gnome town, Candyland, Chocolate factory, Rainbow Town. I mean, we had one that was like dinosaur, we had voodoo New Orleans themed. We came up with like a dozen themes and we had a boy character, girl character. And so we standardized the little ad icon and so the boy character and the girl character. We'd theme to dinosaur theme, voodoo theme, candyland theme, gnome theme. And then the background would also match that. So it was very templatized so we could control the variables. We knew it was like girl character, boy character, background, square. All the squares are same size. And so we would just temporarily test the theme, just run these ads. It would land you in a sign up for a wait list. Game doesn't exist yet. Right. But we would just want to see which one of these ads would be the cheapest to convert the user. Meaning that they'd only have to see it a couple times before they'd click install. And the more impressions, the more expensive it got. And it was funny because we wanted a B testing to be part of the culture of our studio. So we would take bets. We would ask everyone on our team, like, which one do you think is going to win? And so everyone would bet on their theme with money. Probably not kosher these days, but it was very much allowed back then. We'd put the money into a jar, we'd make our bets and we'd run the test for two weeks and then we'd crown the winner. And it was funny because when we Ran all these tests. The one nobody thought would win was Gnome Town, like a gnome themed game. Turns out for the demographic we were going after, they really like gnomes. It's not that the whole demographic did, but there was a niche demographic that if they saw gnome, they would absolutely play the game. So our cac, our cost for user acquisition was in the pennies for that theme. And it was like 50 cents, 70 cents for other themes. And so we're like, I guess we're building a gnome game. You know, like we would have never picked it, but our audience, it's the cheapest theme to acquire a user. And then we're like, all right, well let's test a bunch of different names. Gnome Town, Gnome Village, Gnome story, gnome candy, gnome like. So we ran a bunch of tests like which name converts better? Gnome Town 1. I'm like, all right, we're building a game about gnomes. It's going to be called Gnome Town because our cost to acquire users is 10 cents. And then we build the game, we're like, should it have this storyline, should it have these characters? And so we would test different quests, we would test different mechanisms to earn the yellow digital house. Should you do a spinner, should you finish the quest to get it? Should you just buy it off the little marketplace? And we'd run all these tests. We were running probably like 30 tests a week just to determine the direction of the game and the game mechanics.

That's pretty amazing. And I suppose that's kind of the beauty of the testing, right, Is that none of the people on the team could predict what the best user game was going to be like. I certainly wouldn't have been able to guess that gnomes were going to win. So yeah, as was. Yeah, the data shows something the humans couldn't have got to themselves.

Well, here's what we got wrong though. We tested the conversion. So it turns out if there's a really enthusiastic niche, your numbers will look really great. If you hit that niche. The problem is you don't know the depth of the market. And we couldn't figure out how to measure TAM for a theme. And we ended up landing on choosing a city, a small city where we could flood it with ads to see how deep we'd see like, okay, CAC, $0.10, $0.10, $10, $20 once you hit all the gnome enthusiasts, whoever is not a gnome enthusiast, would be very expensive to acquire. We didn't see that coming. Right. So our game grew so super fast. And then once we acquired every gnome enthusiast in the US we couldn't acquire anyone else because CAC was so high. And we didn't see that coming until we hit a wall about 12 months later after launching the game. So that was another. Learning is like you can run tests but then you need to go like a couple steps deeper.

That's very interesting. Yeah, I can certainly see how you either love it or you hate it if you've got a gnome theme. So I can imagine there is that sort of wall to jump. So talk me through. What do you do then? How do you deal with predicting these longer term things?

I think that's the biggest drawback from a B testing is that it is rather short term focused. Nobody really runs an A B test for a year. Right. You don't have the time. And if you do, by the time you have the learnings, it's retroactively. So like you'll learn what you should have done a year ago. And it doesn't help you a whole lot at that moment or forward thinking because you probably already knew that you were in trouble at that point. And so where I landed was like, AB testing is really helpful in some cases and not helpful in others. And it can be game changing in some ways and detrimental in others. And the art is in picking when a B test is applicable and is a strength and when it's actually not. And the general rule of thumb where I've landed is for optimizations the best, like you have a button, you have something small contained, great thing to test, ad copy, where should you go? Slogans, titles, all these things great for net new, like if you want to launch a whole new product or if you want to do like a step, function change or pivot, it's not that helpful. You have to start from something. So if you're trying to use a B testing to design that thing you're trying to start, you're going to end up with like a Frankenstein with like no opinion. It's just going to be a hodgepodge of micro optimizations which is not cohesive. So I think you have to go with gut and then you can test things around the edges to validate some of the decisions. But it can't be how you start. Like it can't be the spark of the initial product or idea. Now with Gnometon, we knew it was going to be a farming game. We had certain constraints, we had certain budgets, so we were like, we know it's a game, we know it's farming mechanics, we know there's going to be characters, it's going to be a quest system. Like we had all that engine already built out. We just needed to skin it and theme it.

So a B tests are much better for these smaller decisions rather than like the grand strategic sort of vision, I guess. Yeah, it's harder to do a test for like the whole game. You just want to test individual parts, right?

Yeah. And I would also say that one thing I did really appreciate about AB testing is that I learned a lot about my users. So by having my team bet on what we thought was going to be the winning shard, we started getting really good at predicting our users behaviors and getting in the mind of our users. And then we would supplement that with actual focus groups. So we knew that if we put up a virtual good for sale in the marketplace, it would convert like 10%. If we made people spin for it, like you had to do a little spinner and try to win it, we would get close to 100% conversion and people would spend quadruple on it trying to spin, spin to win it versus trying to just buy it from the marketplace. And we also knew if we did a candy themed item versus like a strawberry or chocolate theme did really well. But if we did banana, it didn't. And so we started to know by a b testing these designs, this artwork, we're like, okay, our users like strawberries and they don't like bananas. Can't tell you why. A banana fountain probably looks not great, but a strawberry one sells. A yellow home does better than a pink home. Can't tell you why, but we just know that our users like homes that are yellow because we a B tested a bunch of different colored homes. So towards the end of like the game, because games peak and fall, we knew our users backwards and forwards. We knew the colors they liked, we knew the home styles that they would buy, we knew what kinds of surprises they liked. And animations, the sounds, the more bubbly sounds did better. And so by a B testing, you start, and if you do it frequently and you're trying to unpack why, and you do these on a weekly basis, you really do get into the minds of your users.

That does seem incredibly important. I think it's something that almost every business, they want to know, what do their users want better so they can better serve them. And actually, I know it is a bit of a sketchy culture, but betting on what the users want, like what the tests are gonna be successful, that does provide some sort of good incentive for all your staff to really care about what the users want. Because you know, they got some money riding on the line.

Yeah, it was a playful way to do it. But you want your team to care. I know it sounds super trivial. Of course they should care about the customer, but it's not just caring about them, it's knowing them, like deeply knowing them. And so our target demographic was women, mid to late 40s in non coastal cities. And our team building this was late 20s, early 30s, mostly men in the Bay Area. And so it's, you have to build user empathy. How do you build it? And I think AB testing really helped our team develop that. We almost gamified it. Like how well do you know our users and how well can you predict their behaviors?

I guess that's also a really important learning is that like you are not your users. And it's said so often, particularly in software development, but I think people forget it sometimes. They build stuff that they want to, they would want to use themselves. If you're not your own target demographic.

It'S so much easier to build for yourself. But it also means that if most of the builders are pretty homogenous, then those areas are very competitive. And if you build something that you're not the user of, there's probably less competition there.

So I'd also like to talk a bit about how you go about scaling this, because it said you were running what, 30 tests a week? That seems like a lot. How do you get to that level of testing?

We built on in house. So this was like before all the AB test platforms. We built our own A B test platform in house. And to release anything the way you release it was through the platform. So it didn't make any sense to not test it. If you had to press the button, you'd have to say like to how. What percentage of our users does this release go to? So to not test it, you have to put a hundred percent in like the new release and zero in the test group. So you'd have to actively try to not test it. So you have to make releasing an A B test very easy. You also have to have people care about the results because it can be easy to launch. But it's such a pain in the butt to like two weeks later draft the report and then try to make sense of it and try to see what the users were doing, especially if they like adopted and abandoned it or had some wonky, unexpected behavior trying to make sense of it. And to do that every two weeks across 30 tests, it's a pain. One of the things we did to combat that one was the betting, the gambling. But two, we also held round bags, lunch, brown bags, where we would invite everybody in the company to sit in and learn about how our tests perform and what our hypothesis was. And we opened it up for people to also help us in figuring out like, especially if there was a weird result where some people loved it, people hated it, or people adopted and then dumped it, or adopted but never monetized. And we're like, but if they like it, why wouldn't they buy it? If they tried it, why didn't they share it? So for those head scratchers, people really love to hear about those. And it was a company wide thing. And so knowing that we had this cadence of having to have this presentation to the company also helped us stay motivated in looking at the results and trying to make sense of them.

I like the idea that by trying to teach the rest of the company about what you are doing, that then helps you understand what your own results are. Because if you don't understand it, you're not gonna be able to explain it to everyone else. And I also like the idea that you share your weirdest results. Cause that's gonna be more interesting to people. If you don't quite understand what's going on, then just open it up to everyone else. Yeah, okay, cool. So the Facebook sort of social gaming bubble was pretty short lived, I think. So you wanna talk about what went wrong and how did it fall to pieces?

Well, a few things. Look, intuitively we knew that something was up because we would test a pop up and it would increase monetization or virality. Then we'd add in another pop up and it would increase monetization, virality. And so the user would get two pop ups in a row and it didn't seem to hurt retention, engagement or monetization or virality. We added a third pop up, seemed to just do better. We added a fourth pop up, seemed to do better. And intuitively like, this is a shitty experience. Like you log into the game, it's like, pop up, pop up, pop up, pop up. That doesn't seem like great product design, but the numbers were saying it is. They would either share more or buy the thing and it had no impact on retention. They would still come back, they'd still play the game. So we did a lot of those things where I'm like, that's counterintuitive, that doesn't seem right. And so if you erode the user experience, if you spam users over and over again, they may tolerate it, but at Some point, some point they will just flip. And it's not like they'll flip slowly, they just have enough and you're dead and you can't get them back. But the numbers don't show you that. I wish it would be like, oh, you know that 10th popup really did you in. Like, the engagement fell by 20% from these users, right? They log in less times per day. No, there was nothing, nothing, nothing. And then also turns out that if you spam users and they get pissed, the platform will shut you down. And so Facebook was like, gosh, Zynga and Platum and all of these. Like, the news feed is no longer what your friends are doing. Which is the point of, was the point of Facebook, the newsfeed was to tell you what people are up to, what changed, where they're going, where they were, the pictures they uploaded. The newsfeed was just all, water your plants. Vanessa invited you to buy a gallon of milk, your home needs tending. Right? It was all these Facebook games just spamming the entire news feed. And so it didn't only erode the gaming experience, it eroded the platform experience. And platforms don't like that, right? If you mess up Apple's user experience, they will shut it down. I don't care. Like what API was that they gave you access to, they will shut it down overnight. They don't care. These platforms will protect their user experience above all. And so those are some insights we didn't have, right? We didn't know that if you piss off your users, they churn overnight and you can't get them back. We also didn't know, which is now like pretty obvious, but if you erode an experience on a platform, the platform will shut you down. So overnight Facebook was like, that's it. You gaming companies, you're not allowed to post the newsfeed. And they shut off news feed access for us and we were dead. Shut off newsfeed access meant that we couldn't remind you to water your crops, we couldn't remind you to tend your home, we couldn't remind you of anything. And turns out the games weren't good enough for it to stay in the back of your mind. And so they all stopped coming and we'd been spamming them for months. So the games accumulated all these pop ups and it was just a shit experience. And then everybody jumped ship and it was over so fast.

Yeah, it's kind of a disaster story. It sort of feels like slightly karmic in some ways. Just like, yeah, we were spamming users and then they just revolted. So it's hard to feel too sad about it. But I think there's some important lessons around how you interact with your users there.

Yeah. And that a B test can tell you this is all looking great. But if intuitively you're like, this doesn't seem right, it probably isn't. And maybe you can design a test and figure it out, maybe you can't. But I think at the crux, you still have to use some amount of judgment. And I think if you have a hunch that the AB test is telling you something, but maybe it's not the whole story, dig in, dig in a bit deeper, Run the user study or the focus group, look at other pieces of data, maybe design a different test. Yeah, I think there's just sometimes there might be more to the results you're seeing and that's where the creativity falls into place.

Yeah. I love the idea of combining. Well, we've got some data that says this, but also I'm using my own brain to think about what does this actually mean. I guess the other trade off here is between the short term benefits of you've spammed some users, you've increased immediate engagement, but you also want to optimize for like long term engagement and long term. So do you have any sense of like, how you go about balancing these short and long term objectives?

Like I said, AB tests are really good for short term objectives, not so great for long term objectives. And gosh, if someone listening to this has figured it out, please shoot me a note. My LinkedIn DMs are open. I would love to hear about how you do long term AB testing, where the results are influencing your decisions. Today, it just seems like it doesn't add up. But I think when you're looking at the long term, you have to ask yourself some questions of what do we think our users are going to get from this if they stay on our platform in a year, in two years, in three years, how does this get better for them? And if you're running a test and you're like, this isn't aligned with what we've decided the product should look like a year from now, then it probably doesn't work. And maybe you shouldn't do that in the short term. But I think the question I always push our teams to think through is how does this get better and more useful in a year or two or three or four or five?

I like that. Yeah. So you do have to just like as a product manager, you've got to remind the rest of your team, we do need to think a bit more long term rather than just like, yeah, what's going to get long term? Sprint.

It's compounding value. And so some of the tests we can do, then you can ask yourself not does it convert better today? But does this compound value? And if you ask yourself, does this feature compound value? Then you may design the test a bit differently than just a pure what's the lift on monetization or what's the lift on virality?

Okay, how might you test compounding value?

Well, at box, we knew that if you had all your documents in box that it would be more useful to you. And we could run some analysis on that and see that the more files a user had, the more active they were at box or like, the more likelihood that a company would retain if they had just more files. So it's not hard, both from like analysis perspective, to prove that if you have more files, it's more useful. But just logic, if all your stuff is in one place, you're just going to go to that place more often because that's where you're going to find things you need. And so when we thought about compounding value, we're like, well, how do we get you to put more files in? Because the more files you have in, the more useful this is to you. You know, at some point we realized that for some users, more files wasn't super helpful. And when we dug in, we were trying to figure out, like, why and it wasn't obvious in data. So we ran some focus groups and they're like, well, now we have so many things in there. When I search, I can't find the thing. And so then I have to organize it to find it. And so then we're like, oh, okay, how do. If you have everything in there, yeah, it could be pretty hard to find something. So we added like favorites, we added other shortcuts. So there was a, oh, wait, there's too much. Maybe it's hard to find, maybe we should solve that problem. But overall, when you think through, like, where at a high level, where does this product fit in the user's life? What is the value? Can you describe in a sentence? What is the point? What are you solving for them? What's your value proposition? And then the features should add to that, should push that forward.

I like that. Yeah. Again, we're going back to the idea that you need to understand what do your users want and what problems are you solving for them.

You can't run a B tests in a vacuum. You can't just say I know nothing about your users, nothing about the product, I don't know the value prop. I'm just going to run this, should it be a blue button or red button and I will build an amazing product off of these optimizations. I have yet to see where that works.

So a lot of thinking about context, otherwise you're just going to be doing nonsense. I guess that's true of a lot of things. Working with data, if you just, you can do. Data's cheap, you can do anything with it. But if it doesn't make sense, then.

Look, Rishi, there was this like the rise and fall of the growth pm. I got a job offer a while a long time ago when I was in product to go be head of growth and I was like, I don't want to be head of growth, I don't want to be head of product and growth. But I think product and growth need to go hand in hand because the growth people or the departments that have been built at that time when it was super popular to do growth hacking, AKA a B testing, they didn't care about. They were just like, I just need to get more users in the door. And so like I will design all these tests and like they were separate from the product team so they didn't actually care about the user experience, the value prop or any of that. It was just like I'm compensated on getting as many people through the door as possible. And I was like, I having lived my experience in playdom, I don't think that's the way to do this. That is very short sighted. And so I know it seems obvious that you need to know what your product does and understand it deeply and have user empathy to be able to run good AB tests, but that was not so obvious back in like 2013, 14 and 15.

Yeah, I suppose like the rise of AB testing is happened a lot over the last decade. Just because all these tools have got more powerful, it's become a lot easier for people to do this. So you mentioned Box. I'd love to hear about your time at Box. Was there a similar experimentation culture there as to what you had at playdom?

No, no, no, no, no, no, no, no. And for good reason. When you're changing a button in a game, when you're changing the character, when you're changing the landscape, the color palette, the quest system, users will pick it up or not use it, no big deal. When you're designing software that people rely on to get their jobs done that people rely on extensively to meet their deadlines. You don't want to mess with that. Like, imagine if Riverside just changed all of this overnight right before an interview that you had, and you're fumbling around and your guests can't get in. You are going to be livid. And how much more mad would you get if Riverside's like, oh, sorry, you were in an a B test. We just want to see how you would like this entirely new system, but we'll change it back if it doesn't perform well. What would be your response?

Yes, that happens a lot, actually. Riverside, this is the podcasting platform. They do change their user interface quite frequently. And I'm often, like, not quite sure what I'm doing here. I'll try and look like a professional when we're recording. So, yeah, that is very difficult. When it's your job on the line, you want to understand how your software works.

Yeah. And if you're paying a lot of money for a platform, you don't take kindly to someone saying, oh, we're just testing this on you. You're like, no, I'm paying you money, so I'm not your guinea pig. Go guinea pig someone else. Right. And so you have to be careful. Especially there's this, like, I call it evil word. But it's a prominent thing you should think about if you're an enterprise software called change management. And basically, when you deploy to big enterprises, they have to train all of their employees, and they hold webinars and they hold office hours, and they'll have booths in the lunchroom to get everybody excited about the new piece of software that they're going to roll out, they spend six months to a year rolling something out and getting everybody trained. Now, if you get, you know, a thousand people in your organization trained, here's where the upload button is, here's where the download button. This is how you share here's permission. Do not click this button. This will make it a public L. And then here comes Vanessa, runs an ab test, changed everything. You have to update your instruction manuals. You have to do retraining for everyone. So it's just different than a game, you know, or than, like, an Amazon shopping experience where, like, you're going to buy the thing, whether the buy now button is orange or yellow. But if your job depends on it and the button changed from orange to yellow, and you're like, oh, wait, it's yellow now, maybe I don't press it. I don't know what to do now. Maybe it's a different button. I don't want to get fired for hitting the wrong button and publishing this to everybody. And so it's different.

Yeah, it just seemed like the stakes were a lot higher here. And I'm getting the sense from what you're talking about that there's a disaster story looming here. Can you tell us about that?

Okay, so I came from gaming and I was like, I need to know my users, I need to know what they like, what they don't like. We have this thing that. Because when you launch a feature, some enterprises, some teams, some employees adopt it. But if not, if like 0.1% of my users adopted and are using it, is it worth it to keep it alive? Right. There's UI debt, there's tech debt. We have tested every time we do any launch on any other part of the product. So it's like, why don't we ever deprecate features? Why don't we ever just get rid of them if Nobody, if like 10 people are using them? We have millions of people on the platform. We should just kill the stuff that doesn't work. Right? So I wanted to launch this new feature which needed some space in this menu option. And I was like, well, I wonder if people will find if it's in that menu or if I put it on the button at the top. So that was gonna be my AB test. We'd never really ran one before in the actual web app. And I was like, this will be great. We'll learn a lot about how our users find things in the app. Right. Do they always go to the menu or do they typically look across the top? Where do they first go when they want to find something? And then while I was at it, I was like, you know what? I should add another shard in here. There's this feature in the menu that really crowds it and I think if I got rid of it, nobody would notice. But I should run that as a test instead of running two separate tests. One for a while and then the next one. I decided to do what I did in gaming. A four way test. One with like the control, one with a button at the top, no menu item and keep the old one one with it in the menu item and kill the menu item. And so I created these four shards. And then I just decided. And the engineering team that worked on it was a data team. They were so excited to do a first a B test. They built this platform to be able to do it. We were super jazzed. We didn't tell anyone that we were going to run this. We just said we're launching this new feature and it's going to be a button on the top and telling them that we also were going to have it as a side menu option and that a quarter of our users weren't going to see the thing on the top, they were going to see it in the menu option. So okay, this may seem very logical that this is a bad idea. But at the time we were so excited. We're doing a test and in gaming I would just launch these tests every week, every other week and nobody cared. So we launch it, we put it in for the release to get like compounded and everything get out throughout the evening. And we had teams all over the world using Box. So in Japan they got my test when we were all asleep and turns out there was like this insurance company that really used that feature. I was going to deprecate they were the.01%. And remember I actually, I started by user ID, not by company ID. So four different people at the same company sitting right next to each other could see four different UIs. Now you can imagine how this is going to go, right? So one poor person was like, I need to do this repetitive task over and over again. Oh my God, my button disappeared. Did your button disappear and they went. My button didn't disappear. What do you mean your button disappeared? Well, I can't do my job. I need this button to like file this thing the way I'm supposed to. Like, let me see your screen. Yours is still there. Yeah. Well, reboot your computer, re log out, log back in. Oh my God. Maybe your account got corrupted or maybe you got hacked. So they call customer support and the customer support agent, the way it worked is like it would create a new udid, like a new enterprise ID in that account. They weren't in that shard, they just magically placed in a different shard. So they're like, I have no idea why my screen looks different than your screen looks different than that person's screen. Code red, like sound the alarms ring, the beepers wake SWAT team up. We've got to figure out what's happening here. This is bad. So they wake up a huge chunk of the security team, engineering team, the on call team. Everybody's trying to figure out what's happening. They're up all night. I stroll into the office at 9am before all my engineering team, thinking I'm an overachiever for getting there early after working late. And they're like People waiting for me on Dust, like, did you hear what happened to your product? I was like, what happened to my product? They're like, it's a disaster. I'm like, what's a disaster? They're like, this team can't find this button and we don't know what's happening. We can't replicate the issue. And I was like, I. We're running an a B test. We're just trying to see how people like this new ui and they're like, I'm sorry, what are you doing? Yeah, yeah, yeah. We're running an AB test. We just want to gauge user preferences. And also this button, no one ever uses it. So we figured what would happen if we got rid of it. Like, one, you can't just get rid of things without informing people who are using it that you're going to get rid of the thing. I was like, yeah, but that runs against how you run AB tests. You can't tell people they're in an AB test. You ruined the whole test. They didn't think that was funny. Nor did they care about how you run an A B test. And, like, the sanctity of the data. They were livid. And they're like, you cannot run a B tests, and this is like a very important account that's supposed to renew next month, and now they are livid. And I was like, oh, I. Yeah, it's probably not a good idea to run an A B test on a customer that's supposed to renew next month. That was sad. So then that account manager flips out and they're like, vanessa ran a test on your customer. So within, like, two hours, I get called in as the CEO, the head of sales, ahead of, like, customer success, head of customer support, and I just get reamed. I'm like, what were you thinking? And I was like, I just wanted to get to know our users better. And they're like, nope, we don't care. Like, you're going to cost us this account. This was a disaster. Customers don't want to be around. So I was able to diffuse things and I was able to convince them, like, we do need to understand we have to be able to deprecate features. We just do. Like this platform. We can't just add, add, add forever. It's going to be a disaster. So when we get a feature wrong or it doesn't hit the mark, we have to be able to get rid of it. We can't just keep it forever because 10 people use it now. We could probably Go about it differently. Maybe we like notify them, maybe like it. The test doesn't have to be perfect, so I don't want to piss anyone off. But yeah, we have to be able to like understand what happens when we get rid of features. So I convinced them on that. So we ended up coming up with like a whole like, how do you end of life a feature? How do you test the impact of ending a feature? So we agreed on that. And then when I launched something new, I want to know if it actually hits the goal, which is either increase engagement or increase files uploaded or whatever. I want to know if it's actually worked or didn't work. And that's the only way we're going to get better. As PMs on my team, that was a lot harder of a sale. So what we ended up agreeing on was we would not shard by user id. We would start by company id and then we'd bucket companies into like different categories, large, medium, small. We could categorize them by industry. So we'd be like, okay, want to test this on like, have the same number of companies in legal versus same number of companies that are in healthcare. So we'd have like a good mix. But I would never again start on user id. So no one, everyone at the same company will always have the same experience. Not the perfect way to run an AB test, mathematically speaking, but if you want to survive and not get destroyed by your team, that's probably the right way to do it. And then what was the other concession? Oh, we'd have a blacklist customers we would absolutely not run a test on because they were either in the middle of a deployment in training their customers, or they're about to renew or something. But they were like completely excused from AB testing and that's where we landed.

Okay, I'd say that is. It was probably not hilarious for you at the time, but it's hilarious to me now. Yeah, I can certainly see how like, oh yeah, I'm doing something amazing and then you've just. Yeah, the CEO's not happy. No, it's not a great place to be in career wise. But I'm glad you did manage to sort of eventually convince all your sort of management around this. Do you have any more insight on like, how did you go out diffusing the situation? I'm curious as to like, if you're planning to start your own a B testing program, what's the more sensible way of convincing people this is a good idea?

I would first get their Buy in before you do it. Don't do what I did, which was like, everyone obviously wants us to do this. It makes so much sense. I'm an A B test. If you're in consumer, fine. But if you're in an enterprise company, absolutely get the buy in for the people that it could affect their accounts, sales, customer success, customer support. So like if there is an issue, they're trained and they know what to do about it, they can roll things back. So you definitely want to hear people's concerns because people are going to have a lot of concerns, especially if they've never done it before. And I would come up with protocols for like how you would manage different situations. I would say not to be too dogmatic about the proper way to run an A B test. Yes. Like in an ideal world you want to turn for so much amount of time because he wants to statistical significance and you want as much diversity and so you want it to be totally randomized. But you're going to need a lot more flexibility. The real world doesn't work that way. And having data that you think is good enough to give you directional feedback, I think that's a win. So I know that a lot of people out there who are very into data and a B testing will be like, no, what are you saying? But it's like you have to crawl before you walk. You have to get your organization to buy into the benefits and then I would say share the results broadly once you get the buy in and you can run some tests, share the learnings with everyone who you think would be interested and help them get something out of this effort because it does require a lot of work for so many people in organizations. When you do a program like this that you want to reward them, you want them to get something out of it too. And funny enough, there were some areas we felt more comfortable a B testing with and others that we were less excited to do testing with. For the fear low risk, I would solicit feedback from customer support and customer service. Like, is there anything you, you want to know about? Like if we got rid of this, would they care if we added this? Would they notice like, ooh, we get to play too? I'm like, yeah, give me some suggestions on a B tests. And so you kind of want people to feel like they're part of it too. It is a heavy lift. It does create friction in the system. So just make sure everyone has something to gain from it.

That is really fascinating, the idea of getting support involved because I think support's one of those teams that just gets forgotten about until there's some sort of disaster. So I mean it's kind of sad for support, but yeah, getting them involved because they are going to have to deal with customer requests if something goes wrong. Like your example of the disastrous four way test. So yeah, that's a very cool idea.

Side note, customer support was my secret weapon in being an effective product manager. I would pull in customer support to help. I would manually QA everything before it went out the door because the buck stopped at me, right? Like if we did a release and it was buggy or it was bad or it was something was off by a pixel and just looked sloppy, that's on me. And so I would manually create. It doesn't matter like what time we were done with the build. I manually curate everything and having customer support come in and like if they were interested in getting the build early or getting early access to the build and file bugs on us too, they knew exactly where customers were going to trip up because all they do is see customers trip up and so be like, ah, that, that's going to give us a lot of customer support tickets. You got to change it. People don't like it there. That's a right click. Nope, nobody right clicks. Like you can't do that. And they would just see all the areas that were going to explode tickets in ways that like I couldn't predict. And so having customer support be a part a lot of like your development cycle is so valuable. So valuable. And also having them really like you and your product team is also really helpful. So when there is a bunch of bugs, they'll like jump in, run upstairs, be like, hey, this is looking like it's going to be a big problem. You need to prioritize a fix for this asap. And so customer support teams, the closer you can combine that with product and testing, the much more success I think you'll have.

That seems like a very useful tip. I'm not sure many organizations do that. So yeah, I think it, maybe it should be done more widely. Okay, so how about the data team? Like do you have any advice for the data team on what they need to do to run a more effective a B testing program?

Data teams are super important, they've always been important. But I also feel like people don't understand where they spend the majority of the time. They think that all they do is run reports or people ask them for the most like trivial stuff and they're like, that is not what we do. And So I think the one advice I'd give to data teams is to continuously educate the organization on what are the big data investments they're making and what does that unlock for them. I would try to set up as much self serve as possible so that the data team can work on more strategic stuff rather than like tasks that are not additive to the data pipeline and data strategy. I think some PMs are really well versed in how you can use data and others aren't. And so spending time with the product managers of like, hey, these are the tags we want you to have in your data. This is how we want you to think about it because this is in return what we'll be able to do for you. You just have to educate consistently and in the lens of like this is what it could do for you, this is what it could do for your team and for your customers if you collaborated with us in this way. And data teams change and their ethos and their goals change like every five years as their bigger unlocks or different company priorities. And so I think the education just shouldn't end and data teams should. I almost feel like they should be like a data evangelist within the company because I feel like data teams become so insular because there's always so much to do. They don't have time to go evangelize. But I think it could help them prioritize and help other teams help them as well if there was just more of that education.

Yeah, I do like the idea of every company having a data evangelist. My job title is extremely rare. There should be more of us around.

Exactly.

And certainly I'm right there with you on the idea that self service analytics is the way forward. We talk about this a lot on the show. It's one of those dreams that it's kind of hard to reach. But I think most companies are getting closer.

I think with AI we can at least get the natural language query closer to a query that a system can understand. And I think that's the holy grail, to be honest. I think from the beginning everyone's like, oh, you know, every PM I hired on my team at Playdom would have to learn SQL. And there was this like Stanford class that was online that I would sign them up for and everybody had to learn SQL. And I was like that's kind of a high barrier of entry. And then it got a little more natural language E. And then there was like some suggestions that the system could do for you of like could auto generate some reports and some dashboards. And insights. But I think if you could just ask your data a question in just the way you ask GPT a question, then the training becomes less about how to ask and becomes about what to ask. And that still needs some training. You still need to tell people like, you want to know what your user did yesterday or how that jives with the change that you made three days ago. Right. You still have to get people in the thinking of experimentation, so they have to ask the right question, but they don't have to worry about what words they're using.

So I guess the important skill is understanding what the business problem is you're trying to solve and then at least getting a bit closer to turning that into a data problem. And the actual mechanics, they're getting easier to deal with. All right, so I'd like to talk a bit about your new role. Since you've switched from product to being a venture capitalist and since you're working with early stage companies, I guess there's not that much data there yet on each of the companies. So can you still do experimentation there?

Well, I think you can do experimentation around user acquisition. You can always do that because you just target a bunch of users and see if they convert. Some founders do it, others don't. I don't think it's a leading indicator of where the company's gonna be successful or not. It's just some founders just wanna lean on data to get some early answers. Early on a B testing is hard when you don't have a lot of users. That is true, but understanding your data for the users you do have is still helpful. And so even if you have a thousand, two thousand, five thousand beta users, probably not enough to run a B test, but you can see how often they're using the product. You can see how long their sessions are in the product. You can see what clicks they're doing, what tasks they're accomplishing. And you may learn some new things of like, hey, I really thought they'd come to, I don't know, book a trip. But instead they're just browsing every morning and doing nothing but browsing. Like, what does that mean? Like, well, maybe they're generating ideas, maybe they're building wish. Maybe you should do a wish list builder. Like, I, I don't know. But that is not the behavior you expected to see and you're seeing it. So let's run some tests on that, let's launch some things, let's see what's happening, or let's reach out and do a survey or Video chats with some of these users. So I still think that I love working with teams when they have a thousand, two thousand beta testers because those are your most enthusiastic users. Those are going to be your early adopters, your power users. They're going to be the best cohort that you'll ever have. But it's also the ones with the value proposition resonated and you can really understand why and see how do I make that more broadly applicable to people that have the problem. But it's not as burning, so that's fun. I still think looking at data is super fun. And it's interesting because VCs don't typically understand product data. They understand revenue data where the money comes in and out of. They get that data backwards and forwards. They like put that into spreadsheet and they know your CAC is this, your ltv, is that your CAC to LTV ratio looks great if it's in this perfect little bucket that indicates great companies will invest. But that data shows up like way. It's a lagging indicator that only happens when you've like met the need and understand the value prop and hit it like nail on the head pretty perfectly. And so it's been fun to be a product person inventor because I can get conviction on companies based off their user data, their product data and not their revenue data. And then I know that it'll come like if they figured this out, if the user behavior matches the value proposition, the rest will come. So it's been fun. It's a different use of my product skills. I did not expect to be applicable in this type of role.

That is very cool that even if you do only have like a couple thousand users, you can still do some useful tests, you can still get some sort of interesting results, maybe find out, well, yeah, I guess fewer people to learn from, but you can be able to learn a lot more about them.

Well, those people would be willing to like hop on a call with you, they'd be willing to get on a zoom because your first cohort is your most enthusiastic cohort. And if you make them feel VIP founding members, give them some like badge of first ones here, then they feel some ownership in this product being successful. And they will, they will give you feedback, they will give you data, they will file bugs. And so that first cohort, that's the one to like learn as much as you can qualitative and quantitatively.

I feel like from a data point of view, you don't necessarily think of like, oh, let's have an actual conversation with one of my users. Yeah, but it works. Having chatting with other humans is kind of a brilliant source of data, even if it's a little bit more inefficient to collect compared to clicks. But, you know, sort of doing it a lot.

Sort of. Right. Like, if you saw some weird behavior, you probably come up with like 13 different tests. You'd just like, is it this is that I'm like, Or you could just call them. Be like, why'd you do that? Like, oh, I just. In the mornings, instead of browsing Instagram, now I browse your app. It's just super zen. I love the pretty pictures. And you're like, oh, how would you have found that out? And you would have had to run so many tests. Like, that was just faster. And then you can write a test to validate that. But you just, like, shortcutted a bunch of stuff. Sometimes just talking to your users will help you, like, bypass a lot of brain damage.

I like that. Cool. Okay, so I'm curious, how do you decide which product metrics you want to track?

Ah, I love this question. Okay, I know I've said this about a thousand times, and it could probably be a drinking game on your podcast, but understand your value proposition. Like, what is the point point? Why do you deserve to exist in the life of your user? What is the thing you do for them? The more succinctly you can define that. I save them money on travel, I save them time on accounts reconciliation, I give them higher conversion rate from lead to close, whatever. It is, like, very succinct. And then from there you derive the metrics, right? Then it's like, okay, well, then, like, I want to see leads come in, leads convert. And then it's like, okay, well, what are some leading indicators before that happens? And then what are some of the indicators before that? And then you just go earlier and earlier and earlier. And then you have your leading indicators and your lagging indicators and you track all of those things. But it has to start with the value prop. So many founders just start with, like, the best practices metrics. DAU over MAU sessions per user per day. And then they show me these metrics, I'm like, but this doesn't map to your value prop. Right? If you're looking at dau over MAU and you're a travel app, people don't book travel every day. That's like the wrong metric. And so tell me what your point, what the value is, and then backtrack into the metrics. That show that you're actually delivering on that value. And then let's go earlier and earlier and earlier and figure out if we're on track to that.

Yeah, that's interesting that like the amount of activity of users, that's a very common thing to track, but you don't necessarily always want that actually. Do you have any examples of like, what can go wrong if you have misaligned objectives? So what you're tracking is different to what the business goal is.

Look, I think at Box we had an admin console and my team was the web app, mobile app. So we tracked engagement and so everyone thought like, everybody needs to track engagement, we need everybody to engage. Because when you engage you retain and if you retain, you can you upsell and. But the admin console, the admin chose Box because it's easier to administer this and set the controls and set it and Forget it. No, IT admin wants to go into every single SaaS app's portal every single day. They would hate you for that. And so I remember they're like, ah, we can't get our engagement up. And I'm like, wait, why do you want the engagement up? They're like, well, because then we'll stay top of mind. I'm like, do they want that? And they talk to you for admins like, no, no, we love boxes. We like set it up and then it just like autopilot like provisions and the security settings carry and it follows the org chart and the permissioning. Then it's just like so great that I like set it up and unless something like needs my attention, I just trust Box is doing what it's doing. And I was like, wait, so you guys should be looking at like the less engagement the better because that means that they love you so much more because it's working and they trust you. And everyone's had just exploded. They're like, oh, we want the anti usage. And so then that team changed it to the success is how much customization do they set up upfront? Like how many of our security features do they enable and custom setups. And so what we want is when we launch these different things, we want people to adopt them and set it up in their environment. So is it actually feature adoption and not engagement?

That's absolutely wild that if they've been just trying to boost this metric that they thought was a good idea, they'd have been actively making the product worse. So it'd be data driven disaster.

Exactly.

Okay, all right. So yeah, definitely some lessons to be learned. About, like, think really carefully about, like, what is the right metric for your product? All right, so just to wrap up, what's your final advice for people who want to get better at running testing programs?

It all starts with a why. I think I've been talking about that a lot in this episode. But why do you want a testing program? What are you looking to get out of it? And then from there, who do you need the buy in from? How do you provide value to your customers, to your team members, to the executives? Like, how will the product, how will the company get better because of this? And really think through that. So start with, like, what is the goal of this? What is the crux of it? Why would it improve things? And then get buy in from everyone and then go deliver on it? It sounds trivial, but it's so complicated.

In practice, yes, I feel easier said than done. But I like that there are some definite steps. There's an order to do this, like, get the buy in first, then do the testing, not the other way around. And yeah, lots of cool stuff. All right. Wonderful. Lots of insights there. Thank you so much for your time, Vanessa.

Thanks for having me, Sam.