Product Experimentation at Scale, w/ Claire VO, VP of Product at Optimizely

Hey everybody, I'm Mike Del Sito, co-founder of Product Collective, and thank you very much for joining us for an industry interview. We used to call it Product Lunch, but you might have seen on the page we've changed some things around. So we've had some feedback like, hey, your newsletter, it's Product Brief, you've got Product Lunch. I like— I know you guys, you put on the industry conference. And we're like, okay, so we're starting to use industry more, so you'll start to see that. But If this is the first time you're joining us, all of that will pretty much be irrelevant, but I'll let you know what's ahead for this session. Basically, twice per month, we meet up with product leaders and we talk about topics that are relevant to product people. So I'm really excited for today's session. Obviously, you can already see Claire Vo of Optimizely. So I'm excited to get into the discussion with Claire. Before I do, I just wanted to make a quick announcement that we will be soon announcing the last speaker joining us for the European edition of Industry. Which, Industry is the conference that we put on, used to be once per year in the fall in Cleveland, Ohio, and we're already planning for that. We'll have about 1,000 people with us this October, but at the end of April, we'll be going to Dublin, Ireland. And this is the first European edition of Industry, which we're really excited about. Have all sorts of people from former years joining us, like Bob Mesta, John Vars, Those, they did really well in the past at Industry, so we're excited to bring them back. But we have all sorts of new folks that we'll be featuring as well, like Des Traynor of Intercom, Ryan Singer of Basecamp, and all sorts of people. So we'll be announcing the final speaker quite soon. So you'll have to check that out at industryconference.com. It'll have info about both of the conferences. So check that out. But with all of that out of the way, I definitely wanna dig into the topic for the day, which is experimenting at scale. And to talk about that, we have Claire with us. Thanks a lot for joining us, Claire.

Yeah, thank you for having me.

So I have all sorts of questions for you because this is a topic I love. I definitely don't feel like I'm an expert at it though. So these questions are coming definitely out of curiosity as a product person. And by the way, if you're tuning in for this and you have questions for Claire, there's a little section called Ask a Question. Just list your question there and people can vote on it. And I'll be monitoring that throughout the session. Sometimes I'll pull in questions in the middle of the session. Sometimes it'll be towards the end. All of you are already introducing yourself in the chat section, so that's great. Keep doing that. But if you have specific questions for Claire, it'd be great if you could add them in the ask a question section. But before everybody starts flooding that with questions, Claire, I thought just to start off, You know, I feel like experimentation, especially for product people, it's something that product people like. I mean, it's something that absolutely helps us. I feel like a lot of times on the startup side, it's especially popular. You know, it's a part of the whole lean startup movement. I'm just curious your perspective. Why is experimentation so important as a product person?

Yeah. So as a product person, I started using experimentation. I actually like to tell The first A/B test I ran, I work at a company called Uship. They are an online shipping marketplace and they help customers ship all sorts of bizarro things. And one of the things I was doing as my first A/B test was trying to make shipping horses easier. Apparently, there are about 19 steps you have to fill out to ship a horse. I was like, hmm, what if we made this 6 steps? And that was my first taste of really what experimentation can do is it can help improve your product, it can help improve your customer experience, and you can validate as a product person if your ideas are good or bad from a business perspective through data. And so I think that's just really powerful from a personal perspective. It really inspired me as a product manager in my first job to really take my ideas not as good or bad, but as something that could be tested. And I've worked at a series of startups at varying stages and large enterprises, and I'll say that experimentation was at the center of the most excellently run and then highest performing companies. So take, you know, Claire's brain dataset as, as your sample set. But really, I saw the difference between companies who put experimentation at the center of what they did and how they performed and companies that didn't. And why I think it's so important is it really allows you to diffuse opinions, even your own opinions. It helps you eliminate guessing, which is always fun, and focus on really what, what delivers value for the customer. So I don't think it's just A/B testing. I really think it's a way of running your business differently and what your mindset is towards growth. It's how, as a leader, you think about making decisions. It's not just this like, top-down edict, but it's a democratized process where anyone can have an idea or hypothesis. You can experiment on it and then you can validate it or not. And that's how I see the best companies in the world run their businesses. It's how Facebook runs their business, Amazon and Netflix, all these amazing large companies that we would all aspire to build and grow is how they run their organization. And so if that's how they do it and they put experimentation at the center of what they do, why wouldn't you do the same? And then I think kind of selfishly and practically, I think it delivers some of the most straightforward ROI of anything you can do as a product person, right? You do a thing or you don't do a thing, you test it and you know whether or not it makes your business better or makes your business more money. I see very few things that are as simple and straightforward as that. And especially when you bring designers into the process, it's a really nice way to also put UX and design into the business conversation. And so I think there are nuances to that, but it's really the great part of experimentation is you can prove the impact you made. Yeah.

Which I know as, you know, I've run product for a couple of groups in the past. That's important when you're talking with your CEO or the management team. Sometimes if you're talking about different opinions, their opinions usually hold more weight, but if you're coming with data, That's a different story. So I like that.

Yeah, I kind of, I kind of hate the phrase HIPPO, although I know it's pretty common, the highest paid person's opinion. And I always say like, you're never going to get rid of that opinion. They're always going to have that opinion, but you can diffuse it by saying, well, how would you experiment on that? How would you prove that? How do you suggest I experiment that? And sort of soliciting a more experimentation mindset from your leadership.

Yeah, no, that's, that's a great way to think about it. I like it. And, you know, I had mentioned before, you know, kind of the whole Lean Startup mantra, you know, experimentation is a part of that. But then you named some really big companies, companies that are at massive scale like Netflix, Amazon. Do you feel like it is just as important, maybe even more so for those big companies versus the startups, or, you know, does it have a place in both? Might be a leading question there.

Yeah. So I know that you have me here on this lovely webinar, but I'll let Jeff Bezos speak for it, which he says the success of Amazon is based on how many experiments he does per week, per month, per day. So it's really critical for Amazon's success, which is an extremely large company, that they're experimenting not every year, not every month, not every week, but every single day, and to the tune of tens of thousands of experiments going on at any one time. So I think experimentation is absolutely, absolutely critical across all stages of a business. But I believe in fact it can be more important for established companies because it helps you maintain your competitive advantage and fend off all these new digital disruptors, some of whom may be in the room today. And then I think the other thing is larger companies can get stuck in the way things always have been done, right? So you're, you kind of have this, this state, the stagnation, and experimentation can jolt those teams out of kind of the way business has already operated and open up some pretty exciting opportunities.

That's great. Well, let me ask you this then. How does it differ? You know, when you're that startup where it's, hey, it's just 2 or 3 of us in a room, we're just in the very stage, we don't really have customers versus I don't know how many customers Amazon and Netflix has. It's probably in the hundreds of thousands, you know, how, maybe more than that. How, yeah, so like how does it differ when you're at that scale stage?

Yeah, so I don't know if the audience knows this, but I joined Optimizely about a year ago through the acquisition of my company, which was about 4 people at the time the company was acquired. So I have very recent, early, early stage startup experience. And so know that process of, you know, trying to figure out market product market fit. What is this business we're actually building? And I think Startups have a lot more flexibility in what they can experiment with. As an early-stage startup, you know, you can experiment with what the business even is, right? You can change your entire business model, you can change your go-to-market channels, you can change the product, you can change who you're selling to, you can change basically anything at early enough stage. So that's really exciting. You do some pretty big things. On the downside, most startups don't have as much traffic or sample points, so they might not be able to reach the testing velocity that larger companies can reach. I think larger companies have less flexibility on what they can experiment because the outlines of that business are a lot more firm. You know, they can, you know, that being said, I think, you know, companies like Amazon experiment with entirely new products, entirely new channels every day. So it can be possible to do bigger things. But the positive benefit for larger companies is they have more traffic. So they have the opportunity to do tons of testing. So their testing velocity is really high. On the downside, they need better governance and oversight because a lot more people get involved, a lot more money is involved and people really need insight into what's being tested where, you know, legal gets involved, executives get interested, there are stakeholders that are your peers, the board might even ask about it. And so you have to do all this orchestration across your organization that can get pretty complicated. So for large companies, they get that right. It's very valuable, but that is complex.

Do you? Well, because I can empathize with the product people on that side. I went from very, you know, kind of that 4-person startup to a company with many employees, so to speak. But we had millions of users and there was a lot of money coming through the system. So it was, you know, when I would come up with ideas for experimenting, it was like, yeah, this is pretty risky, Mike. I don't know that we can really do this. I guess, what advice would you have for somebody like me? Like, if I were to rewind back a few years where I know this is a good thing, I know this can help the company, but there is that trepidation because there is more risk involved when, you know, you're at higher revenue stakes. You know, there's more people using the system. Do you have any advice for people that are at that point?

Well, it's so fascinating to me because I think that not experimenting is riskier. Actually, I worked at some, I'll say unnamed, very, very large company, and I was told, you know what, Claire, You're really too obsessed with testing and you just, you need to make decisions from your gut. And I was like, oh my gosh, I just like, I don't think this company is going to go in the right direction. That seems extremely risky. Like, I think I'm pretty smart, but I don't think we should be, you know, betting the company on my guess. So I think not experimenting is riskier. And I think you can paint that portrait for your executive or leadership team pretty easily. I know tons of companies that have launched new products, they've done redesigns, they've even done little tweaks. That have lost them millions and millions of dollars. So with a test, if it performs poorly, or maybe you do like a monitored feature rollout, you can pull the plug really quickly. So you can say, oh, I'm monitoring the data, this is actually doing worse, let's turn it off. And so you can avoid most of the downside. It's much harder to undo a bad decision if you don't have that infrastructure or that experimentation culture in place, right? Because The new decision set, it's usually technically implemented. People don't— haven't ever thought of like pulling the plug and rolling it back based on data. And so you end up losing a lot more with not experimenting. That being said, I think it's fair to say that experimentation— people make mistakes, people launch experiments that lose. And that's where I think governance and oversight really come into play. It's important to know who is monitoring the results. What are your rules for stopping, right? Something out of the gate the first day drops your conversion rate by 30%, even if it hasn't quite reached statistical significance, where are you from a risk tolerance perspective on letting that test continue to run? If you set those rules ahead of time and you have somebody monitoring and responsible, those are all reasonable things that you can mitigate with the right process.

So that— I was just going to ask, like, when you say governance, that's kind of what you really mean is having those set of rules and just somebody making sure that you could Hey, we're either hitting this or we're not. Is that right?

Yeah. Yeah, and you have visibility into it, right, across the organization. And so you know what's running, you know where, you know what the rules are, you know who's responsible. All those things are in place so that the very minimal risk of losing an experiment, or maybe an experiment that has a bug in it, is covered because you have proper process in place. And if your velocity isn't hundreds or thousands of experiments a year, that, that, that process is probably not too onerous on your organization. Got it. I—

a question just came in and I might as well ask it now, sort of on topic, but sure. It was, is there a difference between and prototyping? And maybe I'll add, you know, or is prototyping like, is that potentially one part of experimenting or one form of experimenting, creating prototypes and putting in front of people? Could you talk a little bit about that?

Yeah, absolutely. So I think this, this term experimentation is a fairly broad one. At Optimizely, we specifically mean it to refer to A/B testing, multivariate testing, and product rollout. So sort of these mechanisms by which, with your production users or your real users, you validate an idea with data and statistics. So those are the kind of capabilities that we talk about when we think about experimentation. But absolutely, there are a lot of lighter weight data gathering methodologies that you can use to get early insights into what's performing or not. Those probably won't come with the statistical rigor of something like an A/B test or multivariate test or a rollout, but they are valid in getting early insight into how your product's performing. So those sorts of things would be prototyping, getting, you know, user testing, early user testing, user surveys. All those sorts of things are little, little experiments. They're not statistically valid, but they are ways to gather kind of qualitative data about how your product or design would perform. I think they're worthwhile things, and then I think they're absolutely something that should feed into your ideation queue. So once you prototype something, gotten relatively positive feedback, well then it's how do we productionize this and test it with real-world users? And then how do we circle that back to the next next product we, we prototype. So I think it's a really important part of the process. It's just the distinction I would make is between sort of statistically and scientifically rigorous testing and experimentation or data gathering in sort of more of a qualitative way.

Yeah, that's, that's fair. Let me ask you this, like, let's say I am, you know, I'm going to put myself in the, in the shoes of one of our attendees who might be working— actually, I saw a couple people in the chat, you know, naming companies that they work for. Some of them are big companies. I would say that they're at scale. And let's say I, I buy into this. I want the company to be, you know, to have an experimentation culture, to, you know, use this so we could build better products. But, uh, you know, maybe this isn't the way that things have always been done here.

Yeah.

What advice would you give to somebody like me that I buy into it and I want to get started, but I'm just not sure how to help my organization get started with all this?

Yeah, so I think it's really a cultural shift. So it's moving hearts and minds and people's opinions and how the team does work. And so there's this really simple trick you can do, which is just start asking, how did you test that? And if the answer is we didn't, you can say, well, how would you test that? Or how would you— how did you validate that that was a good idea? How do we know this is adding value? And that's a really good way to just get people to start thinking about, hmm, I actually don't know if it added value, or I didn't actually think it was possible to focus on something like this. And then as people come up with new ideas, it's always the question, how would you test that? Even here at Optimizely, I run our product team. We have lots of product managers and they have debates about the best way to implement our product or the best thing for our customer. And the way I diffuse all those debates are, how would you build a test to validate? This? How would you build a test to know? And so ultimately, those questions really start to seed a culture of data-driven decision-making, of experimentation, and helps you frame every conversation, not as like, I think we should do A/B testing and it's the most important thing and looks like top-down implemented. It's really just asking the questions and letting people answer them themselves. I think the second thing is to create a hero. So who in your organization, and it could possibly be you, will champion experimentation, socialize it, eventually be like the VP of experimentation? Who is going to be that one person that you're going to raise up in the organization and say, this person is responsible for, is going to always talk about experimentation? If it's you, you need to start owning that title and asking your boss, your peers, your colleagues for what you need. And then getting resources and visibility. I think if you have that one person that really cares about it, can take the organization pretty far. And then this is pretty tactical, but I think start with low-risk experiments. It's always the ambition of a brand new product person to come into an organization and say, A/B testing, and I'm going to do the most complicated, crazy thing. And one of two things happen. It takes forever and people are too scared of it and never get tested. Or it's neutral, like it's an inconclusive test. And this kind of the worst outcome for a first experiment. You spend lots of time on it and you don't have— I mean, I think you have stuff to show from inconclusive tests, but from an outsider, you don't have anything to show. So I'm always like, what could be a low-risk, almost win-win experiment you can run? And this is my, my pro tip: try to capture emails where emails aren't being captured. So you have a webpage and you have a newsletter, do an exit intent to capture emails. You're going from zero emails captured because there's no path in it to some emails captured because there's a path to it. And it's like, hey, I had this idea and look, it won. And now we're getting a few more emails a month than we used to. So if you can brainstorm those sort of low-risk win-win experiments just to start with, it might not be the most intellectually honest thing to do, but it helps seed excitement about experimentation.

Yeah, I like it. It gets everybody's feet wet, it sounds like.

Yeah.

Makes a lot of sense. I like that. I had another question come in that might as well, and it's, you know, how do you deal with prioritization when you have tests and we're trying to figure out what's right to do? That's why we're doing the test in the first place. But you also have a product roadmap and you have to execute on your roadmap. How do you sort of manage— are you managing both at the same time or are they one in the same? Sort of. Maybe you could talk a little bit about that.

Yeah. So my question is, why is your product roadmap any different from your experimentation roadmap? Right. You're rolling out new product, you're making product enhancements. Why aren't you testing those to validate that they're adding the value that you intend for them to add? Right. If you have a list of products you release without testing and then incremental tests you do on the side, that's not a true culture of experimentation. That's saying you're doing A/B testing, but you're ignoring the stuff that has big impact. So I would say really, your— every product that you launch should be a test, or you should at least launch versions of your products to test to get to the most enhanced optimal customer experience. So to me, those things should be one and the same. That's why we recommend that companies use both kind of UI/UX front-end design type of testing, as well as deep in their technical stack, integrating an A/B testing solution or an experimentation solution that allows them to experiment with products and that you demand that your product managers and your engineering team roll out everything as an experiment.

I can already picture you at the roadmap meeting. If somebody presents them the roadmap, it's like, well, has this been tested though?

Oh my gosh. I say it 10 times a day. Yeah. I mean, it's, it's tough because there's product managers say, well, this is the right thing to do, or are we really not going to do this feature? And that's not the point. The point is we should use experimentation to say, we did this feature, we intended for it to move this metric and it did, or it didn't. And you need to know that information to become a better product manager. So, I insist that everything is an experiment and I recommend everybody else does too. Yeah.

How about, you know, I know you're at Optimizely now and Optimizely, this is like the lifeblood of the company. It's all about experimentation. You're helping companies do this. So I imagine you've seen things that have worked really well, but also things that haven't worked really well. I'm wondering maybe you could talk about some of those things. Maybe there's a story that stands out that is a highlight or maybe mistakes you've seen along the way that will probably be useful for all of us to know so that we can avoid doing those same things?

Yeah, so there are 3 things I think about, big mistakes that people make when they're scaling experimentation. The first is not resourcing it. So, you know, when you're running 1 or 2 experiments per month, it's really easy for half of a product manager to consider it their job. But if you never add resources, you never give a dedicated developer or a dedicated program manager, it's never going to scale and you're going to be leaving a lot of opportunity on the table. So I see teams under-resource experimentation or not resource at all. The second thing I see is not maintaining discipline and velocity. People will run tons of experiments for 6 months in a row and then stop for 3 or 4 months and then run again for 2 months and then stop for a month. And because of the compounding effects of positive experiment results, you really have a lot of opportunity cost in not running experimentations. So at scale, I think you should be running an experiment every day. If you're not, you have a problem. And then finally, you asked for a story. I think not socializing and sharing results is such a huge miss. The wins from an experimentation program are not just the 30% increase you got in your conversion rate. Or the 10% lift in revenue. It's what did you learn? And often those results and those learnings are kept kind of in the best case in a spreadsheet or wiki, and in the worst case in someone's brain. And so I went into a customer when I was running my company a couple of years ago, and I said, okay, great. You've run 250 experiments last year. That's amazing. Could you give me a download of like, what were the best tests, worst tests, what happened, wins, loss, all that stuff? And they said, well, Joe knows. I was like, you better hope Joe doesn't get hit by a bus because that is a lot of value floating around in Joe's head that you can't share, that you can't spread across the organization. And so I think it's really important to create a repeatable system for managing experimentation results so that it can be shared across the organization. You can maximize value.

Yeah, that's great. Well, thank you for sharing that. I appreciate it.

Yeah.

I'm going to take one more question from the audience, and then I have just one question to close things out. But it's, you know, there might be some times where you're working on a, say, it's a new feature, but that you don't necessarily want everybody to know about quite yet, or, you know, you— something that maybe is a little bit secret on your end. Is there— are there ways to test and experiment with things like that when you're trying to not let the word get out about something in particular? Or how do you, how do you sort of weigh that?

That's really interesting. You know, I've always, I've always, I had my own startup. I've always overshared what our ambitions were, even if I anticipated they changed. I was never, I knew that I was never going to be outcompeted because somebody stole my idea. I was going to be outcompeted because somebody over-executed better than I did. And so I probably have the perspective that I'm much less fearful of, quote unquote, people knowing, because I've just never seen that the reason why a product or company or business fails. It's usually they built the wrong thing or there's not product market fit or they were outperformed in some other, other aspects. So one, I would say maybe reduce your fear. The second is there is a way to target or segment your experimentation. So you're working with a very narrow set of your customers. So we occasionally roll out products here at Optimizely where maybe we don't want to show it to our top enterprise customers that are paying millions of dollars a year because we want to test, field test it with companies that are paying us a little less, a little less risky, and then roll it out higher into the enterprise. So there's absolutely ways you can target and mix how you do experimentation. Again, you might not get to the statistics statistical significance you want because you're reducing your sample size. But if that risk aversion is really important to you, then that's something you can do.

I think that's great feedback. I appreciate that. And so time— we're pretty much at the end of our time. So time goes by really quick with these when I'm learning so much. So I appreciate that. But I'm just wondering if there's one, you know, if for the attendees, if they're going to walk away with one big takeaway, there's one big learning that you want to make sure doesn't fall out of their mind, what would you want that to be?

It's really simple. Always be testing. I just— if every product person in the world was running at least an experiment a day, we would have significantly better businesses. We'd have significantly better products. I think as end customers, we'd have much more delightful experiences. And so my challenge to everyone is get to a point where you're running every day. You have a live experiment. It's a really simple thing. You keep a calendar and X it off. Today I had an experiment or I didn't. It definitely makes a difference in businesses, and I've seen it make differences in careers. So I highly encourage consistent and everyday testing.

I love that. All right, well, thank you so much. I appreciate this, Claire. This session will be recorded, so everybody, you're welcome to, you know, check back here if there is anything that you didn't take a note on or anything like that. Feel free to share it with people. Um, you know, you could search— I know there's some awesome videos of talks that Claire's given in the past that's on YouTube. I've seen some of those, so check those out and check out what Optimizely is up to too. It's a great company. We've had Pete Kooman speak at Industry in the past, so we really appreciate it. And thanks a lot for being here, Claire.

Thanks, y'all. Bye.

All right. And we'll see everybody later. There's more Product Launch Sessions coming up, so feel free to register for those and also check out Industry. I realize I'm wearing the Industry t-shirt. That's actually why I'm a little informal today. So hopefully we'll see some of y'all at Industry in the future too. Until then, I'll see you at the next industry interview.