In this interview, Dimitri (guest) discusses his journey with experimentation in a marketplace business, tracing it back to early tests on fees, site copy, and checkout flows. He explains how the team initially used Optimizely for A/B testing, but after Optimizely shifted to an enterprise‑only pricing model, they stopped testing until they built their own in‑house experimentation tooling. Dimitri emphasizes the need to balance intuition with data, noting that not every idea warrants a test. He illustrates this with a detailed case study of a "Make an Offer" feature: despite seeming intuitive, the feature reduced conversion rates because it increased the time to close a sale. An A/B test revealed that hiding the button improved conversions, but the team had already invested a year in optimizing the feature before pulling the plug. The conversation underscores the importance of testing before building, avoiding sunk‑cost traps, and using data‑driven decisions to guide product development.
Hey, Dimitri, do you want to come on?
Thanks, Mike. I would love to hear about your journey with experimentation as someone that came from a scientific background. How did you get started and when did that happen in your kind of growth? What were the— how— what did your first, like, few tests look like? What were— what were you testing? How, you know, how was the randomization set up and kind of maybe where are you now? So sorry for a barrage of questions.
That's awesome. Yeah, we started, I think, very early on, if I remember correctly, like, you know, and I, it probably actually started with us testing our fees because we were just trying to figure out, like, how to, you know, how to just make more money, frankly. And so, but I just didn't want to just change the fees without understanding the impact. And so just, it's also coming from, like, a technical background. Like, I knew that it's important. the value of testing. We— this is dating me, but if you guys are familiar with Optimizely, they had at the time, you could use it as a small company, a very small company. Now they're only focused on like enterprise, very, very expensive. They— we've been out like now we can't afford Optimizely because I think like they're dealing with like really large corporations. And I think, yeah, we started with like testing our fees and we started testing like, you know, just even simple things like copy on the site. We tested even checkout flows, like different flows just to understand like how people react. And then when Optimizely decided to change their pricing structure, we kind of were like left in a void because we didn't know what to do for a while. So we stopped doing testing. Then we just brought it like we built our own tooling. for some time and we've been, you know, we've been able to just continue experimenting. We still do. I would say that like we probably can do more testing than we actually do, but it's just, there's a level of, you know, don't do testing for a lot just for the sake of testing. Like you have to decide when it makes sense. Like if it's, you know, when would it make sense for your business? Is it worth the time? Because of course it takes time. Like if it's something that's intuition, like listen, like this is just like a no-brainer. do it. But I'll give you an example. We launched this feature called Make an Offer, which I think, you know, I'm sure people understand, right? It's intuitively, it makes so much sense. And I still am upset that it hasn't worked. The kind of like the headline is it didn't end up working at the end. We launched it with that. Actually, we built the feature. So this is a lesson of what not to do. We built the feature because it made intuitive sense, like all, most marketplaces have it. And we launched it and we just saw our conversion rate go down. And it's not because the feature was buggy. It's a great feature. But what we saw was like, is essentially increasing the time to make a sale. So, you know, and so then we ran an experiment where we did an A/B test where we showed the button to some people and we didn't show it to other people. That's probably where we should start. We should have started, right, instead of building the full feature. And we look at the data was undeniable. Not having the button improved our conversion rate. We spent a whole year optimizing that feature and ended up just having to pull the plug. So very painful, a very painful story, but it's just the power of data, right? And so you kind of like, it's sunk cost at the end and just do what's right based on the data.
