#15 – Price Testing: How To Get It Right

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts.

So now we're talking about price testing. We don't do a whole lot of price testing, just some thoughts around price testing. There is very little AB testing software that does price testing and statistical analysis in a meaningful way, whether it's using Frequentist or if using Bayesian bias on statistics. Most of the testing engines might have ports or different variations the RPV revenue per visit or they might report the aov. But determining a winner is not based on price, not based on the RPV or the AOV or based on this variation generated this revenue for me and this variation generated this revenue for me. Determining the winner using Frequentious or Bayesian usually is looking at conversion rates correct. Whether somebody converted or not. What's the conversion rate for this variation?

We usually rely on a conversion metric.

To determine a winner. We look at the revenue metric, but this is secondary metric for us. The only time we're very concerned about the revenue metric, if the variation and the control have a at least 20% difference between them at that point in time, we're like, huh, I wonder why this happened? But that's not enough either because at that point in time we want to look at the actual raw data for the AB test, remove what we call extreme orders. You're running a test, a control versus a variation. The average order value is 100, but somebody decided to come to the control and place an order for $100,000.

Guess what?

The AOV for your control just shot up and you look like your variation just lost horrendously. Well, no, it's a single order. It's an extreme order. That's the reason you see that huge variation in the average order value or the revenue per visit. That's the reason you have to be careful. You have to be able to downlo the data, remove those extreme orders on both sides above the average order value, below the average order value, and then do your analysis. So is it based on a frequentist? Not really. It's more a secondary metric that we use to help us make the decision. But it's not the primary metric.