#50 – Manipulation Or Improvement? Is A/B Testing Ethical?

I remember once we worked with a company. It's absolutely horrible. Okay? But I have to mention this. They sold accessories for cell phones. Okay, that's fine. And like, you know, average order in value is like $10, $12. Now, it doesn't cost them a whole lot to buy those accessories. Probably not 10 cents. And they're selling it for 10. But they made their money on the shipping and handling. So you're buying something for $10, they're charging you $15 and $17 for shipping and handling. And the only way we found this out is because I placed an order on. Just trying to see, and I'm like, what? I'm sure I didn't spend 27 bucks on this. And we go back and we find out, and they didn't even tell us about this. The shipping and handling was just hidden on the website. My name is Khalil Sladeh, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me. Answering some of the difficult CRO questions, dropping insights like, it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today. Wherever you listen to your podcasts. Simba. Another week, another CRO Live. How was your week, Simba? How is it being a new dad? A new father?

So far, it's been good. I've been learning, like, a lot of things. I wanted to say that there's the good and the bad, but there's no bad. Because I was going to say the bad is less sleep, but it's okay. I can. I can manage that. There's the good and the great, so I'm enjoying it. It's been four weeks today.

Amazing. If you had told me, like, in a Hola. I'm like, oh, it's two weeks. Only four weeks.

Yeah, yeah. It's four weeks.

We're about to wrap up this year. I cannot believe that. It just seems like we started yesterday. Fascinating. We're talking today about the topic. And it's funny because when I posted about it, I said, really? In the past I've refused and I know people ask us questions and I just refuse to talk about it. So that I'm like, I guess today you gathered all the questions and I'm like, oh, man, we're going to be talking about. Sure, let's talk about it. Because I had certain views and they might not be very popular and I don't know, let's venture into the land.

And the land of LinkedIn. Last week that was the topic.

Oh, was it?

Yeah. I don't see a lot. A lot of people were talking about how ethical is it or it's not ethical. They're talking about a certain line that should be drawn when it comes to a B testing. But you're going to shed more light because we have four questions today.

Hold on. Yeah, before we talk about this. Okay. And before I start talking, what were the opinions and how did the. Even the conversation start? You have to fill me in because I actually did not follow that conversation.

I'm not really sure how the conversation started, but I think I saw a post from John. I think it's Johnny Longdon from Jennifer. Okay, so he was the one, like, who posted something saying, like, is a B testing unethical? And there were a couple of comments. A lot of people were saying that no AP testing is ethical, but there is some complexity when it comes to AB testing, the price, because there is a line that you have to draw there because sometimes you don't want to show certain customers, like, a lower price and then a higher price. To some, it was just, like, interesting to follow. So I just thought, like, maybe it would be an interesting topic also to talk about in the CRO level today.

Okay, I like it. I like it. But here's the thing. There's so many different topics when it comes to AB testing, and lots of them are practical. Maybe some of them are very tactical. Some of them are very strategic. Some of them, maybe ethics and a B testing are a luxury. Oh, it's an intellectual exercise. But we will go through that intellectual exercise because, yeah, there are some parts of a B testing that are interesting. Let's have at it. Let's have at it.

Okay, cool. So I'll shoot the first question, which is, in your opinion, are there situations where AB testing certain content or features could Be controversial. How should business navigate these ethical gray areas?

I feel over here that AB testing is shoved into this whole. Ethical or not ethical. What do I mean by that? Is it ethical to show wrong information on a website? Is it ethical or unethical to trick your visitors? Not through ab testing. Correct. You have the wrong promises, you're deceiving people, you're showing the wrong price. That's unethical. Correct. It doesn't matter. AB testing or not, you know, now you might say, well, okay, so what if I'm AB testing that? Well, guess what? A B testing is just the tool. Correct. That will show you which one works better. But if you are a B testing something, that's unethical. Yeah, the whole thing is unethical. And I think that is sometimes, most of the time, I would say that it's clear what's ethical and what's not. I remember once we worked with a company. It's absolutely horrible. Okay, But I have to mention this. They sold accessories for cell phones. Okay, that's fine. And like, you know, average order in value is like $10, $12. Now, it doesn't cost them a whole lot to buy those accessories. Probably not 10 cents. And they're selling it for 10. But they made their money on the shipping and handling. So you're buying something for $10, they're charging you 15 and $17 for shipping and handling. And the only way we found this out is because I placed an order on, just trying to see, and I'm like, what? I'm sure I didn't spend 27 bucks on this. And we go back and we find out, and they didn't even tell us about this. The shipping and handling was just hidden on the website. Completely hidden. You wouldn't know that. And their theory is most people buy the cell phone case once and then they move on. And they might then buy another cell phone case in two, three years. So they're like, hey, when somebody's coming, we want to maximize our profit. It's there on the website.

Okay?

But it's so hidden, by the way. And if you don't dig for it, especially in a time where most people expect that even if I'm going to pay shipping and handling, it's going to be very low. I didn't expect to pay 15, $17 on an item that cost me $10. It's there, it's hidden. We felt that was, hey, you're tricking people. I don't know if we went as far as to say unethical. But this probably was unethical, correct? Now, somebody might say this there during the checkout. It's not my fault that you don't really read the copy clearly. Well, yeah, you stuffed it completely on the side. You made the font really small. Okay, so you tricked me. All power to you once, but guess what? I'm not buying from you again. And then the E Commerce seller says, I didn't really expect you anyways to buy from me again. Okay. And in that case, we actually spent quite a bit of time with them, telling them, you're tricking people. And we think we can a B test this to show you that even having this clear would actually help you have more user satisfaction and is going to help your site. Eventually, we convince them to run a B test. And what happens, of course, when we run the A B test, Showing it clearly to people reduces conversion rates. Sort of like, hey, yeah, I'm going to be charging you $17 for an order that's cost $10. The A B test in this case showed you clearly that the ethical practice reduces your conversion rate. The unethical practice, which you already had on the website, does help your conversion rate. And I think all that we can optimize at that point in time is just to show the data to our customer to say, hey, nearly done. Ethical practice is really helping you make money. But is that the way you want to go about doing business? By the way, just to their credits, we showed the data and it was almost like a more than 15% drop in conversions. And their CEO chose to actually show it very clearly. He said, it's okay, yeah, it's a hit, we'll take it. But we'll be very clear. He was assuming that it's going to cost them a lot more to show. So he says, let's show it. And then eventually we start optimizing that shipping and handling charge. So he made the right decision, but for a long time they operated on the wrong decision. Going back to your question, ethical or not ethical, just ask yourself, are you tricking people? Forget about AB testing. Look at the designs, look at the flows, look at the copy that you have on the website. Forget about a B testing completely and say to yourself, is it ethical to have this on my website? A B testing is just a tool, correct to test that. That's when it's very clear. Maybe price testing is a little bit like people really struggle with that and they say, is it ethical, unethical? I have views on that, but we'll leave that maybe for the next Question. We'll see.

That's exactly like what I wanted you to answer. Is price testing ethical or not ethical? Like, how do you strike a balance where you don't cross the line when it comes to price testing? How do you usually do it?

It's funny. Audible, the audiobook company, is running a price testing campaign. How do I know? Because I have unsubscribed from Audible probably a couple of years. Because I would get all these credits, don't use them. Eventually I'm like, okay, let me unsubscribe. And then now some friends are doing a book club. I'm like, oh great, I'll join the book club. I just don't have the time to read the book. So I'm like, I'm going on walks. Let me just get the book on Audible. Okay, long story. I Google Audible, I click on the first results and I see the Flickr effect 3 months, 5.95. And then I'm like, oh great. And then flashes and then goes, you can subscribe now. $14.95. So I know that they're testing. Is it price testing? They're testing an offer corre. But they've showed me the 5.95 and I'm like, I want that going Cognito mode still. I'm hit with the $14.95. I knew that there's price testing going on. I just went on Firefox, hit it, looked for the website, and I landed finally on the 595 subscribed there. Is it ethical for a website to test price? I think merchants are always testing their prices. I think merchants are always trying to find the optimal way for them to to price an item. I think there are many departments that are doing that now. In the past, it did not happen as a b testing. So it happens probably sequentially. Correct. We're going to price this product at $100. Oh, the demand is really not high. Maybe we need to reduce the price. Correct. So we're going to go down to 90. So everybody who was coming to a store was getting the same price. Nowadays, because of the power of ab testing, two people could walk into the store. Correct. Could walk into an e commerce website. One of them sees the item for 100, the other season item for 90. I don't know. What do you think? Is that ethical or not? I don't know. What do you think? How would you feel if you are the consumer who paid 120 and I come on and say, oh, Simba, you paid this 120 for this item. I just bought it for 50 bucks. I'm exaggerating because sometimes you make the difference so little that oh, no big deal. But yeah, I bought it for 50. You bought it for 120. How would that make you feel? Do you think that's ethical? What do you think?

To be honest, I would feel I've been robbed of my 70 bucks. I bought it for 120. You bought it for 50 bucks. I would feel robbed. So I would think that's so unethical.

Okay, so I'm going to ask you a different question. I agree with you. I would feel I would be very annoyed and I might return the item and at least argue with the store if it's not too late. Let's take another example. You buy an item, you buy it for $120 and three days later the same store now is selling that item for 50 bucks. So anybody who walks into the store today is going to pay 50 bucks. You just paid 120 for it. Ethical, not ethical. What do you think of that?

Usually that's what happens during Black Fridays, Cyber Mondays and stuff. But still at the same time, I would still feel robbed or maybe wanted maybe the company to tell me that this item will be going on sale in the next three deal or something like that.

Yeah, I don't know how I would feel. I'll be probably annoyed. Again, I don't know if it's ethical or unethical. That's really a tough question. And I'll give you a real example of where this happened. Tesla, Elon Musk's car. People were buying those Teslas for $100,000. I think you can buy it right now for about 70, 75,000. Imagine you are somebody who just bought a Tesla for $100,000 and the next day the price went down to 90, $15,000. And if you wait another up, it's down to 80,000. So the price keep on going down. Is that ethical? Unethical? And we're talking about a ton of people who are going through that. So it's interesting thinking about price and price. This is variation. This is not AB testing. Again, is it ethical? Isn't it ethical? I don't know. It's tough. Would I be annoyed if I'm that consumer? Now mind you, by the way, I always try and distinguish certain things. Sometimes people use the word unethical on something and I would tell them it has nothing to do with ethics being ethical or unethical, but it's just very annoying. You tricked Me, sometimes people use some words in context and I'm like, maybe this is not the right thing. Now let me talk about AB testing the price. I don't think there's any issues with that. We've done so much a B testing the price, a B testing and we've worked with companies and I remember one particular company where their average order value was 70 bucks. And we kept on increasing the price of the items until we hit 105. We did not work on conversions, did not impact conversion rates. We managed to increase the site revenue by 25, 30% through price setting. How would somebody who just paid 100 bucks feel if somebody else is buying it for 75? In the beginning, if you had come to the website a month ago, you probably would have paid 75. And if you buy the item during Black Friday you probably pay the 75. It's timing that's number one. But just to avoid the whole ethics and non ethical if you're paying a lot of money, if you are paying the higher price for the item. One thing that we've done in the past is after somebody actually places the order so he's seeing the higher price. Correct. And on that order confirmation we go ahead and apply an automatic coupon that discounts the price to the original price. You see how. So everybody is paying the same price. You don't know that your price is going to be discounted. You're just really surprised at the end to say oh, actually we've applied like this and you can do some copy around that. So everybody is happy. But we're able to test that copy to say, oh yeah, okay, the 105 actually did not impact the conversion rate and we actually achieved higher average order value. So everybody wins. Now just think about this. We've done this experiment, okay, 105 is perfect. Now everybody's paying 105 for the item. The examples that I gave you is when the price you've paid 105 and it went down, how about the other way? And you're going to be annoyed. How about you walk into the store right now, you're paying $105 for this item. But I tell you, oh, I bought it a month ago and it was only 75. Are you as annoyed? You're probably not as annoyed, by the way. Oh man. Again, sometimes we mix ethics with our personal feelings and becomes this jumbo, this mess of things and it needs people really to think it.

Yeah, that's a good point. Because if you look at most of the products that are sold especially online. People do pre orders at a cheaper price, like they buy them when they pre order, like at a cheaper price, but then when the product drops, the price will get increased. So I guess it is what it is, right?

Sometimes it is what it is. And you can test it and you can do the method that I just mentioned. And that solves. If you think that there's ethical issues around that. That's the standard practice for us whenever we do AB testing on price A testing.

Okay, so is obtaining explicit consent from users necessary for a B testing?

Oh, gosh. Our friends in Europe will say, of course you have to obtain consent. We have what, JGDPR in the eu, we have CPA in California. Listen, when it involves private data, pii, personal identifiable information, I think you should definitely obtain consent. Now if I'm running an A B test that does not involve any pii, no personal information is there, and most A B test data is anonymized, we're looking at the trend. I'm not looking at a particular user. I actually don't care to get somebody's consent. Now there might be laws, and again, there are people who are very extreme when it comes to that. And then, by the way, most of those laws. Here's the thing, most of the laws and the rules that you see are a response to a situation. It's not like somebody just woke up in the morning and says, oh, we should add this. Usually something bad happens, so you create a law to respond to that thing. Otherwise no one would have thought about a weird law. The problem is, okay, so we started with a bad situation, we created a law for it. And since we're in the process of creating laws, what do humans do? Let's add this, let's add that. And they add so many other things to the law that makes it so, like, okay, is this even reasonable? Or it's not reasonable. And I can give so many examples. And not around personal information and consent, but so many different ways where I think, I'm like, okay, the people who are legislating don't really understand how the Internet works. And we see that by the way, we see it in the US When Zuckerberg goes in front of Congress and I'm like, what kind of questions? And I understand that the congressmen and congresswoman in the Senate don't understand how the Internet works. And it's their staffer who wrote them the questions. But a little bit of education so you don't look like an idiot. But I digress. Anyways, I don't think you should ask for consent unless it involves personal identifiable information. I can't think of a specific a B test that relies on personal identifiable information. Correct. I think just AB tests, at least the way we do it is just looking at general trends. The data is anonymous. So that's the way I think of it, I'm sure. By the way, when I saw this question, I googled, I'm like, let me just see what people have to say. And gosh, some of the stuff that people wrote, I'm like, okay, so you just wrote an article just to rank for ethics and AB testing because you read it. And I'm like, okay, so half of the stuff that you mention over there has nothing to do with ethics or you're just throwing random information. You're just showing, okay, here's 10, 12, 15 different points. And I'm like, okay, half of those have nothing to do with AB testing. All power to you. You managed to rank really up on Google. But no, it has nothing to do on Google. And it's funny because when you Google ethics of a B testing, the very first example I think was for a university professor. And I'm like, oh man. For the most part, he doesn't really even understand a B testing. And you can see from the comments that even the people who are looking at the post and commenting about it don't understand AB testing. So that's where I would stop when it comes to consent.

Okay, so at what point does this customization cross the line into manipulation? And how can businesses avoid like this ethical dilemma?

I'll go back to the fundamental principle. Forget about AB testing. Look at what you are doing. And are you manipulating people? Are you showing the wrong information? Are you trying to manipulate their emotions? That's just not acceptable whether AB testing is involved or not. And even more interesting, think about now and the use of AI. We're talking about it. We're not there yet. We're heading really very fast towards that. So AB testing is not involved, but we're using AI to drive users to take a certain action. And AI, if the goal of AI is to really just get users to take a certain action, AI doesn't necessarily understand ethics and humans struggle with ethics. I don't know how much AI understands ethics nowadays. We hear lots of talk about the need to regulate AI, and that's also a little scary. So many things are just scary and just very confusing. But in this case, again, it's the algorithm. And the algorithm is constantly adapting. The YouTube algorithm will show people videos that maybe are harmful to society, but they generate more clicks. Facebook algorithm will push certain stories up, remove other stories down. Same thing with Twitter. It's the algorithm. ABTC has nothing to do with it. But again, are you crossing the line of morality and ethics? Yeah, sometimes you might be. Now let's just add on top of that another component that if you are a commercial entity, if you are the Twitter and the Facebook, your goal ultimately as a commercial, as for profit business, to maximize the revenue and the profits for your shareholders. I'm not saying ignore ethics, but sometimes also don't go into an extreme level of ethics. Anyways, I'm sure some people wouldn't like that. What I'm saying they're like, what business should ignore ethics? No, they should not ignore ethics. And I think we've seen many instances of corporations ignoring ethics and trying to maximize the profits. And I think anybody who does that eventually pays a very expensive price for it. I'm more of the opinion that the market forces will always correct and will punish those who try and trick people. Yeah, you might be winning in the short term. Long term you're going to lose. I don't care. Eventually you can trick some people some of the time. You cannot trick everybody all the time. Correct. And I think the problem is like when government gets involved and you're trying to regulate, you try and fix one problem, you create a different problem, and so on and so forth. Let me mention this thing to you, which I mean, we started talking about Teslas and the discount and pricing that they give. The reason Tesla discounted this price because in the US we passed a law and says, hey, if you're buying a car less than this price, then you get a government subsidy, like a tax refund of $7,000. So as you're buying something, let's say if it's for $100,000, you don't get that tax refund. It's actually not tax refund, I forgot the name that we use. But this is basically $7,000 that comes to you. But if the price is less than 75,000, you get that. So really, although you're buying a car for 75,000, guess what happens? You get 7,000. So you're buying it for 68,000. Great. So they cut the prices just to encourage people. So that's number one. Number two, okay, so people are doing that. Guess what happens? Because government was trying to encourage people to have electric cars. We don't have the grid to support that many people with electric cars. So it creates a different problem. And manufacturers of just traditional auto cars, gas and whatnot are struggling now that workers there are going on strike. So we fixed one problem, we created many other problems, and so on and so forth. And that seems what governments do. So that's it, man. I stop here.

Yeah. So the next question that I was going to ask you, I think you touched on it, right? About ethical frameworks and principles that business should consider when implementing AB testing. So I just want to get like, your thoughts on something different because I can see an overlap between talking about. So I just want to ask you something, like, about Blackhead CRO, or maybe applying dark patterns in a way or to manipulate users into taking specific actions. What's your thought on those kind of dark patterns? I visit a lot of websites when I'm doing like certain research, maybe about a blog that I'm writing. Then I see, okay, download this thing or this ebook or download this. Yes, I want it. No, I don't want it. I want to remain like that kind of a negative thing that they put. So what's your thought on that?

I always say, no, I like to remain ignorance. And I click on that button. Just leave me alone. That's an obvious example. Correct. Like, you know, nowadays I'm like, who still uses that in 2023? This was really interesting and fascinating back in 2013, 2014, when it first started. And I think it worked for the first couple months, by the way. And then everybody's copying everybody else and you see it and it's like, okay, this is a bit much. But there are dark patterns that you see, and then lots of them. By the way, this is an area. It's around what I call incentives, urgency and scarcity. Oh, order right now so we can ship it to you. That's probably real. But scarcity, we only have two items left. And you always have two items left, no matter what. It's like, I come tomorrow's two. I ordered the two items, but they're still there. There are dark patterns around that. And again, I go back to people will figure this out. And when people figure it out, you're not going to win. I think travel sites use a lot of dark patterns, and they continue to use a lot of dark patterns. And they're trying to trick people to make a decision to buy an airline ticket or make a hotel reservation. And the consumer nowadays is fully aware of all these tricks. Most consumers are, and you got punished by the eu. And I think they find hotels and Expedia and all these travel sites, but the consumer is so fully aware, by the way of what's going on that, okay, so now we use different sites to compare prices and you're not going to trick me. And I'm sure by the way, somebody will figure another dark pattern and it will work for lolbit and then everybody else will copy it and then just like people get annoyed and becomes well known and so on and so forth. Principally, we don't use dark patterns. And what we do, we avoid that because we start with the principle that we believe every user is fully aware. We're not putting an offer to a user who is dumb and who's not aware. We start with the assumption that our users are very smart. We treat them with the respect that they deserve. So when we start with that assumption, we say, oh, we're not going to trick people. We're not going to do this. This is stupid. Because if you assume that your user is smart, you don't use those dark patterns. That makes a lot of sense.

And just to add this, maybe for anyone listening who's interested in this, we have a gold article on the Invest blog around Blackhead CRO and the consequences of Blackhead CRO. So all that stuff, if you want to read and find out more about it, just go on the Invest blog, Blackhead CRO. You'll find it there. I think those are the questions I have for you.

Yeah, okay. Simba, by the way, maybe, maybe I'll just talk. I didn't imagine that we'll do. Also, almost half an hour talking about this topic. I'm like, do I actually have a lot to say about. I guess I do. Thank you for listening. Until next time, let's ask for a favor. If you enjoy the CRO Live hour, wherever you're listening to your podcast, share this episode or any episode that you really liked with somebody else who you think will enjoy it. We would appreciate that. And until next time, happy testing.