#126 Make Your A/B Testing More Effective and Efficient

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey. On datacamp.com. You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization, join us for in depth discussion discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Welcome to Data Framed. This is Ritchie and today we're talking about A B tests. Much of the time in marketing, product design and website optimization, you get tasked with making improvements, but it isn't obvious whether your ideas will work. The key to success is to run an experiment and use data to make decisions on how to proceed. A B tests are the most popular type of data experiment, so they are spectacularly important for every organization that has a website or a product or a marketing team. Personally, I think that a B test is a dreadfully dull name for such an important type of data experiment. However, my campaign to rename them as Data Driven Face Offs is yet to gain much headway, so we shall have to stick to the existing terminology. Sharing her extensive experience in data experimentation is Anjali Mehra, the Senior Director of Product Analytics, Data Science Experimentation and instrumentation at DocuSign. She's worked in marketing analytics and customer analytics and product analytics, so she's got great insights into how A B tests can be used throughout all areas of business. Let's see what she has to say. Hi Anjali, thank you for joining me here. Just to begin with, can you tell us a little bit about what DocuSign does?

Sure. And first of all, just thanks for having me here. Happy to be here on your podcast. So DocuSign now, I think at this time is a household name. It's a market leader in E Signature. We pioneered the E Signature technology and today DocuSign now helps organizations connect and automate how they prepare, sign, act on, and manage agreements.

Overall, you're A senior director. So what does that mean? What do you do at your work?

Yeah, sure. So I'm a senior director of analytics at DocuSign. I lead three different functions within the analytics umbrella, primarily customer analytics, experimentation. That's something we will go deeper into today and then instrumentation. So my role primarily I work with the Product Org product leadership team. I would define the role of me and my team in like a couple broad categories. Number one, defining the KPIs. Now what are the key metrics that matter? We partner a lot with the product team to help them define those. Number two is ongoing now monitoring of KPIs and trends, really with the lens of finding business opportunities to drive growth for the company. And the third piece, and I would say really very critical and strategic piece is experimentation program. And how do you now really build a very robust experimentation program where there is constant innovation and ideation and now hypotheses being built out constantly to be tested and the experimentation program helps in testing out those hypotheses and then really feedback loop back to the business based on the results of those experiments. So those are I would say three key roles or components of my role at DocuSign.

Can you just tell me a little bit about what are the main goals for analytics at DocuSign?

Yeah, absolutely. So I would say the vision for the company and for Analytics Org is aligned now, which is building a world class product and user experience informed by data and customer insights. Now as a part of that vision, again, the key enablers for that vision, which is driven by data analytics, is now number one, now building the right KPIs for the business to monitor the success of our core product and anything additional we are offering on the product suite or on the customer experience. The second piece is really what I mentioned earlier, again now ongoing. How do you build that culture within the company of keeping a very close eye on the key KPIs and now monitoring them on an ongoing basis and really with the lens of finding business opportunities to drive growth. And then now experimentation is another now big enabler for the business. What I referred to earlier, and then I would say largely now an overarching goal is to really enable a data driven culture within the company through a combination of measures, e.g. data literacy programs, building self service tools and dashboards and now bringing data front and center of every conversation, be it about investing in a new product or starting a new campaign, or just like measuring the impact of various investments.

Brilliant. The idea of a data driven culture is something we talk about a lot on this show. And so it is very important to not just go about and have individual data projects, but have the idea of using data be sort of spread throughout the whole company and everything you do. So that's brilliant. But I'd like to talk about experimentation. So when I think of experiments, it's kind of the idea of like someone in a lab coat messing about with Bunsen burners. But what does an experiment mean in the context of data?

Now, the way you can define experiment in the context of data is I will try not to get too scientific here. At the core of it is really it's a process to support or refute a hypothesis. So you always start with a hypothesis and then you run an experiment to either prove the hypothesis or disprove the hypothesis. In my opinion, also, I would say this is one of the best ways to really determine any causal impact to any feature or factor that you are testing. So, yeah, in a nutshell, that's how I will describe experimentation. To measure, measure the impact of any feature or variable that you're changing in the business.

Okay. Supporting or refuting hypotheses. That's a really great summary of the idea of an experiment. So A B tests are maybe the most popular type of data experiment, at least in product analytics. Can you tell us a bit about what is an A B test?

Yes, I think it's the most common nerve testing used in the business. So again, really, to simplify, like a B test, now the name came from, like, now you have two different groups. So ideally you split your audience into two different groups. Let's call them group A or group B. Again, a more common term in the business is like now a control and a test or control and a variation group. And how you split these groups is also important. Again, a very common method is random sampling. So you randomly split your audience into two different groups. And then the idea is the first group, let's call it control or group A, you expose that control group to your current experience, whatever your state of now, whatever your business as usual feature is on the side or anywhere, and then you expose the second group to the new experience that you're trying to test. So again, I'm going to just like now recap. In a nutshell, you split your audience into two different groups, let's call them A and B. One group, you keep the experience the same, and the second group, you expose them or you give them the new experience that you're trying to test. So what this does is really, it gives you a very controlled Environment to test the impact of the feature or the exposure that you're giving to the second group.

Okay, so comparing two groups, I've seen one of the previous companies I worked at their A B tests Champion Challenger test. So the champion was the thing that you're doing at the moment and the challenge was like this new thing that you're trying to test. Is it better or not? I thought that was like quite useful for understanding like what these two groups are. A B test, a bit of a kind of boring name.

Yes, very interesting. I've heard that too. You know, I think you write control and test and control becomes very scientific. I've heard that Challenger now terminology a couple of places. It just makes it very exciting and competitive, I think. Adds kind of a gamification element to it. Like, hey, that's a challenger. Let's see who wins. Right. So it's interesting.

Yeah, I like that gamification of your data experiments. That's pretty cool. All right, can you tell me a bit about how you use A B tests at DocuSign?

Now, as I mentioned maybe earlier, experimentation or A B testing is a very critical strategic component of the business. A B testing plays a very significant role in informing the product roadmap, DocuSign, any campaign strategies and any other major investment decisions. Now it is a very key component of really any kind of strategy and roadmap for DocuSign. And I would say just more generally, broadly, like building an A B testing culture and really adopting an A B testing program offers number of different benefits. It starts with data literacy and education around like what testing is. It's really a mindset and a culture change now rather than just sort of setting up a tool and running a B testing. So that itself offers a lot of benefits now overall for the company culture. And then now it really also drives innovation now in the company where if you really do it right, you can enable a culture with this constant like ideation around, like what new things we can try as a business. And again, we do recommend like have those hypotheses and ideas as much data backed as possible, but not really. There's a constant kind of not really stream of ideas coming through from across the business. And you're testing those and you're building a feedback loop back to the business on the learnings from those experiments and then you're refining your hypotheses. So that's the goal and that's maybe done at DocuSign and that's the goal really at all the places that I've Led experimentation programs. Okay.

And of course like the point of these experiments have some kind of material impact on the business. So can you tell me about an example either from DocuSign or anywhere else you've worked where you' seen a real benefit or a real impact from running one of these A B tests?

Yeah, no, absolutely. And before when I get there. So you're right. I mean the really goal is to drive business growth, drive business impact and you can have a number of different KPIs to as a proxy for business impact. But one thing I just want to even preface that with like a lot of times you may kind of really know have that mindset, hey, if it's winning, if it's not a winner, I really did not drive an impact. I would say every test, every like even a failed experiment is a great learning. So that's something just like it's a pretty kind of known term now in the tech industry. Fail fast or fail forward. Like you fail but you learn from your failures. But having said that, no, I've definitely seen a lot of different like now exciting wins through the experimentation programs I've done at a number of different companies now. So maybe I would like to pick two different examples. One is now really on the UI and the UX elements on your website. And I feel that's really like now you start seeing. It's very interesting how you can really inform user behavior by changing certain elements on the site. This one example, one of the leading e commerce companies I worked for. So you get a ton of traffic on the site and we used to categorize our site traffic or site visitors into like browsers. The folks who want to just like browse around first, right? And spend some time and then purchase a product versus searchers who would just come, they will just go and search for something and you land on a product. We have seen that searchers users who come and search now have a significantly higher conversion rate than folks who just browse. So there was this hypothesis that came from the business teams to say hey, if we can drive more people into search lane, can we really improve the conversion for the company? And again there's always this correlation versus causation. So it was a really great candidate for testing. So that was the hypothesis and we worked with the design team on ideas and designs to really drive more people to search on the site when they land on the site, which is like now you can imagine like really making the search bar bigger. In some pages the search bar was not really collapsed with a magnifying glass, really expanding it everywhere. So that was run as an a B test and that was a big win. We did see a lot of people searching, which we knew that we will influence the search behavior. But it did end up resulting in overall incremental conversion lift. So I think that was really an interesting example of how some very simple changes on your site can drive the downstream impact in such a meaningful way. The second I think example that I've done quite a bit is like really testing the ML algorithms. And that's very interesting because typically if you're familiar when you're building a machine learning algorithm, there's a whole process, there's a control environment when you're building it, there's a training set, validation set, you have some KPIs to measure how effective the algorithm is in a training versus validation set. But when you actually start now, when you put that model into production and you really run now those algorithms on your actual live users coming on your site, it's interesting like now if you don't AB test, I mean, you may not always be driving incremental gains to the business. So that's another example like now where I was working on what the data science team, we were working on building recommendation algorithms for our products on the side. And I think through AB testing, I think it took us multiple cycles to come up really with the right version of the algorithm, with the right mix of business logic added to the algorithm to drive the incremental gains. The first couple of versions actually even end up hurting us and did not really drive any gains. So that was I think a great, I think eye opening, like insight for everybody that how critical it is to really a B test your machine learning algorithms that you are building on your product.

That's a really great sort of selling point for this idea of doing experimentation because often you have this hunch, okay, well it feels really obvious if I do this particular change to the site, then it's going to get a small business or it's going to make small money somehow. And then actually if you run the experiment, it turns out this thing that you knew for sure in your gut turns out to be not true at all and may even hurt your business. So I think that's a really nice example of like why you do actually need to run these experiments. I'd also like to go back to that first example of yours because that was kind of interesting. You were saying that you're doing these experiments around increasing search, but then the actual outcome was sort of increased conversions or so it was actually more people paying you money, but that was kind of a step removed from the experiment. So there was a sort of downstream effect. Is that what you were saying?

Yes. I mean, you can change user behavior. Like now you can say, hey, I'm gonna just maybe make you click here, add buttons, Right. And typically, I think what I've seen a lot of companies, like now you will stop at that, like level one metric trick to say, hey, I drove more searches. Right. But really like now, you don't know if that's just a mix shift that happened. Right. Or did you really truly drive incremental returns for the business? So that's one of the, I would say, best practices now. Try to go downstream as much as possible. I mean, again, it's a fine balance, right? Because it also, like, you know, then that makes your test duration longer and comes at an opportunity cost, but now it really tells you the truly incremental gains that you drove from the test and not just really shifting some user behavior at a very, like now, upper funnel level, which did not translate to gains downstream.

All right, so just going slightly more broader for context. Are there any other types of experiment people ought to know about beyond a B tests?

Yes. No, absolutely. So I think a B testing. Yes. Is the most common used term and people understand that. I mean, there's an extension of a B test called abn, abc, ABCD test. Right. What essentially that means is instead of testing two groups now controller test or champion or challenger, you actually test three different versions of it. So you call it ABC or abcd.

So when would you want to test those sort of three categories or more categories rather than just the two?

Yeah, I think a good example is when you have now multiple ideas, right? So let's say you have strong hypotheses around a certain idea, but there are multiple ways. For example, you can present that to user. So I think going back to the search example, like now, you could now expose the search on every page. You could argue, I'm going to maybe, I don't know, show it at two different places on the page so you can bloody test multiple versions of the design. And that's when you will design like an ABC test. Instead of just designing one version of it, you may design 2, 3, and then see how that spans out which one is the winning version. I think the trade off always is like now, how long does it take to reach statistical significance? Right. So like, the more versions you have, the more variation, your traffic is split out more and this takes longer to get a good read on the test, which is where you can get enough confidence that this is significant. From a statistics point of view, okay, that's like abn. It's an extension of a B test. Another popular now testing type. Again, like, now when I say popular, it's known, but still like, not used very widely is multivariate, where you're changing multiple variables at once and then you assess the. Really, the key of multivariate testing is you can assess the interaction impact between these two variables. So again, like, going back to my example of like, now let's say search bar, let's say on the homepage of your website, you are trying to make the search bar bigger, and then you're also trying to change the HERO image on your homepage, right? And those two elements, because they're on the same page and very close to each other, probably will have an interaction impact. That means, like, now changing one may impact the results of the other. So in those kind of scenarios, that's a good candidate for multivariate testing, where you basically design three or four versions of different combinations of this test. So, for example, you'll have your regular design, your control, and then you'll have a version where you change the search bar but don't change the HERO image. And you have a third version where you change the HERO but don't change the search bar, and then fourth version where you change both. So with this kind of a design, when you look at the results, you can actually sort of really identify the impact of each of those changes, but you can also see the impact of those changes done together. And at times you can see there's a multiplier effect. At times you can see they may nullify each other, or at times you can just see maybe it's negative by doing those two things together. So that's a good example, or that's obviously one use case for multivariate testing. There's a third piece which is, I would say again, like, not common at all. It is a technique, I've used it once, which is called band aid testing. And the concept there is now, unlike a B testing, where you will, let's say now you set up your test and now you do some analysis to know you need to run the test for like four weeks. So you run the test for four weeks and then you decide what the winner is. That's when you make the change at the end of the test. In the bandit testing, the concept is like, now as you're testing, and while you're Learning which version is winning, you start routing more traffic to that version. So you're really capitalizing on the benefit while you're learning from the data, you're not waiting for the test to end to really claim or get that additional incremental return that you are driving through this experiment. So it is more complex to implement and I think that's one of the reason why it hasn't really caught up in the industry that much. But I think that it's a good use case for like now instances where you really don't have much time to wait for the test to end and make the change. Common example is like now let's say some kind of news headlines, right, or subject line like where you only have, let's say very short window, like you cannot really test something and then you can implement that after like four week window.

So that idea of bandit testing is really interesting. So certainly at Data camp when we run a B test quite often you write it takes maybe a month to get statistical significance. So before that you don't know whether the A group or the B group's going to win, but you'll get some results beforehand and, and every single time there'll be someone going, oh look, this method is ahead. It looks like it's better, we should make the change already. And there's this cultural impatience where he's like, okay, I just want to go and implement the thing that's going to win already. But you don't know for sure statistically. So the idea of bandit testing where you can like start saying, okay, well maybe rather than giving 10% of users this new feature, if it looks like it's going to be good, then we'll start rolling out to more people. That seems like a really useful thing to deal with that kind of impatience, particularly if it's your manager going, well, you know, I know there's no statistical significance yet, but it looks like it's good. Go and make this. Just the idea of gradually giving it to more users seems like a good sort of halfway point between just ignoring stats and going for it and doing the sort of the statistically diligent thing.

You're right. But I think just like now what I've mentioned, like keeping in mind the trade off, right? So one is more complex to build. Second, actually it does take longer to get statsig on a banded test, right? Just because you're not really maintaining even split of traffic for a set duration of time, you keep spreading traffic. So you drive from a business point of view, yes, maybe you're capturing more of the incremental revenue, for example, that you're impacting or that you're driving. But to truly get a stat sig read on the test, it actually going to take longer for a bandit test than really an A B test. So I think that's the trade off to keep in mind.

Ah, that's interesting because it sounded like it was going to be a magic speed up, but to get to the end of the test quicker. But maybe not.

Yeah, no it's not. And maybe that's probably one of the reasons, along with the fact that it's complex to build, that it hasn't really caught up in the industry that much yet.

Okay, so maybe we can get into a bit more detail about what the steps involved are for running an A B test. So can you maybe talk me through what the workflow is, the different multiple.

Ways, how like now different companies implement the testing program. But I would say in a nutshell like three broad categories. So one is test planning phase and I would say that's the most critical phase and you should probably spend the most time there. Test execution is the second step and then the last one is like test reporting and building that feedback loop. So let me maybe just go a little bit deeper into each of those. So when I say test planning, that's the step now where you build up almost like some companies call this test charter or test plan, where you really come up with a very solid refined hypothesis. What is your hypothesis that you want to test and that itself, I would say it's a pretty critical piece of the whole testing program or testing lifecycle. And again we can have a whole session on hypothesis now generation. I don't have to go to that many details here, but essentially what we really recommend our stakeholders is they use as much data as possible to build a strong hypothesis. It doesn't have to be like quantitative data all the time. Like at times maybe it's a new idea, you don't have data on your own. But now either quantitative data or user research or even competitive analysis as much as possible you can have a very solid data backed hypothesis. I think it just increases the chances of now that that's really holding out and getting some insights. The second piece of the test plan is establishing a primary KPI. And again it's easier like now said than done because typically I think it's very hard to align or like what's that one KPI that will decide the winner. And I'll give you an example again going back to My E commerce experience right now, a lot of times they just interesting trade offs at play. Like now when you let's say in an E commerce environment changing some elements on the side. So for example like now you are driving let's the end goal is of course like now driving more conversions and revenue and you're changing now let's say something on again like on your landing pages. And the goal is to drive more, add to cart more people to add that product to cart. And then now it is possible through that you may drive, you may make some now changes that may drive more people to add to cart but may not really drive more checkouts and purchases. So in those kind of instances, that's one example. I think it probably almost every test that I've come across has those instances. But it's very critical to really work with your business stakeholders and define and establish that what's the primary KPI that will define what's the success for this test. And I mean we always say they can only be one primary KPI and you can have a bunch of other supporting KPIs, secondary KPIs you can monitor. But like one primary KPI. Yeah, I think those two really are the biggest pieces of test planning. The third piece is more just operationally like now it's always a good practice to establish before you set the test like now on how long long typically the test needs to run again like now the elements and some statistical like calculations that go into that determination some assumptions. I mean I can talk more about that if that will be of interest for this session. But really establishing that in the beginning. So there is expectations that with the stakeholders on like how long some tests will run and actually even broadening it like I would step back and say really the testing team should be maintaining a testing calendar because usually they are like now backlog of things and ideas that you can always like run and test. So it's good to really establish a testing calendar to say hey like now running test one, it's going to take four weeks and then my test two is lined up after that time. So that is the supporting teams who are working on building out that experiment is aware of those timelines that already forms the test planning phase. And then the second now phase is really executing and running that experiment. And that's where like the development piece comes into play. Where typically the testing teams you have to partner with the engineering teams to make sure that experience that you're trying to test. In that example that I mentioned on Search Bar, you Have to work with the front end engineers to make sure there's a version of that page which has a different UI element that you are testing. And then the QA teams play a strong role again here to make sure. Now our data tracking and everything else is really matching between test and control. And then you typically have an AB testing tool that you use to really launch the test. When I say launch the test, it means really, you know what I mentioned in the beginning you have two different versions of your page being shown to two different sets of audience and that's when the test is running. And the third phase is the test reporting where you really report on how did the test perform, what was the winner, if there was a winner, a lot of times there would not be a winner or you would not know. And then what were the learnings. And I think that's what I mentioned in the beginning. The feedback loop really is very, very critical for a successful testing program. You're learning all this, but how do you really come up with one? I think now conclusions and recommendations and how do you feed it back to business that they can refine now? 1. I mean that learnings really are an input to how they think of refining the product roadmap and how they think of the next test or any other test on that feature.

Thank you for the overview. I have to say, when I have a new project, I always want to get started immediately. And it sort of makes me sad that like every single time it's like actually planning stuff before you start doing things is the most important step. So that's a shame. But I would actually like to talk a bit about this. You separated the idea of execution and reporting, but I presume there has to be some kind of monitoring of what's going on during the execution. Like you can't just set up these two different test groups and just leave them for a month and then do the reporting at the end. So can you maybe tell me a bit about like what happens with this sort of intermediate stage reporting and just monitoring what's happening?

Yeah, no, absolutely. And that's a great point, Naruchi. Yes. I think now we need to keep very close eyes on the test metrics while the test is running because the number one reason being is possible, like maybe you did something, it could be also an error at times. Right? Like you did something that really introduced an error in your product or feature. But also it's possible that you influence the user behavior in a way that's really, really hurting you. So let's Say the initial estimate was four weeks for a test that's with a lot of assumptions but it's possible within like now two days you start seeing your order volume going down significantly. So yes, it's very critical to keep very close eye and monitor the test while it is running. So I think now the testing team, again in my experience that the companies that I've run, the testing teams now the testing team is responsible for really keeping a close eye on the test. At times we also ask these stakeholders like if the company has some more like established self service tools where now you can have external folks who can come and monitor the test without really writing some queries and stuff. I think that's always a good investment to build. So there are more eyes on the test. So yeah, typically the reporting piece like now what we do is really break it down to three different steps. We say we call it early read, we call like a mid level read and then we call a final read. Because there's so much assumptions that goes in the time window. Nobody knows for definite if that's the time window we need to run the test on. So again, depending on your business and your metrics, you may define a cadence, but typically I would say definitely and early read at least the first week. I mean I would even say like when you launch the test there should be somebody keeping an eye first few hours as well. Like there's some good practices in certain companies that I've seen that they will say no launches on a Friday because of that risk. Right. Because really it's almost like a feature launch, like you're introducing something new and it may introduce some error so you can put some guardrails in place to protect you from those kind of instances. But yeah, I would say now at least within the first week like it's a good idea to report out maybe a high level like now just the metrics you see. Because the reason being at times, you know, you don't have to wait for the full test to end. You may really find something significantly either like good or bad that may give you enough action item to either like rebuild or reconsider, even like launch it out. Right? Whatever you build and test something else. So yeah, so I would say typically like no break out your, like no reporting into different like milestones. Do an early read, do a mid level read and then do like a final read when the test is ended. Another benefit of doing that is like in the test planning phase, typically you will collect the list of questions you're trying to answer. With this experiment, it's not always like hey, just what's the winner? But you're trying to really interpret how the user behavior changed. Let's say we did abc, right? And how did users react to it? Or did we now cannibalize something else on the business by changing this thing? So I think really doing kind of a slightly more detailed like read out while the test is running maybe around midpoint gives you enough data. You may still not have stat 6 results, but it may give you some insights to really refine your questions and what additional areas you want to look into. Because I mean in the end, right. It is also like analytics team or testing team, like who's reporting on the test and their bandwidth is also limited. So as a good practice, that's the model and framework I've seen working well and you split out your reporting into different milestones and as you are reporting out you collect additional questions and then you really wrap it up with a meaningful set of recommendations and more kind of conclusions that can be acted on.

Okay, I think that point you made about if you accidentally introduced a bug or an error or something into your platform or whatever it is you're testing, then you are going to want to see that very quickly and maybe abort the test. Certainly if you've accidentally disabled the checkout button or something for one of the groups, then you're probably going to want to cancel that experiment pretty quickly.

And unfortunately it happens more often than you will expect when you're testing. We've seen that quite a few times. So you're right. Yeah, I think in those instances you definitely want now somebody to be on point to go and at least stop the test right while the bug is getting fixed.

Okay, so let's talk a bit about personnel because it sounds like an ABA test isn't something that just a statistician or data scientist can run by themselves. There are going to be quite a few people involved. So can you talk us through what are the different roles needed for running one of these experiments?

It is I would say in my experience one of the most cross functional function like our program now in the company. So now it is definitely cannot be done independently by data scientists and testing team. Now you need a very cross functional group of folks involved right from the beginning, from the planning phase. So what I mentioned earlier like test planning, really building that hypothesis, deciding on a primary KPI, what's that one metric that will be your North Star for this test and then also the now really decision maker now some kind of A racy model, say, like, who has the final decision making power on this test? Right. A lot of times it happens you don't have a clear winner, but you get a lot of insights. You may have to still make a decision on whether we really roll this out or we turn this off. And then so really establishing that in the beginning. So I would say the business teams that you're working and the business stakeholders and I will start to establish like, who's really, like no one, that the key decision maker on this test be the product team, marketing team, whichever function you're working with, I think it's really good to establish and making sure that they are really involved from the very beginning in any discussions or decisions about the test. So when I said business team, product marketing, again, like the partner teams like UX and design teams, if it's a UX change you're making or if it's like, let's say you're testing an ML algorithm, right? So then the data science team or machine learning team who build that algorithm, they need to always be also like obviously closely connected to the entire lifecycle of the testing program in the execution phase. A lot of the testing tools in the market are pretty advanced. Now some of the simpler tests, you can actually even implement those like hiding a banner, changing the text on a button, you can really do all that within the testing tool depending on what your underlying platform is. But a lot of times you would need the support of the engineering team to make those changes. So again, really working in a very close collaboration with that team to align on the timelines and when would they be able to deliver that experience that you're trying to test the QA teams to make sure there's end to end testing. And when I say end to end testing, one of the learnings I've had over years, it's not just the testing of the feature, but also the testing of all the data instrumentation which is supporting that, any data instrumentation that you need underlying that feature or change that you're making. Because otherwise in the end, if you really don't test it in the beginning, it has happened. I've seen that when the test is ended and you're like, oh, actually I didn't instrument this piece, so I cannot report on it, which is a big, I would say loss, right. In terms of the opportunity cost. So the QA teams play core critical role here. And then now I mentioned the reporting piece, I mentioned the feedback loop. So again, like now to build a successful testing program, like it cannot be really run in a silo. You have to pass on those insights back to the business teams. So again like not really in close partnership with even actually even to put the recommendations and conclusions. A lot of times you may pull the numbers, but you may need to really meet with the business stakeholders and really the functional teams who are closer to the customer and the the product and the platform to understand how to interpret those numbers and what could be potential recommendations that we can take as next steps based on that. So I would say it really involves very close collaboration among a number of different functions and very cross functional in nature throughout the entire life cycle.

It seems like at least once you have a sort of mature experimentation setup, it does actually involve like just vast numbers of teams. And I'm hoping we're not terrifying any of the listeners in terms of making it sound like it's a really, really hard thing to do. So we can go the other direction to talk about. But for any organization that wants to get started with experimentation, what do you need to do for that first experiment?

Yes, I mentioned like none of those elements you write could seem a little scary that oh my God, it's such a heavy lift to get started. So again that is what I mentioned. Really if you're running like now a scaled out testing program for a large organization, I mean you can always do it in a very light way to start off. So to answer your question of somebody, any organization that's trying to really get their first a B test that's done, what is that basic framework they need in place? I would say number one, like if anything like the key, really the key element is having a strong hypothesis. So identify a use case. It could be anything. Like now it could be changing the experience on the side, changing your email campaign strategy, like how you know your target audience strategy. It could be anything but like now identify a use case and be very clear on the hypothesis that you are trying to test. And then I think what I mentioned earlier like now really establish a primary KPI, like one metric that now you're trying to influence. And I think now if you really want to do it lightweight, where you don't, at least on the business side you need at least one point of contact who's a decision maker on that. So you need to identify who within business now is that person. I mean you need kind of sponsorship from the business team and a buy in from the business team too because in the end you will need engineering resources, you will need some analytics resources. So getting the buy in identifying the use Case and I would say also like now on the sponsorship and buying piece, I mean even doing some modeling, financial modeling to come up with incremental potential, incremental impact, if that use case turns out or proves out its value, could be another good way to get that kind of really initial buy in and sponsorship. And I mean so that's probably what you really need to line up to make sure you have a good use case to test in terms of the actual testing. So I mentioned AB testing tools and there are a bunch of those in the market right now. But if you want to just really test it out, test the waters without really doing a lot of investment, there are ways to really do it on your own too. So for example, when I was in my previous experiences from marketing campaigns, we would just like now build like hey, you're sending out an email campaign. We would just do like now an A B test using Excel. Like no, you just use a control like random function and like now just take randomly split half of that users into a control group and other half in the group that you will like send their campaign to. And then you monitor the results in that fashion looking at like the users that did not get the email versus did got the email. So you can do it in a very manual like basis to start off just without using a tool but to really like know do it right. And I think you would need again like some large companies build their own a B testing tool. Again that's probably not our advice to begin with. But again like if you're looking at the tech stack, maybe go with some. If you're again looking at something, let's say online platform, whatever data stack you have, try to either find a tool that integrates well with it or again in case of something online, I mean typically you will have a tool to track data for your site or mobile app. A lot of those vendors do provide a B testing feature. So that's another good one to maybe like network with them and do a POC on a couple of use cases to get started. And then yeah, I would say maybe that's all you take and it's almost like a flywheel. Like now you take a use case like prove out value, give the learnings insights back and hopefully that kind of works like a POC and then that gives you more kind of buy in to really build it out in a more scaled out way.

Okay, I really like that example of the marketing campaign where you just you know, using very simple tools just to run an A B test by splitting Your user base for different emails and maybe you can track the different number of views or clicks or whatever from those different groups. That's just zero infrastructure to set up compared to maybe some of the user interface kind of changes where it is going to be a little bit more technically involved. So are there any common mistakes that organizations make when they're first starting testing?

Yeah, I mean, I think, you know, it really maps back to what I was saying earlier. Like a lot of times this is excitement and now enthusiasm and also this urgency to say, hey, like I have an idea and I want to just test it. Like, I'm so bullish about it, right? And I want to test it. So I was. It's like spending some time in the planning phase. It may seem a little bit like a slowing you down, but it really does pay back in the end. So, yeah, spending some time on really building a good hypothesis, I've seen a lot of times just now you're testing quite a bit. Right. But not having a strong hypothesis. It just in the end ends up taking longer because you're not really getting good learnings and yeah, just not spending enough time on the planning stage or having multiple KPIs. Like, no, you want to change something and you're testing like a bunch of different KPIs. And a lot of times it happens that those KPIs go in different directions and then it's just unclear what do we do with the results.

So just to expand on that, do you have an example of a good KPI and a bad KPI?

Yeah, it's a fine balance. I think that I was saying earlier, like, you may choose a KPI that's closest to the experience you're changing. Right. So an example again could be going back to my search experience. You change the design of the search bar and you may say, I want to just see how many people are clicking on that search bar now, because now I made it bigger. I would say that's probably not a good KPI. Yes, you should measure that as an operational KPI, but not a primary KPI because you may drive that, but you don't know if that really impacted your business. So try to go a little bit more downstream. In that case, we went to conversion and again there's a trade off. Right. The more downstream you go, when I say downstream conversion revenue, the longer it takes to run your test. So having something a KPI defined very close to the exact experience you're changing may not be, I would say the best idea. Go. Goes slightly I would say one or two levels deeper to get the incremental impact. And at times I would say one of the ways to do that is also like if you know, I mean you can choose a KPI that doesn't have to be let's say all the way down to conversion or orders or purchases all the time. But if you establish over time that there are certain other proxy KPIs to it that have a strong correlation to your really end goal for the business and then you can choose those KPIs as a KPI as long as you're monitoring that, changing that. Let's say again what I mentioned earlier, you improved your add to cart rate, which is a great KPI, but always keeping an eye to make sure that that did not negatively impact your downstream purchase rate or checkout rate.

And do you have any other examples of mistakes that companies make?

I've seen a lot of different examples now. So beyond the planning phase, not spending too much time on the planning phase. Other example I've seen like now ending the test too early. Like when you launch, let's say a new feature within the first couple of days, there's a lot of just variation. It's a lot of like noisy data. And again in the statistics, like Navoral we call it, there are different biases. One of the very common bias is this novelty effect, right? When you launch something new and now your users are used to a certain design, a certain experience, right? And they may not respond to it right away. So it may look like hey, actually this is not working. And maybe at times it may even be like statsic drop. That's what is called like novelty effect. Like first two, three, four days. Typically a rule of thumb that I have always used is at least give it a week, right? There's seasonality in business at times, right? You may have like now weekend. Really your traffic either spikes up or drop depending on your business. So that's I think one mistake I've seen when now you've not given the test enough time. When I say enough time. Depending on the seasonality of your business, it could be a week, it could be two weeks, right? It could be a few days. So now make sure you give it enough time to really like have your users adopt the change that you are changing. Another one is outliers. Like no, again if you look at just aggregate results, let's say end of two weeks, there could be some outliers in the data. Maybe you ran a promo during that period, right? And that's Just that the group really responded very differently to the promotion that you ran beyond the actual feature that you are trying to test. So in those scenarios, one of the best practices we recommend is always like looking at a kind of a time trend on your results, right. And see if there are like some really weird outliers that could be impacting your overall results. That could be another misleading outcome. And I think other mistake I've just seen like now being on either like now far end of the spectrum of this like testing frequency. So either you're testing too little, you're not testing enough, right. Or you're testing too much. It may sound surprising, but there's something called testing too much. Not everything needs to be tested because there is the cost to testing. So that's what I think now over time you build that muscle of like no, hey, what do we need to test? Like no one. I mean for example, if something like no, if you have something. And I'm going to think of an example again like going back to like as an E Commerce some part of your site where you get very few people even go there like very little traffic. If you try to test something on that, it's just going to take you a very, very long time to even get any stats sake. So in those cases, right, if it's not really meaningful, like no impact, you don't want to really waste your testing bandwidth and resources in testing those. I think make sure that you're really identifying areas where there's meaningful impact and something really now does not get that much. I would say audience like yeah, I mean you can probably go with your hypothesis and build it out, right. And just monitor results like pre post that that's okay too. Which is very I would say anti like now testing philosophy. Like typically everybody says test everything but my recommendation would be I think there's a fine balance there. So like you should test as much as possible, but you don't have to test everything.

I thought that point about you can either test too little or too much was really interesting. So thinking about at Data camp we test our registration pages a lot because it's very important to our business for a lot of people to register. But some things where it'd be nice to have tests are courses where ideally we'd have two different versions of each course and you would see which version more learners learn more things from. But that turns out to be very expensive to make because it costs a lot of money to make each version of a course. And then measuring the outcomes is very difficult so we don't do a B testing there. But can you talk me through your thought process for when you should test things or when you shouldn't?

Yeah, that's a good point, Richie. I think example you mentioned that that's exactly in line with one of the use cases we will recommend not to test when in order to do a cost benefit analysis when the cost to really testing that feature or that in the example like building an entirely new course and testing two courses in parallel if the cost is really higher than the potential incremental impact you can drive with that feature. Now in those cases we will recommend not to test, maybe just like really build out the new thing. Right. And you can just monitor results like pre post. The second example would be, you know, at times there's certain features or parts of your platform where you just don't have enough data, enough of an audience to really now get a stat signal read on that change. So again in those cases you may run a test for a few months but that will probably take more bandwidth from everybody to run the test, monitor and do the results. Right. And may not be really a significant impact to the business. Again we will say in those cases, yeah, do your research, build a strong hypothesis, right. And really build out the feature and then you can monitor results like pre post. And I think in certain times it's just like now there's this legal and technical, I would say constraints that you cannot really test in some countries, like different pricing to different users. So in those cases we are just bound by those restrictions that we cannot ab test. And again like now there are methodologies in place that you can look at like a pre post, like what happened before, what happened after you made the change and then accounting for seasonality to look at like year on year, if your business is seasonal, to get some kind of directional sense on what was the impact. But there are certain use cases, I mean I think always keep this in mind, like there's always a trade off. Like when you're testing something there's always an opportunity cost because you could always test something more impactful. So have that in mind and have that lens to there is cost to testing and there's an opportunity cost to testing because you can always test something bigger, better, bolder. So keep that in mind and use that framework.

All right, so let's move on and talk about skills. So you mentioned a lot of different roles involved, but in general what are the skills you need in order to run an A B test?

I would say at the basic and that's the kind of framework I use for any analytics role. Not just AB testing, not just a strong combination of technical and business skills. So when I say technical, like yes, your basic SQL and like understanding of data on the A B testing side, some conceptual understanding of statistics is important, right? So some like foundational things like probability theory, confidence intervals, hypothesis testing, power analysis, again like a lot of the A B testing tools in the market really build it out for you. You don't need to. But when I'm hiring for AB testing like program or AB testing roles in my team, I think some basic foundational knowledge is always helpful. That's just to explain the concepts to your stakeholders and to even like resolve any issues yourself. So that's on the technical side. And yeah, so I think some foundational statistics, some kind of querying, but also just like not really that critical kind of reasoning and being able to like now analyze the trends and really make inferences of that data. But when I said like no a combination of technical and business skills on the business side, really having some exposure to the business domain that you're partnering with is always very critical. And the reason being like now a strong testing team not only is really strong at really setting up the test and doing a good planning around it, but also really coming up with good recommendations and really a critical like now part of coming up with strong recommendations or even like building a strong hypothesis is really now how often can you as an analyst or as a testing person put your business hat on and really now try to understand like how did this, let's say change, inform the user behavior? What does that mean for my business? What are the other implications? Is there any cannibalization? Is there some other opportunity we can leverage by learning this behavior? So that's where I get that comes with time. And as I mentioned like the really testing team or testing function being a very cross functional like role. And now really my advice to be like no again anybody who's looking to get into this function really from the very beginning, yes, I think it's at times tempting to be on the technical side writing queries, running programs and doing the statistical analysis. But as much as possible, insert yourself in the business function, be in those meetings like now really talk to your stakeholders and learn their business. And that's where you really become a strong, I would say analyst. And testing like a B test when you can really put your or give your inputs on the hypotheses or help the team really come up with great ideas to test and then Also when you put together the results really come up with meaningful actionable recommendations and build that feedback loop with your business stakeholders. I would say those two really are the core foundational of these skills to be in that a B testing role. The third component is and again that becomes more relevant in larger organizations kind of program management skills because I mentioned it's such a cross functional role. And now you really need to keep a lot of works in the loop. And how do you really build some kind of scaled out tools? So for example like in our company we use agile like technology so we use Jira Kanban board quite a bit to really say like to communicate to the broader team what tests are running, what stage a test is on. Right. And how do you really build out some dashboards to self service the results of the experiments and how do you make sure like you're managing the testing calendar? How do you managing trade offs? Right. A lot of times it's like a test is running, I don't know, it's delayed. Right. That impacts your follow up test and how do you make those adjustments and really collaborate with business stakeholders to make that fine like adjustments. So sounds like a bunch of different things but I would say again like it really helps you get so much of visibility into the business and really drive the impact in such a meaningful way. It's a very satisfying I would say function and role to be a part of and grow your career in.

Brilliant. Do you have any final advice for the people interested in a B testing or other people managing a B tests?

Yeah, no, I would just say like now again like no. I am very passionate about testing. I've done it in a variety of different companies and industries. I would say testing is one of the most strategic areas for the business that you can invest in. So now both as a business leader and as an analytics practitioner. So my advice for like business leaders would be no, definitely getting executive sponsorship for setting up and building out a testing program in your company. There are multiple ways to set it up. The model that I've seen really successful that could scale is like a hub and spoke model. So building out a testing coe a small group of folks that maintain the center of excellence which can now manage a testing program, testing calendar, kind of some self service tools and educate the rest of the org and then now really invest in data literacy to make sure like now anybody in the org can self serve on the test results and submit ideas, build hypotheses and really building that feedback loop for the analysts or folks who want to get into this space now and build career in a B testing. My advice would be get some fundamental like skills in place, some basic knowledge and statistics, SQL and then just like now really reading about how other companies are leveraging a B testing. I think it's very interesting examples when you start reading like how different industries and companies do it, I think that really sharpens your skills on how you can leverage that in your own space. And I think what I mentioned earlier, along with really building those foundational technical skills, equally important to really build the domain knowledge on the function you partner with so you contribute to the entire life cycle on like ideation, hypotheses, reporting and not just really a standalone like testing implementation.

You make experimentation sound like a lot of fun as well as being a very high impact thing. So that's brilliant. All right, thank you very much, Anjali. I've learned a lot on this episode. I hope our audience has as well. So yeah, thank you for your time.

Thank you Richie for having me. You've been listening to Data Frame, a podcast by datacamp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating, leave a comment, and share episodes you love. That helps us keep delivering insights into all things data. Thanks for listening. Until next time.