Design & analyze geographic-based lift tests | Recast GeoLift Demo

Foreign hello, my name is Chelsea Parlette and I am a consulting statistician here at Recast. And today I'm excited to demo to you the Recast geolift tool. Recast geolift is a powerful tool to both design and analyze geography based marketing incrementality tests which can be an important part of measuring the effectiveness of of your marketing and helping you plan for the future. One of the benefits of Recast geolift is that it's both powerful and flexible. So let's take a look at how the tool actually works. So here I'm on the homepage of the Recast geolift tool and you'll see at the top that there's two tabs. Design and Experiment and Analyze an experiment. We're actually going to start with the Analyze Experiment tab and look at how you might take the data from an already completed Geolift test and analyze it to understand the lift that you saw. So the first thing that we need to do is grab the data from our test. So I'm going to go ahead and click and drag that here. This will auto populate once it's uploaded, allowing you to click through, kind of sanity check what your data looks like. Once you've done that, we can go back over to this tab which shows us a bunch of different columns, many of which auto populated. But we do need to definitely specify what our outcome variable column is. In this case, our test is looking at revenue. So we're going to select revenue. Then the last thing we need to do before ingesting the data is make sure that our date format matches what's actually in the file. This doesn't. So let's go ahead and select the correct one and then we will ingest. Once the data is ingested, you'll be able to see not only the raw data that we were looking at before, but also plots of your different data. Again, just a good way to double check that the data looks as expected. Then once we're satisfied there, we can scroll down to step two where we input a little bit more detail about what happened in our test. So here our test was run in March. It started on March 1st and ended on March 22nd. So we'll go ahead and input those dates. And then we also had a cooldown period which ended on April 4th, so we're actually going to leave that as is. Then we need to go ahead and tell the app which regions were in our test group for this geotest and in this case it was Austin, Boston and Denver. Then we can go ahead and scroll down and choose the outcome variable type. Like we saw in the data, the outcome variable we're looking at in this case is revenue. So we're that and then I'm going to go ahead and select that I increased spend in these locations because that's what actually happened in the test. I increased spend by about $15,000. So I'm going to go ahead and input that here. And now I'm ready to click analyze experiment. This should load pretty quickly and once it does, we'll get all of the results from our actual geolift test. You can see here that the results are available in terms of actual dollars of lift, but also in terms of roi. We also outputted a plot that's useful for digging a little bit deeper into the results of our experiment. Here at the top you can see some graphs that help you understand what happened in the test and in the synthetic control group. On the left hand side we have a waterfall plot that looks at the total revenue in our holdout, the total revenue in our test and what the estimated lift is. On the right hand side we have a really useful graph that shows us as time goes on, what the revenue was in each of these groups. You can see that as time goes on, the test group starts to diverge from the holdout group, which is expected because we spent up in this group, hopefully causing some higher revenue. Below we have a really interesting chart that allows us to see how similar the holdout and test groups were pre experiment and then how they diverged during the experiment. What we're looking for here is pre experiment. So everything in this graph that's in this white portion should match pretty closely. We want to make sure that the trends of our synthetic control and our test group follow very similar patterns before the intervention of our test. Then after the test begins we can see that they start to diverge due to that change in spend that we had in our test group. If you scroll down even further we get another table and this tells us about the weights that are used to create our synthetic control group. In geolift, we don't just use all of the non test regions by themselves as the control group. Rather we use a weighted combination of them to create a group that is as similar as possible to our test group pre experiment. So the algorithm that creates these weights only looks at data pre experiment and creates a weighted synthetic control group that combines all of the potential control regions into our synthetic control group in a way that creates a pattern like we saw above in the graph where the two the synthetic control in the test group follow very similar patterns pre experimental intervention. This table is really useful. You can click in and see even more regions, but it shows you what the different weights are that actually created your synthetic control group. The idea here is to try and estimate a counterfactual what would have happened in our test regions if we didn't change our spend during the experiment. The synthetic control is a great way to try and estimate that counterfactual which we can never actually observe in real life. Now that we've seen what we can do to analyze a geolift experiment, let's switch over to the Design tab to see how we may have designed this geolift experiment in the first place. If we scroll all the way to the top of the page, we're going to click over to this tab and like before, the first step is going to be uploading our data. This is going to be data about your KPI for the different regions that you want in your geotest. Like before, Once it uploads, we can see it populate on the right and we can click through to just make sure the data looks as expected. Again, most of the boxes here auto populate. However, we do have to select what our KPI is. Again, it's going to be revenue. Then we select the correct date format and ingest our data. Like before, it will output a plot. So we can just sanity check that everything looks good. And once it does, we need to input some information about what we actually want our test to look like. So let's say that I want to test increasing spend in my channel. So I'm going to select I want to test increasing spend spend. And like we saw in the data that we uploaded, we are using an outcome variable that is revenue. So then we need to provide an approximate channel roi. This is going to help with some of the power simulations that we'll look at in a second. And for this particular channel, my estimated ROI is 3x. So I'm going to input 3 there and then I need to tell it a little bit about what I want my experiment to look like in terms of time. So depending on the variable this might change. Here I'm going to set an experiment length of 28 days and then I'm going to allow a seven day cooldown period after I have budget to spend about $15,000 for this test. So I'm going to input that there and I'm going to leave these blank for now. But if you had any specifications where you couldn't include a region in your test set or wanted to exclude it entirely. You can use these boxes to do that here. Then we're going to go ahead and click Generate test Markets. Now, if you have some sort of limitation where you only want to consider test markets that have a certain number or certain range of numbers of geos, you can go ahead and use the slider to limit what the app will consider when looking at potential test markets. Now, this may take a minute to run, but when it's done, we'll go ahead and look at the results. All right, so this is done running and now we have a bunch of different potential test markets to choose from. In this table you can see that it lists which regions are in this test market, the baseline revenue, and then it essentially runs a simulated lift test and outputs to us what the simulated lift was in this case and what the model estimated to be the lift. Now, of course there may be outside considerations that the app is not considering when it's suggesting these regions, but all else equal, you probably want to choose test markets that have the lowest percentage error. So here we would probably want to choose one of the top few rows as they'll have the lowest absolute percentage error. Once you've looked through these test markets and decided that you want to deep dive on one, you can go ahead and do that. Let's click through a few of them and we can see that this plot will output something similar to what we saw on the Analyze Express Experiment tab where it shows pre experiment what the trajectories were about, the test group and a potential synthetic control group. What we're looking for here is the same thing we were looking for before of pretty close trajectories pre that experimental period. So this one with Atlanta, Chicago, Cleveland, this is the best one in terms of absolute percentage error. So let's deep dive that. If we scroll down into this step three three box, we can see it's already selected, but we could select any region from this drop down menu and we're going to click Deep Dive these locations. Now that this is done running, let's look at the results. So the first little box tells us what the high confidence and baseline confidence plans are for this design. In this case, the baseline confidence plan is looking at the minimum effect that produces at least 80% power and the amount of spend that it thinks is required based on the ROI you provided in order to create that effect. The high confidence plan usually requires a little bit more spend and targets a bit of a higher effect size than our baseline confidence plan. You can also download the locations of the test market in order to store them, use them or send them to someone else to help run your experiment. Now, if we scroll down here, you can see a deep dive into the actual power analysis. This first graph is a power curve which essentially shows us as the effect size changes, how does the power we have to detect a statistically significant effect change. And these lines tell you where that baseline confidence plan comes from. It's as soon as this power curve hits 80%. The results in this deep dive power analysis essentially simulate a bunch of different lift tests to try and explore what we think might happen when you run this lift test test. In the top graph here you can see examples of simulations where there is no lift. And in the bottom you can see examples of simulation where there is a lift. And you can see that reflected in these right hand plots where you can see that when we do simulate an effect that the test group starts to diverge from the holdout group, showing the causal effect that happened once we changed our spend in this test group. All right, that's recast Geolift. It was so great looking through this tool with you excited to deep dive this A little bit more in future content.