Ryan introduces A/B testing as a statistically sound way to compare ideas by randomly assigning users to variants and measuring outcomes. The course covers selecting what to test, defining KPIs like revenue or playtime, exploratory data analysis, and designing experiments across industries from pharmaceuticals to mobile games.
Chapters
Hi, my name is Ryan and I will be your instructor for this course on customer analytics and A B testing. This is a fascinating subject and I'm looking forward to the opportunity to explore it with you. The primary focus of this course is A B testing. A B testing is a tool that allows you to test two or more different ideas against each other in the real world and to see which one empirically performs better. Because you are running this test in the real world, there is no guessing. You get to know which idea is better under the conditions that matter most. Beyond that, it has many other benefits. It can provide accurate answers quickly, allowing companies to rapidly iterate on ideas. At its core, it is one of the only statistically sound ways to establish causal relationships. We will dive into all of this later in the course. Simply speaking, A B testing works by exposing a unique, randomly assigned group of users to each of the ideas you want to test. Then you can observe these users and by measuring how they behave, untangle the impact of each of your ideas and ultimately determine which is best. If you have users and ideas, then chances are you can run an AB test. It is utilized in fields as diverse as pharmaceutical companies testing the impact of different drugs to mobile games, trying to incentivize users to spend more, and subscription services working to drive user growth, as well as many more use cases beyond these before you can perform an A B test, you must first understand what is worth testing and optimizing for. The first chapter will cover this topic in detail. Once we understand these metrics, we discuss exploratory data analysis in chapter two, which leads naturally to the bulk of the course on how to design and analyze A B tests in practice. These topics will be covered thoroughly in the final two chapters of the course. We will finish off this video by briefly discussing KPIs. Typically, AB tests are run to improve key performance indicators or KPIs. These are the metrics that are most important to the business or organization you are part of. For a drug company, these may be things like remission rates or the likelihood of a particular side effect. For mobile game, it may be something like revenue or playtime per user. Identifying the right KPIs requires a combination of experience, domain knowledge and exploratory data analysis. Experience and knowledge will tell you what is likely to be an important driver of the business, and analysis allows you to uncover relationships that reveal which metrics Dr. Truly measure these drivers. Starting in the next video, we will dive headfirst into exploring a customer data set and begin to work through an example, to identify which metrics are meaningful and which are not. We'll explore this in a way that interleaves the mechanics of doing this analysis with the art that makes up much of the data exploration. We'll continue with similar examples throughout the course to show how the A B testing process grows out of These ideas of KPIs and exploratory data analysis. Now let's work through some exercises on the high level A B testing material we just covered. From there we will dive into some live data and code.
