#57 The Credibility Crisis in Data Science

Hi, I'm Maggie, VP of Curriculum at.

Datacamp, and I know firsthand how overwhelming.

It can be to keep up with.

The rapid changes in data and AI.

But staying ahead doesn't have to be hard.

That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything.

From data and AI literacy for busy.

Executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works.

Power your data and AI transformation journey on datacamp.com.

This week I'll be speaking with Skipper Sebold about the current and looming credibility crisis in data science. Skipper is Director of Data Science at Sivis Analytics, a data science technology and solutions company, and also creator of the Stats Models package for statistical modeling and computing in Python. Skipper is also a data scientist with a beard bigger than mine. I know beard envy is a real thing people. We're going to be talking about how data science is facing a credibility crisis that is manifesting itself in different ways in different industries, how and why expectations aren't met and many stakeholders are disillusioned. We'll also see that if the crisis isn't prevented, the data science labor market may cease to be a seller's market and will have big missed opportunities. But this is not an episode of Black Mirror, so we'll also discuss how to avoid the crisis, taking detours through the role of radio randomized control trials in data science, the rise of methods borrowed from econometrics, and how to set realistic expectations around what data science can and cannot do. Welcome to Data Framed, the weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host Hugo Baun Anderson. You can follow Datacamp on Twitter @Datacamp and Megobown. You can find all our episodes and show notes@datacamp.com.com Community Podcast this is Data Frame. I also wanted to let you know that in the show notes you can find a form to make guest suggestions for the podcast. We welcome all suggestions and encourage you to have as panoramic a view as possible with respect to the diversity and inclusivity of our guests. Also, if you enjoy the podcast, it would be great if you could leave us a rating and a review on itunes, but only if you enjoy it or read out some of our favorite reviews on the podcast. So write one for your chance to be quoted. Now it's time for the show. Hi There, Skipper. And welcome to Data Framed.

Thanks. Happy to be here.

Great to have you on the show. And I'm really excited to talk about all the things you've been thinking about with respect to the current looming future credibility crisis in data science. How we can think about what the science and data science actually is and how to put it in there in a more robust fashion. But before we get into this, I want to find out a bit about you. What are you known for in the data community?

Sure. So I started the Stats Models project, so trying to make it so that people could do econometrics and statistics in Python. This was back in 2008, 2009 or so, kind of before pandas, before scikit learn. I got bit by the Python bug and really wanted to be using Python to do the work that I was doing in graduate school. I was in grad school for economics. So I picked up this bit of code that was part of scipy and took it out of scipy, joined the Google Summer of Code, this program that Google runs for doing open source software, and then kind of ran with it from there.

Cool. And so this 2009, you say.

Yeah, about that.

So this is when like the Sci Fi community was really getting some, getting some steam as well. Right. It was the early days, but, you know, there was a lot of work.

Being done there starting to pick up, I think my first, like, Sci Fi conference, the Scientific Python Conference. I'm fairly certain I was the only social scientist there, if not only, you know, one of a handful. I met Wes McKinney at the one after that. And that's when Pandas was starting to get off the ground and we started kind of talking to each other. We're up to and how we can make our projects complement each other.

Yeah. And matplotlib was being developed around the same time, right?

Yeah, matplotlib was the workhorse at that point for plotting. Scipy was relatively new. Numpy was still kind of relatively new. Pandas definitely did not exist. SciKit learn did not yet exist.

What about IPython? Fernando started IPython.

Yeah, was around, definitely using IPython almost from the start.

Great.

Jupyter notebooks were definitely not a thing. There was no notebooks.

Yeah.

And I was going to say for our listeners, IPython is an integral part of Project Jupyter, which you may know for Jupyter Notebooks and Jupyter Lab and all of these things. And you're also involved in SciPy, which is related to the PI data community. As well. Right?

Yeah, I remember the first couple of PI Data conferences. Spoke at a few of those and just trying to get the community off the ground and focused on how we can use Python to do scientific computing, not only just in traditional backgrounds like engineering, but also in stats and kind of data science.

For sure. That's pretty cool. I mean, the PI data conferences are all over the world now. And I went, I actually went to one in Berlin a couple of years ago, which was incredible. But I've been to several in the US and it's actually a talk that you gave at PI Data LA that kind of inspired the conversation we're going to have today as well.

Yeah, I think that was a good opportunity for me to get the things that were on my mind out into the world, force my thinking down into, into a little bit of a box and then get out there and start talking about it for sure.

So maybe you can also tell us a bit about you work at Civis analytics, so maybe you can tell us a bit about what Sivis does and what you do there.

I work at Civis. Our background, we got our start out of politics. So our CEO was the chief of analytics for the Obama campaign. After the campaign, he was talking to Eric Schmidt and they were talking about how great it was what Dan had put together in terms of people, processes and technologies for campaigns. And the hypothesis was that businesses are craving for the same thing. And that was kind of the genesis of the company. Since then we've branched out from politics and so we are a data science, technology and services company, but we work kind of across all industries, from CPG to healthcare, media, brands, still doing some work in politics, working with people who want to do consumer research, these kind of things.

Cool. And what do you do there?

So I run our data science R and D team and also I'm a product lead. So I spend a lot more of my time these days thinking about product. And the thing that I'm working on we call identity resolution. But the interesting part, the interesting data science part to this is it's basically entity resolution at scale. So how do we know whether two people in different data systems are the same person probabilistically?

Okay, interesting. So it sounds like you're wearing multiple hats at Civis.

I think that's fair to say. Often data scientists are wearing multiple hats, especially as we kind of of progress in our careers and become more embedded in the business, which is something I hope to kind of talk a little bit about today.

I look forward to it. I'm Imagining some sort of stacked hat chart actually. So you mentioned that you're bitten by the Python bug early on. Maybe you can just tell us a bit more about that. I mean, because people have. I got bitten by the bug when I could import CSVs using pandas into jupyter notebooks. Right? This is before notebooks and before PD readcsv.

No, that's right, yeah. So I think this is something I like to tell people a lot. My background, my major up, my last year of undergrad was poetry writing. So I could not have been further from math and data science and programming, but switched over to studying economics. Ended up going to grad school for econ. Really liked econometrics. Anyway, we got into our math econ kind of boot camp and first math econ class and it was taught in Python. So it was kind of the luck of fate because the year before and all the years before that had been taught in this language called Gaussian. But the professor decided to move over to Python using Numpy.

Was that a pretty forward thinking move on the professor's part at that point in time in an econometrics class?

Absolutely, absolutely. I had a few kind of early mentors, advisors who told me that Python was a complete dark horse and if I wanted to be doing this I should be doing R programming. I tried R as well, but working with Python it was just great. It really helped solidify the stuff that I was working on. I feel like I didn't really understand it until I could code it. And once I could code it I was like, oh sure, I totally understand this and now can program and speak competently about it.

I guess that's cool. But then there's the next step of I suppose realizing you had questions in economics and elsewhere that you wanted to solve and the packages didn't exist to solve them. So then you figured out somehow how to develop packages and the stats models was one of the results.

Yeah, for sure.

What does that look like? Going from being a proficient programmer in Python to start developing packages yourself?

I think it went the other way. I jumped in and the thing that was really helpful was there was a problem out there that I needed to solve. I needed to be able to run an OLS regression, get the results out of the regression in a way that was easily understandable. Build a user interface. I didn't know the first thing about programming, but I had some good and patient mentors and co developers along the way. I learned a ton from the open source community just by kind of honestly Failing in public, flailing in public, making pull requests, and having people take the time out of their day to spend hours reading your code and suggest how to be a better programmer. I think being the better programmer definitely came second.

Interesting. And that was really my next question around community, which is what? You know, the scipy community was really burgeoning at that point, and what was the role of that community in helping you to develop what you developed?

I mean, they couldn't have been more helpful. This was kind of before stack overflow. So I just signed up to all of the mailing lists and I just read everything that everyone wrote. I came to figure out who the people were who were just very forthcoming and forthright with their time. Here's what you should try. Maybe you should think about this. And I just saw like a group of researchers and scientists, like, discussing ideas and kind of doing research, engineering, I guess we might call it now, as one of the things that drew me to Python versus the R community at the time. This is like definitely changed in the R community. But jumping on the mailing list, the R help mailing list, kind of back in the day was just like a trial by fire if you didn't know what you were, if you what to say and what you are doing yet. So I found the Python community just to be a lot more welcoming and helpful for me.

Cool. So we're here today to discuss the credibility crisis in data science. What is this crisis to you? And I'm wondering if you can speak to concrete examples.

Sure. So it's a little bit of an analogy really. So the analogy is with this famous paper in econometrics and economics, and it's a paper by Angers and Pischke, which is a response to a paper from Ed lemur in the 80s. And this paper in the 80s was basically saying, like, hey, economics isn't having an impact. Policymakers aren't listening to us. We're over here arguing about minutia, like, let's go ahead and take the con out of econometrics. That was the provocative title for this paper.

That's a great pun.

And then later, you know, 20 years later, there's this paper, the Credibility Revolution. Empirical Economics. How Better research design is taking the con out of econometrics. So their premise was that there had been this maturation in economics and now policymakers took economics seriously. I see a lot of analog to what was in what's happening in data science today, kind of going through this, a little bit of a credibility crisis. And I See some of the solutions to be a little similar to what we saw in econ. So in econ, like some of the things that people were complaining about is there was just a lack of data in the studies. So some of these studies that people were doing had 20 or 30 data points. And they're like, yeah, it's kind of hard to make conclusions for this. And a lot of the economists spent time kind of arguing about just minutiae of what estimator are you using, what functional form are you using, should we log this variable or not? And there wasn't also the theory to just tell people what works and what doesn't. So you can understand, like, why policymakers are like, okay, y' all are having fun over there, but like, you're not actually affecting what we're doing, so we're not going to pay attention to you. And I think the. I see something similar in data science. There's a tendency to focus on minutia, like, what neural network architecture are we using? Are we using R, are we using Python? What method are you using? These kind of things which are not as important to decision makers. So one thing that I've heard just in working here at Civis is this quote I like to say from a CEO of a very large company that everyone would know if I mentioned who they were. And it was basically saying, I have all these data scientists, I have hundreds of data scientists and I have no idea what the fuck they do all day. And just it's like a part of a profession, like part of a class of jobs. That's not where you want to be. You don't want the decision makers and the people who are supposed to be benefiting from your insights not able to discern what it is you do, not understanding what your output is.

And that seems to be a result of many things, but including like a total miss or crisis of communication, right?

Yeah, I think that's exactly right. I think decision makers don't know what data scientists are capable of. They don't know how to communicate with them, what the business needs. And I don't think data scientists necessarily know how to describe the value of what they're doing in terms that the business can understand or also even focus their efforts on things that will have an impact for the business, like going out and understanding what it is that will actually make a difference and then making sure you're doing that. I know it's kind of a cliche now, but you hear people be like, well, have you tried logistic regression? Start simple, first provide value and then go deep, then go further. And a lot of times people want to start the other way. And I think that's part of why we're having trouble communicating.

Sure. And of course we have a lot of companies across a lot of verticals and industries hiring for data scientists, getting data science consultants to tell them to use data, all these techniques to tell them things that can impact, give them insights from data that will impact decision making. How impactful do you think this actually is? I mean do you think companies, a lot of companies like take these insights and go, okay, we're going to base our decision on these insights now?

I think it's highly varied. One, I think a lot of data scientists are like way over promising. And then two, I think the, the number of companies that are super forward thinking and then have kind of data centric thinking embedded in their decision making process are very, very few. You know, maybe there's maybe a handful that are really, really good at this, maybe a dozen. And then there's a long tail of thousands of companies who are not, but are seeing this happen and want, want to be in on it. So there's definitely some people that are getting this right, but the vast, vast, vast majority are not.

And do you think part of the challenge is that data scientists I suppose hailed as, as prophets or they have keys to the gates that A little.

Bit, I think it's also a little bit of a problem of marketing as this, the hype train is underway. You know, part of people's incentives are how can, it's a little cynical but like how can we monetize the hype train? So you see, you know, the, the use of things terms like AI and cognitive and even data science itself to start, like it's okay, what are the, what are the outcomes? What are the outcomes that we're going to enable and instead of what are the fancy methods we're going to use? That kind of thing.

Yeah. So the analogues there, I suppose we've spoken to other. The crisis in communication and expectations not necessarily being matched. Was there anything else that's analogous, you think?

I don't know, it's tough to say like a little bit just what is it that we do as either economists or what is it that we do as data scientists? How do we provide value? What is our process for working? How do we start from a proposition and then go all the way to providing something that will be an input for someone to make decision? I think that's the big analog.

That's great. Cause that actually provides a really nice segue into what I wanted to talk about next, which is in your talk, PI Data LA Talk, which was called what's the Science in Data Science? And we'll link to that in the show notes. You define data sciences using multidisciplinary methods to understand and have a measurable impact on a business or a product.

Yeah.

We'll jump right back into our interview with Skipper after a short segment. So I'm back here with Emily Robinson, who's a data scientist on the growth team here at datacamp to discuss her guidelines for online experiments. Hey there, Emily.

Hi, Hugo.

So we're talking about online experiments, and you have a great post and give a great talk about guidelines for online experiments, also known as AB tests. First up, can you just quickly remind our listeners what an a B test is?

Sure. So the idea behind an a B test is it lets you measure the impact of a change that you make on your website. And how you do this is you randomly assign people visiting your website to either the old experience, which is called the control, or the new experience, which is a treatment group. And then you run that for maybe a week, two weeks, and you compare these two groups, and you see, all right, if I cared about registration rate or subscription rate or conversion rate, buying items on my site, how do those compare between the control and treatment? And because people have been randomly assigned to either one of these, the idea is that everything else is equal. So the only difference between the two groups is the change that you introduced. And so any difference in behavior you see if you know if it's statistically significant, is likely because of that change.

Exactly. And so what you're essentially doing in these experiments is generating numbers that can help you make business decisions. And there's a great saying that generating numbers is easy. Generating numbers you should trust is hard. And you've written that there are many ways that AB testing can go wrong, but most of them won't be obvious. And you've written these guidelines to kind of spell out the types of things that can go wrong and give guidelines around that. So the next guideline that we haven't discussed yet is do not overcomplicate your methods. And I think this is a great maxim for life in general. But in terms of online experiments, what do you mean by this, and how complicated can it get?

I think a common mistake people make when they're starting out doing AB testing at their company is thinking that they need to have the kind of latest and greatest tools. Honestly, it's Similar to aspiring data scientists approaching a problem and saying, all right, let me throw deep learning at it, rather than trying say a linear logistic regression. And the reason I say that is actually there's a lot of experiments that you need to get right to have it set up correctly. So what you're going to get more return from at the beginning is making sure. Are we coming up with hypotheses for these experiments? Have we set up a good system for analyzing and viewing results? Are we avoiding peaking, which is something we've talked about in a previous segment. And basically the return from following some of these good practices and getting these things right is going to be more than using something like multi armed bandit testing or some Bayesian methods. Because the problem with those is that maybe one, honestly no one really understands the limitations of those methods or how they're used, or two, even if you have one or two people who understand that, what if they leave? Are they going to be able to effectively educate the rest of the company on what these methods mean and these inferences? So that's why I say it's good to just start by getting the basic frequented methods right. That being said, companies like LinkedIn or Microsoft or Google who've been experimenting for five or 10 years have data scientists dedicated just to the experimentation platform. They're publishing papers on the advanced research methods they're doing because for them they've got the basics right, so they really need to ink out that last bit of value. Or alternatively, they're dealing with network effects, like on LinkedIn. Right. If you're launching a feature, then it's a two way feature. How do you deal with that for LinkedIn?

Yeah, great. And I think that's really important to really consider. Don't overcomplicate your methods or keep it simple unless you're LinkedIn or Facebook, essentially.

Right, exactly. And again, that's because they have the bandwidth to do this right. They've gotten the basics right. They have data scientists and data engineers whose full time job it is to think about this experimentation platform and who have very specialized training in this.

So your next guideline is be careful of launching things because they don't hurt. Now what do you mean by this and why can this be a dangerous practice?

So by don't hurt, I mean the case where, oh well, you know, the metric didn't go up but nothing went down either, at least significantly. So maybe your P values are all around like 0.5 or 0.7 and it can be very tempting here to say, okay, well you know, we spent this time making this change. Why don't we just go ahead and launch it now? The reason this could be a problem is there may actually be a negative change that's too small to detect that you weren't powered to detect, but that could be meaningful in the long term. Additionally, you have to think about the technical debt or engineering costs that you're incurring by ramping something up. Now, that's not to say you should never launch things when they're, quote, neutral. I actually advocate for deciding before you launch a test whether or not you would want to ramp it up if it is, again, in sort of quotes, neutral. And what you can do for this is one, do a calculation to say, all right, we're running this test for two weeks. What's the negative effect that we could detect in that chance? Are we okay if there might be a negative effect smaller than this? And is this test foundational to some other tests that we want to run? And so even if it may not have an impact in itself, it's something that users have been asking for or something that we need to run this next test that we really think is going to be impactful.

Great. And I love the idea of making. Incorporating kind of the decision function, how you'll act on results into the experimental design process so you know exactly what will happen. In any case, doing that beforehand and not doing it post hoc, that's really exciting. So, Emily, this is all we have time for now, but I look forward to chatting with you again really soon about further guidelines for online experiments.

Thanks, Hugo. I do as well.

After that interlude, it's time to jump back into our chat with Skipper Sebold. So I kind of want to tease apart what this means and what the steps involved are. So I suppose my question is, how does this process relate to the scientific method? How do we put the science in data science, and what are the key steps involved in actually measuring the impact?

Yeah. Okay, so one of the things that's pretty common among data scientists, it's not ubiquitous, but one of the things that's pretty common is a background in the sciences. Right. So everyone has been trained in the scientific method. What is the scientific method? We want to start with a relevant question. We want to come up with a hypothesis about that question, something that we can falsify. Then we want to design our research program. How are we going to design an experiment that's going to allow us to know whether or not our hypothesis is false? We want to analyze the results of that and Then finally we want to either communicate or this is not exactly. It might be part of the scientific method, maybe some applied science. How do we get this into a product? How do we get this into decision makers hands? Or how do we make a product better with what we've discovered? It just doesn't look that different in working as a data scientist day to day than it would working as a scientist in a lab. The question may be a little different. You may not be asking what is the nature of the universe? You may be asking what's the most relevant business question that's driving us nuts right now? Do we know how well our marketing campaigns are performing? Do we know why we are attriting employees, why employees are quitting? Something like that. But from there on out, I don't think it looks terribly different from science and method if you're doing it right.

Yeah. And the one other difference I think is timescale. Right. Of how long a project.

Yeah, that's fair. That's certainly fair.

Three years to publication.

That is certainly fair. Yeah, we're probably talking about months or years. If you work somewhere, if you're lucky to work somewhere, that allows you to think in the year scale.

Great. So I suppose then the first point as you're saying is to figure out why you're there as data science, right?

Yeah. Like go out and talk to as many people as you can and figure out what it is that you need to do to provide value to the company. I mean, if you're, if you're somewhere that already kind of understands this and is just lacking the people who can execute on the data science, then that's great. Quite often the businesses don't even know how to ask this question. Right. They don't even know how to provide the inputs to the data scientists. So the data scientists can be maximally productive. So I think it's a little bit on the data scientists to go out and seek these things out, asking good questions and understanding, hey, what's your job? Hey, what's your job? That kind of thing.

And to make sure that the question as posed to them from a product manager, for example, is the question they're working on in their code and in their process.

How do you mean?

Well, I mean that a product manager can pose a question and you go away and you're actually working, working on something quite different, not actually answering the question that they've asked. Right?

Yeah, that's definitely a possibility. I'm glad that you brought up product managers because I think if you find a Good product manager. A data scientist can be a perfect partner or even a good product manager eventually. But even product managers, they don't always know what's possible with data science. So they don't always know what's the right question to ask. And then if they do know, what's the right question to ask. Yeah, it's entirely possible that the data scientist then goes off and it's like, I'm going to come up with the greatest neural network architecture and I'm going to maximize the accuracy of this model to the hundredth decimal place and it's going to be great. But what they really needed was an interpretable model or something like that. So sometimes the data scientists can lose the force for the trees. And I want to urge people not to do that.

Yeah, exactly. So this is all in, I suppose, from your definition or from your statement, understanding a business or a product. How do you measure the impact of data science?

Yeah, once you understand what the question is, I think measuring the impact, and this again cuts the heart of the scientific method, is teasing out everything that is not related to the thing you want to measure. Right. So you want to be able to isolate the effects of the thing you're trying to measure. That answers the question. So to be concrete, how do our marketing campaigns perform? So you want to run a single campaign and then you want to test whether or not that campaign had, let's say, a better than break even performance. So we went out and spent $100,000. Did we as a business realize gains above $100,000 by doing this campaign? So you just want to wash out everything else and make sure you focus on that and are actually measuring the return to the thing you think you're measuring the return on.

Great. And I think this leads nicely into something you've argued is that the credibility crisis essentially boils down to experimental design. I just wonder if you can expound on this a bit and tell us what it means and why it's the case.

Yeah. So the gold standard of experimental design science, randomized control trial. What this means, the randomized control trial is designed to understand the effect of an intervention, traditionally, whether it's something like a drug treatment effect, a jobs training program. I mentioned a marketing campaign already. What you want to do is design a way to understand whether that intervention had an effect or not. And the best way to do that is to take a group of people, mostly the same or representatively random, and then split them up, give one the treatment and one not the treatment. Another good example of this is just twins. Take two twins that are alike in every component of their life, Similar upbringing, similar genetic makeup, give one a drug treatment and don't give the drug treatment to the other, and then measure whether or not the treatment had an effect. So that's why experimental design is really important.

And this speaks to the point of isolating the effect. Right, of what you're trying to measure.

Exactly. You want the only thing that's different between two groups of people to be the intervention, the thing that you want to measure the effect on. And the randomized control trial is the kind of best way to do this.

So you make it sound easy.

Oh, yeah, it's pretty easy. No, no, it's definitely. I mean, there's a lot of things that you have to overcome to be able to do a randomized control trial. They're the gold standard in medicine and in science. But when you're dealing with more ambiguous things or even business processes, it's a lot harder to get one off the ground or convince people that you want to do it.

Absolutely. And I suppose a lot of our audience will have heard of randomized control trials, or RCTs, in the guise of a B tests, right?

Yeah. So a B test is usually a good introduction to randomized control trials. So think about, like, an outbound campaign for sales. You work in a business that does direct sales, which means reaching out to people who are potentially already your customers, and seeing if they want to buy something else from you or upgrade what they're already buying. So you have this kind of hypothesis that a certain kind of campaign on, like, a certain offer on these people that are already engaged with you is going to have a large effect. You're going to get more sales. So what you do is take all those engaged people, break them up, give one the A treatment, which is an offer that you want to measure the effect on, and another the B treatment, which may be some kind, like the offer that you were already giving them. And then look at the difference afterwards. You want to measure and see whether the effective A was different than the effect of B.

Great. So that's a great example. And thank you for using example, which isn't the color of a button on a webpage. And, I mean, that's really important because what you've stated is a hypothesis that's based upon something rational as well. Right. Like it was. I've discussed this on this podcast last year with Lucas Vermeer, who runs online experimentation@booking.com and, you know, the idea that, let's say you wanted to change the color of a button because it increased contrast or something like that. That can be a good example. Just examples out of the blue, so to speak, aren't necessarily that helpful.

I don't spend a lot of time doing data science that drives that kind of level of product differentiation. So that's definitely a part of something data scientists do. But we're much more centered on, like, processes and kind of human behavioral decision making, like within a business.

Right. And what role does your economics training play in this type of work?

That's a good question. I think it's. It's intuition, seeing a problem and seeing the solution to that problem. So economics is like the bread and butter of economics, is dealing with dirty data. Like they say, it's often unethical in economics to go out and give some people a welfare reform and others not give that welfare reform to other people to see whether or not it has an effect. Often we have to try to make sense of something after the fact. So we already have data that represents the real world. We observe that, and then we have to go back and use methods or techniques to reconstruct something. That looks a lot like a randomized control trial, but is it a randomized control trial?

Oh, I love it. And that actually leads very nicely into the next question I had, which essentially was around what type of pushback you can get in organizations. I mean, I know, for example, that people, if you're trying to do an A B test, managers might be like, if we think one's better than the other, let's just send the one that's better out to everyone. Why do we want to, like, send out stuff that doesn't do as good a job? Right?

Oh, yeah. Getting someone on board with an experimental design from the ground up can be a huge cultural shift for people. People have been doing their jobs probably pretty well for decades sometimes, and they think, or, no, I know how to do things. This is what works. This is what doesn't work. I don't know why we would want to try to kind of waste money and waste time to discover something that we already know. We already know this works. So in the sales example I gave earlier, like, if you go talk to the sales team and explain to them what you're going to do, and you say, hey, okay, I want this group to offer this incentive, and I want this group to offer this incentive, the one that we've always been offering the group that goes, wait, the other one's better. I just want to use that. Their incentives aren't aligned because they get paid on commission. So they say, why would we even do that? That's not going to fly. And then sometimes you need to have a control group. So if you want to test the efficacy of a digital ad, for example, you might want to put out an ad that advertises your brand or whatever, but you need to send the same group of people an ad that is not for your brand. So you can have kind of a baseline to control against and you got to pay to do that. So someone who runs a marketing campaign is not going to spend half their budget on sending a PSA to people or something like that just to measure a baseline. It's just too much money.

So how do we overcome these types of pushbacks? I mean, it seems cultural to a certain extent, but also practical in some ways.

It's cultural, it's practical. I think one way to do it is to use the data that you already have and design a kind of counterfactual experiment using observed data and then demonstrate slow wins, demonstrate a small win. Working with the data that you have using techniques from the social sciences quite often, because that's what they are designed to do. Show what's possible and then slowly build credibility.

Great. And get people on board.

Yeah, yeah. You got to play the game of politics a little bit. I don't even think it's really politics though. I think it's just convincing people of that data driven decision making is going to make them better at their jobs.

Yeah, absolutely. And as you say, people have been doing their jobs, this type of stuff for a long time. And particularly we think about, we hear about data science in tech a lot of the time, but a lot of industries that predate tech are incorporating data science into them. Agriculture for example. So trying to convince people working in agriculture that data science can help them get better returns. Right. That's non trivial.

Absolutely. A lot of early statistics was developed to try to understand what interventions to take to increase crop yield.

Absolutely. So in that case, what can we learn from the social sciences with respect to this burgeoning credibility crisis in data?

Yes, I've kind of been talking around some of the methods that we've about talked developed over the years to try to answer these questions. But maybe if I could just give a few examples. So first start with what was the problem we were trying to solve and then how might we attack that problem? My favorite example is this experiment at a fish market in New York. I think it no longer exists, but it was called the Fulton Fish Market. And the reason this was interesting to economists is that you would think a highly competitive lot of suppliers, there's a lot of buyers all coming into the same place and buying fish, you would expect the price to converge very quickly, expect this to be a very competitive market. And in fact, it wasn't always until economists saw this as like a little test bed of what they could study. But just to kind of describe one thing that they were interested in. Say they just wanted to measure the demand curve for fish so, you know, the price goes up, people buy less. That's the idea. But what we observe is, is a price. We just observe a price at which fish was sold. And as everyone has taken Ecom 101 knows, there's. There's two things that go into determining a price. You know, what's the supply of that thing, how many fish were available that day, and what's the demand? How many buyers showed up that day? So one of the techniques in econometrics is called instrumental variable regression, and it sounds super fancy, but the idea is we just wanted to find something, something that would vary the price of fish but only affects supply of fish and does not affect the demand for fish. So people are still going to show up and buy a fish that day, but let's say that the supply got curtailed or there was more supply than usual for some reason. The thing that they came up with for this paper in this example was weather. So if there were stormy seas one day, the supply of fish would be greatly limited, and so you would see no changes in demand. So you could trace out kind of the demand curve a little bit better by using weather as what's called an instrument. That's kind of one example.

Cool. And we'll actually, we'll link to your, as I said, to your PI data la talk. What's the science in data science where you go through examples like this and techniques like instrumental variables in a bit more technical detail for those interested?

Yeah, a little bit more technical detail. Not too much math. Hope it's still accessible to people. Sure, we'll find out.

Yeah, very cool. We'll jump right back into our chat with Skipper Sebold after a short session segment. All right, Emily, we are back for more guidelines for online experiments. For those of you who are just tuning in, I'm here with Emily Robinson, data scientist on the growth team here at Data Camp. Hi, Emily.

Hi, Hugo.

So we're talking about your guidelines for online experiments we've just talked about. Do not over complicate your methods and be Careful of launching things because they don't hurt. Now your next maxim I love it's have a data scientist and or data analyst involved in the whole process. So why do you need this and what can go wrong if you don't?

So I'll start with a great quote from Sir R.A. fisher and he said to consult the statistician after an experiment is finished is often merely to ask them to conduct a post mortem examination. They can perhaps say what the experiment died of.

I love it. Yeah. Why is that relevant here?

Of course you know that this was not, he was not specifically talking about a B test. This was long before there was online experiments. But why that's relevant here is let's say a team comes to the data scientists after they already have been running an experiment for two weeks. Why? Well, there's a couple of things that could happen. Maybe it turns out they actually weren't measuring the data for their key metric. And so the data scientist is just going to kind of shrug and say, well, you wanted to know you launched this new button or something and you wanted to see what's the click through rate and you weren't measuring clicks. So I have no way of telling you. Was that 10%, was that 20%? I don't know. Another issue they might find is that they come to you and they say, oh, running for two weeks. And like, oh, we're not really seeing a change here, so I guess it didn't work. And you say, well, well actually you are severely underpowered, so you would have had to have a 40% increase in conversion rate to detect a change within two weeks. And the team would say, oh, wow, we never thought we would have done that. We were hoping for a 10% change. And then they've sort of wasted this effort of implementing and launching experiment where it turns out they never would have been able to measure the impact that they think they would have had. And finally, maybe there was some other kind of design flaw in the experiment that just means they're not able to draw the conclusion that they wish to.

Your next suggestion is only include people in your analysis who could have been affected by the change. And this is something we've kind of hinted at before, even when talking about, you know, segmentation of users and that type of stuff. So maybe you can tell us why this is the case.

If you have users in your experiment whose experience could not have been impacted by your change, you're adding noise and reducing your ability to detect an effect. For example, if you're changing the layout of the search page. You should only add users to the experiment if they visit the search page. You could imagine like if users visit the home page and never the search page, obviously they're not going, their behavior is not going to change based on the treatment that you're introducing. So it can get a little more complicated than just that. For example, there's a great paper on triggering where they talk about changing the threshold for a free shipping offer from $35 to $25. In that case, you should actually only put users in the experiment who have cart sizes between 25 and $35, because those are the only people who would see something different between the treatment and control group. Finally, you should start tracking your metrics after the user sees a relevant page. If you're running an experiment on the search page and someone visits your site, buys something from the homepage and then visits your search page entering the experiment, you don't want to count that earlier conversion because that again, it could not have been a result of your change.

This is really all in the name of having more sign signal and less noise.

Exactly.

Great. All right, those have been two fantastic guidelines and we're going to be back soon in a later episode for one more segment to wrap up all of your guidelines for AB testing and online experiments. And I can't wait, Emily.

Me too. Thanks, Hugo.

Time to get straight back into our chat with Scaba. So as it really seems to boil down to design based thinking with respect to experiments and research, could you provide a couple of examples of such thinking?

Yeah, so some examples that are like a little bit more on the business side than the kind of economics history side. One of my favorite papers, it's become one of my favorite papers here is, it's called Courtyard by Marriott Designing a Hotel Facility with Consumer Based Marketing Models. So it sounds kind of boring and I can't believe that my favorite paper is in like quantitative marketing now. But it's, it's so clever.

And Courtyards are great as well. The Marriott Courtyards are really good.

It is like to me the shining example of good, like data driven decision making built on the back of like a very, very good but simple research design. It's also very old. So it's, I think this paper came out in 83, 84. Courtyard by Marriott started in the early 80s. So it also kind of shows how far back starting to do this kind of decision making was happening and doing it well was happening. The premise is, the kind of punchline is that Courtyard by Marriott was a design by we didn't call them this at the time, but it was designed by a bunch of data scientists. It was designed by a bunch of statisticians.

That's awesome.

So how did they do this? So they used another kind of technique that came from quantitative psychology and marketing, which was basically they ran a survey experiment and then they used some econometric methods to do some measurement. So what was the survey experiment that they used used? It's called a conjoint analysis. So some people may be familiar with this. It's kind of now heavily used in product marketing or product management. And the idea is just that you have a bunch of different things about a product that you want to test, a bunch of different hypotheses that you want to test. And only so many people you can ask those of. And you can't ask everybody everything, but you can ask some people some of the things. So to be more concrete, there were seven facets that they wanted to test in this Courtyard by Marriott paper. They saw decline, Marriott saw kind of declining sales. They knew they had two kinds of travelers, business travelers and people traveling with their family. But they didn't know what they wanted, right? So they said, what are the things you care about within these seven facets? What should the rooms look like? What should the food be like? What should our lounge look like? What kind of services do you care about? What kind of security do you care about? So they asked a bunch of questions within each of these facets, something like 50 attributes for each facet and each one of those had eight different answers. So do you want HBO in your room? Is basic cable? Okay, do you want an L shaped pool? Do you want the pool to be in the middle of the hotel? All of these kind of things ran that experiment. And then it was the made some decisions based on what they learned during that experiment and it was wildly successful. When they went back to measure in like kind of the middle of this undertaking, they had used three test cases early and they saw as they grew up this test case cases $200 million increase in sales. And they expected that to grow to within a billion dollars in sales by the early 90s. So like over a decade. They expected a billion dollars in new sales, incremental sales to come from making these decisions, to come up with this new brand. Courtyard by Marriott, which was the new concept, created a bunch of new jobs, 10,000 new jobs for Marriott. And then the whole industry changed. So all of the competitors had to change what they were doing, doing and just the whole idea of what it meant to have kind of affordable but nice lodging changed.

That's incredible. And so they also predicted the return or the increase in sales. How was that? Was that increase met?

Yeah, so they ran a bunch of simulations so they knew what people said they were going to do. They ran a bunch of simulations knowing that some people may not do that. And they predicted the share of the market that Marriott would capture by undertaking this. I think that's probably the thing that the sales numbers, I'm not sure they forecast it but they did forecast the percent of market share they were going to overtake and they were right within 4%. So that's probably the thing that caught the decision maker's eye. Right. Oh man. So you're saying I can get this. Let's start small with a couple of test cases. They're starting to see the results. Then they went all in. And it turned out that by carefully designing how they were going to measure what they thought, they were able to make this prediction and ultimately be right, win the market. It was pretty amazing.

That is an amazing example of a success story in research design based thinking. Are there any unsuccessful examples you can think of?

Yeah, yeah. So I mean it's not all unicorns and roses. There's a lot of places where it's going to be very difficult to either predict success or even measure success. So I think it is important. I mentioned this a little bit earlier, but it is important for data scientists to be upfront about what we can do and what we can't do so that we're not over promising. There's another example that I like that comes from the Yahoo Research group. So they were trying to measure the return on some advertising campaigns. They had pretty much unfettered access to Yahoo data and then they actually worked very closely with something like 25 firms to get information on conversions for advertising. So one of the problems with trying to measure the returns to advertising, it's sometimes hard to capture conversion. So if it turns into an in store sale, how do you make the connection that that actually happened through ads. And they were able to work closely with like department stores and financial services to make sure they were measuring the outcomes. The name of the paper is Unfavorable economics of measuring the returns to Advertising. So it's a little bit bit of a reality check on these things. The conclusion is it's not impossible, but it is very, very difficult. And what makes it difficult is a combination of two things. One is the returns to advertising are very, very small. Campaigns themselves usually have a low spend per person. And the return on investment, therefore, is very, very small. So you may be looking at like a 10% bump if you're lucky. So that's kind of a small effect that you may want to measure. But the. And then compound that with the fact that things like sales in a retail store are super, super noisy. So most people do not buy things, but when they do go buy things, they're really, really expensive. So the standard deviation, you know, how big the variance is of a sale is a lot, lot higher than the average. So when you combine those two things, I want to measure something very precisely, but the thing I'm trying to measure is very noisy. It's really hard to design an experiment to measure that.

And you need to just get so much data, right?

Yeah, exactly, exactly. You have to keep getting data to be able to measure more precisely, which makes things very, very often very, very expensive to do. And if you don't think about these kind of things up front, you may run a very expensive experiment and then come to the conclusion that, oh, I don't have enough information to measure what I thought I was going to measure.

Sure. And I suppose in this particular example, doing some sort of power analysis at the outset will tell you, like, approximately how much data you need and how much that would cost.

Yeah, exactly. This paper actually did kind of a post hoc power analysis. So it went back and looked at 25 different experiments and wanted to know after the fact, would we have been able to measure accurately these things? So what is power? Kind of. First of all, this was always a concept that mystified me when I was first starting to study statistics. But the basic idea is just that if there is an effect, if there is a return to advertising and it is out there to be measured, the concept of power is, do I have enough data to be able to measure it? And the conclusion that they came to was that quite often studies that are trying to measure the returns to advertising are what we call underpowered. So if they wanted to measure a very precise return, they just didn't have enough data to do it. And saying that they did would have been disingenuous.

A psa. It's always good to do some sort of para analysis during the experimental design process.

Yeah, I think so.

When you said post hoc power analysis, I got shivers down my spine.

Sure, sure. People that are kind of more Bayesian in their statistical, philosophical thinking may argue with whether power is a useful concept, but intuitively, I think it makes sense.

Absolutely. And if there are any Bayesians who feel that way out there. Ping me on Twitter and I'd love to have you on the show to go head to head. So, Skipper, this has been a really, I think, thoughtful introduction to the challenges we're facing as a discipline today, particularly, particularly in terms of the relation of data analytics and science, and data science to the decision function. What can we do? How is the credibility crisis preventable?

One of the things that we can do and the talk that I gave and kind of the premise of this whole conversation is that by better research design, by using different methodologies, then you can gain credibility, you can get some wins and you can have an impact. But I don't want to overemphasize just methods. I don't want people to take away that, oh, if I go read an econometrics textbook, then I will be like super credible. I will always deliver results. You have to make sure you're solving the right problem. So making sure you understand the business needs, what it is that people are trying to measure and how to actually have an impact and then go do things correctly. So make sure you're solving the kind of right problem and then slowly build up credibility within your organization and then like more broadly within the profession, I think.

And what about. I always hesitate to use the term soft skills, but I've been too lazy to think of a better term just about in terms of dealing with the bureaucracy and structure and hierarchy in organizations. How does that relate to this conversation?

My prior on the typical data scientists, which may or may not be accurate, is that they come from an academic background. They're very used to long timelines with not a ton of stakeholders in what they are doing. So they may be going off and working on a single problem with their advisor or maybe lab mates very deeply for a long time. And the business world is slightly different in that you're going to have multiple stakeholders and people are going to kind of lose faith very quickly. It's not just methods, it's not just good research design. It's good project management, good communication, letting people know where you are and being kind of honest with them about. Am I stuck? This is an assumption that I've made. It may not be true. Here's when you can expect results. Go out and read a good book on project management because even if it's not in your description as a, as a data scientist, you're going to build up credibility and you're going to have more soft skills if you approach things like that.

Great. Do you have any recommendations for books on project management?

Yeah, there's one that we read around here. We have several book clubs. And one of the things that we read in book club recently was this project management for the unofficial project manager. And the premise is like kind of exactly what I just stated. Like whether you like it or not, you're a project manager. You may not know it and you may not even know what that means because no one has ever talked to you explicitly about that. Here's how we think you can be effective at this. Know where you're going before you start that kind of thing. Which is also kind of the opposite of some academic pursuits. Like sometimes you can find very interesting, interesting things by just by going down rabbit holes. And that's something not necessarily always to be avoided, but often to be avoided in a business context.

Okay, great. And we'll link to that book in the show notes as well. To push even a bit further. What could happen if the credibility crisis isn't prevented?

This is kind of what keeps me up at night a little bit. And I think the really cynical answer is that the job market for data scientists kind of drops out. So it ceases to be a seller's market. In that data science, data scientists will be hired less and paid less. You hear this. This is the sexiest job of the 21st century. Those kind of things again and again. And that is pretty far along in the kind of Gartner hype cycle into the trough of disillusionment here that I hinted at earlier. And if that continues, then I think you'll cease to see data scientists be like a really emphasized and hot job title, which would not be great for us. I think it's a lot of fun to be a data scientist.

I choose encouragement. Where we are in the Gartner hype cycle, I suppose is a function of industry as well. I think in a lot of industries we're still at the peak of inflated expectations.

Yeah, that's probably true. I mean it's a little bit of a balance and it depends who I've talked to that morning. But yeah, that's definitely true.

So that was your cynical response. What else could happen if the crisis isn't prevented?

I don't know. I'm not sure I have a really good answer to that. Nothing. We go back to the status quo. People go back to making gut based decisions and the kind of. The promise of data science is never actually realized. So it's not just. It's somewhere in between the trough of disillusionment and kind of overhyped expectations. There is value in making evidence based data driven decisions. And it's kind of up to data scientists to prove that.

Yeah, agreed. So to wrap up, you know, we've talked about a lot of your different interests, but something I like to ask is just what one of your favorite data science techniques or methodologies.

Sure. So I think logistic regression is a great one. I joke a little bit. It's quite often we ask people, have you tried logistic regression? First, but a little less glib. We've been looking a lot at using factorization machines a lot here. It's a cool technique that kind of allows you to explore the interactions between data in a way that it's tractable. So you can kind of compare nonlinear effects really, really, really easily. And we found we got a lot of use out of factorization machines.

Awesome. So to wrap up, I'm wondering if you have a final call to action for our listeners out there.

I'll leave people with kind of one hard call to action. It's something that I talk about in my talk at PI Data la and that's like the benefits of having a journal club at work, having a bunch of like minded colleagues and going out in the literature, finding a paper, that's cool, reading it, breaking it down and discussing it with your colleagues on like a weekly basis. I can't kind of overestimate how much value and how many ideas we've generated just by doing something like that.

Fantastic. Skipper, it's been such a pleasure having you on the show.

Yeah, it's great talking to you, thanks for having me.

Thanks for joining the conversation with Skipper about the current and looming credibility crisis in data science. Part of the challenge we now face is that decision makers often don't know what data scientists can and cannot do do. And there's a lack of communication around actual business needs. On top of this, data scientists often don't have the tools to communicate the value they themselves can create or even focus their efforts on work that will be impactful for the business. As Skipper said, and I quote, one, I think a lot of data scientists are way over promising and two, I think the number of companies that are super forward thinking and then have data centric thinking embedded in their decision making process are very, very few. Few. There's maybe a handful that are really, really good at this, maybe a dozen. And then there is a long tale of thousands of companies who are not, but are seeing this happen and want to be in on it, end quote. The solutions as we saw, involve good, robust and reproducible systems. Of experimental design, which have been developed a lot in the social sciences and economics. But Skipper also made clear not to overemphasize methods. He made strikingly clear that you need to understand business needs and be solving the right business problems. Figure out what people are trying to measure and how to actually have an impact and do things correctly. Next week I'll be speaking with Debbie Barebaches about the importance of critical thinking in data science. Debbie is a physicist, TV host and data scientist and is currently the Chief Data Scientist at Metis in New York City. In a world and a professional space plagued by buzz terms like AI, big data, deep learning and neural networks, conversations around skill sets and less than productive programming language war, what has happened to critical thinking in data science and data thinking in general? What type of critical thinking skills are even necessary as data science, AI and machine learning become even more present in all of our lives? And how spread do they need to be across organizations and society? Join us next week to find out. I'm your host Hugo Bown Anderson. You can follow Datacamp on Twitter @Datacamp and Megobound. You can find all our episodes and show notes at datacamp. Com Community Podcast.