#9 Data Science and Online Experiments at Etsy

Hi, I'm Maggie, VP of Curriculum at.

Datacamp, and I know firsthand how overwhelming.

It can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works.

Power your data and AI transformation journey.

On datacamp.com in this episode of Data Framed, a DataCamp podcast, I'll be speaking with Emily Robinson, a data analyst at Etsy. Emily and I will be talking about online experimentation at Etsy, an e commerce website focused on handmade or vintage items and supplies, and how data analysis and data science are essential to their business. I'm Hugo Bown Anderson, a data scientist at DataCamp, and this is Dataframed. Welcome to Data Framed, a weekly DataCamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter hugobown and datacamp@datacamp. You can find all our episodes and show notes@datacamp.com Community Podcast Hey Emily, and welcome to Data Framed.

Hi Hugo, thank you for having me.

Such a pleasure to have you on the show. I'm super excited to have you here because, you know, we're delving deep into what data science can do, what it's capable of, what it looks like on the ground for working data scientists and data analysts. And it's a really great opportunity to discuss these things, kind of talking around enlightened experiments at Etsy. But before we delve into your work at Etsy, I'd like to find out a bit about you. How did you get into data science?

Sure. So I have a social science background, so I had a pretty heavy statistics background. I did a statistics minor in undergrad and then I went on straight from there into a PhD program in Organizational Behavior at INSEAD, which is a business school in France and Singapore. And so I was in the PhD. So the plan was, you know, five or six years of that, but then while I was in it, I decided to off track at the Masters. So I did two years instead and finished up in June 2020 16. And then when I was thinking about where I wanted to go from there, data science was appealing to me for a couple reasons. So it was a growing exciting field and it fit with my background of statistics and academic social sciences research. So the research I was doing, there's a very similar process to when you're tackling a data science question in that you come up with the question and you design a way to answer it, you collect and analyze the data and then you have to present it. And you know, in both social science and data science research, you can be presenting to a lot of different audiences. So either people who are, you know, non technical or non experts, people who are sort of vaguely in your broader field but maybe don't know as much about your narrow area, or you know, people who have technical 10 more years of experience in your narrow area and they're judging your work. So learning how to tailor your work to these different audiences. So in thinking about how to best transition, I decided to go to Metis, which is a data science bootcamp that's three months full time. So I wanted to fill in the gaps that I had in Python and machine learning and Git and version control. And it also gave me some time to build up a portfolio of data science projects that I could share through a blog and GitHub. If you want to learn any more about my background, I was interviewed by the R Forwards group, which is a task force working to promote women and other underrepresented groups in our programming language. And I believe the link will be included in the show notes.

It definitely will. And this is great. In a matter of maybe 120 seconds there, you've actually touched upon several points which are really interesting in terms of people trying to break into data science and establish data scientists in terms of the importance of statistics, experimental design, communicating results which, which, which is incredibly important, building a portfolio and filling in the gaps. I mean you mentioned you had a statistics background, but filling in the gaps when need be with respect to programming, version control, these types of things. And before we do dive in to talk about your work at Etsy, could you speak to how much statistics and or programming you need to know in your work and how much you can pick up on the fly as well?

Sure. So I think something interesting here is that depending on which subspecialty of data science you're in, the programming and statistics needs vary. So for example, we have some machine learning engineers here at Etsy who work on the search ranking team and they generally come from a pretty heavy programming computer science background, but don't know much statistics versus in my work I definitely do some programming, but I certainly would not say you need A degree or even multiple computer science classes to do it. But I use a lot of statistics because I'm working on the experimental design. And I think something that really captures. One nice way to think about this is type A versus type B data science, which as was shared in a core answer. And basically the type A is the analysis is sort of a traditional statistician and the type B is building machine learning models. And I would also say you definitely can learn on the job. So before I came to work at Etsy, I didn't know much SQL. I just sort of knew the basics that I could learn through a short online course. And I've been able to pick up a lot here. But that's helped partly is that I came, I was already able to offer something in statistics and my R skills in making graphs. And so I was offering something while I was learning these other skills like SQL and like scalding, which is what we use to write our big data jobs.

So you started with a math heavy background, particularly in stats, and filled in the gaps with respect to your programming chops, do you think? I mean, it's commonly said that that may be easier than, than knowing a lot of computer science and trying to learn all the stats and calculus later on. What are your thoughts on that?

That's probably true. And I think that's partly because there are a lot of great online resources for learning programming. I mean, Data Camp of course is one, but there are many others. I'm not as familiar with the stats ones just because I haven't had the need to learn it that way. I learned it sort of more traditionally through my college UNDERRAD courses. And I also think the motivation there can be more difficult in the sense of with programming you sort of get immediate feedback and motivation like, wow, I can make this graph, I can change this data set. And I wasn't able to do that an hour ago. Whereas you might not get. You can run with a stats thing even if now, okay, you can run a linear regression and sort of understand it. You could kind of do that before anyway, even having no idea what's going on behind the scene or what assumptions there are. And it can feel like it can take a little longer for getting a stats foundation to pay off.

Can you tell us a bit about Etsy and its business model? What happens there?

Sure. So Etsy is a global marketplace for unique and creative goods. So we don't have any inventory ourselves, but instead we connect our about 1.9 million sellers to over 30 million buyers. So we have about 45 million unique items. And the sellers range from people who just want to make a little extra money selling their knitting to people who do this full time and have a small team working with them and sell thousands of items a year. And Etsy makes money when our sellers sell because we take about 3.5% of the sale price and we make a little bit of money when they list. It costs 20 cents to list. And then we also offer some seller services like shipping labels and promoting listings. But basically overall we do well when our sellers do well. And that's how Etsy continues to grow. And our mission is to help our sellers thrive and buyers to find these creative and unique goods.

And you said from about 1.9 million sellers.

Yes.

Where are all the sellers?

So they're almost in every country in the world and close to every county in the United States. Um, so there's, it's, it really is a global marketplace. And that's one of the great things is, you know, if you're interested, you can get something right from, say, your little small neighborhood. So for me, maybe Brooklyn or Manhattan, but you can also get something from Japan or you can get a special Icelandic clothing item and order that directly from a seller.

Fantastic. So what type of business challenges does Etsy face that that data science and data analysis can, can help with and impact?

Yeah. So I would say data science is really spread throughout the company. So we have a small team of data scientists who really who work on our recommendation modules. So if you've been to Etsy, we have a section called our picks for you. So basically we've tried to learn what you like and what other items you might be interested in. Then we also have data scientists working on our ranking models. So when you search for jewelry, you get millions of items back. How do we pick which ones we show on the first page and what do we take into account? So maybe where you're searching from or what time of day. And then finally we also have the data analyst team, which I'm on. And so we do data science for human consumption. So we embed with different teams and I work with the search team. And so I specifically work a lot on experimentation and designing and analyzing the experiments and planning them. But we also have people working with the marketing team and they can do things like make it really easy. If the marketers want to target people with an email campaign who bought in the last 10 months in Germany, our marketing analysts can really help them easily access what users those are or they can make dashboards. So Even people who can't, we use Looker, which is a business intelligence tool that basically queries from our SQL database. And so even people who don't know any programming, if we've set up a dashboard for them, they can then say we have, okay, what is our total sales over this year? And they can easily sort of drag and drop and figure out, okay, what about if we look at Germany? What about if we look only at sales? Sales by sellers in March, maybe in Iceland or in Europe. And so we basically are really meant to release all this data that Etsy collects as an e commerce website and help people use that to make product decisions.

Great. And so that's an internal tool.

Yes. So we really work mainly internally facing as data analysts versus the data science team works on these directly on these products like our picks for your ranking algorithm that people outside, etcc.

I really want to delve into what type of experiments you work with and enjoy the most. In a second I'd like to first jump into the other types of challenges you mentioned were a recommendation challenge, which is really an exciting burgeoning field. We see that when we go to Netflix or go to Amazon we get recommendations that are on point occasionally but. But for the most part maybe relatively odd. So there's a lot of work to be to be done there. You also mentioned the ranking algorithm, which is when I search for something, it'll rank what the search results come in as. Right. And I suppose there's some sort of matching problem happening there as well.

Yeah. So this is really. So I knew almost nothing about search before I started Etsy. So one of the first things I did was just how does, how does search work? And so really in this there's sort of two problems. One is what items do we even return at all for a search? So how do we retrieve items? And then the next step is, okay, given we now have this set of 1 million items that has been returned, how do we rank them? And so we've been really focusing more on the generally Etsy, we're more concerned with the ranking problem because more often than not we already have enough items. So let's make sure we have the relevant items. But something that's really challenging here is Amazon about, you know, maybe 80% of the time there is, you know, a clear thing you're looking for. You search vacuum cleaner. You probably want the best vacuum cleaner. And maybe there's some things here that, okay, some people, you know, may want the best regardless of price. Some people may be more price sensitive. But you know, or they're looking for Harry Potter and they want, maybe they want the Harry Potter book, maybe the movie, and then maybe after that, like Harry Potter merchandise. But you have pretty good idea of what they're looking for. Versus, if you search Harry Potter on Etsy, you know, we return hundreds of thousands of things and we have no idea what people are looking for. So some people could be looking for a mug or a poster or, you know, a scarf or just, you know, hundreds of thousands of things. And so how do we figure out what's going to be useful for people? And so one way we're trying to tackle that is let's have users help us. So make new search filters. We have something called guided search. So if you search one of these vague, broad queries, we suggest, hey, maybe would you like to search instead Harry Potter mug or Harry Potter scarf? And letting them help us figure out what kind of items they're looking for or offer inspiration. If instead what they're looking for is, oh, I just kind of want to browse around, we can help with that, too.

That's awesome. Is there some sort of buyer onboarding?

So we don't have buyer onboarding now, but we are trying to make a more guided experience. So one thing we did for the first time this holiday season was offer a holiday gift guide. And so most likely if you go to the homepage or maybe at the bottom of your search page, you'll see things like Spirit Sipper or Book Club mvp. And if you click on those, you'll find all these items. And what I'm really excited about is we didn't do kind of the traditional gifts for him or gift for her or gifts for your grandma. Instead, we adopted these Personas to really try to help people get inspiration and figure out what Etsy can offer. And we have Editor's Picks modules where basically we do things like gifts for your mother under $30 around Mother's Day. But I think that's definitely something we, we're working on more, is how do we let buyers know what Etsy is? Because sometimes people come and don't realize it's a marketplace and don't realize that, you know, they're buying directly from sellers and not from Etsy. And so how do we build that understanding and excitement about Etsy when people come for the first time?

So you mentioned that you work as a data analyst. Could you speak more to the difference between being a data analyst and what data scientists do and how that involves human consumption?

Sure. So this is something that's I think really in flux across the industry is titles, because what. So for example, we recently had someone move to Spotify from our data analyst team and she has now the title data scientists, but has said, you know, I'm working on very similar things. And in fact Spotify and I believe Facebook as well recently made their data analyst change their title to data scientists. But here at Etsy, what it means is that as my boss put it, once the data science team works for machine consumption and the data analyst for human consumption. And so the data science team is more isolated in the sense that the data analysts we really embed closely with product teams. So I work with search and another data analyst might work with marketing, another one with our seller team, but we all work with partner teams versus the data science team here really works a little bit more isolated on these specific modules and on things like our promoted listings. So this is where a seller says, okay, I'm willing, I would like. On the search page we have certain rows that are dedicated to these promoted listings. And so sellers basically bid and say, okay, every time I get a click, I'm willing to pay 20 cents or, or 30 cents. And so there's a very complicated model that figures out, okay, every time you search for something, how do we fill those promoted listing rows? And so that's something the data science team works on. And so in that case, what they're doing, right, they're building models and maybe they're running their own experiments, but they're very rarely delivering something to other people to understand the data more versus working as a data analyst team. That's usually our primary goal is helping people. How do we make decisions? Should we launch this experiment? We advise our partner team, or how do we make this data more accessible for them? Or how do we help them understand we should be working? We have this great opportunity to change this page because we think it's not doing as well as it should or this page has a high bounce rate. Sometimes it's proactive from us. We make these discoveries and other times it comes as requests from our partner teams what they're interested in learning.

It's now time to dive into a segment called Data Science Toolbox with Mike Lee Williams, a research engineer at Cloudera Fast Forward Labs. Hi, Mike.

Hi, Hugo. Let's talk today about machine learning interpretability.

I can't wait. What does it mean for an algorithm or a machine learning model to be interpretable?

So if you ask people who work on this, you'll get a different answer from every person but the definition I like the most, just the most pragmatic, is an interpretable algorithm is one whose predictions you can explain.

So what's an example of an interpretable algorithm?

My favorite example, and it's not really machine learning, but I like it because it's so clear, is the APGAR score that babies are assigned at birth. When that happens, nurses measure five things, things like skin color and pulse and so on, and they give each of those things a score from 0 to 2. They add the five numbers together and get a number out of 10, and that number turns out to be predictive of the baby's long term health chances. Obviously, it's a heuristic. In the language of machine learning and data science, we could call this a linear model. It's a funny linear model because we've set every coefficient to be one. But a linear model it is. And we could make it more accurate by using machine learning to build a nonlinear model. We could do all sorts of things to make this model more accurate, but we would lose interpretability. And the fact that you can do this calculation, this APGAR score in your head, means that you, as the nurse, can quickly figure out what needs to change to improve the score. This less accurate but more interpretable approach saves lives precisely because it's a model whose predictions you can explain.

So in general, why is it important for algorithms to be interpretable?

So there are lots of reasons, but I want to focus on what I think is the most important. A model whose predictions you can explain is one you can trust. And obviously that's good for business. If I can trust a model to be accurate, I feel warm and fuzzy. But there are many contexts where that trust regulatory or ethical requirement. A bad model is dangerous, it's discriminatory or otherwise unsafe. The concern is that a model might seem fine when you validate it against your held out training data. But the question is, is it getting those examples right for the right reasons? Is it really generalizing or is it embedding dangerous misunderstandings about how the world works that you're only going to find out when you unleash this model on the real world is much easier to spot that kind of problem. A model that is right for dangerously wrong reasons. If the model can offer an explanation of why it thinks it's right?

Yeah. And isn't there a trade off between how well an algorithm can perform and how interpretable it is?

That's right. You get nothing in this world for free. And the bad news is that generally the more accurate a model is the less interpretable it is. So a nonlinear model like a neural network is super accurate. And if you do feature engineering, if you, you know, use your brain power to make it even better, is going to be more accurate still. But both of those things make it harder to explain the behavior of the trained model. So if you think back to that APGAR score we talked about, when a nurse is reasoning about what changes would drive the score up or down, really what they're doing without realizing it, is a very simple form of calculus. And good luck doing any form of calculus if you've got something like a random forest with engineered features. Now, that's not to say there's not a place for uninterpretable, very accurate models. It's a trade off that depends on where you're going to be using the model.

Right. So what work has your team at Cloudera Fast Forward Labs been doing in this area?

Our job really is to help our clients do machine learning better. And we came at this problem with essentially that trade off in mind, that trade off between interpretability and accuracy. And with the question, can you have it both ways? Can you have a model that is accurate and interpretable? And the answer is maybe, kind of. So we found two ways. The first way is to start with an interpretable model, and very carefully, in a very controlled way, give it some freedom to be a little bit more accurate. So an example of that, if you think back again to the APGAR score, we have this linear model where the coefficients are 1. What if I allowed the coefficients to be integers still, but numbers other than one? We'd retain some of the interpretability, but. But we'd significantly, potentially increase the accuracy. That specific approach, by the way, has a real name. It's super sparse linear integer modeling, if you want to look it up. The other approach we took, and I think this is a more generic approach of use in more situations, is to build a model however you like and then to apply techniques after the fact to try and interpret it. We used a tool called Lime, which was created by Marco Ribeiro at the University of Washington and his collaborators. It takes a model and treats it as a black box and tries to understand what's going on inside the black box. The way it does that is it slightly perturbs the input features and makes a note of the effect of those perturbations on the output of the model. So if I change feature one by a small amount and it changes the output. I've now learned something about the relationship between feature one and the output. And that's a kind of an explanation. We used it to build a proof of concept prototype of a customer churn model. So customer churn is the rate at which you lose customers. And it's great to be able to identify customers you're at risk of losing. But with a tool like Lime, you can go further. You can say, not only am I going to lose this customer, but here are the things about that customer that the model finds concerning, which raises the possibility of not just understanding what's going to happen in the future, but changing the future, which really is a machine learning superpower.

Thanks, Mike, for that great introduction to machine learning interpretability. Listeners will include more resources in the show Notes. Time to get straight back into our chat with Emily Robinson. So when people hear the word experiment, a lot of the time they'll think of beakers and pipettes and lab coats. What are online experiments and how are they run?

So online experimentation also goes by the name A B testing. And generally what it means is that you randomly assign visitors to your website to one of two or maybe three or four experiences. So for example, if you have a product person that thinks a buy button in red rather than white will make people buy more, you can test that with experimentation. So you randomly assign people to see the red or the white button, and because of that random assignment, on average the only difference should be the button. So you can compare buying rates. And this sounds pretty simple in theory, but it's actually pretty complicated. And I gave a whole talk about it at a meetup in New York, which you can watch online. And the other thing is it's also newer, so this has only been possible in the last 10, maybe 20 years. And so you have a lot of papers now coming out about best practices around online experimentation. And finally, the reason we do this is people might think, okay, why don't just watch conversion rate over the past week, launch your new feature and then watch conversion rate again. And the idea being here is be a flat line and then it spikes up and then it's another flat line and great, your new feature is better. Or maybe it just stays flat. Oh, your new feature failed. But conversion rate are the other metrics you're interested in. Conversion rate being like what percentage of visitors buy. Or maybe your interest is in clicks. Basically these are very noisy measures that change because of holidays or bugs or time of day or who knows what. And so you're really not able to look at a time series to tell the difference because generally the change you're going to make is pretty small. It's not changing click through rate from 60 to 70%, it's changing it from 60 to 61% which is well within the normal variance. And so online experimentation really gives us a way to confidently say okay, your change made a difference or didn't make a difference.

Are there any challenges with the random assignment part of this experiment?

So we have our own in house system that takes care of it for us. What we do is you basically you have a name for your experiment and you hash that with a cookie or a device id. Generally that works pretty well. But where that can run into problems is if you clear your cookies you'll get a new assignment, but maybe you're the same person or cross device. So I am browsing on my, you know, Etsy app and then I go to my desktop. I may be in, you know, version A in the app and version B on my desktop because those have two different cookies and device IDs that are randomly assigned even though I'm one person. And you know, I think maybe at the, I don't know much on the engineering side, but I think it's again, it's a relatively straightforward system to implement. We have it running smoothly but we're really something to remember always is we're randomly assigning kind of browsers or app versions and not people.

So I love the example of red and white buttons for explanatory purposes. Are there examples that experiments that you found more exciting that you could speak to or speak about? I don't want to probe too much into company strategy either.

No, that's fine. And the nice thing is we can generally. Well, I can't. I can sort of share which experiments were a success and which ones were running. And one reason I can do that is well one, anyone could discover it themselves when an experiment's running, like say you open a bunch of incognito windows but especially who discovers it for us is our sellers. So we're always surprised. So you know, we change a ranking algorithm or we change something for people in Canada for 10% of the population and this Canadian seller will find out and will post on our forums about it. So we've also now started proactively communicating what experiments are going on so that our sellers can understand more why their stats or sales might be changing and why we're running these different tests. So one interesting one which was we ran a test on the bottom of our search page to add a module that had your recently viewed items. So if you'd have seen any items you will then get like this populated with, you know, if you've seen more than six with six items and then a see more. And so I was, I was fairly confident before we started this. So we didn't know how many people reached the bottom of the page because we didn't have anything that fired when that happened. And we can kind of estimate it by okay, if you click to the next page we guess, you know, you saw the bottom of the page, but there might be a lot of people who dropped off without clicking next. So that's probably an underestimation. So I didn't think it was going to be that impactful. I was also like, oh how, I don't know, how helpful is it going to be to see the items you've already seen once again? But actually that was a very successful experiment. What we did to implement there is we did start to have modules and things firing when people reach the bottom of the page. That's something we've iterated on and now we try different content. So we're looking at having the holiday gift guide there or having an RPIX 4U or having multiple modules not just recently viewed. So that was a really interesting one to work on.

That's really cool. So something else before we move on. You mentioned that a B testing online experimentation can sound simple in theory, but it's actually quite complicated. So there are all types of challenges with respect to running online experiments and a B testing in general. You can hack your P values. There are all types of different techniques people use, whether it be Bayesian A B testing or there are multi arm based bandit reinforcement learning solutions for this type of stuff. There's traditional frequentist statistical hypothesis testing. What do you do and what do you enjoy doing and where do you see this field going? I know I just threw a whole bunch of questions at you, but let's riff on that for a second.

Sure. So we use sort of standard frequentist statistics and part of why I thought this would be easy is as I mentioned, Etsy has their own in house system and it does things like it calculates the metrics for us so I can log on every day when our new data has come in and I can see how many people are in the experiment, what percentage bought in A versus B. It does the statistical test, it tells you the P value. So okay, great, that sounds pretty straightforward. We're not doing multi arm bandit or something like that. But even within that, one thing you mentioned is peaking. And so what's been exciting to me to work on is how can I help educate people and understand how to better run experiments. And one thing that complicates this a bit is that people usually want their experiments to work, of course, you know, both because they think it's going to work since that's why they bothered to try this new thing, but also because then it credits their team with, hey, we got X amount of extra money for Etsy with our experiment because we increased conversion rate. One thing that's been challenging is for people without stats education, understanding why peaking is bad and what that does. One thing we've recently implemented was a system around this where we worked together with the engineering team who does the experimentation system to come up with a couple different ways of displaying data. So essentially, if you have 80% power for a 1% change in your metric, which is generally the standard we put for how long we wanted an experiment, we will either display no change if the P value is high or x percent increase or decrease. But what we've also done is if you're not powered yet, we say waiting for data, but if you have, say, you know, we power is dependent on the, you know, the percent change. So it may take 10 days to run for a 1% change, but if you actually have a 5% change, it's only going to take three days. So how do you help people, you know, discover that massive change without also having a peaking issue of if you check every day, if P dip below, you know, 0.05, that's going to happen much more often than 5% of the time.

Just for those hypothesis testing newbies out there, and correct me if I'm wrong, the idea of peaking is if you do check your experiment, check the results every minute or every five minutes, you will occasionally just randomly see that white button is doing a lot, performing a lot better than the red just due to random fluctuations.

Exactly. And so my brother David Robinson actually did a post on this specifically about does Bayesian stop it. But he started out with saying, okay, what happens if we run an experiment 20 days? For 20 days, there's no difference. You just simulate this null experiment, but you check every day and you stop if p dips below 0.05. And what he shows is that this doesn't happen 5% of the time, which is sort of promised that, okay, sometimes you're going to get what's called a false positive. It happens over 20% of the time. And so that's what you have to guard against is this increased false positive rate if you're checking all the time. And so what we've done is both education, but we now also, if your p dips below 0.01, we actually do display the change, even if you're not powered for 1% change. And we're working to educate people. Okay, how do I know again, is this peaking or is this a real change? So we talk about things like the confidence interval. So one thing we've started doing is visualizing those. So we'll have a little line which has a Y axis at zero, and we'll show in a little rectangle the confidence interval. And so people can start to grasp, okay, if it's really. If it's a really big rectangle that just barely doesn't overlap with zero, you know, maybe not be too confident there, but if it's a tiny little rectangle that's, you know, way far from zero, okay, we probably have a real change going on there.

That's awesome. Because what that's doing is visualizing and quantifying uncertainty, which is part of our jobs as data scientists, statisticians and data analysts.

Exactly. And so that's something that we have to do is communicate that uncertainty. Because often people will just want an answer. They'll get like, is this change in a metric real? And I'll sort of say, statistics can't do that. We can't ever say something is real. That's where we have to take into context. But we can have evidence for this is more or less likely. And so we've really been working on figuring out ways to help other people, like the product teams we work with, understand that.

So I've got one more question about experiments that really just came to mind. I don't even know if it's well formed, but I'm going to try. Try it, try it anyway. In scientific research, in laboratory research, there are governmental organizations that make sure that the ethics of experiments are in line with certain, certain regulations. Is there the possibility in the future that this will happen to businesses? I don't know whether in your case, because what we're discussing is experiments with respect to ux, for example, or search results. Maybe there is in that case. But if we're talking about Facebook, for example, choosing to show us different types of media or whatever it is as a media outlet. Do you think so? Okay. The question is, if we're running experiments and the general populace are the lab rats, do we need to consider the ethics of it?

Yes, and I think we do. And one thing you mentioned Facebook. So there was a paper Released I think some years ago. But basically they tried out, okay, if we change the sentiment of they choose in the newsfeed, obviously usually most people have enough friends that you could populate this with. You can't show everything or at least you can't show obviously everything in the first 10 slots. So what do you show? And I believe they experimented on changing the sentiment of the statuses that were shown on your newsfeed and found that that seemed to influence people's viewers emotions and there was some controversy there about okay, what is the ethics? Because you know, they were showing, it wasn't just like, okay, show some people happier stuff. They were showing people more sad or more depressing things. And they also, you know, there was another paper that talked about people who were writing statuses and didn't post it. And looking at, looking at that and then people found that controversial because, you know, I didn't know like I have my like random thoughts and I never even posted it. But wait, Facebook has been recording this? It turns out that I believe. No, they just sort of record whether or not you typed anything and how long it was. Not the words, but still. And I think, you know, part of this is how it's done in medical and social science research is, you know, how strictly it's regulated and what it has to go through depends. You basically fill out a form that says, you know, could this cause harm? Could, you know, does this, is this with vulnerable populations like children or a prison population? And depending on your answers of what population you're affecting and how, how much, how invasive or your intervention is, then that will be subject to different rules. And I think it's an interesting question. I think that the idea is how, you know, part of how you can regulate that is people doing this research is in an academic institution and the institutions does it themselves because they have motivation, you know, financial motivation and maybe losing a license or whatnot. But how would you do this with private companies? How would you make sure people aren't running rogue experiments? But I think something a little similar that's coming down the pipe is this general data protection regulation in Europe which has things like the right to be forgotten, the right to understand what data a company has about you. And this is coming into force soon and it's really going to affect people because often how you run experiments is based on the data you have about people. So it's going to change the way we can work if you can't keep certain data or you have to delete data. So what if you have attrition from your experiment population, et cetera. So I do think it's definitely important to think about the ethics, and I'm just wondering how that could work in practice for regulating the experiments at companies like Facebook or Google that can really impact people's lives and health.

And as you say, incentives are slightly misaligned in the sense that maximum transparency ethically and for the civilian US Civilians and citizens would seem an ideal place to go to. But we're talking about businesses who need to keep a lot of what they do proprietary in order to, you know, they. They need to run their business as well.

Yeah. And the other idea here is that, you know, one thing that's done in academic research is these risks. You also talk about the benefits. Right. So medical trial can be quite risky. You know, people can sometimes die or get more sick. But the idea is, within some bounds, it's worth it if maybe the benefit is getting this drug that can save thousands of lives. And so I think the other thought is how, you know, in terms of businesses defending it, you could say something like, well, we made the sad statuses more extreme so that we can learn. And then we learn that that can be harmful. And so now we know to pull it back more than we're doing. Or, you know, Google might experiment on, I don't know, something, what happens when you type in how to commit suicide? That's a huge thing there of seeing. Okay, maybe if you put a suicide hotline at the top there, or resources versus, and not, for example, I don't know, articles about how to do this, that could be hugely beneficial. But the question is, how do you know that's effective without an experiment? Because maybe could it be more harmful to just launch it and just roll this out without having any way to measure. Okay, did this have the beneficial impact we hoped it did?

Yeah. That's super interesting. So you mentioned these ideas of data protection, and I'm wondering, what type of data do online businesses have access to that allow them to make substantive business decisions?

Sure. So I would say, so some businesses track and have access to kind of all that you do, as long as you keep the same cookie or you're logged into the same device. So that's how you find these. These ads that follow you around the Internet. Right. You look at ugg boots once, and there you go, you have ugg boots, you know, popping up in the ads in your news articles for the next couple weeks. So at Etsy, though, we focus. We sort of.

I just want to say I'm actually currently getting a lot of ads for Data Camp.

Yeah. Yeah, right, I see.

I need to tell them.

Yeah, you need. Yeah, it's. It's. Yeah, I. Sometimes you'll find, I think, yeah, someone posted like Hadley Wickham, following him around the Internet and a little Data Camp course advert.

Brilliant.

Yeah. So at Etsy though, we just generally focus on. I'm not even sure we have. Certainly on my team we don't use it. I don't think we really have or are interested in data of what people do off site. So we do have. But we are interested in things like marketing team is very interested in. Okay, where do people come from? How many people click our Pinterest or Facebook advertisements and we're interested in long term effects. So where it can happen. Okay, someone who saw this, this module a couple weeks ago, you know, this RPIX view. Okay, maybe they didn't click and buy it right away, but maybe they bought it two weeks later. And so we want to keep tracking that online behavior on the website. So that's really the data that we use. And that's partly because we don't on site, we don't have any off site advertisements. So the promoted listings I talk about, the only ads we have are items from our sellers. And sometimes you'll find, you know, okay, the sock is promoted listing, but also it's servicing organically in our organic rankings two rows later. So we don't really have to concern ourselves with anything with, okay, what are our most effectiveness outside ads that we can do.

Now it's time for a segment called Stack Overflow Diaries with datacamp curriculum lead Cara Wu. Hey, Kara.

Hey, Hugo. I've got a Python question for you today. A user wants to know how to rename columns in a Pandas data frame. In their specific case, the columns are named $A, $B, $C et cetera and they want to rename them A, B.

And C. That's a great question. It's something I do every day due to the plight of CSV files, among many other reasons. There are several ways to do this though, right?

Yes. The first, which is the accepted answer, is to create a list of the new names and assign it to the columns attribute of the data frame. This works if you know exactly what column names you want and what order they should appear in. A second option in a similar vein is to use the Pandas dataframe method setaxis, which can also take a list of new column names and overwrite the old names. A third option is to Use the Rename method for Pandas dataframes. You can call Rename with a dictionary that maps the old column names as keys to the new column names as values. You can also call Rename with a function that will convert the column names into the form you want. So, for the example in this question, since all of the column names are in the same format and just need the first character removed, you could call Rename with a function that drops the first character of each column name, or one that looks for the dollar sign symbol and replaces it with an empty string.

So in the answers, we do find several ways to rename columns in a data frame, right?

Yes, and I really like this question in part because it shows how many different approaches one can take to solving programming problems. But more than that, I also really like the specific answers here that represent different levels of complexity, but also different levels of robustness to different situations.

So can you unpack that slightly for us, Kara?

What I mean by that is the first answer where you're creating a list of new column names and assigning it to the columns attribute works just fine when you have a small number of columns and you know exactly what they are. But if you had to rename 100 columns, it'd be a huge pain. On the other hand, writing a function to convert the column names into the desired format may seem like more upfront effort, but it would save a lot of time in situations where you had many columns or where you don't know the exact column names in advance.

Thanks, Cara. Those are all super useful ways to rename columns, depending on your use case. See you next time.

You bet, Hugo.

After that interlude, it's time to jump back into our chat with Emily. So we spoke a bit about data ethics. I was wondering if there are any other challenges moving forward into the future of data science and big data that. That you're interested in thinking about.

Yeah, so I think, I think this is. Comes into force in a couple ways. So there's some. Sometimes it's with people's individual projects. So for example, people have scraped dating profiles and done things about that and even released the data without considering the impact that that could have. Then you also have things on a company level. So with the diesel cheating scandal at Volkswagen, an engineer actually went to jail over that. And so the idea here being you can get in trouble for something you're asked to do and you need to push back. But that's a very clear case of this is illegal. And most likely the engineer knew that when he was being Asked it or she was being asked it. But there's many other cases that is talked about in this book I really like by Kathy o' Neill called Weapons of Mass Destruction.

That is a great name.

Yeah, yeah. WMDs. And what these are, are algorithms that are she classifies as destructive, important, so affecting some critical outcome and secretive or unaccountable. And so one thing she talks about is teaching ratings. So this idea of, you know, these ratings of like, okay, how much impact did this teacher have? How good is this teacher? And one thing that people try to do with that is basically, okay, you predict, okay, these students in second grade. I predict in third grade based on their second grade scores. You know, they'll get these scores. And then you look at it in third grade and you say, actually they scored a lot better. So this teacher must be good. But there are some problems with this that she talks about in her book where people's scores can fluctuate, you know, year to year, even though they don't, you know, these are experienced teachers who aren't changing anything. People get fired over this and can't hold anyone accountable. Not even that. The school administrator just says, I don't know. This is the number that we got, you know, or people, this can be in recidivism. So predicting whether someone's going to commit a crime again and predicting this at the parole hearing and not understanding maybe, okay, it's taking into account zip code, which is correlated with race, which is what it's really using. So it's really using and saying, okay, it's really using race as an input and oftentimes saying, okay, if you're African American, you're going to be more likely to commit a crime. So we're not going to parole you. When of course, you're not actually allowed to consider race directly, but it can come into these algorithms in these secretive ways.

And in that particular example, there are a lot of things at play. But what we have one thing happening there is societal biases and human biases being actually fed into the algorithm and creating that type of algorithmic bias as well.

Exactly. So sometimes people can say this, well, if it improves my prediction, like, why? Why can't I use it? And one problem there, of course, is, okay, why would. Why would African Americans get arrested more? And the naive answer is, maybe they commit more crimes, but actually it's that they're policed more. So the neighborhoods, maybe they go back to has a higher police presence or they're arrested for crimes that they wouldn't arrest. White people for like, maybe, you know, carrying a small amount of marijuana. And so those societal biases get magnified in these algorithms and again, sometimes not knowingly. And so you can hide behind the algorithm and say, it's just the numbers, we're not racist. When really that's what is happening there. And that's why you have to be very careful.

A lot of people outside data science may not appreciate how much of a social sport it actually is. What role does community play in data science for you?

So community to me is a huge component of it. So the, as I mentioned, I took Metis and I learned Python there. I'd already knew R in undergrad. I was very fortunate to go to Rice University when Hadley Wickham was teaching and designed some of the courses, the statistics courses. So that was. I had Hadley in my introductory stats class and then I took other classes from other teachers, but that had been designed by him and were taught by his grad students. So that was a really great foundation. And the primary reason Etsy lets you work in R or Python as an analyst, you know, whatever you're more effective in. And the primary reason I usually work in R rather than Python is the R community. So it's very friendly and welcoming. It's also a bit smaller than the Python community, as Python is of course used. You know, part of the advantages is it's used for many things by many different people, whereas R is usually a little more academic and statistics rather than, say, general programmers. And R also is things like the R Ladies group. So I'm on the New York City board for that. And this is a group basically with chapters in more than 40 cities where there's like a bit of a global organization. And the idea is to promote and welcome women into the R community by hosting meetups, which can be talks or tutorials or just casual get togethers, or we even have a book club in New York that we host every couple months where we come together to this lovely bookstore and we've read a book about data, and one of those was Weapons of Mass Destruction. And so we all get together and talk about that. So I've really enjoyed getting to be a part of the R community, which also is thriving on Twitter. And I highly recommend you check it out. I mean, people there are very accessible. You can ask a question with the RSTATS hashtag, and if it's a question about, say, dplyr or these tidyverse packages, Hadley himself will answer. And for those listeners who maybe aren't in R, Hadley Wickham is someone who's designed many packages for the R community. There was an article written about him called the man who Revolutionized R and is just an enormous presence and contributor, but also so friendly to, you know, newcomers and people he doesn't know and will often jump in and talk to, you know, people who are just asking the world this rstats question. If he can answer it, he'll answer it back on Twitter.

We've discussed about the sense of community in data science and in the R community. Can you speak to the challenges concerning diversity in data science?

Sure. So I think that what's interesting here is that statistics is almost 50, 50 men and women, but data science isn't. And a concern of mine is that there's a growing gap between, as I discussed earlier, these type A so stats and analysis data scientists and the type B machine learning. Machine learning is much more male dominated and it's also becoming more prestigious and better paid than this more type A work. And something that's interesting is the history of programming. You see, this happened in front end development. Basically when front end development started becoming a thing rather than say full stack, more women went there and it became less prestigious and lower pay. And historically in occupation, you see that when women enter an occupation, wages go down. So it does, you know, and also my own interests lie in Taipei and I'm just a little concerned, are we going to start even though the data science community in certain parts, like, you know, in R, there's a meetup, not a meetup, a conference that I'm talking at in Austin called Day to Day Texas that has an R track and it's I think almost right around 50% women, which is awesome. But you look at something like a machine learning heavy conference and sometimes you see all male panels or nothing like that. And so that is a little concerning to me and trying to figure out, okay, how can we keep diversity in all different parts of data science. And also I think I often focus on gender diversity, but making sure to expand that and think about, you know, racial diversity and other types, you know, background diversity. So not just people who have PhDs or master's degrees or bachelor's degrees from prestigious institutions, but maybe some adults coming from other fields, age diversity, not just people in their 20s or 30s, but people in their 40s are people with kids. So I do think that's something that's always going to take work. But I'm also really excited to see how many really good allies we have in the community and how interested people are in Working on this. So, for example, Hadley Wickham made a commitment, so he speaks a lot, and he committed in 2018 to only speak. The only meetups he's going to speak at is our ladies meetups, which is just, I think, a great commitment. So he's still doing conferences, but these are the meetups he's doing, which I think is really exciting.

What type of steps as a community can we take to increase diversity and be more welcoming?

Sure. So I think one thing is definitely thinking about amplifying. So this is something I'm trying to work on. So if you are probably most of us are an ally in some way, as in, like, maybe as a white woman, I can be an ally to women of color. And thinking about, okay, how can I amplify, you know, as my reach grows and my reputation grows, how can I work to showcase people who might not, you know, be having the opportunities they. They should be because they're. They're assumed not to be as technical or, you know, they have these other challenges. So I think that's one thing you can. You should. You can definitely focus, focus on. And I think the other thing is I really like to a quote I heard, which there can definitely be some pushback into this idea of privilege being like, hey, you know, if you're someone that you can be said, oh, you know, you're privileged, you can take that as, what do you mean? Like, I've worked hard, you know, I haven't had it easy. And perhaps you're not privileged in some way.

Right.

Maybe you're a gay white man and you've had challenges because of that. And I think the idea is it's not meant you shouldn't. Feeling guilty or shame is not going to help anyone, but use your privilege. So if you maybe with income donate to groups that hack, that like black girls code or these other things that promote women of color. Going into tech, there's this great group I just found about Hack the Hood, where people in minority communities learn tech skills by working with small businesses and helping design websites for these small businesses. So whether it's through money or time and mentoring, doing tutorials, working on the hiring at your company, trying to ensure that, hey, are we only hiring white men? Because don't just blame the pipeline, okay? Maybe most of your applicants are white men. So what that means is not to throw up your hands, but think about, where are we advertising is the problem that we're getting most of our applicants through our network and our networks are mostly white men or, you know, we're not taking advantage of these women in tech communities where we can post jobs. Or is it that, you know, we're not showing in our job description that will be a welcoming environment. And we're using terms like rockstar or, you know, we have, you know, we work 80 hours a week that are a turn off for people who, you know, aren't young. Mostly, you know, males who are in their 20s who can just dedicate their whole lives to their jobs, even though that's probably not necessary.

We've discussed a lot of interesting techniques and methodologies in data science, data analysis, statistics. What are some of the techniques or one of the techniques that you enjoy using the most?

I really enjoy the tidyverse, which we've talked about. Something interesting here is that Jenny Bryan, who is working at RStudio on leave from UBC as a professor, she had a really great interview recently where she closed with in machine learning. It seems like everyone wants to make a contribution there. And I'm like, you go for it. I'm going to be over here getting data out of Excel spreadsheets.

That's awesome.

And what I mean, it may not be what's quote unquote hot right now, but it's important to do these things that help improve your data analysis workflows and thinking and tooling. And what I love about the tidyverse is it just makes these everyday data analysis tasks easier. And so, yeah, it's not as cool maybe as saying, like, oh, I'm doing a deep convolutional neural network, but 90% of the time that may not actually be what's going to be useful for the, for the person you're helping make a business decision or make a product. So I think, you know, that's, honestly what's very exciting to me is just how do we empower people to make better decisions to have a better data analysis workflow.

That's awesome. So with all that having been said, do you have a final call of action to our listeners?

Yeah, and I think this goes back to what we talked about community. So I would say is, you know, try to contribute that to that community. So if you're a more advanced or experienced, be kind to beginners. You know, don't, don't, don't shame people who, you know, use a for loop when you think they should be using apply functions. You can definitely share with them. Hey, you know, this, this might be a cool thing for you to learn. Did you know you could do it like this? But it's not productive to just be like, oh you're bad, this is terrible and try to give back to the community and connect. I think data science is much more fun when you have peers and mentors and friends. I've met so many people through the R Ladies Group in New York, through Jared Lander's New York Open Statistical Programming Meetup that happens once once a month through Twitter and you know, and you can give back and contribute to this whether it's through writing a package or sharing packages or writing blog post tutorials. So you know, find, find your community and try to try to contribute to it if you can.

Emily, it's been such a pleasure talking to you and thank you so much for coming on the show.

Thank you. This has been great.

Thanks for joining Our conversation with Emily about data Science and online experimentation at Etsy we saw the challenges faced in online experimentation at Etsy and the types of questions data science can solve for the business, from ranking algorithms to UI changes and the introduction of new features. We also dove into the role of community and the current diversity challenge faced by the field, along with concrete ways to deal with it, such as amplifying and being aware of our own personal and societal biases. Make sure to check out our next episode, a conversation with Roger Pennsylvania, professor in the Department of Biostatistics at Johns Hopkins, Co Director of the Johns Hopkins Data Science Lab and the Coursera Data Science Specialization, and well seasoned podcaster on not so Standard Deviations and the Effort Report. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com community podcast.

Sam.