#11 Data Science at BuzzFeed and the Digital Media Landscape

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey.

On datacamp.com in this episode of Data Framed, a DataCamp podcast, I'll be speaking with Adam Kelleher, principal Data Scientist at BuzzFeed and Adjunct Assistant professor at Columbia University. Today, we'll be talking about how data science is impacting decision making at buzzfeed and the modern digital media landscape in general. I'm Hugo Bown Anderson, a Data Scientist at DataCamp, and this is Dataframe. Welcome to DataFrame, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter hugobown and datacamp@datacamp. You can find all our episodes and show notes@datacamp.com Community Podcast hi Adam, and welcome to Data Framed.

Hey Hugo, thanks for having me.

It's such a pleasure to have you on the show. So we're here today to talk about the impact of data science on the modern digital landscape and to talk about causality in data science. But before getting into these, these, these topics, I want to find out a bit about you and your work as a data scientist. So I think a lot of people wonder exactly what, what data scientists do. I'd like to first know what your colleagues would say that you do.

Oh, I guess they'd say I daydream about complicated math that probably won't be super useful. But really, I'd say they'd say I actively evangelize causal inference. It's something that I've been working on for a while. I've seen a lot of kind of funniness in data science methods. Not just here, but kind of everywhere, kind of all over in data science blogs, for example, or just discussions about data science. So I can get pretty annoying about experiment methodology, about good kind of analysis methods. And so that's. Yeah, I guess that's. That's probably what they'd say.

Cool. So we're going to dive into a lot of that later about causal inference, about experimental methodology in online businesses. So we've just heard what your colleagues will say that you do. What do you actually do?

Recently it's, it's really been mostly engineering work actually. I've been building some tools for doing large scale machine learning. Not just large data sets, but also large numbers of variables. And I've been working on a Python package called Causality. So you can pip install Causality if you want to play with it. It makes causal inference just really, really easy. I think that's an important thing to spread the use of it. So there's basically no math background required, very little machine learning background and you can work with it. So really just those two things. My kind of key machine learning projects and working on causal inference tools.

That's cool. And I actually remember the first time we spoke, I told you I love the fact that in a Python environment you can import causality into your network.

Yes, I agree. It's like import gravity.

Yeah, exactly. But you also teach, right?

I do, right, yes. I'm teaching at Columbia. I'll be teaching causal inference, applied causal inference for data science this coming semester in the spring. I just finished teaching machine learning products this fall about productionizing machine learning algorithms.

And could you just give us the elevator pitch on what that means productionizing machine learning algorithms?

Oh, yeah, sure. So when you're prototyping something, you're usually working maybe on your laptop or on a local desktop machine, building a machine learning algorithm, probably in Jupyter, working with data that's local to your computer. And in order to put it into use on a website or in an application, you need to put it up on a service somewhere, maybe in the cloud, maybe on your own hardware. And there's just very different methods, it's a very different workflow, very different tools that you would use. So it's really taking your prototype algorithm from your local machine and putting it up in a production environment.

That's pretty cool. I mean that'll provide a lot of value to the students in the sense that, you know, you learn, you can learn machine learning, that type of stuff in at college, but actually learning how to do it on a daily basis in practice in industry is what you're really talking about, right?

Yeah, yeah. I designed the course to fill in the gaps. I really look at where the gaps are and that's what drove my teaching causal inference in particular and also the machine learning products course.

So one of the things I love, one of the many things I love about data science is that we all have a lot of different backgrounds and diverse skill sets and diverse areas of expertise. We're all coming together to try and solve real world problems. But what that means is that a lot of data scientists don't know how anybody else got into data science. So how did you get started in data science?

Yeah, it's, it's kind of roundabout, you know, it's something that I didn't know I wanted to do originally. And so I just kind of wanted to understand how the world works and how we might model it. So I decided to study physics because that was the best thing I knew of for modeling how the world works. Social science to me didn't seem hard enough. It seemed like something I could pick up kind of after the fact, which seems crazy of course, looking back on it, but that's what 16 year old me thought. So that's when I decided to study physics. So I graduated with a doctorate in physics, studying mathematical cosmology in particular. But while I was there I was studying graph theory in particular as it applies to politics and partisanship. So I was actually standing outside of a classroom one day, I saw a paper on the wall. It was about community structure and congressional co sponsorship networks. So it's asking the question, which congressman co sponsor bills with which other congressmen? And if they co sponsor together, you draw a connection between the two congresspeople. So if you ask the question, how can I arrange these congresspeople into groups such that there are the most connections within groups and the fewest between groups, you're asking the question of how do I optimize the modularity or how do organize this network into clusters that are tightly interconnected in a way that you would expect beyond chance. So when you do that, you can actually put a number on how modular the arrangement is. And that number is actually a direct measure of partisanship and co sponsoring. If you look at it, there's kind of a two side split right along party lines. And you can quantify how partisan things are. I thought that was the coolest thing in the world. So my brother and I scraped some data off of thomas.gov, this was kind of back before it was made more easily available. And we recreated this study, approached the professor who did the work originally, who happened to be at unc. And we just started working with understanding community structure on the blogosphere, the partisan blogosphere. And that led me into more of a data science background. And when I was graduating I had the choice either continue going into gravitational physics, which looked pretty, pretty promising actually. The decadal priorities for US physics research were being set at the time, or that is toward gravitational physics in particular with LIGO and the gravity wave observatories, or go into data science. My brother was working at BuzzFeed, which was a very small company back then. It was about a fifth the current size. So it was a bit of a risk to take that path, but it seemed promising enough. And they were implementing the pound tracking technology which my brother invented. And I was a data scientist on because that seemed like a particularly interesting project coming with a graph analysis background. So that was the choice I was faced with. Clearly the choice was buzzfeed. The graph analysis problem that I had coming in was just too interesting to turn down what a trajectory.

And the graph analysis challenges you're describing, I mean you did them or you worked on them with respect to studying, researching political partisanship. But I presume you these types of techniques can answer all types of social network problems, whether we're trying to analyze echo chambers in social networks or even looking at network effects in studying epidemiology, for example.

Sure. Yeah. And so that was kind of my interest in it at BuzzFeed. More toward the epidemiology side because if you ask what makes content go viral, what do you mean exactly by virality? So there's this notion in epidemiology of it's like a three state model where there a susceptible individual, an infectious individual and then a recovered individual. And so I kind of thought to apply that type of analysis toward our content at BuzzFeed where we could kind of understand virality as kind of this diffusion problem. So a lot of people have approached the problem that way. It turns out not to work super well in practice. There are more practical methods. But that was kind of my first thinking, attacking the problem coming in. That was pretty fun.

That's incredibly interesting. So what are some examples of the use of data science at buzzfeed that you've found interesting?

Sure. So let's see. I guess the kind of key one that I think the company was really built on is this feedback cycle where you produce content. That's the first step. You see how the content performed by looking at the data that you produced around it. That is the performance metrics, how many page views did it get, how many shares did it get, and so on like that. That informs the content production process and says, you know, this might be working. This article about this particular celebrity might be working, for example, or this one might not be working. And then the author can hypothesize about why they think it did or it didn't work. They can produce a new piece of content to test their hypothesis and then repeat the process. They look at the metrics again and so on it actually, if you look at it kind of step back and look at this collection of authors writing all of this content and performing this process and learning not just from their own experimentations, but from each other's. It looks a lot like an evolutionary algorithm where you create a landscape of organisms which would be the content. You have a fitness function which is your set of KPIs for how the content performed. And then you have a mutational process where the people look at how each of the content performed, take what they think is working, what they don't think is working, create new content and see how that performs.

Has any of the techniques or methodologies that are used in analyzing this evolutionary data been useful in your work at BuzzFeed?

I haven't used it in particular with fitness functions, for example, or trying to simulate the evolutionary process. The data really doesn't lend itself well to that. It's just so hard to quantify what makes a particular piece of content perform well. I think there's some promise in that area, maybe from representation learning, where we can figure out what aspects of a piece of content drive its performance. But we've always just used people for the fitness function. They see what makes this piece of content unique and they come up with ideas, maybe from reading the comments on the article or talking to their friends about it or their colleagues about what's causing its performance, and then they just keep iterating like that.

So you've described this content feedback cycle. Do you also run experiments?

We do. So that's kind of the bread and butter of a lot of what the tech team does. So we'll make a product change. We'll want to see how the product change changes content performance or changes site performance. And then so we'll just run an experiment. Some people will be exposed to the product change, others won't. It's pretty cool actually. It's at such a large scale, there's so many other variables we can measure alongside of just the metrics that we're interested in. And we're actually able to automate the methodology. So the experiments are so repeatable that it really opens up a lot of opportunities relative to experiments. You might see in the social sciences, for example, where you generally have a bit smaller scale data. Except of course, nowadays where we're doing more large scale computational social science. So this is a kind of cool frontier in science and I think of a B testing kind of as a social science experiment because we really are experimenting with lots of hundreds of thousands of individuals in an experiment and we can actually measure pretty small causal effects. It's pretty cool.

And what type of questions do you ask in these experiments?

They can generally be pretty simple. They could be, for example, what's the effect of this font change on people's engagement with, with a link that we're increasing or decreasing the font size as of. Or they could be big site changes, for example, adding new modules on the site or changing the way the site's navigated and seeing how that changes engagement metrics, for example.

And how about use of language in let's say article titles or something like that?

Yeah, that's a very interesting one. We have an algorithm that runs on the site to do the experimenting for us. It's not actually an AB test, it's a multi arm bandit. So the multi arm bandit will measure how each of the titles is performing and show each of the titles more or less often, depending on its performance. So it does that to try to find a winner. So the process generally looks like an author is going to publish an article. So they'll write several titles for the article and then publish it. At publishing time they start showing impressions for the article. So it shows each of the different titles and then as one becomes a winner, that one gets shown more and more often until it's declared the winner, at which point the experiment or kind of pseudo experiment is terminated and that article continues to retain that title.

That was a great explanation of description of the multi arm bandit technique. And in fact, for those of you who haven't heard multi arm bandit before and have heard the word reinforcement learning floating around, this is the first example of reinforcement learning that you would generally see because the ones that are successful are reinforced in some sense.

Right? Yep. So we, we try those ones more often. Yep. Yeah. So it's a pretty, it's a pretty cool problem setting. And it's, it's actually a bit different from a B testing in the sense that you don't necessarily get statistical significance as quickly as you might with an A B test. Instead you're minimizing the, the regret that is the, in this case the clicks that you're missing by showing the wrong variant. You could get statistical significance faster by showing a poorly performing variant more often. So you learn what its bad performance is more quickly or. But you really don't care about getting statistics on the bad performers, you really just want to pick the winner.

You mentioned that we're talking about data collection and evaluation at a very large scale. What type of scale are we talking about?

I don't have our numbers on what our active users are on the site, but I could look up those and see if I can get them to you afterwards.

Yeah, sure. What about order of magnitude?

Oh, sure, order of magnitude. So one of these experiments might have. If we run it, I guess some of the smaller ones I was running had 50,000 or so. Some of the larger ones could be millions. So it's really a pretty big scale, especially compared to older social science experiments.

And you also mentioned that you're automating a lot of these methodologies these days. Can you tell us a bit more about that?

Sure. So one thing that you might worry about, for example, is just computing error bars on your effect estimates. So you can do that in kind of a simple way just making what's called a normal approximation, where you say, okay, the statistic I'm calculating should take on a normal distribution for large numbers of samples because it's just an average. So my error bars on that should be just the kind of Gaussian standard error bars. And so those are some error bars you might calculate for it. But it might turn out that that's not actually the best procedure to use because your samples aren't independent from each other. And so you want to make a methodology improvement to that. Now, one person might realize this and they might improve it for their own experiments, but if you have a framework for doing a B testing, then you can implement that change into the frameworks. And so now it's not just a single individual doing the analysis on their own. And that best practice not being disseminated well, but now it's the entire team is using the same framework, and you can accumulate these best practices into the experiments analysis.

We'll jump right back into our interview with Adam after a short segment. Let's now jump into a segment called Blog Post of the Week with datacamp curriculum lead Spencer Boucher. What you got for us today, Spencer?

My pick this week isn't actually a blog post, Hugo, but instead it's a paper by Leo Breiman in the journal Statistical Science back in 2001. A lot of hubbub gets made over the advantages and disadvantages of statistics versus machine learning. But this paper, which is called Statistical the Two Cultures, it really laid out the divide early on in an accessible way. Even if you've never heard of Breiman there's a pretty good chance that you've used some of his stuff. He's the guy who pioneered the random forest algorithm, and he actually has the trademark on the name itself.

So what's the paper about?

Breiman lays out the difference between what he saw as the two statistical modeling, data modeling and algorithmic modeling. The primary goal of data modeling culture is to use your data to figure out a model that explains how nature gets from a set of inputs to a particular output. These days, statistical modeling is usually what people will say when they're talking about what Breiman calls data modeling. From this perspective, the key is to identify a model that most accurately reflects the way that nature actually works.

Okay, so what's the other modeling culture?

Right. So on the other hand, you've got the algorithmic modeling culture, which is now usually just referred to as machine learning these days. It still tries to find a function that maps input variables to an output variable, but mainly attempts to mimic nature's output as accurately as possible. It doesn't concern itself with reflecting the way that nature actually works, necessarily.

So would I be right in saying that the difference really boils down to looking for mechanisms and making predictions?

Yep, that's pretty much spot on.

So the former is incredibly important in, for example, basic science, but perhaps the latter more important to much of industry, although this is certainly debatable. So which side does the author fall on?

Well, so it shouldn't come as much of a surprise that Bryman identifies very heavily with the algorithmic or machine learning culture, seeing as his random forest algorithm is pretty much the poster child for that statistical culture. Breiman's got several beefs with traditional statistical modeling culture, actually. First of all, it often just doesn't perform as well as cutting edge algorithmic models can, which lock statisticians out of interesting prediction problems that they could otherwise be involved in. Second, sometimes it's actually pretty presumptuous to assume that a statistician can come up with a parametric model that really is analogous to whatever nature's doing. Like most things, however, there's clearly not a cut and dry right answer. And even Breiman recognizes that data modeling has its rightful place.

So how relevant is this discussion in today's data science landscape?

You could absolutely make the case that these days there's way more than just two cultures. On the data modeling side, for example, you've got the divide between Bayesians and Frequentists. The algorithmic modeling side can break down any number of ways as well. The so called five Tribes of Machine Learning, for example, which are laid out in the book. The master algorithm is one really well known example, and maybe that's a topic for another day. Your takeaway here shouldn't be that data science is divided or fragmented, though in reality, it's really exciting how deep and rich the world of statistical modeling has become in the last few decades. Your tool belt as a data scientist is only going to continue to get larger, and the problems that you'll be able to solve will only keep expanding.

Thanks, Spencer, for that thoughtful introduction to the many cultures of statistical modeling. After that interlude, it's time to jump back into our chat with Adam Kelleher. What are the major challenges that face the modern digital landscape, digital media landscape that you're working on and interested in?

Oh yeah, that's a cool question. It's not really what my main focus is, but I can talk a little bit about my role in that kind of domain. There's a ton of content out there to compete with, so you have to be good at producing and distributing the content. So a lot of it's just kind of reinforcing that feedback loop as well as we can. So better reporting, reporting better metrics, measuring things faster, measuring more variables that describe our content and correlating those with the metrics and so on, all driving this feedback process. On the other side of it, it's producing the content faster so we can do the feedback loop even quicker. And then distributing the content is tricky. You could just promote a post on Facebook for example, but are you promoting the right thing at the right time on the right page? For example, many publishers have lots and lots and lots of pages, so ourselves included. And so you want to promote the right thing on the right page at the right time. And so it's a fairly complex problem. That's an interesting one.

And so how can data science help to solve challenges like this?

Yeah, that's tricky. There are specific tools that are useful. So for the feedback process we want to get the numbers in front of the people who are going to use them. So for that you need to build dashboards. We have a couple of internal tools for that. One of them is very simple, it's called dashboard and that's probably the most frequent, frequently used one. I think probably because of its simplicity. If you make the numbers really easy to use, then non technical people can read them and make use out of them. The more complicated a tool is, the harder it can be for people to use it. So while you're getting them more information and you're hoping to improve the feedback cycle, you might actually be hurting yourself in the process by making fewer and fewer people engage with the reports. So I think that one's successful because it's so simple. We have other tools that we do use for more in depth reporting. Looker is a great tool that we use for generating dashboards for our internal data, so people who are more interested in the more in depth reports can use that.

And what type of data science techniques and methodologies do these tools employ?

A lot of it is really just basic averaging, ranking those basic types of statistics. You can get a lot of value just by looking at the data and just kind of coming up with hypotheses for why things are what they are. If you look at content performance, for example, it's a long tailed distribution in terms of most KPIs, so just page views, for example, there are a small number of articles who generate most of the page views. And so that fact is something that you can use to say, okay, what are these few things that are really performing really well and what about them do I think is performing really well? If all of them have something in common, then you have some notion of what's probably making them perform really well. And so you can test that idea by producing more things with that in common with those items. And you'll pretty often find that it does seem to be what's driving the performance. The iteration task is simple enough that if we go into some more fairly complex analysis on these types of things, we're probably not even going to get the statistics we need because the media landscape changes so quickly. What does well today probably won't do well tomorrow. So it's really just kind of about simple, quick iteration on basic statistics.

What other types of techniques are prevalent?

Other techniques include content tagging, for example. It is nice to be able to extract some of the attributes of a piece of content and then see how that might be influencing its performance. So you could do a little bit of that. There are tools out there and new APIs actually that are leveraging advances in machine learning recently, for example image tagging or video tagging. So those types of things have actually been pretty useful for feature extraction, even if the basic statistics approach kind of dominates the feedback loop.

And for these types of things, whether it be image tagging or text content tagging, is it generally deep learning that.

You guys use a lot of? It's actually manual. It's hard to train an algorithm kind of working at our scale to do a lot of the more complex types of analysis. So we do use some external APIs, like Clarify's API, for example, and Google's APIs for image recognition. And we use some of that to do some meta tagging. But we also have a lot of internal tagging that our editors will do when they're writing a piece of content and they want to test some idea of theirs over a slightly longer period of time when we'll be generating enough content to get statistics to test the idea.

So, Adam, you mentioned that something that's essential to to your work at BuzzFeed is efficiency and making things faster. So are you guys working on making faster experimentation processes?

Yeah, that's actually a bit of what I've been working on recently. So these tools for productionizing machine learning algorithms really are workflow tools for developing algorithms more quickly. For example, our test cycles can go on GPU machines instead of our local machines. So that's a big win because it makes the algorithm iteration process faster and it also makes it EAS to get things into production. So we can just actually use the local version of your of your algorithm instead of developing it on your jupyter notebook, develop it with this new framework and then it just immediately goes into production. So it's actually taken a lot of the extra engineering overhead out of it.

That's really cool.

Yeah, it's pretty cool because you can build new products out of it fairly quickly. Somebody has an idea, you can take it from prototype to production just really fast, which means more tests, more product changes and so on like that.

Is there anything you've seen recently that you've really liked or that's caught your eye?

A cool experiment we did recently was infinite quizzes that's newly out on the mobile version of the site. So if you take a quiz, you can keep scrolling through and it'll be another quiz. So the recommender for that's one that I built.

Cool. So how infinite is it?

That's a good question. I actually, you know, that was a problem that I was worried about, actually. I didn't get a chance to go into depth on it. So the question then becomes, what's the typical cycle length? Because if each article goes into another similar article, then there should be, first of all, a finite number of articles. So there's a maximum cycle length. You could go through all of the articles, but in general you're going to arrive back at one that you've seen before at some point. So then what's the typical cycle length? And that's not something that I got to check out.

A lot of what buzzfeed needs to do is make a really cool experience for people who are on your platform.

Right.

So what type of data does BuzzFeed have access to on people who visit it in order to make these types of decisions?

Sure. A lot of it's pretty standard. It's the types of action logs that most websites will generate. So when you perform an action on a website, whether it's a scroll or a click, or just viewing a page, so the page load event or having an impression of a link, that is seeing a link in your browser sharing an article, all of those are actions and those tend to be logged. So people generally tend to collect more data than they actually use, just so they have it if they do have a use for it later on. So if you look at the network logs in your browser, you can usually see these things happening as query parameters on request for images. So it'll be like a one by one pixel gif that's being requested and you'll see the data that's being recorded on the query parameters. So most websites will have something like that and you can actually see what's being recorded. So we do that kind of thing. We've also got a technology that we built in house. It's called Pound. That's actually the project that brought me to the company in the first place. So this one is a lot like page view tracking, but also on the page view is a referring user or rather a referring client. So all of this is anonymized on your browser and things are kind of hashed into random strings. But we can kind of de randomize it on our end because we know some extra data about the randomization process and we can kind of build out the graph. Of course, all of the IDs are still anonymized. They're just numbers to us of who shared what with whom. And we can kind of build up the diffusion network out of that. It was a much more useful idea a bit early on when the landscape was a little bit less complex. It doesn't work on some different platforms and it doesn't work for some different content types, which is kind of a newer problem with the technology. Still a very cool technology, Very cool data set.

Yeah, it sounds really interesting. Could you just remind us what a diffusion network is?

Oh, sure. So the idea is you have lots of individuals. Imagine these individuals as points. So these are our users and each of these people might be connected to one another through some kind of communication channel. Maybe they're friends on Facebook, maybe they're friends on Twitter. Maybe they just have each other's phone numbers and they can text each other links to BuzzFeed articles. So we have all of these communication channels and those you can imagine as being connections between the points. So this whole thing, the collection of individuals and communication channels, forms a network or a graph, as you would call it. And so really, when we're talking about content spreading on the Internet, we're talking about content spreading over the collection of all of these individuals and diffusion channels. So that's our diffusion network or diffusion graph, or you might call it an influence graph as well.

This is something we've kind of been skirting around. You're interested in mathematical methods for social research, and of course, the type of data sets we're talking about. There's a lot of room for movement with respect to social research. What social questions interest you the most currently? And how can math help us to answer them?

Oh, yeah, that's an interesting one. So this actually gets a bit more toward my interest in causal inference. So in one approach to causal inference, pioneered by Judea Pearl, among other people, is a causal graphical model approach to causal inference.

Could you just remind us what causal inference is in itself?

Oh, sure, yeah. So the basic idea of causal inferences is you really want to understand what causes what and how it causes it. So I could say viewing an advertisement might cause you to buy a product. And so it's really not guaranteed that viewing the ad is going to cause the product, but it might make you more likely to buy the product. And so the causal effect there we would be interested in is the increase in your likelihood of buying the product. So that's kind of the basic idea of causal inference in general. There are many possible causes and many possible effects that you're interested in. And so you can regard each of the causes, again, as a node in a graph and the causal relationships between cause and effect as an arrow pointing from cause to effect. And the effect can now be a cause for some other effect. You can kind of chain these things together into a graph which is directed graph directed from cause to effect. And it contains no cycle. So it's acyclic. What that means is a cause can't cause an effect which then causes the original cause, because then it would have to go backwards in time, which is not causal.

Yeah. So how can these graphs help us think about the world?

Yeah, that's an interesting and complicated question. So part of it is they help us reason about what causes we need to consider when we're thinking about a causal problem. For example, I could be looking at the effect of viewing an article that is an advertisement on your propensity to buy a product. But if the article was an article about that product, and your propensity to buy the product is driven by your interest in the product, then viewing the article and buying the product could be related to each other by your interest in the product originally and not your interest as created by viewing the advertisement. So the product then. I'm sorry, the advertisement could actually have no effect and you would still find a positive correlation between viewing the advertisement and buying the product just because of that latent interest in the product. Right.

So this is exactly the statement, correlation doesn't imply causation.

Correct? Yeah, so I just described confounding. So you could actually draw these variables in a causal graph where your interest in the product causes your viewing the page, your interest in the product causes your buying the product, and there may or may not be an arrow between viewing the page and buying the product. So this is the quite. Yes, that's the basic picture.

So why is it then important to think about causality in modern data science in general?

Sure. So the example I just gave is a very simple one, but you can generally have much more complex pictures than this. And generally the way we understand causal relationships is by performing experiments. But the problem is that you're not always able to perform an experiment. We've had cases at BuzzFeed like that. So in particular we had a title optimizer where a naive analysis of this showed that if you use the title optimizer, you'll see your content and performance improve by about five times. So of course, you know, everyone should use the title optimizer. You'll see a 500% improvement in KPIs. You know, why would anyone not? Well, of course the optimizer wasn't nearly that effective. The problem is that people writing articles would have to opt in to using it. They would do that by writing several variations of the title for their article. And so the people who tended to write more variations of the title were also the people who tended to just be better writers. They're willing to put that extra effort in. So there was a correlation between using the algorithm and writing good content, that if you just looked at the correlation between the two, you would see this gigantic performance improvement. Whereas the true performance improvement was was less than the 500% that we measured this way. This was a case where we actually couldn't use an Experiment, because we can't force people to not use it or force people to use it. Even if we did, they might use it begrudgingly, or it might be quite different than what they would use kind of in the long term. So we're just watching an observational system that we had no control over. And we were tasked with trying to figure out what the real effect of this optimizer is. So this is a case where we have confounding. Again, we know the variable we'd like to control for. So as a proxy for our author's skill, we just use who the author is, and then you can control for that at least somewhat, and you can see the effect size decrease.

So when you first told me there was a 5x improvement in performance, I was already skeptical. Right, because that's a huge increase in KPIs, right?

Yeah, it's enormous. And this doesn't just happen in our context. You really can see it all over the place. It's been observed in cancer and chemotherapy. For example. If you receive chemotherapy, it looks like there's a negative relationship between whether or not you'll recover.

Why is that strange?

Right? Yeah. Can you guess? It's tricky. The problem is people who receive chemo.

Their case is already quite bad.

Exactly. Yes. So the idea is they're already very sick, they have a lower baseline recovery rate. If they're receiving chemo, then people who aren't receiving chemo, they're already much, much less likely to recover. So if you take this group who has a very low recovery rate and then give them the chemotherapy, which somewhat increases their recovery rate, it's not enough to compensate for the already very low baseline. And you'll find a negative relationship between people receiving chemotherapy and recovering from cancer.

Wow, that's incredibly interesting. Is this a variation on Simpson's paradox?

Yeah. So what I described is exactly just Simpson's paradox. It's the idea that if you condition on something, you can see, in effect, flip flop, in this case, you would have this naive, unconditional measurement between whether or not they recover and whether or not they receive chemotherapy. And then if you then condition on how advanced their case is, you'll see the relationship change sign, that is chemotherapy will go from having a negative apparent effect to its true effect, which is a positive effect. If you didn't know about the causal structure of the system, you would have a very strange result. There's a funny case that Perel goes into in his book, which is a doctor finds that a medicine is effective for the population overall, but has a negative effect on both men and women. So this is actually something that can happen. So the strange result that comes out of that is if you know the sex of the patient, then you shouldn't give them the medicine. Otherwise you should. Of course, that result is garbage. You can read his book, and I encourage people to read his book Causality if you want to find the actual solution to that problem.

And we'll include a link to Pearl's book in the show notes. And we'll also include a link to Simpson's Paradox so you can see some visualizations of exactly what we're talking about here. We'll return to our chat with Adam after a short segment. Let's now jump into a segment called Data Science Toolbox with Mike Lee Williams, a research engineer at Cloudera Fast Forward Labs. Hi, Mike.

Hi Hugo. Today I want to talk about probabilistic programming.

Fantastic. What is probabilistic programming, Mike?

It's a programming paradigm that makes it easier to learn from incomplete, imperfect data. And all data is incomplete or imperfect in some way. That's because we can't run a B tests forever. We make our measurements using imprecise equipment. So it's a big deal. The right ways to handle data like this, such as Bayesian inference, have historically been mathematically complicated, computationally expensive, a hassle. Probabilistic programming languages usually do at least two things to make life easier for us. First, they come with well tested, fast batteries included, implementations of the latest and greatest algorithms to do Bayesian inference. And second, where a regular programming language knows about primitive data types like strings and integers and arrays. Probabilistic programming languages know about data types that naturally occur in the context of uncertainty. So that's things like random variables and distributions.

Okay, great. So let's back up a bit and can you tell me what Bayesian inference is?

Bayesian inference is a strategy for taking what you know about the world before you conduct an experiment and the results of the experiment and combining those things to come up with a new statement about what you know about the world. In other words, it's a way to learn from data. The mathematical basics are natural, uncontroversial rules of probability, and we figured them out centuries ago. But the practical engineering details are the difficult bit. Applying that idea, that very simple idea, to interesting problems with big data. And lots of unknowns are still being worked out. Probabilistic programming languages like Stan and PYMC3 know about these engineering developments and make it easier for mortal data scientists like you and me to use this approach.

Okay, I got that. So what are some appropriate use cases for Bayesian inference?

Right, Right.

Bayesian inference is hard. And sure, probabilistic programming makes it easier, but why bother in the first place? Well, there are tons of reasons, but I'm going to focus on my favorite, which is risk. The output of the Bayesian approach is a probability distribution, not a number. Let's say I'm trying to predict the profit I'll make if I apply a particular investment strategy. I've got some training data and vanilla linear regression using Scikit Learn or whatever will give me a single number estimate, say $100, which in some sense is the model's best guess. But the Bayesian approach will go further. It will give me a distribution. Maybe the most likely outcome really is $100, but the probability of that might only be 20%. And there's a 5% chance I make $200, which would be great, but there's also a 10% chance I'm going to lose everything having that complete picture. The probability distribution. Being able to describe the likelihood of all possible outcomes, not just the most likely, is invaluable in situations when you're dealing with risk. And that's not just the obvious situations like gambling and investment banking. If it costs a million dollars to change the design of a website, the cost of being wrong about the relative popularity of layout A versus layout B is significant. So, among other things, Bayesian inference is great when you're dealing with risk, which is a lot of the time that makes sense.

And those are some great use cases. So what data products have you built at Cloudera, Fast Forward Labs that use probabilistic programming?

We built a proof of concept web application that models the New York City real estate market. And what that does using these techniques is instead of just saying a house in this neighborhood will cost a million dollars or whatever, it knows the predicted distribution of prices, which allows it to help you answer questions like, let's say I have a $700,000. What's the probability I'll be able to afford a house here next year? Or I could turn that question upside down and say, I want an 80% chance of being able to afford a house here in two years time. How do I need now? And the prototype also used hierarchical modeling, which is a technique that allows us to deal with areas of the city where no real estate transactions took place in a particular time. In our training data, it allows us to fill in the blanks probabilistically.

So what tools are available in probabilistic programming to estimate these distributions of interest.

Well, for our prototype, we built the probabilistic core using a language, a probabilistic programming language called Stan that's the best known perhaps of the probabilistic programming languages, along with a Python library called PyMC3. I like both, and I'd probably start a new project with PYMC3 these days. There are other options. Edward is great if you're into deep learning, and if you're in the Scala ecosystem, you're going to want to check out Figaro. The algorithmic challenge these systems solve for you though, is the same efficiently sampling from a probability distribution. And you can do this the brain dead brute force way. And I think it's a really valuable exercise to do that for simple problems. So I wrote an article about it for O'Reilly that you'll check out. But for interesting problems, you'll need a faster algorithm, and the current favorite is one that rejoices in the absurd name Hamiltonian Monte Carlo with a no U turn sampler. It's fast to run, but it is not easy to code up. The good news, however, of course is that it is built into Stan and PyMC3, so you don't need to code it up yourself. And they make life even easier. As I mentioned right at the start, they build into the language's primitive objects that represent things that come up in Bayesian inference, that's random variables and distributions. So once you're over the learning curve, the programs are short and they run quickly. So if you fancy a break from your pandas, your R, your SQL play with some probabilistic programming. It's a really fun challenge.

Fantastic. That was a great introduction to probabilistic programming, Mike, and we'll definitely include a bunch of resources in the show. Notes. Thanks for the chat, Mike.

Always a pleasure, Hugo.

Time to get straight back into our chat with Adam Kelleher. We've spoken about correlation not actually being causation, but we all know that if you make if you state correlation, a lot of people will still interpret it as causation, Right?

Right. Yeah. So that's a problem you run into in practical context. Right? So as a scientist, you can be as care as you will about making causal claims when it's causal and not making causal claims when you know it's just correlative. But you have to communicate those results to lay people. Lay people probably don't have such a fine understanding of correlation and causation as the scientist does. So then what's going to happen is if you don't kind of strongly qualify exactly why something is not causal, you'll tend to see people interpreting it as causal, even if it's not. There was an example from the news recently about B vitamins causing cancer. The title of the article was something like can B vitamins cause cancer? And so they're making causal questions in the statement of the article. And the results of the study are just correlative results. They give a list of things that they've controlled for, but there are other things that they might not have controlled for. For example, is the person sleeping? Well, because we know some cancers are caused by poor sleep pattern. Some B vitamin consumption is for energy. So if you sleep poorly, you'll probably consume more B vitamins. And so bad sleep might be a confounder for cancer and B vitamin use. Yeah, I mean, even if you do adjust for that, it's so hard to correct bias. Well, in studies where we know the sources of a significant amount of the bias, and that is the incorrectness of our causal effect estimates, you can find, for example, there are studies in advertising where they use causal inference methods and compare them to actually running an ad effectiveness experiment. And naively, you can find something like 1,200% bias, which is enormous. And the reason for that is because of activity. If a person is an active Internet user, they're more likely to be exposed to the advertisement. If they're more active, they're more likely to buy a product online. So if you control for activity, you can, as they do in this paper, you can reduce the effect down to something like 800%, but it still doesn't make the problem go away. It's still 800% biased. And you've controlled for what you know to be kind of a very large source of bias. So, yeah, it's interesting. Which isn't to say that it can't be done. Facebook and Kellogg did a similar study. They took a big nonlinear model with thousands of covariates to do propensity score matching and went from about 400 bias in their INIT ad effectiveness estimate to only about 25% bias. So that's getting to the level where you're not going to get a sign change in the ad effect. And that's pretty often all you care about, whether the ad was positively effective or not. I mean, obviously often you do care about how effective it is. If you're comparing lots of different ads and whether you want to invest in more in one than Another, but at least you're showing reliably the correct sign.

So these are some very cool examples of how confusing correlation and causation can get people in all types of disciplines and fields into trouble. How could confusing correlation and causation get a hypothetical buzzfeed data scientist such as yourself or a colleague hypothetical? Once again, how could confusing correlation and causation get buzzfeed data scientists into trouble?

Sure. Yeah. So there's a fun toy example. I actually saw it on a blog post somewhere. So I use it as a toy example because people have actually made this incorrect analysis before. So suppose you have a bunch of articles and you want to find what's the best length of a title to get the highest click rate that I can get. So what you'll do is you'll take articles that are all of a specific title length and take all their click rates and average them together for each possible title length. So all articles of eight words, all articles of nine words, all articles of ten words, and so on. And what these people did is they found a peak right at 15 to 18 words. And then it kind of drops off as titles get longer. So what you would say just by looking at that graph very naively is the ideal title length is about 15 to 18 words. And of course this is correlation is, not causation. So they're making that mistake here. So if you actually implemented the policy of saying everyone should write articles 15 to 18 words long, you could actually see a negative result from that. The real optimal title could have been something like eight to ten words long. It could have just been that your best authors were writing articles about 15 to 18 words long. Even though they're not the optimal length, they're still getting better click throughs because they're your best authors. So what you did was by implementing this policy of writing 15 to 18 word titles, you took people who were actually writing eight to 10 word titles this more optimal title length. You kept everything else that is in particular their skill held fixed when you made this changed. And so you would end up seeing the click through rates dropping as a result.

Yeah. Wow. So that's a really instructive toy example. And you actually told me about a concrete and actual example in which thinking about causality correctly has helped people at BuzzFeed.

Right, sure, yeah. So this is actually related to the title optimizer we talked about. We made a change where for some articles we were going to use a different algorithm. So instead of using a multi armed bandit, we'll use a Bayesian A B testing. So this has the advantage that it gets its statistical significance faster. So the idea there is we find the best title as quickly as possible and then we can use that title to promote the article elsewhere. So there's actually a conference talk recently, a data engineering conference that Lucy Wang, one of our employees at BuzzFeed, gave. And so you can check out that one for a bit more detail around this. But the idea was this new algorithm we knew wasn't minimizing regret anymore, so we expected to see fewer clicks going to those articles. But the analysis, actually the naive analysis that is, showed the opposite result result. When we controlled for author like we did before with the observational study with the same, that is the multi arbanded algorithm, we saw the results flip again. We saw the new algorithm was performing worse as expected and the old algorithm was performing better. So that was a cool kind of inversion that we expected to see and that we did see.

That's awesome because it shows that thinking causally has actually helped in making business decisions. But it is, it isn't seriously complicated. The causal graph and the way you've explained it are relatively straightforward in these cases.

Yeah. It's a bit tricky though, and I'd caution people about using it kind of too confidently. The trouble is that these are fairly simple situations and we had a fairly good idea of how the system worked exactly. But in general, you won't know exactly the right things to control for. In that ad effectiveness study with a 1,200% bias, they only controlled for activity bias as well as they could and they saw the bias and the effect dropped from 1,200% to 800%. So there's still quite a lot of bias there.

Yeah.

So if you ended up with 800% bias, would the approach generally be to look for another confounding variable and condition on that?

Yeah. So that's the approach that you would take? Yes. There's an orthogonal approach you could try to take. There are a few kind of structure based approaches to controlling for. Sorry for getting rid of bias. These are not conditioning based approaches. These are things like instrumental variables and mechanism based methods. They're a bit more advanced and they require a lot more knowledge of what the graph looks like, but they're definitely worth learning about and they're good to have in your data scientist toolkit.

Awesome. And for anybody wanting to know more about this, we'll also include, among other things, a series of wonderful blog posts that Adam has written on all of this causal business.

Yeah, thanks. I'm trying to get this, this type of information out there because it's not something that I've come across a lot. And really the types of questions you're asked as a data scientist are almost always causal. Types of questions it would be what would happen if we did this differently? And so you only have this kind of observational data to work with unless you run an experiment. You're not always able to run an experiment. So you're kind of left with this causal vacuum.

We've talked about a lot of different techniques and methodologies in data science. What's one that you enjoy using a lot?

Oh man. Yeah, it's a fun question. You might not believe. It is not neural networks.

I love it. I love it.

At our scale, it's a bit tricky. We don't have the scale of the data sets for one thing, or the kind of technological sophistication to build up really big neural networks and train them. Well, if you look for example at Google Translate, how that one works, they put out a paper on this one, they had something like 96 Nvidia K GPUs running for like seven days. I did like a ballpark calculation. I think that would have been like $300,000 just for that one training run.

That's wild.

It's enormous. So I mean, it's a little bit crazy for a company of our scale to invest that much in a single training run of a single outgrowth just wouldn't be responsible. So we can do some cool things with neural networks title modeling using things like third party APIs that make these things more reasonable. For a somewhat smaller company, we can use things like LSTMs for title modeling. Not to understand the full content of the title, like reading it like a person might, but at least for really smart engrammy and kind of picking up on structure and adjacent words. So we can do some cool stuff with that, just not like the real kind of like Google Translate level of neural network learning. I'd say my favorite algorithm is probably logistic regression. It's easy, it's extremely interpretable. The coefficients just give you the increase in the likelihood of an outcome given a unit increase in a variable. So that can actually be really actionable. So if you want to, if you take a data set and you think that the things you're controlling for with it make your results causal, then you can actually causally interpret these likelihood changes. And also you can use logistic regression for propensity score estimates. It's just a nice easy way to do propensity score estimation. So that's kind of a requirement for a lot of causal inference methods. Yeah. So I think that's probably my favorite one. It's generally a go to.

Yeah. And speaking to the fact that it's interpretable means we can all actually talk about it.

Right? Yeah. I could tell you I found this value for this coefficient. And so what this means is that if I increase, for example, maybe we did a title analysis like we described and. And we controlled for the right things. So now I can tell you if you increase the title length by one word, then I expect your click rate to increase by 10%, something like this. So, yeah, those are the types of conclusions you can make if you use regression analysis, even with very simple models in a way that's carefully causal.

No doubt. So my last question for you is, do you have some sort of final call to action for our listeners, for the aspiring and established data scientists listening out there?

I think if we want to do large scale causal inference and choose the right variables to control for at a scale where we're not able to look at each variable and how each of them relates to each other, now each of them relates to an outcome of interest, then I think we need to be able to build causal graphs at that scale so have causal graph discovery algorithms. Just because the human approach of looking at all pairs of interactions between variables just doesn't scale to very large numbers of variables. So I think we need to think precisely with the Perlean approach to causality that is worrying about the graph and the structure of the relationships between the variables, as opposed to the kind of more typical potential outcomes approach where you're really focused on the cause, the effect and the determinants of the cause.

And as we've discussed at length in this conversation, this can have huge potential effects from, from the type of work you do to medical diagnosis to epidemiology, all across the board.

Yep, that's right. So some of these things you can see the problems because we've done experiments and we've compared them with observational data and we can see that this is a real thing to be worried about. So I think we're getting to a point where technology is just getting better and better for solving these problems. It's a pretty cool time to be working in this area.

It really is. Adam, it's been an absolute pleasure having you on the show.

Thanks for having me, Hugo. I've enjoyed our conversation.

As have I. Thanks for joining. Our conversation with Adam about data science at buzzfeed and the modern digital media landscape. We saw the types of experiments and data science workflows built around the feedback cycle of producing content, gauging performance and looking at the data and how this workflow can be used to achieve viral. We also saw how instead of classical AB testing, both multi armed bandits and Bayesian A B testing can be used to ensure optimal content as well as a dive into the role of causality in thinking about modern data science. Make sure to check out our next episode, a conversation with Katie Huff, Assistant professor at the University of Illinois at Urbana Champaign in the Department of Nuclear Plasma and Radiological Engineering. I'll be chatting with Katie about data science, nuclear engineering, the importance of interdisciplinary data science and the open source, among many other things. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and datacamp datacamp. You can find all our episodes and show notes@data camp.com community podcast.

Sam.