#4 How Data Science and Machine Learning are Shaping Digital Advertising (with Claudia Perlich)

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Claudia Perlisch, chief scientist at Distillery, a role in which she designs, develops, analyzes and optimizes the machine learning algorithms that drive digital advertising. We'll be discussing the role of data science in the online advertising world, the predictability of humans, how her team builds real time bidding algorithms and detects bots online, along with the ethical implications of all of these evolving concepts. I'm Hugo Bown Anderson, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems data science can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBound and you can also follow DataCamp at DataCamp. You can find all of our episodes and show notes@datacamp.com community podcast. Hi Claudia, and welcome to Data Framed.

Well, thank you so much for having me.

It's such a pleasure to have you on the show and I'm really excited today to be talking about how data science and machine learning are shaping and reshaping digital advertising. But before we get there, I'd like to find out a bit about you.

Okay.

What are you known for in the data science community?

So I have a couple of different hobbies. Even within data science, I think what most people may know before, and that's almost where my quote, fame started, I used to participate in a lot of data mining competitions. So you may recall the Netflix price where you could make a million dollars if you were substantially better at at recommending movies than their existing algorithm. And much earlier than that there have been in the field, really in the geek world of machine learning competitions where people are trying to build the most accurate model on a data set that was provided by the organizer. And I've been participating in those for quite a while. And then I won three in a row between 2007 and 2009 on one on breast cancer prediction. The other one was on training prediction from telecommunication and also one of the Netflix data set, although we didn't get the million dollars on that one. So that's a little bit of my initial claim to fame that really helped being perceived as a kind of hardcore part of the machine learning community.

Great. And so I suppose one of the most famous platforms where this can happen now is Kaggle.

Yes. So Kaggle is basically the next gen. When it became much more mainstream as machine learning and BEC data picked up, they provided a very nice Interface where now, not just organizers of this conference, but in general, nonprofit organizations, companies, all of them have a very easy way of interfacing with a huge community of thousands of people who have fun building these models.

So this speaks to some of how you got into data science, but I want to probe a bit more into what is there in your background that led you to data science, what type of skills you develop or what, what, what jobs did you have or what did you study that that led you to data science?

Right.

So I grew up in East Germany, and other than knowing that I was good in math, I didn't really have any convictions and no clear idea what I wanted to do with my life. And so my dad took me aside and said, look, they will need computers everywhere, so why don't you study computer science and if you're good in math, and you should be fine with that. And that was really how I picked my first choice for my undergrad, being computer science. And his words were even more prophetic if you think about what happened to data science now being really extremely hot in all kinds of different application areas. I migrated into Data Science in 95 as an exchange student at CU Boulder when I took my first class on artificial intelligence and artificial neural networks. And just love the fact that you could learn so much about the world and all of these different fields by looking at it through the lens of data, aside from these kind of very measurable challenges of building the best possible models. So that world just appealed to me from the first time I basically tipped my toe into data.

And I think this, actually, this is a nice lens through which to view the emergence of data science from your background, right, because you were good at math, you studied computer science, and then you moved into model building and predictive analytics. And these, I suppose, are three of the things which people associate most with whatever data science is these days.

The challenge with data science, and I would argue even today, really with artificial intelligence, is that it really depends on the background of the person you're talking to. So I think overall you are right with your characterization. Generally, there is a sense that you also want statistics as well as some domain knowledge in the mix here. I personally have found that the fact that for my Ph.D. i moved from computer science into a business school here at nyu really shaped my focus away from the purely algorithmic towards being much more interested in what kind of problems you can solve with these tools and algorithms. I think this is really where the birth of data science in its kind of broad application originates. It's in using data in combination with algorithms to solve some very specific problems.

And so in that way it's actually domain centric and question centric.

It should definitely start with a good question. You can't just jump into a data set and dig around there and hope to find gold as it is sometimes ported. You do need to first really understand what are the things you want to do. Is there any decision you can make? Because if there's nothing you can do, then you don't need to waste your time on looking around in data. So I really like to start with the constraints of the problem to then being led down the path towards what data is most appropriate and what algorithm can help me solve it.

This is great because I really wanted to talk about your work in digital advertising and your work as chief scientist at Distillery. So perhaps you can speak to the types of questions that you use data science to solve in that work.

So before we get into this, the story of my life really progresses from getting a PhD in Information Systems in a business school, deciding that rather than taking an academic career path, I really wanted to focus on these applications, which brought me to IBM, Watson where I stayed for six years. And then I was lured into the world of digital advertising really by it's almost the promise of a golden land. It's like a big sandbox to play in. Digital advertising has an incredible data footprint where you can experiment and really push these algorithms to the limit. So it's a huge experimentational field where you can understand how well you can predict human behavior, which is typically somewhat limited, but a lot better than random. And when these methods, in this conversation between correlation versus causation can help you to understand causality. And when they don't. So for me, the excitement really was in not just the sheer amount of data, but also the ability to try these things, put these algorithms to the test and see how they perform kind of in the real world.

Great. So can you give me the elevator pitch on digital advertising?

I don't think I can give you an elevator pitch of digital advertising. I can tell you what we do and what I've been really enjoying to do for the last eight years. So you may be familiar with the rise of the programmatic advertising world. And what that means is that advertising are now being sold in real time auctions. So every time you interact with your digital device, whether this is reading news stories on the web or using an app, you will be exposed to many different ads. But these ads really were bought in real time. As the page loaded in one of these auctions. And so what distilleries specifically initially started out doing it was the promise of being able to pick extremely selectively the right person and moment by using predictive modeling that informed the automated bidding when such a good opportunity showed up, and then adjusting bid prices for that. So this is the core promise of programmatic advertising, where you see everybody all day long and you can choose with very high precision when to interact with a customer. Now, around this core problem, there are lot of interesting, quirky, fun, other things that can keep a data scientist excited. So we have, for instance, problems around fraud. There are a lot of the moment you have an open market where people can buy and sell the same way you need on ebay, reputation. All of a sudden, even in that environment, you run into scenarios where this is not a really a real person who is sitting there, who may or may not see the ad, but in fact it's a bot that was written for the sheer purpose of selling ads. So that was a very interesting discovery back in 2012, where our models looked really at predicting certain outcomes. And the thing is, usually people are not that predictable. And then when we try to understand what was going on. So the performance was too good to be true, because bots really are deterministic, they're easy to predict. And that was one example of these side problems that we're having. On the other end of the spectrum is we're already spending all of this effort of machine learning and AI on huge data footprints, just forbidding. But then the clients come back and said, well, this is really amazing. You have this great performance. What did you do? And other than shrugging and saying, well, we build a predictive model in, I don't know, 500,000 dimensions. That wasn't the right answer because they really wanted to understand what we may have found out about their potential future customers. And so increasingly now we are looking into translating back what this artificial intelligence kind of found in this vast different behavioral patterns to be able to not just show the right ad, but to answer more strategic questions about why do my customers actually buy my brand? What's their perspective on the value proposition of the product? And some of this you find encoded in these models that are very good at products predicting. And now increasingly we work with augmented reality to just give this information back to brands to help them understand what their customers are really doing.

Yeah, absolutely. So there we have a question posed in non data science terms. I mean, well, a job to do to build algorithms that will predict whether people will click or not in order to make real time bids in these real time auctions. But then another non data science question emerges which is what type of why are our customers doing this? So it's really a translation in both directions, which I think is incredibly interesting. And it really speaks to this idea that data science doesn't exist in a vacuum as you stated initially, that it responds to real world questions. And we also need to, as data scientists, translate our results to non data science people, whether they be customers or managers or people where we're consulting.

One of the most interesting quirks to this is as you said, we're predicting people clicking. We actually learned the hard way that it's a really bad idea to use powerful machine learning to learn when people click on ads. The reason is people occasionally click on ads because they're interested in the product, but much more often it's an accident when you're trying to either close it or just change the window. But the fact it's an accident doesn't mean that that's random. And these algorithms are good enough to find out how to predict the accidents. And turns out it's much easier to predict accidents because they're typically either contextual that people don't pay attention. So for instance, we see very, very high click through rates on the flashlight app because there people fumble in the dark and there's a very good chance that you accidentally hit the ad. Other scenarios might be just people with very kind of bad fine motor skills or eyesight problems just tend to be more prone to clicking, which definitely doesn't mean that they're truly interested in the product. So the interaction between really smart technology and optimization metrics that were kind of okay for a very long time, but now we have to educate back saying this is not a good idea to combine these two things because the technology is actually too powerful. We need to think a lot harder what we're optimizing for because we may end up doing the exact wrong thing.

And I suppose what's telling is that in all these cases the things that are easiest to predict are the things you don't want to predict, whether it be bots clicking or people using the flashlight app. I think I saw you give a talk once in which you had another very telling example of wanting to advertise airport related stuff or travel related stuff for people at airports. And I'm going to get this wrong. So I'd love for you to tell me that.

That example again, you remember correctly. It's, it's really a progression of what I just talked about. So if you really listen to me saying we have bots that go and visit websites and then we have accidental clicks, so what should we optimize towards? And one thing I try to do is say, well, nobody accidentally goes to a physical location. What we could try to do is predict whether people will go to a store or an interesting location. And so I did two different experiments. One was predicting who will go to a car dealership, and that one actually was very successful. Really interesting to see the market research the different brands that consumer or prospective buyers look at before they then choose and go to that Mercedes car dealership. And since it worked so well. And there is this other group that everybody always wants to reach, which is the frequent traveler, presumably because they have a lot of money, probably bad conscience, and need to bring gifts home. So it's always a great audience to reach. And where would you expect to find them if not in airports? But it turns out that the people who are much easier to find in airports are all the people who work there from the baggage handler or the people at the check in. And they spend their whole day on their digital devices because they're there every day and have very typical patterns of behavior. So again, by not thinking very clearly, what else could explain what you're looking for? We found mostly employees of JFK rather than the elusive frequent traveler.

Now it's time for a segment called Tales from the Open Source. Today we're going to hear from Alan Nicol, co founder and CTO at Rasa, who develop open source tools for conversational software. Hi Alan.

Hey there, Hugo. Thanks for having me on the show.

It's such a pleasure. What are you going to tell us about today?

I'd like to give a bit of background on a pair of open source tools for building conversational software. So that means chatbots, voice assistants, and anything else where you interact with the computer through natural language. Browser NLU is the first part and it says, okay, I've received a message. What is this person trying to say? And then Raza Core is the second part. And it takes that information, looks at the history of the conversation, and decides what to do next. They're relatively new, but they're already used by thousands of developers.

That sounds really cool. How did this originate?

Back in 2015, we were building some bots on Slack and we realized that the developer tools just weren't there to build really great conversational experiences. And then in 2016, Facebook announced that they were going to open up messenger as a platform, which meant that tens of thousands of developers would be building chatbots using keyword matching and 1000 if statements. And we knew from experience that that just doesn't scale. So Rasa, NLU and Core are built entirely on machine learning and everything that your system understands and is able to do is learned from real conversations. We're trying to get people to move away from thinking of conversations as flowcharts because it just doesn't work for anything beyond a hello World app.

And what's been the response from the developer community at large?

We were totally blown away when we first released Rasa nlu and I think it was just the right time and the right place. It was a few months after the messenger platform opened up and everyone still seriously developing bots wanted to be independent of the language understanding APIs that the big tech companies were offering and we saw a huge amount of traffic right away and we've seen over 60,000 downloads.

In the first year.

We've learned so much from our users and contributors. Open source is really unique in that way. People get stuck and then they care enough to really write a good issue and lots of them then actually go back and contribute to the project, which is amazing. Rasa also builds on a bunch of amazing open source libraries like Spacy, Scikit, Learn and Keras.

So what's next for Rasa?

Our take is that Rasa, NLU and Core are meaningful steps in the right direction. But making natural language interfaces to computers is an infinitely hard problem and we've definitely not solved it. So we do a lot of applied research and sometimes we publish papers, but mostly we care about turning that research into code people can actually use. There's so many ways that natural language understanding and dialogue need to improve and I'm really excited about the things that we've got on the roadmap for Rasa.

Thanks Alan for that dive into Rasa and open source conversational software. For those interested, check out Rasa AI that's spelt R A S A in case you can't understand our silly accents. Thanks once again Alan.

Real pleasure to chat. Hugo.

Let's now jump back into our interview with Claudia Perlish. You've hinted at this, but part of your job is to predict human behavior and you've hinted that humans aren't so predictable. But how predictable are they? We?

How predictable are we? Well, I was really fascinated by and this has nothing to do with advertising. You may follow kind of these competitions that AIs are now engaging to, starting with the chess games back in the 90s over recently we had jeopardy that Watson won to go. But none of them really has that much to do with predicting human behavior and more so with strategy. What is fascinating that apparently now we have algorithms that can predict when people try really, really hard to be not predictable. So apparently we finally have beaten the world best poker players using algorithms that analyzes faces. And apparently we give away a lot more than we think. But that's, that's kind of a side fun story.

So the poker face really a real thing?

No, not to the machine. It may work with other people, but the machine can still see right through. But in our daily activity, I think a lot about us is very predictable, at least in my case. I mean, you can very quickly figure out what my daily habits are. The other thing that is very predictable are all kinds of consideration activities, things that require gathering information and take a couple of, I don't know, days, weeks, months to come to a conclusion. So for instance, buying a new refrigerator, for most people, that's a serious thing. And you spend some time thinking about it and you will have plenty of digital traces of that activity that helps marketers identify, oh yeah, these people are in the market for that product. When you look at the more kind of spur of the moment activities in those cases, you might be able to identify that this person is even prone to, to kind of a buy on short notice of an apple while walking over the Union Square market because the person likes apple. But can you predict that at this particular moment the person will feel like buying an apple and seeing one that they want? Probably not. So there is a huge range and you have seen this, you probably heard about one of the studies done on Facebook, data that was really concerning from a privacy perspective. This is not about predicting future behavior. But what researchers showed is there are a lot of parts of our personality and behaviors that we may not want to make public, but that can be very easily inferred. So for instance, sexual orientation, political perspectives, all of these things are really easy to infer from a machine learning algorithm just given your day to day activity.

And so that isn't something necessarily that will help distillery do their job though, right? Or is it?

This is typically not what we're interested in in the sense that we are looking to optimize very specific metrics by the marketer, such as number of new customers signing up for a service on their website. And for that, I'm not really interested in any of these concerns about who you really are and things like sexual orientation. And it's also perfectly anonymized in the sense that I don't know any personally identifiable information about you. And even your browsing and digital activity can basically be hashed and obscured. So it doesn't really mean anything. Now this being said, I think in general when you now turn it around and go back to the client and they want to know, but what did you find? I think at that point the boundary between where do I start infringing at audiences kind of privacy when I share certain correlations with character traits becomes really interesting. And one of the concerning examples, for instance even from our frequent traveler, we did find flight attendant sites, but we also found gay dating sites. Now is this something that I should be seeing? Is this something I should have that such an easy access towards? And often when I tell about the story, people in the audience feel somewhat ill at ease about the fact that this is even something that can be that easily revealed.

And this is something relatively new in the technological landscape. So presumably there isn't even legislation hasn't caught up with these types of challenges yet from a societal level.

So we are really struggling right now finding new ways of possibly self constraining how we interact with these technologies and where the line is, what type of.

Data do you have access to that helps you predict human behavior at distillery.

The data sources in digital advertising are really coming from many, many different places and different players have very different access rights. So you really see specialization. Obviously Facebook knows everything you do on Facebook and will provide versions of that data to their advertisers. We have access through these real time auctions basically to every auction that happens. And we're talking about 100 billion events every day. So 100 billion times we are being told that this particular device is right now looking at this particular content. And so you have this constant stream that ultimately then gets assembled into an activity history, very granular nature like the URL and the news for instance that you read alongside with location information if the bit request came from your mobile device. So if you're just standing on the corner and you're bored and you're playing, I don't know, Candy Crush 15 and there's an ad showing your phone just told me that you're standing there unless you were very diligent about switching off the gps. And so this is kind of one of the primary sources is actually the environment itself through which ads are being sold. And in addition to that you also have many data vendors who are providing additional information that they have collected of similar granular form.

Interesting. So do you have access to for example, how, how much time people spend on websites, even like cursor activity, if they have other apps open any of this type of stuff.

So the details of your web activity typically remain behind the scenes. So what you do with your cursor really requires an integration in your browser, which is far beyond anything that is available kind of broadly in the advertising environment. Now, sometimes the ad itself could have technology that, for instance, tracks how long it is in view and whether or not you went with the cursor over it. Something similar happens in the mobile space when you use your digital devices. What's called the SDK. It's basically almost the operating system. It's the fundamental software that is underlying most of the apps that people develop. And they themselves might collect data about what you do with the app, but also other apps that are being installed. And there's kind of an ongoing attempt from, for instance, Apple to restrict that apps stick to the rules and only kind of look at their own data and only share their own data. But there's a lot that comes directly from these deeply integrated parts of the software stack that is providing you apps.

We've talked about the masses of data that businesses such as Distillery have with respect to how, you know, a lot of people are interacting with online environments, with the online world, which is taking up more and more of our, of our daily lives. What are ways in which these masses of data can, can be used for. For social research?

So there have been really great pieces of social research recently, and I think especially around the rise of fake news and how people interact with information, propagate information that lead ultimately to these kind of information bubbles that are being enhanced by the AI itself that's trying to predict what you may want to read about. And so there are a number of researchers that do incredibly important work because it goes beyond just understanding the social nature of our modern generations. It really comes to the fundamental questions of how do we now progress with democracy moving forward if we no longer have even a remote hope for objective information when things are shared algorithmically? And so some of the work Gilad Lotan has done, I also really recommend the work that's done at the Data and Society Group here out of New York City. And then you also have various of these pieces coming from Google and Microsoft. So you see a much increased need for this understanding. And now we also have much more access to what people actually do and understanding how these processes work.

So consumers have a relatively complex relationship with advertising. What value can data science in the online advertising space add to the consumer's.

Experience My overarching sense is that by helping valuable content to monetize, we are part of the ecosystem that allows publishers to be ultimately somewhat independent. With the decline of subscription, many publishers and even blogs where everybody can express themselves have to rely on advertising as a primary source of income. I'm not necessarily convinced that we are truly providing the much needed information. I think having a possibly less disruptive experience and have advertising be part of the fabric that it fits both my interest and the kind of topic of the site where it's being displayed is an acceptable compromise. Whereas what I'm seeing with a lot of concern as advertising has increasingly focused more on viewability as a metric, meaning, for instance, that advertisers only want to pay for viewable ads. This makes initially sense from the perspective of the advertiser, but it then puts the publisher in a really difficult position because as a result of it, you have these absolutely terrible experiences as the user where the ad is kind of following you around on the page and there's no way to get rid of it. And then you have of course, high click through rates because every time you try to close it, something happens. And eventually I think that really only fosters the installation of ad blockers, which now becomes a bigger concern to publishers that are trying to provide independent and free content to readers. If large groups of readers install ad blockers.

That's a great point, because I think there is a certain balancing act that we're talking about whereby as a publisher, you want to get your stuff out there as much as possible, but you don't necessarily want to spam people, at least to the point of annoying them enough for them to take that type of action.

Exactly, yes.

So we're kind of going down the path of discussing a few ethical implications of your work and data science in general. I'd like to go down this path a bit further. You hinted to the idea of biases in data and algorithmic bias, and I was wondering if you could speak to some more challenges involved in these areas today.

So, with the vast deployment of automated systems, there have been an increased number of concerns on the ethical side of the implications that these algorithms may have. The simplest or earliest one I think you could refer to is the information bubble, where the algorithm isn't necessarily biased, it's just really good at figuring out what you like to hear. And as a result, when it comes to more important things like political information, if you only hear the side of the story that you like to hear, not only does it kind of reinforce your precision but it gives you the delusion of being absolutely right and certain about it, and you no longer have to question yourself or seek the dialogue with other opinions. So I think this is one of the early concerns that has nothing to do with even the technology being biased, but the way our brain processes information in the interaction with a kind of preselection that appeases us very much.

And this is even known now in popular culture as echo chamber.

Right, Exactly. This is another term that we have for that. Now, the next generation of concerns that were brought forth are with respect to users in areas, for instance, as predictive policing, or even something as simple as job recommendations on various job sites, where the concern is that we have for the better of the worse. Our society has certain biases, our behavior is not up to the overall standard that we want it to be. And as a result, if you now train models on behavioral data where, for instance, you have never hired a woman for this position, therefore you have no data of a woman ever being successful, therefore none of the candidates that the algorithm will find will be female. So the concern is that we could somewhat accidentally propagate or potentially even increase biases that was existent in the data that was used to build a model that now behaves exactly as we used to and not necessarily true to audio.

So this is an example, as you state, of algorithms encoding already existent societal or human biases, which is something we need to be very cognizant of moving forward. I know something else you're interested in, though, is the ability for algorithms to create their own biases, which may not be even existing in the data. So I'd love to hear your thoughts on that.

So, in the experience, and we touched on this earlier when we talked about bots and clicks and even the people working in airports, one of the things I understood is that ultimately when you build a predictive model, it's just doing exactly this. It's going to find the easiest thing to explain wherever it finds the most signal or the most information. And that becomes a problem when different groups of your population have more or less signal, more or less information. And so the example that I like to bring forth for people to consider, if, for instance, a group of people has a consistently lower usage of technology, and as such I have less data points about that person, I will be much less likely to target the person either with advertising or with a job offer, simply because the model can never quite be sure that this is the right choice to make. And there are other easier things to predict. So if you look for instance, at Jobs, if for some reason it is easier to predict success for one gender than the other, although both are equally likely to succeed, what happens if you simply use your algorithm to rank candidates? You can easily see very strong majorities of the same gender in the top 10 candidates presented. Although originally 50% of the people who succeeded in that role were actually male, so there was a balanced male female representation in the data. That's the concern that I'm having where a lot of the conversations today around making sure that your training set is unbiased. It is not enough to ensure that your training set is what I call first order unbiased, meaning you have the exact representation that you want, you still have to take the responsibility for taking action on the predictions because the predictions can be biased again. And then you, as the user or as the platform, have to make a choice to present a again equalized outcome, to pick the top N candidates from both genders, for instance.

And you're aware of this and clearly trying to do these types of things in your work, but do you think enough people are aware of this? And if not, is educating them part of our job?

So I had an interesting experience. I went and gave a keynote at Predictive Analytics World this fall in New York City and I spoke exactly about this. And after my presentation, the general chair walked up and asked the audience, well, how many of you knew that this happened? And I think intuitively, most data scientists are kind of aware of it, but in this audience, I would say maybe of the 200 people, we had 10, 15 hands going up. And the rest of them may have an inkling, but possibly not fully thought through all the way to the implication of what that means and even the bigger challenge, what now to do about it. So I still like to give that same talk, although I have been giving it for at least one and a half years now, because I do find it very important that as a community we understand the implications of our work and that it's not enough to delegate it even to legal restrictions or things like de biasing data sets. We still need to take responsibility for the usage of this technology.

Now it's time for a segment called Statistical Pitfalls. I'm here with Michael Bettencourt, applied statistician and one of the core developers of the open source statistical modeling platform. Stan, great to have you on the show, Mike.

Thanks, Hugo. It's great to be here.

So you're here today to tell us about a common statistical trap that we all fall into, that of the tyranny of the mean, right?

Very much so. In fact, this is one of my favorite pitfalls. So many mistakes in statistics are made when the mean of a population is confused with a typical individual in that population. In particular, when someone tries to use an average individual to characterize a typical individual.

And by average individual, you mean an individual whose features are given by the average or mean characteristics of everyone in the population.

Right. So, for example, the proxy individual who has the average height of everyone in the population, the average weight, the average distance from arm to shoulder, or whatever feature that we're taking into account.

So why is this such a bad idea?

Well, we already know that the mean doesn't tell us anything about the variation in the population. But we're not trying to characterize the entire population here.

Right.

We're just trying to represent a typical individual in the population. Unfortunately, if we consider more than a few features, then almost no individuals look anything like the mean.

Really?

Yes, it's an extremely counter to a phenomenon. So let's consider what happens as we look at more and more features. We'll start by considering a single feature, say, height of the individuals in the population. Now, if we went out and sampled random individuals in the population, then most of the samples would have heights near the mean height of the population.

So the mean is pretty typical in one dimension then.

Right. But that quickly falls apart as we start looking at more and more features. For example, what happens if we consider both the height and the weight of individuals in our population? How many individuals have both average height and average weight? Well, there have to be fewer than before, because now each individual has two ways in which they can deviate away from the mean. And as we consider more features, we get more and more ways in which the individuals can deviate away from the mean. And the probability that an individual doesn't vary in at least one of those ways quickly falls to zero. In other words, the neighborhood around the mean quickly depopulates as we consider more and more features. Even when considering as few as five features, almost no individual in the population looks anything like the mean. That average individual is completely atypical of the population.

So why is this relevant to practicing data scientists and statisticians?

Well, this behavior is immediately important. We're trying to design products or processes that are meant to be suitable to most of the population. For example, let's consider designing a bike helmet. If the dimensions of the helmet are based on tight tolerances around the average head, then the resulting product will be uncomfortable or unsafe for nearly everyone in the population. If we Want a helmet that will be effective for a substantial percent of the population, then we can't design for a single individual. The helmets have to be adjustable, or at least big enough for the biggest heads.

Interesting.

But this is just one example. Mathematically, this behavior is a manifestation of a phenomenon called concentration of measurement. And this arises anytime we use a probability distribution to characterize a high dimensional space. This could be, for example, a distribution that characterizes the features in a population like we talked about before. Could be a distribution that characterizes the variability of data in a measurement process, or it could even be the distribution of parameters in Bayesian inference. Anytime we have a probability distribution on a space with more than a few dimensions, the neighborhoods of high probability will be very far away from the mean. We have to be constantly vigilant of the tyranny of the mean.

Mike, thanks for that introduction to the subtle but ever present statistical pitfall known as the tyranny of the mean.

My pleasure, Hugo. Be careful in those high dimensional spaces, everyone.

Time to get straight back into our chat with Claudia. And so in terms of responsibility, what, what is the role of data scientists to think about data ethics, particularly in a world where we're reaching a point where, I mean, advertisers may know us better than ourselves?

I wouldn't quite go that far. At least I don't think we need to worry about that specifically, but.

Okay, so I just think in the case there's the anecdotal example of if someone's. We discussed cars before. If, if, if someone has displayed interest in sports cars, maybe you advertise flashy, flashy cars to them, but if they display an interest in sports cars, your algorithm knows that they may be in debt. They also have a history of alcohol abuse. These types of things. What, what type of ethical considerations need to be in place to, to, to help in, in this type of situation?

So first off, I think it is important to have an honest and open conversation about it. What I have perceived is you basically have two different groups here, people who do data science for a living and very rightly concerned citizens, often with insufficient depth of understanding of what is even controllable or can be known about these algorithms. And in some sense, I am the best police for data science because I'm the one closest to building them and observing these things. A lot of the examples I talk about, whether this is the case of JFK or even clicks and bots, a lot of this happens behind the scenes. And it's really my kind of curiosity and diligence to Find those things. And I would like us to have more open discourse what we expect from this technology and what the comparison that we want to put into place is the right level. And what I want to talk about here is not exactly the direction that you are going with some of these abuse cases, but more so when we are looking at failures of machine learning and AI, when there is an accident by a self driving car, when we have mislabeled pictures showing up that could possibly be offensive, what is the right expectation? My sense is that society feels that this technology has to be perfect. And I think this is where the disconnect in the conversation is. Because when you are doing this for a living, you do understand that ultimately these systems can be a lot better and can do a lot of good. For instance, diagnosing rare diseases that the doctor that you happen to go to in some rural area has never encountered before. But will that system be perfect? Almost truly not. And so the answer to how do we as a society engage with that, in my opinion has to be one of realistic expectations and a sense of collaboration between machine and human with a shared responsibility for the action that ultimately we choose to take based on the recommendations that, that we get. What if I do observe specific cases that hint of, for instance, that some people that I observe in the advertising environment are suicidal? Is there something I should do? Do I need to point out to the brand that this might be a group of constituents that they somehow have responsibility for? I'm not sure, but I feel I would like at least want to be able to speak up without being pushed into the corner of privacy violating, because privacy doesn't make these things go away, they just become invisible.

Exactly. So you're speaking to this idea of openness and transparency, which I think is incredibly important. And it also, I think it's heartening and helps that there are people such as yourself who are on one side working as data scientists in such businesses, but are also communicators and explainers and take that duty upon themselves to go out and speak about these types of issues in public fora, which is very welcome and necessary, I think.

Thank you. I appreciate that.

We've discussed a lot about the modern data science landscape. What does the future of data science look like to you.

Now? You're asking me to really predict the future.

Exactly.

So on the one hand side, I think the appreciation for really all the upside potential that data has will continue. So I don't think this is a fluke. I'm really excited about the fact that even Though big data as a hype term is coming to its end, but the increased sensitivity that we as a society, but also institutions and firms have that they should be more data based or data driven in their decisions, I think that's very important. I think it's also very important as these systems exist, that we as a society become more data literate. Because recommender systems are not going to go away. And we need to understand that we are living in these kind of filter bubbles or echo chambers that were mentioned before. What does it mean for data science itself? So first off, I'm not worried of us automating ourselves. I mean we are automating ourselves all the time. But I think the demand for human skill and supervision of data science systems will only rise and technology can really not make up for good human intuition and the crucial vocabulary it can play in. Exactly. These concerns we have express some of the things that go wrong and when we can trust the machinery and how we should interact with it. The tooling is incredibly elegant today. If you Compare that to 20 years ago, I think we will see more of that and tooling really being broadly available through either cloud providers or many other open access tools. I do believe that the current excitement about deep learning will come to a realization that it's not the answer to every problem. Deep learning is very good for very specific types of problems and they are really around areas that have a lot of signal. So we're talking about vision where you have very clear rules of the physical world that can be exploited. We're talking about language. We will get still better about translation and automatic conversion of audio to text. And obviously we have seen this in reinforcement learning which is where all of these games go and so on come from. But there will be a lot of space for good old kind of solid statistics just on bigger data and simple models. So I'm quite optimistic for the field with the understanding that these different tools will find their different places.

And speaking of solid statistics, what's one of your favorite techniques or methodologies for data science? Not necessarily favorite, something you just enjoy implementing or doing.

I'm very old fashioned in the sense that I don't trust myself looking at graphs. Graphs are great if I want to tell stories. So if I want to tell the story about people fumbling in the dark, then it's very nice to kind of illuminate the these things with information about click through rates. But I really like to look at data almost running over my screen. But that's probably just me being really weird. That's okay, that wasn't what you were asking. I have somewhat ironically taken almost the opposite development than the field. I started out doing artificial neural Networks back in 95 and then I downgraded if you want to decision trees in 2004 for my dissertation. And today I really value the simplicity and elegance and also transparency that you can get from linear models like logistic regression or even just simple indexing that you would probably refer to as a form of naive bayes because it's so much easier to look under the hood and understand what might be going on there. And it really has become my go to tool over the last, I would say 10, 15 years. In fact, I won all of my data mining competitions using some form of a logistic model.

Firstly, I love the idea of just you watching data stream across multiple screens. Secondly, I think your passion for interpretable models, for decision trees, for linear models, where you can actually communicate what certain things mean in these models also speaks to what we were discussing before your role as a communicator. So you can take the output of what one of these models put outputs and speak to a data science manager or someone in HR or whatever it is, or someone in the advertising space who isn't technical about the results of these models. Right.

So this is exactly, I think, why I gravitate towards it. Because initially I was geeking out over the fancy stuff and I've come to realize that if you want to impact the world, it doesn't matter what you find exciting. What matters is what you can get other people excited about. And so depending on the kind of sophistication level, it's often a really great idea to have the worst possible model. That's the nearest neighbor. The nearest neighbor is awful. It almost never has really good performance compared to some of the more sophisticated because it doesn't really learn anything. You just kind of find something that's similar, but it's very difficult to know what similar means. But it has one huge advantage. That's exactly how people think. That's the reason why in advertising we talk about lookalike models. Now what we build is not a lookalike models, but to people to understand, yeah, we find other consumers who look like your consumers. That makes sense. And they can relate and they can embrace the technology and start giving it at least a try. And then after a couple of iterations, you can swap out that awful nearest neighbor and give them a really good predictive model and they will be very happy moving forward.

And I suppose it's about establishing trust as well in that sense that. But you may have a model that performs better, but if nobody has any idea what it's doing, they don't know why they should have faith in it or trust it.

So trust is definitely very, very important here. And the other part is simply get them involved because that's what you can do with nearest neighbors. You can say, yeah, here are the five most similar other cases. And then the person can say, nah, that one. That one doesn't count because that was completely different. Okay, let's delete it. And I just work with the four. So it has this nice communication where they feel that they have become part of something. And at least that was the case in one of the projects at IBM. Trust was a component, but it was also that they felt taken seriously and part of the process and relearned when our models actually had no data. And we would have to build something entirely different for those cases where the customer just knew that this was not appropriate.

Fantastic. So my final question is, do you have a final call to action for our listeners who are aspiring and working data scientists alike?

My sense is, number one, just keep your curiosity and your skepticism. I mean, have fun with what you do, but always don't take yourself too seriously and definitely not your model. So having some appreciation when you find out why something went wrong, that's much more fun and interesting than finding out that something went right. So as a philosophy, moving forward, being cautious with the things that you build, and I think that plays into being responsible when you hand them over and clear where you think the limitations are. But first and foremost, just keep your excitement for it because that will keep you sharp and being able to identify these things.

Claudia, thank you so much for coming on the show. This has been such a great pleasure chatting with you.

Thank you so much for having me.

Thanks for joining. Our conversation with Claudia Perlisch about the evolving role of data science in the online advertising world. We discovered a lot about the predictability of humans, but also that our algorithms will often pick out the targets that are easiest to describe, such as online bots. We also saw the importance of an ongoing and increasing dialogue between data scientists and the population at large in a world that is becoming increasingly defined by the data we all produce. Make sure to check out our next episode, a conversation with Ben Skranker, a data scientist at Convoy, a company dedicated to revolutionizing the North American trucking industry with data science.

Sam.