#50 Weapons of Math Destruction

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming.

It can be to keep up with.

The rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com wow.

In this episode 50, our season one finale of Data Framed, a Data Camp Podcast, I have the great pleasure of speaking with Cathy o', Neill, data scientist, investigative journalist, consultant, algorithmic auditor, and author of the critically acclaimed book Weapons of Math Destruction. Kathy and I will discuss the ingredients that make up Weapons of Math Destruction, which are algorithms and models that are important in society, secret and harmful from models that decide whether you keep your job, a credit card, or insurance, or algorithms that decide how we're policed, sentenced to prison, or given parole. Kathy and I will be discussing the current lack of fairness in artificial intelligence, how societal biases are perpetuated by algorithms, and how both transparency and auditability of algorithms will be necessary for a fairer future. What does this mean in practice? Stick around to find out. As Kathy says, fairness is a statistical concept. It's a notion that we need to understand at an aggregate level. And moreover, data science doesn't just predict the future, it causes the future. I'm Hugo Baun Anderson, a Data Scientist at DataCamp, and this is Dataframe. Welcome to DataFrame, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bownanderson. You can follow me on Twitter Hugo Bown and Data Camp Data Camp. You can find all our episodes and show notes@datacamp.com Community podcast listeners, as always, check out the show notes for more material on the conversation today. I've also included a survey in the show notes and a link to a form where you can make suggestions for future episodes. I'd really appreciate it if you take the survey so I can make sure that we're producing episodes that you want to hear. Now back to our regularly scheduled programming. Before we dive in, I'd like to say thank you all for tuning in all year and thank you to all our wonderful guests. We have had so much fun producing these 50 episodes and cannot wait to be back on the airwaves in 2019. We'll be back with season two early in 2019. And to keep you thinking, curious and data focused in between seasons, we're having a Data Framed Challenge. The winner will get to join me on a segment here on Data Framed. The challenge is to listen to as many episodes as you can and to tweet excerpts that you find illuminating to at datacamp, Ugobaun and the relevant guest using the hashtag Data Framed Challenge. That's Data Framed Challenge. At the start of season two, we'll randomly select the sender of one of these tweets to join me on a podcast segment. The more tweets you send, the more chance you have. However, we'll delete the duplicates first. That's Data Framed Challenge. Hi there, Kathy, and welcome to Data Framed.

Thank you. I'm glad to be here.

It's such a great pleasure to have you on the show, and I'm really excited to be here to talk about a lot of different things surrounding data science and ethics and algorithmic bias and all of these types of things. But before we get into the nitty gritty, I'd love to know a bit about you. So perhaps you can start off by telling us what you do and what you're known for in the data community.

Well, I'm a data science consultant and I started a company to audit algorithms. But I guess I've been a data scientist for almost as long as there's been that title. And actually I would argue that I was a data scientist before that title existed, because I worked as a quant in finance starting in 2007, and I think of that as a data science job, even though other people might not agree. So, I mean, and the reason I'm being I'm waffling is because when I enter data science, maybe I would say in 2011, you know, large question in my mind was like, to what extent is this a thing? And so I. I wrote a book actually, called Doing Data Science just to explore that question. I co authored it with Rachel Shutt. The idea of that book was like, what is data science? Is it a thing? Is it new? Is it important? What you know, is it powerful? Is it too powerful? Things like that. So what am I known for? I think I'm known for being like a gadfly, for sort of calling out bullshit, for possibly, you know, people think of me as overly negative about the field. I think of myself as the antidote to Kool Aid.

Well, yeah, I think of A lot of the work you've done as kind of a restoring or restorative force to a lot of what's happening at blinding speed in tech, in data science, in how algorithms are becoming more and more a part of our daily lives.

Yeah, that sounds nice. Thank you.

You're welcome. And I like kind of the way you motivated your book that you co wrote, doing data science in terms of, you know, exploring what data science is, because you actually have a nice working definition in there, which is something along the lines of like a data savvy, quantitatively minded, coding literate problem solver. And that's how I like to think of data science work in general as well.

Yeah, you know, I've actually kind of discarded that definition in preference for a new definition which I just came up with like a couple months ago. But I'm into. And this is a way of distinguishing the new part of data science from the old part of data science. So it's not, it's not really a definition of data science per se, but it is what is sort of a definition of what I worry about in data science, if you will, which is that data science doesn't just predict the future, it causes the future. So that distinguishes it from astronomers. Astronomers use a lot of quantitative techniques, they use tons of data. They're not new. So are they data scientists? In the first definition that you just told me from my book, probably yes. But in the second definition, no. Because the point is that they can tell us when Halley's comet is coming back, but they're not going to affect when Halley's comment is coming back. And that's the thing that data science, or I should say data scientists need to understand about what they're doing. For the most part they are not just predicting, but they're causing. And that's where it gets these society wide feedback loops that we need to start worrying about.

Agree completely. And I look forward to delving into these feedback loops. And this idea of a feedback loop in data science work and in algorithms and modeling is one of the key ingredients of what you call a weapon of math destruction, which I really look forward to getting back to. But I like the idea that you've moved on from a quasi definition you had in, in your book doing data Science. Because a question, I mean, doing data science was, was written five or six years ago now. Is that right?

Yeah, 2012 I think.

Yeah. Right. So I'm wondering, looking back on that, if you were to rewrite it or do it again. What, what do you think is worth talking about now that you couldn't see then?

Well, I mean, to be clear, each chapter was a different lecture. In a class at Columbia, it was taken from those lectures, including a lecture I gave about finance, including a lecture that we had the data scientists from Square Speak. We had people that were probably not considered data scientists, but statisticians speaking, etc. So it was like a grab bag. And in that way, it was actually really cool because it was all over the place and broad, and we could see how sort of these techniques were broadly applicable, various techniques. And we could also go into, you know, networks in one chapter and, you know, time series in another. And that was. That was neat because we could sort of like have a survey, if you will, of stuff, but it wasn't meant to be a deep dive in any given direction. If I rewrote it now, I would probably. If I kept with that survey approach, I would be surveying a totally different world because we have very different kinds of things going on now. I guess we also have some sort of. Through streams, like, we have some. Some things that are still happening that were happening then that we'd emphasize more. I think in particular, I would spend a lot more time on recommendation engines, although we do have a chapter on recommendation engines from the former CEO of Hunch, I believe, to try to understand a person by 20 questions and then sort of recommend what kind of iPhone they should buy or something like that. But nowadays I spend a lot more time exploring things like, to what extent do the YouTube recommendations radicalize our youth?

That's really interesting because I think what that does is it puts data science and data science work, as we've been discussing already, into a broader societal context and assesses and communicates around the impact of all the work that happens in data science. So I think that provides a nice segue into a lot of the work you've done, which culminated in your book Weapons of Math Destruction, that I'd like to spend a bit of time on. Could you tell me kind of the basic ingredients of what a weapon of math destruction actually is?

Sure. A weapon of math destruction is. Is a kind of algorithm that I feel we're not worrying enough about. It's important, and it's secret and it's destructive. Those are the three characteristics important.

By.

By important, I mean it's widespread, it's scaled, it's used on a lot of people for important decisions. I usually think of the categories of decisions in the following. Like, financial decisions. So it could be a credit card or an Insurance or housing or livelihood decisions like, do you get a job, do you keep your job, Are you good at your job, do you get a raise or liberty? So how are you policed? How are you sentenced to prison, how are you given parole, your actual liberty? And then the fourth category would be information. So how are you fed information? How is your environment online in particular informed through algorithms? And what kind of long term effects are those having on different parts of the population? So those are the four categories. They're important. One of the things that I absolutely insist on when we talk about weapons of mass destruction or algorithms or regulation in particular, is that we really focus in on important algorithms. There's just too many algorithms to worry about. So we have to sort of triage and think about which ones actually matter to people. And then the second thing is that they're secret. Almost all of these are secret. Even people don't even know that they exist, never mind understand how they work. And then finally they're making important secret decisions about people's lives and they fuck up. Like they, they make mistakes and it's destructive for that individual who doesn't get the opportunity or the job or the credit card or the housing opportunity, or they get imprisoned too long. So it's destructive for them. But as an observation, this goes back to the feedback loop thing is it's not just destructive for an individual, but it actually sort of undermines the original goal of the algorithm and creates a destructive feedback loop on the level of society.

Yeah, and a point you make in your book, which we may get to, is that they can also feed into each other and exacerbate conditions that already exist in society, such as being unfair on already underrepresented groups. So before we get there though, could you provide, I mean, you've provided a nice kind of framework of the different buckets of these algorithms and WMDs and where they fall. But could you provide a few concrete examples of what you consider to be the most harmful WMDs?

Yeah, I'll give you a few. And I choose these in part because they're horrible, but also because they all fail in totally different ways. And I want to make the point that there's like not one solution to this problem. So the first one comes from the world of teaching, public school teaching. So there was this thing called the value added model for teachers, which was used to fire a bunch of teachers. And unfairly, because it turned out it was not much better than a random number generator, didn't contain a lot of information about a Specific teacher. And in instances where, where it did seem to be sort of an extreme value, they. It was often it was manipulated by previous teachers cheating. So like you couldn't really control your numbers, but if you're a previous teacher cheated, then your number would go down. So it was like this crazy system.

Yeah, because if, if I remember correctly, the baseline is set by where your students were in the previous year or who taught them. Right.

Yeah. The idea was like, how well did your students do relative to their expected performance in a standardized test? And it was a very noisy question in terms of statistics. Unless the previous teacher in the previous year had cheated on the, on those kids tests. And those kids did extremely well relative to what they actually understood, which would force them, of course, to do extremely badly the next year, even if you're a good teacher. So it would look really bad for you. But long story short, it was normally speaking when there wasn't cheating involved, just a terrible statistical non robust model. And yet it was being used to fire people. So that's the first example. The next example is this example from hiring, which is a story about Kyle Beam, this young man who noticed in a personality test that he had to take to get a job that he failed, he noticed some of the questions were exactly the same questions that he had been given in a mental health assessment in, when he was being treated for bipolar disorder. So that was like an embedded, illegal mental health assessment that, you know, is illegal under the Americans with Disability act, which makes it illegal for any kind of health exam, including a mental health exam, to be administered as part of. Part of a hiring process. So that's, that's another example. But in, and I should add that like, it wasn't just one. One job. It was, you know, Kyle ended up taking seven different versions of this test. I should say he ended up taking the same exact test several different times when he applied to seven different chain stores, all of them in the Atlanta, Georgia area. So he wasn't just precluded from that one job. He was precluded from almost any minimum wage work in the area. And it wasn't just him. It was like anybody who would have failed that mental health assessment, which is, you know, to say, a vast community of people with mental health status. So that's a great example of the feedback loop I was mentioning. You know, because of the scale of this Cronos test. It wasn't just destructive for the individual, but it was undermining the exact goal of personality tests and also undermining the overall goal of the ada, which is to avoid the systematic filtering out of a subpopulation. And so that's the second example. And the third example I would give is what we call recidivism risk algorithms in the criminal justice system, where you have basically again, questionnaires that end up with a score for recidivism risk that is handed to a judge and being and told to the judge like, this is objective scientific measurement of somebody's risk of recidivism. Recidivism being the likelihood of being arrested after leaving prison. And the problem with that is. Well, there's lots of problems with that. But the very immediate problem with that is that the questions on the questionnaire are almost entirely proxies for race and class. So they ask questions like, did you grow up in a high crime neighborhood? I mean, you grew up in a high crime neighborhood if you're a poor black person? Fact. Like, that's almost the definition of high crime neighborhood. That's where the police are sent to arrest people historically, from the broken windows policy, the theory of policing, to the present day. And by the way, I should add, like, in part that has been propagated by another algorithm which is predictive policing. So you're being asked all these proxies for poverty, proxies for race and class. Other questions are like, are you a member of a gang? Do you have mental health problems? Do you have addiction problems? A lot of this kind of information is only available because, or only held against you if you are poor and, you know, richer people, white people get treated, they don't get punished for this kind of thing. So long story short, it's a, you know, basically a test to see how poor you are and how minority you are. And then if your score is higher, which you, which it is, if you are poor and if you're black, then you get sent to prison for longer. Now, I should say, like, as toxic as that algorithm is and as obvious as it is that, that it creates negative feedback loops, one of the things that sort of the jury is still out on is like whether that is actually that different from what we, we have already. We have already sort of racist classist system, not to mention judges, and we have evidence for that. And the idea was we're going to get better, we're going to be more scientific, we're going to be more objective. It's not at all clear that kind of scoring system would do so. Nor is it clear, by the way, because there's been lots of, well, not lots, but there's been some amount of testing since my book came out about how judges actually use these scoring systems. It's not clear that they use them the way that they're intended. And there's all sorts of evidence now that judges either ignore them or they ignore them in certain cases. But listen to them other cases. Like for example, they ignore them in black court courtrooms and they, they use them in white courtrooms. So they actually like keep a lot of people and especially if they're being used for pre trial detention considerations, like they'll let white people out of incarceration pre trial, but then they're, they're going to ignore them in urban districts where they're going to keep black people incarcerated before trial. Long story short, there's also a lot of questions around how they're actually being used. But it's, it's a great example of a weapon of mouth destruction sort of just created as if it just the nature of algorithms will make things more fair. I mean, I guess going to your earlier point, no algorithm is perfect and we couldn't expect that to be perfect. But it's the reason these sort of society wide destructive feedback loops get propagated, get created by these algorithms isn't just because of their imperfect, it's because they're being used as I said in that example. But more broadly they're more like funneling people in different classes and different, for different genders or races or different mental health status or disability status. They're funneling them onto a path which they were sort of quote unquote already on, depending on their demographics.

Yeah. And I think speaking to your point of the fact that these algorithms may not be creating new biases, I mean they may as well, but that they're encoding societal biases and keeping people on a path they, they may have been on already. I think something distinct from that is that they're actually scalable as well. Right.

Yeah. Right. So we shouldn't be surprised. Of course, now that we say it out loud, like they're propagating past practices, they're automating the status quo. They're just doing what was done in the past and acting like, oh, since this happened in the past in a pattern, it's we should predict that it will happen in the future. But the way they're being actually utilized it means not just that it's we predict it will happen, but we're going to cause it to happen. If you are more likely to pay back a loan, you're more likely to get a loan. So the People who are, who are deemed less likely are going to be cut out of the system. They're not going to be offered a credit card. And since all the algorithms work in concert and similarly to each other, this becomes like a rule, and it's highly scaled, even if it's not the exact same algorithm, which it was in the case of Kyle Bean with the Kronos algorithm. Same exact algorithm being used in. But even if it isn't, the fact is data scientists do their job similarly across, across different companies in the same industry. So online credit decisioning is going to be based on similar kinds of demographic questions.

We'll jump right back into our interview with Cathy o' Neill after a short segment. Now it's time for a segment called Data Science Best Practices. I'm here with Heather Nolis, a machine learning engineer at T Mobile. Hey, Heather.

Hey, Hugo.

Heather, we have a lot of problems in data science and a huge practical one is running code on different computers, operating systems in production and in the cloud and so on, right?

Absolutely. So for me, whenever I started a new job, the worst part is that first day, few days where you just spend your time trying to get your computer to work like you'd expect a development or data science computer to work. It requires a lot of setup and you have to install the correct operating systems, get your programming languages installed, find the IDE that everybody else is using, make sure that you have the right version, then download all of your right packages, any libraries that you want, and so on. This takes a long time and it's really complicated. And for me, it's, like I said, the most painful part of starting a new job. And no matter how well somebody thinks that they've documented the process to set up a computer to run their code, they probably haven't. This becomes an even bigger problem when you're thinking about running something in production or on the cloud or on tons of other computers where you have to have multiple versions stably exist in a scalable capacity. Because how can you make sure that the setup is correct on all of those machines?

Are there any solutions to this?

There are today. I would like to tell you about one called Docker. Just imagine that you can write all of the machines for setting up your computer once in a very clear, understandable, legible language, and then have that setup happen on any machine that you want. That is essentially what Docker does for data scientists. The most useful thing is it eliminates the machine setup and allows for seamless code handoff. You're working on a model on your machine, you really need it to run on your coworker's machine. You can just wrap it all up in a Docker container and hand it over. And they don't even need to develop an R or Python or whatever. They don't need to have any language installed on their computer.

In fact, yeah, I'm a huge fan of Docker. Can you tell us a bit about how it works?

A Docker file is a set of instructions that tells the machine how to set itself up and then to run your code. When a Docker file is finished building, we call that a snapshot machine, and that's a Docker image. It's just like a picture in time of what this little miniature machine looks like. When the image is run, it creates a container which is a mini virtual machine. Containers are really cool because you can have a lot of them running on your computer and they don't interact unless you tell them to. So, for instance, when I'm writing programs in Python 2.7 and 3.6 on the same machine, I run into a lot of trouble because those libraries communicate with each other and just a huge hassle to keep them from interacting. But if I do it in containers, I can have a container written in Python 2.7 and a container written in Python 3.6 running on the same machine at the same time, and neither one of them complains. This is wonderful for production environments where people are writing in all sorts of languages, because each container is isolated, it allows for all sorts of code to be used, scaled and maintained on a single server. And so for data scientists, the important thing is if you can wrap up your models into a Docker file and no setup is needed, then you can just hand that to an engineer and your code will just run. Even cooler than this is, others can do the work of constructing a Docker image for you. So you're like, I don't know anything about Docker. I want to do this for the very first time. You can go to Docker Hub and find images that other people has already created. So you can find images for Python, images for R, images for Python 2.6 with a really specific package installed. There's probably a Docker container for fortran if you want to. Like absolutely anything that you want, somebody's probably started making it.

And do you use Docker AT T Mobile?

We do so AT T Mobile. Our goal is to create machine learning models that run constantly in production, in customer facing ways, not wanting to create R models, and then recreate them in Python we decided to use R to create all of our APIs. Docker is perfect to do this because the setup of our model is really complicated. When you go to run our models, we do deep learning, and so we need R a bunch of R packages, including Keras. Keras runs Python on the back end, and then you need a bunch of really specific Linux libraries to make sure that your Python's running correctly. And then we need a way for our API to pass the security requirements for T Mobile as a whole, because you can't just have these APIs open out in the world. And our initial Docker image for doing this topped six gigs, which caused a ton of trouble. It actually took down production clusters, but after a lot of work we've gotten it down to 1.85 gigs, which is well within the range of acceptable by our DevOps team. And we're going to release it in a few weeks on Docker Hub. So it includes all the necessary functionality and security features that you would need to run a deep learning R tensorflow Keras model in production. And it will be open sourced because we're super passionate about using R in production and we want to empower data scientists who are most comfortable in R to go ahead and make their own APIs without having to rely on a Python dev to eventually do that work for them.

Are there any potential pitfalls you'd like to warn our listeners about?

Learning Docker can be a little bit confusing at first, just because it's a totally different way of setting up a file. And if you've never used containers before, it can take a little bit of a brain switch to get used to. But after that it's really simple. And then aside from that, the other thing that you just really have to look out for is if you want your containers to be productions table they do have to be really small. By really small, I mean probably under two gigs when you go putting out 20 gig Docker containers. Because it was easy, DevOps doesn't get too happy. And with R that means eliminating a lot of the extra packages. And R is not super, super great at tracking packages. So you just have to be really thoughtful whenever you install things into your Docker image and about how it will affect the size. After that, it's pretty intuitive.

Thanks Heather, for that introduction to Docker. After that interlude, it's time to jump back into our chat with Kathy. I also think, I mean, there are a lot of different avenues we can take here, and for people who want more after this conversation, I highly recommend Kathy's book, Weapons of Map Destruction. Something I'd like to focus on is that in all of these models, you know, the value added model for teaching, the hiring model, these models to predict recidivism rate, one really important aspect of these is that they're not interpretable. We can't tell why they make the predictions they do. The fact that they're black box in that sense, and the relationship between this inability to interpret them, the inability of a teacher to go and say, why have you given me this rating? And they're pointed to the algorithm and the fact that this combined with the scalability really makes en masse lack of accountability and lack of fairness correct.

Yeah, I mean, it's exactly right. I mean, and I talked about that as in fact a characteristic of a weapon of mass destruction, that it's secret. And that's a really important part of it, because when you have something that's important and secret, like it's almost always going to be destructive. There's no, there's no, you know, a good data science model has feedback loops and it incorporates its mistakes, but there's no reason for their mistakes to be incorporated when we don't alert people to them. So that's. This sort of lack of accountability is a real problem for the model, but it's also obviously a real problem for the people who are scored incorrectly because they have no appeal, there's no due process. And to that point, there were six teachers in Houston that won a lawsuit. They were fired based on their value added model scores. They sued and won. And the judge found that their due process rights had been violated. And I'm sort of sitting around waiting for that to happen in every other example that I have mentioned, but also in lots and lots of other examples that are similar where you have this secret important decision made about you. Like, why is that?

Okay, so this is a retroactive, I suppose. I don't want to use the word solution, but a way of dealing with what has happened. I agree that action needs to be taken across the board. I'm wondering what some viable solutions are to stop this happening in future. And I love the fact that we open this conversation with you telling us that you work in consulting now, in particular in algorithmic audits, and I'm wondering if that will be a part of the solution going forward and what else we can do as a data science community to make sure that we're accountable.

I mean, yes, I mean, so there's two different approaches, and one of them is transparency, and one of Them is auditability. And honestly, I think we need to consider both very carefully. We have to think about what it means for something to be transparent. Certainly it wouldn't be very useful to hand over the source code to the teachers to tell them, oh, this is how you're being evaluated. Here are the coefficients that we have trained on this data. No, that would not be useful. So we need to understand what we mean by transparency. And I sort of worked out a kind of idea that I think is worth a try. It's kind of a sensitivity analysis. I mean, that's a technical term, but really what it looks like is, you know, hey, first confirm that the data that you have about me is correct. Next, what if, you know, what if something had changed a little bit? What if this kid had gotten a slightly better score? What if that kid hadn't been in my class? What if I had yet another kid? What if I'd been teaching a different school? What if I'd been teaching in a different classroom in my school? What if I'd had, you know, 30 kids instead of 20? How would my score change? And it's not going to prove everything. It would catch obvious errors. It would catch obvious instabilities, which actually that algorithm in particular had. So, you know, if you found out that your score would go from bad to good based on one small change, then you would know that this is a bogus algorithm. So that's one idea at the level of transparency. But I would insist on suggesting that, you know, you really don't know whether an algorithm is fair just knowing your own, how your own score works. Even if you really, really understood your own score, you wouldn't know if it's fair. Fairness is a statistical concept. It's a notion that we need to understand at an aggregate level. So I am pushing for the idea of auditing as just as important as transparency, really, to ask the questions along the lines of, for whom does this algorithm fail? Does this fail more often for black people or white people does fail more often for women than for men, et cetera. And that's a question you cannot get to just by understanding your own score or whether your own data is correct or incorrect. It's a question that has to be asked at a much higher level, which much more access. Now, to your point that I myself have an algorithmic auditing company. I do, but guess what? It doesn't have that many customers, sadly. And it's a result of the fact that algorithms essentially don't have that much scrutiny where there's not much leverage to convince somebody to audit their algorithms. I have some clients and those clients are great and I love them. They are clients who really do want to know whether their algorithm is working as intended. And they want to know either for their own sake, because money's on the line or their reputation's online, or yeah, for some third party, on behalf of some third party like the, the investors or their customers or the public at large. They want to know whether it's working. What I really started my company for though, is to understand to audit algorithms that I think are, generally speaking, the algorithms that companies don't want to have audited. If you see where I'm going with this, like it's those algorithms that are profiting from racism or profiting from bypassing the Americans with Disability Act. Those are the very algorithms that I want to be auditing. But I don't have, I don't have those clients yet. And I don't have them because we're still living in a sort of plausible deniability situation with respect to algorithms.

So it may not currently be within these companies interests to be audited. Right. So where do these incentives or where do you see them coming from? I can imagine the end game could be legislators catching up with technology. Another thing we currently have is that data scientists and the community as a whole are in relative positions of being able to make requests to their own company. So you could imagine we're having this conversation now around checklists vs oaths vs codes of conduct within the data science community as a whole. And you could imagine algorithmic audits becoming part of a checklist or an oath or a code of conduct. So I'm wondering where you see the incentives for companies in late stage capitalism coming from.

Yeah, I mean, I know there's a lot of really ethical data scientists out there and I love them all, but I don't expect their power to be sufficient to get their company that they work for to start worrying about this in general. So I think it has to come from fear, honestly. And that's either fear of federal regulators not holding my breath for that to happen, or fear of litigation so that, you know, essentially their, their compliance officer says, you have to do this or else we're taking on too much risk and we're going to get screwed. Just in his example, like Kyle Bean was applying to work at Kroger's grocery store when he got red lighted by that Kronos algorithm. So Kroger's grocery store was licensing the Kronos algorithm with a license Agreement that said they wouldn't understand the algorithm that Cronos had built, but they understood that if there was any, they had this indemnification clause, extra contract on top of their licensing agreement that said if there's any problem with this algorithm, Cronos would pay for the problem. So they would take on the risk. But you know, Kronos is not a very big company. It was working with seven huge companies just in the Atlanta, Georgia area taking on the risk, which is stupid because honestly, like the fair hiring law, the ada, the onus is on the large company, not on some small data vendor. So when, when Kyle's father, who's a lawyer, sued, he filed a class action lawsuit, seven class action lawsuits against every one of those large companies. Those large companies are on the hook for the settlement if it, if it ends up as a settlement and Kronos is going to go bankrupt very, very quickly if that ends up being settled for lots of money. So I, it's just one example, but it's I think a very important example to demonstrate the fact that the companies using these algorithms for HR or what have you. And that's often the framework, the setup is that like somebody, some third small company builds the algorithm and then licenses it to some large, either company or government agency in the case of government predictive policing or recidivism or for that matter, teacher evaluation. And they can't actually, they can't just offshore the risk because it's those large companies that are going to be on the hook for the, for the lawsuits. Right now the world is that those large companies do not, they do not see the risk, they do not acknowledge the risk. And for so far they've gotten away with it.

Your discussion of Cronos there really reminded me something that really surprised me when reading Weapons of Mass Destruction was how, I mean I knew about a lot of these cases, but about a lot of the data vendors and small companies that build these models, I'd heard of hardly any of them. That kind of shocked me with respect to how much impact they are having and can have in the future on society.

Yeah, you know, it's this kind of funny thing where we as a society are waking up and that's a very important thing. The public itself is starting to say, hey wait, algorithms aren't necessarily fair. But how do we know that? It's because we use Google Search and we, we use Facebook and we see these, I would say, consumer facing algorithms one, one by one, you know, on a daily basis. And so we see the flaws of those things. And we see the longer term sort of societal effects of being outraged by the news we see on Facebook every day. Those happen to be sort of obvious examples of problematic algorithms, but they're also, they also happen to be like some of the hardest, biggest, most complex algorithms out there. I would not know actually how to go about auditing them. I mean, let me put it this way. There, there'd be like a thousand different ways to audit them, and you'd have to sort of think really hard about each way of how to, how to set up a test. Whereas just asking whether a specific personality test or application filter, which is also used, you know, an algorithm that filters applications for jobs, whether that is legal, is a much more finite, doable question. But because of the nature of those algorithms, like when we send in an application for a job, we don't even know our application is being filtered by an algorithm. So how is the public going to find out it's wrong or their application was wrongly classified? It's completely hidden from our view. And I would say that most of the algorithms that are having strong effects on our lives, college admissions offices all use algorithms now too. Like, we don't know about them. We can't, and we can't complain if they go wrong because we just were never made aware of them. And yet those are the ones that desperately need to be audited.

And so in terms of where people can find out more about these types of algorithms and the challenges we're facing as a society, I know, for example, the recidivism work ProPublica has done a lot of great work on, on that I follow Data and Society and I now institute. But I'm wondering, do you have any suggestions for where people can read more widely about what's happening now?

I mean, the good news is that there's lots and lots of people thinking about this. The bad news is ProPublica AI now, any kind of sort of outside group, even with the best intentions, doesn't have access to, to these algorithms. You know, I mean, that's a large part of why I did not go that route. I'm not an academic. I'm not. I don't have the goal of having a sort of think tank that audits algorithms from the outside, because you can't, you literally can't audit algorithms that are HR algorithms from the outside. You have to be invited in. So that's why I started a company that theoretically anyway, could be invited in to audit an algorithm. But then the problem I still have, you know, in spite of the fact that I'm willing to sign a non disclosure agreement is that nobody wants my services because of this plausible deniability issue. Literally. There are people that I have talked to that they want my services, but then their corporate lawyers come on the phone and they say if what if you find a problem with our algorithm that we don't know how to fix and then like later on somebody sues us and in discovery it's found that we knew there was a problem with this algorithm. That's no good. We can't use your services. Goodbye.

We jump right back into our interview with Kathy after a short segment. Now it's time for a segment called Data Science Best Practices. I'm here with Ben Skranker, an independent data science consultant. Hi Ben.

Hi Hugo. It's great to be back on your show. Do you know what color a unit test is?

No, but I do know that I'm red green. Color blind.

It is a bit of a trick question, but your unit tests should be red green, green.

Listeners may recall that we've discussed unit tests in a previous segment.

Awesome. To recap, unit tests are a key software engineering tool in our arsenal to make sure code is correct. They help you catch errors immediately as you write your code, and you can check if one of your rocket scientist coworkers pushed code which broke the build. Righteous data scientists always run the unit test before pushing their changes.

So how does this magic happen?

Ben, Unit tests work by providing a framework that runs your code before the entire application is complete. The framework will call your functions or instantiate objects and then check that assertions are true, which lets you find out immediately if you coded your brilliant idea correctly when everything is still fresh in your mind and easiest to fix. A good framework will even set up the resources to run a test such as a fake database or simulated data before calling your test. Those of you who heard the VV and UQ segment on how to think about correctness of scientific models will recall that this is the verification step and how you provide proof that your code correctly implements the model.

So Ben, do you have a favorite unit test framework in Python?

Check out Unit Test and in R. Hadley Wickham's test that really the most important thing is that you use a unit test framework. Look for the one which provides the least friction for you and your team.

So what is this chromatic magic you mentioned?

Red, Green Green is one of the best philosophies for writing unit tests and part of the test driven development approach. The idea is to make the feedback loop on whether the code you wrote is correct as tight as possible. This will boost your productivity and increase the quality of your work, essentially for free.

So how does it work?

The first red means that you should write your unit test code and write a stub for the function you want to test. Then when you run the unit test, it should fail, hence the name red. This gives you confidence that your unit test will fail when the code doesn't work.

And the first green.

The first green is means that you then implement the code and get the unit test to work. At this stage, you only care about correctness. Don't forget Knuth's comment that premature optimization is the root of all evil in programming. Cool. And the final green, the final grain, means that now that your code and unit test work, you can refactor it to make it faster, cleaner, or add other improvements. Because you have a unit test, you know that your optimizations, which you should make only if necessary work correctly. Remember, only refactor in the presence of working tests. Martin Fowler may have said something to that effect.

I'm excited to try this out next time I write some tests.

Yeah, once you start writing code this way, you will have much more confidence in your system and be more productive because unit tests enable you to catch most bugs sooner while you are first implementing the code. Bugs are just easier to fix when you still have all the code stacked up in your head.

Thanks Ben, for that dive into test driven development. Time to get straight back into our chat with Kathy. So I'd like to move slightly and just think about the broader context of data science and the data revolution. And I'm wondering what other important challenges you think we're now facing with respect to the amount of data there is, data privacy, and all the work that's happening.

I mean, I'd say the biggest problem is that, you know, we live in a putatively free society and we're having a lot of problem thinking about how to deal with this in a large part because it doesn't, it doesn't give way to that many individual stories. Like I think I found a few stories like Kyle beam story, etc. They found some teachers who were fired unfairly by their value added model. But the way our policy making works in this country is like they need to find victims and people get outraged and then they complain and then the policymakers pass laws and the nature of this statistical harm is different and it's harder to measure and so it's harder to imagine laws being passed and that's the best case scenario when you live in a society that actually cares. I guess the best best case scenario might be be happening in Europe where they actually do pass laws, although I think it's much more focused on privacy and less focused on this kind of algorithmic discrimination. But in terms of what I worry about the most, I'm looking at places like China with their social credit score, which are intrinsically not trying to be fair. They are just sort of explicitly trying to nudge people or strong arm people really into behaving well and sort of their social control mechanism systems. And they are going to be very, very successful.

So what lessons do you think we can take from history to help approach this issue? And I mean in particular from, you know, previous technological revolutions such as the Industrial revolution. But there may be others.

Well, I mean, I. So there's lots of different answers to that. One of them is like, it took us a while to catch up with pollution because it was like, who in particular is harmed by a polluted river? So it's kind of an external, what is it called? Externality. And so we have externalities here which we are not keeping track of in the same kind of way. And it took us a while to actually care enough about our environment to worry about how chemicals change it. But we ended up doing that in large part because of the book Silent Spring, but other things as well. And then another example I like to give is if you think about the exciting new invention called the car. Like, people were super excited by the car, but it was also really dangerous. And over time we have kept track of car related deaths and we have like lowered them quite a bit because of inventions like the seat belts and the crash chest dummies, et cetera. And we started paying attention to like, what makes something safer. Not to say that they're totally safe, because they're not. They're still not. But we have traded the convenience for the risk. I feel like best case scenario, in our future interactions with algorithms, we're going to be doing a similar kind of trade where we're like, we need algorithms that are so efficient and convenient, but we have to be aware of that risk. And the first step of that is to sort of measure the deaths. We measured car deaths, car related deaths. We need to measure algorithmic related harm. And that goes back to the point I was making at least twice already, which is that we aren't just, we aren't not aware currently of the harm because we're, because it's invisible to us. And so when I Talk to policymakers, which I do, I beg them to not to regulate algorithms by saying, you know, here's how you have to make an algorithm, because I think that would be possibly too restrictive. But regulate algorithms and saying, tell us how this is going wrong. Measure your harm, show us who's getting harmed. That's the very first step in understanding how to make things safer.

And I think this speaks also to a greater general cultural and contextual challenge we're facing is in that as part of the political cycle, the amount of deaths incurred in a society forms a fundamental part in a lot of respects, you know, in America and other countries. But the amount of unfairness and poverty isn't necessarily something that's discussed in the same framework. Right.

Can you say that again?

Yes. So deaths are something which are immediately quantifiable and be able to brought to legislators and politicians as part of the political cycle, whereas the amount of, of poverty isn't necessarily something that is as interesting in the news cycle and the political cycle.

Yeah, that's, that's a good point. I mean, it's, it's harder to quantify inequality than it is to quantify deaths. And you know, that goes back to like the question of what does our political system respond to, if anything? I mean, right now it's just a complete show. But like, you know, even in the best of times, it responds better to, you know, stories of cruelty and death than it does to silent mistakes that nevertheless cost people real opportunities. So it's, you know, it's hard to measure what is an opportunity loss cost to like not getting a particular job or not being. Here's another one that's even relevant to current lawsuits going on with Facebook. Like not being shown an ad for a job that you might have wanted because Facebook's getting in trouble for showing ads to only to young people. So it's like an age issue or only to men, so. So women don't get to see stem related job ads. And so. And so how much harm is that for a given person? You know, it's, it's not obviously the most harmful thing that's ever happened to someone. So it's not as exciting at a policy level, but if it happens systemically, which it does, like it's a problem for society.

Yeah. And that speaks to another really important point in terms of accountability and transparency that you can be shown stuff in your online experience that I, and I'm showing something totally different and legislators are showing something entirely different. And this is something that we, this type of targeting is a relatively new phenomenon.

That's right. I mean, it's. One of the reasons it's so hard to pin down is that it's going to my earlier point, like, you get to see what you get to see. But. But that's not a statistical statement about what people get to see. Flipping that from in the other direction. It is an amazing tool for predatory. For predatory actions like payday lending or for profit colleges. It's like they can't believe how lucky they got. They used to have a lot of trouble locating their victims, desperate poor people. But now it's like they couldn't be happier because they've got this. This system and it's called the Internet, that finds them for. For them and cheaply and en masse and scaling and in a way that's exceedingly easy to scale. So it's. It's a fantasy come true for. For those kinds of bad actors. But then the question becomes, how do we even keep track of that if they are actually going after those people that are, in sort of a very real way, voiceless and don't have the political capital to make their problems a priority?

So I've got one final question for you, Cathy. You know, we're in the business of data science education here at datacamp. Because of that, a lot of our listeners will be the data analysts and data scientists of the future. And I'd like to know what you'd like to see them do in their practice. That isn't happening yet.

I just wrote a paper. It's not out yet, but it will be out pretty soon, about ethics and artificial intelligence with a philosopher named Hannah Gunn. It's called the Ethical Matrix. And actually, no, no, it's called. But I think it's something along the lines of the ethical matrix. And at least it introduces this concept of an ethical matrix. And it's a very simple idea. The idea is to broaden our definition of what it means for an algorithm to work. So when you ask somebody, does this algorithm work? They always say, yes. And then you say, what do you mean? And then you say, oh, it's quite efficient. And so, you know, you're like, oh, but beyond that, does it work? And you know, that's when it becomes like, what do you mean is that. And then even if they want to go there, they're like, is that an infinitely complicated question that I don't know how to attack? So the idea of this ethical matrix is a sort of. To give a rubric to address this question. And it's Something that I claim we should do before we start building an algorithm that we should do with everyone who is involved. And so like to that point, the first step in building an ethical matrix is to understand who the stakeholders are and to get those stakeholders involved in the construction of the, of the matrix and to embed the values of the stakeholders, you know, in a balanced way relative to their concerns. So the rows are the stakeholders, the columns are the concerns, and then youth goes through each cell of the matrix and try to decide are these stakeholders at high risk for this concern to go very, very wrong? It's basically as simple as that. But our theory is that if this is, if this becomes part of the yoga of building a data driven algorithm, then it will theoretically at least help us consider much more broadly what it means for an algorithm to work, what it means for it to have long term negative consequences, things to monitor for making sure that they're not going wrong, et cetera. And it will bring us, from the, from the narrow point of view, it's working because it's working for me and I'm making money, which is like I call the one by one ethical matrix. The stakeholder is me or my company and the only concern is profit. Broaden that out to look at all the people that we're affecting, look at all this, including maybe the environment. Look at all of the concerns they might have, fairness, transparency, you know, false positives, false negatives, and consider all of these and balance their concerns explicitly.

Well, I for one, I'm really excited about reading this paper and we'll include a link in the show, notes to it as well.

Great.

When it's out.

When it's out, yeah. Great.

Fantastic. Look, Kathy, thank you so much for coming on the show. I've enjoyed this conversation so much.

Great, thank you for having me. Thanks. Hugo.

Thanks for joining our conversation with Cathy o' Neill about weapons of math destruction, the harm that algorithms can do, and what we can do as a community and society at large to begin to correct these issues. We saw that the three defining characteristics of WMDs are that they're important, they're secret and they're harmful. They're generally highly scalable black box models that affect real lives on the ground, whether it be teachers being fired for the output of the value added model, job applicants being asked proxies for mental health assessment questions in the interview process, or recidivism models that are biased against underrepresented groups being fed into parole hearings. We saw that potential solutions are offered by a future in which algorithms are both more transparent and auditable. Cathy herself works in the algorithmic auditing space, and we discuss several key techniques, such as a sensitivity analysis which figures out how sensitive any given algorithm is to its inputs. And Kathy gave the example of the teaching value added model being overly sensitive in that if you make small changes to a teacher's data, it can alter the output wildly. We also discussed the concept and practice of an ethical matrix, which lays out all the stakeholders and the different ways the algorithm in question can impact them. As Cathy said, if the ethical matrix becomes part of the yoga of building a data driven algorithm, then it will theoretically at least help us consider much more broadly what it means for an algorithm to work, what it means for it to have long term negative consequences. And my dear listeners, remember data science doesn't predict the future, it causes the future. Thank you all for tuning in. All year long we have had so much fun producing these 50 episodes and cannot wait to be back on the proverbial airwaves in 2019. Don't forget to take the survey in the show notes and to make any suggestions for future episodes. Thank you all once again my listeners. And thank you to all my fantastic guests who've been on the show this year. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com community podcast.