#45 Decision Intelligence and Data Science (with Cassie Kozyrkov)

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Cassie Koziakov, Chief Decision Scientist at Google Cloud. Cassie and I will be talking about data science, decision making and decision intelligence, which Cassie thinks of as data science augmented with the social and managerial sciences. We'll talk about the different and evolving models for how the fruits of data science work can be used to inform robust decision making, along with the pros and cons of all the models for embedding data scientists in organizations relative to the decision function. We'll tackle head on why so many organizations fail at using data to robustly inform decision making, along with best practices for working with data, such as not verifying your results on the data that inspired your models. As Cassie says, split your damn data. I'm Hugo Bown Anderson, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter hugobown and datacamp@datacamp. You can find all our episodes and show notes@datacamp.com community podcast. Hi there, Cassie, and welcome to Data Framed.

Hi. Thanks, Hugo. Honored to be here.

It's a great pleasure to have you on the show and I'm really excited to have you here to talk about data science in general, decision making, decision science, and decision intelligence. But before that, I'd like to find out a bit about you. First, I want to know what your colleagues would say that you do.

Oh, goodness. Well, it rather depends on the colleague, I think. But I think the consensus would be that they'd say that I have some expertise in applied data science especially, and I help Google teams and our cloud customers apply machine learning effectively.

Great. Could you tell me a bit about what applied data science means to you?

Yeah. So when it comes to data science, let's start with what data science means and then we'll dig deeper. So data science to me is that umbrella discipline that has underneath it statistical inference, machine learning and analytics, or data mining. And the difference between these three for me boils down not to the algorithms that are used in them, because if you're smart, you can use any algorithm for any of them. Also not down to the tools, but really boils down to the number of decisions under uncertainty that you want to make with them. With data mining, you really want to get inspired. There are no specific decisions that you want yet, but you want to see what your Data inspires you to start thinking and dreaming about statistical inference is one or a few really important decisions under uncertainty. And then machine learning and AI, they boil down to a recipe for repeated decision making. So many, many decisions under uncertainty. I actually see data science as a discipline of decision making, turning information into action. Now, where is this applied versus research side of things? Researchers focus more on enabling fundamental tools that other people will use to solve business problems, whereas applied folks will go and find among the available tools what they need to solve those problems. So I'm not focused on how do I develop a new neural network architecture. I'm focused more on there seems to be this germ of an idea in a business leader. How do we bring that out, make it real, build the team that's going to end up working on that and then ensuring that the entire process from start to finish is well thought out and executed, and then at the end there is a safe, reliable result.

And I think framing this form of data science and applied data science as a sub discipline of decision making is something we're going to unpack more and more in this conversation. So that really sets the scene very nicely. So you said that your colleagues would view you as an expert on applied data science and thinking about how to use machine learning effectively. Now, what do you actually do? Are they pretty on track with what you do?

They're close, but I think at the heart of what I care about is this notion of type 3 error in statistics. For those that don't remember your errors, let's have a quick reminder. Type 1 is incorrectly rejecting a null hypothesis. Type 2 is incorrectly failing to reject a null hypothesis. And type three is correctly rejecting the wrong null hypothesis. Or if you prefer a Bayesian statement on the same thing, it's all the right math to solve entirely the wrong problem.

Great. So could you give me an example of a type three error?

Yeah. So it's any rabbit hole that a data scientist goes down meticulously, carefully answering a question that just didn't need to be answered. So maybe this will be a familiar thing to some of the data scientists listening. And I hope you don't suffer from this too much, because that's a bit of a newbie gotcha. But it kind of goes like this. There you are finished with most of your work for the week. It's maybe 4pm on a Friday, and you're excited for a nice free weekend, because that's the whole point of, you know, being in industry and not being in academia anymore, Right? Just kidding.

Anyways, you're not kidding at all.

Okay.

I'm kidding some.

Sure.

A bit, yeah. All right, great.

Go on.

Fine. I'm not kidding at all. So there you are, and you're just ready to go home and say, a product manager comes up to you and with this sense of urgency in the voice, wants to get a specific measurement from you or a specific question answered. And you think to yourself, my goodness, but that is difficult. That's going to take me at least all weekend and, you know, well into the nights as well. I'm going to first have to figure out getting the data. I'm going to have to sync up with data engineers, I'm going to have to look up all these methods in the textbook. This is going to be a difficult thing. But look, I'm a great data scientist and I can do this and I can do this correctly and I can make sure that all the statistical assumptions are met. And come Monday morning, I'm going to deliver this thing perfectly. And so on Monday morning, you come on bloodied knees and you lift this result up to that product manager and they kind of poke their head, look at you and go, oh, I didn't even realize that that was what I was asking for. So there you were, meticulously, very correctly solving this problem, but rather uselessly as well. And it goes nowhere. The product manager doesn't use it for anything, just gets abandoned behind the sofa of lost results. So that's a type three error.

And so how do you stop that happening, presumably on the Friday afternoon? Presumably involves communication. Right?

Communication is one of those things, but process as well. So the data science team should know what the other stakeholders that they depend on are responsible for and what it looks like for those pieces of work to be completed correctly. So I like to talk about this wide versus deep approach to data science. So a rigorous approach versus a more shallow inspiration gathering approach. And the second one is always good as long as you don't end up wasting your data on it, and hopefully you'll prod me about that shortly. But as long as you have data allocated to inspiration, having a light, gentle look at it is always a good idea. Putting your eyes on that data and seeing what it inspires you to think about it helps you frame your ideas.

And so in that case, we're thinking about some sort of rapid prototyping in order to.

We're thinking about something even more basic. We're thinking about just plotting the thing. And that is separate from a very careful pursuit, rigorously of a specific and important goal. So first, separating these two and saying the former, that broad, wide, shallow approach that is always, that's indicated for every patient, let's just put it like that doctor prescribes that always, as long as you have the data to spare for it, do it. But don't take your results too seriously and don't do anything too meticulous. On the other hand, this more rigorous approach, this takes a lot of effort and it takes a lot of effort, not just from the data science team and the context for that, how the question is being asked, what assumptions are palatable and so forth, that is actually the responsibility of the decision maker, the business leader. And they have to have done their part properly in order for that meticulous work to make sense. So if you're going to go and do rigorous things, you need to make sure that that work was properly framed for you.

Right. So in this case of the data scientists going and spending their weekend working on this problem, doing essentially work that the product manager didn't think they'd be doing, a way to solve that would have been doing some sort of rapid data viz, exploratory data analysis and then having a conversation with a product manager about what they really wanted.

I would say actually the other way around, a conversation with the product manager about what they really wanted first. And if what they want is something emotional to get a feel for, something that needs to be unpacked a little more. And perhaps what they're looking for is possible, perhaps it isn't. Perhaps taking a look at the data generates the inspiration that is required for what they want. Perhaps they're hoping that the data scientist is a genie in a magic lamp that grants wishes that can't be granted. So talking to them and figuring out what they want should actually be the first step. But better even than that would be an organization where there isn't this adversarial relationship that assumes that the product manager doesn't know their part. Better to staff the project with trained, skilled decision makers who know how to do their part. And the data scientist will simply check the requests coming in. And if the request has a certain characteristic, they will tend to go for no work or light work. And if the request has a different kind of characteristic, they will go and do things carefully, rigorously and meticulously at the level of rigor and complexity requested by that skilled business leader.

I love it. So then we're actually talking about specifically with clarity, defining roles and defining a process around the work.

Absolutely. So this can get pretty big and interesting of how you arrange these teams and how you arrange these processes. In its lightest form, it can be a matter of who talks to whom in what order, but it can be much bigger than that.

Great. And this is something we're going to delve into later in the conversation is kind of common organizational models for this type of work. But before that, I want to prod you a bit. And something you mentioned earlier was the idea of wasting your data. And maybe you could tell me what you mean by that.

Sure. Well, we all learned something rather straightforward and obvious sounding in statistics and data science class, and then unfortunately, we end up forgetting it a little bit. And it's something that we really shouldn't forget, and that is that a data point can be used for inspiration or rigor, but not both. If you're dealing with uncertainty, if you want to go beyond your data, because when you're assessing whether or not your opinion actually holds up in reality and in general, then you need to make sure that you check that opinion on something that you didn't use to form the opinion. Because we humans are the sorts of creatures that find Elvis's face in a piece of toast. And if we use the same piece of toast to get inspired to wonder whether toasts look like Elvis, and then to also answer whether toast does in general look like Elvis, we have a problem. You're going to need to go to a different piece of toast. And so you can use data for inspiration or rigor, but not both. And so if you use all the data that you have for getting inspired, for figuring out what questions you even want to ask, then you have no data left over to rigorously answer them.

And I think similar comparisons can be made to null hypothesis significant testing. Right. So, for example, you'll do exploratory data analysis, start to notice something, and then do a test there. But because you were inspired in your null hypothesis and alternative hypothesis by the original data, you may actually be overfitting your model of the world to that data set.

Yeah. And I think that that kind of thing actually happens in practice in the real world because of the way in which students are taught in class. So it makes sense in class for you to get a look at the conditions of a toy data set, see what sorts of assumptions might or might not hold in that data set, and then see what it looks like when you apply a particular method to that data set. And that poor toy data set would have been torn to shreds with respect to what you could actually reasonably learn from it by all the thousands of Times that the student and professor are torturing this poor little data set. But that's okay, because all you're supposed to do in class is see how the math interacts with the data. But you get used to this idea that you're first allowed to look and examine this data set and then you're allowed to apply the algorithm or statistical test to it. In real life, though, you end up running into exactly this problem where you invalidate your own conclusions by going through that process. You really should not use the same data set for both purposes. You shouldn't pick your statistical hypothesis and test it right then and there. I mean, think about it like this. Here you are with X variable and Y variable, nice scatter plot, and you take this little data set and you plot it and you see sort of the ghost of an X upward, up and to the right lift in this little point cloud that you've just plotted. Well, you've just seen this. And so you ask yourself, well, maybe I can put a straight line through it and see whether I statistically significantly have a positive correlation there. Well, congratulations, you are going to get the result that yes, you do statistically significantly have a positive correlation. On account of having been inspired to ask the question in the first place by how these particular points fell onto your scatter plot. The conclusion you make might be entirely unrelated to reality. If you're inspired to do that by this data set, go get another data set from the same process in physical reality and make sure that your inspiration holds up there. We humans, we do see patterns that are convenient, interesting to whatever we're interested in and might touch reality at no point at all.

There are several ways that we think about battling this endemic issue. One that you mentioned, of course, is after noticing things in an exploratory analysis of your data set and coming up with hypotheses, then going and gathering more data generated by the same processes. Another, of course, is pre registering techniques. Before looking at any data initially, I was wondering if there are any other ways that you've thought about or you think maybe worth discussing to help with this challenge.

The problem really is the psychological element of data analysis. How you're looking for things, how your mind tricks you as you look for things, and the mathematical techniques that are supposed to help you do things like validate under extreme circumstances with cross validation. Those are really easy to break. They don't actually protect you from the going about things the wrong way psychologically. So what I suggest people do when they start thinking about this is if you were Pitted against a data scientist who absolutely wants to lead you astray and trick you into all kinds of things. When you give them certain constraints on their process, can they still give you a bad result? Can they still mess with you? Can they still trick you? And most of those methods out there, in fact, I can't think of one off the top of my head that wouldn't be. Most of them are susceptible to that kind of mucking about. And unfortunately, as a good data scientist, you are likely to trick yourself in that same manner.

That's very interesting because I think it hints at the fact that we actually, due to a lot of our cognitive and psychological biases, we don't necessarily have good techniques. We need to develop processes, but we don't necessarily have good techniques to deal with this. Yet.

When you talk about pre registration of a study, that is less of a technique and more of a statement that in these data you're not going to go and adjust your perspective and question. So you're saying wherever your hypothesis comes from in advance of gathering and processing these data, it is now fixed. That's kind of how it should be anyway. So even when you ask the question and separate the two, you're actually talking about two aspects of the same thing. If you want to form a hypothesis, go and explore data, but nuke that data set from orbit. If you're going to go and do some rigorous process where you have the intention of taking yourself seriously, you should have your entire question, all the assumptions, all the code, even submitted before, ideally before the data is collected, but certainly before the data hits that code.

We'll jump right back into our interview with Kassy Kozikov after a short segment. And now we've got another segment with Justin Boyce. He's been telling us stories about probability distributions and you may remember that last time he promised to come back and tell us about the exponential distribution. Justin, tell us the exponential story.

This is one of my favorite stories. But just like each episode of Game of Thrones reviews what happened in previous episodes, I need to set this story up with a little review.

Of course. Please do.

We talked last time about Poisson processes. The story behind a Poisson process is set at a bus stop in the quirky town of Poissonville. The time I have to wait at the bus stop for the arrival of the next bus is completely independent of when the previous bus arrived. Processes that arrive with this kind of memoryless process, that is that the past has no bearing on the present or future, are called Poisson processes.

So the exponential distribution describes something about Poisson processes.

Exactly. Here's the story. The time between arrivals of a Poisson process is exponentially distributed. So the amount of time I have to wait for a bus at the bus stop in Poissonville is exponentially distributed.

Right. And I think New York sometimes resembles Poissonville.

Yeah, I bet. Now the exponential distribution is parameterized by a single parameter called the rate, which describes how rapidly Poisson processes arrive.

Now I know students sometimes stumble over some of the names of these distributions. Can you tell us how the exponential distribution is different from the Poisson distribution?

Sure. I often have students confuse the exponential and Poisson distributions in my classes as well. The Poisson distribution has this story. The number of arrivals of a Poisson process in a given time interval is Poisson distributed. So this is a discrete distribution describing the integer number of arrivals. Conversely, the exponential distribution describes the time between arrivals of a Poisson process. Time is a continuous quantity, meaning that it can take any non negative real number in this case.

Thanks for clarifying that, Justin. Can you give us an example of a Poisson process and describe what is Poisson distributed and what is exponentially distributed there?

Well, one of my favorite examples is no hitters in Major League Baseball. A no hitter is when a team allows no hits by the opponent in a baseball game. And we can model this as a Poisson process. It is very rare. To date, there have only been 299 of them in the history of the Major league. So here's a quiz. What distribution would describe the number of no hitters in a season?

Ah, so you're making me do story matching now. Great. So the number of Poisson processes that arrive in a given amount of time is Poisson distributed. No hitters are Poisson processes, and the given amount of time is a season. So the number of no hitters in a season is Poisson distributed.

Very well explained. I would expect nothing less from the host of my favorite podcast. Now, the time between no hitters. Now that's exponentially distributed. And we see everything from two no hitters on the same day that happened twice to the longest gap of 6,364 games.

Wow. It happened twice that no hitters happened right after each other. How does that work?

It turns out that the most probable waiting time for the arrival of a Poisson process is zero. So it's not surprising. But the exponential distribution has a pretty heavy tail. So even though zero is the most probable, most of the probability mass is to the right of zero. The mean is actually given by the inverse of the rate.

Now that's an interesting distribution.

It is. Its probability density function is a simple exponential decay away from zero. It is particularly useful in analyzing a data set of times to see if they come from a Poisson process or not. For example, you might look at the time between hits on a website. If they follow an exponential distribution, they could have arisen from a single Poisson process. But if they do not, then they definitely did not arise from a single Poisson process.

We could go on about this distribution all day, I'm sure. How about instead we talk about another distribution next time and I jump back into our interview.

Sounds great. See you again soon.

Awesome. After that interlude, it's time to jump back into our chat with Cassie. So I'd like to move now and talk about decision intelligence, which you yourself are chief decision scientist at Google Cloud and you work in decision intelligence. And I was wondering if you could frame for us what decision intelligence actually is and how it differs from data science as a whole.

So I like to think of decision intelligence as data science plus plus, augmented with the social and managerial sciences and with a focus on solving real business problems and turning information into action. So it's oriented on decision making. If we had to start again with designing a discipline like that, we would want to ask every science out there, what does that science have to say about how do we turn information into action? How do we actually do it? Because of the sort of animal that we are. And if we want to build a reliable system or reliable result for a specific goal, how do we do that in a way that actually reaches that goal rather than takes a nasty detour along the way? So it's very process oriented, it's very decision oriented. But of course a large chunk of that is applied data science.

So can you tell me why data science plus plus, why do we have two pluses there?

Ah, plus plus as in upgrade it with the next thing. I suppose in the. In the same way as in syntax, you would have. I. There's a. There's just some cuteness there, I suppose. But think of the upgrade like this. A data scientist is taught how to analyze survey data and how to think through a lot of the careful mathematical stuff, how to deal with what happens if that data are continuous, what if they're categorical, what if the person was using some sliding scale, etc. How many questions, how do we correct for this many questions, that sort of thing. But what they're not taught, not directly in their training, is how do you construct that Survey, how do you make sure that that survey minimizes, say, response bias, which is where the user or participant simply lies to you and gives you the wrong answer to your question? And how do you think about what the original purpose of that survey is? I mean, why are we doing this in the first place? And was a survey the right way to go about it? And how did we decide what was worth measuring? Those things are not typically taught to data scientists. So if data scientists want their work to be useful, then someone, whether it's themselves or a teammate, who has the skills to think through that stuff has to be participating.

Great. And is it important? Or is best case scenario, the data scientists being involved in every step of the process, from data collection, experimental design, question design, through to actual decision making?

Well, look, that depends on what your budget is, right? If you have infinite money, perhaps you might be able to hire one of the very, very rare unicorns who've actually thought about all of it and who are skilled in all of it. There are not that many of those people. And if you intend to hire them, you're going to have to pay them. So intending to staff your projects in that manner, well, no wonder you're going to be complaining about a talent shortage. So the reality of it is that you're going to have to work with interdisciplinary teams. And also, even if you have someone who gets all of it in a large scale project, there's still more work than someone can do with the number of hours that they are in the day. And so why do you really need all these identical copies of the perfectly knowledgeable worker if they're going to have to work on different parts of the process in any case? So the data scientist upscaling to perfection and then owning all of it. It's a nice dream, but it doesn't sound very practical. Instead, I imagine that they would be best suited to the part that they have spent the most time learning. And what they should really worry about more is how the baton is passed from colleagues who are responsible for other parts of the process and having the skills to check that that part was done well enough for their own work to be worthwhile. Because unfortunately, data science is right in the middle of the process and it relies on the bookends. And if the bookends, like that decision making side and that product leadership side and that social science side, if that wasn't done correctly, or if downstream you have no way to put this into production reliably, and even if the prototype has beautiful math, it will be too much of a mess to actually use in practice, then there is no point to the data scientists work. It all becomes a type 3 error. So they're going to be working with a very interdisciplinary team probably, and they should focus on the parts where they can have the best impact.

So in terms of decision making, I want to know about these teams. I love that your response to my previous question was the reality of the situation is I want to know more about reality and I want to know more about the practical nature of how data scientists and their work are included or embedded in decision making processes. So could you tell me a bit about the most common organizational models for how data scientists are included in this?

Yeah, sure. An obvious way to do it is to collect a whole lot of data scientists and put them together into a centralized data science team. And that tends to be guarded jealously by their data science director, who buffers them from the most egregious type 3 error requests and makes sure that the rest of the organization uses them to a good purpose, or at least to the most impactful business purpose. And the junior data scientists in that structure, they don't need to navigate the politics. There's another model which is simply embedding a data scientist in a large engineering team and telling them to be useful. And there's the decision support model. This is where you append the data scientist to a leader and the data scientist helps that leader make decisions. And then of course there is the data scientist owning most of the process, especially the decision making. So here data science is responsible for framing decision context, figuring out which questions are even worth asking and then also taking ownership of answering them.

So we have the pure data science team, the embedded in engineering, decision support and data scientists as decision maker. And I think you know, the fifth.

Will be the decision intelligence option, which is none of these.

I look forward to discussing that. And it seems like generally this order goes from less decision making to to more decision making on the part of the data scientist. Is that fair to say?

Fair enough.

And so what are the pros and cons of being at different points along.

This spectrum with a super centralized one? An obvious con is that if you are a small scrappy organization, forget it, you're not going to be able to have this large data science org. Another con is that they tend to be put towards what the business already knows is worth doing properly and carefully. So in some sense this is a pro. They're going to be associated with the most delicate or high value questions in the business. The con is that there's less flexibility to help out the broader organization to seize unusual opportunities because there is this sort of single point through which all the requests come, and that tends to homogenize the requests a little bit. And that also means that individual data scientists will have very low contact with the decision function. Now, that might be a pro for them. Maybe that's a stressful thing for a junior data scientist, but it's very hard for their work and their contribution to have visibility in this way. And all of this is really at the mercy of data science leadership. So if their data science director does not know what they're doing, we'll have a problem. And the industry is really suffering from a shortage of data science leaders. There are people who call themselves data science leaders or analytics managers, but these folks might not really know how to play the organizational politics. They might not have a good business sense, or maybe they are primarily leaders. They have all those that knows for impact, but they don't understand how to make a data science team effective. So there can be some problems with that. The embedded in engineering pro is that you get to influence engineering. However, you end up doing a variety of tasks which may or may not have anything to do with data science. Quite often the engineering team doesn't really know what sort of animal you are and doesn't really know what you're for, doesn't know if you're useful. They think of you as a sort of not so good programmer. What's wrong with you? And what is this stuff that you're constantly fussing about on whiteboards? You might not be seen as very useful and you might find yourself taking on product management tasks that you might not want to do, that you didn't think you were going to have to do and that you didn't train for. So you end up with non specialist tasks and there's no buffer against politics for you there.

And is this something that also happens as we move more towards data scientists who work in decision support and as decision makers themselves?

There's some element of this as well. With decision support, a good leader quickly figures out how to make you useful. So you don't spend an awful lot of time sort of wafting about figuring out how to even contribute in the first place. Now it might be that your best contribution is nothing to do with complex methodology that you spent so many years in grad school studying. And your data science tasks might end up getting diluted with a whole host of other things you might be working on. But your value does tend to get better protected under this setup.

And how about for a data scientist as an actual decision maker?

So of course the pro there is that you don't have this loss in moving between the data science, engineering and decision functions because the data scientist owns all of those things. The con is that in order to do it, you need to really get several black belts. And if you don't have them, you might think you're being useful, but you might be doing more damage than good. So maybe you think you're good at understanding business impact, but really what you're much better at is doing the math and you end up pushing the organization down rabbit holes, bad rabbit holes, at a much worse rate than they would have had without you. So you really do need these multiple black belts and you need to understand that you have to train for these things separately because the standard training program just does not prepare you to be a two in one or three in one worker. So in practice, this is a rare animal.

And then of course, the fifth model that you mentioned in passing that I'd like to focus on now is data scientist as decision intelligence operative. What happens here?

So there will be some allocation of time and human resources towards the analytics or data mining side of data science. And so there will be an ongoing pulse check for the company. So there will just be this broad light touch analytics going on all the time. And whoever is best at that model of working under data science will be doing that and will be partially driven by what leadership wants, but will also be driven by the explore over exploit attitude. Then if something else is going to be requested, there will be certain stages in the project life cycle that have to be completed in order to get that work. So it's sort of like a combination of those two models where you get embedded in engineering or you get embedded with decision making, but that match happens out of a centralized pool of labor and it happens based on the project being framed in the required way. So for example, you might have the decision support framing where you need statistical assistance on a project. In order for that to happen, there has to be certain steps, like a selection if you're going to go the frequentist way, a selection of the default action, what the decision maker actually wants to do by default, an understanding of what it takes to convince them, what their metrics are that sort of passes a social science function there, what population they're thinking about, what assumptions they're willing to deal with their that will be someone from social science or from data science working with the decision maker to help them frame their decision Context. And then once that is all ready, then you staff folks who can actually do the heavy lifting, the calculations, the data staff to the project. And of course you need to staff data engineering to that project as well. So when everyone comes together, they know what they are there for.

And this actually speaks to kind of a broader challenge. I mean, we've discussed this previously, but this idea that a lot of people want to hire data scientists or do machine learning or state of the art deep learning or AI before they even know what questions they want to answer. Right?

Yeah. So what you should do, here's my advice for everybody. If you don't know what you want, think of your data as a big old pile of photographs in the attic and think of analytics or data mining as the person or function that is going to go to that attic. And their opportunity to actually go to the attic and look at the data is going to be supported by data engineering. They're going to go to that attic, they're going to append this big box of photographs on the floor, they're going to look at them and then they are going to summarize what they see there to the people waiting patiently in the house and ask those folks whether they are considering doing anything more with it. That kind of approach always makes sense. You never know what's in this pile of photographs and you never know whether it's worth doing anything serious with it. But also because it's a pile of photographs and you don't know who took it and for what purpose, you should never learn anything beyond what is there. So we citizens, we already know how to think about a pile of photographs or a photo you find on the side of the road. The only thing that you can reasonably say about it is, hey, this is what is here. Does that inspire me? Does it make me dream? Does it make me want to ask other questions about the world? Sure, perhaps. But do I take any of that seriously? No, of course not. It's some photograph. And data science is essentially Photoshop, as we all know. And we don't know much about how that photo was taken or why, we can't really make serious decisions based on it. But taking a look always makes sense as long as you continue to think about it reasonably in the same way as you would think about those photos. So that's always good for every project. And if any team, any organization says, I'd like to know a little more about my data, I'd like to get into mining my data, looking at my data, finding out what's in there, that is always a good thing. But now if you don't actually control the quality of that data, you might end up doing very careful, rigorous things with it. And the photographs were all, I don't know, blank, Right? There's no point to it. Or maybe they were all taken in a way that's entirely unreliable for the question that you want to answer because you didn't actually plan the data collection. So if you look at the photographs that I take in my travels, you notice there's all these super touristy landmarks, and yet somehow I'm the only person pictured in the photograph at that landmark. You can't conclude anything about how many people go to these landmarks based on my pile of photographs. But you can still take a look as long as you don't take them too seriously. And then you might start thinking about the sorts of things you might want to do with them. And when you start figuring out what you might like to do, then you start planning the entire process as directed towards that goal. And then it makes sense to start thinking about hiring people who can do that extra stuff.

We'll jump right back into our interview with Cassie after a short segment. Welcome back to Machines that Multitask with Frederica from Cloudera Fast Forward Labs. Frederica, you built a prototype newsi that is powered by a multitask neural network. How would one build such a model?

So newsi puts current news into perspective. It relies on a multitask neural network that predicts news article category for sensationalist tabloid articles. Task one and more buttoned up broadsheet articles Task two. Users can take any article they're interested in and analyze it as a broadsheet or tabloid article to gain insight into current trends and other biases in news reporting. So we decided to build this multitask model in Pytorch, which is really an excellent deep learning framework developed by Facebook. Pytorch makes it easy to add neural network layers to a model, from simple linear layers to more specialized ones like convolutional or recurrent layers. So turning a single task into a multitask neural network really isn't all that hard. A single task neural network for classification has one output node. A multitask neural network has multiple output nodes, one for each task. These output nodes connect to the inner layers of the neural network, the ones that process and transform the input data to extract useful features for prediction. The multiple output nodes of a neural network can share these inner layers. So each output node receives activation from all inner layers. In this case the inner layers are likely to extract features that are useful across all tasks. Alternatively, you can connect some output nodes to only some of these inner layers. These inner layers then specialized in serving information to that particular output node only. In other words, they can specialize in one task, the one associated with that output node. Basically, we can use neural network architecture. The way we connect layers to nodes to nudge some parts of the network to extract features from data that serve only one task, and to nudge other of the network to extract features useful across all tasks. This flexibility allows us to extract both task agnostic features, the one that serve all tasks, and task specific features, the ones that serve only one task.

Great, so why would we want the network to learn both task specific and task agnostic features? What's the benefit there?

Yeah, so some tasks are related, but only loosely related. In which case you want to learn some shared or task agnostic features and some task specific features for better model performance. What's great is that given the right network architecture, you don't even need to worry about which features should be shared. The network will figure it out by itself during training. As a model learns to master supervised tasks, it figures out in unsupervised fashion the relationships between tasks. Which features can and should be shared and which ones cannot and should not. What's more, once you've got a trained model, you can use the weights of this model to initialize another model. That way you can make use of the knowledge you've gained in the past to benefit you in the present and future. Task agnostic representations are especially promising since they're likely to generalize better. We've abstracted out task specific information in the end, in fact, we put quite some effort into obtaining clean task agnostic representation.

That's interesting. How did you do that?

Yeah, so we used adversarial learning, popularized by generative adversarial networks or GANs. People have used GANS to generate realistic looking headshots of celebrities that don't actually exist. Gans learn the color distribution of pixels from the headshots of real celebrities and then use that distribution to generate fake ones. GANS basically consist of two neural networks that compete against each other. One, the so called generator, takes noise and learns to distort it. The output of the generator is then passed on to a so called discriminator, which in addition to the distorted noise, also sees the some real images. The discriminator learns to distinguish real from fake images. The generator tries to deceive the discriminator as they both learn to get better at their respective tasks. In the end you have a generator that's pretty good at generating fake images. So in our model we only used the discriminator part. It received activations only from the task agnostic representations of the model. The discriminator had to predict whether the model was currently processing an article from a broadsheet or which is task one or tabloid our task two. If it could do a good job then clearly there was task specific information in the task agnostic layer. We punished the model for good discriminated performance. So over time the model learned to remove task specific information from the shared layer. So to me that's actually what's really exciting about applied machine learning. Research innovations are popularized in one context, adversarial learning and gans, but have much broader applicability. So we basically used adversarial learning as a regular browser for clean task agnostic representations.

Thanks Frederica for that journey into the workings of how to build a multitask learning model. Time to get straight back into our chat with Cassy So Cassie, given the variety of different models for embedding data scientists in the decision making process, I'm wondering why so many orgs fail fail at using data science to inform decision making properly and robustly.

Well, this comes down to a problem of turning information into action and how decision makers are organized and trained to do that. So it may be a case of the decision maker actually doesn't know what their own role in the process is, and they don't know how to properly frame a decision context for a data science project that isn't simply data mining and analytics. This wide and shallow approach without the decision maker taking control of the process, what always makes sense is a nice shallow wide data mining approach. Mine everything for inspiration. Don't take yourself too seriously, don't spend a whole lot of effort, and if you just stick to this and you really truly don't take yourself more seriously than you're supposed to, the biggest danger is overspending on personnel. Maybe you've ended up hiring a bunch of professors and now you've used them for tasks that they consider to be far too easy given their intense training. But what tends to happen is that decision makers don't end up owning the dive into the careful, rigorous stuff properly. So maybe they just hire a bunch of data scientists and they leave them in a room all on their own, they don't give them any instructions and then they are surprised when the only thing that comes out of that room is research white papers. Maybe there's a case of all those folks pursuing research and rigor for its own sake, because that's the most comfortable thing, their comfort mode from their research training. And those folks aren't really qualified to diagnose what is useful for the business and the decision function just leaves them alone. It might be a case of the entire organization not understanding that there's a difference between inspiration and rigor and how to use data for these things and how much effort each one takes. So another failure is where you get the opposite. You end up using data for inspiration, and then you think that you've done something rigorous there where you really haven't, and you start taking those results much more seriously than you ought to. You become overconfident and you run headlong into a wall. Another problem that organizations have is that it's very convenient to use the outputs of data science work as a way to bludgeon your fellow decision makers in a meeting. So everyone wants to argue and put forth their personal opinion about a thing that cannot be solved with data. Really, it has to do maybe with the strategy of the organization. And instead of sticking with humility, owning what you don't know, and using argument to discuss with your fellow decision makers what should be done next, you bring some inscrutable report that's covered in equations and you say, because my magical data scientists have said this is the truth. But you know about statistical inference, you know that the question matters more than the answer does almost. And if all you do is you bring an answer, well, it may or may not be an answer to the question that is being asked or assumed by everybody else. It's like that Douglas Adams thing where you just bring 42 to the meeting and you say, look at all these arcane equations that have gotten us to 42. And because it says 42, I'm correct, actually doesn't make much sense. Takes a lot of effort and wastes a lot of time. And then there's also an element of misguided and misdirected delegation of decision responsibilities. That's where you have someone who wants to have decision responsibility and they want decision making to be done rigorously, but they want that for more decisions than they actually have time to deal with. And so they sort of fool themselves into thinking that they can be in that decision maker role without spending the time to actually frame decision context, think through assumptions, work with the data science teams and so forth. And so what ends up happening is that people junior to them end up usurping those roles and make the decision however they make it. Maybe they make it rigorously, maybe they don't, and then spend all of the data science team's effort in persuading or convincing this pretend decision maker that it's actually their idea. Now, there's an element of fast there which could just be avoided if decision responsibility were delegated appropriately. There's no need for this usurping thing. If you don't have the time to put in the effort that it takes, then hand over that decision to someone who does have that time. If they're going to pursue it carefully, rigorously, and in this intense statistical way. Or say we're going to base it on inspiration. It's going to be a light case of analytics and plotting, but we're not going to allow ourselves to become more confident than our methods deserve. So really, most of that disconnect has to do either with the people hired or with the decision makers themselves, not knowing what their own role is. Since they are the ones who kick off the whole process, it really does matter that they have the skills to do it.

And is there another disconnect with respect to how many people in an org can speak data in the sense that, you know, data literacy and data fluency aren't necessarily spread or distributed across organizations? I suppose my question is, in organizations, you've seen how data literacy spread through the org, and how would you like to see this change?

So I'm not going to speak specifically to Google here. I'm going to speak much more generally about all of us at once in this world. Data literacy is in a sorry state, at least from my perspective. I really wish that we were better at this stuff, and we are surprisingly good at thinking through photograph data. And we're fairly reasonable. Fairly reasonable. We still might do some silly things, but we're fairly reasonable about that. And we're fairly reasonable about laughing and saying, oh, ha ha. Just because it's in a book doesn't mean it's true. But somehow, when it involves math and data, we start to pronounce data with a capital D, like it is some source of objective truth that's entirely disconnected from the humans that decided to collect it in the first place and made decisions over how they were going to collect it and why. So data literacy isn't a sorry state. And what I keep seeing in the world at large is that we lack the humility to say, well, if we had no one on the team who could have played this role, who had the skills to take on the decision maker's part, then we shouldn't take ourselves too seriously. Instead, what one sees out there in the wild is that there are these teams that are staffed with very meticulous mathematical minds and unskilled decision makers. So what I see missing in the world at large is teams with the humility to say taking ourselves seriously actually takes work and it takes skills. And if we lack those skills, we're not going to be able to do it. The best that we can get from this is the same kind of thing that we get from looking at a pile of photos. And that's actually still something. It's amazing that we have the ability to take an SD card, which means nothing to you when it's lying in the palm of your hand and you plug it into your computer and you use some visualization software, I don't know, Microsoft Paint or something, and now you can get inspired and see what's there. That's an incredibly powerful thing that's good for everybody. Everyone should be doing more of that on more data types. But not to assume that just any old data plus very complex mathematics can create something out of nothing. Certainty out of uncertainty, for example, a good decision process where the fundamental initial skills were lacking. I like to say inspiration is cheap, but rigor is expensive. And if you're not willing to pay up, don't expect that there is some magical formula that's going to give it to you. Without that, data literacy, please don't be trying to do very complicated things.

Right? So this is the present state of data science, decision making, decision intelligence and data literacy. What do the futures and their intersection of data science and decision science look like to you?

As we start to do more with data? I hope to see the world taking the quality of the decision skills kicking off and also guiding those projects growing. We can't really afford to be automating things with data at scale and have that all based on bad decision making skills. That's going to be a disaster for the companies doing it. So we'll have to move towards taking those skills more seriously and not treating it just as something that you have a flair for or a talent. But even though I imagine that whether we learn it now or we learn it the hard way later, that those skills are going to get better. They don't have to be carried entirely by the people with decision responsibility currently delegated to them. There is another option, and that other option is to hire a helper who can do that rigorous thinking for you. The pot of decision making that is a science can be done by a Scientist helping the decision maker who is owning the part that has to do with intuition and politics and so forth. So you can hire a helper to upgrade your skills if you don't want to go and learn this stuff yourself. But I do think that on the whole, the future involves us taking that first bit much, much more seriously.

So towards that future, my final question is, do you have a final call to action for all our listeners out there?

Yeah, two One is it's time to start shifting our focus away from only research and more to a choice of whether you want to be doing research or you want to be doing applied stuff. These are both equally valuable, important approaches. One of them is tremendously understaffed right now. One could argue both are. And it's a really exciting time if you want to get into that sphere, because that's going to become more and more important and as those general purpose techniques that the researchers make become more readily available for application. An analogy that I have for this is researchers might be folks who build microwaves, new and better microwaves, whereas applied folks think about innovating in the kitchen, in recipes at scale. And I want to point out that if you want to say, create McDonald's. Just because you don't have to wire your own microwave doesn't mean it's easy. So this is an exciting time for a new area of investigation and new discipline. And the other thing I want to leave you with is the world is generating more and more data. We really owe it to ourselves to make that data useful. Wasting all of our time and resources on type 3 error after type 3 error is a very sad state of affairs. So it's really time for us to take this seriously because we just have so much of it. Let's do good, beneficial things with it.

I love that because it really brings this conversation full circle in terms of. You stated at the start of our conversation that a big part of what you do is to help teams avoid or lower the rate of type 3 error in data science. We've come full circle essentially, and that's one of the calls to action here, right, that we all work together and use the data and our modeling techniques and question asking capabilities to lower type 3 errors more and more.

Yeah. And as I think back on our conversation, I think that a disservice that I did to decision intelligence as a whole is that I really spoke to you a lot about data scientists, I spoke a lot about decision makers, I vaguely mentioned social scientists. But it's a much, much more diverse game and I really left out all the other people that should be involved, the engineers, the reliability folks, the ethicists, the designers. There is a lot of important work to be done in this space by a great variety of people, and I want to ask everyone who's thinking about sneaking off right now because this doesn't apply to them to reconsider. Decision making is important for all of us, and if we're going to do this seriously and at scale, then there is a role for everyone to play. If you have anything to say about turning information into action.

I couldn't agree more. And Cassie, it's been such a pleasure having you on the show.

Thank you so much.

Thanks for joining. Our conversation with Cassie we saw the trade offs of the five different ways of embedding data scientists and data science work in organizations relative to the decision function. There was a pure data science team or center of Excellence model, then having data scientists embedded in engineering, then data science explicitly as decision support, then as decision maker, and finally the decision intelligence model. This final decision intelligence model, as Cassie told us, can be thought of as data science augmented with the social and managerial sciences. Also, make sure to check out our next episode, a conversation with Arnab Chatterjee. Arnab is a Senior Expert and Associate Partner in the Pharmaceutical and Medical products group at McKinsey and Company and we'll be cutting through the hype about artificial intelligence and machine learning in healthcare by looking at practical applications and how they help the industry evolve. Join us for an insider's account into what has worked in healthcare, from machine learning models being used to predict nearly everything in clinical settings, to imaging analytics for disease diagnosis to wound therapeutics. Will robots and AI replace disciplines such as radiology, ophthalmology and dermatology? How have the moving parts of data science work evolved in healthcare? What does the future of data science, ML and AI in healthcare hold? Join us to find out all this and more. I'm your host Hugo Bown Anderson. You can follow me on Twitter hugobown and datacampatacamp. You can find all our episodes and show notes@datacamp.com community podcast.