#1 Data Science, Past, Present and Future

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Hilary Mason about the past, present and future of data science. Hillary is the VP of Research at Cloudera Fast Forward Labs, a machine intelligence research company, and the data scientist in residence at accel. Previously she was the Chief Scientist at Bitly. If you want to hear about where data science has come from, where it is now, and the direction it's heading, you've come to the right place. Along the way, we'll delve into the ethics of machine learning, the challenges of AI, automation, and the roles of humanity and empathy in data science. I'm Hugo Baun Andersen, a Data Scientist at DataCamp, and this is Dataframe. Welcome to Dataframe, a weekly data camp podcast exploring what data science looks like on the ground for working data scientists and what problems data science can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBown and you can also follow DataCamp at DataCamp. You can find all of our episodes and show notes@datacamp.com community podcast hi Hilary and welcome to Data Framed.

Thank you.

So great to have you in the show. I'm really excited to be having a conversation today about data science, what it is, where it's come from, the past, present and future of this emerging discipline. But before we get into that, I'd like to find out a bit about you.

Okay, sounds like fun.

So how did you get into data science?

Oh, it's a good question because, you know, when I started my career, data science didn't exist. And so I started an academic computer science in machine learning a long time ago, about 20 years ago, and realized after some time I actually really like building products and systems that touch real people and real data and that furthermore, most of the interesting data was not in academia. It was actually in companies that were starting to collect this data on the Internet as a side effect of their business operations. And so I ended up Moving back to New York City, which is where I grew up, and joining some startups to work on hard algorithmic problems that would open up some product possibilities. And so data science started to emerge for a variety of reasons about that time. And this we're talking about 10 years ago. And so I've been pretty involved with it for a long time. And I think whether we call it data science or machine learning, it's really just different perspectives on using data to build interesting applications. And so it's been a while.

So when you say touch real people using real data, we're talking about seeing an effect on the ground. And I presume the startups that you started working for, when you came to back to New York City, you were working on these types of projects.

Yeah. So it's really the difference between working on an algorithm that may satisfy some theoretical requirements, you're working on a toy data set, and then looking at data actually generated by people or from human behavior that can then be used to provide some application or a service. And so one of the first companies I worked with was actually building models of career progression and career evolution by looking at a few million resumes. And so this was long before LinkedIn had any of these similar features. So we were able to actually see that, you know, oh, if you're a software engineer and you want to be a CEO one day, here are the jobs that other people tend to get in the meantime that will will take you from point A to point B. Or we would see other things. Like if you start as a lawyer, there was a 50% chance you'd be out of law in five years, whereas accountants would stay in law or stay in accounting. 90 something percent chance they'd still be accountants five years later. And that's what I mean by real data. It was giving us this insight into human behavior that was previously not out of reach, but was too expensive to apply to these sorts of fairly trivial problems.

And so this is an example of what you'd refer to as a data product.

Yes, the product piece was the part where a person could actually come along and ask these questions. But the data science part was getting these resumes, parsing these resumes, building the models, and then actually hooking it up to the product in a useful way.

Yeah. Cool. And what other types of projects were you working on when it came to New York?

Looking at Data extracted from 3D environments to understand and infer likely intents and actions. So looking at data from things like World of Warcraft or Second Life, which was Cool. At the time. I will remind you to try to figure out what people were attempting to accomplish in those environments. And that also had an interesting real time aspect to the classification problem. I also ended up working at a company called Bit Ly, which is a social media analytics company. In 2009, I was there for four years as their chief scientist. And that was really at a time when there was not a defined practice of analysis of social media data. There was a nascent computational social science movement and people were really just starting to get their heads around what we could learn about human behavior through this kind of data. And so it was a very exciting time to be able to play with it and think about the products we might be able to build.

Did you work for the City of New York at any point?

So I was on Mayor Bloomberg's technology advisory committee. So I wouldn't say work, but maybe volunteered is a better word.

I recall that Bloomberg was involved in all types of a variety of data science innovations and initiatives.

Mayor Bloomberg, when he was our mayor, was very involved in encouraging the technology industry in general. I think he realized that we couldn't rely entirely on finance as a city for a healthy economy and in, in many ways using data more effectively. So building the Mayor's Office of Analytics and using their own data as an internal tool to guide the use of scarce resources and you know, in many cases actually to make city services more efficient to actually, you know, save and improve lives. And then also he was responsible for large initiatives like the Cornell Technion Project, which brought Cornell University, which has just opened their new building on Roosevelt. That's pretty exciting to see that actually come to fruition.

That's really cool. And I recall there was an example of an initiative to do with ambulances and allocation of city resources in that sense.

Sure. And this is one of my favorite projects to talk about because it is fairly trivial data science work that actually leads to a very important impact. This is work that was done when Mike Flowers was the Chief Analytics Officer. I believe the work was done by Lauren Talbot, who's one of the statisticians on his team. And they looked at where ambulances should be sitting in order to be optimally located for the likely distribution of incoming calls. And they found that of course they're not actually sitting in those locations. They figured out that what the ambulance drivers really wanted was 24 hour bathrooms and coffee and other services so that they could actually be comfortable. They found them those services in those more optimal locations and actually managed to reduce the ambulance response Time fairly significantly. And I love this project for several reasons. So one is that it actually shows that data science can have a significant impact in the real world. So it's not just are you going to watch this movie on Netflix or are you going to buy this thing on Amazon? It's actually making our city more efficient and saving lives. Then the second part of it is that it's actually about going outside and asking people why are you sitting where you're sitting? What's important to you in getting to an optimal answer to this question and taking those human factors and into account. And the last bit, of course, is that the math is pretty trivial and well understood, but it's still hugely impactful when it's applied appropriately. And so this is why I love this example.

I think it's incredible. And as you spoke to the ideas of not such complex math, having a well formed question and also communication, the fact that data science doesn't exist in a vacuum. Understanding the problem and actually you, as you said, they found a solution and realized it wasn't happening in the real world and then went and spoke with ambulance drivers to, to figure that out. And something I admire a great deal about, about you is your emphasis on the role of communication in data science. So maybe you could speak to that a bit.

Oh, I think the best data scientists are people who are pretty empathic and are able to understand, you know, what is important to solve. I mean, the, the truth is that framing the questions is where the challenge is, finding the answers a trivial exercise or an impossible one. And when you think about it that way, a really great data scientist can sit down with somebody, understand the thing they're trying to make a decision about or what they need to know. Go away, MacGyver up some analysis with the data that is available or could be available and the tools they have at hand, they can go back to that person and explain to them what they've learned in a way that lets that other person make a better decision. And when you frame data science work in that kind of context, it becomes really about how well you can understand somebody else's domain and somebody else's needs and then how well you can do your own work to satisfy those needs. And it's not about whose math is the hardest math, it's really about how do I get to a robust problem definition that I can solve that will actually give someone an insight they didn't otherwise have.

So we're here to talk about data science as a function of time, where it came from and the direction in which it's heading, what factors led to the emergence of data science as a discipline.

So it's not an accident that data Science emerged about 10 years ago because it is a technological artifact in the sense that technology had progressed to the point where the multiple things that a data scientist does could be combined in one professional role, those things being actually write code and build models. So there's a technical skill set and tools set that had to be created. The data had to be available, which was also not the case before 10 years ago, or it was very expensive to make that data available. And you also needed a set of problems and processes and ways of thinking about the world that let you put all these pieces together. And that's the broader communication and empathy piece. Right. And so we hit a point where previously, it's not as if this was new work at all. People have been using databases for nearly 100 years for solving business problems. But it was newly affordable and newly so easy that one person could take on everything from the sort of problem formulation through to the analysis to the visualization, to the communication, to the eventual decision making as well, in a way that it just hadn't been before. And it opened the door to the creation of this new job role of data science as something that is itself distinct.

And where did we see it emerge? Which disciplines or fields?

So I see, and keep in mind, my background is computer science, so I have a bit of a bias here, but I do see data science as a blend of. It's essentially as if computer scientists had stolen a lot of the wisdom of statistics. And so it is a blend of computer science, statistics. And then we're also seeing more influence from social sciences now as well.

Yeah. So, and also, as you said before, communication, journalism, storytelling, or all of these.

As well, just the kinds of business fluency skills that we expect most professionals to have today.

So where is data science now and what's it capable of?

So data science has become a real thing, which still kind of astounds me that there are potentially thousands of people running around with that job tag and it is accepted as a role in an organization. And it's something that, you know, if you mention that you do data science at a dinner party, you're probably not going to get people turning around and walking away like people think. It's actually an interesting thing to do. And I also believe we're starting to see data scientists make large contributions to their organizations, and not universally, and there are certainly still challenges to overcome. But the value of data Science, from a business point of view is pretty clear at this point. The questions I have really are how will the practice of data science be changing over the next five years? Will we still be using that title or will we all be AI monkeys or something else? Even though of course the fundamental skills will remain the same. Those things I'm not entirely clear on yet.

And so we're definitely going to talk about the future of data science very soon. But I want to know which, which industries, which fields do you see data science having been capable of having the most impact now?

So right now, and here I'm speaking through the lens of our work at Cloudera Fast Forward Labs, I mean, we see huge impacts across industries, but some are more mature than others. So particularly in finance, they're not necessarily in the places you'd expect. We see large progress being made. And this is largely because these companies have a lot of data already in a fairly like finance has a long history of making data useful. And so there is already a culture of being fairly data driven in place in many of these companies. And they're also very interested in extending those capabilities to new kinds of data. And so that's a place where we've seen people starting to make unstructured data useful in the ways that only structured data has been useful before, by which I mean things like tax. And so that's certainly one area. Another area where I see a lot of impact is in the pharmaceutical and healthcare space where you have, if you can shave a few percentage points off the cost of certain exploration activities, you have a very clear win. And data is certainly a tool for doing that. We work pretty heavily in media as well. That's things like understanding your audience, helping them find content they'll love, helping them engage with that content, making sure it's shared optimally across different platforms. It's not any one place, but really pretty distributed. When I started the work at Fast Forward Labs about three and a half years ago, I thought we'd probably end up working in one industry or maybe two, but that really hasn't been the case. And it turns out that the thing that everyone has in common is the data and the math is the same. So whether we're generating celebrity gossip, reporting on fashion, or we're writing a program to generate language about portfolio performance, it's the same mathematics and same techniques that enable those data applications. And so we've seen pretty broad use.

That's cool. And I like the examples you give, because finance and pharma, for example, I think speak to this idea we were discussing earlier, that data science existed in certain disciplines before the term came about. Right. So in finance and pharma, they've been doing data science, or analogs of data science, for decades, if not longer.

Absolutely. And the same in insurance, though I found that insurance as an industry is only now really picking up on modern data science. It's a pretty exciting time if you work in insurance analytics right now.

Yeah, absolutely. And I think you also spoke to the idea that the math, a lot of the time remains the same. The applications may change depending on industry, but there is an abstraction that the same techniques will apply. And that's something that's deeply integrated into your work at Fast Forward Labs, right?

It is. And so we do our own program of applied research, looking at emerging capabilities and attempting to make them useful to our clients ahead of where they otherwise would be. And so we publish reports which are a description of what the thing is and how it works at both a conceptual and a technical level every quarter, along with working software prototypes that demonstrate it applied to a business problem. But we try to choose problems where people have a fair amount of empathy for it, and so they can look at the prototype and say, okay, I understand how this technique can be applied to my work. And so, just to give you one concrete example, we recently did a report on algorithmic interpretability. So these are new algorithmic techniques that you can run on top of black box algorithms, such as neural networks or other deep learning approaches to, at a very high level, permute the inputs and look at how the outputs change and then infer the significant features in the classifications that those black box systems are making. And our business problem demonstration was on a black box model of churn. For a telecom, this was a real problem. We advised one of our customers about where the interpretability capability was used to be able to see which features of the customer were leading them to churn. So were they paying too much? Were they on an old technology? And you could also change those things and get a model of how the prediction changed, which actually enabled new marketing and new customer service actions.

That's awesome. So, once again, this takes us back to the idea of being able to communicate technical data science results to stakeholders.

Exactly that. So the idea is that an executive or an engineer can look at this and say, okay, I get it. I see how it works. I can now apply it to my problem. Whether that problem might be doing something like bias testing for regulatory compliance using the same mathematical technique, or it might be something like inspecting a prediction model for you know when and where to spend resources, whatever it is. You can get a very good intuition for the algorithmic capability and then figure out how to transfer that into your specific domain and problem set.

I think that's a great example. And I actually. I think your colleague Mike Williams gave a webinar on this, didn't he?

Yes, he did.

Yeah, I tuned into that. That was awesome.

Oh, that's very cool. That's up on our blog, if you.

Well, we'll put that in the show notes. Definitely.

Awesome.

Now it's time for a segment called Statistical Pitfalls. I'm here with Michael Bettencourt, core developer of the open source statistical modeling platform Stan, who also goes around at parties calling himself a once and future physicist, masquerading as a statistician. What's up, Mike?

Thanks for having me, Hugo. It's true, I am really fun at parties.

You sure are. But today you're here to tell us about a common statistical trap. Selection bias.

Indeed. So selection biases are really present anytime we collect data. Before talking about generalities, however, I want to first consider how subtle it can be by considering one of the most famous examples of selection bias, which came out of an analysis in World War II led by the statistician Abraham Wald. Now, the Allied military was interested in how they could most efficiently use their limited resources to supplement the armor of the damaged aircraft that were returning from combat missions. The planes returning would be damaged in particular locations, and popular thinking was just to add more protection to where the damage was observed. This thinking, however, implicitly assumed that the returning planes were representative of the entire population of damaged planes. Walden and his team recognized that the returning planes were in fact a bias sample because they didn't include the planes that had never actually made it back from their missions. Once they understood the selection bias in their data, they recommended a seemingly counterintuitive strategy of reinforcing armor in places where the turning bombers were undamaged. What they had realized was that if potential damage was uniformly distributed across each plane, then the damage on the returning planes actually demonstrated what damage was survivable. It's harm to the undamaged areas that was likely to be critical and prevent the planes from returning.

And this is an example of selection bias, right?

Yes, exactly. One of the most common pitfalls in statistics is the misunderstanding that the data in hand are fully representative of the system being studied. Were this true, we'd be able to infer arbitrarily precise insights about the system as we collect more and more data. Unfortunately, in Practice, our measurements are never quite as perfect as we assume them to be. And the reality is that anytime we collect data, we neglect some of it. If we want to ensure accurate inferences, then we have to consider how these selection biases might be corrupting our data. In particular, we have to consider how our measurement process preferentially ignores data with some properties while preserving others. If we don't, then the inferences we draw from even very large data sets will always be limited in how much they can teach us about the systems that we're analyzing.

So how big of a problem can selection bias be and how do you combat it?

Well, the exact consequences of selection bias depend both on the selection mechanism itself and how precise of an inference we need in a given application. For example, if we're a company and we're trying to understand how a niche demographic interacts with our product, then even small selection biases can seriously distort our insights. Ultimately, the only way to quantify the magnitude of these effects and determine whether or not selection bias needs to be taken into account is to build a mathematical model of the full measurement process. Fortunately, there's a growing collection of tools to facilitate this kind of complex modeling. In particular, problems with programming languages like Stan.

Mike, thank you for that great description of the common statistical pitfall of selection bias.

Thanks for having me, Hugo.

Now back to our interview with Hilary Mason. So we've discussed a bunch about what data science is. Is capable of, but we've also heard that data science is, well, data scientist is one of the sexiest jobs, or the sexiest job of the 21st century. So there's a lot of hype around the term data science. And with such hype, I think there's also a healthy skepticism that needs to be invoked. What isn't data science capable of? What can't data science do?

I mean, that's a really great question, but it also comes from this framing where we assume that the default is that data science can do anything, right? So it's pretty clear that data science can often tell us what to expect or, you know, what might happen, but not why. And so if you want to understand the why, you really have to go talk to people. You have to understand that a lot of that knowledge is, you know, based. It's in someone's brain, it's from their experience. And this is things like, and this applies to the entire umbrella of data capabilities, from analytics up to these more complex, you know, and interesting neural network models. You might get a result that you just don't know why it's doing what it's doing, or you might see something in your data that you can't explain. So, like one company, I looked at their analytics, it was an E commerce company, and they had a unexplained but repeated trend of increased orders in March. And this is something that you can see very clearly in the data. I had sufficient data to be able to make a prediction for the following year. You could understand the trend from a mathematical point of view, but you could not explain to anyone why that was happening. And it turned out to take quite a bit of digging to figure out what was going on there. And what the story actually was is that there was a subset of a few products that had gotten written into some elementary school curriculums and the purchasing decisions for school districts had to be made a year ahead, by March. And so what you were seeing there was this bizarre artifact of arbitrary deadlines and a set of customers they didn't even know they had. And that was not the kind of answer we were ever going to get to just from looking at the graph.

Knowing something about the domain and actually delving in to the results is essential in this case.

Yes. So data science, the science part, will only take you so far.

We also see a rise in awareness about such challenges as algorithmic bias, whether it be algorithms encoding societal biases or human biases, or algorithms creating their own biases as well. Is this something you're actively thinking about?

Absolutely. And as part of this interpretability research, we hosted a research fellow named Julius Adebayo who did some fantastic work on bias, extending the work that ProPublica had done on the recidivism data set where a, you know, black box proprietary algorithm was making sentencing recommendations. That's also on our blog. But yes, this is something that we focus not just on algorithmic bias, but on the ethical implications of the use of every capability we talk about. We have a chapter on that in every report we write. That's really to help people understand the kinds of decisions they have to make in designing around these algorithms, and then also to give them an excuse to have the conversation and to think about the ways in which something could go awry and impact the people on the other end of the product.

So what are the biggest concerns for you and Fast Forward Labs when thinking about ethical aspects of data science?

I mean, the biggest concern I have is a very basic one, which is that the ethical considerations are rarely part of the data product designer planning process. And so it's really a high level concern that the Only time I see this routinely considered in the product development process is when it's a regulatory compliance issue as well as. And so if a company has a legal obligation to not discriminate, then there certainly is a review and a lot of thinking about how to validate that. There's no discrimination in an algorithmic system. But if that legal requirement is missing, it is still not a given that they'll even be thinking about data bias in the data or bias in the results or how that may impact people in the products they eventually release. And so that's, that's like level zero.

And so part of our challenge there then is also the fact that tech is faster than legislation, right?

Yes. And I don't personally think that legislation is necessarily the answer. I think the answer is that we are still developing a practice of what I'll call data product development here, which is. And data product could just be a model in a report, or it could be an internal tool. It doesn't have to be a consumer facing beautiful application or Google Maps.

Right.

Google Maps being my favorite data product, because you don't need to know anything about the data and the algorithms behind it, which are incredibly technically impressive in order to use it effectively. But that aside that we as a field are still evolving the practice. And you know, you can see this when you look from one company to another, if you're a software engineer, you're probably going to encounter pretty much the same process from when you move from company to company. That is not the case with data science and it's certainly not the case with data product development. And so as this practice emerges, I would like to see us as a community consider ethics as a first class design principle in our work.

This, you stated was, was level zero.

Yes.

And what's. What's built on top of that?

Well, level one is where we can start talking about the specific problems we've already seen emerge and specific things to watch out for. And I can give you a pl plenty of examples, but I think we're still stuck at that very beginning part.

So bundled in with all of this, what data science is, what it isn't, there are a lot of buzzwords floating around that are very substantive as well, but I need your help to demystify them. Examples such as deep learning and artificial intelligence are probably the most prevalent that have gained currency and getting a lot of attention. What terms and language do you think need to be clarified in the data science space, particularly with regards to what they are actually capable of?

I spend a huge amount of my time just clarifying the use of different terms in a given room, because we can't take for granted that when someone says AI, they mean the same thing that I would mean when I say it. And the meaning of these words has changed dramatically in the last few years, and I expect we'll continue to do so. Right now, at this moment, In December of 2017, we have seen a huge increase in the popularity of deep learning neural network techniques for very good reason. It's opening up capabilities that were simply impossible five years ago. Things like robust image object recognition, doing video classification, looking at audio in a way that is something that is actually novel. And beyond that, being able to model text and language in a way that is completely novel. So looking at things like word embeddings and sentence embeddings, and these give us a bunch of new tools for addressing an entirely new and interesting class of problems.

And you actually have a report on word embeddings that came out recently, is that right?

We do. We framed it around summarization, so the report is called summarization, but it's essentially about word and sentence embeddings in order to do robust extractive summarization of documents. And it was a lot of fun. We have a great prototype for that where you get a Chrome extension you can run on any English language article on the web and see the summarization run in real time and play with the different network architectures in order to see the different kinds of summaries that get extracted, which is a lot of fun.

Oh, that's awesome. I know what I'm doing tonight. Sorry I cut you off. Talking about the capabilities of deep learning in particular.

Right. So deep learning is, again, it's one of these terms where there's no hard limit for how many layers you need in a network to be deep. So at this point, anything that's pretty much a neural network, even if it's one or two layers, is deep learning. And it itself is one set of techniques under the broad umbrella of machine learning. But when we think about AI and the way that term has come to be used now, I mean, historically, it was the field of research inside of computer science that gave birth to machine learning. And that was because there was such disillusionment with AI from a funding and sort of accomplishment perspective, People were so overly optimistic that they. That researchers essentially had to rebrand. And it also coincided with the use of probabilistic and statistical techniques. So AI fell out of favor as the term of art and started to show up, up really more in sci fi and in movies, but now it's come back, and it's come back largely as a result of the rise of deep learning and the capabilities there and the hints we see of, you know, more intelligent machines. But AI itself today is not a technical term. It's largely a marketing term. And it's one that I think shows a little bit more enthusiasm for the capabilities that what. Than what may actually be possible given the state of the technology.

Yeah. And I think you made an interesting point that AI is also a term that has been in the cultural consciousness from, from science fiction. So people have a general idea that, you know, sentient robots and, and these types of things are artificial intelligence. And we see that. We, we see headlines play on that. I, I can't quote any off the top of my head, but you see headlines such as artificial intelligence creates copy of itself in a way that Google can't understand. And, you know, stuff like that. And we need to be careful about that.

Right, right. And one of the things I found most fascinating about the emergence of AI as a term now is the way it has changed the language that people use to talk about what is still fundamentally a computer program, in that it implies this, this kind of anthropomorphism. And we talk about the AI, and I'm putting this in air quotes as if it's a person in a way that just changes the expectations people have of its capabilities. And I do think we have to be pretty careful with that sort of language and the impression that it encourages people to take away.

And I think part of this anthropomorphic process is that we have had this term ported from science fiction because we all know Blade Runner. Right. And what replicants are and, and these types of questions being posed in that space.

Yes. And the Matrix and all the Philip King.

So these are a bunch of interesting challenges that I think we're facing as a discipline. Are there any other major challenges that you think currently face the data science community?

I mean, do you think that imprecise ethics, no standards of practice, and a lack of consistent vocabulary are not enough challenges for us today?

I definitely think so. We haven't really delved into the inconsistent practices though, so maybe you can speak a bit to that.

Yeah, I mean, it's really an artifact of the. That data science is still a fairly new professional role, and as such it tends to get shoved in with software engineering, or sometimes it gets shoved in with, you know, traditional analytics and sort of the CFO sort of framing on the world, but it's not managed using its own process. And so this, this is a very controversial statement, but if you are going to run an agile by the book process, it is terrible for data science in the sense that it is the ideal process for software engineering. But when you start out with a software project, you generally know that the thing, the exact thing you want to build is achievable. And what you're figuring out in your sprints is the methodology by which you will achieve it. But you are not inventing anything new, you're not doing experimentation in data science. You are doing experimentation. You have a question, you are trying to get to an answer, and you don't necessarily know at the beginning if it's going to work. And if you do know, then I'd say what you're doing is a little bit more analytics than it is data science. And so trying to shove this into the established practice that works very well for engineering does not work for science. And I've seen many companies where they end up with a lot of wasted effort and friction because they don't manage data science as its own thing.

And what are the most important aspects of the data science process that you think need to become more rigorous or develop a process or methodology on whether it be data mining, documenting data lineage through to the actual development of a product.

So all of these things are important, right? So you need to know what data you have, where that data comes from, why it looks the way it looks. Did someone make a decision about, you know, a database field being of a specific length or a specific type, and if so, why did they make that decision and what might you be losing? All of that is important to being able to do accurate data work. But from a data science practice point of view, when you set out to do a project, you generally start with a question or something you're trying to understand. And I always encourage people to write that down and to write it down in plain language so that anyone in the business can understand it, not just people who are technical. And that is actually harder than you think it is. So I've had plenty of data scientists be like, oh, that's easy, I'll have it to you in 10 minutes. And then two days later they're like, oh, man, this is actually not so easy.

I love that, because that actually speaks to the fact that data science answers questions that exist independent of data science, if that makes sense.

I'm thinking about it. I think it does.

So we can pose the question in the world before data science exists. And then data science is a tool to answer this question, which we can pose without using the language.

Oh my gosh, you're right. This is beautiful. So question two, once you have a problem statement, is really, what are the error metrics by which you'll know you have a successful solution to the question you have posed? And hopefully these are quantitative error metrics, but sometimes they aren't. And I've also found as someone managing data science teams, we have a lot of shame when we don't have proper quantitative error metrics. And so I want people to admit before they even start the work that this is how we're going to know that we've solved the problem and this is just the way it is. And these are problems like working on search engine algorithms, where you know, you can pull together a couple of things that give you some notion of whether your algorithm is better than random. But getting to a true quantitative metric really requires a volume of user data that may or may not be available to you, depending on what kind of company and product you're working on. Right? So you might have to admit that there are no quantitative metrics, and that's okay. And then the next thing you need to answer is really, what is the product or business utility of this work? So why are we doing this? And I always like to phrase this as, you know, assuming we can answer the question successfully, what is the first thing we'll do with it? And that phrasing is very careful because I find that well run data science practices have multiple uses in mind for pretty much every piece of work. So everything you do opens up the ability to do something else or to do something new, faster and cheaper than you could have done it it before, which also speaks to a set of requirements around practice. Or if you have a team, they need to be sharing business definitions of the data, they need to be sharing code and capabilities. And so a lot of things fall out of just having a really nice process around how you frame the problems that you're going to explore, how you make sure that they're worthwhile and impactful. Because we also have this problem where we as a group tend to get very excited about interesting things that are not necessarily impactful. And you don't want someone vanishing down a rabbit hole for a few months and coming back with something that's not really useful. And then how you know when to stop spending time on something. Because one of the big differences between academic computer science work and data science work is that you're in a business, you don't generally have two years to think about one problem. In fact, that's one of my favorite interview questions is, is, you know, okay, what's the proper approach to thinking about this problem? And then, okay, that's a one year solution. What's your one week version and what's your one day version?

Yeah. And the desired metric or how well your model performs or whatever it may be, may be a function of how much time you have for the project, Right?

Absolutely. Or what your budget is for testing and all of these things. And so it's really developing a discipline around those aspects, which if we look at our sister disciplines of software engineering, those sort of exploratory aspects are much less important than they are in data.

Science in terms of these processes and methodologies being developed and made rigorous. Is there a concern that this may happen in silos?

That's an interesting question because it does happen in silos. So I've seen this done well at many companies, but they all do it differently. And that's something that I think will change over the coming years. And part of it will descend from our tooling. So as our tooling gets more advanced, it will encode some of this process and practice in it. And you can think about the way GitHub has had a huge influence on software engineering workflows and pull requests. And I think that's a good analog for where the tools will take us. But we'll also, hopefully we'll converge on some combinations of how, what good work.

Looks like and what timescale do you predict this happening on. And I know I'm really asking you to do data science now.

I have no data at hand, so what I'm doing is purely intuition. I think it'll happen in the next few years because all of this has happened much faster than I would have predicted. And so I want to say it's 10 years off, but I actually think it's more like three to four years off path.

Now for something completely different, we have a segment called Tales from the Open Source. Today we'll hear from Chris Fonsbeck, assistant professor of Biostatistics at Vanderbilt University and a core contributor of PMC3. What's up, Chris?

Hi, Hugo. It's good to be here. Thanks for inviting me.

It is always a pleasure to chat with you. Chris, what are you going to tell us about today?

Well, I'd like to introduce you to an open source project for probabilistic programming. But before I do that, I suppose I should tell you what probabilistic programming is. Well, if you think about classical programming, about the types of variables you can use when writing a program, what do you think about integers, floats, strings, arrays, things like that? In a probabilistic program, we build things using variables that represent random quantities. Here we're talking about things like probability distributions, normal distributions, binomial distributions, and the like. Even more advanced things like stochastic processes. What do we need those things for? Well, mostly we use probabilistic programming to build Bayesian statistical models. Bayesian models are really good for estimation and prediction in a variety of scientific applications. For example, I use them in my work to model infectious diseases and to conduct statistical meta analyses. But they can be applied wherever classical statistics are used. Regression modeling, group comparison, spatial data analysis, you name it. They're super flexible and super powerful, and they let you build useful models that are easy to understand and interpret because they're based on probabilities and most people have an intuition about them. The problem is the technical details of using these models are complicated. The good news is today there are quite a few easy to use software packages for building Bayesian models using probabilistic programming, and PMC3 is one of these. The project's been around for a long time, around 15 years, and it's been used a lot in academia and industry to build statistical models. And one of the main strengths of PMC is that it abstracts away all of the mechanics of doing Bayesian model fitting. And that lets you specify models in a manner not unlike standing in front of a whiteboard and writing out your model using mathematical notation. And this allows users to concentrate on their particular problem of interest rather than the technical details of the underlying computational methods. It gives you everything you need to build virtually any model you can think of. Now, remember those random variable types that I mentioned? PMC has over 40 different probability distributions built in, and it makes it really easy to write your own if you have something more complicated in mind. But I think the most compelling feature of PyMC3 is that it provides efficient and easy to use implementations of the latest Bayesian computational methods. This means two different things. Markov Chain, Monte Carlo, or MCMC as it's known, and variational inference. Vi. You may have heard of MCMC before, associated with software like bugs or jags, but PyMC implements newer, more efficient methods that model gradients to better approximate unknown variables in your model. To make things performant, PYMC3 is built around the Theano library, which is a popular framework for deep learning that's fast and stable. And this, combined with our implemented variational inference methods, makes PYMC suitable for analyzing very large data sets, which are increasingly common today. So if you're in the market for a powerful open source statistical analysis package for Python, I'd encourage you to take a closer look at PyMC3. The project repository includes a large suite of jupyter notebooks with well developed examples of the sorts of analyses that are possible. I also strongly encourage folks who are interested in contributing to a scientific computing project like this to visit our GitHub page. We have a slew of ideas for future development that we'd like to implement, as well as many outstanding issues and support tasks and we could really use your help. And you can find us at GitHub.com pmc devs pymc3 thanks Chris, for an.

Illuminating introduction to the world of probabilistic programming and PyMC3. Let's get back into our interview with Hillary. So, in 2010. So that was seven years ago now, nearly eight years ago, you, you wrote with Chris Wiggins a piece called Taxonomy of Data Science in which you proposed what you refer to as one possible taxonomy of what a data scientist does.

That's right.

How has what a working data scientist does changed since then? And has anything surprised you about this?

Well, if you, if you look at what we wrote, and it was, you know, a short essay to just put down in writing a thing that we had not seen put down in writing yet. So keep in mind, seven years ago, data science was not really a viable career option. It was not a common job. The phrase had just really come into, you know, use beyond a couple of companies. And what we wrote down is so obvious. If you look at it now, you're like, what are these people thinking that they have to spend their time to articulate, you know, something that's so very clear. But it was not clear seven years ago. And so I'm glad you reminded me of that because it really does help put the time scales here into perspective that this has only really been, you know, a viable practice in itself. Seven years. So maybe I'll have to shorten my prediction and say that perhaps we'll standardize on that process in the next two to three years, not three to four.

But it is still a telling piece because, I mean, it makes it clear how, you know, being able to interact with the shell, the terminal is incredibly, incredibly important. And I still have aspiring data scientists come and ask me at Data Camp, is shell as necessary as people say it is? And these types of things. So I think in a world where there's a plethora of all types of tools to know and learn, it does set a good baseline for what people need to be doing.

I mean, I think so, but of course I have a huge bias. I'm still a huge fan of awk and other bathrooms capabilities. And so, yeah, I think that's, it's not necessary, but it does give you a speed advantage in a bunch of situations for sure.

So we've touched upon what the future of data science looks like to you, but I want to ask you a relatively general question, which is what does the data science landscape look like to you in the coming two, five and or 10 years?

Oh, that's a fun one. Right. So I think two years looks largely like today in the sense that the kinds of problems we're solving won't have changed very much. And our tooling will progress, but it will ease some of the challenges we currently face. And by that I mean we'll look at, we'll, you know, take things like once I've trained a model, how do I deploy it and monitor its quality over time and deal with retraining. And that will be something that, that's covered by standard, hopefully infrastructure tooling that won't be a custom set of little nutty scripts that only the person who created the thing can actually manage to run appropriately, which is where we are right now. I think in two years we know what that's going to look like. We'll have a lot better tooling around model deployment and monitoring. We may have more standard tooling around things like AB testing and multi armed bandit testing and some better experimentation tools. I think within enterprise we'll see better data provenance and data sharing tools, and so even things like feature catalogs. But this is all stuff that is essentially an engineering problem in that we know it's possible, we know how to build it just sort of hasn't happened yet. Right?

Yeah.

Five years is where it starts to get interesting because that's enough time for a creeping tide of commoditization to come in. And we may see some really interesting capabilities emerge from some of the AutoML work that's being done, which will change perhaps our fundamental approach to machine learning. And so, as excited as everyone is today about deep learning, the vast majority of deployed machine learning systems are not using deep learning. And deep learning is by far not the best choice, certainly not for the beginning of those projects where you're just trying to get something in place and perhaps you even want to know why it works the way it does?

Yeah. And could you remind me what AutoML is?

So it's automatic machine learning and it has a couple of different meanings currently. Again, back to the clarity around vocabulary. So some companies are using it to mean things like tooling for citizen data scientists. You won't need professional data scientists anymore. I think that particular framing is kind of bullshit, largely because, you know, you can give someone a button that will, will sort of select the right classifier and do the hyperparameter tuning for them, but that doesn't mean that they'll, it's doing what it's doing and at a level where they can actually do useful work.

And this speaks to interpretability once again, having someone who's able to translate from machine language, well, not literally machine language, but from computational language, from the math to the real world situation, having that interface.

Right. And so you also have Some use of AutoML where you have automated parameter tuning. You have the ability to take trained features from one model and use them in another model without a person having to make a decision about that. And there's some really exciting research work there that we have yet to see broadly impact practice. And so I think that at the five year timescale, if that is going to pan out, we will see it and it may change the way we think about, you know, design of our algorithms entirely. It's pretty exciting stuff to think about using deep learning to design deep learning systems. It's a little bit meta, but I.

Think this idea of pre trained models and transfer learning is going to be huge and also in the deep learning space.

Right, Absolutely. And I also wonder if you're a large organization, you have many data scientists, they should not all be working in isolation. And so if one is training on a certain set of data, those things should transfer to other related problems. You end up perhaps with a kind of network effect capability that might happen. Again, this is all supposition because this is the kind of stuff where today we see hints of it working. We don't yet know what the real impact is going to be and we certainly don't know how it will change process and practice and we don't know how it'll change our broader set of tools. Tools. And so, so there's a ton of exciting stuff in that five year time frame that we can start to imagine, but we can't, you know, quite. It's still a. Let's just say we have a lot of uncertainty in that future.

And I'm presuming there's even more uncertainty in, in the, your ten year prediction.

Oh, the ten year predictions are, you know, really far out there. Will we even have data science in 10 years? I mean, again, I remember a world where we didn't. And it wouldn't surprise me if the title falls, you know, goes the way of the webmaster. And you know, maybe data science becomes a tool set that we expect every engineer to have or maybe we expect every, you know, business professional to have it. I'm not saying that's what I believe. I do think we'll still need the specialists to sort of build and tune and monitor those machines. But I also, I could imagine that being a, like there's a potential set of universes out there where that's the case. I'm personally more excited about. Like in 10 years, I expect we will make rapid progress in understanding of language and emotion, which will open up a whole nother set of, you know, potential applications and capabilities. And we also haven't talked about hardware at all. But even today we see, you know, systems and infrastructure have to evolve to run across fairly heterogeneous hardware. So you now have CPUs, GPUs, you might have, you know, interesting sort of GPUs running on the edge. You have stuff running on mobile. I think 10 years from now, machine learning will be running everywhere. And we may not even be carrying around our phone bricks. We might be carrying, you know, maybe it's a hairpin or a little watch that has all of that computational power and we're using a variety of ambient interfaces, like who even knows. But all of those things will involve machine learning in some form.

And as you mentioned, aspects of machine learning such as feature extraction, feature engineering, these, these things are well on their way to being automated to a certain extent. And whether data science exists as the field it does today, whether it has a new name, processes, machine learning will still be incredibly important. I'm wondering, what parts of the data science process do you think are less likely to become automated? What are the most valuable skills for data scientists to be, to be working on?

So I think personally the ability to frame the best problems to work on is a skill that is underappreciated and unlikely to be automated.

That's a great answer.

Yeah, I mean, it's the hardest part. Again, generally the answers are trivial or impossible, but the problem statement is where the majority of the work goes. And knowing what's useful and valuable is still something that's, it's really hard for people to do well I don't think we'll, we'll get to the point where machines can just, you know, sort of drop in and do it across all domains. I also think that we in data science tend to neglect some of the human aspects of the products we build. And so I got into a fun debate a few months ago with someone who was arguing that a business professional who spends the majority of time working at their computer answering email could never be replaced, but cashier could. And I think it really speaks to the assumption that they were making about the role of the cashier. So do you think the cashier in a store is there to push the buttons, or do you actually think they're there to be a human presence and interact with someone and say, how's your day? I think we tend not to pay enough attention to the important work of human relationships around the data systems that we build.

So my final question is for aspiring data scientists and well seasoned data scientists alike. Do you have a final call to action or something you'd like to tell them?

Oh, that's a lot of pressure to put on one question.

I know.

I mean, so the thing I would tell aspiring data scientists that I think the seasoned ones probably know already is just to follow what's interesting in the sense that like, this field didn't exist seven years ago and here we are today. And so don't have any hard expectations about what your life is going to look like seven years from now. Just follow interesting problems and interesting people and technologies and that's, that's where you're going to find the really hard, fun, impactful work.

Hillary, it's been such a pleasure having you on the show. Show.

Oh, this has been really a lot of fun. Thank you so much.

Of course, I had a great time as well. Thanks for joining our conversation with Hillary Mason about the past, present and future of data science. As we discovered, the future of data science will include a wide array of machine learning applications operating in everyday life. Many common data science techniques, such as feature extraction may be automated away, but in Hillary's words, the ability to frame the best problems to work on is a skill that is underappreciated and unlikely to be automated. We also heard a call to arms to approach major challenges facing our discipline, such as imprecise ethics, no standards of practice and a lack of consistent vocabulary. Along with an in depth study of what data science currently looks like on the ground. Make sure to check out our next episode, a conversation with Chris Volinski, Assistant Vice President for Big Data Research at AT&T Labs and a member of the seven person four country team, Belcor's Pragmatic Chaos that won the one million dollar Netflix Prize, an open competition for improving Netflix's online recommendation system. It.