#28 Organizing Data Science Teams (with Jacqueline Nolis)

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Jonathan Nolis, a data science leader in the Seattle area. With over a decade of experience, Jonathan is currently running a consulting firm, helping Fortune 500 companies with data science, machine learning, and artificial intelligence. We'll be talking about how to organize a data science team and the do's and don'ts of managing them. With a dive into best practices for hiring data scientists, we also tackle questions such as what can't data science do? And what's a more important skill for a data scientist? The ability to use the most sophisticated deep learning models or being able to make good PowerPoint slides. The answer may surprise you, but then again, it may not. There's only one way to find out. Keep listening. I'm Hugo Bauen Anderson, a Data Scientist at DataCamp, and this is Data Framed. Welcome to Data, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter Hugo Bowne and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com community podcast. Hi, Jonathan, and welcome to Data Framed.

Hi. How's it going?

Really good. Really excited to have you on the show. And before we jump in to our conversation, I've got a quick question for you. My first question is, in terms of skills that data scientists need to work on a daily basis, do you think it's more important to be able to develop sophisticated machine learning models or to be able to give a PowerPoint presentation?

Oh, I'm going to say PowerPoint presentation, but I feel like that's a controversial answer. But I actually feel really strongly about this.

Tell me a bit more about that.

So when you think about the job of a data scientist, usually what that entails is taking data, understanding it, processing it, really trying to figure out what's going on. And then once you've figured out what's going on, trying to convince someone else of what they should do or what's there or what your feeling is. But there's this really important part that is the convincing part. And knowing how to make a good PowerPoint presentation is really important in a business setting to be able to convey all those thoughts and ideas you've learned. And so having the ability to do that is way more important than being able to use the most advanced machine learning models. Right. If you can use a linear regression and you can give a good PowerPoint that can often be much more powerful than being able to do deep learning recurrent neural networks, but not being able to take what you've learned and convey it to other people.

That's great. And I love that you mentioned regression because this is an example where you can show people relatively straightforwardly, even non technical people, why the model does what it does. You can explain if you tweak one parameter, why the output that they're interested in changes.

Yeah, and I think that's something that is often undervalued. I think there's this kind of traditional notion in data science that the higher accuracy the better and that's the most important. You know, and I think like things like Kaggle competitions really emphasize this, that the more accurate you can get, the higher are your R squared, the better things are going to. But there's actually a lot to doing models, right? There's understanding what are the things that are important than the model. There's the convincing people that your model is good. There's lots of things that go on and it's often that stuff is more important than necessarily getting the technically most accurate answer. And so something like a logistic regression is. It's really great because it's really easy to understand what's important. It's easy to explain to other people what's going on. And that's, you know, that's having things not be a black box is often really useful.

Absolutely. And I also think you mentioned that a data scientist's job is to convince people of the power of their models and to help make business decisions. And another point is not only do you want to make PowerPoint presentations, but you want to, for example, you know, have figures that people understand, use colors that people like as well.

Right. You know, if you're making a plot and there's something that's good that could come out of the plot and there's something that's bad. Right. Like maybe there's a bar for opportunity and a bar for risk. And if you color the bars where the good one is green and the bad one is red, when someone's going to look at that chart, they're going to much more quickly understand what's going on than if you made them two different shades of blue and that's really small. But if you think about making good visualizations, good presentations, there are giant sets of these small decisions that altogether compile and to do people quickly understand what you're talking about and accept it? Or do people feel confused and uncertain and don't necessarily want to listen to what you're saying.

And the other thing, of course is that I think you've written about, if I recall correctly, is even a data scientist needs to be able to choose their meeting times correctly. For example, it's probably better to try to convince someone of something at 11am than directly after lunch.

Yeah, there's that famous study that I'm going to butcher, but it was around seeing who gets parole, I think. And the prisoners who got their parole hearings right before lunch did a lot worse than the people who got it right after lunch. And so just having your meeting at the right time could often be influential in the decision. And that's really infuriating when you think about data science. Right. Because data science is all about getting the best evidence and using the best techniques to try and really understand what's going on. So to find a really cool finding and then have it not be listened to because your meeting was at the wrong time, that's just unfortunate. But there's lots of these sorts of decisions that happen and being thoughtful of them is really valuable.

So you mentioned R squared, and very recently you wrote a post on Medium which actually the first image in it is r squared equals 0.042. And this is a great post about why data science projects may not work and how we should feel when they don't work. And I thought maybe you could give us a bit of insight into your feelings about this.

Yeah, so this really, this post was really kind of a culmination of a thing that's been happening a lot in my career, which was I would be given, you know, or I would have some new idea for a project. Right. Maybe it's trying to predict which customers are likely to churn at a company. You know, so I'd have some say, like, hey, we can use data science here at this company to try and predict which customer is going to churn. And we can use that, the customer's transactions and we can use when they called the call center and we can put all these things in and make a machine learning model and guess which customers will turn. And I come up with this idea and people would say, okay, great, go try it. And I'd be all excited and then I'd go try and do it and it wouldn't work. You know, maybe the data really wasn't quite there or maybe I would actually have the right data and I make the model. But for whatever reason it actually didn't work very well. So like, maybe it did turn out that knowing what transactions happen doesn't really actually tell you much about which customers are likely to turn. And what would happen to me in these situations is I'd get really depressed and I'd think, oh, if I was only a better data scientist, I could have made this work if I had had a few more techniques or if I had had a few more projects under my belt so I'd known what to look out for, you know, if only I was better, this thing would have succeeded.

And in fact, the, the top highlight on your medium post is, so you say each time I feel awful about myself, comma and then the top highlight is with a lingering belief that the project would have worked if only I had been a better data scientist. And what that says to me is that a lot of people identify with this or consider it very important to them.

Yeah. And I think if you look out, if you go and sample 100 random blog posts about data science, I pretty sure that at least 95 would be talking about how great data science is and how it's going to change the world and cure cancer and you can use data to improve your company. And it's data is the new oil. And there are all these optimistic things. And so it creates this environment where this feeling is, is that data science is easy and it's going to be very fruitful. And so when you get people trying to do these things in practice, the reality is you're doing a new risky venture. Right? No one's tried predicting churn with transactions before. No one's tried clustering these customers before. You're doing all these things that no one has actually tried before. And, and so naturally it makes sense that most of the time it doesn't work. If it did work, people would have probably tried it already. And so because of that, we have this environment where the hype is that everything is easy and good. And the reality is that it's difficult and hard. And the natural inclination when people don't succeed at problems is they don't want to talk about it. And so I feel like most people run into the situation where they try doing machine learning model, they try doing some data science and it doesn't work, but then they can't really talk to anyone else about it. And so you just assume, well, since no one else is talking about how they fail all the time, it must just be that I'm bad. But I don't think that's true. I think that happens to most of us and dare I say all of us.

And as data scientists, we should be absolutely aware that statistically it will happen to most of us some of the time.

Yeah, it's like it should happen to some of us all the time. And you know when you see those news articles about those companies doing really cool data science and everything going great, they don't make articles about all the times things didn't work. So there's just a huge selection bias going on.

And this happens in basic research as well, right? That negative results aren't publishable for the most part.

Yeah. And I feel like there's a movement to start publishing them more and you know, just getting more open about what people try. That doesn't work. And I think data science is an especially good field where, you know, we would be especially receptive to that kind of philosophy.

And in this post you also, you mention several reasons why you may not be successful on any given task. Two in particular you mention is that the data just doesn't contain a signal in it. And you give the example of it'd be ridiculous to predict the weather based on rolling die.

Right.

But you also mention that a signal may exist but your model isn't. Right. But then you chop down that particular possibility really stating that if a signal exists, when you try a bunch of models, you'll find it even if it's, if it's weak. So it's usually the fact that the data just doesn't contain the signal you're interested in.

Yeah, and that's really, from my personal experience, is that if there's some relationship, you know, if it turns out that transactions can help you predict churn, then trying like even a linear regression will pick up some sort of, you know, some sort of correlation, you know, you'll get some sort of success and then you can try using better techniques and choosing better features and you can do a lot of things to improve it. But usually the very simple approach will still work. And in fact, I think this comes from this thing I've been thinking about a lot, which is this notion that machine learning models very rarely will pick up something that a human couldn't detect themselves. And so what do I mean by that? I mean that if you took customer data, if you took a bunch of history of transactions and you said I'm going to personally take a guess if this customer number 27, if they will churn or not. If you as a human can do that pretty well, then a machine learning algorithm can probably do it too. Right? You know, maybe it's that, well, if you haven't made any transactions in a year, that's a sign you're not going to come back. Or maybe it's, you know, if your transaction are getting less in value, it's coming back. But if you can kind of just explain as a human by looking at a couple data points what's happening, then your model will probably work. Conversely, if you can't even as a human, if you can't look at the data points and try and predict what's going to happen, then a machine learning algorithm probably won't either. So let me give you an example from my career. At one point I was working for a software company and this software company would put out software and before the software was released, it would have to go through testing. And so what would happen is people would use the software a lot and it would create a lot of in app telemetry. So you know, it create a lot of data around. Well then someone clicked here and then someone did this and then, you know, the app was a little sluggish and things like that and it create like tons and tons of logs of telemetry. And the idea was, hey, what if we use machine learning and data science to try and make it so that we didn't need a human to tell if the software passed the test or not? What if we could just look at that telemetry and tell if then by looking at the telemetry, tell if it will pass or not. And the company I was working for ended up spending like two years and millions of dollars trying to do all these different data science and machine learning techniques, only to realize that it just wasn't going to work. And intuitively, if you had looked at the very beginning and said, okay, well here's a terabyte of telemetry data collected from the app, tell if it's going to pass the test or not. The tests were really complex things. They're like, the app must feel responsive and it must have no grammatical errors and things like that and things you would never be able to tell just by looking at telemetry. And so as a human, if you couldn't look at the telemetry and tell if it would pass or not, there'd have been no way. It's very unlikely that a machine learning algorithm is going to suddenly detect a thing that you wouldn't have noticed either. And so let me bring this all back. So then going back to why do the simple approaches work? Well, it's like if you with your eye can figure out right away, oh, if this is what the rules are here, I can think this Customer is going churn, or I think this software is going to pass the test. If you can do that, then a simple algorithm can probably pick it up too.

And essentially then a more complex algorithm won't just, I think, as you say in this post, won't just give you signal where a simple algorithm did not, but a more complex algorithm will give you some sort of lift or it will give you, perhaps it'll turn a good algorithm to a great algorithm, but it won't just produce something from nothing.

Right, exactly. Like it may catch the edge cases. Right. You know, you're your linear and logistic regressions, they may not be able to notice all the different possible rules and things like that, but they can pick up on the base idea of what's going on. And so if they can't pick up on anything, then it is very unlikely that by putting in a deep algorithm, a super hard algorithm, you're actually going to get a huge success.

So this has been a whirlwind introduction to, I think, a lot of your interests. What you think about. I want to step back a bit and consider you historically in some sense. I'd like to know how you got into data science originally, what your trajectory was.

Yeah. So I would say my story starts with me going to college. And I went to a college not knowing what I wanted to study. And after my first semester I realized, you know, I really like mathematics, I'm going to get a degree in that. And I had no idea what kind of things you could do with a math degree or anything like that. I knew intellectually that people said, oh, businesses hire math majors. But I didn't actually know what that was like or what the job would look like or I didn't really know anything. But I figured, I like math, it'll all work itself out. So I ended up getting an undergrad degree and I got a master's degree in mathematics too, because at some point during my undergrad and master's I decided, oh, I want to be a professor. And then later during that undergrad and master's, I realized, oh, I hate math research. I don't want to do that at all. So then I ended up working at a company called vistaprint. And this was before data science was a term. So at the time this was a role called business analytics. And I didn't really know what that meant, but it sounded interesting. And so there I ended up doing a bunch of cool stuff around creating forecasting algorithms for sales and helping them optimize their recommendation engine and things like that. So I ended up doing that for a while and then I realized, wow, having a degree in math and having all this applied math knowledge isn't that helpful without knowing statistics and knowing how to work with data and all those things. So I ended up going back for a doctorate in industrial engineering. And my particular research had to do with how do you, like, optimize electric vehicle route networks? So if you're Tesla, where do you put charging stations? And if you have electric buses, where do you have them stop and recharge their batteries? All these sorts of cool math problems, but not super related to the data science. But I was finishing my degree and then I started getting into consulting and I realized there's actually this huge opportunity where there are lots of these companies have all this data and they just need help figuring out what to do with it. And so I think by that point data science had decided to be a term. And so at that point I became a data scientist and did that for quite a few years. I eventually that consulting company I was with went under, that was unrelated to me. And then I ended up being a director of analytics at a consulting firm called Linadi, where I started the analytics team from scratch. Started as just me, ended up growing to a team of seven and then I decided that I wanted to do something new. That team was pretty much running itself. And so I became an independent consultant. And so now I am working for Nolis llc.

Great. And what do you do generally in your independent consultancy work?

So there are lots of companies that are trying to start getting into data science, machine learning, AI. And if you don't have any of that, trying to start having that is not easy. And so I help companies with that kind of process. So right now I'm helping T mobile looking into growing their AI and how can you use AI and natural language processing within call centers and things like that, and really growing that space up there.

That sounds really exciting. So T Mobile, of course, is telecommunications and we see that data science can have a huge impact in telecommunications. I'm wondering more generally, what verticals do you currently see data science having the most impact in or being capable of the most impact?

So I'm going to pivot that question slightly, if you don't mind.

No, not at all. I've got no idea where this is going to go, so I'm excited.

Okay. So the question of which verticals are right for data science, I think in some way it's more a question of which verticals are farthest along in the data science journey. So like you imagine tech companies, Google, Microsoft, Amazon, they're pretty far in, they really understand data science. They have big teams in some of the other verticals. You can imagine they're much earlier. You know, retail is getting there, healthcare in some ways is getting there, in some ways it just barely starting. It really depends. And when I think about data science, I really think it actually breaks into three separate fields. And I think for each one of those subfields it really the different companies and different verticals can approach them differently. So allow me to talk about those three different fields.

I'd love that.

When I think of data science, I think it's really three topics. One topic is business intelligence and that's really around taking data that the company has and getting it in front of the right people. So that can mean taking data and putting it in dashboards or weekly reports or you know, even in emails that get sent out every day, but really taking data and getting it to the right place. And business intelligence generally doesn't have that much analysis of the data. As a bi person, your job is not to try and figure out what it all means. Your job is to get it to the right people so they can figure out what it means. The second area of data science is what I call decision science. And I say what I call, but I didn't coin the term and I don't know who did and it's killing me. So if someone's listening to this and they know who coined the term decision science, please email me, get in touch. Yeah, so decision science is really around taking data and using it to help a company make a decision. So for instance, that could be trying to figure out, hey, which of our products is the right product, you know, which of the products should we stop stocking? Or we are noticing that this segment of customers is churning, can you help us figure out why? Or even, you know, what is the best way to split up our customers so that we market to them differently. But it's really, you know, this is really around the, you know, creating PowerPoints and you know, really trying to get people to understand what is happening from what we've seen in the data. And then the last field is what I think about as machine learning. And so this is the area of data science. That's how can we take data science models and put them continuously into production. So for instance, making a website's recommendation engine or creating the model that chooses what price are we going to quote you when you look at our website? Or trying to predict each day, trying to decide which customers are going to get the email. All these things that are continuously running, that's really like the machine learning part of data science. And each one of these fields is different and each one is important. And I think for different verticals and industries, they've progressed differently along each one of those.

And there are interactions as well. Right. For example, machine learning can impact decision science.

Yeah. And they really kind of overlap quite a bit because you could imagine that the decision science work can also influence what you're going to put in your dashboard, which relates to bi and you know, the machine learning models can influence what you report out in the decision science and the decision science folks can really influence what you know. Does it even make sense to put a machine learning model in place in the first place? I think from a skill sets perspective, they're kind of, each one is different, but they have a fair amount of overlap. So when you think about business intelligence, that's much more around understanding storing data in databases. It's understanding tableau power, BI and, you know, the right visualization approaches. When it comes to decision science, that's much more around using Python and R to be able to take a big data set and get some meaning out of it and then put it into a meaningful report. And then the machine learning component, it's much more around the software engineering. So getting it so that you're, you create a model, you test that it works, that you deploy that code into a test environment, then you deploy it into production. And so it's by far the closest to software engineering. So I think when you realize that data science is these three distinctions, it makes it a lot more approachable because I think if you lump them all together, you kind of think, wow, I need to understand software engineering and I need to understand visualization approaches and I need to understand how to make an effective report. Like, how can I, a lowly, you know, a lowly human being ever be able to do all of this and become a data scientist? But because it's actually much more specialized, it's much easier for any one person to enter this field. And usually which part of this field you're best fit for, you will just naively fall into. So people who are, you know, who really have a good business understanding and think about people and that sort of component are often the ones who end up as the decision scientists and the people who really like software engineering and thinking about, well, what's the right way to store this data in a big cluster, you end up doing that kind of work. So it. It's often naturally sorting.

Exactly. And I think that will help a lot of newcomers who, as you say, can find the world of data science incredibly overwhelming. Get started.

Yeah, I saw that. There's this discussion people have been having online around those like, this is what a data scientist looks like infographic that has like 100 things listed that the person knows. And it's like, no, any one data scientist maybe knows four of those, and that's plenty.

Exactly. The unicorn is rare. We'll jump right back into our interview with Jonathan after a short segment. It's now time for a segment called Data Science Best Practices. I'm here with Eric Ma, a data scientist and research investigator at Novartis who helped with our Python Network analysis curriculum here at Data Camp. Today, once again, we'll import software engineering best practices as Data science best practices. What are you going to tell us about today, Eric?

Version control.

Uh oh, you're talking about git, right?

Yeah. So git is one way to do version control. Sure. More importantly, I cannot count the number of times version control saved my projects from my own mistakes. I think for analytics folks there's a. There's a key version control pattern that we need to master, pun not intended, that is branch and merge.

So tell me about branches, Eric.

Branches are a lifesaver. If I'm embarking on something high risk, then I'll begin by creating a new branch in my repository. If things work out, then I'll merge back into master. But if I've messed things up somewhere in the middle, such as say, modifying a piece of code I should not have, then all I have to do is. Is go back to delete the branch, switch back to master, and I'll be back at where I first started. What branches do then is isolate my risky work from the rest of my project.

I've been saved by branches more than a few times. Also, I remember that version control systems also let you go back in history too.

Exactly. So in case I made a mistake that I needed to undo and that mistake occurred like three commits before I can simply call on git, revert and be done with that.

Just now you mentioned branching and merging branches help us isolate new work from stable previous work. What about merging?

The merge part comes from traditional software engineering. When doing an analysis, I often end up writing code. I usually strive to do the code right, that is refactoring, structuring things logically, writing tests for them, basically applying good software engineering practices to my data analysis code. At some point after branching, I want to merge that code into the project's code base. Now when merging, I'll usually send in a pull request from the branch I was developing on into the master branch. Then I'll tag a colleague or collaborator to review the changes that I've made. And through the code review process I basically get another pair of eyes on my code. And sometimes that means catching errors in the analysis that I might have been blind to. It's like real time peer review.

Great stuff, Eric. Any last words for our podcast listeners?

I think this is what I'd like to offer that is, Though we're not necessarily software engineers, I think it pays off for data scientists to think like one tested code. Logical organization, documentation of code intent. This all helps us data scientists do better analyses. And it can also really help with a handoff from data science teams into software engineering teams that put the data analysis into production.

I'll also add that we all need to persevere with Git. Sometimes it can be difficult and throw you wacky errors. Even us who have been using it for years now still struggle occasionally, don't we?

Definitely agree. From my first introduction to git@scipy until now, it's been about three years of continuous practice with my code. I think a few tips to get practice include one Practicing on toy repositories that I create locally and two that thing I mentioned on my first appearance on Data Framed contributing to open source.

Thanks Eric.

Thanks for having me here, Hugo.

After that interlude, it's time to jump back into our chat with Jonathan Nolis. So Jonathan, we've seen clearly that data science can be highly impactful, but as with any endeavor, I'd like to invoke a healthy skepticism. And I'm wondering, to your mind, what can't data science do? Or what isn't data science capable of?

So I think that there's this naive assumption that if you use data in a decision, then that decision will be better. So you know, if you are deciding where to locate a new factory, then having data on every possible bit of information you can know about all the different places will intrinsically cause you to make a better decision. Or for instance, in my last company we did a lot of loyalty program design. So knowing having all the historic data on a company's customers before designing the new loyalty program will intrinsically get you a better designed program than not having that information. So there's that assumption that more information and more data is better and therefore data science should make every decision better. But often in practice, data can sometimes make decisions Harder or get a less good solution, right? So for instance, if the data you have is sufficiently limited, then it could be that using data actually causes you to make a less informed decision. So for instance, if you are designing a new loyalty program for a company, so like let's say you're designing for Starbucks, a new program for you get a certain number of coffees, you get a free coffee. Knowing everything you possibly could about how customers behaved in the old program doesn't actually tell you very much about how they will behave once a new program is available. But if you assume that how they behave in the old program will be just like how they behave in the new program, you may make a decision that is actually less good than had you not thought about data at all.

So is this speaking to the idea that an underlying assumption of data science is that the past will be a good predictor of the future?

Yes, exactly. So that the past would be a good predictor of future. Or if you have customers that look similar to other customers, how they will all behave similarly. You know, there's, there's lots of. Actually, if you really step back and think about it, when you're doing a data science analysis, you're often making a ton of assumptions and those assumptions often have the possibility of leading you to a bad place, which is, if you think about it, can get really dark. Because then it's like, why do anything ever? Like, why should everyone even care about data science? But like, no, it's so good. But like, you just have to be careful. So here's just another example, right? Suppose you're doing an analysis for a company and you find that customers who buy Product X end up spending way more after that. So then you now have this bit of knowledge that the customers who have bought Product X spend more in the future. That piece of information doesn't necessarily tell you that much, right? Because from it you can infer that A, if we give everyone else Product X, they will then buy more. Or B, you could say, hey, all those customers who buy Product X, they're high value customers. Maybe we could get rid of Product X and it would still be high value customers. Or maybe if you took Product X and gave it to the people who didn't, maybe you give it to the people who didn't buy Product X, maybe they'd be annoyed because Product X doesn't work for them at all, right? There's all these actual different things that could come out of the analysis that buying customers who bought Product X buy more in the future and that finding that if you buy Product X or the people who have bought Product X buy more in the future, that finding could be valuable. But it could also be incredibly dangerous if you make an assumption off it that holds not true. And so that doesn't mean finding that initial analysis wasn't valuable in the first place. It's just you have to be very thoughtful of what are the conclusions you draw from it.

Absolutely. And then what decisions and business decisions to make afterwards. Because that's what's not clear in this case as well.

Right, Exactly. And I think most of the time when we use data science to make decisions, it is not necessarily clear what is the exact implication you can pull from it. Which is not to say that you shouldn't pull the data at all. It's just that it is not sufficient to have good data to then make a good decision. Back to the original question of what can't data science do? Is it often can't tell you the final conclusion you should draw? It can just present evidence.

Fantastic. So we're here today to talk, among other things, about organizing data science teams and good organizational structure for data science teams. But before we do this, I'd like to know what are the most common pitfalls that, that you've seen people make when organizing data science teams.

So I think there are a couple big areas people struggle with. For one, there's a big question of how should you organize data science within the bigger company? One approach people take is they build what's called a center of excellence, right. Where you take all of your data scientists, you put them together in a space and you say, well, the data scientists will all talk together, they'll share what they've learned, they'll really grow. And that's the best way to organize our data scientists. Another way you can do it is you can say, I'm going to distribute the data scientists. I'm going to put data scientists in each part of the company. Finance is going to get data scientists and marketing is going to get data scientists, and, you know, the supply chain is going to get data scientists. And you say, well, this is the best way to organize because those data scientists will be really focused on what's important to them and really be able to help out that particular part of the organization. So these are actually two totally different ways of organizing, and they can have really different results. So thinking about what's the right way to do it for your company is difficult to do, especially because often data science grows organically. And so to be able to coordinate Hiring and distribution is just not an easy thing to do and it may.

Change as a function of time and as a function of industry and the actual business needs of the company in question?

Yeah, it's really not a one size fits all. Like for some companies one approach works and for some it's really about being distributed and things like that.

Are there any other pitfalls that you commonly see?

Yeah, so I think similar to that is how do you actually store the knowledge that the data science team generates? People often don't think very hard about how you're going to store the knowledge that you gain from data science. So for instance, if you learn that customers who buy Product X spend more in the future, how are you going to store that so that people remember it? Right? How are you going to keep that knowledge around so that it's not lost the moment the person who did the analysis has left or the person who is the recipient of that analysis is left? And if you think about it, there's actually a lot of things that have to get stored, right? There's the things that you've learned from the analysis, there's the code to do the analysis, there's the knowledge of how do you actually execute that code to do the analysis. And so the best data science teams are the ones that can really handle the ability to have people changes, the ability to have changes in who's working on what and have all that knowledge be stored. And the worst data science teams are the ones where all that knowledge is stored in like one person. And if that one person quits or whatever, then that knowledge is lost. If you think about it like that is hugely expensive for an organization to have someone do lots of different analyses and help out a company a lot and then just suddenly lose all that the moment the person quits.

Interesting. So in a word, essentially it's how to take the knowledge gained and how to store it, but also the distribution of the knowledge.

Exactly. Companies that do this really well, they will make knowledge sharing hubs, places where when you do analysis, you drop it right in and they'll come up with standards for their code. Code always has to be stored in this particular way. I think that I really get caught up a lot is when you do an analysis, make sure that someone else can run it by pressing a single button. If you're thinking very practically, if you're writing R code and you have to run this script and, and then that creates a data set that you then run this script, and then you run this script and you have to Remember all those steps, no one else is going to be able to do that. So to be able to say, oh, to enforce a rule that hey, anytime you do an analysis, it has to be that someone can run one script in one place and it runs everything else correctly, those things can be really helpful to an organization and a data science team, but isn't something that everyone always does.

Agreed. And not only other people may not be able to run it, but me in a week or less may not be able to run what I've been doing.

Right, yeah, that's how I learned this lesson. It only takes you having getting burned yourself by this a couple of times. Also, I think 10 years ago when I started in this field, we didn't have the tools to do it that we do now. In R, for instance, you can have R code that automatically pulls from your database and runs an analysis and creates a word doc with all of your output. You can have all of that happen with a single, a single R script can do all of those things. And that just wasn't possible 10 years ago.

That's right. So having talked about the common pitfalls in your mind, what is the ideal organizational structure for a data science team?

So for me I've found that what works best is what I think of as distributing the data science within the business, but culturally being centered as a single group. So by that I mean, you know, if you're going to have lots of different data scientists, some who have to work on supply chain and some who work on marketing, like really have those people embedded in those parts of the organization. So really have them get to know the problems and the people and what's going on there. But that being said, make sure that everyone, all the data scientists culturally feel like they're connected together. So have, you know, even like small things like team lunches and quarterly outings where we talk about our career goals and you know, things like that that get them to feel like they are part of the data science team. That combination I think really works the best. So distribute the work and on a day to day basis have people embedded, but make sure you have everyone coming back as a single group culturally to keep, you know, to keep people feeling like one core team.

Yeah, that's cool. And I'm, I'm wondering whether even, you know, this idea of storing and distributing knowledge in a knowledge repository for, for example, or a wiki can actually help with this kind of culture of a single group that they're, you know, checking each other's code every now and Then seeing what each other are working on in this kind of knowledge. Knowledge base.

Yeah, exactly. And you know, if, if we're culturally in one group, then when, you know, when that group is meeting once a week or quarter or whatever, when we're meeting together, we'll be talking and I'll say, hey, let me tell you about this analysis I did. And like, oh, wow, that would have been great over here. And like, you build those, those bridges and you get things continuing to work together and that kind of cultural cohesion, like, is really valuable, especially to your point of keeping the knowledge around.

Yeah. And in this particular model, would. Let me get this right, Would a data scientist on a growth team report directly to the VP of growth, or would they also report to a chief data scientist or something like that?

So I would, my heart tells me they would report to the chief data scientist, but. But dotted line to the other group. But I think that really, like, I think in practice it really depends on the actual organization. So I think, you know, it's. This is, you said ideal and I'm kind of punting by saying, well, it depends. But I think that kind of exactly who reports to who is much more around the culture of the company you're working in.

And is there an inherent challenge here though, that if you report to someone, but there are dotted lines also that you get, you get cross. Cross lines and it can actually get confusing?

Yeah.

I think the problem is because of the field of data science, you're always going to get dotted lines. If you only report just to a data science team, you will never actually do work for people who care. You need some dotted line to get direction on what is important to the business. If you report just to that part of the business, then you will be isolated and your data science work won't be coordinated with other parts of the company, which can be really problematic. And I think just more than other fields like software development, we really in data science have to deal with the ambiguity of who reports to who.

Yeah, I agree completely. How does your distinction between the different types of data science play into this ideal org structure? To be very specific, the distinction between what we discussed earlier. Business intelligence work, decision science, machine learning.

Yeah, So I think those are really three different types of work. And I think data scientists listening to this podcast would likely agree that business intelligence, decision science and machine learning are really quite different. But people outside this field often may struggle to understand the difference. And many times those are the people doing the hiring and sending. Just to a point we're talking about giving those dotted lines of work. So it's very natural that a person who feels like they're best at decision science may be asked to do BI work, or a person who is really thoughtful about machine learning may suddenly find themselves doing decision science work. And so this crosswork happens all the time. But the more you can kind of, as a team, try and enforce a structure that keeps that work separate, the better, because I've really seen teams struggle where people are doing kinds of work that they don't like, which ends up causing them to be really dissatisfied, which causes the team's productivity to go down and everyone's just unhappy. And like, that's just not a good way to run a team.

We'll jump right back into our interview with Jonathan Nolis after a short segment. We're back with another interlude to talk Data Framed for Social Good with Peter Bull, co founder of Driven Data and believer in the power of data to make a difference.

Hey Hugo, I'm glad to be back. We've had some great conversations about the history and the context of data science for good. So now I'm looking forward to shar specific ways that your listeners can get involved.

All right, well, there's a fair number of folks in our audience that are in the process of becoming data scientists. Let's say I'm a student or an aspiring data scientist. How do I get involved doing data for good?

Well, there are a number of great opportunities depending on your availability, your location, and your interest. One great way to get involved if you're a full time student is through Data Science for Social Good fellowship programs.

What's a fellowship program?

A fellowship program is usually either a summer or semester long program where students are funded to work on a particular project. These end up being like an internship where students can work on a data science team and tackle that project. In 2013, a group at the University of Chicago laid the groundwork for this model. The program, called simply Data Science for Social Good, puts teams of data science students together to spend an entire summer working on a project for a nonprofit or government group. We've seen some very cool work come out of these fellowships, including a project I particularly like where one of the teams built a predictive model for the City of Chicago that helped them target inspections for lead paint. This is a great example of a tangible algorithm that helps use city resources that we all pay for with our tax dollars more efficiently.

Awesome. So how can people get involved in this?

Well, since 2013, a number of expansions have happened in this program. Not Only have they run in Chicago, but also in Atlanta. And currently there are programs in Portugal, which is called DSSG Europe, and in Seattle at the University of Washington. Even if you're not a student, these programs are often looking for mentors and projects to work on. So you should look at their sites, which we'll put in the show notes.

Great. So what if we have listeners that are students or learners that can't do one of these fellowships?

That's a great question. One of the great things about being a learner is that you have control over the data you choose to work with. As you build out a portfolio of analyses, I encourage you to look for data sets that focus on a social impact problem. Government open data portals, the humanitarian data exchange, and even driven data are great places to find these kinds of data sets. It's a core competency for data scientists to be able to communicate their work. So thinking about the impact of your analysis up front when choosing your data will help you to frame the value of the work you've done down the line.

Thanks, Peter. These sound like compelling ways to get involved. I'm looking forward to hearing about other ways to get involved next time we chat. Time to get straight back into our chat with Jonathan. So I think something we're circling around is the fact that data science teams and data science individuals can actually sometimes be stuck between a rock and a hard place in the sense that we do have a lot of dotted lines everywhere. And in all honesty, a lot of the time you'll have engineers on one side that you're waiting on stuff to be implemented and marketeers and biz ops on the other, for example. So I'm just wondering what are practical ways to deal with this unique position for data scientists and data science teams?

Yeah, and you know, I really love that question because it gets to the thing just so we are talking about and just as talking about the beginning. Whereas to me it feels like data science is much less about understanding complex mathematical approaches or working with giant massive big data sets and it's much more about how do you get people to work together and make decisions using data. And yeah, just on a day to day basis, as a data scientist you're going to be given questions from marketing and biz ops and you're given questions from the engineering teams and you're going to have people not wanting to do things you really wish they would do. And trying to manage all of that is difficult, but it's very much a part of the job. And the more as a field we Think bigger than just what is the next newest technique and more about how do we handle these kinds of approaches, I think the better we will be. So the question is, what is the actual practical way of dealing with this? So I have two answers to that. One is kind of a cop out. And the kind of a cop out answer is hire people who are good at this stuff in the first place. So when it comes to hiring, if you are a person on a data science team, who does hiring really try and hire people who you think would be effective at communicating with people outside of data science? Because if you're given the choice to hire someone who knows the best techniques or knows how to communicate, the person with the communication skills is going to really be the person who helps your team out. Because just as we were talking about at the beginning of this podcast, data science is about getting people to make decisions effectively. And so a core component of that is communication. So that's the kind of cop out answer, because most of us don't actually have the ability to choose who's hired. So a more practical answer for me, I found that the most effective thing is really just trusting that everyone you're working with is trying to do what they believe is in the group's best interest. So when the engineering people who you're working with are telling you they can't do a thing, it's not because they're malicious or don't like you or don't respect you. It's because they are trying to do what they think is best. And same with marketing and the business teams. And so the more you trust other people to make good decisions, I find the better things go. And another way of wording this is have empathy for people outside of data science. So really have empathy and understand, like, what are the struggles of the business person? What is that poor engineering team struggling with that they're having difficulty getting you the data? The more empathy you can employ in a situation, the easier these sorts of decisions end up being.

And I think what you referred to as the cop out answer actually plays into that a great deal. Because if you hire people who are good at working with other people, you can trust everybody as much as possible.

Yeah. And oftentimes when you hire people who are good at the communicating and good at more than just purely getting a optimal answer to a textbooks math problem, you're going to get more different perspectives, you're going to get more viewpoints, you'll have a more diverse team, and it just ends up being much more successful than hiring around technically, who has the technically best technical skills.

Yeah. And actually this discussion reminded me of several points in you've got a fantastic series of posts on medium. There are four parts of how your hiring process and how to hire data scientists. Listeners definitely check out all of Jonathan's hiring data scientist posts. The reason I mention this now is that we, when you have a section on the take home challenge, which is a real business problem that you give people during the interview process and you encourage, almost actually force them, I think to email someone on your team at some point to even see how they frame interacting with collaborators in that type of workspace.

Yeah, because you know, when we're interviewing, you know, you really want to understand how's the person going to do at the job you're hiring them for. And oftentimes that job involves communicating with others. So as part of the case studies we would give out at Linadi, I written it so that the person taking it had to. They didn't, they needed a data dictionary to be able to do almost anything. And I didn't give them a data dictionary. And I said in the instructions, if you want a data dictionary, please just email me. And so they almost universally would. And that email was very informative on how do they think about handling communication and did they make the request clear. And so that was very helpful. And that email was, you know, very much like what they would have to do on the job.

In a previous discussion we had talking about the distinctions between decision scientists and data scientists, you raised an interesting point which is you asked the question, is your product as a data scientist or a decision scientist, a model or is it an idea? And I'm wondering if you could unpack that slightly because I think that's a wonderful question that perhaps we need to think about more.

Yeah. So it's very easy as a data scientist to think what you're building and what you're doing is making models. So like for instance, maybe you're making a segmentation model or maybe you're making a customer lifetime value model or you know, whatever you're making models. And for some people that is their job really. It's just literally make a model, put in production and call it a day. Those are the people I thought about, you know, I refer to as being in machine learning. But for most of us, the job is much more around delivering an actual idea. Right. So it's not just creating a survival model and survival curves. It's really saying, okay, I have learned that these are the indicators that a customer is not likely to survive. And so I need to convey the idea of, hey, these are the signs that someone's not going to survive. And by survive, I don't mean die, I mean stop being a customer. These are the signs someone's not going to be a customer. And so when you're thinking about doing marketing, you should try and market around these particular ideas. And so when you pivot from thinking about delivering a model to an idea, then a lot of things come into more focus. Right. So just like we were talking about before, you know, how do you give the right presentation to convey that idea? How do you, how do you get people convinced that you are a trustworthy person, that your ideas are sound? You know, there are lots of things that come into convincing someone of something, and it's more than just a single model that does that.

We've spoken a lot about the different types of data science work that, that can be done. And you spoke to the fact that it's important to recognize whether people really are interested and adept and skilled in business intelligence, decision science, machine learning. However, when, when you were director of insights and analytics at Lenardi, you made it very clear that you are hiring generalist data scientists, not specialists. And I'd like you to speak a bit more to the role of generalists and specialists in data science as a whole today and how you see this evolving in the future.

So I think to the point earlier about there being business intelligence work, decision science work, machine learning work, I think it is very reasonable, acceptable, and wise to focus in one of those three areas. So for me, most of my work has been in the decision science space. I've done a fair amount in BI and machine learning, but most of my work is in decision science. And so for each person, having a specialization is generally good. But just as we were discussing earlier, what will happen is oftentimes, for whatever reason, your company doesn't have that much or your team doesn't have that much of that type of work to do right now? You know, it's like, oh, we don't have any decision science work to do, but we really could use someone to make this visualization. And so having the ability to switch between one or the other, it's not something you have to be able to do, but you should be able to want to try it. And you know, for me, a lot of my career growth has just come from a situation where someone asked me to do something I wasn't, I didn't know how to do before. And I just said, well, I'll try and I learned it. And so when I hire, I really look for people who are generalists because I think it's important that they feel comfortable that if they were given something new, they would be able to learn it. When you think about running a team, right, you could have a team where everyone's literally generalists and technically anyone can do anything, right? If you have BI work, anyone can do it. Decision science work, anyone can do it. In some ways it's very good for running a team because you don't have to worry about having. If anyone's available to do work, they should be able to do it. And as a employee, it's very good to be able to work in that space because it means that if something new happens, you can feel comfortable trying to do it. Generalization is good, but the downside of that is that people like doing the stuff they like doing, right? Some people really like making dashboards, and some people really like building machine learning models and coding them so they can run in production. And some people like giving presentations. And so if you can get your team to specialize, then you get to do give people what they like to do. And then, you know, if you like people, you should like to give them what they like to do. So by allowing for more specialization, you can then make your team happier. But then the downside of specialization is that, say you have someone on your team who likes making the presentations, you have someone who likes loading the data, you have all these different special roles. Well, now anytime you need to do one task, you have to have like five people involved, right? You have to have the data loader and then the person does the analysis and then the visualizer, then the presenter. And each time you have a new person involved, getting knowledge from one person to the other is hard. It takes time. That person loses context, and so that can make your team run less efficiently. So it's like I was constantly in the struggle to try and decide, well, how much should I allow people to specialize versus how much should I say? Everyone here technically can do everything. So I will try and give, you know, I will give someone a piece of work, even if they're not the best at it, and they will learn how to do it. And at the end of the day, I really landed on I'm going to do that generalist approach of I'm going to assume anyone can do everything. And while some people are better at things than others, I'm not going to feel uncomfortable giving something to someone a little bit out of their comfort. Zone. But I'm not sure that's the right approach. And so if a listener has some evidence that, no, here's a situation where our team, really highly specialized and it worked extremely well, I would love to hear about it.

So, yeah, as would I. And look, Jonathan, embedded in that narrative, I think, is one of the most important points for people kind of getting into data science is really to demonstrate the willingness and ability to learn new things on the fly almost constantly.

Right? Yeah. And in fact, let me tell you a story from my very first job at vistaprint. I was a month out of school, out of a master's, first real job, and I was working at the company, and they had this situation where the company, there's a bug on the website and they lost a fair amount of money. And they only realized there was a bug because the marketing dashboard went down a bit. And so marketing noticed, and then eventually they found the bug. And so a marketing executive came to me and he said, you're on the forecasting team. Can you. The bug was on a Tuesday, and he said, you're on the forecasting team. Can you pull the last five Tuesdays worth of data? Because I want to show that we can analyze Tuesdays and see that this issue occurred. And so at the time, I realized, hey, this is actually a deeply complex mathematical problem of if you have a sales forecast and every day you have a certain amount of sales come in, how can you know when sales are so much lower than baseline that it's a problem? In fact, how can you even know what the baseline is? There's weekly seasonality, there's yearly seasonality. Our sales are going up each, you know, each day because our company is growing. So there's all these different factors. And how can you tell, okay, this is so bad, we should do something about it? And. And I realized, like, that problem is way out of my comfort zone. No one's ever tackled it before. And all the executive wants is these five Tuesdays. So do I give them the five Tuesdays, or do I make this problem way more complicated and complain that we're not doing it the right way? As you can imagine, I did the latter and I complained a lot. And eventually this ultimately led to me running a team where we built this tool that would actually every day analyze all these different components and try and figure out, okay, is today a day that we're considering except sufficiently bad that we need to do something? And to do that, I had to learn a ton about statistical quality control. I had to learn a lot about different forecasting methods and I had to really get out of my comfort zone. But because I did that and because I had learned how to learn, I really succeeded in that environment. And I think that's what makes the best data scientists is the people who have learned how to learn, which is not easy. But if you can learn how to do it, it's very valuable.

Agreed. As a final question, I'm wondering if you have a final call to action to our listeners out there.

Yeah, I would say I've got two calls of action, which is maybe breaking it because you're only supposed to have one. So two 1. If you are interested in having me consult for you or your organization around data science, in particular around how do you grow, expand your team, hiring or potentially coming in with new approaches, please reach out to me. If you are not in a position where you're looking to hire a consultant, but you still find me mildly entertaining, you should check me out on Twitter. And my Twitter account is skytetra and that's sky with an E at the end. So S K Y E T E T R A and we'll also link.

To you on Twitter in the show notes all right, Jonathan, thank you so much for coming on the show and the wonderful chat.

Thank you. It's been a lot of fun.

Thanks for joining Our conversation with Jonathan about data science team organization and the ins and outs, the do's and don'ts of managing them. We saw that the most common pitfalls that people make when organizing data science teams are not thinking deeply enough about the organization, such as having data scientists distributed throughout or having a center of excellence. We also saw that all too often organizations don't have the right data engineering infrastructure in place and don't think enough about how to store and distribute knowledge gained. On top of this, we saw the absolute necessity of the softer skills in data science, such as making good slides, great figures and best times for meetings to convince key stakeholders of your work because a deep convolutional neural net isn't worth a damn if you can't convince anybody of its value. Also make sure to check out our next episode, a conversation with Amoju Miller, a senior machine learning data scientist at GitHub who has over a decade of experience in computational intelligence. Amadju and I will talk about the role of data science in product development at GitHub, what it means to use computation to build products to solve real life decision making, practical challenges, and what building data products at GitHub actually looks like. On top of this, we'll discuss GitHub as a platform for work not just technical and, as Emoju has called it, a collaborative work environment centered around humans. I'm your host, Hugo Bown Anderson. You can follow me on Twitter Hugo Bowne and datacamp at datacamp, you can find all our episodes and show notes@datacamp.com community podcast.