Best Practices for Hiring Data Scientists

Sam. Sa. Sam. Foreign.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join in. In the meanwhile though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments, depending on what platform we're watching on. And yeah, tell us something that you'd like to get out of today's webinar. If you have any questions at any point throughout the session, then please let us know using the chat in the comments as well. And I'll be saving them for the Q and A that we will host at the end. Quick note as well that the session is being recorded and the recording will be emailed to everyone that has registered for the event. If you haven't done that already, head over to datacamp.com/webinars and you'll be able to see today's event right at the top. But yeah, if not, have a look at our other events and yeah, you can find them there. I've also put a link to datacamp.com forward/webinars in the chat as well, so you can go find it there. I'll also be sharing more links to extra goodies as well throughout the session in the chat, so keep your eyes peeled in there. But yeah, let us know where you're joining from. I'll be back to repeat these messages for any new joiner shortly, but until then, enjoy the background music.

It. Sa. Sam. Foreign.

And thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about 30 seconds or so. We've just been waiting so everyone has had a chance to join. Let us know where you're joining from using the chat or the comments, depending on what platform you're watching on. And yeah, tell us something that you'd like to get out of today's webinar. If you have any questions at any point throughout the session, then please please let us know using the chat in the comments as well. And I'll be saving all of your questions for the Q and A that we'll have at the end. For anyone that's just joined, the session is being recorded and the recording will be emailed to everyone that's registered for the event. If you haven't registered for the event already. Then head over to datacount.comwebinars and you'll be able to see today's session as well as all of our upcoming sessions as well. I've put a link to that in the chat so that you can check that out as well. Keep your eyes peeled in the chat as well. I'm going to be sharing resources that are relevant for session throughout the session as well. So yeah, keep an eye out for those and yeah, send us all of your questions as well.

Brilliant.

I think that's everything from me and I think we're about ready to get started. So now I'll hand you over to your host for today's session. Richie. Richie, please take it away.

Hi, welcome to the webinar. This is Richie. So hiring is a pretty high stakes activity for any organization. So if you hire someone who doesn't have the right skills, you can end up spending a lot of money and wasting a lot of time training someone who you'd have to get rid of a few months later. And aside from the cost, that's a fairly traumatic experience for both the employee and the manager. On the flip side, if you make your hiring process so thorough that you're sure that you have the perfect candidate, then you end up with a time consuming mess that's just frustrating for everyone involved and gives the new hires a bad impression of the company. So doing hiring better is beneficial for both managers who are hiring and for the job seekers. But despite these high stakes, hiring data scientists is something that's often done pretty badly at a lot of organizations and the interview process doesn't match the skills needed at the job. So today we're going to figure out how to do it right. And guiding us on how to do better data science hiring is Isaac Slavitt. He's a co founder and the principal data scientist at Driven Data where he leads client engagements and spearheads the development of the data Science competition platform. Now Isaac's been doing some research interviewing data science hiring managers on what to do and what not to do. And so today we're going to find out exactly what he found from his research. So please take it away Isaac.

Great. Thank you Richie. Greetings to everyone in the chat. It's great to see people checking in from all over the world, as Richie said. I'm Isaac Slavitt. I'm one of the co founders of Driven Data. One of the fun things about being at Driven Data is organizing challenges that have lots and lots of data scientists from all over the world, but also Running a team where our data scientists work on projects of all kinds, all kinds of data science projects for generally large nonprofits and NGOs. But to have a data science team, we have to hire data scientists. So that's something that we do on a fairly routine basis. But also a lot of my friends and colleagues are also in the field of data science, and this is something that we talk about a lot. So I started getting interested in this topic last year, and I'll talk about why I decided to talk to a bunch of data science hiring managers, but ended up pulling a lot of those thoughts into this talk here. So the, the title today is Best Practices for Hiring Data Scientists, but there's also a sort of more negative and funnier way of saying it, which is that data science hiring is kind of broken right now. There are a lot of common practices that are just not great. They're painful for the hirers, they take a lot of time and they're really sort of distracting. And they're also a little bit hostile to the candidates as well. So there are a lot of, there are a lot of ways to think about this, and we'll talk through some of those things today and hopefully get into some actionable kind of tactics that people can maybe try out in their organizations. So again today we, you know, we don't have enough time to lay out a perfect data science hiring process from end to end. There are other resources that I can point people to that are good for thinking about how to create a new one from scratch, but there are a lot of things that we do just sort of by default that aren't as effective as they could be and that basically everyone dislikes. And so I ended up, because this comes up so often in conversation, I decided to take a more structured approach to learning about what people are doing out in the field. And again today we're trying to think about ways that we could maybe tweak our process and different things to try out. If you are starting from scratch, though, there's a great talk that I highly recommend. This is Drew Conway, who's well respected in the field and one of the early data science thought leaders. He gave a talk at ODSC 2018 about building a data science hiring pipeline to create a team. And he really goes from the beginning, like, how do you structure screening, interviews, projects, things like that. And a lot of his takeaways are things that we also came to believe independently. So it's nice to see that this is a great resource, but to learn what people are doing just empirically seeing what's going on out there. I was really inspired by this paper last year where a few mlops practitioners went out into the field and they did what they called an ethnographic interview study. So they had conversations, structured conversations with a whole lot of people in the MLOPS field. And they were able to pull out some patterns and interesting things just to sort of give a picture of the state of, state of the art, what people are doing, what they're thinking about their work. And so I tried to do something similar across a lot of different industries about data science hiring. And most of the people I talked to were data science hiring managers or they were on the talent acquisition side. But this was across finance, biotech, funded startups, things like that. And there were a lot of commonalities really quickly. There are certain things that almost everybody kind of agreed with or things that everyone was struggling with. I think before we get into this, it makes sense to just think quickly, why is this hard? Why is this harder than hiring an admin assistant or hiring a mechanic? What is it about data science that makes this so tough? A few things just in general. One thing is the data roles are highly sought after right now, or at least they have been the last few years. And so part of this is just a numbers game, like if you open up a data science role and the comp is, is reasonable and the work is interesting. Most of the people I spoke with said that it wasn't uncommon for them to get hundreds and hundreds of applicants. And a lot of the platforms now make it easy, like in order to apply, you sort of just click a button. So even just screening and trying to think about this many people is difficult for hiring managers. But another thing about this is that bad hires are at least perceived to be extremely undesirable, as Richie alluded to earlier. So in a role like this, where there's a lot of autonomy and a lot of judgment and people are sort of just, they're building things where you need to trust the outputs. A bad hire is a big deal. Like if you were to hire a dishwasher at a restaurant, if you try them out and for a day it doesn't go well, there's not going to be any long term damage. Maybe you just don't invite them back. But someone building models, someone making recommendations to senior leadership, if that turns out not to have been good, it can be a significant event for the organization. And almost every single hiring person I spoke with found that interviewing was hugely distracting for their team. Taking A lot of time. And these are data scientists and data science managers. These aren't fungible employees. Anytime they were spending on this, they weren't spending doing the R and D that really they were there for. These data skills are just very difficult to assess. And part of the reason is that data science is the center of a bunch of different fields. And so you can't just be good at math, stats, kinds of things. There's also the whole programming and software engineering side and the business side, and then communicating with senior leaders. There are so many. Each of these is kind of its own professional specialty. And resumes have just become very unhelpful in this respect. So other than looking at what the person person's title was in their last few jobs and even title that's like, it's so inconsistent between companies that that's not very helpful. You know, everyone has the same kind of keywords under technical skills. Everyone puts a handful of personal projects. It's just, it's very difficult for a hiring manager to look at the resumes and get useful signal from that. And then advanced degrees and past roles are not a great proxy for whether the candidates will be a good fit for the actual work at the organization. Or at least that's what all of the hiring managers I spoke with felt. And you know, we don't need to dwell on this, but if a lot of efforts have been made to quantify how, you know the resources involved in hiring software people or data scientists. And even if you make really, really conservative estimates for number of candidates and how long you spend per candidate and the cost estimate per candidate, this gets really expensive by almost any metric, but especially time. The amount of time that teams spend doing technical assessment and interviews is huge at organizations where they are often hiring people. So there's no sense in taking a role like data science and thinking about it in just a completely qualitative way. We want to use the skills of our field to think about this problem like a data scientist would. So, you know, you can, you can pose this task like, what are we trying to predict? Well, we're trying to predict should we hire a particular candidate or not. And you know, as we all know, this is like a binary classification problem. And we are used to thinking about this in terms of a confusion matrix where we have true positives, true negatives, and then false positives, false negatives. In this setup, a true positive is somebody who we hire, and that was a good decision. And a true negative is someone we don't hire. And that was a Good decision where a false negative is someone we don't hire, but they would have been good and a false positive is someone we do hire and that doesn't work out. Actually, important to note here, we never get feedback or labels to think like a data scientist on the people we don't hire, the people we don't hire, we don't actually know for sure whether that would have been good or bad. But this is the conceptual setup here. But these negatives, the false negatives and the false positives are not the same. And this is a huge pattern that came out from talking to almost every single hiring manager, which is they didn't feel that a false negative and a false positive were the same because this isn't. We're not just making a prediction about the role. We actually need to. We need to decide what we're going to do given all of the information we've collected. And so we really, we get some information from the questions we ask in our process and the decision we make about whether to hire or not. It's not as simple as, well, you know, are they a little bit better than 50, 50? Not like that. So again, data scientist hat this decision, this binary decision we're trying to make, there are a couple ways to think about making that decision. And if you ask like a hedge fund manager, what's the right way to make a decision like this, they would say to you, maximize your expected value. That's like, formally, that's like the optimal policy for anything. You want to maximize your expected value on average. But again, if we're, if we don't think about a bad hire and someone we don't hire as the same, those sorts of false positives and false negatives, if those aren't the same, then as humans, we don't really feel like we just want to maximize our average hire quality. What most hiring managers, from what they said, they're kind of trying to put an upper bound on the amount they regret the decision. So in other words, they're not really trying to, on average, hire people that are basically good. What they're really trying to do is they're trying to minimize the amount of times that they have a bad hire. That's important because it totally changes the way that you pose your process. And there are kind of two philosophies on this One we might call the Netflix philosophy. So famously, Netflix has really high compensation and they have a difficult interview process, but they also have this culture of firing fast. It's in their philosophy. You can read it it's written down where if things aren't working out even a little bit, you know, they give the person a severance package and they part ways. And that's just a known quantity. So when someone gets fired there, it's, you know, unfortunate, but people know that that's part of the deal. The other philosophy you might call the Google philosophy, and I'm using this as a stand in for a lot of kind of like soft environments, tech companies with foosball tables and things like that where they have a difficult hiring process. But once you're in, if you talk to your friends who work at Google, it's not like people never get fired. But on the other hand, it's an event when someone gets fired. It's not like that's happening left and right. And so people on the team take that, you know, personally. They, they start to think about whether their job's at risk and it's, it's sort of, it's a really undesirable outcome when that happens. So one of the themes that emerged from all of these talks with hiring managers is that most people perceived that they were in the sort of Google philosophy environment where if they were to fire somebody, even someone they had hired a couple months ago, that would be a really big deal for the team, you know, a bad way. And so they were very afraid of it. So you might think, all right, we really don't want to make a bad hire. Let's just make our process really hard. And that is one way to do it. But there are downsides, and we'll talk about that in a minute. So I think, and not to belabor this point, the math doesn't really matter here. But just to finish out the thought of us thinking about this like data scientists, there's a useful framework we can use that comes from the literature in our field. So if we think we're kind of designing a test for candidates to see how well they'll do once they're here. A nice analogy for that is like the SAT College standardized test or the GRE graduate school standardized test here in the United States. And the whole statistical model for this kind of thing is called item response theory. And the basic idea is that each test taker has some latent ability parameter that we can't observe directly, but we're trying to measure. And each of the questions we design, which for us would be like parts of our screening process, has some difficulty parameter. And the more ability someone has, the higher probability they have of correctly answering some questions. Question with Difficulty parameter. But I just want to note, and this is how a lot of us think about our hiring processes. You ask things that are easier or harder and the person is better or worse. This is not the model that the standards organizations use, because it's too simple. It's missing one thing. And so the next layer on this that people usually add is this thing called an information function or discrimination. And discrimination is how useful each question is in separating people who know more from people who know less. So we're not going to delve too deep on this, but I just want to grab that point because we'll talk about it a lot more in the next few minutes. Here's one quote that I just wanted to raise from one of the interviews. And this was from a People analytics partner. And so that's someone who works in hiring in HR and at a major tech company, household Name. And she said, I wish we could show a radar chart for a candidate about what was evaluated, how advanced each different skill category was, and how certain we are about that. So then we could make an informed decision based on the actual job description instead of just looking like, is this person perfect or not? And right now, we can't do that. And most of us, I would say, if we're honest about our processes right now, we can't really do this. This is an ideal state. So, I mean, you talk to people, you get a good anecdotal view of the fields. But one other thing we can do is we can just look at the literature, and there's been a really robust academic literature over the last, say, almost century about what works and what doesn't in hiring, screening. And regardless of the position, whether it's highly technical or unskilled, the research is pretty consistent in what works and what doesn't. So here's a paper that summarizes, like, the last 85 years. But it's at this point, it was in the 90s, like, late 90s. But these results replicate pretty consistently. And they had this validity measure, which is like how well you predicted whether someone would be a good fit for the job. And I just want to point out a couple of things here. First of all, the highest thing on this entire list is called work sample tests. And that's a lot of what we'll talk about in the next few minutes. So we'll get into what that is. But that is the most predictive of whether a person will be successful in the job. Another thing I want to point out here is one of the things that's Very high up there is employment interviews structured. And another thing I want to point out is they also have employment interviews unstructured. And that's like how well planned and sort of replicable the flow. And what gets asked is structured, has a high validity, unstructured has a much lower validity. And then job knowledge test is another thing that's way high up there on the list. So that gets into like the knowledge questions. We'll talk about that too. Before we move on, I just want to note a couple things. First, there are a lot of things on here that are very commonly used, maybe by a lot of us, like job experience, years, reference checks, biographical data, very low validity, a lot of those things, years in education, low validity, just throwing that out there. And then GMA tests is high. That is like intelligence tests, depending on where you are, that may or may not be legal. It's not a common practice here. And so we don't do those. But they're also kind of just a proxy for whether the person is good at doing the relevant tasks for the job. So. And then this pull quote from the famous Daniel Kahneman and a couple of other co authors in a recent book. Research has shown that work sample tests are among the best predictors of on the job performance. All right, let's talk about that then. So work samples, code tasks and knowledge questions are basically, this is what people do for technical assessment in our field. And these are the three kind of broad categories of what we ask candidates to do. We'll just take them one by one. First, work samples. If you go online and you look at what data science people think about work samples, think about our field. They are extremely polarizing and divisive. And so the thing that most people think about right away is take homes. And that's like giving a candidate like a four hour kind of realistic task, maybe realistic, and asking them to do it and submit their results. And this is incredibly divisive online. Some people absolutely hate them, some people love them. There's also kind of a perception that these take homes are free work. I think it's a little bit naive for people to think that a company can get like useful results by parceling it out in dribs and drabs to candidates. But that is a, it's a widespread perception that take home is free work. It's certainly a demand on people's personal time. But there's also this perception out there that if you ask someone to do a take home, you know they're, they're being exploited in some way. However, some people, like this person, say that they love take homes because they really get really anxious during interviews and if they're put on the spot to code, they don't perform well. And I mean, that's like, that's something that we don't necessarily want to screen for because most of our jobs are not like, we're not, we don't have a whole room of people watching us while we try to do something. And so some candidates really prefer take homes. They prefer this asynchronous approach. Some people are conflicted. I mean, there's a lot of, there's a lot of opinions on this, but basically you'll see a lot of people on both sides. But the people that don't do well on live coding tests, their basic position is, I'm great at my job, I would be so good at this role. But by asking, putting me on the spot to do a quick, you know, task that I'm not prepared for, I won't do well. Some people just think they're, they're bad. So that's, that's the Internet's mental model, which is like you either do take homes or you do live tech interviews. And when people talk about it, that's how they tend to frame it. However, most organizations don't do just one or the other. And even organizations that do take homes tend to have some component of live tech interviews and vice versa. So first of all, this isn't correct at all. I mean, just by going by what people are doing out in the field, it's usually a mix of some kind. Some places don't do take home assignments, but if they don't do a take home assignment, it's almost always the case that they have hours and hours. And at some places, like if you interview at Google or Amazon, it could be like six or seven hours of live interviews, some of which are coding, some of which are, you know, like whiteboard puzzles and things like that. And so either way, it's sort of a big imposition on the candidate's time and it's hard. But there's another, another aspect in which this is wrong, which is that take home assignments are not the only kind of work sample. And that gets lost, I think, in this debate. So one of the major takeaways that I just want to raise from a lot of these interviews is you can kind of think about work samples as breaking down along two axes. So one axis in here, it's the Y axis, is whether you actually have the person implement something all the way like in code. Or you could have them sketch it out either verbally or on a whiteboard or something. Then another aspect here broken down on the X axis is, is this synchronous together at the same time or asynchronous? So obviously a take home assignment is full implementation, but asynchronous. But there are lots of things where you could give someone a little realistic task and you do it together. And it's not necessarily that you stand there watching them with like a sort of neutral expression while they struggle on something. Sometimes you can just pair on it so you can work on it together as if you were colleagues, which is really what you're interviewing the person for, to be a colleague. Another thing that hiring managers raised as being really helpful was you might not ask the person to implement something in code, but you give them a realistic setup and then you have them talk through all of the considerations and sketch out a work plan. That's a great way to get at the more wisdom and judgment and experience aspects where you can really start to see by how a person talks through an end to end data science project from elicitation and scoping all the way through implementation, validation and maybe even deployment about whether they've been through this before and whether they know what to think about. Another interesting thing here is one of the people I spoke with who's at extremely large post IPO tech unicorn that everyone here has heard of, probably a lot of people use. He said, I find that the conversation that takes place with a candidate going over their case study is the most telling part of the whole interview process and it's not even close. Digging deeper to understand the thinking and asking hypotheticals gives more insight than everything else combined. And I tend to agree. In our process we have a case study, but it's really the conversation after where you can switch things up a little bit and talk about rationale. What are some of the pros and cons for the decisions? What are some other ways we could do this? You use this model kind of how does that work? And you can, you can start to really get a sense for how people work through a problem and not just look at the work and kind of say, yeah, I like this, this is how I would do it, or you know, this isn't up to my standards. So really when you think about it like each step of the entire process you can turn into more of a work sample rather than just like a sort of canned question or puzzle. So if you're working on code, maybe you pair program Something as if you were working together already. Like think about the way that you work with your colleagues or your direct reports and maybe you try to set things up so that you're kind of working that way together. If you do code reviews in your organization, you might consider having the candidate either review some code or you both could review code that the candidate wrote or something like that. So again, taking something that's more of a performance or like an unusual thing for a person to be asked to do and taking it into more of the what do we do on the job and how can we make this represent that accurately? And then these sort of project design interviews where again you, you talk through things or you sketch them out without having to do them. This is something that we do all the time on our data science teams. And so getting a sense for how a person approaches this rather than just a static like, you know, how do you solve this kind of programming puzzle? And we'll talk about those too. But one thing I cannot emphasize enough is this aspect of skin in the game. So one of the things that drives candidates crazy is when they spend their time, their personal time, sometimes it's family time or time that they should have off or whatever, doing like a take home or something or an all day interview on site and then they just don't hear back. And so one way that companies can address this is, you know, committing up front, putting some skin in the game. Like if you do our take home, you know, you spend your four hours, we will go over it with you. You know, obviously like if it's not plausible or not even close to being to standard, you have to use your judgment. But basically like if you spend this time working, we will work with you. Some companies even famously like pay people. Like if you do our, we'll pay you for the time for the case study that we asked you to do. That's pretty hard to do at a lot of organizations just because there's accounting stuff for paying know, miscellaneous people. But the, the idea is more important where it's like we respect your time as a candidate and you need to be so careful about using these take homes or work samples as the top of funnel screen, like as the first layer. Like if you ask 200 people to spend four hours each, you've just sort of used like 800 hours of people's time out there in the world. And people never forget sort of violations of this skin in the game rule. Like it's not uncommon to see when companies get discussed on Hacker News or Reddit or something. They say like, hey, my company's hiring, you should, you know, email me. Someone else will comment and say, I did their take home two years ago and they never replied back. And companies acquire reputations for this kind of thing. So if you're going to ask people to spend their time, this is just sort of a public service announcement. But it's important to have skin in the game. So those work samples. Let's talk about coding questions and, and coding questions are fair, they're in scope. Because at Driven Data, we've actually given sort of a talk over the last few years about how data science basically is software, a specialized kind of software, but it is software. Data scientists write code. We do it for a purpose. We don't do it just to build software systems, but that is what we do. One of the things that we do, and it's one of the important parts of it, it's how we actually implement the things that we're trying to do. So code questions are in scope, but that doesn't mean that any kind of coding question makes sense. So here's one thing that was so common, actually, I don't want to put a number on it because it's not exact, but I would say like at least half, but probably 2/3 of the hiring managers I spoke with said something like this, which is right now we mainly ask CS algorithms and coding questions like puzzles. And in our Slack hiring channel, people are complaining about lack of consistency or like, none of, you know, something other people said was like, most of the candidates don't do very well on this and this is such a shame and a big missed opportunity to use the time productively in our hiring processes.

So.

You may be wondering, why does everyone do these data structure and algorithm questions? Well, who doesn't love data structured algorithm questions? And the answer is nobody loves these really. So what they come from is in the software engineering field, these questions are incredibly common. And even software engineers hate being asked this question. This stuff like, here's a red black tree. How would you walk the nodes on this tree structure or something like that. Or sorting questions. How would you implement bubble sort in C? So software engineers hate these questions. And in software it's just known that before you do interviews, you need to grind leetcode. Leetcode being the site where you practice doing all these puzzles with like dynamic programming and recursion and tree walking and all this stuff. Software engineers hate this and that software where you could even make an argument, you could make the argument that CS fundamentals Or like data structure and algorithm swap is very relevant. A lot of software engineers don't agree with that. But for data science it's even worse. When was the last time a data scientist needed to reverse invert a binary tree? It doesn't come up. It's not a good signal for whether someone is a data scientist who gets things done and you know, knows the things that we need to know. But this is a major category of question that companies are asking. And the why is because other people do it, they're used to hearing about it. There are platforms that have these question banks and things like that. And no one would ever kind of like get criticized for doing this just because it's such a common thing. But it's not good. We really shouldn't be doing this. There's this famous case where Max Howell, who is a software engineer, he wrote the homebrew package, which is like a open source package management for Mac os. And he interviewed at Google where lots of people use his software and they asked him sort of trivia CS questions and he was upset by this. This started a big discussion in all the usual spots about whether these are good questions. He in data science it's even worse, but we still do it and it's just not. It's not good. So what should we do? Actually this is just. I just wanted to show. This is. I think this is a software engineer and he was saying that asking these puzzle and whiteboard questions turns me off to the company. It makes me think that they don't know what they want, they don't know what they're doing. We have other options. This is going back to the idea of the work sample. Why don't we turn. As long as we're going to have the person coding, why don't we turn that into another work sample, which again is the most proven way of figuring out whether someone's good for a role or not. Work sample. Let's make it realistic. Instead of asking them to invert a binary tree or do a dynamic programming optimization problem, what if we had them write code to explore data, to visualize data, to build models or even another thing that's interesting that I just want to surface as an insight. Some places incorporate a comprehension part of this where you're not trying to see if the candidate remembers all the Pandas syntax, but you have them look at another piece of code from another data scientist and understand what it's doing and spot any potential sources of danger, ask them to explain simple to medium piece of code and Ask them to do something realistic sometimes. And this is actually really cool. One thing I just wanted to flag for this group is some places have the interviewer write the code while the candidate leads the thinking. Another way of. In the workplace, you don't need to remember the syntax for like a window function when you're writing a SQL query or something, you could look that up. And so if the interviewer just basically wants to know whether the person understands joins and how to write queries and things like that, the interviewer can drive and have the candidate walk them through it. Yet another way to make it so it's not putting the person on the spot about whether they remember all the trivia about syntax and stuff and seeing whether they can think through data science problems. So in these three categories, work samples, code questions, there's also knowledge questions. And I'm including in knowledge the sort of behavioral stuff where people ask about past roles, ask about experience questions, but also specifics like how does a gradient boosting model work? So again, here we need to go back to what the fundamental problem is that we're up against. And it's this crazy Venn diagram. So, you know, it's hard if you're hiring for just one of these ellipses. If you're just hiring software engineers, there's a lot you can ask about software engineering, best practices, version control, object oriented programming, the list goes on. For business know, translating business requirements into implementation plans or things like that. Or statistics. There's a, there's, that's a whole field, you know, there are just statistics roles where people ask only statistics questions. We somehow have to ask some portfolio of questions that gets at a lot of these things. And it's very difficult to figure that out. But we also need to be conscious of a certain aspect of this, which is that all these ellipses are kind of fair game. And so there's this whole category of questions that I would call theoretically in scope. I'm putting one up. Doesn't really matter what the specifics are here, but this is a statistics question you could ask. And theoretically it's fair because statistics is one of the things data scientists should know a fair amount of statistics in order to know what they're doing so they don't use the wrong models or violate assumptions, things like that. So theoretically you could ask kind of anything from the statistics literature that's not insane. But here's a question where it's like you ask someone to define or like write the equation for a consistent estimator of some Parameter. All right, again, theoretically, it's okay, but this is the kind of question that candidates hate. I would hate it. I mean, I can answer it kind of like, I get the idea, but I'd never be able to write this equation up just from memory. You know, maybe I could verbally talk through it a little bit, but it's putting me on the spot for something that never comes up in my job. And as long as I have the intuition for it, this is the kind of thing that I could look up. So candidates hate this, but it's also not a good question, because you either know it or you don't. And if you don't know it, which is, I'm guessing, fairly common, you ask this to, like, nine or ask this to 10 data scientists. Nine of them probably wouldn't just be able to spit it out because it just either you either know it or you don't. It's just not. It doesn't add that much information to the process. Going back to that idea of discrimination, we only have time to ask a certain portfolio of questions. You know, thinking about this, like giving someone an SAT test or designing an SAT test, what do we ask? What? Not just is it hard or not, but how much information have we added by how they answer this? And so we like to. I sometimes refer to this as, like, trivia questions where you're really testing. You know, is this something the person happens to have focused on a lot? Is it something they've encountered recently? Like, do they know the trick? If you ask them a certain question they've heard recently, like, they know the trick. They could just answer it. And it's sort of like if you're an admissions officer at a college, you could get rich report cards, which are like, you know, in all the STEM classes, the person excelled, but then in these humanities classes, it was a different story. And just saying, pass, fail. Okay, well, assuming the person didn't fail too too many classes, which they probably didn't, that doesn't tell you very much. And we're data scientists. We don't want lossy information. We want rich information. So we should design questions that give us rich information, even if the person doesn't get it right. So I would argue that if you're going to ask these questions, which is fine, you know, we probably should have some kind of statistics quant component to our interviews. Ask for questions that you can really dig into and that someone can grapple with even if they don't get the right answer. There are multiple levels to it where they can have the intuition for it. They can do some work. Maybe it doesn't work out to the right number or something, but you can kind of see how they approach it. So here's a question. I won't read the question, but it's an interesting question. Like, there are a lot of different ways you might think about tackling it, but you'd get it. You'd get an idea for whether the person can think about distributions. You get an idea for how a person translates a sort of complicated, you know, technical posed question into more of a like, how would we work this? What's our attack on this? And even if they don't get the right answer, which maybe a lot of people wouldn't, you see how they work through a problem, how they can communicate where they are, how they're thinking about it. This is the kind of thing that is both difficult and rich, meaning that it doesn't really matter whether they get it right or wrong. We learned something through the process. And all of our questions can be made like this. The coding questions can be made like this. Like, you can see if you design your questions, you know, it's not just whether the person got the right answer for, like, the bubble sort. It's how did they go about it? Did they add tests? Like, all of those things can be rich sources of information if we aren't just treating it like a trivia test. And these layered questions going back to what that one person said they really wanted about each candidate, this gives us more of a holistic view of the candidate. So these layers of nuance let us actually look at strengths and weaknesses and not just like, is this person perfect and did they answer everything correctly? So, first of all, this is better for comparisons. If you have two candidates and they both got a question right or they both got it wrong, that's kind of the end of the road, at least with regard to that question. And then you're like, just counting, did this person get lots of things right or not as many. Instead of saying, like, this person was able to kind of talk through the problem. They had good intuition about setting it up. Like, they didn't realize that it was this particular distribution they had to use, but they would have gotten there, especially with a hint. Or this candidate really struggled just to kind of, like, reform the question as a way to solve. Those are the kind of comparisons that let us think about the actual job and say, like, in this role, the person will have to. They'll have a team of technical experts. They can rely on for like minutia. But what they really need to be able to do is have a good sense for how to set up the problem and have good communication skills to make sure they're understanding the problem correctly because they're going to be scoping it for others to work on and actually get into the weeds of what the job description is and what we've learned from these questions that we're asking in our interview process. And finally, it's better for us holding ourselves accountable where we're not just every single time we do this, looking for the perfect data scientist that has 20 years of experience and knows absolutely everything. And that's where you know, if someone has good intuition, I'm just going to make this up. But let's say they have good intuition about the concerns about putting something in production. That's much more important to me about whether they've used Google Cloud or aws. Like have they, you know, a smart person who has the good intuition for the basic ideas should be able to learn using S3 rather than blobstore on Azure or something that's small. Those kinds of keyword questions are not relevant and we're doing ourselves a disservice by wasting our time when we over focus on that. And we're really doing the candidates a disservice. So what we need to do is figure out how we can get these nuanced views and, and the answer, as always, and the kind of argument I'm making here and the academic research is making is we need to figure out how to turn our trivia questions into work samples. Everything can be turned into a realistic work sample if you're kind of crafty about it. Finally, and this is the last point, we kind of touched on it, but candidate experience is incredibly important and incredibly under discussed. So here's VP of Computational biology at a really big life sciences company that probably most people here have heard of. He said Another thing we haven't solved is how to get people excited about the role once offers are extended. And I would say from these conversations with like 40 plus hiring managers, one of the biggest frustrations after you know, sort of like sheer number of applicants and you know, all the time that was getting spent is once you have found people that you are excited about hiring, you've invested so much time in this person because you also have to sort of average out all the time you invested in other people that didn't work out. And so once you've invested all this time, if the person doesn't take the Offer that is such a bad outcome. People hate, hate that. And they hate it for obvious reasons, because it feels like a waste. The candidate obviously spent all this time. You spent all this time your team could have been doing other stuff and they don't take the offer. And you know, it's just unfortunate, but it's not just unfortunate. It's not like a luck thing. There are things we can't control. There's personal stuff. Maybe the candidate doesn't want to move, maybe they're not ready. There's kind of logistical stuff like comp. A lot of times we as hiring managers don't control that. You know, it's like an HR thing. We have a band that we can be within. If the person got another role that was double, you know, that's too bad that happens. It's not really something we can do. But a lot of times, and this is highly underestimated, the candidate is highly factoring in the process that they have just been through. Because while you are simulating them working with you, they are simulating what it's like for them to work with you on your team.

And.

And the only experience they have typically is through this process. And if the process is incredibly hostile and just feels bad, even if they did well enough to get an offer, they will factor that in. They can't not factor it in. So earlier we posed it like this, which is, do we hire the person? Do we not hire? We're not in control like that. We only get to choose about whether we make an offer or don't make an offer and how we run the process. People really need to think about this. I don't want to belabor this point too much, but it's not just like we're looking at them and saying, are they good or not? They're looking at us and saying, are they going to pay me enough or not? They are forming a mental model of how you think about them, which means that if you ask all hostile hard questions and the interviewers are all kind of like come off as. Even if they're not trying to be, if they come off as patronizing, that will have an impact on whether the person accepts or not. So all of this, again, work sample. Make it a good simulation for them about what it's like to work with you as colleagues. It's better for you. You get better information by turning it into a work sample. And it's better for them just to sync up, make sure we have time for some questions. What are the key things here? First of all work samples. I've said it a thousand times, we're all tired of hearing it, but it's true. The academic literature supports this. Anecdotally, if you talk to hiring managers, they prefer this. Even a lot of people feel guilty about their process because they're not doing this. It's not that hard to take. Oh, we ask three statistics questions and try to find a way to turn it into a little bit more of a realistic exercise. Sometimes that's the form. Do you talk through it, pair on it, rather than just putting them on the spot? Sometimes it's the question itself. Can we ask something that's a little bit more relevant to the task at hand? Or give them a real world example that gets at the same statistical concept that we're thinking about here. Again, the follow up, the talking through how they approach it. Don't just look at this and say, I don't like this. This isn't how I would do it, or oh, this is perfect. It's like, what were they thinking? Because code standards, like whether the person does the right spacing for the equal sign, that can be taught. That's a silly thing to look at and think about. Will this person work out on our team? That's a cultural thing of like, do they come from an academic lab where nobody cared about spacing and stuff like that? That's easy. That takes like two days of pairing or something to get over. We want to think through the thought process, choosing these rich layered questions that give us more information, that add that discrimination parameter, that aren't just pass, fail avoiding trivia questions. And especially, guys, we've got to stop asking data structure and algorithm questions. For data science roles. It is only a filter. So in other words, if a person can answer a really hard data structure and algorithm question, I mean, they're probably a good programmer, but the inverse is not true. If they can't answer it, it means nothing. It's not a good question. Choose something else. Choose something realistic. Then we can flip the script, reduce the pressure, reduce the anxiety. Maybe you drive for some of it. For the coding part, we're not testing people on whether they remember the partition syntax. We're testing them on whether they can do this work for real. Ask design questions to get at that communication. Can they, you know, do they ask follow ups? Do they get the real picture or are they just spouting stuff, you know, that they think you want to hear? And interview them like you would interview a colleague. Because that's the role that we're thinking about we're trying to see if they would be a colleague. So don't make it a stilted, you know, I'm testing you because I'm perfect. Interview them and set up the interviews as if you're working together. Finally just kind of throwing this out there. We help organizations hire data people and we've also built some tools that we're kind of experimenting with to make this technical assessment faster and easier. So if either of these things are interesting, definitely feel free to reach out. But other than that, Richie, if there are questions, I'd love to field those.

Thank you so much for that. That was really informative. We got a ton of questions from the audience. For anyone who hasn't asked a question yet, then please do ask them now. I have to say, I did love that you were talking about item response theory earlier on that section. So all of data camp skill assessments are based on item response theory. And I've never quite made that leap that testing skills is exactly what you do in job interviews. So obviously item response theory is going to be a thing there as well. All right, so let's go to audience questions. So the first question comes from Harry Nebas saying, what are the new skills needed? So, yeah, generative AI big thing. Has this changed? Has this changed what you need? What skills you need?

I mean, that's a really good question and I'll answer it from the perspective of a data science generalist, which a lot of us work on teams where we're not LLM specialists. Only if you were hiring someone for to be like a prompt engineer or something like that, it would be a different story. But I need to draw a distinction between machine learning engineers, like people who work at Google Brain or a lot of these new startups that are all AI companies. If someone, you know, the big sort of branch here is, is the person sort of like in the Weeds and Pytorch code fine tuning models or are they working with LLMs as part of a larger kind of data science thing where it's really more, you want a generalist who understands this and if it's a machine learning engineer, I'm not really the person to speak to that. But you can really get into like, you can get into the weeds of how transformers work. You know, all the aspects of, you know, going from how things change from Bert to now and, you know, asking them some of the hard kind of mathematical questions that you would ask someone who needs generally to have a PhD or at least some research background for the data science generalist roles I would really focus more on intuition and engineering. And so if they have a sense for how transformers work, what they're good at and what they aren't good at, how this fits into a more holistic picture of a data science team, what are some of the considerations they would have? Let's say you can give them behavioral questions. Like, let's say our CTO comes in and he just met with the board and everyone's excited about LLMs. We want to incorporate LLMs into our product. As a, as a data scientist, how would you respond to that? That's important because that's actually happening a lot. So, again, taking it from the realm of questions that you could theoretically ask, like, I could theoretically ask a candidate about, like, what all the different parts are of the transformer blocks. But is that the job they'll be doing? Always back to the job they'll be doing. And if the job is knowing about LLMs enough to know, you know, what's a good idea, what's going to take, you know, two weeks to prototype versus what's like a team of people working for six months, those are the kinds of things you would suss out if you were talking to a data scientist about this.

That's really great advice. And again, it seems to be going back to the idea that make it relevant to the job and don't ask people about, like, what, what's a transformer algorithm? Nice. Okay, so next question comes from junior saying, for senior hires, how do we get more signal from noise, particularly if they got company projects that they maybe can't share, actually. But that's an interesting question in general is what happens when people don't have this public portfolio of work because everything's private and confidential.

So I will say that the answers were all over the map from the hiring managers that I spoke with. Some liked looking at personal projects, some would have liked looking at personal projects, but they didn't because they couldn't do it. For the number of candidates they have, this may be a hot take. I don't really know, but I don't like doing this because. And it's sad because I think a lot of people spend a lot of time on, like, a portfolio or they're given the advice that you should have personal projects. And, like, if you've done pro bono work or you done something in your free time, by all means, like, it's great to put on the resume still, because then when someone's talking to you, they can ask you about it, and that gives you a chance to demonstrate expertise and it gives them a chance to elicit your thought process. But I don't like looking at personal projects because it's not fair. First of all, that's the big thing. There are a lot of people who work for companies where the work doesn't transfer and people have families, people take care of parents. You know, people have lots of stuff going on. So the fact that someone doesn't spend all of their free time doing data science shouldn't overly lead me in one direction or another. You, you don't have to ignore that. Like if someone's a prolific open source contributor, like that's, that's really cool. You, I mean clearly they know how to use version control and they're good at programming. If they do this a lot and their pull requests get accepted, don't ignore that. But it's not, it's a fairness issue. And then also there's like a sample bias thing here where people are only showing you typically the things that, you know, they feel are good enough to show. It's never clear how much of that, a lot of them are school projects. I don't know whether they did a lot. You know, I don't have time to go through and look at the git blame and see who wrote every line of it. And so I don't want to get the wrong idea. And ultimately I'm going to have to rely on my questions anyway just for standardization. So I really, you know, I advise hiring managers don't ignore any information. We'd be bad data scientists if we do that. But don't over index on it. And it's, it's kind of impractical anyway because there are usually too many people to look at.

Okay, I guess I have a question for you then. Related to that is, is this something you want to be looking at at the start of the process as like a sort of casual filter or is it something that comes towards the end?

It's something that I use as a tool in you're talking to people. A lot of times part of the process is a screening call, like 30 minute get informational. And that's a great time to just ask about past work. Or if they don't have personal portfolio projects like on GitHub, you can ask them what they do in their current role, not violating any whatever confidentiality, but have them talk through things. It's great there. But making it a part of like a standard part of the screening is hard to do and just not, I mean, I've found, and a Lot of the hiring managers I spoke with, they'll use it as a positive signal so it can help the candidate. But you never want to use it as a, as like a screening tool, like, oh, this person doesn't have GitHub. Well, there are a lot of reasons for that. So you know, that's my advice.

All right, so next question. I think you covered this a little bit, but just want to recap how you think about take home stuff versus live exams.

First, let me just say what we do, which I think works pretty well. We have a take home based process and if you do that, the take home is great. You see how people solve the code, everything like that. I love talking about the code. Make a real time aspect to that as well. Because the hypotheticals you ask where you've seen how they can do something asynchronously using all the resources, the time when you get to ask more questions to see just what people know from their past experience and judgment is like, oh, this was posed as a classification. What if we instead posed it as a regression? How would we have to change the setup here? Like how would you approach that hypotheticals like that or you decided to use like a random forest here. What are the, you know, what other models would you consider and what are the kind of pros and cons? How would you think through that if that was a task that you were assigned? So you, it's not either. Or if you have a take home, you can still have like a real, real time component or you can have a take home and then a real time component where it's a smaller task. But for places that are doing a real time component. The only thing I would say is it's easy to drastically overestimate the amount of progress someone can make on a new task that's been posed in like an hour. And so you really want to make it make the task kind of simple to medium. You want a task where there's a straightforward core of it, but there are a lot of like cool ins and outs that the person might uncover. And so you want it to be unlikely that the person just can't make any progress. Like that's good information because that person might not. If the task is fair, that may be not a good fit, but you want it to be. You want the person to kind of quickly get the idea of what they're trying to do. But if it's like a data analysis exercise, have there be some weird stuff like don't make it a trick, you know, where you hide the core thing that person's trying to do in a weird thing. But make it so that some rows are missing data. Is that realistic? Yeah, make it so that some rows have, they're not randomly different. And so if the person doesn't uncover that in the hour, it's okay. But if they do, you're like, oh, they really know how to interrogate a data set. So scope down, but add layers. It's all about the layers.

I like that layering approach. Yeah. So some easy things, some extra hard stretch goals as well. Excellent. All right, I think we've got time for one more. This one comes from Alex. So how does bias influence your decision? Can you imagine a data scientist in his 70s or a young woman who's expecting to work with a young, a young stamina team? People are working hard. I think that means. Yeah. So yeah. How do you think about bias in hiring?

So the way I think about bias, which let's all agree that's undesirable, the way I think about bias is the more contrived the things are, the more biased they will tend to be. Because you want to select for bonafide job specific job related skills and characteristics. And if you're somehow selecting for age or gender expression or whatever, that's not a good test. It just doesn't work well. And again, I hate to sound like a broken record, but how do we not select for things that don't make sense? Well, it's designing screening steps and assessment steps that are as related to the work as possible. And the research suggests this, but anecdotally, like everyone just knows this to be true. And that's why it's still so frustrating to me that places, they're like, yeah, we want to have a work sample. But like, nobody has time to design. These questions don't have to be crazy, like grab one from the Internet. Because the, the idea is how does the person approach a problem? It's not like can they solve some insane unsolved research question in two hours. So if you want to get rid of bias, make everything about the job description and design accordingly. Not about, you know, how recently was someone in college so they had like a linear algebra class. Do they know what they need to know for the role and nothing more. That's, that's all I would. There's no way to solve that, unfortunately with humans. But there are also some tips and tricks. Like if you're circulating a take home to a panel, there's no reason to include the name. Right. Why give people other Sort of external, illogical, sort of like things that can introduce bias, just blind it. There are some companies that make it a practice that when they send out the candidates, take home to a couple of data scientists. The data scientists don't know who it is. They can't look them up on LinkedIn, see if they went to the same college, stuff like that.

That idea of blinding stuff just seem very useful for reducing bias. All right, brilliant. I think with that, we're going to wrap up. Before I do, I just want to say we've got a break from webinars next week, so we'll be back again on the 18th with a session on AI. So best practices for developing generative AI products. Hope to see you then. Please go to datacamp.com webinar webinars. Thank you again, Isaac, for that. That was like, incredibly informative. Like there's so much you can do wrong with hiring that it's good to avoid all. All those problems. Excellent. Thank you to Reese for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. I will see you again in two weeks. All right, thank you.

Bye. All right.