#38 Data Products, Dashboards and Rapid Prototyping

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Tanya Casciarelli, a founding partner of TCB Analytics, a Boston based data consultancy. Tania started her career in bioinformatics and has applied her experience to other industries such as healthcare, finance, retail and sports. We'll be talking about what it means to be a data consultant, the wide range of industries that Tania works in, the impact of data products in her work, and the importance of rapid prototyping and getting MVPs or minimum viable products out the door. How does Tanya balance the trade off between rapid prototyping and building fully mature data products? How does this play out in particular cases in the healthcare and telecommunications spaces? How has her ability to do this evolved as a function of open source software development? We'll also dive into how general data literacy has evolved, how it can help decision making in business more generally, the data science skills gap, and how many data science hiring processes are broken and how to fix them. I'm Hugo Baun Anderson, a Data Scientist at DataCamp and this is Dataframe. Welcome to DataFrame, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on TwitterGoBown and DataCampataCamp. You can find all our episodes and show notes@datacamp.com community podcast. Hi there Tanya, and welcome to Data Framed.

Thank you Hugo. Glad to be here.

It's great to have you on the show and I'm really excited to talk about your work in data consulting, data data products, and especially this idea of rapid prototyping that I know you're a huge proponent of. But before all of that, I'd like to find out a bit about you. So maybe you could start by telling us what you're known for in the data community.

Sure. What I'm known for in the data community. It's Definitely a loaded question. And it's one of those things, where have you ever done what do people think you do and what do you actually do?

Exactly.

So I think some people know I was a pretty early R user in the community. I got a little bit lucky in that I was an undergrad student at Northeastern and I worked on a co op, which is when you work six months and then you go to school the other six months. And the PI at the lab I was working at said, you're gonna learn this thing called R. And this was back in 2005 or six. And I had no clue what it was. And There was no RStudio, there was no data camp, which makes it incredibly easy now to learn it. But I printed out the entire CRAN documentation, which is like, I don't know, 250 pages at the time or something, and brought it home over my summer break and just read it and thought I'm in deep trouble. But sure enough it worked out. I had some really cool mentors that were just super smart and got to publish a paper of some of the work I did learning R. Just basically diving in head first, blind, and it turned out to be a good thing to learn.

Great. And what do you do now?

Yeah, so the other thing is that I started a consulting firm back in 2015 after working at mostly startups, I worked at one large company, Biogen, and I think both were very different and fun, wild experiences, but I always wanted to do my own thing. And so after seeing so many different use cases and vendor pitches and just what can be done out there and how much opportunity there is, it just became a no brainer. As I was consulting on the side, it became too much to have a full time job and consult. So I just made the leap and here we are three years later.

Fantastic. And so you've given us a bit of insight in there of how you actually got into data science, but maybe you can tell us a bit more about your history.

Sure. I always knew I liked computers and technology. I was a huge sports nerd. I basically played every sport under the sun. So between basketball and video games, computer games and somehow stumbling into QBasic in 6th grade, I was definitely on my way to knowing I wanted to pursue doing something with technology, something with computers. So ended up at Northeastern University and dual majored in computer science and I got interested in biology actually, which is why I ended up in bioinformatics and biotech ultimately when I graduated. But bioinformatics was really not even well known back in 2005, there were no degree programs like there are now for it, just like there were no data science programs. And now there are.

Exactly. And I was going to say the greater Boston area is a really great place for biotech as well, right?

The best. Yeah, probably the best in the world. I mean, people between all of the universities, the hospitals, there's teaching hospitals. Yeah. I got really fortunate to be able to work at Children's Hospital for the Harvard MIT division for Dana Farber Cancer Institute. It's one of the best in the world and that was really great. So networking early on and meeting a lot of just incredibly talented people was definitely a key factor to my decision to go to Boston for school.

Great. And now you really think about business questions outside the realm of, you know, healthcare and bioinformatics and that type of stuff, right?

Yeah. So we still have a large focus in healthcare. It was kind of funny. I sort of full circle came back to healthcare after exploring some other industries. But we do, yeah, we work in, in healthcare, life sciences, but also sports, some retail and consumer packaged goods, telecom. So really every industry now is generating data. So it permeates and spans multiple verticals.

And which verticals or industries do you see data science having the most serious impact on currently and through the lens of your experience as a consultant?

Yeah, I mean any, any of the industries that are generating data as a result of just day to day business operations are always good ones and just ripe for opportunity. Because not only can you optimize their internal benchmarks, things like how can we save time on this operation we're doing, or how can we optimize what we're selling based on targeting audiences? And I think anytime you have transactional data, which is obviously very large in quantity and longitudinal and typically pretty structured, those are also really great opportunities. But healthcare in general just has so much different data across the spectrum to improve outcomes for so many people and patients across the world that that's where my passion lies in terms of which vertical I enjoy the most.

And what type of questions are you interested in answering or most relevant or is data science kind of most well equipped to answer in life sciences and healthcare and that type of stuff?

Yeah, I think it really always comes back to the data. Is it? I started out analyzing gene expression data and genetic data and back when I was doing it, the data was still noisy. There was a lot of noise to signal where we want to obviously have more signal to noise. So we actually started also looking at electronic medical record data and claims data, which is cleaner I mean, it's still messy, but it was much easier to have an impact sooner. So when you go on the research side with the genetics and genomics, there's just so much that has to get done. Even if you make a finding, you publish, hey, we have a potential new biomarker, but then what? You have to go through clinical trial and we know that those take up to 12 years and hundreds of millions of dollars. Whereas if we can say with these patients, we know that if you intervene this way, a phone call versus a text, this person is going to respond to a text and take their medication when they should. And that is something that is not expensive and is something we can do immediately to have a positive effect on patient outcomes. So it's two very different worlds and two very different sort of questions that we're trying to answer, at least in healthcare.

Great. And I'm sure you get data in all a whole variety of different forms. So is this a challenge thinking about, you know, the amount and heterogeneity of the data and how to even convey insight from that to non technical stakeholders?

Yeah, I think people still don't understand the messiness of data and how much time gets spent on cleaning it and standardizing it and joining it and what can go wrong in that process. You make one wrong join and you know you've completely inflated the number of sales or you accidentally find and replace the wrong pattern. It's just there's so many different things that can go wrong in that process that I think conveying that is still very tricky. And it's the statistic is that it's 80% or so of any data analysis project is just getting the data ready for analysis. So I think the more we can get people hands on and dirty with the data themselves, the more they'll start to understand. And I think everyone at any level, C level, entry level, should be looking and diving into data the same way that you were expected to start using email 20 years ago.

Absolutely. And I think one example that I've heard from several people who work in analytics in health is the great example is if you have a patient record or something like that, having a doctor's scribbles in the margins, something handwritten. And we all know how horrible doctor's hand handwriting can be as well.

Yes. Yeah. I mean the other problem is we've created all these forms that they're supposed to go in and fill out. You know, like EPIC has a massive software solution that doctors can go in and check the boxes if they're smoker or none and check how many packs. But typically because they're trying to save time and they're seeing a lot of patients, they go to the bottom comment box and just write it all in free text and they type it at least so we don't have to read their handwriting. But now you've got doctors saying things in different ways and it turned into natural language processing, which becomes significantly harder than obviously having just that structured form of data.

So you mentioned in passing there are other sectors like retail, consumer goods, telecommunications that maybe aren't quite as mature in how they think about data science, but there are steps being made there. So maybe you can tell us a bit about that.

Yeah, a couple interesting problems that we worked on was actually, this was a while ago, one of these. I'm interested in it because it was, I think early on for this type of work, they were conducting surveys. They were a famous gum manufacturer and they had user surveys that queried people about the gum, how do you like the taste, how long does it last, the flavor? And we were using that data to try to predict new product launch successes. So using old product surveys that were really successful at launch, we then tried to predict, they gave I think thousands, a couple thousand people, different types of gum to try that they were going to then bring to the market. That was an interesting one. I don't think we ever got to see whether our predictions panned out or not, which is one of the holy grail problems in some of these fields. But that was one. And another more recent one was with massive consumer goods supplier who sells products online and they just have every single thing documented on what's happening on their website. If a user clicks a page, if they click a link, if they add something to their cart, which most E commerce sites are now collecting that data and storing it. But they're all, so many of them are at this point place where they don't know what to do with it next. So we came in to help them try to optimize product placement on their site, figure out when people were leaving and abandoning their cart and try to make changes to resolve those issues essentially and sell more products.

Yeah, and of course that's one of the reasons that what has now become modern data science, a lot of the techniques did emerge in tech. Right. Because we were able to, once you have the data foundation set up as a tech company to actually get all the data and start to work with it.

Yeah, I think companies have been collecting data for a while now and they've Caught on. Some are just starting now, but we're definitely at a tipping point where now they need to figure out what to do with it.

Yep, absolutely. I want to just pop back to this, this gum example. I find it very interesting because I'm a gum guy as well, and I've got a lot of questions around gum, which we probably shouldn't, shouldn't delve into too much. I was wondering, you said you may not have found out whether the predictions had panned out, but were there actionables from the work that you'd done for the. That the business could take from the insight that you got out of the data?

Yeah, I believe they took several. We ranked them essentially, and I think they took the top three or so and did a more focused study on those. So you're always trying to figure out where do we spend our money, where do we spend our marketing budget and so on. And so they took those and conducted a more focused study on those products and then decided from there which ones they were going to try to mass produce. So definitely action items. They took it and ran with it. And who knows? I may have tried the gum when it came out.

So I do want to say two things about gum. Firstly, I feel like gum flavor has decreased in duration as I've gotten older, but it may actually just be my olfactory system being less sensitive. Secondly, I was in Schiphol or Schiphol Airport. I've got no idea how to pronounce it. The airport in Amsterdam a while ago, ago tried to buy gum. They don't sell gum in the airport at all. And I asked why? And they said it's because people will put it under chairs and on the ground. So they just stopped selling. And I said, but people can bring it in. And they said, yeah, but we're minimizing it in the way we can. And I was like, okay, okay, deli dude, that's fine.

I mean, I, I kind of respect that.

Yeah, no, it's, it's, it's, it's incredible. And it's like he was so upfront about it as well. It's, it's hyper, hyper rational and logical. We don't only want to go from data to insight all the time. A lot of the time we want to make a decision based around that insight.

Right?

Yes. And that's what you always want to be doing. I mean, there's some cases where insight may be valuable depending on what it is. But ultimately I want to drive people to make decisions that are then measurable. So that we can determine yes there was a success or no there wasn't.

And is the role of the data scientist, and this may be a provocative or ill formed question, is the role of the data scientist to make that decision or provide insight into making that decision or to just provide results from the data?

It'll depend on the seniority level of the person, how much they know about the domain or the company, but I think they're positioned to sure, because they probably know the data better than anyone in the company, what management does with it, or if you're a co founder or chief data officer, you can probably make those decisions yourself. But I think ultimately we're building, we're not replacing humans and our natural instincts, but we're building decision recommendation engines in a way. So we're trying to guide and optimize the decision making process. Rather than put your finger in the air and just take a guess.

Awesome. I love it. And something you mentioned earlier was that people who aren't necessarily data scientists will more and more hopefully be able to get into the weeds with data, whether it be data exploration or basic statistical modeling or cleaning data and understanding how messy data can be. And how do you see that with people who you do consulting work for? I mean people at C level or whatever it may be. Would you like to see these types of people become more data literate in the next five to 10 years?

Yeah, for sure. It does everything from alleviate whether it's unreasonable expectations and how long something actually takes to get done. It helps them really position the value proposal from the analysis better and in a smarter way helps them realize what's actually possible. So oftentimes, as we know, there can be empty promises made maybe by a salesperson and you absolutely cannot do what they said because either you don't have the data or the data is just too, there's not enough of it or it's too messy. So absolutely, I mean it will, it will help in a number of ways. Foreign.

We'll jump right back into our interview with Tania Casciarelli after a short segment. Now it's time for a segment called Statistical Distributions and Their Stories with Justin Boyce, a lecturer at Caltech and a Data Camp instructor. Hi Justin.

Hi Hugo. Great to be back. I want to talk about probability distributions and simple ways to think about them. Now, I'm not going to be rigorous with mathematical definitions, but we'll just say that a distribution provides information about the probability of different outcomes of an experiment or observation. So an important step in statistical modeling is to Attach a distribution to a given experiment.

Right. So I know you like to talk about thinking probabilistically. And of course, distributions are an important tool for doing so. Right?

Exactly. If you can match a distribution to your experiment, you can analyze outcomes using the many properties of the distribution, such as its probability mass function or probability density function, in the continuous case, its cumulative density function, its moments like the mean, etc.

Can you give me an example?

Sure. We can start with a very simple one. Lets say our experiment is a simple single coin flip and we want to know the probability that the coin lands heads.

It's 50 50, right. Do I need a distribution to describe that?

You're right. The probability that an unbiased coin lands Heads is 1/2. But we can encode that in a probability distribution. This distribution has a name. It's called the Bernoulli distribution, named after Jacob Bernoulli. Knowing the distribution, we also know its probability mass function. For example, if we assign heads a value of 1 and tails a value of 0, we know that the probability of getting 1 is 1/2 and the probability of getting 0 is 1/2.

But what about if the coin is biased? Let's say a trickster has a coin that lands heads 75% of the time.

The distribution is still Bernoulli, but the parameter of the distribution, the probability of getting one, is now 0.75. So the Bernoulli distribution has a single parameter, the probability of having an outcome of one.

So what else is Bernoulli distributed?

Great question. This is where the skill of assigning probability distributions to experiments comes in. The Bernoulli distribution has a story. The story of the Bernoulli distribution is this. The outcome of the flip of a coin with bias. P is Bernoulli distributed. In making this story, I have defined the parameter of the Bernoulli distribution, which I called P. Now a coin flip is an example of a Bernoulli trial, which is an experiment that has outcomes that can be encoded as 0 or 1. So really, the story of the Bernoulli distribution is the outcome of a Bernoulli trial is Bernoulli distributed. So any experiment that matches this story can be described with a Bernoulli distribution. An example is having a monkey choose the left banana or the right banana.

Or a randomly chosen user clicking an ad banner on a website, or a.

Given mutation occurring in DNA replication. All of these can be Bernoulli distributed, just with different parameters, P. So it's.

Really just twisting around nouns to make the story match exactly.

And once you match the distribution, you know all Those delicious things about the probabilities as given by the distribution functions and the mass or density functions. And now that we understand what we have to do to match an experiment to a distribution, I'd like to tell you more stories for more distributions so your listeners can use them to model what they are working on.

I'd like that too, Justin, but I'm afraid we're out of time for this segment.

Oh, man. Okay, we only got to do the Bernoulli distribution. Do you want to have me on again so we can do some more?

Sure. Justin, always a pleasure to have you on.

All right, it's a date.

After that interlude, it's time to jump back into our chat with Tanya. I'd be right in saying you're a huge fan of dashboards, right?

Yes.

So maybe you can give us your take on dashboards and data products in general. Tell us what they are to you and then let us know kind of what their role is in your work and data science for business in general.

Sure. So a dashboard is really just a great way for people to consume and interact with data quickly. An ideal dashboard, in my opinion, and I think also maybe Edward Tufte's, is that you should be able to look at something, a visualization or a dashboard, and take away insight or an actionable item within 10 seconds or less. And if you didn't accomplish that, you need to go back to the drawing board, redesign it, make it simpler. Because otherwise you're basically better off just looking at a big Excel sheet of numbers and try to weed through it and figure out what your answers are. So dashboards are great for that, for distilling a bunch of different data down into something consumable. The reason I like them for data products. And what is a data product? It's either something that your company sells, it could be your entire product. I've worked at companies that were just data driven product companies. So as an example, I worked for a telecom company that had this proprietary data on customer switching. So I don't know what your providers are in Australia, but here we have like T Mobile, Sprint, Verizon. You could imagine that if you switched from T Mobile to Verizon, which people do, because nowadays it's a poaching game, right? Because everyone has a cell phone. We had all those switches in the country, the date it happened, the phone number, and there were about 50,000 of those happening a day. And obviously you can imagine a CMO at Sprint would love to know where customers that are almost out of contract when those Contracts existed. High value customers that are coming off contract from AP Verizon are densely located in Southern California. So now Sprint says we're going to run a marketing campaign there. So we built a product around that was put in the hands of these providers as a competitive insights tool. And that was essentially our entire business model. But the data was our product, it had to be correct, it had to be clean, it had to scale. The product needed to be user friendly and completely usable. And I think that's a huge use case at a lot of companies now, especially startups that are building essentially their profits from data itself, for sure.

And in that case, and maybe in general, these data products tend to abstract over the data in order to show the insights as immediately as possible.

Yeah, exactly.

And what's the role of interactivity in data products, do you think?

Yeah, especially in rapid prototyping, which I think we'll talk about in a little bit. It's really a way to make something real and tangible to someone. So it's, you come up with an idea, you go through a whole bunch of requirements, planning and then you get the data and you build something and you realize, oh crap, I didn't even think of this. Because of the complexity of data and how many unexpected things can come up, you're never going to be able to just document and know ahead of time what's going to be in there. So we build something quickly to allow the user to interact with it and get an idea for how this might work, put it in the hands of a customer, get their feedback before spending a ton of time and money on building a fully fledged product.

This is a really nice segue into this idea of rapid prototyping and I'd love to hear more about your approach to the trade off between rapid prototyping of data products and dashboards and building fully matured data products.

Sure, yeah, it's. So there's the old sort of waterfall technique they call it, where you, you know, you spend six months building out your first phase of the product. I always like to use the analogy that you could end up building a Ford truck when your client really wanted just a Prius.

I love it. Yeah.

So you have to essentially not over complicate things. You want to keep it simple, you want to ship quickly and iterate and get that feedback from the customer. Because I've been in situations where a client or a company has over engineered something and made it so complicated the user was overwhelmed. It just sat on the shelf and collected dust and was never used because we never tested the market. That's what you're doing, the data product. You need to test your market before you go get funding and build a company. You should be doing that for data products as well.

Yeah. And you want to demonstrate at least potential value asap so that everyone's in or at least some key stakeholders are in. Right?

Yeah. And it seems like common sense, but you'd be amazed. I still run into this where it's just over, over engineered and not what the customer actually wanted. A lot of people also like to think maybe they know their domain and their clients better than they do. But at the end of the day, sometimes they just don't. And there could be something unforeseen and sometimes the client doesn't even know what they want. So let's put something in front of them, get them to react to it and then start to define our requirements and iterate that way. And that's an agile prototyping method that I swear by and it's been successful. There are drawbacks and one of them being, let's say you put something in front of a client quickly, you don't always have time to fully QA it. You don't always have time to catch every single edge case. And people sometimes misconstrue that as well. This is completely wrong. It's a broken product and we don't trust this data anymore. But there needs to be, I think, a shift in thinking. You're always building, there's never final versions, everything is a draft because otherwise, you know, you spend two years developing something and it either never gets out the door or you built the wrong thing for sure.

And it sounds like essentially what we need is this type of wrapper prototyping, but hand in hand with some serious management of expectations around it as well.

For sure, managing expectations and also getting the client highly involved in the QA process if possible. So I always try to ask for a helping hand, whether they have a junior analyst or anyone that can come in and get some extra eyes on, on what we're building. Just because sometimes we don't have a dedicated QA team on stuff, sometimes we need the extra help and they know their data better than us. So we try to bake that into our projects. Now is we want you guys to look at it, make sure it doesn't look crazy because you know your data better than we do at this point.

Absolutely. Do any illustrative examples spring to mind.

Of catching wrong data?

Yeah or no, just of the power of rapid prototyping. Before building out fully mature products.

Yeah, for sure. So the telecom example that I talked about, we had an idea and it was called Voice of the Customer. So we knew that people were switching from saferizing to T Mobile in vast quantities, but we didn't know the why. So we wanted to build some sort of social media monitoring tool, start monitoring of events that were happening, like iPhone releases. And it was just me, it was a. Well, it was a 12 person company, but it was just me in charge of this product. And I had just hired, I think, two fresh grads out of college who were just learning R and everything about data. And so we got together and we literally just went to Twitter and started searching to see if people were talking about switching. Turns out they were. People love to go to Twitter to complain about things. So you'd see a lot of things like, I'm switching to T Mobile because Verizon service sucks. And from that tweet, there's so much information there. Right. They're talking about the reason they're leaving Verizon. They're talking about who they're switching to. And so we started to quantify that using just basic language processing and built a prototype. The first one was just in Ruby on Rails at the time. And then we had actually built some R Shiny apps as well. So we were doing a bunch of different things in parallel, testing out different designs and ideas, and when we realized, hey, there's a viable product here because we put it in front of a client, they were very interested. Then we decided, well, let's go to. It was called Gnip at the time. They were the Twitter firehose provider. I believe they got acquired by Twitter. We decided to purchase data in bulk. So initially we were just scraping whatever tweets we could get, which is a very small percentage of the Twitter firehose. But there's no reason to go and buy the Twitter firehose before. It just wouldn't make sense. You need to make sure that the data is there and that it's viable and then you cast your net wider.

Yeah. And it's what they want as well. Whatever the management is that you're dealing with and working with.

Yeah, yeah, exactly.

Incredible. So I love that you've mentioned R and shiny. Now we're not going to get too much into the programming language.

Ring.

When I say R because of my accent, it doesn't sound like an R. So it sounds like what the Doctor sounds like.

You're at the Doctor.

Yeah, exactly. And shiny, which for our listeners out there, Shiny is a wonderful technology for rapidly prototyping dashboards. We've got great courses on Shiny at data camp. RStudio has a lot of great resources. But my question for you, Tanya, is these technologies have and are evolving so, so quickly. So I'm wondering how your ability to actually do the work you do has evolved as a function of the tech and open source software development since, you know, 2005 when you first learned R by reading the entire CRAN documentation.

No way. I didn't read the whole thing, I'm sure. But yeah, I was crying myself to sleep. Now it's so crazy how it's changed. But yeah, I constantly feel like I can't keep up. Right. And that goes back to probably good old imposter syndrome. But if you don't have a little bit of imposter syndrome, I think that you're probably doing something wrong. Because if you're not just aware of how much there is out there to know. Because I know what I don't know and there's a lot that I don't know, but I try to. What's nice is having clients kind of dictate what I need to know and learn. So if I'm going to need a columnar database because I know this client's going to have a ton of data, it needs to be HIPAA compliant, maybe for healthcare. Then I brush up on my AWS and redshift. If my client is going to be doing large batch processing jobs, maybe we look at Spark or Hadoop. Spark is definitely outpacing Hadoop now. But luckily R has built all. There's been all of these packages that really awesome people have made to make my life easier. I always thought, for example, I wanted to learn D3 because I saw those really sexy visualizations in the New York Times and I thought, crap, now I have to learn D3 and JavaScript and all these other things. But then Ramnath came along and built R charts and I could basically build D3 charts just knowing R. And similarly with Shiny, I can build websites now with web applications just using R. So I've been fortunate and I think the R community is a big reason why I'm able to be successful in my current consultancy.

Great. So there's literally stuff that you do now that's part of your daily bread and butter that you wouldn't have been able to do pre these tech being developed.

Absolutely, yeah.

So I also know that something you're very interested in and passionate about is kind of thinking about the data scientist hiring process. Yes, and I hesitate to use the word opinionated, but I think you have very, very good, strong opinions on it. So maybe you can tell us a bit about how you feel about all of this.

I do. It's a pretty broken process. I've been through some interviews myself that I thought were just horrendous experiences and I've been through solid, pretty good interviews. And I mean, the interview process is the first, the first experience that someone has with your company. You want it to be a good experience. And so I hate whiteboarding. I call it the. It's the waterboarding essentially of interviewing.

I've never heard that before. That's incredible.

It puts people in this just weird state of trying to almost dehumanize them and some kind of situation that would never happen. It's like you'd be hacking in the movie Swordfish with a gun to your head or something. But yeah, I gave a talk on this at Strata and I have formed a lot of ideas around it. And honestly, the reason I started getting interested in it was because I was working at startups where I was in charge of hiring a lot of people and interviewing a lot of people. And it's just such a time sink for everybody involved. So I wanted to just streamline it for myself, honestly, and for the candidates as well. So I came up with a small test. And people have opinions about this too. Like you shouldn't expect them to spend their time doing something. However, this is like maybe to a couple hours, two to four hours. You should never expect more than four hours from anyone. And they take it on their own time. We give them some, some hints and what they should. Some questions they should answer. There's really no right answer. It's just, you go, you evaluate their thinking process and how they document their code and everything and they come in and present and it might be an hour to your stakeholders. You put them in front of maybe your business stakeholders and see how they convey technical concepts. Or if you want them to be very technical, focused people and client facing or facing the engineers or statisticians in your business and have them present to them. So it's still better than an all day gauntlet, which I like to call it the eight hours of just going through and meeting everyone in the company. But those are kind of the big no nos, is the, I think the whiteboarding and just having someone spend an entire day at your office.

Absolutely. And we'll put the link to your Strata talk, which is online in the show notes as well.

Oh, cool. Okay.

We'll jump right back into our interview with Tanya after a short segment. Now it's time for another installment of Statistical Lesson of the Week. I'm here with Emily Robinson, a data scientist on the growth team here at Datacamp. Hi, Emily.

Hi Hugo. Thanks for having me on again. This time I'd like to discuss the concept of power.

All right, Hour of Power so what is it?

Power is another important part of statistical hypothesis testing. Power is the probability that if there is a difference of a certain size, you will detect it and reject the null hypothesis. Generally, the rule of thumb is that you want 80% power. When you're designing a test, you should do a power analysis at the beginning. This is because you may find there's no point in running a test.

Why not?

Say you're a psychologist designing a laboratory study to see if offering a gold star as a reward will increase the percentage of people finishing a task. You only have funding for 100 participants. If you run a power calculation, you would find that if you pick a task where you expect in the control group, only 10% would complete it. The gold star would need to increase that to more than 35% for you to be able to detect the effect. That's because even if your gold star increased the completion rate to 20%, you just wouldn't have enough data to distinguish between those groups. If you don't expect there to be such a dramatic increase, there's no point in running the test. You'll almost certainly get a non significant result and won't have learned anything.

It's certainly always good to follow statistical best practices. Are there any other issues besides potentially falsely concluding there's no difference?

Well, when your study is underpowered, there will be a larger variance in your estimates. When you do reject the null hypothesis, your estimate of the difference between groups tends to be higher than when you reject it. With larger sample sizes, you'll end up overestimating the size of effects. If you want to read more, check out the Nature article Power why Small Sample Size Undermines the Reliability of Neuroscience.

What are some other mistakes people make in regards to power?

Well, sometimes people will do what are called post hoc power analyses when test results are not significant. In fact, journal editors may even ask academics for them. There's a great paper about this problem called the Abuse of power, the Pervasive Fallacy of Power Calculations for Data Analysis by John M. Hoing and Dennis M. Hazy. People compute what's called observed power, the power of the test for the observed value of the test statistic. The argument goes that if the power is low, the evidence of the null hypothesis is weak. It's not that there's definitely no difference, it's just that you didn't have enough data. The problem is that the observed significance level of a test, the p value, also determines the observed power. For any test, the observed power is a one to one function of the p value, and non significant p values always correspond to low observed powers. And we face a paradox here. A higher p value, usually taken as more evidence for the null hypothesis, corresponds to lower power, taken as less evidence for the null.

Interesting. How has power been important in your own work?

Power is a critical concept in a B testing before doing a test, we do a power analysis to determine whether we should even run a test and if so, how long we should run it for. If you don't do a power analysis, you can fall victim to a problem called peaking where you check every day whether there's a significance difference between groups and stop the test if there is. This will result in a huge increase in your false positive rate.

This has been very educational, Emily. Thank you. Where can our listeners go to learn more?

In addition to the papers I mentioned, Julius Silgi has a great blog post on AB testing that covers power. She also created an accompanying shiny app so you can calculate how your power level changes with your effect size and population.

Awesome. Thanks Emily. I think this discussion will be very helpful to our listeners.

Thanks Hugo. Looking forward to doing more segments.

Me too. Time to get straight back into our chat with Tanya and something I think definitely coupled with with this line of questioning is, you know, something you're also speaking to is you don't want someone who can do everything necessarily. We're not looking for the data science unicorn and I'm wondering what your thoughts are on the data science skills gap essentially, and how as educators we can approach arming future data scientists with the skills that they'll need.

Sure, yeah, that's there's a huge skills gap, but it's not as unattainable as some people think. There's a lot of hype around machine learning and now AI and advanced statistical modeling, but at least in my experience that's really only about maybe 20% of the projects I've worked on, probably also 20% of the companies I've worked in. There's a ton of need to just be able to take the step from Excel to basic data munging in R Python, and I've taught courses to non coders, non technical people on the basics of R. We've had great success with it. People enjoy it and they feel empowered to now go and do something a little bit more sophisticated than what they were previously able to do in Excel. And there's a lot of need for that. Just literally taking different sheets of data, joining them up, doing some basic QC with missing data and, and being able to just do basic summary statistics like aggregating and finally maybe putting it in a tableau dashboard. And we've taught courses that are part time over eight weeks that get you to that point. So people get intimidated. And I can see why, because the field hasn't been super welcoming to some extent where you're not a real data scientist if you don't know every single sort algorithm in existence and can write it on a whiteboard while someone stares at you. Or you're not a real data scientist if you don't have a PhD in statistics. So we need to start to compartmentalize and understand what we actually need into different skill sets.

Yeah. And I do think this also speaks to a general statistical literacy and data literacy, which, as you say, the people who are using, you know, the tens of millions or perhaps more people who use spreadsheets once a week or more, getting them up to speed on a bit more of the robust programming and statistical concepts that will help them do their job.

Yep, exactly.

So this leads nicely into my next question because we're really talking about the future now. And I'm wondering what the, what the future of data science looks like to you. And this is a prediction problem, right?

Yeah. Let me, let me build a model for you real quick.

Build me a dashboard.

Wow. I mean, I think that we will start to see more advanced cases for predictive modeling, machine learning. But like we said before, companies are just now starting to ask questions of their data, seeing what they have, realizing it's dirty. The prediction part always comes later. So I do think you're going to see an increase in that. Even though some people are starting to say the big data and machine learning hype is the wave has passed. But I think it's still just beginning. We were kind of ahead of our time when we were doing this type of work at a biotech company I worked at back in 2008 or so. But people are starting to get it. The more success stories you see with it, the more it's going to catch on. I think you're going to start seeing more uses across industries we might not have expected in government, in small research labs, and even, you know, the, the army is starting to use some Techniques, you're going to see just a lot of different applications of using data to improve outcomes for any, any case you can really think of, I mean, you can improve employee retention, you can improve, you can try to cure cancer. We can, there's a million different ways that we can use it more effectively.

Absolutely. So in that sense, what the prediction is that we're going to see it move laterally into kind of all facets of life and society and business.

Yeah, I just think it's going to be more, more adopted, widespread. And I think a big key indicator of that has actually been its involvement in the recent election and in sports. With sports gambling actually being legalized here pretty soon, people are starting to get it. That's a big way I teach it too is we use sports data so that people have a little bit of fun and they understand it. And once you use analogies like that, it starts to click and people realize, oh wow, I could use this, you know, in accounting or to sell more inventory.

Yeah, that's one of the most important things for education and learning in general is making it relevant to the learner.

Yeah, we always try to use relevant data sets at corporate training so that people get it and they understand the why. When we do just these part time, kind of fun ones, we'll use a fun data set and then we ask the students, so how can you see this applying to your current work environment? And it's pretty interesting because they can just see how it applies laterally. Once they've done, you calculate Tom Brady's touchdown conversion percentage and then you go and calculate percentage of budget used.

Yeah, that's fantastic. So what's one of your favorite data sciencey techniques or methodologies, something you enjoy doing?

I love Shiny. You mentioned it before and I just think it's given data scientists such cool way to put their work out there and express themselves. And I used to just build R scripts in isolation that maybe generated a CSV file. But now with R markdown and Shiny, we can really showcase what we've done and, and make it something real that people can touch and see. And that is one of my favorite things to do. I mean I build fantasy sports dashboards for fun. So it's definitely my favorite part of it I think is visualizing and putting together those dashboards.

Great. And I don't know if I recall Mara Avrick, who's also in Boston area, she's a big fantasy sports fan as well, right?

She is. We worked together actually on a project, it was a sports API that we, we built an RWrapper for and she's a huge hoops nerd. She knows more about basketball than probably anyone I know. And yeah, I keep trying to team up with her to, to take over the sports betting market. So we'll see.

Fantastic. Well, let me know how that pans out. So Tanya, my final question is do you, do you have a final call to action for all our listeners out there?

Final call to action is just dive in, don't be afraid. Ours not scary. Python's not even scary. But I honestly love R. I use both. But just find a data set that you might be interested in. It could be anything. It could be about food, it could be about sports, movies, and look up some tutorials. Data camp is a great place to start. I always plug you guys and awesome.

I do too.

Yeah, I'm not paid, I swear for this, this promotion. But yeah, it's just dive in and get your hands dirty. And the R community is very friendly and helpful. There's lots of places to go, there's Slack channels you can join. So definitely just get started. If you're interested in playing with data.

Absolutely. And I always tell people the ask stats hashtag on Twitter is a great way to get involved. You'll ask something and most of the time you'll get an answer really, really quickly. In all honesty.

Yeah. And our studio also just launched our community section where people ask questions and get answers. So if stack overflow isn't working out or you just don't even want to face the wrath of some stack overflowers, you can go to rstudio community.

And what we'll also do, Tanya, is post links to a whole bunch of pieces you've written. There's a great one on rapid prototyping, another one on hiring in the data science space. We'll post all of these in the show notes as well.

Cool. That'd be great. And I'm always happy to answer questions if people want to reach out over Twitter. You can find my email at my at our website tcbanalytics.com as well.

Fantastic. Thank you so much for coming on the show, Tanya. It's been an absolute pleasure.

Thank you, Hugo.

Thanks for joining Our conversation with Tanya about data consulting, building data products, rapid prototyping, decision making, the data science skills gap, and the process of hiring data scientists. We saw how important it is to get data products out the door while managing expectations of those using them for decision making. Tanya demonstrated this with some great examples, including one that helped to inform telecommunications networks about when and why their customers were leaving. We also saw how the ability to build analytics dashboards is so important in helping businesses make decisions from their data, and how open source technologies such as shiny dashboards have lowered the barrier to entry for data analysts and data scientists to do so. One other incredibly important takeaway and is that we need to promote data literacy at all levels of society and organizations. In Tanya's words, everyone at any level C level, entry level should be looking and diving into data the same way that you were expected to start using email 20 years ago. Also, make sure to check out our next episode a conversation with Eric Colson, Chief Algorithms Officer at Stitch Fix, an online personal styling service, reinventing the shopping experience by delivering one to one personalization to their clients through the combination of data science and human judgment. Eric is responsible for the creation of dozens of algorithms at Stitch Fix that are pervasive to nearly every function of the company, from merchandise inventory and marketing to forecasting, demand operations and the styling recommender system. We'll be talking about all of this and more. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com community podcast now it's time for a segment called Data Frame Bloopers with Hugo and Hugo. Hi. This is a test about to record for Tanya. Just testing my setup. Hi there Hugo, and welcome to Data Framed. Hi Hugo. So great to be here. I love the podcast. It's really, really, really, really pretty. Kind of. Okay, thanks Hugo. Those are really kind, ambiguous words you use there. That's wonderfully underwhelming criticism. So we're here today to talk about how to grow rambunctious antipodean beards, your approach to occasionally looking like a giant hipster doofus, and your relationship with your ugly Wu Tang Christmas sweater. But first, I'd like to know a bit about you. How did you get involved with Data originally? This is a story as old as time itself, and the story is called Data Day to Day Data Science Today End of Test.