#6 Citizen Data Science

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Dave Robinson, a data scientist at Stack Overflow. We'll be speaking about the evolving importance of citizen data science and a future in which data literacy is considered a necessary skill to navigate the world. Similar to literacy today. We'll speak about many of Dave's projects, including his analysis of Trump's tweets that demonstrated the stark contrast between Trump's own tweets and those of his PR machine. We'll also speak about ways for journalists, software engineers, scientists, and all walks of life to get up and running doing data science and analysis. I'm Hugo Baun Andersen, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems data science can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter Hugo Bowne and you can also follow datacamp at datacamp. You can find all of our episodes and show notes@datacamp.com community podcast. Dave welcome to Dataframe.

Hi Hugo, thanks so much for having me.

Such a pleasure to have you on the show. I'm. I'm so excited to have you here to discuss data science, what it looks like on the ground as a working data scientist, what it's capable of, and how everybody can be a data scientist. This modern concept of a citizen data scientist. But before we get there, I want to talk about you. What are you known for in the data science world?

So I do a few things within the data science community. So I blog about data science and also programming and education at my blog at Variance Explained. So there I usually just find interesting data sets and analyze them. So a lot of them are about baseball, sometimes about about programming or about some things about biology. And I also tried to teach statistics and programming concepts like how to analyze particular kinds of data and how to make R code run really quickly and I share my general opinions about data science education. I've also written a book, Text Mining with R, that was published by O'Reilly in Summer with my colleague Julia Silge, and that's a book about analyzing text data using a particular set of tools that we call Tidy Tools, particularly the tidytext package that Julie and I developed. And besides that, besides Tidy Text, I also develop a couple open source R packages like Broom for taking statistical models and turning them into tidy data frames, gganimate for creating animated graphs with ggplot2 and fuzzy join that combines data frames based on inexact matching of their columns. So I did these couple of contributions to GitHub and to CRAN to these packages of R that other people can use, and I've created three data Camp courses. So my latest one was introduction to the Tidyverse that came out in November 2017. It's an introduction to a set of tools that we call the tidyverse, dplyr and ggplot2 that allow for data transformation and visualization that in a way that fits together really intuitively. And the course is designed for people to take it, even if it's their first introduction to R. I'm going to talk a little about some of that philosophy I have for how to introduce people to R and to programming later in our podcast.

That is a fantastic introduction to you, Dave, and you've touched upon a number of things which we'll delve into deeper soon, such as the type of data sets you've written about and enjoy writing about what it means to actually explore a data set and publish these results online, and how citizens can actually do that as well. You mentioned briefly your educational efforts, which we'll get to your books and of course Tidy Tools and the tidyverse, which as we'll get to, is a very interesting way for aspiring data scientists, aspiring programmers, and people who want to discover more about the world through data. The tidyverse provides a wonderful set of tools for people to get started with, and that's a little teaser for some things we'll get into later. How did you get into data science?

So it started with I was doing my PhD in computational biology, so worked in the field of genomics, where I generally analyzed how we could tell whether genes were turned on or off within a cell. And it was a project where I was really involved in I really was working more with programming and with statistics than I was with the actual biology. So I'd work with biologists, but I try and really analyze these large data sets so in 2015, when I was finishing my PhD, I was originally planning on looking for academic jobs. And then I happened to get reached out to on Twitter by from Stack Overflow, that's the programming Q and A website that I work for now. And he'd seen this post that I'd written a couple of years earlier. So a couple years earlier I'd answered this question about the beta distribution, particularly how can someone understand the beta distribution intuitively? And these years earlier I'd written this detailed answer. And right as I was looking for a job, he'd happened to find it. And someone had the idea, what if we, they've been looking for a data scientist and someone had the idea what if we just hired that guy? And.

And was this answer, was this the answer that involved the use of baseball statistics?

Yeah. So for me, I really liked explaining the beta distribution based on trying to figure out someone's batting average. So in baseball, someone goes up to bat, let's say 100 times, and if they get a hit 30 of those times, their batting average is 0.3. And estimating that about a player is one of the ways you can tell who's a better player or worse player.

Wait, just hold up, hold up for a second there. You're telling me that statistics can be explained not just through flipping coins, but through real world applications like baseball.

It's really important to find analogies that speak to you personally. It's also sometimes one to find analogies that speak to your audience. And one thing that concerns me about that analogy is it is particularly useful to Americans who follow baseball. But I think it's one way of attacking an educational problem from many sides. Some people want to read the math directly, read the equations. Some people want some really general examples. Example, say you're flipping a coin and you're figuring out whether it's fair. And some people like examples that use some real data and use some real problems and baseball provides a lot of those. Someone else might have done it with clicks on a website, which is what, which is usually what I would be using these methods for in my action, in my work at Stack Overflow. Someone else could do it based on analyzing text within literature or identifying images. There's. Yeah, there's many ways to teach statistical concepts and I think everyone trying to teach them in different ways leads to this really healthy ecosystem.

So you mentioned that you got started doing computational biology and genomics at Princeton.

Who I actually should say is that I actually wrote a, took that answer and the long term ended up writing Many more posts about it where I explored how baseball could be used to explain the method of Empirical Bayes. And I built out, I built it out into an entire ebook which is available, which available on my website is Introduction to Empirical Bayes.

You mentioned that you got started in computational biology and genomics, and this sounds like science to me. Well, it clearly is. But is that data science?

I'd say that in a sense I was doing data science sort of before there was a term for it. I think a lot of current data scientists have been doing this kind of work for a long time. The idea is, I think within a scientific department you have some people that are really interested. You also are really interested in biological questions. They'll be really interested in what genes affect this disease, how can I change the genes within a tumor to treat it. And other people will be really interested in computer science questions. They'll be really interested in how can I make this algorithm run faster, how can I sort this list, how can I build a database that can be queried? Well, and I think then data scientists tend to straddle both of those fields. They're people that are really interested in learning, programming and then taking scientific data and working with it. And for me, it really then was. It was a process of I learned enough about computer science that I could, I could apply these methods that I could program and work with large data sets. I wouldn't be doing things by hand. And then I learned just enough biology that I know what kinds of problems they would need to solve. And then I really liked, I'd rather grab some data, find out some things from it and move on to kind of move on to the next data set. And I'd work with collaborators to do that. And I think a lot of people in my field have that same kind of impulse. And as a result, it's a very good position for us to be where we can then move on to when we now work at a web company and I and instead of analyzing biological data, I analyze data about visitors or I could go to data set of tweets and I can analyze it and make some graphs and find out things about it. Is once you get what's your really favorite thing to do is grab data, understand how you need to reshape it and process it and how to draw a draw and then communicate conclusions. That's. I think that's where the data science lies. It's not just in biology, it's not just in computer science.

And so it's taking any data set really and figuring out how to extract information from it, which of course involves a lot of reshaping, filtering, munging, data cleaning, but extracting information from it and then being able to communicate that information.

Exactly. Those are the things that are in common across any data analysis, no matter what field you're working in. There's data transformation, say, how can I clean this? And how can I reshape it? There's statistical inference for how can I separate signal from noise? There's prediction for, I've got some inputs and I want to class fire or predict the outputs. There's visualization for how can I better understand this by making it into a graph. And there's communication. And sometimes these are sort of taken for granted within the sciences as they, those are just something that everyone does. Data scientists are people that are really interested in that process and interested in making it more effective.

And it just so happens that computers are a great way to do this now with the amount of data we have and the languages that allow us to reshape data and produce figures and do statistical inference.

Absolutely. You know, one of the founding figures of data science was named John Tookie, and he was a Princeton statistician who lived, I think, a bit early for the, he was, let me see, he was born In, I think, 1915 and did, did a lot of his best work in the center of the century. And then computing was very early, but he was, he was sort of, he was really laying a lot of the, lot of the groundwork. He, he invented some new, some kinds of graphs that would start to, that were able to better communicate data. He largely invented the idea of exploratory data analysis, of getting a data set and then figuring out what, what factors within it might influence each other through an iterative process of asking questions and getting answers out of it. And what I find interesting about that is that he was, he was scraping the, he was using these very early computers to solve these problems. And as computers have gotten more powerful, it's been a lot of people like him that have been able to move the data science field forward and keep up with it. And I absolutely do think that, yeah, I do think that computers are, some are sort of bound on how, how effective data science can be.

So you've spoken to the fact that there are so many different types of data sets that, that have interested you and that you've worked with, from web data in your current position to computational biology genomics data sets, to using the Twitter API. What, what data science projects have you worked on or been involved in that you think are the most impactful on society or telling about society, or that you've just loved the most yourself.

So one set of analyses I've done that I really love is I work for Stack Overflow and it's a programming question and answer website. So when people Google a programming question about, say, how do I sort a list in Java, or how can I get to make JavaScript do this particular thing on a website, they'll often end up on one of our questions. And with 20 million hits a day, we get a lot of great data about what programming languages people are using. And we get, and we get to analyze that to see how these languages and technologies that are used change over time and are different around the world. So an example of a cool analysis we did is what kinds of programming languages tend to be visited late at night versus between 9 to 5? So we can see that, for example, the programming languages built tools built by Microsoft like C Sharp and Excel, they tend to be visited a lot between nine to five. They tend to be ones that people use to do their day jobs. And there's other languages like Haskell, a functional programming language that is much more visited late at night. So that's the kind of thing that people would do in hobbies or maybe as passion projects. And I just think it's really cool the kinds of insights that we can have and that we can share with the world based on this data. And that can all be found on the Stack Overflow blog.

That's really cool.

But another analysis, probably the one I'm best known for, was in the summer of 2016. I some people had observed something about Donald Trump's Twitter account while he was running for president, and that was that about a lot of his tweets were sort of typical candidate tweets. They were tweets that contained, say, a picture and him saying, thank you, North Carolina, or I'll be on Fox News tonight. It would be these, these announcements. But then there also were a lot of tweets that were a lot angrier and kind of, kind of very passionate or somewhat unhinged. And someone had had the realization, and I went through and used data analysis to confirm this, that the difference between these tweets is the ones that Trump writes and the ones that his campaign writes. And the difference is that Trump writes all the tweets from, from the Android and his campaign uses an iPhone. So by looking at that data from each tweet, you can figure out whether it was Donald Trump, the candidate Tweeting or his campaign tweeting for him. And the particular analysis I did show, it showed some really big differences between the two devices. And the biggest was that Donald Trump was much angrier. You can use sentiment analysis and that's a tool we really use a lot in my book to tell our what kinds of words does he use that that the words the Android use the iPhone wouldn't. And see that they're much angrier. More likely to use crazy, bad, sad was the iPhone was more likely to use positive words like win or join or thank you.

Fantastic. And you noticed something about the temporal distribution as well, didn't you?

Yeah, exactly. Trump tended to tweet earlier in the morning and late at night for himself, while the campaign was more common, retweeting from 9 to 5. A few of the differences he some of the really telling signs. Trump himself doesn't use hashtags and or photos or links. His tweets are almost always just text. And it was the campaign off in wood. So that's one of the ways you could tell there was a really big difference between the two devices. And yeah, that that analysis got a lot of attention. I get interviewed in a number of.

News articles and you're on CNN recently, weren't you?

Or yet CNN did a documentary about Trump's Twitter account and how important it's been to the last couple of years and his particular political journey. And yeah, I was one of the people interviewed for it.

Cool. So I need your help now, Dave, to demystify something. If somebody read, a lot of people will have read your analysis of Trump's Trump's Twitter account. And what you've written is an incredible finished product that a lot of people may read it and think that you went in having a hypothesis, had a very strong idea of what you were going to find, and you just did the analysis. But that isn't really how data science or data analysis works, is it?

Exactly. So when I went into this analysis, I was interested in answering this question, but I'd never actually queried Twitter data before. One of the first things I had to do was look into how to do that, how to work with the Twitter R package and API. And because I had these basic programming skills, these things they developed in other projects, I was able to to use documentation and figure out how to query Twitter. So once I downloaded the data set, I basically, by using these publicly available tools, I was able to bring in my data set and then by using the same kinds of tools that I've been working on for my book, for analyzing text and other tools for visualization. The same kinds of tools that I would use for visualizing any data that I had, I was able to build to put this post together. So one of the things I love about the data science world is that a little knowledge of tools can be very flexible in terms of the kinds of problems you can answer for.

Sure. And presumably you also were talking about John Tookie and his ideas of exploratory data analysis, otherwise known as eda. Presumably that also played a huge role in what you did.

Oh yeah. I think there's a huge part of practice that comes into doing a data analysis like that. Just I think in total I spent about 12 hours on that blog post and downloading the data, analyzing it, finding some conclusions and writing it up. And it's not, it's not a particularly polished blog post. Certainly I think something that I put on my company blog would usually be a bit more polished. I might spend a bit more time on it. But I think it's really once you've done a lot of data analyses, I'd done a lot of for biological data and then I done a lot within my company and then I'd done some for other other blog posts. It's a really fun process and it's one where your habits can build on themselves. Once you learn how to use a tool, you can keep using it in the future and it kind of speeds itself up. You start to learn to build these tools faster and faster. So that's really one of the, one of the reasons, as I'll talk about in a second, that I really recommend people that are interested in data science start a blog and write blog posts analyzing particular data sets. There's just no better practice for really for understanding how the whole process of exploratory data analysis fits together through to communication.

Let's now jump into a segment called Blog Post of the Week with Data Camp curriculum lead Spencer Boucher. What's up Spencer?

Hi Hugo.

So Spencer, you're here to talk about a blog post that you read this week and loved.

Yes, I sure am. This week I really enjoyed a bunch of great reads. But one standout was Robert Chang's medium post advice for new and junior data scientists. We'll include a link to this in the show notes.

So tell us a bit about Robert Chang to set the scene.

Robert first became a well known data science blogger back while he was working as a data scientist at Twitter. Now he's on the data science team at Airbnb.

And why did you enjoy the post? So much.

All right, so Hugo, you know how it's so easy to get caught up in the math and the coding that we forget to take a step back sometimes and think about data science as a process and as a career? In his post, Robert invites us to do just that by laying out a set of six principles that he wishes he had known early on in his data science career.

And which principles did you most identify with, Spencer?

My favorite has got to be the importance of declaring your desire to learn early and shamelessly, as he puts it. Imposter syndrome is such a huge problem in data science. Feeling like you don't know as much as your colleagues do can cause us to try to pretend that we already know things that we don't just because we don't want to look silly. In reality, though, you might be surprised just how far others will go to teach you things if you're just sincere about what you don't know but want to learn.

Were there any other standouts?

Yeah. So Robert also recommends paying very close attention to all of your dissatisfactions instead of focusing on the negative aspects about what you're unhappy with. Let these things guide your next steps and take your data science work to the next level. And what about you, Hugo? I know that you've read this post. Did anything jump out at you?

Oh, look, Spencer, there's so much meat in there. I think one of my favorites is identify the tools that will help you solve your problem. Robert discusses this in terms of technologies. For example, should I use Python or SQL for my particular problem? But it remains the same for all types of tools, such as which models to use. For example, why throw deep learning at a problem that a logistic regression can solve? Well, think about your problem, your question, and let that define your choices. Otherwise, everything becomes a nail.

Absolutely. Any others, Hugo?

Yeah, I agree completely with Robert that teaching is the best way to test your understanding of the subject and improve your skills. Not only that, explaining and communicating. Write blog posts, for example, explaining what you're doing and your work will become richer. Oh man, I want to read Robert's post again right now.

Yeah, totally. Robert goes into detail about everything we've talked about and much more in his post. So be sure to check it out, listeners.

And if you enjoy it, be sure to check in for the next episode of Data Framed. Our guest will be none other than Robert Chang himself, here to talk about data science at Airbnb. Thanks for telling us about your favorite blog post of the week, Spencer.

Until next time, Hugo.

After that interlude, it's time to jump back into our chat with Dave. Something you mentioned which really caught my attention was that you hadn't had much experience with the Twitter API or interacting with, with Twitter using a programming language before. And I think that speaks to a really interesting fact that as data scientists, what we do is we're learning stuff on the fly constantly. We may bring skills such as your knowledge of R already into play to use the Twitter R package and API, but people reading your blog may think you're a well seasoned tweet analyzer. And I think a lot of people on the outside, as well as aspiring data scientists may consider themselves non technical or non computational or even non mathematical and consider data science out of reach. So my question for you is, can anybody do data science?

That's a really interesting question. I think there's a, there's a, there are some people that would say no, that would say it's kind of dangerous to say everyone should learn some data science, everyone should learn some statistics. And the reason some people give for saying that it's not a good message to spread is that there's a lot of danger in statistics and machine learning being misapplied. What particularly dangerous case is we're seeing a reproducibility crisis in science where people that had, had done these scientific studies where they try and say, detect particular psychological effects and they, they're discovering that, that all these papers that have been published that looked like they were statistically backed up, other people aren't able to get the same results and reasons for that is abuse of statistics. Famous example would be P hacking, that is hacking a P value by looking at P values, a way that scientists can tell whether an effect is real or just noise. And if you try looking at a bunch of different parts of your experiment, like if you run an experiment on 1000 people and you say, and you don't get an effect, but you say, oh, I'll only look at the men, or I'll only look at the women, or I'll only look at people over 40 if you slice and dice your data enough, you can make an effect kind of appear out of nothing, but it won't be real. And that's a danger. And not having enough statistical education can lead to it. And another big problem is algorithmic bias. And this has gotten a lot of attention recently as there's a problem where if people design say, a machine learning algorithm to decide who gets a bank loan, if they just take some data and train on it without really thinking through some of the implications. They can end up building racists or sexist systems without even intended to ones that discriminate against people based on, on where they live or, or say how they act online. And these can have real consequences. They've had consequences in, within finance, within the tech industry in terms of if you just listen to an algorithm's recommendations without, without properly controlling for some confounding factors. I think the, I wouldn't give a full exploration of it here, but there's a lot of really great work being done to understand all these dangers. So everyone agrees there's that danger. And the question is, what is the solution? How can we try and protect against it? And here's where I would differ with some people. Some people I think would say this shows you have to. Everyone needs very rigorous statistical training and you can't be allowed near numbers until you've done it. You can't be allowed to implement a machine learning algorithm until you've had all this training in it. Therefore we should be careful about trying to say anyone can analyze data. It's just going to lead to data being analyzed poorly. So I understand that approach, but I think it's not the right way to handle the problem. So Hadley Wickham is a really influential developer in R. He wrote the popular dplyr and ggplot2 packages. And I really like how he compared this approach to absence based sex education where they say you just can never have sex. There are these dangers behind it. Only have sex once you're, once you're married. So he says this, this approach of saying you should ban everyone from statistics until they've learned all these tools is like abstinence based stats education. You should only do statistics if you're in a committed long term relationship with professional statisticians. So this has the exact same problem that abstinence only sex education has, which is people go and do it anyway. If you start just trying to put up these walls and say, well, I'm not going to teach you any statistics because you're just going to misapply it or well, if you really wanted to, if you want to do machine learning, you would better get an entire PhD in this subject first. You're just making it completely unattainable. So that's why I talk about what I think of as the citizen data science. The idea of someone who doesn't work as a data scientist, doesn't have statistical training, but still learns things that are useful for them to analyze their data and learns the pitfalls and the approaches they can use to handle data safely and learns them early on and starts applying them. I think it's better really to be upfront with education and say everyone should learn some basics of programming, of statistics, of data analysis, of visualization. I think it's useful for software engineers, it's useful for journalists. It's useful for scientists across the entire spectrum, psychologists, biologists, physicists. It's useful for executives in companies that might want to understand their own data. I think that learning a little bit about how to handle data, to understand relationships, do statistical inference, to build graphs and communicate about data, I think it's useful for everyone. In the same way that writing is useful for everyone, not just professional writers, for sure.

And knowing where to look for help as well. And where to ask for help.

Oh, exactly. And that involves building a welcoming community where if you say I want to do this, the answer isn't, well, you shouldn't be analyzing data. You're not a statistician. The answer should be great. Here are the things you need to know. Basically building a welcoming environment and one that has all the resources ready to say if you need help with this, here are the things that you. Here are the things we're going to have ready for you to learn to use it. So I look back at my trump analysis and I realize nothing in there was particularly advanced statistics. Nothing in there I really needed my PhD for. That's the kind of thing that a journalist could learn if they were committed to learning some, some R and then and some basics of data visualization and some programming, they could have taken that trump data, found the same patterns that I had and written about it. And I think a world where journalists are able to parse data that way is one where you'd see a lot of great journalism happen. I think the same thing is true across almost any field. I think people are better at serving their business if they're able to analyze the data.

So yeah, so what we're really talking about. This kind of struck me when you mentioned the idea of writing. We're talking about some sort of. Of citizen data science actually involves some sort of basic data science literacy across the community.

Exactly. Data literacy would be exactly how I'd say it. And I note that that's different from anyone can be a data scientist. Being a data scientist would be. Is professionally. It involves having like any field, it involves having skill, experience and a lot of focus in that particular area. So I think there are a lot of paths to being a data scientist, but it does Evolve a good amount of commitment. On the other hand, almost anyone can be data literate. They can understand. I'll download some data, I'll build a, build some graphs out of it, run a few regressions to understand how variable X affects variable Y and understand some of the pitfalls that underlie it. And even if they aren't doing the programming themselves, they might be able to talk to someone else who is doing and understand the pitfalls and the dangers that the. So I think data literacy is not a, either you are or you aren't ready to work with data. It's really a continuum. And I think the deeper everyone gets into it, the more these skills spread throughout the field, the healthier the entire community can be.

So we've touched upon the fact that when doing data analysis or having some sort of data literacy in the modern age, programming ability is some sort of predictability prerequisite. Maybe not extreme programming ability, but I think a lot of people, for a lot of people, programming ability is a potential barrier to entry for data science. What technologies do you see emerging that are reducing this barrier?

So I think this is one of the biggest challenges facing not just the modern data science community, but the entire world is how can we spread programming ability most effectively? And I think there are two fronts to this. I think there are two ways that we can approach it. One is making programming more intuitive, and the other is making education more effective and available. So in terms of making programming more intuitive, I really like what Hal Abelson was a computer scientist who said programs must be written for people to read and only incidentally for machines to execute. So the idea is that when programs were first being developed, it was all about how can I get the computer to solve this problem. But 50 years later, we're in a very different world. We're somewhere where we, we really should be thinking of the problem being how can we get programmers to understand this program? How can we design our tools in a way that are easy for people to use? I think a lot of people are doing really great work here, but one of them is Hadley Wickham. So I mentioned him before. He created the dplyr and ggplot two packages that are the center of this set of tools called the tidyverse. And, and Hadley likes to say these tools are developed for humans. That is, there's a lot of thought put into how can you make them fit together intuitively in ways that are easy for beginners to learn. So a big thing there is consistency. An example would be when you have many functions that these many tools that all work with strings, a few ways you can make them consistent and therefore more easy to remember is you can name them in similar ways. He starts them all with, with str, underscore. These are all tools that work with text strings and they're named, so they work together. And another way is to make sure that they all take their arguments, that is their inputs, in the same order. So say that you always have the string that's being operated on be the first argument, the first input to this function. So there's a lot of details to that, and he lays it out in a great document called the Tidy Tools Manifesto. But the really important thing is the consideration that when one's building tools to think, how can it be beginner friendly? How can it be something that people that are new to programming can learn without making lots of avoidable errors along the lines of, oh, I forgot that this function had this name, but this one had this name, or oh, I forgot that this requires this format, but this should be in another format. He thinks a lot about making these tools intuitive.

I think the naming of the tools is incredibly important in terms of having a structure in which we can actually implement the techniques we want to in a kind of consistent way. I also think something that Hadley is really locked in on, and the Tidyverse in general, is that we all have ways. We have a collective way of thinking about data already in our head before computation. So we think about GDPs of countries or birth rates or whatever it may be, and we can talk about them, and we can talk about in a particular year or which has the highest birth rate. These types of things and the actual computational structures that have been built in dplyr, for example, and ggplot mimic the patterns that we actually think in cognitively. So you will write code that mimics the way we think about the data set and talk about it, right?

Exactly. I think a lot of the center of that, and it's happening not just within the Tidyverse and not just within R, is based around the data frame, the data frames particular structure. I think it's revolutionizing a lot of, of a lot of the ways people build data science tools, it's the way of structuring data to think of it in rows and columns. Hadley likes to define these in terms of each row gets one observation, each column gets one variable, and building tools that take that as their natural input and thinking about transforming your data in this rectangular format with rows and columns. And it's amazing to look back just a few decades at how much data analysis and how much programming was usually done using objects that were shaped just very differently. People thought in terms of lists and they thought in terms of dictionaries that map a key to a value. But the idea of the data frame as being the natural unit of analyzing data, it's just very powerful. Another place that's happening is Pandas in Python has made a lot of these tools really easy for humans to work with with by structuring it all around data frames.

So Dave, as educators, what can we do to lower the barrier to entry for data science and programming?

Yeah, so I think educators have a really important challenge going on right now to make programming and data science education more available and more effective. And I think for that there's two really important considerations. One is just diversity of learning styles. Everyone learns in different ways, so we need to teach a lot of different ways. Some people really like learning from books. So Hadley Wickham has a fantastic, fantastic book R for Data Science that serves as a great introduction and our own book, Text Mining with R. The thing that we've done is make this available for free online as well as being sold. And that's one good way to make it make educational resources really widespread. I also think datacamp is doing really terrific work for people that learn well from videos and from interactive exercises through learning. By doing both of these handle the other problem which is scale, which is learning in colleges is just not enough anymore. There's not enough first world colleges to teach the amount of data science that we that the world really needs. And it's not realistic with how fast the field changes. People are going to be entering the field later in their life after their formal education and people are going to need to keep learning. So I think tools that such as massive open online courses like Data Cap and resources like books, resources like, like documentation that goes with programming. There's all these things that need that there's this real mass effort to improve them and make them more and more accessible to people that might be new to programming. I think the R community is doing a really fantastic job. Now another place that it's been really productive is the rstudio has this community.rstudio.org website that let's basically discussion forum that is really a welcoming place for beginners to go and get some feedback and build a network for how they can learn. So that's, I think some of the ways that that education can be made, can be made more available. But I think it's really also important to Consider the order that concepts are introduced and the, really the ways people are introduced to it so that it can be well oriented towards citizen data scientists, towards people that, that aren't looking for an entire PhD or an entire career in data science, but want to learn enough to do their jobs better. And for me, that comes down to the really important educational concept of teach powerful concepts quickly teach people the tools that they can immediately apply. So my introduction to the Tidyverse is based entirely around this. I don't start by teaching programming concepts. I don't start by teaching variables and loops and functions and lists. And these are a lot of programming concepts that people might typically start with. I start with the data. I say, here's a data set, we're going to be analyzing it, let's ask some questions, let's get some answers. And by the second chapter of it, they're already building graphs out of this data and starting to really gain insights from it. I think that's a really important approach. When you have someone like, say a journalist, someone who's busy but, but wants to use some R in their work, they're not looking to go through a long educational process. They're looking to get something done right away. And that's how they can build. They can learn something, use it and build on it to keep doing better and better work.

That's awesome because speaking to that, I think is this idea of learning in order to be able to do something straight away, as you say, which is analogous to what we were discussing earlier, that learning statistics, not necessarily to find out about the binomial distribution, but to find out something about, about what's happening in baseball statistics or genomics or gerrymandering or voting systems.

Right, Exactly.

People are having it practical.

Yeah, people. The world is filled with problems and questions that people want solutions and answers for. And then and challenge involves them being able to get their hands dirty with the data. So if you give people a programming lesson and you say it will be useful someday, that's not, not, that's not going to, they're not going to stick with it. But if you give people a, if you tell people, here's how you can start solving questions with your data right now, that's the way that you can really build their data literacy.

And that's actually something that's really cool in, in your Intro to Tardiverse course because a lot of introductory data science programming courses will, will like tell you how to print integers 1 through 10 or 0 through 9 if we're zero index in a for loop, Right? As opposed to showing from the Gapminder data set how to plot how literacy has improved since 1980 to now in over a bunch of nations.

Right? Exactly. This is something that this thing I feel very strongly about with regard to an educational philosophy of starting with by have a goal for what you should get students doing and get them doing it very quickly. I'm really looking forward to seeing seeing whether that course is successful in accomplishing those goals, collecting some data on that, and really seeing how it can help spread unreal understanding of R and let people work with their own data.

Now, for something completely different, we're going to dive into a segment called Rich, Famous and Popular with Greg Wilson, who wrangles instructor training here at datacamp. What's up, Greg? G'.

Day.

What are you going to tell us about today?

I'd like to talk a bit about git and about how we can use data science to make it accessible to the 99% of humanity who aren't using version control today. Now, some people claim that Git is intuitive, but it's actually the most complicated program I've ever tried to teach, and I've taught people Emacs. Some of that complexity comes from its inconsistent command syntax and jargon, but I think there's a deeper cause. Way back in 1968, Edgar Dijkstra argued that arbitrary use of go to statements led to programs that were hard for both human beings and compilers to understand, and that we should instead restrict ourselves to a handful of control structures, like for loops, if else statements and subroutines. And that might all seem obvious today, but it wasn't at the time. It took a decade for programmers to realize that limiting themselves to a small subset of all possible flows of control when building software actually made them more productive. We haven't had that collective epiphany yet when it comes to distributed version control systems like gif. At its core, Git is a way to construct and manipulate a graph that represents changes made to a project. It allows programmers to make almost any change to that graph they can imagine, and the results can be just as hard to understand as the tangled flowcharts of our distant ancestors. But in practice, though, most of us only change the graph in a small number of ways. That got me wondering whether version control will have its own structured revolution, and I therefore have a proposal. Step 1 Download data from several thousand large projects on GitHub. Step 2 Use data science methods to find patterns in those projects graphs. Step 3 Select a small set of common subgraphs that will cover most everyday uses. Step 4 Build a tool that provides those and only those to users so that they've got a more structured environment, a more predictable way of working. And then Step five is profitable. Now, Step three is a bit speculative. I have no evidence that usage patterns will actually fit a long tail distribution, but I think most of us would be surprised if that wasn't true. Step four is the one that will lead to all the shouting, as happened when structured languages eliminated goto statements. A minority of very vocal programmers will point at fringe cases that can't be handled by your chosen set of simple constructs. However, everyone else, the people that Microsoft's Scott Hanselman calls dark matter developers, will thank you.

And thank you, Greg. If anyone in the audience is interested in giving this a try, please get in touch. We'd love to hear from you. Thanks Greg, and looking forward to speaking with you again.

Thanks, Hugo.

Time to get straight back into our chat with Dave Robinson. So we've discussed a lot about what us as educators can do to help people get started with data science. How can people out there get started with data science? What type of action can our aspiring data science listeners take?

So I think the two steps are find some data that you're interested in and a question you have about it, and then learn how to work with it using programming and statistical tools. So finding data I think there's a lot of really terrific data resources and and there's this great community of open data where people share where people have been companies and communities and academic institutions have been sharing a lot of their data that people can analyze. Kaggle has many. There have been some R packages that share data sets. My datacamp course is built around country statistics in the gapminder data set, and it's worth contributing to that if you can. If you have interesting data, it's probably worth sharing it. So that's so once you have data and whatever you want to work with, then how can you learn to program with it and do analyses on it? I think that's worth just getting in touch with the right educational resources. I mentioned some before. R for Data Science is a terrific introduction if you're interested in learning R. My own book, Text Mining with R is great if you want to if you want to analyze text data. And if I really do recommend datacamp, I think it's a great way to build Python and R skills towards analyzing your own data sets. But it is important that when you're learning you have something, you have a goal that you're interested in working with yourself. Another really important educational resource for anyone is their network. So finding people that are already data scientists or are also interested in building their skills in data science and learning from them and sharing resources with them. So I really recommend the rstats hashtag on Twitter. So I tweet a lot and I think there's a really friendly and welcoming group there. And if you go in there and you ask advice, people often will help. Or if you share work you've done, people will share it themselves. You can also go to meetups in your local city. So I think one global organization that organizes a lot of terrific meetups is R ladies. So especially for women that work in R, it's a really great way to look for your local chapter. They're in New York, in London, in Istanbul. So a lot of cities have these. These are meetups. And it's worth going and meeting some people and building this network. It'll help you learn and build your skills faster. And if you're interested in being a professional data scientist, I especially recommend starting a blog. So that takes a lot of time, but it's really worth it. There are a lot of things that you can, that you can share in a blog and build your skills. So one thing you can do is find an interesting data set, analyze it, share the results. So that lets you improve your skills not just at analyzing the data, but at communicating the results. If you're more experienced in statistics or mathematics, you can teach a statistical concept in your blog post. I think that's something that I built a few posts around. And I think it's really important to share the kinds of knowledge that you have that might that not everyone does.

And that's actually how you got your current job. You said earlier was essentially explaining the statistical concept of the beta distribution.

Exactly. And the thing that astonished me about that was it's not a particularly advanced or expert statistical concept. It's the kind of thing that would be introduced in an introductory probability course. So it's not that it was such a advanced concept. It's not the thing I was most expert in or that very few people knew. It's just something that I knew that professionals knew but a lot of the world hadn't been exposed to. And people like the software engineer that founded and was and ended up contacting me about the job, it's something they were able to learn from.

Fantastic.

I have a blog post on variance explained about this. It's called Advice to Aspiring Data Scientists. Start a blog where I really share a lot of my arguments for this. And I also make an offer. I say, if you're interested in breaking into data science and you write your first blog post, you should tweet at me at Drob and I'll share it with my followers. So I have a good number of followers and also importantly, a lot of them are really are data scientists that are really interested in spreading the work of people that are newer to the field.

I love this idea because writing a blog post really forces you to go from collecting data through to exploratory data analysis, through to either modeling or statistical inference, whatever you do do, to visualization and actually communicating your results. So it takes you through all the steps that are involved in data science, right?

Exactly. And once out in the. Yeah, it's not just that one sound in the world that can help you get a job. It's the entire process of constructing that is like a data science project in microcosm.

And actually, recently you shared a blog post from a Metis boot camp student, right?

Yes, that's right. Someone was learning to. Someone was in a data science boot camp called Metis and they'd heard about my offer and shared this article with me. And it was a really terrific data analysis where it looked at comments on the FCC's website about net neutrality and it discovered that more than 1.3 million comments about against net neutrality were faked. So they were all like. They were all almost the same comment, but they were changed in just a few words. They were created by a bot. And he. By detecting that, he was able to say, in fact, the. While the comments on the page looked like they were a mix of positive and negative, they were the real ones made by real people were overwhelmingly positive. So he shared this post and I tweeted some about it and ended up getting a lot of attention. So this is someone who's very early in their career. Jeff Kao, who Eddie got interviewed by the Washington Post and shared in all these. In all these publications and might have a real effect on the future of net neutrality in the country. So I think it's another really terrific example of someone relatively early in their data science career having an enormous impact.

And I think it's probably worth getting slightly more technical with respect to how we blog, especially for aspiring data scientists and discuss the advent and how prominent jupyter notebooks are. Our markdown is now for writing computational narratives where you can explore and explain your entire data analysis and then creating blogs from that. I think you use Jekyll for your blog Is that right?

I do, yes. And it's built with Knitter. So Knitter is a way that I can write a notebook where I intersperse text with code and it generates the posts for me so that I can show my text, my code and my figures all at once. I think it's, it's a really. Notebooks are a brilliant data science technology because they can be used to share results within a company. They can be used to share results within the. That can be used even for work that you don't plan to share. So in the future you can understand how it was executed. Yeah, I think it's great how the notebook technology has kind of LED straight line from there to blogging.

And our authoring format on our datacamp community is entirely through notebooks. I will push a notebook to a GitHub repository in order to create a post on the datacamp community section, which is very cool and a lot of fun. So we, we've talked about a lot of different data science techniques, methodologies. What's one of your favorites or something you just love doing when doing data science? What really gets you going?

So this is a simple technique, but it's one that I think is really underrated and is really kind of one of my favorites. It's learn to put something on a log scale. So that is take it from a numbers that goes 1, 2, 3, 4, 5, 6 and think instead of a scale that goes 1, 10, 100, 1000. So that's really important when graphing because so many sets of numbers that we work with in the real world exist on scales that are much larger that are these multiple different orders of magnitude. So an example would be if I look at the GDP per capita within each country, I can see that some countries have a GDP per capita in the hundreds and some in the tens of thousands. And if I try and plot that just on a scale, on a scale where it goes 1,000, 2,000, 3,000, it's going to cram a lot of interesting and important data into one small part of the graph. But once you turn to a log scale, it ends up being much more meaningful. So I find there's just so many visualizations and so many statistical methods that become more usable once you get used to thinking about data sometimes on a log scale and switch back and forth when it's used. Useful.

Dave, that was incredible. I don't think I've ever heard someone explain why log is so important so well, let alone without having the ability to actually visualize it. So doing that on a podcast is a feat of wonderful data science education?

I think so.

We've discussed a lot about modern data science. What does future data science look like to you? What's going to happen in the next two, five, ten years?

So I think it's a really exciting time to be a data scientist. There's a lot of new tools being developed. For me, the really interesting, interesting question is that there's a convergence between tools that make data science tasks easier without programming and tools that make programming easier. So examples of the former tools that don't require programming include Tableau and Looker and I think Periscope. There's some tools that are very good, but they're sometimes meant to automate away your data scientists. They like to say, you don't need a data scientist anymore. You'll have this tool or you can use this without programming. And the others and those tools I think are getting better and better. But from the other side, programming is becoming more and more accessible. I think the Tidyverse is doing terrific things for that in R. I think Pandas is doing terrific things for that in Python. And so I think these two are converging where the question is going to be are the non programming tools going to get so good, no one needs to program or are the programming tools going to get so intuitive that everyone will program? And in the long term, I think programming will win. I think that programming as part of data analysis will be a typical and valuable business school skill across many fields, just like writing or public speaking. So not everyone writes as part of their job, but it is considered an important part of so many jobs. I think any CEO would be expected to do some writing. Any. A lot of people in software engineering are expected to be able to communicate effectively. And I think that these data literacy skills, as programming becomes more and more accessible, are going to become more and more widespread. So I think I'd love to see a future where in 10 or more years where tools like dplyr and ggplot2 or whatever's next in that space become really, really widespread across a wide variety of fields beyond, beyond professional data scientists.

So, Dave, it is always tough to say goodbye to you because I always enjoy our conversations so very much and we could wax lyrical for hours on such things. But in closing, I'd like to know if you have any final call to action to our listeners here.

Yeah, so I've provided some advice here and I definitely say I really recommend read about data science. I'd say the book R for Data Scientists by Hallie Wickham and certainly recommend checking out my book Talking with Julia Silge, Text Mining with R A Tidy Approach. I'd recommend that people blog about data science and if you do, you should tweet me about it. That's drob on Twitter and that'll let me share it and have more people read it. And I'd recommend people take my datacamp course. It's Introduction to the Tidyverse. It's a great introduction to R the people that haven't rather haven't programmed. It's great for people that want to be introduced to the dplyr and ggplot 2 packages or just want to review how these how maybe they've used the dplyr and gplyr2 packages. Want to see how they fit together. I recommend checking out the course and.

I'll add to that that it's a great introduction to computational ways of dealing with data and exploring a computational way to mimic how we cognitively think about data and the patterns that emerge there.

Yeah, I really like to think so. I think it gives a real taste of what a day in the life of a data scientist is like.

Like absolutely. Dave, it's been an absolute pleasure having you on the show. Thank you.

Thanks so much, Hugo. I had a great time.

Thanks for joining everybody. In this conversation with Dave Robinson, we heard about the increasing importance of data literacy in society and how everybody out there can take action to become more data literate, whether it be by working on small projects that interest you, you reaching out to the data science community at large for help and advice, or getting started by writing blogs of your very own projects. Make sure to tune in for the next episode of Data Framed, where I'll be chatting with Robert Chang about data science at Airbnb and Twitter.

Sam.