DataChats | Episode 17 | An Interview with Daniel Chen

All right, data campers, welcome to yet another episode of Data Chats. We're here in the beautiful suburb of Cambridge, Massachusetts. Had a great day recording cleaning data in Python course. I'm Hugo Bown Anderson, a data scientist and course developer at datacamp, and our instructor for this wonderful course is Daniel Chen.

So, I'm Daniel. I am a graduate student at Virginia Tech. I am in the genetics, bioinformatics, and computational biology program there, and I work at the Social and Decision analytics lab up in Northern Virginia.

So you're a grad student, but you do a lot of other things as well, Daniel, so maybe you can tell us.

Yes. So I guess I'll talk to you about how I got started. So I got my master's in Public health at Columbia in the Department of Epidemiology. And it was during then when I took a course on Intro to Data Science taught by Jared Lander. So hopefully some of you guys may or may not know of him. And so pretty much him and the other instructors for the course, Rachel Schutt and Kerr Patel, pretty much got me started, got me not so scared about using R. I had already known. I've already known Python or how to program in Python just for random scripting that I've done, but this was the first time I was really introduced to analytics. So that's pretty much how I've gotten started. And since then, it was during that course where I got my first. How do I say this? I was. I was an attendee of a software carpentry workshop. And it was then when I realized that what I actually knew from just my basic playing around in Python, I can actually extend that knowledge to be able to start teaching other people. And so I got involved with software carpentry, and I think by now I'm coming close to my third or fourth year being involved with them. So with software carpentry, I got a lot of practice teaching Bash, teaching Git, teaching Python and R. And over the course of the years was software carpentry and data carpentry. They focus a lot on good data practices or just good software practices in general. And so I've been trying to do my best in, you know, learning all of those techniques. And because I got to teach them all the time, you get a lot better just by teaching. And so now I'm here today trying to teach the rest of the world everything that I've learned or that's taken me about three years to learn.

So absolutely. And you mentioned that Jared Lander taught one of the courses you took in your masters and you also work with him on and off, right?

Yeah, so with Jarrod Lander. So I guess the Intro to Data Science course I took at Columbia was essentially like a semester long job interview that I didn't know was happening. And so at the end of the course Jared sort of like asked me if I would be okay or if I wanted to help with his teaching load for like consulting purposes at other companies around the country. Actually we can actually say around the world now, so. So yeah, so I've been teaching, you know, basic R. I've done some teaching on, you know, machine learning in R. And right now most of my teachings these days are around shiny, which is, you know, how you make an interactive website with your data, but that's more.

R specific than Python and this is R focused. But I'm under the impression you're also writing a book for Python, right? Maybe you can tell us a bit about that.

So I'm also writing a book called Pandas for Everyone and it's supposed to be just very similar to the course today that we've been recording, except more of just Pandas in general. So I'll show you. The book goes over how do we even import data? So, so it's a lot like the other data camp courses and I guess it comes from me teaching with software carpentry and now where I am at school where I have a bunch of these students who want to start doing data analysis and they come to me with like the same set of questions. And I guess just through teaching experience I've learned like these are the things that people are having a lot of troubles with. And because I've been teaching so much with software carpentry, it's, I slowly have like a nice mapped out order on what techniques or what things people should learn in what order. And so right now I'm just trying to put that down in paper form and I guess the goal with that is to try to come up with like a semester long university course and a book that kind of very ties in with it because the university structure is actually really difficult to hold a proper, like just a course on data science because of, you know, you don't meet every day, you meet a few hours a day, a few times a week.

So, so I'm sure a lot of people out there want to know when could they expect to see this book and who's, who's publishing it.

Oh, so the book is published through Pearson. Really hope I don't get this wrong. This is Bad. So it's Pearson. It's one of the Addison Wesley data series. If you guys look up Jared Lander and R for everyone, it's in the same series as that book. And hopefully it'll be out end of this year, early next year. It should be coming out pretty soon.

That's the end of 2017. Early 2017. 2018.

Yeah. So end of 2017, early 2018, yeah.

So the other question that I'm sure is on everybody's lips, it's obviously on my lips, is we've talked about how you do a whole bunch of training and consulting and software carpentry stuff in R, how you're doing a Pandas Python cleaning data in Python course. Here, a book on Pandas and Python. Tell us about R and Python and yourself and maybe your history of both.

So, like I said, I learned data analysis. The first thing I did, how to better frame this the best way. Actually, that's not even the right way. Take your time. I've learned how to do data analysis in R first. So at the time, I didn't know anything about data analysis. I was actually doing all of my stuff in Excel, like most people when they start out. And using R was really scary at the start, I'm not going to lie about that. But I would say between R and Python, it really doesn't matter. You going to learn one language, just learn it well, because what you learn in one will translate to the other. So a lot of the, you know, this tidy data stuff that actually I mentioned in the video, it comes from Hadley Wickham, which is very prominent in the R world. But all of that information or that, like, way of thinking about data still applies in Python. You still have the same problems in cleaning data. It doesn't matter if I give you a CSV, whether you pick R or Python, it really doesn't matter how you're going to analyze it, you're still gonna go through the general, same general steps. So in terms of R and Python, I would say really, it doesn't matter which one you pick, just pick one and learn it. Well. Um, but for me, I started with R and the way I started learning R was I think I learned how to load in data as like an R matrix and use one of, like, the matrix manipulations to like, turn a row of data into like something that looks like rows and columns. And that was the first thing I did in R was if I ever got data, because I was scraping something in Python and it just came back as like a row or like one vector of values. I would save that out to a CSV, load it into R, do my matrix manipulation. So I get a nice rectangular data set and write that back out and do the rest in like, Excel or something. So that's, that's what I did in R. And then slowly I learned other R functions, like, oh, instead of just transforming this data into something rectangular, I can do other things like make a calculation or make a new column based off of other columns. And so I slowly got more and more into R, just because I didn't want to start reading and writing out files over and over again. So that's how I slowly built up the knowledge on how to use R. And then you learn more and more. And then it becomes like, oh, I don't really need Excel anymore. Like, I can just do everything I need in R. And then from there, because I've been teaching with software carpentry, you learn, it's like a snowball effect. It gets really hard to start for a lot of people just because it is scary, especially if you don't have someone showing you how to do something. But then once you get started, it's very easy to learn that one little thing to extend your knowledge. And then you build on that. You build on that until you pretty much know how to, like, do whatever you need. You can wrangle data to whatever format you need. And yeah, so that's how I pretty much learned R. And then from there it became a problem of, oh, I did. I learned how to, like, scrape data from the web in Python, even though you can do it in R. I learned how to do it using Beautiful Soup. And so then the problem was, oh, I scraped all this data using Beautiful Soup, but I only know how to analyze data in R. And then you learn about this package called Pandas. And then the same process happened. I learned how to load something in Pandas, maybe do this one thing that I know that has, like, some correspondence to how I do it in R. And then I'll write it out and then load the rest and do all my analysis in R. And then slowly it became, well, the same issue. Instead of just jumping back and forth between languages, I just slowly started learning how to do more and more data stuff in Python. And then so that's pretty much how I learned R in Python. So you can do. You can come at it the same exact way, just the opposite way, really. Like, learn that one bit in Python, work at it until you know enough, you're gonna come to some use case like In R, like my use case, I would say Shiny is pretty cool. As of yet, there isn't something as simple and almost one click, but it's many clicks and typing like shiny. So maybe that'll be like the use case for people coming from Python. They have this data set, they want to manipulate it and they have this nice data set. Now they want to create a website for people to interact with. And so maybe that'll be the way to drag you into the R world. And this day and age, it doesn't hurt to know more than one language.

I agree completely. A Pythonista would respond, maybe you'd want to use Flask or Django and Bokeh for example, instead of going to the R ecosystem. I may disagree with that, but this isn't really about my opinions. You mentioned that if someone came to you and said should I learn R or Python? Your response generally is pick one language and run with it and learn it. Well, yeah, my question is, are there qualities in this. Let's say a student came and asked you this, Are there particular qualities or things they'd want to do that would make you go actually use Python? Like if they said I want to embed this in a whole bunch of web apps, do full stack development, do web scraping, would you then maybe say Python? Or if they said I want to focus on statistics and data viz, would you then direct them to R?

And what other qualities would I mean for me? I guess if you're coming from a computer science background, chances are you already learned Python. Just because you're learning, you're going down like a computer science major. Pandas would probably make a lot more sense just because then you don't have to worry about learning the syntax of R, because there are some weird quirks in R. Then again, at the same time, if you're coming from a statistics background, more and more statistics program programs, after you take your intro, SAS or Jump or whatever those other stats programs, more and more professors are teaching R in their classes. So in that respect you might just know R for general computing purposes in R, in which case you already know how to fit models because you're being taught how to do this in a stats program. So it doesn't hurt to just continuing to continue to go down the R route in that respect.

So the R ecosystem has rstudio in particular has something very clever where they've abstracted over the shell the terminal command line, right? So you can get statisticians or biologists or social scientists jumping into rstudio and not Having to do any command line.

Right.

If that happened, let's say you could open notebooks without having to use the command line at all. Do you think that would actually open up the Python ecosystem to a lot more people?

Yeah, I mean, I mean, like whatever.

Scary, right?

Command line is scary. And I guess this ties back into software carpentry, if you ever look at any of their material.

Definitely do. All data campers out there check out software carpentry and data carpentry.

Yeah. So during the Software data carpentry workshops, the first thing that we teach in the workshop is Bash, because we know that the terminal, the blinking cursor, if you've never communicated with a computer that way, it is really scary. But at the same time it's really important because we're going into this big data world where your data sets are not going to fit on your laptop. The only way you're going to be able to work on these data sets are on remote servers. So you have to know Bash or some terminal application just to be able to communicate with your data. And then also there's little things like how do you load in data sets like the dot dot just to signify one folder up. That's a bash thing. So that's sort of why the terminal is really scary. And you kind of have to know it, but you don't have to know too much of it. Most of the terminal commands you need are just, how do I get to a folder and how do I run a program?

For sure.

But I definitely feel like, yeah, once Python becomes one of those, like double click, you get a nice RStudio environment. I mean, there are pretty good Python IDEs like rodeo, run by Y hat and Spyder, which is open source ships with Anaconda, those are also like, you know, if you're home by yourself and you're really afraid, like, how do I run this notebook? Because it requires some terminal thing. Look at some of the ides that ship with Anaconda or something. And I think with Anaconda, so I'm also primarily on Linux, so I don't get to see what happens in Windows and Mac. But I think if you install Anaconda, you get a nice launcher that lets you double click. So you can open a notebook without running the notebook thing in a terminal.

That's right.

You can run all this stuff straight from your applications toolbar. So it's nice that the barrier for entry for everything in general is just going down. So, yeah, to go back to your point, yeah. Does something like Python similar to what RStudio has done exists. I would say Anaconda pretty much is lowering that bar already. You just get a different environment like notebooks versus like a four panel ide. So. Absolutely.

So we've, we've navigated quite, quite beautifully around this idea of data science, which is what we're talking about constantly. And I hesitate because data science doesn't have any. I mean, it's a vague, it's a vague term, it's a buzzword in a lot of ways. So maybe you could speak to what it, what it means to you personally and professionally and what your interest is.

So I guess, yeah. What is a data scientist? If you ever look at like job applications, they go from something, anything between system administrators to statisticians to database managers, et cetera. Um, but I guess just my views personally, you pretty much are the system administrator, database manager, slash analyst and statistician. Well, or at least you should be able to speak basic, the basic lingo to be able to communicate between one group and another if, if you're in a corporate environment or something. Databases are their own separate beast. They have people who write how databases should look. That may or may not be your job, but you should still know that there is a database. That's where all your data exists. How do you get this data? That is your job as a data scientist. From a statistician's point of view, at the end goal, whatever company you or somebody, or even if you're just playing with data on your own, you're trying to answer a question and that requires some kind of statistical knowledge. You're trying to fit a model just so you can make some decision or figure out something about the data you're working with. Does that mean that you need to know every single statistical test out there? No. As long as you can get most of the way done, you'll understand your data like this course. By the time you're done tidying your data, you should have a good sense of what's in your data. You've seen this for so long, you, you've managed to understand what each column means. You should be able to formulate like a pretty simple, or not even simple. You should be able to form like any question, given the data that you're gi. That you have. So you know, whether you decide to run like a linear regression, for example, or a T test, you should be able to at least understand the basic statistics. Uh, but in no way or shape or form are you required to understand the entire realm of statistics. That's kind of unheard of unless you're like a doctoral or a professor in statistics or something. But what's great about R. Python and pretty much the world of open source and open science is that a lot of this stuff you can find online or if you go to, if you're lucky enough to be in close to some kind of large city, like there are meetups everywhere where you can learn that brand new statistical technique and you know, you can apply it to your work. And you know, again, you've started with some core of knowledge and you expanded it a little bit more and then, you know, you snowball and you learn more and more and eventually you'll learn a bunch of statistical techniques which, you know, maybe you won't know that esoteric one for that weird data set. Then, okay, you'll ask the statistician, whoever you're working with. But you can get pretty much most of your work done. At the very least, you'll be able to communicate your problem to someone else. And if anything, that is your job as a data scientist, or your job as a scientist in general or someone working in general. You should be able to communicate your ideas or your problems to someone else, whether they're in your field or not. Communicate your problems, communicate your ideas, communicate your findings. So that's pretty much what I think a data scientist is. I mean, it has the word scientist in there. You should probably behave like a scientist and communicate results or explain what your problem is.

Yeah, I mean, creating science. Science doesn't exist in a head. Right. It exists in a method of communication and communal knowledge.

Right.

And communal wisdom. That's really, really interesting. We have a lot of aspiring data scientists who come to data camp to learn how to be data scientists. A lot of people transitioning from other positions, a lot of working professionals, and we get the question a lot. If I want to kind of get myself on a path to being a data scientist, what are the best things I can be doing? And maybe you can speak to that.

So definitely, if you're watching this video, definitely continue watching whatever video you have next queued up. Like, we have already mentioned the software carpentry, data carpentry stuff. Those are all Creative Commons and they'll teach you the core skills that you would know what other things you would do. Me personally, I listen to a lot of podcasts, so there is. Should I list them?

Sure.

Off the top of my head.

Absolutely. Whatever springs to mind.

So we have not. So standard deviations. That's a great one. Yeah. So that one's done by. Oh God, I'm gonna Roger. Roger Peng and Hillary Parker. So that's more just.

I'm glad we got that right.

Yeah. So that's just more. I got a sticker from Hillary too that was signed. So there's no reason why I should kind of, I guess when things are recorded you get really nervous about whether he's safe or not. Sure. So that's just more about what's going on in the data science field in general. And it's just like a nice talk, like what we're doing. About what, what's going on in data science. What's the new package that has an rpun. So, so there's that. Some of the more technical ones that I listen to, there's Partially derivative, I should probably. There's a bunch of others. Partially Derivative is a nice one where it's three people, they're all usually drinking beer, talking about something data science related.

Fantastic.

It's kind of nice. The other one that I would like that's more technical is there's one called Data Skeptic. And what I like about those is they have like these, they have general discussions about what the latest thing going on in data science is as well. But they have like these mini series which is just like a small five minute burst about what is a P value or what is unsupervised learning or what is a random forest stuff like that. Which if you're demystifying. Yeah. So me as I come from the applied side of things. So for me, just listening to those mini series or something just to get a high level overview of what some algorithm is doing means a lot more to me because I don't have the proper math background to read the mathematical proof of what's going on. And just understanding what's going on if you end up looking or coming across the math or just applying it in your day to day, just having an overview of what's going on, it's probably in my opinion because I come from the applied side better than just knowing the nitty gritty math stuff.

Absolutely.

So I guess those are the three podcasts that I can list off the top of my head that I pretty much listen to.

That's great advice. And you also mentioned previously, try to make it to meetups as well. Which.

Yeah, so the meetups, if you can, will. I mean, one, you'll get in touch with people in your local area, two, you'll pretty much learn something you've never learned before or, or you'll see a technique that you've never heard before that's really relevant to the problem you're having. And so it's. And usually at these things there's food or drink involved.

And recruiters also.

And recruiters. Yeah. You definitely want a job hunt if.

You'Re on a job hunt.

Yeah, that's. Yeah, if you're definitely on a job hunt. A lot of these meetups, the first thing they do at the beginning is who has a job that they're looking for people and they'll talk about the job openings they have. And then you meet the recruiters or sometimes like even the CEO of a company or at a startup, they're there looking for people. And a lot of these, I would even say you don't have to be like a full blown data scientist to one get hired, but you definitely don't need to be a full blown data scientist. Talk to these people. I mean, you have the CEO of a startup, it really, you know, after maybe down a couple of beers, you'll have enough liquid courage, go up to them and like talk to them. Like what, what types of problems they're working on? Even ask them, like, you have your inspiring data scientist. Where do you want to start? You know, a lot of these people who go to meetups are more than happy to share information. It's sort of the nature of why are you attending this, this like three hour block of time out at the end of a workday. I mean, like the people who go to these things are all of the same mind. So if you're a really, if you're like an aspiring data scientist and you want to learn or learn more, there's going to be someone on the other side who knows a lot and just wants to find people who are as motivated to spend three hours at the end of a day to learn something and work for them. So it works both ways. And meetups are a great opportunity to learn and get hired.

Absolutely. All right, thanks, Dan. It's been an absolute pleasure. Yep, we'll see you guys in the course.