R, Data Science, & Computational Biology

As far as a first language goes, it doesn't really matter if you're coming in from a data science point of view, what's the most important thing is when you see a data set that is quote unquote messy, can you like in your head write the general sequence of steps to make it clean again? I borrow all the terminology from the R world, which is like the concept of tidy data. And so if you can see a data set and know the steps on making it tidy, at that point it really doesn't matter what language you use because you can literally just look up like in the R world now in Tidyverse, it's like pivot longer or wider. So you would just Google like pivot longer, wider tidy R and then on the Python side it'll be like pivot longer wider like python. One of those words will show up some search result. And I think that's probably the more important thing is just knowing the steps on processing data and then just treating programming as the thing to get get you there. Bandwidth for Change log is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out@roar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean Droplets. Manage Kubernetes, Manage databases, spaces, object storage, volume block storage, advanced networking like virtual private clouds and cloud firewalls. Develop tooling with a robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150000 businesses that develop, manage and scale their applications with DigitalOcean. Head to dot co changelog to get started with a $100 credit. Again dot co changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@change dog.com community and follow us on Twitter. We're @practicalaifm.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How you doing Chris?

I'm doing very well today, Daniel. How's it going?

It's going great. Over the weekend on Saturday morning my time I gave A workshop at ODSC Europe Open Data Science Conference Europe. And that was a good time. Virtual conferences are kind of fun because I get to connect. There were people joining from all over the world, so that was cool to get people joining into the workshop from all over and get to discuss some fun things with them. We did some transfer learning and reinforcement learning and Gans with TensorFlow, which was fun. Transfer learning is very much the bedrock of a lot of, you know, industry work.

Sure is.

Reinforcement learning in GANS is like, for me, it's. It's a really fun topic to play around with and have some fun with. It definitely has some practical application, but it's just kind of fun to get into. So that was a good time. It was a fun weekend in that sense. What about yourself?

Well, over the weekend just enjoying cooler weather. It's been nice. We're getting into, you know, about to get into the fall here and that was pretty pleasant last week. I know that we may talk a little bit about public orientation, government type stuff potentially today in terms of conferences and stuff. And was doing some work with the Air Force. It's amazing to see, you know, government, you know, following industry into advanced technologies, things that support AI technology, doing software development better. That was the last week for me and it's just, just a pleasure.

Yeah, yeah. Cool. Yeah, it's definitely good to get some cooler weather. I'm sure for you too. I can't imagine we've mentioned this before on the podcast. I just don't know. I don't know if I could survive Georgia and.

I struggle with chilling outside right now.

Yeah, it's awesome.

It's amazing when you get through a Georgia summer which is all very humid and hot, and you get to this first inkling of spring and everything is.

Like, ah, yeah, yeah, yeah. Well, as you mentioned, in talking about sort of government and that vertical, which I know you're involved a lot in, recently got connected to the R conference which I've spoken at before, the New York City version, which was a lot of fun. I think that was two or three years ago. I was, I was there, but now they do, of course they're doing virtual conferences this year. I attended the one online recently was super high quality and a lot of fun. They had a great system for it, but they're doing several of these that are related to different verticals. There's going to be one coming up that is related to government and related sectors and what used to be the sort of DC conference but is now, of course virtual and got connected to that. So we're actually going to, as the podcast, we're going to kind of do a media partnership with that conference and that's going to be really cool. And as part of that, they helped arrange today for us to have a conversation with Daniel Chen, who's with us, who is a Lander analytics data scientist. He's a PhD candidate as well at Virginia Tech and a former RStudio intern. So welcome Daniel. It's great to, great to have you on the show.

Hi, thanks for having me.

Yeah, we're going to have to navigate this to Daniel situation. Chris, Best of luck to you.

If it makes it easier, you can call me Dan.

Yeah, I may still respond to that accidentally, but we'll navigate this. I think we'll get through. I don't know about you guys, but it always bothered me when there's two people named the same. Sometimes people want to call that Daniel squared or something, but that always really bothered me because it seems to me to be actually two Daniel, not Daniel squared. Like there's two of them. That's two Daniel. It's not like I'm multiplied by Daniel Chen.

You're not multiplied by yourself there, right?

I, I don't know if that's just my own peculiarity.

No, you know, my name's Chris. I had, there were Chris's all over the place growing up. So I, I, I understand you feel like you're, you're melting into the background when there's like five Chris's in your class. I get it.

Yeah. Well, on that note, Daniel, Daniel Chen, if you want to just let us know a little bit about yourself, about your background, how you got into doing what you're doing now, the path that you took to there, I'd love to hear a little bit about it.

Yeah. So I grew up in New York City and I guess I've, my dad is a software engineer. My parents, when they came to America, they both studied computer science in college. My mom doesn't do computer science related stuff anymore, but I've always had a computer at home since, you know, when I was a kid. So like back in the day these were just like old company, hand me downs. And around high school I went to one of the math and science high schools in New York City. And it was interesting because sophomore year it was a requirement for all the sophomores to take one semester of computer science and one semester of technical drawing. And then the people who like computer science at the time could take the AP class, et cetera, et cetera but it was interesting the fact that every single sophomore student in high school was exposed to programming in some way, shape or form.

Yeah, that's super interesting.

Yeah. We actually looking back on it, I have no idea how the instructors handled got all the material in because we covered in one semester. We first went through netlogo, which is drawing turtles, and I made a little blackjack game for that small project. Then we went through Scheme, which is a Lisp language to talk about lists and functions. And then towards the end we got introduced to Python where it was like make your own prisoner dilemma kind of algorithm and will compete it in the class. And that was all in one semester. So as an educator I start teaching data science. Now I'm always baffled when I think back, how do they make that work? Because there's no way I would be able to teach all of that stuff even in a semester. So that was super interesting.

Yeah. Did it seem overwhelming at the time or did it just seem new and exciting stuff or different stuff?

Yeah, so it was new and exciting. But as I started teaching more, I didn't realize it at the time, but yes, there's always at the. Because of it was a math and science high school. Like clearly there are people who have done this stuff before in the past and then there are the people who like have seen this for the first time. And so I was in a camp of like I've actually never programmed before. But then there's all these kids who like knew the answer as the questions being written on the board and I'm staring at a blank like piece of paper like how do I do this? And so it was actually, that was like one of the like I don't think I'm ever going to do this for a living moment.

Yeah, it pushed you into that place rather than sort of further inspiration, at least at the time.

Yeah, well like I've always been interested about like just tech and things, but yeah, programming definitely at the time seemed this is not for me kind of ordeal. Yeah. And then fast forward a little bit to my undergraduate years. I ended up getting a computer science minor just because I was like, you know what, I'm just gonna go do it. Just learn how to program formally. And that's when I realized looking back the whole people who have seen it before versus not seen it before. My intro classes, they were relatively easy for me, even though it was like for example, the 1C class. I've never actually programmed in C before, but I didn't have to think about print statement debugging that's not a brand new concept at the time. Or if statements and loops are no longer something I need to struggle with because I've seen it before in the past. And then I actually felt bad for some of my students. I picked up my computer science minor, junior senior year of my undergraduate career and then I felt bad for like the freshmen coming in who were like, they've wanted computer science as like their degree but they've like, but they've never seen it before and like they actually struggled really hard. So like that's when I like had those, those feelings back in high school again.

Gives you empathy, doesn't it?

Yeah, yeah, yeah. That's when I actually started realizing like, hey wait, I've actually seen this before and that's why it's easy for me. So. And in some way that carried forward. So after that I got my master's in public health in epidemiology, which is somewhat relevant these days. And it was a two year program and the second year I ended up taking a Intro to Data science class with some of my MPH friends and that's where I met Jared. So that'll eventually tie in somehow. And so it was during that Intro to Data Science class where I sort of really understood what data science and what could you actually do. During the time when I was doing my master's we talked a lot about linear regression, logistic regression, survival analysis and all of the EPI concepts associated with that. But I never knew what random forest was or clustering and all of that stuff until I took the data science class. And that's when it was sort of like, oh, if you can just think of something, something already exists to make that happen in some way, shape or form. So it was really eye opening in that sense that whatever you can imagine, you can probably make it happen in some sort of. That was great. And then from my MPH, I entered my current PhD program. Fast forward like till today, since I started. I am now doing my dissertation topic on data science education in the medical and biomedical sciences.

So do you think that those, I mean it sounds like that those experiences of, you know, in high school when you were introduced to computer science and then when you're introduced and you kind of, your vision was expanded to see all these different methods and the possibilities later in your education, do you think that like pushed you to this specific interest in data science education or like what is it you feel about data science education that like, I know there's a lot of gaps out there and a lot to be addressed. But how did your specific interest in that develop and what are you kind of hoping to learn and contribute through what you're doing now and in your current PhD work?

Yeah, so I guess like, in terms of like pivotal moments in my life, it would definitely be taking that data science class during my master's program. And part of it was Jared was a inspiring teacher. Jared Kayer and Rachel, they taught the class and it was a very, actually it was a very difficult class, but like if you struggle through it, there was so much that you learned from it. And also during that class, it was the first time I attended a software carpentry workshop. And so those two things put together sort of put me on the road where I am now. So during that software carpentry workshop, So a little background of software carpentry, which is now the Carpentries, they are a non profit organization focused on teaching scientists the computing skills that they sort of never were learned, never taught. And so I attended that first workshop. I sort of knew a little bit of Python from like undergrad and high school years and had been sort of like playing around in Bash and git because for some weird reason I decided to install Linux on a computer where no one that I work with uses Linux. And so the stuff that they taught during their workshop were like all of those pieces. It was a little bit of Python, some Bash and some Git. And I thought to myself, like, hey, like I can actually do this. It's like not that much of a jump from what I currently know. So that's how I got into the education area. So like the following semester I signed up to be a carpentry's instructor. This was back in like 2014 or so. And that's where I met Greg Wilson, who was the instructor trainer at the time. He currently works at our studio. But that's sort of where I picked up all of the fundamental parts of like teaching this stuff. Didn't know that like this would actually turn into like a career or like a dissertation topic. But that's sort of when I realized or started thinking about like what makes a good teacher thinking about students, like how to convey topics in some coherent way for people who are new to this. And I did that for enough times or over a large enough period of time that I eventually wrote it all down into a book called Pandas for Everyone. And so that is my attempt of teaching Python from a data science perspective using Python.

Yeah, that's awesome. So you mentioned Jared a couple times. It's Jared Lander. He is very involved in the R world. So if you're listening and you're here in part of the R community, you probably know that name already. But he's also involved in the R conferences that I mentioned, like the one that's coming up later this fall, the podcast is involved with as well. And he actually was a previous guest on the podcast as well, all the way back on episode number seven, which seems like another age ago.

It does.

And, you know, I don't know if, as part of that data science class with him, Daniel, if this was part of it, but I remember him just talking this really great, sort of giving a really great overview of the landscape of, like, machine learning or AI techniques and, like, where certain things fit in and like, how to kind of orient yourself in terms of how, for example, deep learning fits into, like, the spectrum of other techniques. So that was very, very useful.

So I got a question for you, Dan, and it's something that really caught me. When you said it a few minutes ago. You were, you were referring to that first data science class when you were taking your master's as a pivotal moment for you. I'm kind of wondering. We have other students out there listening, and they're kind of trying to figure out where they want to go. What was it about that class that you found inspiring? You talked a little bit about the fact that if you'd get through it and when you could, you, you know, it would help you. But what was it that really grabbed you about that? What was it that you found beautiful about data science at that particular moment?

Yeah, so there's two parts to it. One was the people, and then the actual. And then second was the actual data science material. And so this was a. A class. So the people that you're interacting with are probably the going to be more important than anything else. And what I've also learned, this doesn't apply to actual Jared's case, but one of the things I learned over the years, like, what makes a good teacher doesn't necessarily mean you have to master the material. Like, being a good teacher is different from knowing the material, but it was sort of like the way the whole entire class was taught. Jared taught the technical lab component, and he was also a carpentry's instructor at the time. And so it was sort of that style of actually live coding in the class to go through the lab material. That was really good as a student to see because one like it just slows you down. Instead of flipping through slide decks, it literally will just slow you down and you see the like typo error process and stuff like that, which is a lot to take in when you're a student seeing it all for the first time. But I want to believe that subconsciously it does really help a lot. Just seeing the error process. And then Kayer, Patel and Rager shut, they taught like the general data science landscape portion of it. And that's where I learned about like, how does this apply to like everything else? Like there are so many techniques and methods outside of what I was learning in my epidemiology classes that I just didn't know existed. And so just learning about those methods and just understanding, or not really understanding at the time, but just seeing how they, what they are, how they work, just understanding the heuristics of how they function under the hood. I saw so much. It was eye opening for me just to see how this could just be applied in the health space. Granted, I was doing a master's, so a lot of the stuff that we were learning in the data science class, I believe that if I were to do a PhD in epidemiology, I would have seen some of that stuff eventually. But it was more just like I was doing a master's. I was. There was so much new information about a field already coming in and then you just threw in this analytics component and it was just like, whoa, we can just do this for everything. So it was sort of like that eye opening moment for me where it was just the teachers were great, so it kept me motivated. And then the material itself, I just was able to make so many more connections to what I was currently learning. And so that sort of just kept pushing me forward. Foreign.

Is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite web browser to changelog.com once again. That's changelog.com.

Changelog. It's better. Yeah.

So I was really interested to hear that you kind of, as you were going through that data science class, you saw kind of a new world open to you in terms of how these techniques could be applied specifically like in the medical space or in epidemiology, like you were talking about. Do you feel like that those communities now are aware of those methodologies and like data science and AI is really like taking a foothold in those industries, or do you still kind of see it as like, maybe a bit of an uncomfortable mixing right now and people like, still learning where things are being applied. What do you think is like, the kind of current state of those things and how do you see it progressing forward?

So it's definitely been more adopted in the medical space, especially with, like, deep learning stuff being so good at image recognition. Like, that's a prime case for, you know, looking at medical imaging. But it's tricky for other parts of medicine because a lot of what we learned in epidemiology courses and biostats courses is trying to do inference on our, like, data. And so EPI as a field is one way you can think about it is it is the field of setting up all of your observational experiments. So when you do the stats you have, you're a little bit more comfortable with like, what is actually like a cause and effect. And so if you take that part in mind, it gets a little tricky because there's so many machine learning methods that are really just black boxes that really don't give you what, like any sort of inference, it's really just made for a prediction. And so you have to be careful using these methods in a medical context if they are like these black box methods, because if it predicts something wrong, it becomes harder to figure out why did the model predict this wrong. And usually at the other end of this is someone's life on the line.

Yeah, the consequences are high.

Yeah. So, yes, there is a place for all of the, you know, AI ML stuff in medicine. And you just have to be more careful when you're trying to put a model into production, I guess, than like, you know, your regular company. I guess that's the other way to put it. At the end of the day, like in health, like the end of that model is going to affect someone's life versus, you know, some bottom line, I guess. Right.

And I imagine that that, that kind of ties into some of your feelings about, you know, good, good co and like the carpentry stuff that you were talking about as well in terms of understanding, like the implications of the code you're writing and how, how to test it and how to deal with, you know, debugging models and all of those things.

Yeah. So that the next part question or problem is like, not everyone was as fortunate as me. Like, I went into a public health program or a medically related program and then got thrown into data science and then went down that track. So a lot of people who are actually practitioners or physicians on the medical end when they want to do, like, research. They typically are just doing research from Excel sheets because that's what they know or that's what they went through school with doing. They weren't taught all of the techniques and methods and skills from, like, computer science or data science or just programming in general. And so, yeah, that's sort of where the carpentry stuff comes in, where, you know, now it's our time to teach all of the researchers, like, the skills that they haven't, like, actually formally learned. And they just went through their life, you know, patching stuff together because, you know, programming was the means to get their work done. And it was just, you know, they just had to program something or do some kind of analysis just to get the result that they needed. They just struggled with a tool because they never really had formal training. And so that's eventually how I came to my dissertation topic, which was I've been teaching for so long, you know, I read, like, education books, like, for fun, and I've always had this interest in the medical space. So find an advisor who will let me mash those two things together. And I got super lucky at tech. So my current advisor is Anne Brown. And yeah, I got super lucky just getting to meet her through the Virginia Tech library. So if you are a student, definitely go befriend a librarian. And, you know, because if you think about what the people in the libraries do, like they've been doing data science, like, since libraries were a thing.

That's a great point right there. So one thing I'm curious about is you have a lot of experience in both Python and R. You know, on the Python side, you wrote Pandas for everyone to share that learning. On the R side, you're giving a talk at the R conference focusing on government and public sector. And I'm wondering, those are two different tools within the data science toolkit, if you will. And how do you see those? At what point do you turn to R and say, that's, you know, the, the particular problem I'm trying to solve right now lends itself better to R in your view, versus when would you turn to Python? How do you. You hit. Since you have them both. And often those two communities, you know, people kind of do an either or, but, you know, for the benefit of someone who might want to consider both, how do you see that? Where. Where is each one stronger for you personally?

Yeah. So currently, like today, the way I pick the language is like, who am I working with? So if I'm working with my advisor, I'm probably Working Python. If I'm working with someone else who does R, I'll probably use R. That's today. If you're currently an R user and you go through my book, there was a tweet like a couple weeks ago that was actually like, this book is great if you're an R user, because, like, I make so many references to R things in the Python book. And it's not super explicit, but it's one of those, like, if you know, you know, kind of moments and what's actually, like, interesting these days or now, like, it really doesn't matter which language. If it's your first language, it doesn't matter. Eventually, like, you're going to end up learning both. Just, I almost feel like it's the nature of just doing data science.

It's the nature of programming.

It's the nature of programming lots of.

Languages for different things.

Yeah, yeah. And so as far as the first language goes, it doesn't really matter if you're coming in from a data science point of view. Like, I always make the distinctions between, like, data science and computer science. But if you're coming from a data science point of view, what's the most important thing is when you see a data set that is quote, unquote, messy, can you, like, in your head, write the general sequence of steps to make it clean again? And I borrow all the terminology from the R world, which is like the concept of tidy data. And so if you can see a data set and know the steps on making it tidy, then at that point it really doesn't matter what language you use, because you can literally just look up, like in the R world. Now in Tidyverse, it's like pivot longer or wider. So you would just Google, like, pivot longer, wider, tidy R. And then on the Python side it'll be like pivot longer, wider, like Python. But in Python it's melt and pivot. So one of those words will show up like some search result. And I think that's probably the more important thing is just like knowing the steps on processing data versus and then just treating programming as like the thing to get you there. Because if you struggle, if you're just starting off, you don't know the steps, you don't know, like the terminology or how to clean data. And then you're also trying to learn a brand new language. So when something goes wrong, you don't know if it's like the sequence, like the overall sequence was wrong or was it like an actual programming typo mistake. And that's what you want to separate as much as possible. So just pick one. Learn how to manipulate data. And once you're comfortable with that, it becomes super easy to transition to another language. Like, when I did my data science course in my master's program, it was actually all done in R. It was actually all done pre. Tidyverse was like formalized as a thing. But I worked with processing data for a good year or two, and then that's when I sort of when I actually understood what tidying data meant. That transition into Python was super easy. And then that's sort of why the ordering of the book that I put together. Or like, there was a lot of stuff in the book that was sort of like, I learned all this from my transition into R. And so that's why there's so many random R things in the Python book.

Yeah, that's awesome. And it sounds like we were talking a little bit before the show about kind of your personal data processing pipeline. Like, when any of us kind of go into a project, somehow we have to set up a set of, you know, scripts, programs, folders, files, config data sources, whatever that is, to. To define our project and the structure of, like, the pipeline that we're using. It sounded like that's something that you think about quite a bit. What, what is your. As you're kind of now also thinking about data science education, a lot of. What are your thoughts as far as when you're talking to students, when you're thinking about how to educate them around your project structure? What are some of the main things that you know really can benefit you as you set up a project, a new project, whether that be just something that's analytics or whether that be a machine learning project. What are some ways that you can help yourself down the line when you start out a project?

Yeah, so I am in academia, so there's three papers that like, sort of talk through this entire process. The first one that I read that sort of introduced me to all this is by William Noble. And the title is called A Quick Guide to Organizing Computational Biology Projects. And that was probably the first time I've seen in academic writing. Literally, how do you set up the folders in a project? So you have like an output or an output folder, you have like a scripts folder, you have a docs folder, you have a readme file in the top level, stuff like that. And then the two other papers that sort of like expand on this from the carpentries folks, is there's a paper called Best Practices for Scientific computing that was written in 2014 and then in 2017 there was another one that said, good enough practices for scientific computing. So you can see how like doing good or best practices is actually pretty difficult. But it really all does stem from one of the core pieces is really having a folder structure so that your scripts can find the data that you're working with. And it's focused around the idea of, yes, it works on my machine, but it needs to work on someone else's machine or another one of your machines or the cloud as well, without having to change a whole bunch of file paths. In an ideal world, it runs on your computer with a command and it will run on another computer with the same exact command without you having to change anything. So that's sort of the overall overview of what I focus a lot on. And then there's, you know, then there's like the super technical parts of like, yes, git is a thing. Version control is a thing that you have to know when you're trying to collaborate. That's just sort of the nature of the beast. The good thing is the Carpentries has a git lesson. So if you want to learn it on your own, it is written down somewhere. And I've this past summer put together a few workshops that are on the carpentry's like YouTube page on like the actual super like complicated collaboration aspects of using Git and GitHub. But yeah, so most of my stuff really does focus around, you know, project organization is the actual like cornerstone or centerpiece to managing like a project.

Yeah, it's interesting like you talked about the one paper being like best practices and then they went to like good enough practices that that concept definitely resonates with me. I was wondering your own. Because there is like software engineering sort of best practices in industry where like I think now, like if you're working on a project, you have a GitHub repo, if it's not connected to some sort of CICD and you don't have some of sort, some sort of portable way of deploying this thing, maybe with Docker or something like that, that's kind of like what people are doing a lot. But that's like a, that's a lot of things for a, like someone in academia or like a new data scientist to learn, it can be rather burdensome. So like, yeah, I guess, daunting. Yeah, daunting is a good word. So like, in terms of like people that are starting out as data scientists, do you think that's something like, as they're embedded in an organization. Should, should they sort of strive for, you know, eventually kind of learning all of those software engineering best practices and like adding that to their workflow. Or do you think there is a sort of in between where you know, the workflow of a data scientist is, it is different.

Right.

There's different data concerns and all of those things. So how much of a software engineer does a data scientist have to be? I guess is the end question that I'm going for.

Yeah, so that's the other big dilemma is a lot of workflows from data science are actually like anti patterns from software engineering.

Right.

As a data scientist we primarily work in scripts that execute from top to bottom. Very rarely do we end up writing classes or things like using those software engineering tools in a data science analysis. We will write functions that's good, but we don't necessarily create packages. That is considered maybe a best practice, but it's a lot more stuff. So just writing a function is good enough. But then what happens when you have 50 functions? There's this tension between, well not tension but there, there is like the way you program things and from a data science perspective is going to be different from software engineering that's just going to happen. It's kind of interesting when you hear stories about data scientists working with the engineers and then like when their code bases need to mesh and that becomes a different question and problem on its own. But from at least from what I'm working on now, which is the data science perspective, but it's like catered towards the biomedical sciences and those people, we even need to go like an even step further back from that, from like we're thinking about best practices and that stuff because these are the people who are so new to this field that if you talk about like Docker and CICD integrations like those are like letters that they've never seen put together before in that order. And so one of the I guess byproducts of my dissertation is this, I guess you can call it a book slash lesson plan that's called DS4BioMed. So data science for the biomedical sciences. Literally the first thing I talk about is we're just going to talk about spreadsheets for now because it is probably something that they're most familiar with in terms of a data perspective. Spreadsheets are one way you can think about spreadsheets. It is a GUI for your data set. And so people like looking at things and being able to click on things. And so how do we go from Spreadsheets to data Science Pipeline is sort of where I'm focusing more of my time these days. And so, yeah, I just finished the first spreadsheet module, so I can actually talk about this. And putting that part together, I sort of realized that, yes, we can actually introduce those tidy data concepts in the spreadsheet section, which is like, if you've ever loaded up an Excel sheet. First of all, as a data scientist, when I see an Excel sheet, I'm already preparing myself over if I get a CSV file. And so why do I cringe when I see an Excel file? Well, it's because sometimes we have multiple tables like in the same sheet, and it's like from A to M is like one table and from like P to Z is like another table and you have to load those tables separately. Like those are like data issues that happen when you're loading in data into R, Python or whatever language. But from a lot of people who don't actually work with programming languages, that's great. They get to see everything at the same time. So it's sort of like identifying those bad habits and trying to show them why they're not conducive if you want to load them into a programming language is sort of like, that's where I'm at right now. And so it all also comes down to the whole mantra of you want to have empathy for the people who are learning this stuff. And if all they get away from that first workshop is structuring their spreadsheets better, I'll be okay with that. I'll be happy with that. So it's always about making like these small incremental improvements every time you start a new project. And that happens if you're like a full blown data scientist as well. Like, yeah, maybe you have the whole project structure thing working for you and you can have your computer, you know, all your code work on whatever machine that your code base is deployed on. What would be the next step for you that might be trying to learn like one of the continuous integration services or like using Docker or something. So there's always something that you can do to improve like your workflow. And I guess like that, that does take a lot of effort on like one's end because you do have to do a lot of like, introspection of like, what can be improved and like, I guess like the way I've always seen it, for me it was easier for me to do it because it's always like, oh, what is the best practice? And then I Read about it and it's like, that is way too complicated. And then like six months later it's like, that seems doable now because I've learned all the other stuff in the middle that gets me there. So there's different entry points towards picking up practices from software engineering. But at the end of the day, data science pipelines or workflows really don't mesh with software engineering stuff. In software engineering, your end product is probably a library or this big program thing, versus in data science, it's really this pipeline of scripts that create this model and then this model gets handed off to like the software engineers to implement somewhere else. So like those things are just going to be different and it makes sense that the best practices on both sides aren't going to be the same. But you know, if you make incremental progress, you know you'll eventually get to a good spot.

So you're really thinking a lot about trying to get people working in these areas, bioinformatics and other related areas, to think about using data science techniques. If you were to look into the future and let's say that you've accomplished your goals of getting these people to use these sorts of techniques in their workflows, what are some of the example things that you envision them being able to do with data science techniques that maybe they wouldn't have been able to do if they kind of followed the same workflows that they have been using for quite some time?

So one of the main takeaways would be just working with multiple sources of data at the same time. We have a system where, you know, every local department, organization, government at the government level, et cetera. They're doing reports of case counts, for example, on a daily basis. And they don't necessarily all come in as like one. They're not all combined together for you in this current pandemic? Yes, you can find data sources that are doing the aggregation for you. Back in 2014, during the Ebola outbreak, that wasn't necessarily the case. We were getting daily reports from different countries as like PDF files, for example. And so being able to work with multiple data sources is going to be one of those like, skills that are going to be super important. And how it all ties back into like, why use a data science approach? And when I say that like, why use a programming language to do that kind of analysis over something like spreadsheets is that goes into like one of the most important things when you're working with data is like, you always want to keep your raw Data completely intact. And so this way if there is an improvement or something in your actual data science code, in an ideal world, you just rerun your code over a new set of data and then you get your updated results right away. That's probably the most important like idea that we have even in like today's Covid world. That's sort of the reason why like you'll hear like recommend recommendations changing over the past couple of months is because in the beginning we weren't. The data itself didn't show conclusion. But as more data came in, if you were to rerun your analysis over and over again over new courses of data, you might actually find a new outcome. And so like currently we are in real time living the scientific process. And part of that process is, you know, making sure that if you do have new data sources coming in, you can still rerun your analysis and you know that part is reproducible. And then you know, as more data comes in, your conclusions may change. So that's sort of how that all ties into current times. But it really is something that is really just fundamental to data science as a whole since we're always querying data from the world and we want our pipeline to be there so that as new data comes in we can have an updated model at the other end.

That makes perfect sense. So we talked a little bit a while ago about the fact that you're doing a talk at the R conference and I was wondering if you could share a little bit about what your talk talking about and what is your message to the R community and give us a little insight into that. Would love to hear what's of interest to you.

So my previous R conference talks have always been around the topic about like this data pipelining part of data science. So like, you know, I think the last talk I gave last year was, you know, something around like I'm going to teach you how to make a make file so you can like make your reports super useful. Yeah, it is. So last year, so 2019, I was one of the interns at RStudio and I worked on a package called Grade this, which is the auto grader system for R code. So it can yes, tell you if like you're an instructor, like here's the correct answer and then the student can type in some R code and compare the results. That's like the easier way you can grade code, the more complicated way you can grade code is like looking at the code itself. So like in more technical terms it's looking at the abstract Syntax tree. And so you're literally comparing, like if the student put in, for example, log of three and your solution is log of two, you want a sentence that essentially says you put in three, where the answer should have been two. So creating that sentence is a lot more complicated than it may or may not seem if you didn't think it was a hard problem. So during that process I learned a lot about R's way of handling code expressions. And so this year I am trying to teach that to regular people. How that ties in to the general, to the greater R ecosystem is if you've worked with tidyverse packages, you'll notice that you are allowed to pass in like column names without having them like quoted in strings, but they look like regular variable names, which is like terrifying from like a Python user's perspective because lazy eval is not a thing in the Python world, but it's like there in the R world. So it's sort of trying to introduce those topics is sort of my goal for like the next series of talks. Why does this help? This is sort of the transition of if you want to write your own tidyverse compatible packages for your own work, this is sort of like what you need to know to make that happen. So yes, the talk is more towards like the software engineering side of things, but it's one of those, like, hey, if you want to have your own work plug into this whole ecosystem, like, how would you go about doing it? So this is my part of trying to make an incremental improvement for myself and for the greater community.

Do you think as you dug into the underlying mechanics of how R processes expressions, do you think that's influenced how you write your R in terms of just your general programming? And like, has it made you more sympathetic in terms of like how you, how you write your R with that kind of better underlying understanding?

So as far as like, like regular day to day, like if you were to just tell me to run some type of analysis on a data set now, it doesn't affect that part of it. If anything, I have a lot more sympathy for the people who develop these packages. Like last year I literally read the advanced art book like three times over and over again without understanding like what I was doing during my internship. And then like after seeing it and having it like mesh in my brain for like a year, like I'm reading it like again, rereading it again, and it like makes total sense now. So as far as like a day to day thing, it doesn't affect it that much. But when I am writing functions and things that might need to end up like a collection of functions, if I start writing a collection of functions, whether they make it into a package or not, I am more mindful of certain things. Mainly around like dependencies is what I'm really mindful about. One of the things that sort of surprised me last year was, you know, if I wanted to, you know, do like some kind of grep search for a string in tidyverse world, I would just instinctively use something from the string R package or something like that. But you don't need the entire string R dependency, which if all you're doing is a simple grep call, just use the regular built in grep L, it's fine. I've sort of realized that yes, when you are a package developer, all of the engineering hurdles are now like your problem. Like your, your job is to make like the end user's life easy and then you deal with all of the engineering burden on your end. And so I definitely appreciate that a lot more. But like, as far as like my day to day, it's mainly just like, you know, just write more functions and like, try to keep working on like the best practices and stuff like that. And then when I have a new student that I'm working with, that sort of like grounds me back, like, okay, this is where I once was. And then so how do I get them to like some other point, the next like level in their life and programming, like, how do I make that transition, like less violent for them?

Yeah, no, I, I, I really appreciate what, what you just said actually about dependencies. Like, you can reduce, I don't know what the right word is, I guess your liability or your, your potential debugging issues in the future. If you only need for like, I've, I've done this sometimes where it's like, oh, I need like a sigmoid function or something like, well, I could like import any number of packages where I could call like sigmoid parentheses, like give my thing, but I could also just write that in a couple lines of code and just embed the function in my own code so that like it's super clear what's going on. I don't have an external dependency. I think that's something that like is underrated a lot. So I really appreciate you bringing that point to the surface and I'm super interested to hear like the other insights you have from your talk at the R conference. We'll definitely look forward to that and we'll also link in our show notes to a bunch of the things you've mentioned, the carpentry courses, your book, various packages you've mentioned, and all of that. We'll also link to Jared Lander's episode if people want to go back and listen to that. But yeah, we really appreciate you joining us. I think if I'm right, that people can find out more about the R Conference at rstats. AI is the correct website and we'll of course link to that as well. But yeah, it's been a huge pleasure to get to chat with you, Daniel. Looking forward to spending some more, more time together at the conference.

Yeah, thanks for having me. It's been really fun.

The R Conference is where our enthusiasts and data scientists gather to explore, share and inspire ideas. It's happening December 2nd through 4th and we have a discount code for you if you're interested. Use code practical AI20. That's good for 20% off every ticket type, including the conference and all workshops. This episode was hosted by Daniel Whitenack with Chris Benson. Our special guest, Daniel Chen. Thanks for coming on the show, Daniel. Our music is produced by the one and only Breakmaster Cylinder. And we are brought to you by amazing sponsors. Thanks again to Fastly, Linode and Rollbar for their continued support. That's all for now. We'll talk to you again next week.

Sam.